Kubeflow Community Distribution

This repository helps you install the Kubeflow Platform on popular Kubernetes clusters such as KinD, Minikube, Rancher, EKS, AKS, and GKE. The manifests include all Kubeflow components (Pipelines, KServe, etc.), the Kubeflow Central Dashboard, and other applications that comprise the Kubeflow Platform. The platform is used and adapted around the world by small and large enterprises with strict legal, security, and multi-tenancy requirements for machine learning platforms. It also helps academic users explore the end-to-end capabilities of the Kubeflow Platform and contribute to it.

For a stable and conservative experience, we recommend using the latest stable release. However, please consult the more up-to-date documentation in the master branch. You can also install the master branch of kubeflow/community-distribution by following the instructions here and providing us with feedback. Our continuous integration workflow runs end-to-end installation and tests on pull requests and pushes to the master branch in the upstream kubeflow/community-distribution repository; see the full Kubeflow integration test workflow for details.

We plan to publish two releases per year, for example 26.03 and 26.10, before the corresponding KubeCon events in Europe and North America.

As a community, we make a best effort to support each release for six months. There is commercial support available if needed.

Table of Contents

• Overview of the Kubeflow Platform
• Kubeflow Component Versions
• Installation
  • Prerequisites
  • Install with a Single Command
  • Install Individual Components
  • Connect to Your Kubeflow Cluster
  • Change Default User Name
  • Change Default User Password
• Upgrading and Extending
  • Version-Specific Upgrade Notes
• Release Process
• Security
• Pre-commit Hooks
• Architecture
• Frequently Asked Questions

Overview of the Kubeflow Platform

• This repository is owned by the Community Distribution Maintainers.
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website.
• Our channel on the CNCF Slack is #kubeflow-community-distribution.
• You can also find our biweekly meetings, including the commentable Agenda.
• If you want to contribute, please take a look at the CONTRIBUTING.md.

The Kubeflow Platform/Manifests repository organizes installation manifests into three main directories:

Directory	Purpose
applications	Kubeflow's official components, maintained by the respective Kubeflow WGs
common	Common services, maintained by the Manifests WG
experimental	Third-party integrations and platform experiments (e.g., Ray or security improvements)

All components are deployable with kustomize. In addition, experimental Helm charts for selected components are available under experimental/helm/. You can choose to deploy the entire Kubeflow platform or individual components.

Kubeflow Component Versions

Kubeflow Version: Master

This repository periodically synchronizes all official Kubeflow components from the respective upstream repositories. The following matrix shows the Git revision included for each component and its approximate resource requirements. CPU and memory figures are the maximum of actual usage and configured requests; storage figures reflect PVC requirements:

Component	Local Manifests Path	Upstream Revision	CPU (millicores)	Memory (Mi)	PVC Storage (GB)
Trainer	applications/trainer/upstream	v2.4.0	8m	143Mi	0GB
Kubeflow Notebooks	applications/notebooks-v1/upstream/	v1.11.1	43m	806Mi	0GB
Kubeflow Dashboard	applications/dashboard/upstream/	v2.0.1	10m	302Mi	0GB
Katib	applications/katib/upstream	v0.19.0	13m	476Mi	10GB
KServe UI	applications/kserve/kserve-ui	v1.1.0	6m	259Mi	0GB
KServe	applications/kserve/kserve	v0.21.0	600m	1200Mi	0GB
Kubeflow Pipelines	applications/pipeline/upstream	2.18.0	970m	3552Mi	35GB
Kubeflow Hub	applications/hub/upstream	v0.3.18	510m	2112Mi	20GB
Spark Operator	applications/spark/spark-operator	2.5.2	9m	41Mi	0GB
Istio	common/istio	1.31.3	750m	2364Mi	0GB
Knative	common/knative/knative-serving common/knative/knative-eventing	v1.24.0 v1.24.0	1450m	1038Mi	0GB
Cert Manager	common/cert-manager	1.21.2	3m	128Mi	0GB
Dex	common/dex	2.46.1	3m	27Mi	0GB
OAuth2-Proxy	common/oauth2-proxy	7.15.5	3m	27Mi	0GB
Total			4377m	12316Mi	65GB

Installation

This section covers the installation from scratch. For the in-place upgrade guide, please jump to the Upgrading and Extending section.

Although our master branch has extended automated tests and is very stable with trunk-based development, please consider using a stable release tag/branch for a more conservative experience.

We provide two options for installing the official Kubeflow components and common services with Kustomize. The aim is to simplify installation and help users build Kubeflow distributions by adapting the manifests:

1. Single-command installation of all components under applications and common
2. Multi-command, individual component installation for applications and common

The example directory contains the Kustomize configuration used by the single-command installation.

:warning: In both options, we use a default email ([email protected]) and password (12341234). For any production Kubeflow deployment, you should change the default password by following the relevant section.

Prerequisites

• For the specific Kubernetes version per release, consult the release notes.
• The KinD script below installs the Kubernetes, Kustomize, and kubectl versions tested in continuous integration.
• We use KinD by default but also support Minikube, Rancher, EKS, AKS, and GKE. GKE may require the adjustments documented here; OpenShift is also an option.

ARM64 / aarch64 note

Kubeflow on ARM64/aarch64 may not be fully supported yet because some OCI images might not be available for linux/arm64. If you hit image pull errors such as “no matching manifest for linux/arm64”, please track/report details in kubeflow/community-distribution#2745 and take a look at the Google Summer of Code project for Kubeflow on ARM64.

---

NOTE

kubectl apply commands may fail on the first try. This is inherent in how Kubernetes and kubectl work (e.g., CR must be created after CRD becomes ready). The solution is to simply re-run the command until it succeeds. For the single-line command, we have included a bash one-liner to retry the command.

---

Install with a Single Command

Prerequisites

• 16+ GB of RAM recommended.
• 8 CPU cores recommended.
• You can exclude components from the example/kustomization.yaml to fit Kubeflow into 4-8 GB of memory and 2-4 CPU cores.
• kind, kubectl and kustomize from ./tests/install_KinD_create_KinD_cluster_install_kustomize.sh
• podman (better) or docker to run the OCI images for the Kind cluster.
• Linux kernel subsystem changes to support many pods:
  • sudo sysctl fs.inotify.max_user_instances=2280
  • sudo sysctl fs.inotify.max_user_watches=1255360

Create Kind Cluster

./tests/install_KinD_create_KinD_cluster_install_kustomize.sh

Save Kubeconfig

kind get kubeconfig --name kubeflow > /tmp/kubeflow-config
export KUBECONFIG=/tmp/kubeflow-config

Optionally create a Secret based on existing credentials if you run into Docker Hub pull limits

docker login
kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=$HOME/.docker/config.json \
    --type=kubernetes.io/dockerconfigjson

You can install all Kubeflow official components (residing under applications) and all common services (residing under common) using the following command:

while ! kustomize build example | kubectl apply --server-side --force-conflicts -f -; do echo "Retrying to apply resources"; sleep 20; done

Once everything is installed successfully, you can access the Kubeflow Central Dashboard by logging in to your cluster.

Congratulations! You can now start experimenting and running your end-to-end ML workflows with Kubeflow.

Install Individual Components

In this section, we will install each Kubeflow official component (under applications) and each common service (under common) separately, using just kubectl and kustomize. This can help you a lot when debugging your installation or if you are encountering installation problems.

Executing all the following commands produces the same result as the single-command installation above. The purpose of this section is to:

• Provide a description of each component and insight on how it gets installed.
• Enable the user or distribution owner to pick and choose only the components he needs.

Note: Many component installation steps below reference scripts from the tests/ directory. These scripts are the same ones used by our CI/CD integration test workflow, which ensures the documentation is continuously verified. The scripts also provide better error messages and include sanity checks.

Kubeflow Namespace

Create the namespaces where the Kubeflow components will reside. We are in the transition from kubeflow to kubeflow-system.

Install the Kubeflow namespace:

kustomize build common/kubeflow-namespace/base | kubectl apply -f -

Cert-manager

Cert-manager is used by many Kubeflow components to provide certificates for admission webhooks.

./tests/cert_manager_install.sh

Istio

Istio is used by most Kubeflow components to secure their traffic, enforce network authorization, and implement routing policies. This installation uses Istio CNI, which eliminates the need for privileged init containers and improves compatibility with Pod Security Standards. If you use Cilium CNI on your cluster, you must configure it properly for Istio as shown here; otherwise, you will encounter RBAC access denied on the central dashboard.

Install Istio:

./tests/istio-cni_install.sh

For Google Kubernetes Engine (GKE), use the GKE-specific overlay instead:

kustomize build common/istio/istio-install/overlays/gke | kubectl apply -f -

Oauth2-proxy

The oauth2-proxy extends your Istio Ingress-Gateway capabilities to function as an OIDC client. It supports user sessions as well as proper token-based machine-to-machine authentication. Authorization which is completely different from authentication is handled via Kubernetes RBAC and Istio authorizationpolicies.

echo "Installing oauth2-proxy..."

# Only uncomment ONE of the following overlays, as they are mutually exclusive.
# See `common/oauth2-proxy/overlays/` for more options.

# OPTION 1: works on most clusters, does NOT allow K8s service account
#           tokens to be used from outside the cluster via the Istio ingress-gateway.
#
kustomize build common/oauth2-proxy/overlays/m2m-dex-only/ | kubectl apply -f -
kubectl wait --for=condition=Ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy

# Option 2: works on Kind, K3D, Rancher, GKE, and many other clusters with the proper configuration, and allows K8s service account tokens to be used
#           from outside the cluster via the Istio ingress-gateway. For example, for automation with GitHub Actions.
#           In the end, you need to patch the issuer and jwksUri fields in the request authentication resource in the istio-system namespace
#           as done in /common/oauth2-proxy/overlays/m2m-dex-and-kind/kustomization.yaml.
#           Please follow the guidelines in the section Upgrading and Extending below for patching.
#           curl --insecure -H "Authorization: Bearer `cat /var/run/secrets/kubernetes.io/serviceaccount/token`"  https://kubernetes.default/.well-known/openid-configuration
#           from a pod in the cluster should provide you with the issuer of your cluster.
#
#kustomize build common/oauth2-proxy/overlays/m2m-dex-and-kind/ | kubectl apply -f -
#kubectl wait --for=condition=Ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy
#kubectl wait --for=condition=Ready pod -l 'app.kubernetes.io/name=cluster-jwks-proxy' --timeout=180s -n istio-system

# OPTION 3: works on most EKS clusters with K8s service account
#           tokens to be used from outside the cluster via the Istio ingress-gateway.
#           You have to adjust AWS_REGION and CLUSTER_ID in common/oauth2-proxy/overlays/m2m-dex-and-eks/ first.
#
#kustomize build common/oauth2-proxy/overlays/m2m-dex-and-eks/ | kubectl apply -f -
#kubectl wait --for=condition=Ready pod -l 'app.kubernetes.io/name=oauth2-proxy' --timeout=180s -n oauth2-proxy

If and after you finish the installation with Kubernetes service account token support, you should be able to create and use the tokens:

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80
TOKEN="$(kubectl -n $KF_PROFILE_NAMESPACE create token default-editor)"
client = kfp.Client(host="http://localhost:8080/pipeline", existing_token=token)
curl -v "localhost:8080/jupyter/api/namespaces/${$KF_PROFILE_NAMESPACE}/notebooks" -H "Authorization: Bearer ${TOKEN}"

If you want to use OAuth2 Proxy without Dex and connect it directly to your own IDP, you can refer to this document. However, you can also keep Dex and extend it with connectors to your own IDP as explained in the Dex section below.

Kubeflow Istio Resources

Create the Kubeflow Gateway kubeflow-gateway and ClusterRole kubeflow-istio-admin.

Install Kubeflow Istio resources:

kustomize build common/istio/kubeflow-istio-resources/base | kubectl apply -f -

Multi-tenancy components (Kubeflow roles)

./tests/multi_tenancy_install.sh

Dex

Dex is an OpenID Connect (OIDC) identity provider with multiple authentication backends. In this default installation, it includes a static user with the email [email protected]. By default, the user's password is 12341234. For any production Kubeflow deployment, you should change the default password by following the relevant section.

Install Dex:

./tests/dex_install.sh

To connect to your desired identity providers (LDAP, GitHub, Google, Microsoft, OIDC, SAML, GitLab), please take a look at https://dexidp.io/docs/connectors/oidc/. We recommend using OIDC in general since it is compatible with most providers. For example, Azure in the following example. You need to modify https://github.com/kubeflow/community-distribution/blob/master/common/dex/overlays/oauth2-proxy/config-map.yaml and add some environment variables in https://github.com/kubeflow/community-distribution/blob/master/common/dex/base/deployment.yaml by adding a patch section in your main Kustomization file. For guidance, please check out Upgrading and Extending.

apiVersion: v1
kind: ConfigMap
metadata:
  name: dex
data:
  config.yaml: |
    issuer: https://$KUBEFLOW_INGRESS_URL/dex
    storage:
      type: kubernetes
      config:
        inCluster: true
    web:
      http: 0.0.0.0:5556
    logger:
      level: "debug"
      format: text
    oauth2:
      skipApprovalScreen: true
    enablePasswordDB: true
    #### WARNING: YOU SHOULD NOT USE THE DEFAULT STATIC PASSWORDS
    #### and patch /common/dex/base/dex-passwords.yaml in a Kustomize overlay or remove it
    staticPasswords:
    - email: [email protected]
      hashFromEnv: DEX_USER_PASSWORD
      username: user
      userID: "15841185641784"
    staticClients:
    # https://github.com/dexidp/dex/pull/1664
    - idEnv: OIDC_CLIENT_ID
      redirectURIs: ["/oauth2/callback"]
      name: 'Dex Login Application'
      secretEnv: OIDC_CLIENT_SECRET
    #### Here come the connectors to OIDC providers such as Azure, GCP, GitHub, GitLab, etc.
    #### Connector config values starting with a "$" will read from the environment.
    connectors:
    - type: oidc
      id: azure
      name: azure
      config:
        issuer: https://login.microsoftonline.com/$TENANT_ID/v2.0
        redirectURI: https://$KUBEFLOW_INGRESS_URL/dex/callback
        clientID: $AZURE_CLIENT_ID
        clientSecret: $AZURE_CLIENT_SECRET
        insecureSkipEmailVerified: true
        scopes:
        - openid
        - profile
        - email
        #- groups # groups might be used in the future

For Keycloak, we have rough guidelines in https://github.com/kubeflow/community-distribution/blob/master/common/dex/README.md.

Dashboard

./tests/dashboard_install.sh

Admission Webhook

The PodDefaults admission webhook is installed by the Dashboard step above.

Knative (used by KServe)

./tests/knative_serving_install.sh

Optionally, you can install Knative Eventing, which can be used for inference request logging:

./tests/knative_eventing_install.sh

KServe model serving and KServe UI

./tests/kserve_install.sh

The KServe control plane and the KServe Models Web Application are installed in the kserve namespace. User InferenceService resources remain in the profile namespaces, and the Models Web Application is still reached through the Kubeflow gateway at /kserve-endpoints/.

Kubeflow Pipelines

Kubeflow Pipelines offers two deployment options to choose from, each designed for different use cases and operational preferences. The traditional database-based approach stores pipeline definitions in an external database, while the Kubernetes native API mode leverages Kubernetes custom resources for pipeline definition storage and management.

The default artifact store is now SeaweedFS, as explained here. All overlays now set SeaweedFS as the default S3-compatible artifact store for Pipelines. Each overlay replaces minio-service to route S3 traffic to SeaweedFS and patches the Argo Workflow controller to use it.

Pipeline Definitions Stored in the Database

Install the Multi-User Kubeflow Pipelines official Kubeflow component:

./tests/pipelines_install.sh

This installs Argo with the runasnonroot Emissary executor. You remain responsible for assessing the security risks of containers running as root and deciding whether the main Kubeflow Pipelines containers should run without root privileges. We strongly recommend using restricted Pod Security Standards for all user-accessible OCI containers. KFP 2.16.0 and later can set runAsNonRoot, runAsUser, and runAsGroup through the SDK and the KFP API server configuration.

Pipeline Definitions Stored as Kubernetes Resources

Kubeflow Pipelines can be deployed in Kubernetes native API mode, which stores pipeline definitions as Kubernetes custom resources (Pipeline and PipelineVersion kinds) instead of using external storage. This mode provides better integration with Kubernetes native tooling and GitOps workflows.

kustomize build applications/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user-k8s-native | kubectl apply -f -

Using the KFP SDK with Kubernetes Native API Mode:

For detailed pipeline compilation instructions, please refer to the Kubeflow Pipelines compilation guide.

Differences in Kubernetes Native API Mode:

• Pipeline definitions are stored as Pipeline and PipelineVersion custom resources in Kubernetes.
• Pipeline validation is handled through Kubernetes admission webhooks.
• The REST API transparently handles the translation to Kubernetes API calls.

Benefits of Kubernetes Native Mode: This approach is ideal for organizations that prefer Kubernetes-native workflows and want to manage pipelines using standard Kubernetes tools and practices. Pipeline definitions can be managed through multiple interfaces: direct kubectl commands, the Kubeflow Pipelines REST API, and the KFP UI for user-friendly pipeline management.

Katib

./tests/katib_install.sh

Workspaces (Notebooks 2.0)

./tests/workspaces_install.sh

This installs the Workspaces controller, backend and frontend, which provide the Workspaces and Workspaces Administrator pages of the Central Dashboard.

Trainer

./tests/trainer_install.sh

Spark Operator

./tests/spark_install.sh

User Namespaces

Finally, create a new namespace for the default user (named kubeflow-user-example-com).

export KF_PROFILE=kubeflow-user-example-com
./tests/kubeflow_profile_install.sh

Connect to Your Kubeflow Cluster

After installation, it will take some time for all Pods to become ready. Ensure all Pods are ready before trying to connect; otherwise, you might encounter unexpected errors. To check that all Kubeflow-related Pods are ready, use the following commands:

kubectl get pods -n cert-manager
kubectl get pods -n istio-system
kubectl get pods -n auth
kubectl get pods -n oauth2-proxy
kubectl get pods -n knative-serving
kubectl get pods -n kubeflow
kubectl get pods -n kubeflow-user-example-com

Port-Forward

The default way of accessing Kubeflow is via port-forwarding. This enables you to get started quickly without imposing any requirements on your environment. Run the following to port-forward Istio's Ingress-Gateway to local port 8080:

kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80

After running the command, you can access the Kubeflow Central Dashboard by doing the following:

1. Open your browser and visit http://localhost:8080. You should see the Dex login screen.
2. Log in with the default user's credentials. The default email address is [email protected], and the default password is 12341234.

NodePort / LoadBalancer / Ingress

To connect to Kubeflow using NodePort / LoadBalancer / Ingress, you need to set up HTTPS. The reason is that many of our web applications (e.g., Tensorboard Web Application, Jupyter Web Application, Katib UI) use Secure Cookies, so accessing Kubeflow with HTTP over a non-localhost domain does not work.

Exposing your Kubeflow cluster with proper HTTPS is a straightforward process but depends on your environment. You can expose the istio-ingressgateway service in the istio-system namespace via nginx-ingress or any other ingress provider. For security reasons, only use ClusterIP on the service, not NodePort or something similarly dangerous. There is third-party commercial support available.

---

NOTE

If you absolutely need to expose Kubeflow over HTTP, you can disable the Secure Cookies feature by setting the APP_SECURE_COOKIES environment variable to false in every relevant web application. This is not recommended, as it poses security risks.

---

Change Default User Name

For security-sensitive installations, replace the default Kubeflow username and email address before deployment. To configure these values for the default user:

1. Edit common/dex/overlays/oauth2-proxy/config-map.yaml and fill the relevant field with your email and preferred username:

   ...
     staticPasswords:
     - email: <REPLACE_WITH_YOUR_EMAIL>
       username: <REPLACE_WITH_PREFERRED_USERNAME>
   

Change Default User Password

If you have an identity provider (LDAP, GitHub, Google, Microsoft, OIDC, SAML, GitLab) available, you should use that instead of static passwords and connect it to oauth2-proxy or Dex as explained in the sections above. This is best practice instead of using static passwords.

For security reasons, we do not want to use the default static password for the default Kubeflow user when installing in security-sensitive environments. Instead, you should define your own password and apply it either before creating the cluster or after creating the cluster.

Pick a password for the default user, with email [email protected], and hash it using bcrypt:

```sh
python3 -c 'from passlib.hash import bcrypt; import getpass; print(bcrypt.using(rounds=12, ident="2y").hash(getpass.getpass()))'
```

With the required packages, such as passlib, installed, running the command locally prompts for a password and prints its hash:

python3 -c 'from passlib.hash import bcrypt; import getpass; print(bcrypt.using(rounds=12, ident="2y").hash(getpass.getpass()))'
Password:       <--- Enter the password here
$2y$12$vIm8CANhuWui0J1p3jYeGeuM28Qcn76IFMaFWvZCG5ZkKZ4MjTF4u <--- GENERATED_HASH_FOR_ENTERED_PASSWORD

Before Creating the Cluster:

1. Edit common/dex/base/dex-passwords.yaml and fill the relevant field with the hash of the password you chose:

   ...
     stringData:
       DEX_USER_PASSWORD: <REPLACE_WITH_HASH>
   

After Creating the Cluster:

1. Delete the existing secret dex-passwords in the auth namespace using the following command:

   kubectl delete secret dex-passwords -n auth
   

2. Create the secret dex-passwords with the new hash using the following command:

   kubectl create secret generic dex-passwords --from-literal=DEX_USER_PASSWORD='REPLACE_WITH_HASH' -n auth
   

3. Recreate the dex pod in the auth namespace using the following command:

   kubectl delete pods --all -n auth
   

4. Try to log in using the new Dex password.

Upgrading and Extending

For modifications and in-place upgrades of the Kubeflow platform, we provide a rough description for advanced users:

• Never edit the manifests directly; use Kustomize overlays and components on top of the example.yaml.
• This allows you to upgrade by just referencing the new manifests, building with Kustomize, and running kubectl apply again.
• You might have to adjust your overlays and components if needed.
• You might need to prune old resources. For that, you would add labels to all your resources from the start.
• With labels, you can use kubectl apply with --prune and --dry-run to list prunable resources.
• Nevertheless, with a bit of Kubernetes knowledge, one should be able to upgrade.

Handling Immutable Field Errors

Some Kubernetes resources have immutable fields (e.g., spec.selector on Deployments). When an upstream component changes these fields between versions, kubectl apply will reject the update with an error like:

The Deployment "example" is invalid: spec.selector: Invalid value: field is immutable

The solution is to delete the affected resource before reapplying:

kubectl delete deployment <name> -n <namespace> --ignore-not-found
# Then reapply your manifests

This will cause brief downtime for the affected controller but is safe since the new manifests will recreate it.

Version-Specific Upgrade Notes

Kubeflow Pipelines 2.18.0

Review the Pipelines 2.18 upgrade guide, including custom roles, artifact endpoints, package mirrors, and database preparation. The first adoption of the shared TensorBoard signing key briefly interrupts the Pipelines user interface and invalidates existing TensorBoard URLs; reopen the viewers afterward. Automatic artifact previews can contain sample data or model configuration, and legacy cache entries with unknown ownership are no longer reused by default.

The matching Python packages are kfp==2.18.0 and kfp-kubernetes==2.18.0. The existing version 1 SDK compatibility tests remain separate.

1.9 -> 1.9.1 (oauth2-proxy migration)

• We switched to oauth2-proxy, which needs additional attention (cleanup istio-system once).

1.9.1 -> 1.10

kubectl delete clusterrolebinding meta-controller-cluster-role-binding

1.10.2 -> 1.11.0

• Migrates from minio to seaweedfs. Delete minio and optionally migrate your data via S3 commands to seaweedfs.

26.03 -> 26.03.1

1. JobSet Controller Manager (#3428, introduced by #3413): The upstream JobSet upgrade from v0.10.1 to v0.11.0 changed spec.selector.matchLabels from 1 label to 4 labels, which is an immutable field. Delete the Deployment before applying:

   kubectl delete deployment jobset-controller-manager -n kubeflow-system --ignore-not-found
   

2. KServe LLMiSVC RoleBinding: KServe 0.16.x -> 0.17.0+ restructured LLMiSVC role bindings. Clean up the old binding:

   kubectl delete clusterrolebinding llmisvc-manager-rolebinding --ignore-not-found
   

3. Kubeflow Dashboard 2.0.0 (#3469): The Central Dashboard, Profile Controller + KFAM, and PodDefaults webhook moved to kubeflow/dashboard and must be cleaned up first before the new manifests can be applied. Delete the old 26.03 resources once before kubectl apply. Do NOT delete any CustomResourceDefinition or Namespace, so that existing Profile objects and therefore their namespaces survive:

   # Admission webhook (PodDefaults)
   kubectl delete -n kubeflow --ignore-not-found \
     serviceaccount/admission-webhook-service-account \
     service/admission-webhook-service \
     deployment/admission-webhook-deployment \
     certificate.cert-manager.io/admission-webhook-cert \
     issuer.cert-manager.io/admission-webhook-selfsigned-issuer
   kubectl delete --ignore-not-found \
     clusterrole/admission-webhook-cluster-role \
     clusterrole/admission-webhook-kubeflow-poddefaults-admin \
     clusterrole/admission-webhook-kubeflow-poddefaults-edit \
     clusterrole/admission-webhook-kubeflow-poddefaults-view \
     clusterrolebinding/admission-webhook-cluster-role-binding \
     mutatingwebhookconfiguration/admission-webhook-mutating-webhook-configuration
   
   # Central Dashboard
   kubectl delete -n kubeflow --ignore-not-found \
     serviceaccount/centraldashboard \
     role.rbac.authorization.k8s.io/centraldashboard \
     rolebinding.rbac.authorization.k8s.io/centraldashboard \
     configmap/centraldashboard-config \
     configmap/centraldashboard-parameters \
     service/centraldashboard \
     deployment/centraldashboard \
     virtualservice.networking.istio.io/centraldashboard \
     authorizationpolicy.security.istio.io/central-dashboard
   kubectl delete --ignore-not-found \
     clusterrole/centraldashboard \
     clusterrolebinding/centraldashboard
   
   # Profile Controller + KFAM (keep the Profiles CRD and all Profile objects)
   kubectl delete -n kubeflow --ignore-not-found \
     serviceaccount/profiles-controller-service-account \
     role.rbac.authorization.k8s.io/profiles-leader-election-role \
     rolebinding.rbac.authorization.k8s.io/profiles-leader-election-rolebinding \
     service/profiles-kfam \
     deployment/profiles-deployment \
     virtualservice.networking.istio.io/profiles-kfam \
     authorizationpolicy.security.istio.io/profiles-kfam
   kubectl delete --ignore-not-found \
     clusterrolebinding/profiles-cluster-rolebinding
   

   The hashed ConfigMap resources (namespace-labels-data-*, profiles-config-*) are recreated under new hashes by the new manifests and can be left untouched.

   WARNING: Never delete the Profiles CRD; it would delete every profile namespace.

4. KServe Models Web Application:

   # Legacy KServe Models Web Application (renamed from kserve-models-web-app to
   # kserve-models-web-application). The old misnamed stack is no longer in the
   # manifests, keeps the unbound kserve-models-web-app ServiceAccount, and still
   # claims the /kserve-endpoints route, which causes RBAC 403s on SubjectAccessReview requests.
   kubectl delete -n kubeflow --ignore-not-found \
     serviceaccount/kserve-models-web-app \
     service/kserve-models-web-app \
     deployment/kserve-models-web-app \
     virtualservice.networking.istio.io/kserve-models-web-app
   

5. KServe namespace relocation: KServe control-plane resources and the Models Web Application moved from the kubeflow namespace to the kserve namespace. This migration requires a short planned control-plane interruption. Do not delete KServe CRDs, user InferenceService objects, or profile namespaces.

   The standard upstream bundle also uses the current upstream workload labels. Update custom monitoring and policies that select the legacy app: kserve pod label to select the control-plane label instead.

   First remove the unused cluster-scoped authorization for the disabled local model node agent:

   kubectl delete --ignore-not-found \
     clusterrolebinding/kserve-localmodelnode-agent-rolebinding \
     clusterrole/kserve-localmodelnode-agent-role
   

   Before applying the new manifests, scale the old controllers to zero. Controllers in different namespaces use different leader-election Leases and must not run concurrently:

   kubectl scale -n kubeflow --replicas=0 \
     deployment/kserve-controller-manager \
     deployment/kserve-localmodel-controller-manager \
     deployment/llmisvc-controller-manager
   

   Remove the old Models Web Application route before applying the new manifests so the old and new VirtualServices do not claim /kserve-endpoints/ concurrently. Then remove the remaining old namespaced resources:

   kubectl delete -n kubeflow --ignore-not-found \
     virtualservice.networking.istio.io/kserve-models-web-application
   kubectl delete -n kubeflow --ignore-not-found \
     serviceaccount/kserve-models-web-application \
     configmap/kserve-models-web-application-config \
     service/kserve-models-web-application \
     deployment/kserve-models-web-application \
     authorizationpolicy.security.istio.io/kserve-models-web-application \
     networkpolicy.networking.k8s.io/kserve-models-web-application
   

   Apply the new release manifests using the normal upgrade procedure. Then verify the new KServe controllers, Models Web Application, certificates, and webhook endpoints in the kserve namespace:

   kubectl rollout status -n kserve --timeout=120s \
     deployment/kserve-controller-manager \
     deployment/kserve-localmodel-controller-manager \
     deployment/llmisvc-controller-manager \
     deployment/kserve-models-web-application
   kubectl wait -n kserve --for=condition=Ready --timeout=120s \
     certificate.cert-manager.io/serving-cert \
     certificate.cert-manager.io/llmisvc-serving-cert \
     certificate.cert-manager.io/localmodel-serving-cert
   kubectl get endpoints -n kserve \
     kserve-webhook-server-service \
     llmisvc-webhook-server-service \
     localmodel-webhook-server-service
   

   After the new control plane is ready, delete the old kubeflow-scoped resources:

   kubectl delete -n kubeflow --ignore-not-found \
     serviceaccount/kserve-controller-manager \
     serviceaccount/kserve-localmodel-controller-manager \
     serviceaccount/kserve-localmodelnode-agent \
     serviceaccount/llmisvc-controller-manager \
     networkpolicy.networking.k8s.io/kserve \
     role.rbac.authorization.k8s.io/kserve-leader-election-role \
     role.rbac.authorization.k8s.io/llmisvc-leader-election-role \
     rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding \
     rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding \
     configmap/inferenceservice-config \
     secret/kserve-webhook-server-secret \
     secret/kserve-webhook-server-cert \
     secret/llmisvc-webhook-server-cert \
     secret/localmodel-webhook-server-cert \
     service/kserve-controller-manager-metrics-service \
     service/kserve-controller-manager-service \
     service/kserve-webhook-server-service \
     service/llmisvc-controller-manager-service \
     service/llmisvc-webhook-server-service \
     service/localmodel-webhook-server-service \
     deployment.apps/kserve-controller-manager \
     deployment.apps/kserve-localmodel-controller-manager \
     deployment.apps/llmisvc-controller-manager \
     daemonset.apps/kserve-localmodelnode-agent \
     lease.coordination.k8s.io/kserve-controller-manager-leader-lock \
     lease.coordination.k8s.io/llminferenceservice-kserve-controller-manager \
     certificate.cert-manager.io/serving-cert \
     certificate.cert-manager.io/llmisvc-serving-cert \
     certificate.cert-manager.io/localmodel-serving-cert \
     issuer.cert-manager.io/selfsigned-issuer
   

26.03.1 -> 26.10.0

1. Optional Notebooks v1 cleanup: After retiring all Notebooks v1 and TensorBoard workloads, delete the following legacy resources, including the three legacy custom resource definitions. Preserve namespaces, persistent volumes and persistent volume claims:

WARNING: Confirm that no Notebook, Tensorboard or PVCViewer objects remain in any namespace before cleanup. Deleting these definitions deletes all remaining instances and can cascade to owned resources.

kubectl delete --ignore-not-found \
  mutatingwebhookconfiguration/pvcviewer-mutating-webhook-configuration \
  validatingwebhookconfiguration/pvcviewer-validating-webhook-configuration

kubectl delete --namespace kubeflow --ignore-not-found \
  deployment/jupyter-web-app-deployment \
  deployment/notebook-controller-deployment \
  deployment/pvcviewer-controller-manager \
  deployment/tensorboard-controller-deployment \
  deployment/tensorboards-web-app-deployment \
  deployment/volumes-web-app-deployment \
  service/jupyter-web-app-service \
  service/notebook-controller-service \
  service/pvcviewer-controller-manager-metrics-service \
  service/pvcviewer-webhook-service \
  service/tensorboard-controller-controller-manager-metrics-service \
  service/tensorboards-web-app-service \
  service/volumes-web-app-service \
  serviceaccount/jupyter-web-app-service-account \
  serviceaccount/notebook-controller-service-account \
  serviceaccount/pvcviewer-controller-manager \
  serviceaccount/tensorboard-controller-controller-manager \
  serviceaccount/tensorboards-web-app-service-account \
  serviceaccount/volumes-web-app-service-account \
  role.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role \
  role.rbac.authorization.k8s.io/notebook-controller-leader-election-role \
  role.rbac.authorization.k8s.io/pvcviewer-leader-election-role \
  role.rbac.authorization.k8s.io/tensorboard-controller-leader-election-role \
  rolebinding.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role-binding \
  rolebinding.rbac.authorization.k8s.io/notebook-controller-leader-election-rolebinding \
  rolebinding.rbac.authorization.k8s.io/pvcviewer-leader-election-rolebinding \
  rolebinding.rbac.authorization.k8s.io/tensorboard-controller-leader-election-rolebinding \
  configmap/jupyter-web-app-logos \
  certificate.cert-manager.io/pvcviewer-serving-cert \
  issuer.cert-manager.io/pvcviewer-selfsigned-issuer \
  destinationrule.networking.istio.io/jupyter-web-app \
  destinationrule.networking.istio.io/tensorboards-web-app \
  destinationrule.networking.istio.io/volumes-web-app \
  virtualservice.networking.istio.io/jupyter-web-app-jupyter-web-app \
  virtualservice.networking.istio.io/tensorboards-web-app-tensorboards-web-app \
  virtualservice.networking.istio.io/volumes-web-app-volumes-web-app \
  authorizationpolicy.security.istio.io/jupyter-web-app \
  authorizationpolicy.security.istio.io/tensorboards-web-app \
  authorizationpolicy.security.istio.io/volumes-web-app

kubectl delete --ignore-not-found \
  clusterrole/jupyter-web-app-cluster-role \
  clusterrole/jupyter-web-app-kubeflow-notebook-ui-admin \
  clusterrole/jupyter-web-app-kubeflow-notebook-ui-edit \
  clusterrole/jupyter-web-app-kubeflow-notebook-ui-view \
  clusterrole/notebook-controller-kubeflow-notebooks-admin \
  clusterrole/notebook-controller-kubeflow-notebooks-edit \
  clusterrole/notebook-controller-kubeflow-notebooks-view \
  clusterrole/notebook-controller-role \
  clusterrole/pvcviewer-metrics-reader \
  clusterrole/pvcviewer-proxy-role \
  clusterrole/pvcviewer-role \
  clusterrole/tensorboard-controller-manager-role \
  clusterrole/tensorboard-controller-metrics-reader \
  clusterrole/tensorboard-controller-proxy-role \
  clusterrole/tensorboards-web-app-cluster-role \
  clusterrole/tensorboards-web-app-kubeflow-tensorboard-ui-admin \
  clusterrole/tensorboards-web-app-kubeflow-tensorboard-ui-edit \
  clusterrole/tensorboards-web-app-kubeflow-tensorboard-ui-view \
  clusterrole/volumes-web-app-cluster-role \
  clusterrole/volumes-web-app-kubeflow-volume-ui-admin \
  clusterrole/volumes-web-app-kubeflow-volume-ui-edit \
  clusterrole/volumes-web-app-kubeflow-volume-ui-view \
  clusterrolebinding/jupyter-web-app-cluster-role-binding \
  clusterrolebinding/notebook-controller-role-binding \
  clusterrolebinding/pvcviewer-manager-rolebinding \
  clusterrolebinding/pvcviewer-proxy-rolebinding \
  clusterrolebinding/tensorboard-controller-manager-rolebinding \
  clusterrolebinding/tensorboard-controller-proxy-rolebinding \
  clusterrolebinding/tensorboards-web-app-cluster-role-binding \
  clusterrolebinding/volumes-web-app-cluster-role-binding

kubectl delete --ignore-not-found \
  customresourcedefinition/notebooks.kubeflow.org \
  customresourcedefinition/pvcviewers.kubeflow.org \
  customresourcedefinition/tensorboards.tensorboard.kubeflow.org

Version-specific hashed ConfigMaps can be left untouched.

1. Kubeflow Pipelines user interface (#3661): The ml-pipeline-ui Deployment now uses the Recreate update strategy. Kubernetes stored the default rolling update parameters in the existing Deployment, and kubectl apply --server-side does not remove them, so applying the new manifests fails with spec.strategy.rollingUpdate: Forbidden. Switch the update strategy before applying the new manifests. The command changes nothing when the Deployment already uses Recreate:

   kubectl patch deployment ml-pipeline-ui -n kubeflow --type=merge --patch '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'
   

Release Process

Kubeflow release handbook.

Security

To view all past security scans, head to the Image Extracting and Security Scanning GitHub Action workflow. In the logs of the workflow, you can expand the Run image extracting and security scanning script step to view the CVE logs. You will find a per-image CVE scan and a JSON dump of per-WorkingGroup aggregated metrics. You can run the Python script from the workflow file locally on your machine to obtain the detailed JSON files for any git commit.

For more information please consult the SECURITY.md.

Pre-commit Hooks

This repository uses pre-commit hooks to ensure code quality and consistency. The following hooks are configured:

1. Black - Python code formatter.
2. Yamllint - YAML file linter.
3. Shellcheck - Shell script static analysis.

To use these hooks:

1. Install pre-commit:

   pip install pre-commit
   

2. Install the git hooks:

   pre-commit install
   

The hooks will run automatically on git commit. You can also run them manually:

pre-commit run

Architecture

Frequently Asked Questions

• Q: What versions of Istio, Knative, Cert-Manager, Argo, ... are compatible with Kubeflow? A: Please refer to each individual component's documentation for a dependency compatibility range. For Istio, Knative, Dex, Cert-Manager, and OAuth2 Proxy, the versions in common are the ones we have validated.
• Q: Can I use Kubeflow in an air-gapped environment? A: Yes you can. You just need to get the list of images from our trivy CVE scanning script, mirror them and replace the references in the manifests with kustomize components and overlays, see Upgrading and Extending. You could also use a simple kyverno policy to replace the images at runtime, which could be easier to maintain.
• Q: Why does Kubeflow use Istio CNI instead of standard Istio? A: Istio CNI provides better security by eliminating the need for privileged init containers, making it more compatible with Pod Security Standards (PSS). It also enables native sidecars support introduced in Kubernetes 1.28, which helps address issues with init containers and application lifecycle management.
• Q: Why does Istio CNI fail on Google Kubernetes Engine (GKE) with "read-only file system" errors? A: GKE mounts /opt/cni/bin as read-only for security reasons. Use the GKE-specific overlay: kubectl apply -k common/istio/istio-install/overlays/gke (or overlays/ambient-gke for ambient mode). These overlays use GKE's writable CNI directory at /home/kubernetes/bin. For details, see Istio CNI Prerequisites.