Skip to content

Use a Datalab virtual cluster

Declare each virtual Kubernetes cluster under spec.vclusters:

apiVersion: pkg.internal/v1beta2
kind: Datalab
metadata:
  name: s-joe
  namespace: workspace
spec:
  users: [joe]
  vclusters:
    vc0:
      storage: 5Gi
      groups:
      - name: namespace-demo
        namespace: demo
        role: edit

Each entry creates one runtime in status.runtimeNamespace. Add vc1 alongside vc0 to create another cluster. An empty map creates none. Sessions and their environment-controlled vclusters remain separate.

API access

The virtual API for vc0 is:

https://s-joe.lab.acme.com/.vc0

The Gateway removes /.vc0 and forwards to that instance with backend TLS validation. Workload Ingress and Gateway synchronization is disabled; use Kubernetes port forwarding to reach a workload Service.

Roles and profiles

The API validates issuer, Datalab audience, signature, expiry, and configured roles under resource_access.<datalab>.roles. Kubernetes RBAC controls which operations those roles permit.

Caller Virtual API access
A user in spec.users Explicit additional profiles, through the Datalab users group
A Datalab administrator Additional profiles and each instance's built-in vcluster_<instance>_cluster-admin role
The Datalab service account None by default; its only default role is ws_api
An identity with an assigned vcluster_<instance>_<group> role The group's configured namespace and Kubernetes role

groups defines additional profiles with view, edit, or admin access. The provider grants their client roles to both Datalab groups. Optional OAuth scopes select which assigned roles appear in Datalab client tokens. Roles are specific to an instance. See token exchange for a token limited to one selected profile.

A downloaded profile selects the endpoint and default namespace. It neither grants access nor reduces permissions already present in the caller's token. The cluster-admin profile cannot elevate a namespace caller, and a namespace profile cannot restrict a caller who already has the admin role.

Provider Datalab already assigns workspace administrators ws_admin and each instance's vcluster_<instance>_cluster-admin role through the managed administrators group. No additional manual role assignment is needed. The API checks the instance role in the token. An unrelated manual assignment of ws_admin alone does not grant virtual API access.

Download a profile

Resolve its Secret reference from status.connections; do not construct the name. Each profile Secret contains kubeconfig, the Datalab client's public client-id, and issuer-url. Public CA certificates can be included. It contains no client secret, token, private key, or client certificate. The provider keeps the recovery kubeconfig private.

ready: true reports runtime, routing, role, scope, and RBAC readiness. It does not imply caller authorization or require caller credentials to be present. The kubeconfig includes a Python 3 credential helper. Additional profiles select --token-only and read an access token file supplied by the caller. They require no OAuth client ID or client secret and make no token requests. The built-in cluster-admin profile retains the Datalab client ID and issuer. It accepts a caller token or a separate client secret for device login as a user. It never requests a service-account token. No separate helper file is needed on the caller's machine.

The following cases use Datalab s-joe, instance vc0, and the additional namespace-demo profile from the example above. Use Bash with kubectl, jq, base64, and Python 3 installed. Set the host context used to read Datalab metadata and download Secrets:

set -euo pipefail
umask 077
host_context=your-host-context

Replace your-host-context with the platform context. Download commands use that context explicitly. Commands with --kubeconfig connect to the virtual cluster. If you cannot read the host Datalab or its Secrets, obtain the files from the platform operator and continue with the local steps.

Case Local kubeconfig file Credential supplied by the caller
Human device login s-joe-vc0.kubeconfig Datalab OAuth client secret; browser login identifies the user
Delegated access s-joe-vcluster-vc0_namespace-demo.kubeconfig An exchanged user access token for the selected profile

These are local filenames. Always resolve the host Secret names and namespaces from Datalab status. The download commands require a ready profile.

Human access

This case logs in as your Keycloak user. The client secret authenticates the Datalab client; your user identity and assigned roles control API access.

1. Download the profile with device login

Select the built-in cluster-admin profile. Its helper supports device login even when your user has only an additional namespace role.

read -r profile_namespace profile_secret < <(
  kubectl --context "${host_context}" -n workspace get datalab s-joe -o json | \
    jq -er '.status.connections[] |
      select(.type == "vcluster" and .instance == "vc0" and
             .group == "cluster-admin" and .ready == true) |
      .secretRef | [.namespace, .name] | @tsv'
)
kubectl --context "${host_context}" -n "${profile_namespace}" \
  get secret "${profile_secret}" -o jsonpath='{.data.kubeconfig}' | \
  base64 --decode > s-joe-vc0.kubeconfig

2. Download the OAuth client secret

Resolve the ready client credential reference from status.oauth2. Save only its decoded client secret value in a private file. The helper does not read a Secret YAML document.

unset DATALAB_ACCESS_TOKEN_FILE
mkdir -p "$HOME/.config/datalab"
chmod 700 "$HOME/.config/datalab"
export DATALAB_CLIENT_SECRET_FILE="$HOME/.config/datalab/s-joe-client-secret"
read -r oauth_namespace oauth_secret < <(
  kubectl --context "${host_context}" -n workspace get datalab s-joe -o json | \
    jq -er '.status.oauth2[] |
      select(.type == "client" and .ready == true) |
      .secretRef | [.namespace, .name] | @tsv'
)
kubectl --context "${host_context}" -n "${oauth_namespace}" \
  get secret "${oauth_secret}" -o json | \
  jq -er '.data.clientSecret // .data.client_secret' | \
  base64 --decode > "$DATALAB_CLIENT_SECRET_FILE"
chmod 600 "$DATALAB_CLIENT_SECRET_FILE"

You can instead obtain that value through the platform's credential process. If DATALAB_CLIENT_SECRET_FILE is unset, the helper asks for the secret in an interactive terminal without showing or saving the input. A valid cached access token needs no secret. Refresh and new login need the secret again.

3. Start kubectl and approve the device login

For a user with vcluster_vc0_cluster-admin, run:

kubectl --kubeconfig ./s-joe-vc0.kubeconfig get namespaces
kubectl --kubeconfig ./s-joe-vc0.kubeconfig auth can-i '*' '*' --all-namespaces

On the first command, the helper prints a verification URL and a user code. Open the URL in a browser, enter the code, and sign in as your Keycloak user. The browser can run on another device. After approval, kubectl continues. The authorization check returns yes for a user whose token contains the built-in administrator role and whose binding is ready.

The default login requests openid offline_access s-joe:interactive. The interactive scope permits ws_access, ws_admin, and built-in vCluster administrator roles that your user holds. It does not request all additional namespace profiles.

4. Select scopes for other assigned user roles

For a user assigned vcluster_vc0_namespace-demo, add its OAuth scope to a local copy of the device-login profile:

kubectl --kubeconfig ./s-joe-vc0.kubeconfig config view --raw -o json | \
  jq '.users[].user.exec.args += [
    "--scope", "s-joe:interactive",
    "--scope", "s-joe:vclusters_vc0_namespace-demo"
  ]' > s-joe-vc0-human.kubeconfig
kubectl --kubeconfig ./s-joe-vc0-human.kubeconfig -n demo get pods
kubectl --kubeconfig ./s-joe-vc0-human.kubeconfig -n demo auth can-i create deployments

JSON is also a valid kubeconfig format. This copy keeps the embedded helper, endpoint, and CA settings. Add another --scope pair for each additional profile you need. Each scope includes only roles that your user holds. A user with only the example namespace role gets edit access in demo, not cluster administration. A user who also holds the built-in administrator role retains that wider access when requesting s-joe:interactive.

5. Reuse the login

The helper saves the access token, offline refresh token, and the local login deadline in:

~/.cache/datalab/oidc/<hash>.json

It uses $XDG_CACHE_HOME/datalab/oidc when XDG_CACHE_HOME is set. Set DATALAB_TOKEN_CACHE_DIR to select another private directory. The directory has mode 0700; token files have mode 0600. The hash separates issuer, client ID, and scopes. Different scopes can require a separate device login. The helper does not store the client secret in this cache.

Later kubectl commands reuse a valid token without a browser prompt. Near expiry, the helper requests a new token with the cached offline refresh token and the client secret. It saves the updated token response, including any replacement refresh token. Refresh keeps the user's current assigned permissions within the login's requested scopes. If Keycloak rejects the refresh token with invalid_grant, it starts device login again. There is no background refresh while kubectl is idle. In a new shell, set DATALAB_CLIENT_SECRET_FILE again if refresh or login is needed.

To sign in as another user, run the standalone helper from this repository:

python3 scripts/vcluster-credentials.py \
  --issuer https://identity.acme.com/realms/acme --client-id s-joe \
  --force-login >/dev/null

Use the same issuer, client ID, scopes, cache directory, and secret source as the downloaded profile. Add the same --scope pairs for the modified profile. For a private issuer CA, add --issuer-ca /path/to/issuer-ca.pem. Generated device-login profiles already contain the configured issuer CA.

Human login lifetime

CLI device login requests offline access by default. Access tokens stay short-lived; the CLI obtains replacements without another device prompt while the offline session remains valid. Refresh does not extend the local login deadline.

Limit Default or setting Effect on kubectl
Access token Realm or client Access Token Lifespan; upstream default is 5 minutes The helper refreshes it when needed.
CLI cached login 30 days after device approval The helper starts device login again after this deadline.
Client offline idle timeout Provider sets 30 days The session can expire if no refresh occurs within this period.
Client offline maximum Provider sets 30 days Keycloak applies this limit when the realm enables Offline Session Max Limited.
Realm offline limits and revocation Platform settings Shorter realm limits or revoked offline access can end the login sooner.

The realm must permit offline_access, and the user must have the standard realm offline_access role. Provider Datalab adds the built-in offline scope to the client's optional scopes. Keycloak normally grants its role through the realm's default roles. Ordinary SSO Session Idle/Max limits do not govern an offline session. Check Realm settings > Sessions and Clients > s-joe > Advanced for the deployed offline limits. The device code has a separate approval timeout.

An existing cache from a login without offline_access is not reused for the new scope set. Download a current kubeconfig and approve device login once. The new cache then supports automatic refresh for the CLI login period. See Keycloak offline access and login lifetime.

[!NOTE] On Keycloak versions older than 26.8.1, device login token refresh is unreliable; use --force-login to start a new device login when refresh fails.

Delegated access

This case uses an access token issued for a human user and passed to another caller. The recipient needs the selected kubeconfig and token, with no client secret or browser login.

1. Obtain a token for the selected profile

The user or platform service performs token exchange with the user's access token as subject_token. For the example profile, request:

scope=s-joe:vclusters_vc0_namespace-demo
audience=s-joe
requested_token_type=urn:ietf:params:oauth:token-type:access_token

The OAuth scope is s-joe:vclusters_vc0_namespace-demo; the corresponding Keycloak client role is vcluster_vc0_namespace-demo. Exchange can select only a role the user holds. Before passing on the result, the issuing service must check that resource_access.s-joe.roles contains that role and omits administrator and unrelated profile roles. ws_api can also be present. The token keeps the human user's subject. Selecting a kubeconfig alone does not restrict a token with wider permissions.

Give the recipient the returned JWT access token through a private file. Keep the client secret, original user token, and refresh token with the user or service that performs exchange. An ID token cannot replace this access token.

2. Download the selected profile

Using the host context from the download setup, select namespace-demo:

read -r profile_namespace profile_secret < <(
  kubectl --context "${host_context}" -n workspace get datalab s-joe -o json | \
    jq -er '.status.connections[] |
      select(.type == "vcluster" and .instance == "vc0" and
             .group == "namespace-demo" and .ready == true) |
      .secretRef | [.namespace, .name] | @tsv'
)
kubectl --context "${host_context}" -n "${profile_namespace}" \
  get secret "${profile_secret}" -o jsonpath='{.data.kubeconfig}' | \
  base64 --decode > s-joe-vcluster-vc0_namespace-demo.kubeconfig

The platform operator can supply this file if the recipient has no host access. The profile uses --token-only: it cannot start device login or make refresh requests.

3. Set the token file and run kubectl

Save the received access token as ~/.config/datalab/s-joe-vc0-namespace-demo.token. The file must contain only the JWT, without a JSON wrapper or Bearer prefix. Then run:

unset DATALAB_CLIENT_SECRET_FILE
export DATALAB_ACCESS_TOKEN_FILE="$HOME/.config/datalab/s-joe-vc0-namespace-demo.token"
chmod 600 "$DATALAB_ACCESS_TOKEN_FILE"
kubectl --kubeconfig ./s-joe-vcluster-vc0_namespace-demo.kubeconfig -n demo get pods
kubectl --kubeconfig ./s-joe-vcluster-vc0_namespace-demo.kubeconfig -n demo \
  auth can-i create deployments
kubectl --kubeconfig ./s-joe-vcluster-vc0_namespace-demo.kubeconfig \
  auth can-i create namespaces

The example edit profile permits Deployment creation in demo; namespace creation is denied. DATALAB_ACCESS_TOKEN_FILE contains a file path, not the token value. The helper reads that file and returns the token to kubectl. It makes no login or refresh request and does not use the device-login cache.

Delegated token lifetime and revocation

The received token's exp claim is its expiry time. The exchange response's expires_in reports its remaining lifetime when issued. Do not assume a fixed duration or that exchange starts a new full session. The helper sets ExecCredential.expirationTimestamp from exp. When the token expires, ask the issuing user or service for a new exchange result and replace the token file. The recipient has no refresh token to renew it.

To display the expiry in UTC without printing the token, run:

python3 - "$DATALAB_ACCESS_TOKEN_FILE" <<'PY'
import base64
from datetime import datetime, timezone
import json
from pathlib import Path
import sys

payload = Path(sys.argv[1]).read_text().strip().split('.')[1]
claims = json.loads(base64.urlsafe_b64decode(payload + '=' * (-len(payload) % 4)))
print(datetime.fromtimestamp(claims['exp'], timezone.utc).isoformat())
PY

This command reads the expiry claim; the virtual API verifies the signature and decides whether to accept the token.

The virtual API validates JWTs locally. It does not ask Keycloak whether each token has been revoked. Removing a user's Keycloak role, ending the login session, rotating the client secret, or deleting a local token file does not reliably invalidate copies of an already issued JWT at this API. Keycloak also documents that revoking the original access token does not revoke an exchanged access token. See token exchange revocation.

Without an API authorization change, expect an issued token to remain usable until exp. To deny access sooner, the platform operator can remove the affected configured profile or its Kubernetes RBAC binding. This removes access for all callers that depend on that role or binding, not just one token. Apply changes through the source configuration and allow for reconciliation. This setup has no individual JWT revocation list. The same limit applies to access tokens cached after human device login. Revoke its offline session in Keycloak to stop further CLI refresh. Ordinary browser logout does not revoke offline access. Neither action erases issued JWTs.

Action Human device login Delegated access token
Access token expires Helper attempts refresh with the client secret. Recipient must obtain a new token.
Offline session expires or is revoked Refresh stops; a new device login is needed. An issued token can remain usable until exp.
Keycloak role assignment is removed New tokens lose the role; issued tokens can retain it. An issued token can retain the role until exp.
Configured profile or required RBAC binding is removed Access is denied after the API configuration or binding changes. The same denial applies to the delegated token.

Machine access

An identity administrator must explicitly assign the required vcluster role to the existing Datalab service account. For example, vcluster_vc0_namespace-demo permits the configured operations in demo. Do not add this account to a human group. Request the profile's optional scope when using a separately assigned role; the default machine token has only ws_api. Obtain the token before using the downloaded profile.

Obtain the Datalab client secret from the source in status.oauth2 through an authorized Secret-delivery process. This is the <datalab>-oauth-client Secret. Use it to authenticate a token request with these parameters:

grant_type=client_credentials
client_id=s-joe
scope=s-joe:vclusters_vc0_namespace-demo

Save the returned access token in a private file and use DATALAB_ACCESS_TOKEN_FILE as shown above. The downloaded helper accepts only an access token file for additional profiles. The built-in cluster-admin helper uses a supplied client secret for user device login, not machine access. Client credentials alone produce a token with only ws_api and cannot access the virtual API. An explicitly granted machine role still requires its matching optional scope in a separate token request. A profile download supplies no credentials or role grants.

Secret bindings

Select a Datalab connection by type and instance name:

spec:
  databases:
    pg0:
      names: [analytics]
      storage: 5Gi
  vclusters:
    vc0:
      secretBindings:
      - type: postgres
        name: pg0
        target:
          namespace: applications
          name: database-credentials

The provider publishes each service connection in status.connections, with its type, name, secretRef, and readiness. A binding selects the same type and name pair. The mapping uses that connection's secretRef.

The connection sources are:

Type Name Published host Secret
s3 primary The Secret with the same name as the Datalab
postgres A key in spec.databases A connection copy in the Datalab namespace
mongo A key in spec.documentStores A connection copy in the Datalab namespace
redis A key in spec.cacheStores A connection copy in the Datalab namespace
registry primary A connection copy in the Datalab namespace
custom A predefined Secret name A connection copy in the Datalab namespace

Only declared service instances can be selected. The storage source namespace comes from EnvironmentConfig.data.storage.secretNamespace, or the claim namespace when unset. Use a custom connection for credentials such as a GitHub token.

The operator can list predefined custom Secrets in the EnvironmentConfig:

data:
  customSecrets:
  - name: apikey-1
  - name: apikey-2

The provider checks these names in each Datalab's runtime namespace. Set namespace on an entry to use a shared source namespace. Names must be unique. It observes the source and does not create or change it. A custom connection is listed in status.connections after its source is confirmed ready. Its ready value stays false until the published copy is available and current. Missing custom sources do not block Datalab readiness.

Bind an available custom connection through the same list:

spec:
  vclusters:
    vc0:
      secretBindings:
      - type: custom
        name: apikey-1
        target:
          namespace: applications
          name: application-credentials

vCluster creates the target namespace and syncs the whole Secret, including source updates and deletion. Secret keys keep their source names. Only ready, unambiguous connections generate mappings. Missing or unready references are skipped; reconciliation adds or removes mappings as source readiness changes. Source readiness does not confirm delivery inside the virtual cluster.

Platform settings

Merge these values into the Datalab's EnvironmentConfig:

data:
  iam:
    issuerURL: https://identity.acme.com/realms/acme
  ingress:
    domain: lab.acme.com
  httpRoute:
    enabled: true
    parentName: default
    parentNamespace: contour
    sectionName: workspaces-https
    httpSectionName: workspaces-http
  vcluster:
    storage: 5Gi
    storageClasses:
      allowed: [sbs-default]
      default: sbs-default

For a private issuer CA, set vcluster.issuerCA. An optional HTTPS vcluster.discoveryURL can provide separate discovery routing with the same issuer. For a private Gateway CA, set ingress.certificateAuthority.

Host requirements

Apply dependencies/rbac.yaml. The host must serve ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding. The Gateway must support HTTPRoute prefix rewriting, BackendTLSPolicy, and Kubernetes streaming requests. vcluster.backendTLS.apiVersion defaults to gateway.networking.k8s.io/v1; set it to the version served by your platform.

Set network.serviceCIDR and network.podCIDRs to the host ranges. Virtual workloads follow the environment's network policy. Managed databases in the same runtime namespace are reachable through their generated connection endpoints. Cross-namespace backends need an approved network.internalEgress rule. Virtual clusters share the host namespace and node kernel; their RBAC boundaries apply to their virtual APIs.

Storage and control-plane resources

The vcluster EnvironmentConfig section supplies defaults for all declared instances. spec.vclusters.<name>.storage overrides the control-plane PVC size. vcluster.storageClassName selects its class; otherwise the default class is used. Only classes in vcluster.storageClasses.allowed are exposed to virtual workloads. A host admission policy enforces this list for each syncer. Control-plane, workload, database, and session PVCs share the existing defaults.overallQuota.storage limit.

Set vcluster.resources.requests and vcluster.resources.limits in the EnvironmentConfig to change control-plane resources. Both the native vCluster configuration and the host StatefulSet use these values. The defaults are CPU 200m and memory 1Gi for requests, and CPU 4 and memory 4Gi for limits. Ephemeral storage defaults to a 1Gi request and a 10Gi limit. Each supplied field overrides its default.

SQLite settings

To limit SQLite to two open connections and one idle connection, merge:

data:
  vcluster:
    backingStore:
      database:
        embedded:
          enabled: true
          dataSource: "sqlite:///data/state.db?_journal=TRUNCATE&_locking=NORMAL&cache=shared&_busy_timeout=30000&_txlock=immediate&_synchronous=FULL"
          extraArgs:
          - --datastore-max-open-connections=2
          - --datastore-max-idle-connections=1

Datalab STAC catalog

The STAC workload example runs stac-fastapi-pgstac in virtual namespace demo. Its PostgreSQL database is a Datalab-managed service on the host. Add this request alongside vclusters:

databases:
  pg0:
    type: postgres
    names: [catalog]
    storage: 5Gi

PgSTAC requires PostGIS and a schema migration. The example migration Job uses the generated PostgreSQL credentials from the Datalab Secret. It creates a separate catalog login with the pgstac_ingest role. The API uses that login and the standard PostgreSQL settings: PGHOST, PGPORT, PGUSER, PGPASSWORD, and PGDATABASE.

PostgreSQL needs DNS and access to the host Kubernetes API for its controller. In a namespace with denied egress, approve these targets for the PostgreSQL Pods. Namespace selectors do not match host-network API endpoints on all CNIs. The integration has a PostgreSQL-specific policy for this case.

The example expects catalog-postgres-admin and catalog-postgres Secrets in virtual namespace demo. The first contains migration credentials. The second contains the API login. Supply these through your Secret process. Build the image from tests/vcluster/catalog.Dockerfile, push it to your registry, and replace __CATALOG_IMAGE__ with that image reference before applying the example. Wait for catalog-migrate to complete before starting the API Deployment. Remove the migration Secret after it completes.

Connect to the catalog through the generated profile:

kubectl --kubeconfig ./s-joe-vc0.kubeconfig -n demo \
  port-forward service/catalog 8080:8080

The catalog is available at http://localhost:8080. The example enables STAC transactions so a client can add collections and items. Its catalog records remain in PostgreSQL when the API Pod restarts.

Run tests/integration/run.bash verify-vcluster to check both virtual clusters, caller role boundaries, public profiles, persistent volumes, and the catalog. The catalog test creates a collection and item, searches it, restarts the API, and repeats the search after a control-plane restart.

Removal

Removing a group removes its role and binding after reconciliation while retaining its virtual namespace data. Removing an instance removes its runtime and control-plane PVC. Other instances, databases, sessions, and the shared namespace remain. Deletion follows the host StorageClass retention policy. Back up control-plane and database volumes before removing an instance or Datalab.

Removing a Keycloak role assignment or client prevents new tokens with those grants. Already-issued JWTs can remain valid until expiry because the API validates them locally. To deny access sooner, remove the affected configured role or Kubernetes binding. Allow for reconciliation before checking revocation.