Skip to content

S3 client access

The read-only S3 endpoint is experimental. Range reads currently return HTTP 200 with Content-Range, rather than HTTP 206.

Enable the endpoint

Set httpRoute.enableS3: true in the selected EnvironmentConfig, with Gateway Data exposure configured under Service addresses. This adds the S3 endpoint to each Data-enabled Datalab's browser hostname.

Use a Gateway API controller that supports HTTPRoute header regular expressions, such as Contour. Add a ReferenceGrant if its certificate Secret is in another namespace. With S3 disabled, Data requests use the browser service. The network policies must permit access from the Gateway and access to object storage and IAM.

S3 access to s-jane

The Jane example creates the Datalab s-jane and selects datalab-custom. With the example EnvironmentConfig, its endpoint is https://s-jane.lab.acme.com and its issuer is https://identity.acme.com/realms/acme. Replace these example DNS names with your environment's values.

Keycloak checks CLIENT_SECRET and issues a JWT. The AWS CLI sends it as AWS_SESSION_TOKEN in the X-Amz-Security-Token header. NGINX checks the tenant ID and passes the token to oauth2-proxy, which checks its signature, issuer, audience, expiry, and Datalab role. After authorization, rclone serve s3 checks the original AWS SigV4 request using s-jane as both signing values. It serves read-only data through separate upstream storage credentials.

Credential field Value
AccessKeyId s-jane
SecretAccessKey s-jane
SessionToken Keycloak JWT access token
Expiration Token expiry

The signing values are public tenant identifiers. They do not grant access without a valid JWT. The token must include s-jane in aud and at least one of ws_access, ws_admin, or ws_api under resource_access.s-jane.roles. These are Keycloak client roles, separate from OAuth scopes. All three permit reads. The upstream storage policy limits which buckets and objects the Datalab can read. Missing or invalid tokens cannot use a browser cookie or an unsigned request as a fallback.

The generated Datalab client always adds its own audience. Other OAuth clients must emit the same audience and tenant roles. Service-account tokens do not need a verified email address.

The following Bash commands use the credential helper from the repository root. It uses the client credentials grant with s-jane as the client ID. At the prompt, enter clientSecret from the Secret referenced by the client's status.oauth2[].secretRef. The commands read bucket s-jane with path-style requests. Change the bucket name if your storage uses another name.

Set the client secret for either method:

CLIENT_SECRET="$(python3 -c 'import getpass; print(getpass.getpass("CLIENT_SECRET: "))')"
export CLIENT_SECRET

Manual token retrieval

export AWS_ACCESS_KEY_ID=s-jane AWS_SECRET_ACCESS_KEY=s-jane
export AWS_ENDPOINT_URL=https://s-jane.lab.acme.com
export AWS_DEFAULT_REGION=us-east-1
export AWS_PROFILE=s-jane
aws configure set s3.addressing_style path --profile s-jane

credentials="$(python3 tests/s3/s3-credentials.py \
  --tenant s-jane \
  --issuer https://identity.acme.com/realms/acme)"
AWS_SESSION_TOKEN="$(printf '%s' "$credentials" | \
  python3 -c 'import json, sys; print(json.load(sys.stdin)["SessionToken"])')"
export AWS_SESSION_TOKEN
unset credentials

aws s3 ls s3://s-jane/

Run the token retrieval command again after the token expires, or use automatic renewal below.

Automatic token renewal with credential_process

Clear the manual AWS settings so that the CLI uses the profile's credential process. The CLI obtains new credentials when needed, using the helper's Expiration value.

unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
unset AWS_CONFIG_FILE AWS_PROFILE AWS_ENDPOINT_URL
unset AWS_DEFAULT_REGION AWS_REGION

export AWS_PROFILE=s-jane

aws configure set region us-east-1 --profile s-jane
aws configure set endpoint_url https://s-jane.lab.acme.com --profile s-jane
aws configure set s3.addressing_style path --profile s-jane
aws configure set credential_process \
  "python3 \"$PWD/tests/s3/s3-credentials.py\" --tenant s-jane --issuer https://identity.acme.com/realms/acme" \
  --profile s-jane

aws s3 ls s3://s-jane/

Keep CLIENT_SECRET available in the environment of the AWS CLI or SDK process. If you move the repository, update the helper's absolute path in the profile.

The endpoint supports list, HEAD, GET, and byte-range GET requests. It remains read only even when the Data browser has write permissions. Writes and presigned S3 URLs are rejected. packageR download and share links use the browser route.