Crossplane with Workload Identity

On this page

When running Crossplane on managed Kubernetes clusters (EKS, AKS, GKE), you can leverage Kubernetes Workload Identity to grant Crossplane access to pull packages from private cloud container registries. This allows Crossplane to install providers, functions, and configurations from registries like AWS ECR, Azure ACR, and Google Artifact Registry without managing static credentials.

Important

This guide configures the Crossplane package manager to pull packages from private registries. However, packages reference container images that run as separate pods (providers and functions).

Two-step image pull process:

Crossplane package manager pulls the package, extracts the package contents (CRDs, XRDs) and creates deployments
Kubernetes nodes pull the runtime container images when creating provider/function pods

This guide only covers step 1. For step 2, ensure your Kubernetes nodes have permissions to pull images from the private registry. This is typically configured at the cluster level:

AWS EKS: Node IAM role with ECR pull permissions
Azure AKS: Kubelet managed identity with AcrPull role
GCP GKE: Node service account with Artifact Registry reader role

Without node-level access, package installation succeeds but pods fail with ImagePullBackOff.

Overview

To enable Crossplane package manager access to private registries, configure service account annotations during installation. The crossplane service account in the crossplane-system namespace requires specific annotations for each cloud provider:

AWS EKS: IAM Roles for Service Accounts (IRSA)
Azure AKS: Azure Workload Identity
Google Cloud GKE: GKE Workload Identity

Select your cloud provider below for detailed setup instructions:

AWS EKS Integration

Configure Crossplane to pull packages from Amazon ECR using IAM Roles for Service Accounts (IRSA).

Prerequisites

An Amazon EKS cluster with OIDC provider enabled
AWS CLI installed and configured
kubectl configured to access your EKS cluster
Permissions to create IAM roles and policies

Enable OIDC Provider

If your EKS cluster doesn’t have an OIDC provider, enable it:

1eksctl utils associate-iam-oidc-provider \
2  --cluster=<CLUSTER_NAME> \
3  --approve

Verify the OIDC provider:

1aws eks describe-cluster \
2  --name <CLUSTER_NAME> \
3  --query "cluster.identity.oidc.issuer" \
4  --output text

Create IAM Policy for ECR Access

Create an IAM policy that grants permissions to pull images from ECR:

 1cat > crossplane-ecr-policy.json <<EOF
 2{
 3  "Version": "2012-10-17",
 4  "Statement": [
 5    {
 6      "Effect": "Allow",
 7      "Action": [
 8        "ecr:GetAuthorizationToken"
 9      ],
10      "Resource": "*"
11    },
12    {
13      "Effect": "Allow",
14      "Action": [
15        "ecr:BatchCheckLayerAvailability",
16        "ecr:GetDownloadUrlForLayer",
17        "ecr:BatchGetImage"
18      ],
19      "Resource": "arn:aws:ecr:<REGION>:<ACCOUNT_ID>:repository/*"
20    }
21  ]
22}
23EOF
24
25aws iam create-policy \
26  --policy-name CrossplaneECRPolicy \
27  --policy-document file://crossplane-ecr-policy.json

Note

Replace <REGION> and <ACCOUNT_ID> with your AWS region and account ID. You can restrict the Resource to specific repositories if needed.

Create IAM Role with Trust Policy

Create an IAM role that can be assumed by the Crossplane service account:

 1export CLUSTER_NAME=<your-cluster-name>
 2export AWS_REGION=<your-aws-region>
 3export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
 4export OIDC_PROVIDER=$(aws eks describe-cluster --name $CLUSTER_NAME --region $AWS_REGION --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
 5
 6cat > trust-policy.json <<EOF
 7{
 8  "Version": "2012-10-17",
 9  "Statement": [
10    {
11      "Effect": "Allow",
12      "Principal": {
13        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
14      },
15      "Action": "sts:AssumeRoleWithWebIdentity",
16      "Condition": {
17        "StringEquals": {
18          "${OIDC_PROVIDER}:sub": "system:serviceaccount:crossplane-system:crossplane",
19          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
20        }
21      }
22    }
23  ]
24}
25EOF
26
27aws iam create-role \
28  --role-name CrossplaneECRRole \
29  --assume-role-policy-document file://trust-policy.json

Attach Policy to Role

Attach the ECR policy to the IAM role:

1aws iam attach-role-policy \
2  --role-name CrossplaneECRRole \
3  --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/CrossplaneECRPolicy

Install Crossplane with IRSA Annotation

Install Crossplane with the service account annotation:

1helm upgrade --install crossplane \
2  crossplane-stable/crossplane \
3  --namespace crossplane-system \
4  --create-namespace \
5  --set "serviceAccount.customAnnotations.eks\.amazonaws\.com/role-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/CrossplaneECRRole"

Verify Configuration

Check that the service account has the correct annotation:

1kubectl get sa crossplane -n crossplane-system -o yaml

Expected output should include:

1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  annotations:
5    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/CrossplaneECRRole
6  name: crossplane
7  namespace: crossplane-system

Test Package Installation from ECR

Once configured, you can install Crossplane packages (Providers, Functions, Configurations) from your ECR registry. Here’s an example using a Provider:

1kubectl apply -f - <<EOF
2apiVersion: pkg.crossplane.io/v1
3kind: Provider
4metadata:
5  name: provider-aws-s3
6spec:
7  package: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/crossplane/provider-aws-s3:v1.2.0
8EOF

Check the provider installation status:

1kubectl get provider provider-aws-s3
2kubectl describe provider provider-aws-s3

Troubleshooting

Failed to Get Authorization Token

Error:

failed to get authorization token: AccessDeniedException

Solution:

Verify the IAM role has the ecr:GetAuthorizationToken permission
Check the trust policy allows the service account to assume the role
Ensure the service account annotation matches the IAM role ARN exactly

Invalid Identity Token

Error:

An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation

Solution:

Verify OIDC provider is enabled on your EKS cluster
Check the trust policy condition matches your cluster’s OIDC provider
Ensure the service account namespace and name in the condition are correct

Access Denied to ECR Repository

Error:

denied: User: arn:aws:sts::123456789012:assumed-role/CrossplaneECRRole is not authorized to perform: ecr:BatchGetImage

Solution:

Verify the IAM policy includes ecr:BatchGetImage, ecr:BatchCheckLayerAvailability, and ecr:GetDownloadUrlForLayer
Check the policy’s Resource includes your ECR repository ARN
Ensure the IAM policy is attached to the IAM role

Service Account Annotation Not Applied

If the service account doesn’t have the annotation after installation:

1# Update via Helm
2helm upgrade crossplane \
3  crossplane-stable/crossplane \
4  --namespace crossplane-system \
5  --reuse-values \
6  --set "serviceAccount.customAnnotations.eks\.amazonaws\.com/role-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/CrossplaneECRRole"
7
8# Restart Crossplane
9kubectl rollout restart deployment/crossplane -n crossplane-system

Check Crossplane Logs

View logs for authentication issues:

1kubectl logs -n crossplane-system deployment/crossplane --all-containers -f

Look for:

AWS credential errors
ECR authentication failures
Image pull errors with specific repository paths

Additional Resources

Azure AKS Integration

Configure Crossplane to pull packages from Azure Container Registry (ACR) using Azure Workload Identity.

Prerequisites

An AKS cluster with Workload Identity enabled
Azure CLI installed and configured
kubectl configured to access your AKS cluster
Permissions to create Azure managed identities and role assignments

Enable Workload Identity on AKS

If your AKS cluster doesn’t have Workload Identity enabled, update it:

1export RESOURCE_GROUP=<your-resource-group>
2export CLUSTER_NAME=<your-cluster-name>
3
4az aks update \
5  --resource-group $RESOURCE_GROUP \
6  --name $CLUSTER_NAME \
7  --enable-oidc-issuer \
8  --enable-workload-identity

Get the OIDC issuer URL:

1export AKS_OIDC_ISSUER=$(az aks show \
2  --resource-group $RESOURCE_GROUP \
3  --name $CLUSTER_NAME \
4  --query "oidcIssuerProfile.issuerUrl" \
5  --output tsv)
6
7echo $AKS_OIDC_ISSUER

Create Azure Managed Identity

Create a managed identity for Crossplane:

 1export IDENTITY_NAME=crossplane-acr-identity
 2
 3az identity create \
 4  --name $IDENTITY_NAME \
 5  --resource-group $RESOURCE_GROUP
 6
 7export USER_ASSIGNED_CLIENT_ID=$(az identity show \
 8  --name $IDENTITY_NAME \
 9  --resource-group $RESOURCE_GROUP \
10  --query 'clientId' \
11  --output tsv)
12
13export USER_ASSIGNED_OBJECT_ID=$(az identity show \
14  --name $IDENTITY_NAME \
15  --resource-group $RESOURCE_GROUP \
16  --query 'principalId' \
17  --output tsv)
18
19echo "Client ID: $USER_ASSIGNED_CLIENT_ID"
20echo "Object ID: $USER_ASSIGNED_OBJECT_ID"

Assign ACR Pull Role

Grant the managed identity permission to pull from ACR:

 1export ACR_NAME=<your-acr-name>
 2
 3export ACR_ID=$(az acr show \
 4  --name $ACR_NAME \
 5  --query 'id' \
 6  --output tsv)
 7
 8az role assignment create \
 9  --assignee-object-id $USER_ASSIGNED_OBJECT_ID \
10  --assignee-principal-type ServicePrincipal \
11  --role AcrPull \
12  --scope $ACR_ID

Create Federated Identity Credential

Create a federated identity credential that establishes trust between the managed identity and the Kubernetes service account:

1az identity federated-credential create \
2  --name crossplane-federated-credential \
3  --identity-name $IDENTITY_NAME \
4  --resource-group $RESOURCE_GROUP \
5  --issuer $AKS_OIDC_ISSUER \
6  --subject system:serviceaccount:crossplane-system:crossplane \
7  --audience api://AzureADTokenExchange

Install Crossplane with Workload Identity Configuration

Get the tenant ID:

1export AZURE_TENANT_ID=$(az account show --query tenantId --output tsv)

Install Crossplane with the workload identity annotations and label:

1helm upgrade --install crossplane \
2  crossplane-stable/crossplane \
3  --namespace crossplane-system \
4  --create-namespace \
5  --set "serviceAccount.customAnnotations.azure\.workload\.identity/client-id=$USER_ASSIGNED_CLIENT_ID" \
6  --set "serviceAccount.customAnnotations.azure\.workload\.identity/tenant-id=$AZURE_TENANT_ID" \
7  --set-string 'customLabels.azure\.workload\.identity/use=true'

Note

Azure Workload Identity requires:

Service account annotations for the client ID and tenant ID
Label azure.workload.identity/use: "true" on pods (applied via customLabels)

The customLabels setting applies the label to all Crossplane resources. The Azure Workload Identity webhook uses this label on pods to inject the necessary environment variables and token volumes. Use --set-string to ensure the value is treated as a string rather than a boolean.

Verify Configuration

Check that the service account has the correct annotations:

1kubectl get sa crossplane -n crossplane-system -o yaml

Expected output should include:

1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  annotations:
5    azure.workload.identity/client-id: <client-id>
6    azure.workload.identity/tenant-id: <tenant-id>
7  name: crossplane
8  namespace: crossplane-system

Check that the deployment has the required labels:

1kubectl get deployment crossplane -n crossplane-system -o yaml

Expected output should include:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  labels:
 5    azure.workload.identity/use: "true"
 6  name: crossplane
 7  namespace: crossplane-system
 8spec:
 9  template:
10    metadata:
11      labels:
12        azure.workload.identity/use: "true"

Test Package Installation from ACR

Once configured, you can install Crossplane packages (Providers, Functions, Configurations) from your ACR. Here’s an example using a Provider:

1kubectl apply -f - <<EOF
2apiVersion: pkg.crossplane.io/v1
3kind: Provider
4metadata:
5  name: provider-azure-storage
6spec:
7  package: ${ACR_NAME}.azurecr.io/crossplane/provider-azure-storage:v1.2.0
8EOF

Check the provider installation status:

1kubectl get provider provider-azure-storage
2kubectl describe provider provider-azure-storage

Troubleshooting

Unauthorized: Authentication Required

Error:

unauthorized: authentication required

Solution:

Verify the managed identity has AcrPull role on the ACR
Check the federated credential is configured correctly
Ensure the service account annotations match the managed identity client ID and tenant ID

Failed to Resolve Reference

Error:

failed to resolve reference: failed to fetch oauth token

Solution:

Verify Workload Identity is enabled on the AKS cluster
Check the OIDC issuer URL in the federated credential matches your cluster’s OIDC issuer
Ensure the subject in the federated credential matches system:serviceaccount:crossplane-system:crossplane

Invalid Federated Token

Error:

invalid federated token

Solution:

Verify the federated credential audience is set to api://AzureADTokenExchange
Check that the OIDC issuer URL is correct
Ensure the service account namespace and name match the federated credential subject

Service Account Annotation Not Applied

If the service account doesn’t have the annotations after installation:

 1# Update via Helm
 2helm upgrade crossplane \
 3  crossplane-stable/crossplane \
 4  --namespace crossplane-system \
 5  --reuse-values \
 6  --set "serviceAccount.customAnnotations.azure\.workload\.identity/client-id=$USER_ASSIGNED_CLIENT_ID" \
 7  --set "serviceAccount.customAnnotations.azure\.workload\.identity/tenant-id=$AZURE_TENANT_ID"
 8
 9# Restart Crossplane
10kubectl rollout restart deployment/crossplane -n crossplane-system

Check Crossplane Logs

View logs for authentication issues:

1kubectl logs -n crossplane-system deployment/crossplane --all-containers -f

Look for:

Azure authentication errors
ACR authentication failures
Image pull errors with specific repository paths

Additional Resources

Google Cloud GKE Integration

Configure Crossplane to pull packages from Google Artifact Registry using GKE Workload Identity.

Prerequisites

A GKE cluster with Workload Identity enabled
gcloud CLI installed and configured
kubectl configured to access your GKE cluster
Permissions to create service accounts and IAM bindings

Enable Workload Identity on GKE

If your GKE cluster doesn’t have Workload Identity enabled, create a new cluster with it enabled or update an existing cluster:

New cluster:

1export PROJECT_ID=<your-project-id>
2export CLUSTER_NAME=<your-cluster-name>
3export REGION=<your-region>
4
5gcloud container clusters create $CLUSTER_NAME \
6  --region=$REGION \
7  --workload-pool=${PROJECT_ID}.svc.id.goog

Existing cluster:

1gcloud container clusters update $CLUSTER_NAME \
2  --region=$REGION \
3  --workload-pool=${PROJECT_ID}.svc.id.goog

Create Google Service Account

Create a Google Cloud service account for Crossplane:

1export GSA_NAME=crossplane-gar-sa
2
3gcloud iam service-accounts create $GSA_NAME \
4  --display-name="Crossplane Artifact Registry Service Account" \
5  --project=$PROJECT_ID

Get the full service account email:

1export GSA_EMAIL=${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
2echo $GSA_EMAIL

Grant Artifact Registry Permissions

Grant the service account permissions to read from Artifact Registry:

1gcloud projects add-iam-policy-binding $PROJECT_ID \
2  --member="serviceAccount:${GSA_EMAIL}" \
3  --role="roles/artifactregistry.reader"

For specific repository access, use:

1export REPOSITORY=<your-repository-name>
2export REPOSITORY_LOCATION=<repository-location>
3
4gcloud artifacts repositories add-iam-policy-binding $REPOSITORY \
5  --location=$REPOSITORY_LOCATION \
6  --member="serviceAccount:${GSA_EMAIL}" \
7  --role="roles/artifactregistry.reader"

Create IAM Policy Binding

Create an IAM policy binding between the Google service account and the Kubernetes service account:

1gcloud iam service-accounts add-iam-policy-binding $GSA_EMAIL \
2  --role roles/iam.workloadIdentityUser \
3  --member "serviceAccount:${PROJECT_ID}.svc.id.goog[crossplane-system/crossplane]"

Install Crossplane with Workload Identity Annotation

Install Crossplane with the service account annotation:

1helm upgrade --install crossplane \
2  crossplane-stable/crossplane \
3  --namespace crossplane-system \
4  --create-namespace \
5  --set "serviceAccount.customAnnotations.iam\.gke\.io/gcp-service-account=$GSA_EMAIL"

Verify Configuration

Check that the service account has the correct annotation:

1kubectl get sa crossplane -n crossplane-system -o yaml

Expected output should include:

1apiVersion: v1
2kind: ServiceAccount
3metadata:
4  annotations:
5    iam.gke.io/gcp-service-account: crossplane-gar-sa@project-id.iam.gserviceaccount.com
6  name: crossplane
7  namespace: crossplane-system

Test Package Installation from Artifact Registry

Once configured, you can install Crossplane packages (Providers, Functions, Configurations) from your Artifact Registry. Here’s an example using a Provider:

1kubectl apply -f - <<EOF
2apiVersion: pkg.crossplane.io/v1
3kind: Provider
4metadata:
5  name: provider-gcp-storage
6spec:
7  package: us-docker.pkg.dev/${PROJECT_ID}/crossplane/provider-gcp-storage:v1.2.0
8EOF

Check the provider installation status:

1kubectl get provider provider-gcp-storage
2kubectl describe provider provider-gcp-storage

Troubleshooting

Permission Denied

Error:

PERMISSION_DENIED: Permission denied on resource

Solution:

Verify the Google service account has roles/artifactregistry.reader role
Check the IAM policy binding exists between the Google and Kubernetes service accounts
Ensure the service account annotation matches the Google service account email exactly

Failed to Fetch OAuth Token

Error:

failed to fetch oauth token

Solution:

Verify Workload Identity is enabled on the GKE cluster
Check the IAM policy binding allows the Kubernetes service account to impersonate the Google service account
Ensure the workload pool matches your project: ${PROJECT_ID}.svc.id.goog

Invalid Identity Token

Error:

invalid identity token

Solution:

Verify the IAM policy binding member format: serviceAccount:${PROJECT_ID}.svc.id.goog[crossplane-system/crossplane]
Check that Workload Identity is enabled on the node pool
Ensure the service account annotation is correct

Service Account Annotation Not Applied

If the service account doesn’t have the annotation after installation:

1# Update via Helm
2helm upgrade crossplane \
3  crossplane-stable/crossplane \
4  --namespace crossplane-system \
5  --reuse-values \
6  --set "serviceAccount.customAnnotations.iam\.gke\.io/gcp-service-account=$GSA_EMAIL"
7
8# Restart Crossplane
9kubectl rollout restart deployment/crossplane -n crossplane-system

Verify Workload Identity Configuration

Test the Workload Identity configuration:

1# Check if Workload Identity is enabled on the cluster
2gcloud container clusters describe $CLUSTER_NAME \
3  --region=$REGION \
4  --format="value(workloadIdentityConfig.workloadPool)"
5
6# Verify IAM policy binding
7gcloud iam service-accounts get-iam-policy $GSA_EMAIL

Check Crossplane Logs

View logs for authentication issues:

1kubectl logs -n crossplane-system deployment/crossplane --all-containers -f

Look for:

Google Cloud authentication errors
Artifact Registry authentication failures
Image pull errors with specific repository paths

Additional Resources

Configure After Installation

If Crossplane is already installed, you can update the service account annotations using Helm upgrade with the appropriate --set flags shown in your cloud provider’s tab above. After updating, restart the Crossplane deployment:

1kubectl rollout restart deployment/crossplane -n crossplane-system