折腾 Zhēteng

Google Cloud Kubernetes Binary (Image) Authorization

Created: Modified: 12 min read

NOTE

This is a walkthrough I wrote in 2019 and migrated from my Notion. Things may have changed a lot since then.

This post is based on the official tutorial: Securing Your GKE Deployments with Binary Authorization

Why

Verify the container image is verified (or comply the whatever policy) before deploying it on a Kubenete cluster

This implies, you must have a GKE cluster first

  • Create a cluster: gcloud beta container clusters create --enable-binauthz --zone us-central1-a binauthz-codelab
    • Assume we already have a project, this can be done in the cloud shell of that project.
    • The cluster name will be binauthz-codelab
    • not sure about the beta, not sure if it is GA now
    • looks like --enable-binauthz is necessary, but not sure what it is doing
    • not sure if --zone us-central1-a is required.
  • Get the created cluster to the context for kubectl by: gcloud container clusters get-credentials binauthz-codelab --zone us-central1-a
    • Then, kubectl will always be processing that GKE cluster binauthz-codelab
  • Assume we have a docker image: FROM alphine; CMD tail -f /dev/null. This image will simply keep waiting forever.
    • we need to push it the google container registry (GCR), so that GKE can pull it from there.
      • To do so, we need to name our image correctly
        • like: docker build -t [us.gcr.io/${project_id}/hello-world](http://us.gcr.io/${project_id}/hello-world) ./
        • Note the [us.gcr.io/{project_id}/hello-world) does not have a *tag, (e.g: xxx:tag),* GCR will make add a :latest at the end automatically in the registry.
    • gcloud auth configure-docker —quiet
    • docker push us.gcr.io/${project_id}/hello-world
    • Then we should be able see it in the UI
  • Now we need to deploy the docker to our GKE cluster
    • kubectl create deployment hello-world --image=us.gcr.io/${project_id}/hello-word
    • kubectl get pods to check the pods are running

so we can see, here we can deploy whatever an image from our registry

But, when things go wild, we can’t be sure all the images in the registry are verified to be deployed, for example, a dev image to prod cluster.

so we need to verify before use, and now comes the binary authorization

Background

  • Many GCP features are implemented, and made available as an API, so we can find them in the API marketplace
    • Each enabled API will add a service account (most the case, not thoroughly checked)
      • Form: “service-{PROJECT_NUMBER}@gcp-sa-{service name}.iam.gserviceaccount.com”
      • Why: Sometime (actually very commonly) the service (the API) want to read data from another service (API) to achieve some functionalities, we have to grant the permission of accessing the data from the second service to the service account created by the first service.
  • Most of the gcloud commands should be just a wrapper of their corresponding REST requests. So, nearly all the gcloud commands below should all have a curl replacement. Also true for the opposite way.

APIs Required for the Codelab proj

  • Container Analysis API container.googleapis.com
    • For everything about GKE cluster
    • And Docker container image registry
  • Binary Authorization API binaryauthorization.googleapis.com
    • Main role here
    • This will add a service-{PROJECT_NUMBER}@gcp-sa-binaryauthorization.iam.gserviceaccount.com
  • KMS API cloudkms.googleapis.com
    • To manage (generate, use, refer?) cryptographic signing keys.

Enable the APIs:

## enable GKE to create and manage your cluster
gcloud services enable container.googleapis.com
## enable BinAuthz to manage a policy on the cluster
gcloud services enable binaryauthorization.googleapis.com
## enable KMS to manage cryptographic signing keys
gcloud services enable cloudkms.googleapis.com

Moving pieces in this game

1. Container Analysis API

This is the guy that actually prevents an arbitrary image being deployed

This is part of the container API

But might be relevant to the --enable-binauthz flag.

It has a ‘note’ service. We can create a note.

cat > ./create_note_request.json << EOM
{
  "attestation": {
    "hint": {
      "human_readable_name": "This note represents an attestation authority"
    }
  }
}
EOM
 
NOTE_ID=my-attestor-note
 
curl -vvv -X POST \
    -H "Content-Type: application/json"  \
    -H "Authorization: Bearer $(gcloud auth print-access-token)"  \
    --data-binary @./create_note_request.json  \
    "https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/?noteId=${NOTE_ID}"
 
## Verify by:
curl -vvv  \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    "https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/${NOTE_ID}"

Note that:

  • JSON payload itself is the note, although the codelab calls it create_note_request.json
  • The NOTE_ID will be used later. And this is the only identifier of the note.
  • The curl to the containeranalysis API. There is only REST api and client API. There is no gcloud container xxx counterpart of it, not even in gcloud beta container

The note must also be registered with the binary authorization service (detailed later)

2. Binary Authorization

First, we need an attestor

ATTESTOR_ID=my-binauthz-attestor
 
gcloud container binauthz attestors create $ATTESTOR_ID \
    --attestation-authority-note=$NOTE_ID \
    --attestation-authority-note-project=${PROJECT_ID}
 
## verify by:
gcloud container binauthz attestors list

Second we need to make sure the binary authz service account can access data from the container analysis service (grant IAM permission)

PROJECT_NUMBER=$(gcloud projects describe "${PROJECT_ID}"  --format="value(projectNumber)")
BINAUTHZ_SA_EMAIL="service-${PROJECT_NUMBER}@gcp-sa-binaryauthorization.iam.gserviceaccount.com"
 
cat > ./iam_request.json << EOM
{
  'resource': 'projects/${PROJECT_ID}/notes/${NOTE_ID}',
  'policy': {
    'bindings': [
      {
        'role': 'roles/containeranalysis.notes.occurrences.viewer',
        'members': [
          'serviceAccount:${BINAUTHZ_SA_EMAIL}'
        ]
      }
    ]
  }
}
EOM
 
## to grant the necessary IAM role
curl -X POST  \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    --data-binary @./iam_request.json \
    "https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/${NOTE_ID}:setIamPolicy"

3. KMP, create and keys

We need a asymmetric-signing pair, create is with KMS and link it with our attestor.

KEY_LOCATION=global
KEYRING=binauthz-keys
KEY_NAME=codelab-key
KEY_VERSION=1
 
## Key ring is created first. We will needs a location
gcloud kms keyrings create "${KEYRING}" --location="${KEY_LOCATION}"
 
## Creates a key pair from the key ring and location. And the key
## will be identified as KEY_NAME
gcloud kms keys create "${KEY_NAME}" \
    --keyring="${KEYRING}" --location="${KEY_LOCATION}" \
    --purpose asymmetric-signing  --default-algorithm="ec-sign-p256-sha256"
 
## Link the attestor created by binary authz with the key
## Not sure why we need that many info about the key though.
gcloud beta container binauthz attestors public-keys add  \
    --attestor="${ATTESTOR_ID}"  \
    --keyversion-project="${PROJECT_ID}"  \
    --keyversion-location="${KEY_LOCATION}" \
    --keyversion-keyring="${KEYRING}" \
    --keyversion-key="${KEY_NAME}" \
    --keyversion="${KEY_VERSION}"

Note that, we can check the created key in the page: https://pantheon.corp.google.com/security/kms

After linking, we should be able to see we now have +1 public key:

gcloud container binauthz attestors list

4. Real game starts by an attestor signing an image

First, always use digest for signing and prod deployment, do not use tag

tag can point to different builds (and it is supposed to do so)

digest is unique for each build

So we now needs to get the digest from a build (which can be identified by tag like :latest)

## container_path should be us.gcr.io/${project_id}/hello-world
DIGEST=$(gcloud container images describe ${CONTAINER_PATH}:latest \
    --format='get(image_summary.digest)')

Second, sign

gcloud beta container binauthz attestations sign-and-create  \
    --artifact-url="${CONTAINER_PATH}@${DIGEST}" \
    --attestor="${ATTESTOR_ID}" \
    --attestor-project="${PROJECT_ID}" \
    --keyversion-project="${PROJECT_ID}" \
    --keyversion-location="${KEY_LOCATION}" \
    --keyversion-keyring="${KEYRING}" \
    --keyversion-key="${KEY_NAME}" \
    --keyversion="${KEY_VERSION}"

Note:

  • The artifact-url points to the image: us.gcr.io/${project_id}/hello-world@{digest}
    • Note that, this is kind of a full url of the image with digest
  • The way to identify a build is: us.gcr.io/{digest}
  • Needs the Attestor’s ID
    • And needs the KEY_RING and KEY_NAME, note that one attestor can have multiple public keys associated, so we need to know which one to use here.
    • But we still needs a lot of other information
      • keyversion-project
      • attestor-project
        • So the attestor project can be separated?!
      • keyversion-location
        • This is user defined before
      • keyversion
        • This is user defined before

This makes an attestation

To verify attestation:

gcloud container binauthz attestations list \
   --attestor=$ATTESTOR_ID --attestor-project=${PROJECT_ID}

5. Policy should match. If not, update it

We need to make sure we have a policy defined on binary authz service matching our attestation mechanism

cat << EOF > updated_policy.yaml
    globalPolicyEvaluationMode: ENABLE
    defaultAdmissionRule:
      evaluationMode: REQUIRE_ATTESTATION
      enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG
      requireAttestationsBy:
      - projects/${PROJECT_ID}/attestors/${ATTESTOR_ID}
EOF
 
gcloud container binauthz policy import updated_policy.yaml

Note that:

  • The requireAttestationBy: field
    • It specifies the required attestor
    • It is a list, so, new problem: AND OR?
  • The evaluationMode is REQUIRE_ATTESTATION
    • Must be a pre-set rule
  • defaultAdmissionRule
    • considering the final command: gcloud container binauthz policy
      • This

6. Create deployment with attested image

kubectl create deployment hello-world-signed --image="${CONTAINER_PATH}@${DIGEST}"

This will get the deployment created.

kubectl get pods

7. Cleanup

Delete cluster gcloud container clusters delete binauthz-codelab --zone us-central1-a

Delete image: gcloud container images delete ${container_path}@${digest} --force-delete-tags

Delete attestor: gcloud container binauthz attestors delete my-binauthz-attestor

Delete container analysis note:

curl -vvv -X DELETE  \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    "https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/${NOTE_ID}"

Command list

gcloud (or gcloud beta)

  • container
    • clusters
      • create
        • gcloud beta container clusters create --enable-binauthz --zone us-central1-a binauthz-codelab
        • To create a GKE cluster
      • get-credentials
        • gcloud container clusters get-credentials binauthz-codelab --zone us-central1-a
        • To set the GKE cluster to current context
      • delete
    • binauthz
      • attestors
        • create
          • gcloud container binauthz attestors create $ATTESTOR_ID \ --attestation-authority-note=$NOTE_ID \ --attestation-authority-note-project=${PROJECT_ID}
          • Identified by ATTESTOR_ID
          • Linked with note by NOTE_ID
            • Also needs the owner project of the note (?)
        • public-keys
          • add
            • gcloud beta container binauthz attestors public-keys add \ --attestor="${ATTESTOR_ID}" \ --keyversion-project="${PROJECT_ID}" \ --keyversion-location="${KEY_LOCATION}" \ --keyversion-keyring="${KEYRING}" \ --keyversion-key="${KEY_NAME}" \ --keyversion="${KEY_VERSION}"
            • To add a key (generated by KMS) to an attestor
        • list
          • gcloud container binauthz attestors list
          • To list created attestors, also show the number of public keys of each attestors
        • delete
          • gcloud container binauthz attestors delete my-binauthz-attestor
      • attestations
        • sign-and-create
          • gcloud beta container binauthz attestations sign-and-create \ --artifact-url="${CONTAINER_PATH}@${DIGEST}" \ --attestor="${ATTESTOR_ID}" \ --attestor-project="${PROJECT_ID}" \ --keyversion-project="${PROJECT_ID}" \ --keyversion-location="${KEY_LOCATION}" \ --keyversion-keyring="${KEYRING}" \ --keyversion-key="${KEY_NAME}" \ --keyversion="${KEY_VERSION}"
            • I guess --attestor-project is the owner project of ATTESTOR_ID
              • because ATTESTOR_ID is definitely not universally unique
            • and --keyversion-project is the owner project of the KMS key info (key name, key ring, etc)
          • To sign an image, and creates an attestation
        • list
          • gcloud container binauthz attestations list \ --attestor=$ATTESTOR_ID --attestor-project=${PROJECT_ID}
      • policy
        • import
          • gcloud container binauthz policy import updated_policy.yaml
          • To import/update the policy
            • Note that, there is only one policy
    • images
      • describe
        • gcloud container images describe ${CONTAINER_PATH}:latest \ --format='get(image_summary.digest)
          • This prints out the image digest
      • delete
        • gcloud container images delete ${container_path}@${digest} --force-delete-tags
  • kms
    • keyrings
      • create
        • gcloud kms keyrings create "${KEYRING}" --location="${KEY_LOCATION}"
        • To create a keyring
    • keys
      • create
        • gcloud kms keys create "${KEY_NAME}" \ --keyring="${KEYRING}" --location="${KEY_LOCATION}" \ --purpose asymmetric-signing --default-algorithm="ec-sign-p256-sha256"
        • To create a key from key ring
  • auth
    • configure-docker
      • gcloud auth configure-docker —quiet
      • To set some docker context, so that docker push xxx will work correct
  • services
    • enable
      • gcloud services enable container.googleapis.com
      • To enable some API services

kubectl

  • create
    • deployment
      • kubectl create deployment hello-world --image=us.gcr.io/${project_id}/hello-word
      • kubectl create deployment hello-world-signed --image="${CONTAINER_PATH}@${DIGEST}"
      • To deploy an image to the current cluster
  • get
    • pods
      • kubectl get pods
      • To get the deployed pods in the current cluster

curl

  • post
    • containeranalysis.googleapis.com/v1/projects/{note_id}
      • To create a note, there is no gcloud comamnd counterpart
  • get
    • containeranalysis.googleapis.com/v1/projects/{note_id}
      • To retrieve the created note
  • delete
    • containeranalysis.googleapis.com/v1/projects/{note_id}
      • To delete the created note

docker

  • push
    • docker push us.gcr.io/${project_id}/hello-world
  • build
    • docker build -t [us.gcr.io/${project_id}/hello-world](http://us.gcr.io/${project_id}/hello-world) ./

Interconnections

  • Note is just a note ( I guess). It does not enable the verification. It probably will just add a comment or log somewhere when creating deployment from an attested image
  • When an attestor is linked with a note, it probably will be able to add the note to its log or the image owner project’s log with the content in the note, when an attestation is created with the image. To label an human readable label to say the image is attested.
  • Policy is the real enabler of the checking, more specifically, it is the gobalPolicyEvluationMode: ENABLE
  • Policy can require attestation from multiple attestors from multiple projects (see the project ID and attestor ID in the yaml file)
    • I feel like there are missing pieces, cause fetching the attestor from another project should require some permissions
    • Note that the roles/containeranalysis.notes.occurrences.viewer should not be relevant to this (I guess)
  • Attestors are granted with keys generated from KMS
    • But where is the permission, I don’t see anywhere saying the attestor’s owner project or its service account has the permission to get the key from the KMS owner project
      • Well, these are Google-managed service accounts, their concrete permissions are hidden, but pretty much all needed to do whatever the API supposed to do.

GCP Projects (Not confirmed yet)

Cluster Project (aka. Deployment Project)

This is the project pulling the image (MLLP docker image) from the Container Project

The Cluster must be created with --enable-binauthz option gcloud beta container clusters create --enable-binauthz --zone <zone> <cluster name>

  • Or it can be updated with the binauthz support

This project stores the deployment policy.

Needs to confirm:

  • Needs to enable containeranalysis and binaryauthorization API?
  • Needs to enable container.googleapis.com?

Container Project (Aka. Builder Project)

This is the the project contains the built docker image

To use binary authorization feature, we do NOT need to change anything on this project (in many cases, this project is not managed by the image users)

Attestor Project

Note that, each attestation is actually a container analysis note occurrence. There is actually no real ‘attestation’ type.

Needs container analysis API, because container analysis notes are stored in this project. Analysis note creation is done by REST request, not gcloud commands.

Needs binary authorization API. Only then we have a service account: service-${ATTESTOR_PROJECT_NUMBER}@gcp-sa-binaryauthorization.iam.gserviceaccount.com and we can bind role: roles/containeranalysis.notes.occurrences.viewer to it.

Usually, the signing keys are also stored in this project (though I think the key can be stored in another Key Project, but in practice, I don’t see much benefit of doing like so…)

The attestors are created and stored in this project, they should have their public key with them registered.

Note Project

Needs container analysis API, analysis note and note occurrences are store in this project

Attestation Project

The project that stores attestations.

KMS Key Project

If using the GCP KMS service to provide the PKIX key, we can use the GCP KMS service

API Explanation

Container Analysis

Store Container Analysis Note and Note Occurrences

  • Note Project

Binary Authorization

Create attestor, attestation, apply policy

  • Attestor Project
    • To create attestor
  • Attestation Project
    • To create attestation
  • Deployer Project
    • To import and use policy, enable binary authorization

KMS

Create cryptographic keys

  • KMS Project

Project and used APIs

Extra Permission Settings

Attestor To Access Container Analysis Note

Account: Attestor’s binary authorization service account

Target Resource: projects/{NOTE_ID}

Role: roles/containeranalysis.notes.occurrences.viewer

Deployer To Attestor

Account: Deployer’s binary authorization service account

Target Resource: projects/{ATTESTOR_ID}

Role: roles/binaryauthorization.attestorsVerifier

Set on ${ATTESTOR_PROJECT} with gcloud cmd

Multiple Project Setup

Multiple Project Setup

References

Binary Authorization Document:

https://cloud.google.com/binary-authorization/docs/

Official CodeLab of Binary Authorization (Single Project Setup):

https://codelabs.developers.google.com/codelabs/cloud-binauthz-intro/#0

Official Multi-Project Setup Document:

https://cloud.google.com/binary-authorization/docs/multi-project-setup-cli

Graph View