registry.k8s.io

Kubernetes's multi-cloud image registry.

For more details on the design and implementation see the source repo: https://github.com/kubernetes/registry.k8s.io

Most of the deployment configuration is in infra/gcp/terraform/k8s-infra-oci-proxy (staging instance for development only) and infra/gcp/terraform/k8s-infra-oci-proxy-prod (production / end users).

Managing Kubernetes container registries

This directory is for tools and things that are used to administer the GCR/AR repositories used to publish official container images for Kubernetes.

• Staging repos
  • Requirements
  • Creating staging repos
  • Enabling automatic builds
  • Image Promoter

Staging repos

Kubernetes subprojects may use a dedicated staging GCP project to build and host container images. We refer to the GCR provided by each staging project as a staging repository. Images are promoted from staging repositories into the main Kubernetes image registry (registry.k8s.io).

Access to each staging project is governed by a Google Group, which grants the ability to manually trigger GCB or push container images in the event that automated builds via something like prow.k8s.io are not setup or are broken.

Requirements

The rule of thumb is that staging repositories should be used to host artifacts produced by code that is part of the Kubernetes project, as defined by presence in one of the project-owned GitHub Organizations.

In other words, code that is not part of the Kubernetes project should not have its artifacts hosted in staging repos. SIG K8s Infra may make exceptions to this policy on a case-by-case basis.

For example:

• CRI-O is not part of the kubernetes project, it does not meet the requirements to get a staging repo
• While etcd and coredns are not part of the kubernetes project, we do bundle them with kubernetes as part of the release, so for this specific case are allowing a staging repo to host them (solely within the context of the kubernetes project)

Creating staging repos

3. Add the project name to the infra.staging.projects list defined in infra/gcp/infra.yaml

4. One your PR merges:

   • a postsubmit job will create the necessary google group
   • whoever approved your PR will run the necessary bash script(s) to create the staging repo

Enabling automatic builds

Once your staging repo is up and running, you can enable automatic build and push. For more info, see the instructions here

NOTE: All sub-projects are strongly encouraged to use this mechanism, though it is not mandatory yet. Over time this will become the primary way to build and push images, and anything else will become exceptional.

Creating image promoter manifests

1. Enable automatic builds and ensure and image has been published by Cloud Build to the staging repo.

2. Create 3 files:

   • images/k8s-staging-<project-name>/OWNERS
   • images/k8s-staging-<project-name>/images.yaml
   • manifests/k8s-staging-<project-name>/promoter-manifest.yaml

The promoter-manifest.yaml file will house the credentials and other registry metadata, whereas the images.yaml file will hold only the image data. You can look at the existing staging repos for examples.

NOTE: For images/k8s-staging-<project-name>/images.yaml, it must not be empty as an image must be proposed for promotion when you are opening this PR.

The separation between promoter-manifest.yaml and images.yaml is there to prevent a single PR from modifying the source registry information as well as the image information. Any changes to the manifest/... folder is expected to be 1-time only during project setup.

Be sure to add the project owners to the images/k8s-staging-<project-name>/OWNERS file to increase the number of people who can approve new images for promotion for your project.

Image Promoter

Image promotion roughly follows the following steps:

1. Push your image to one of the above staging docker repos e.g., gcr.io/k8s-staging-coredns
2. Fork this git repo
3. Add the image into the promoter manifest e.g., if you pushed gcr.io/k8s-staging-coredns/foo:1.3, then add a "foo" image entry into the manifest in images/k8s-staging-coredns/images.yaml
4. Create a PR to this git repo for your changes
5. The PR should trigger a pull-k8sio-cip job which will validate and dry-run your changes; check that the k8s-ci-robot responds 'Job succeeded' for it.
6. Merge the PR. Your image will be promoted by one of two jobs:
   • post-k8sio-image-promo is a postsubmit that runs immediately after merge
   • ci-k8sio-cip is a periodic job that runs every four hours in case there are transient failures of the postsubmit jobs
7. Published images will be available at registry.k8s.io

We've written some tooling to simplify the creation of image promotion pull requests, which is described in detail here.