retagger

A tool to handle the retagging of third-party docker images and make them available in their own registries.

What does retagger do, exactly?

retagger is first and foremost a CircleCI workflow that runs every day at 21:30 UTC and on every merge to the main branch. It utilizes skopeo and custom golang code to take upstream docker images, rename them if necessary, and push them to Giant Swarm's container registries: gsoci.azurecr.io and giantswarm-registry.cn-shanghai.cr.aliyuncs.com. It is capable of working with v1, v2, and OCI registries, as well as retagging multi-architecture images.

💡Please note it is not responsible for pushing images to neither docker.io/giantswarm, nor azurecr.io/giantswarm container registries.

Every image it copies to gsoci.azurecr.io is signed after the copy, see Signed images.

The Aliyun mirror

Upstream images go to gsoci.azurecr.io only. Aliyun is fed from gsoci afterwards, by the mirror-to-aliyun job running on Giant Swarm's self-hosted CircleCI runner inside China (giantswarm/galaxy-runner), where the hop to Aliyun is short:

1. The gsoci jobs record every tag they copied, together with the digest they wrote, in a .copied file in the workspace: skopeo sync --digestfile for the skopeo files, retagger run (one record per executor) for the renamed images files. A copy that failed is not recorded.
2. One retag-aliyun-<file> job per images file reads those records and copies each image from gsoci to Aliyun under the same name and tag. It reads the source by digest, not by tag: the runner reads a different gsoci geo-replica than the one written to, and a lagging replica could otherwise hand it the previous image of a moved tag such as latest. A copy that fails, typically because the image has not replicated yet, is retried a few times (copy_attempts) and otherwise reported.
3. The mirror jobs wait for all the gsoci jobs, and run even when some of them failed: only successful copies are recorded, so one file's failure does not hold back the others.

A tag missing from Aliyun is copied to gsoci again by the next run (the filters check both registries), and so recorded and mirrored again.

How to add your image to the job

You've come to the right place. Pick one of the following methods. For both methods, ensure that the repository exists in the desired container registry first.

Plain copy

You do not need any customizations. Great!

1. Find a skopeo-*.yaml file in images matching your upstream container registry's name. Create a new one, if necessary.
2. Add a tag, SHA, or a semantic version constraint for your image. Refer to Skopeo section or existing files for format definition.
3. If you haven't created a new file, that's it. You're set. Otherwise, continue following the steps.
4. Open CircleCI config, add a filter-skopeo-tags job for your file, and add that job to the requires of validate-filter-errors and retag-gsoci. Then add <your file>.filtered under matrix.parameters.images_file of both the retag-gsoci (retag-registry) and the skopeo retag-aliyun (mirror-to-aliyun) jobs.

Manual copy

You need to copy a few tags, it's a one-off situation. You can use docker pull/docker tag/docker push combination or the below skopeo snippet:

$ skopeo sync --src docker --dest docker --all --keep-going crossplane/crossplane:v1.11.0 docker.io/giantswarm/

Signed images

Every tag retagger copies to gsoci.azurecr.io/giantswarm/ is signed right after the copy with cosign keyless signing: a short-lived certificate from Fulcio for the CircleCI job's OIDC identity, the signature recorded in the Rekor transparency log and stored next to the image as an OCI referrer (cosign v3's bundle format, the same the architect orb uses for images Giant Swarm builds). What is signed is the digest of the manifest the registry serves for the tag, the image index for a multi-architecture image. The signature is made the moment the gsoci copy lands. The copy to Aliyun is made later by a separate job (The Aliyun mirror), so its failure or stall only affects that mirror, never whether the gsoci copy is signed.

The signatures carry this identity:

Issuer	https://oidc.circleci.com
Subject	https://circleci.com/api/v2/projects/0a65bbac-fae8-4aca-bc8c-23d51903006b/pipeline-definitions/b8ddcac6-0699-5c43-a349-7906ac0d4a36

The subject is the CircleCI pipeline definition of this repository, the shape Fulcio issues for every CircleCI job. Images built by the architect orb carry the same shape with their own project, so one keyless attestor admits mirrored and built images alike: issuer https://oidc.circleci.com, subject matching ^https://circleci\.com/api/v2/projects/[a-f0-9-]+/pipeline-definitions/[a-f0-9-]+$.

Verify a mirrored image (cosign v3 or newer):

cosign verify \
  --certificate-oidc-issuer https://oidc.circleci.com \
  --certificate-identity-regexp '^https://circleci\.com/api/v2/projects/[a-f0-9-]+/pipeline-definitions/[a-f0-9-]+$' \
  gsoci.azurecr.io/giantswarm/storage-initializer:v0.20.0

Admit mirrored and built images with one Kyverno verifyImages attestor:

attestors:
  - entries:
      - keyless:
          issuer: https://oidc.circleci.com
          subjectRegExp: ^https://circleci\.com/api/v2/projects/[a-f0-9-]+/pipeline-definitions/[a-f0-9-]+$
          rekor:
            url: https://rekor.sigstore.dev

retagger sign <images file> does the signing (see sign.go): it lists the tags the file governs (a skopeo file the way retagger filter does, a renamed images file from its rules the way retagger run does), resolves each at the registry and signs every digest that does not already verify against the job's own identity, so it is idempotent. The retag-registry job runs it over the .filtered file, the tags the run copied; the retag-renamed-images job runs retagger run --sign, which signs each tag right after its copy. The pipeline parameter sign-all signs every tag the files govern instead (the unfiltered skopeo files, and the renamed-images files sharded by executor like the copies): that is the one-off pass for tags mirrored before signing existed, and the repair of a run whose signing failed. Trigger it on main:

curl -X POST -H "Circle-Token: $CIRCLE_TOKEN" -H "Content-Type: application/json" \
  -d '{"branch": "main", "parameters": {"sign-all": true}}' \
  https://circleci.com/api/v2/project/gh/giantswarm/retagger/pipeline

The copies in the Aliyun registry are not signed, and neither are the Trivy vulnerability databases the retag-image-name jobs copy: OCI artifacts no pod runs. Mirrors in the Docker schema 1 manifest format (a few images from before 2019) cannot carry a cosign signature at all; retagger sign reports them as unsignable and moves on. Only copies that predate the change can exist: etcd:v3.3 is still on gsoci as schema 1, but gsoci now rejects a schema 1 push (PUT manifests: UNKNOWN), so a new one never arrives; giantswarm/bats:0.4.0 was the first to be noticed. Such an image has to be rebuilt as an OCI or Docker schema 2 image before it can be mirrored.

Failed copies

skopeo sync --keep-going logs a copy it could not make and still exits zero. So that a tag missing from the registry does not go unnoticed, the last step of a retag-registry job runs retagger verify-copied <images file> <images file>.copied: it compares the tags the filtered file asked for with the --digestfile record of what landed, names every tag without a line and fails the job. The reason is in the job's skopeo.log artifact. The Aliyun mirror still runs, for what did land.

Image list formats

Skopeo

The basic file format looks as follows: images/skopeo-registry-example-com.yaml

registry.example.com:
    images:
        redis:
            - "1.0"
            - "2.0"
            - "sha256:0000000000000000000000000000000011111111111111111111111111111111"
    images-by-semver:
        alpine: ">= 3.17"

The full specification is available in upstream skopeo-sync docs. Semantic version constraint documentation is available in Masterminds/semver docs.

Custom image builds

Custom container image builds were moved to: https://github.com/giantswarm/custom-container-images.

Renamed images

Renamed images are represented as an array of RenamedImages objects. Please see the definition below:

type RenamedImage struct {
	// Image is the full name of the image to pull.
	// Example: "alpine", "docker.io/giantswarm/app-operator", or
	// "ghcr.io/fluxcd/kustomize-controller"
	Image string `yaml:"image"`
	// TagOrPattern is used to filter image tags. All tags matching the pattern
	// will be retagged. Required if SHA is specified.
	// Example: "v1.[234].*" or ".*-stable"
	TagOrPattern string `yaml:"tag_or_pattern,omitempty"`
	// SHA is used to filter image tags. If SHA is specified, it will take
	// precedence over TagOrPattern. However TagOrPattern is still required!
	// Example: 234cb88d3020898631af0ccbbcca9a66ae7306ecd30c9720690858c1b007d2a0
	SHA string `yaml:"sha,omitempty"`
	// Semver is used to filter image tags by semantic version constraints. All
	// tags satisfying the constraint will be retagged.
	Semver string `yaml:"semver,omitempty"`
	// Filter is a regexp pattern used to extract a part of the tag for Semver
	// comparison. First matched group will be supplied for semver comparison.
	// Example:
	//   Filter: "(.+)-alpine"  ->  Image tag: "3.12-alpine" -> Comparison: "3.12>=3.10"
	//   Semver: ">= 3.10"          Extracted group: "3.12"
	Filter string `yaml:"filter,omitempty"`
	// AddTagSuffix is an extra string to append to the tag.
	// Example: "giantswarm", the tag would become "<tag>-giantswarm"
	AddTagSuffix string `yaml:"add_tag_suffix,omitempty"`
	// OverrideRepoName allows user to rewrite the name of the image entirely.
	// Example: "alpinegit", so "alpine" would become
	// "gsoci.azurecr.io/giantswarm/alpinegit". A slash keeps a nested path:
	// "kagent/controller" becomes "gsoci.azurecr.io/giantswarm/kagent/controller".
	OverrideRepoName string `yaml:"override_repo_name,omitempty"`
	// StripSemverPrefix removes the initial 'v' in 'v1.2.3' if enabled. Works
	// only when Semver is defined.
	StripSemverPrefix bool `yaml:"strip_semver_prefix,omitempty"`
}

Contributing

Please refer to CONTRIBUTING.md.