Signing software artifacts has many obvious benefits such as code integrity or developer (author) authentication. Yet it's oftentimes neglected, creating a software ripe for supply chain attacks. One of the reasons why people can't be bothered to sign their code is that existing tools - such as PGP - aren't very user friendly and require extensive security and/or cryptography knowledge.
Signing software can be easy though thanks to sigstore and its cosign CLI! In this article we will learn how cosign works and integrates with other sigstore components (fulcio and rekor). More importantly, we will learn how to use it to sign container image the easy way, both with and without keys, as well how we can use it to verify produced signatures and integrity of the signed software.
Note: This is a "hands-on" followup to my previous article Sigstore: A Solution to Software Supply Chain Security, which explains what's sigstore and how its components work.
Setting Up
Before we sign anything, we first need all the CLI tools for each of sigstore's components - that is cosign, fulcio and rekor. The first of them - cosign - which we need to actually sign anything, can be installed as binary or as Docker image. For the for first option, download the appropriate binary from release page and put it somewhere in your $PATH. Additionally, considering that we're dealing with security tooling, it's recommended to verify authenticity and integrity of the binary. You can do that using the commands shown on release page.
If you prefer to use Docker image, then you can use the following:
skopeo inspect docker://gcr.io/projectsigstore/cosign:v1.0.0
{
"Name": "gcr.io/projectsigstore/cosign",
"Digest": "sha256:5e88d8f6162c04da4fa7d63b032bac34d8c906b48e88057263d67b059ace7de4",
...
}
docker pull gcr.io/projectsigstore/cosign:v1.0.0
docker run --rm gcr.io/projectsigstore/cosign:v1.0.0
USAGE
cosign [flags] <subcommand>
...
For the second component - fulcio - we won't need to install anything because we will be using the public instance of fulcio. The public-good service is available at https://fulcio.sigstore.dev/api/v1 and API documentation can be found here.
Lastly, there's rekor and its CLI called rekor-cli. Same as with fulcio, we don't need to install rekor as it's available at https://rekor.sigstore.dev along with the Swagger definition here.
We will however want to install the CLI so that we interact with the rekor server. The binaries are available in GitHub release page. If you're on linux you can use the following:
wget -O rekor-cli https://github.com/sigstore/rekor/releases/download/v0.3.0/rekor-cli-linux-amd64
chmod +x rekor-cli
# Move it into $PATH directory...
./rekor-cli
Rekor command line interface tool
Usage:
rekor [command]
...
And again, as mentioned with cosign, you should be careful with what binaries you're using. Therefore you might want to verify rekor-cli binary using the process outlined here.
The Hard Way
With all the tools ready, we can start signing artifacts. To get better understanding about what goes on under covers, we will first try doing it the "hard way", that is - without all the fancy tools.
First we will need an artifact. For this demo we will use "hello world" Docker image created using following Dockerfile:
FROM alpine:3.14
ENTRYPOINT ["echo", "Hello sigstore"]
We however cannot sign the image itself, instead we will sign its digest:
# Generate artifact
docker build -t dockerhub-username/sigstore-hello .
# Generate artifact digest for signing
cosign generate martinheinz/sigstore-hello > artifact
Next we need an ephemeral keypair to sign the digest with. We can use cosign commands for this, but considering that this is the "hard way", let's use openssl directly:
openssl ecparam -genkey -name prime256v1 > ec_private.pem # Create keypair, same as `cosign generate-key-pair`
openssl ec -in ec_private.pem -pubout > ec_public.pem # Extract public key, same as `cosign public-key`
Now we're ready to sign it and while we're at it we can also verify the signature:
# Sign artifact digest, same as `cosign sign`
openssl dgst -sha256 -sign ec_private.pem artifact > artifact.sig
# Verify using public key
openssl dgst -sha256 -verify ec_public.pem -signature artifact.sig artifact
Verified OK
Now that we signed the artifact with our private key, we want to have a proof that we were the ones who really did it. For this we need code signing certificate from fulcio. To get it, we have to authenticate with OIDC provider to get an ID token, which serves as proof of our identity for fulcio.
After that, we sign our email address which we used to authenticate using the previously used private key. We do this to prove that we have possession of the private key at the time of signing.
Finally, we ask fulcio for code signing certificate, by giving it ID token as form of authorization, the signed email address and our public key:
# ... Get token from OIDC provider
# ... Store ID token in `id_token` file
# Sign email address (to prove possession of private key)
echo "martin7.heinz@gmail.com" > email
openssl dgst -sha256 -sign ec_private.pem email > email.sig
# Submit token, public key and signed email to fulcio
curl -X POST "https://fulcio.sigstore.dev/api/v1/signingCert" \
-H "Authorization: Bearer $(cat id_token)" \
-H "accept: application/pem-certificate-chain" \
-H "Content-Type: application/json" -d \
{
"publicKey": {
"content": "$(base64 ec_public.pem)",
"algorithm": "ecdsa"
},
"signedEmailAddress": "$(base64 email.sig)"
}
One problem with this "hard way" approach is that it's not really feasible to simulate the authentication and retrieval of ID token. Therefore, in the above snippet this step is omitted and we skip directly to submitting everything to fulcio.
Alternatively, you could also skip the interaction with fulcio entirely and use your public key instead. This approach is shown in https://github.com/sigstore/rekor/blob/main/types.md#pkixx509.
Next we can proceed with uploading the record to the transparency log (rekor). Here we show both the option with our public key and signing certificate from fulcio. When using the certificate from fulcio, we can also delete the keypair as we no longer need it:
# Delete keypair (if using signing certificate from fulcio)
rm -rf ec_private.pem ec_public.pem
# With our public key
rekor-cli upload --artifact artifact --signature artifact.sig --public-key=ec_public.pem --pki-format=x509
...
# With cert from fulcio
rekor-cli upload --artifact artifact --signatire artifact.sig --public-key sigingCertChain.pem --pki-format x509
Created entry at index 33612, available at:
https://rekor.sigstore.dev/api/v1/log/entries/2f77b399fd8a162f44c75c596fb0e5917ed2f314348e135874fae9e14eff69e3
# Inspect entry
curl https://rekor.sigstore.dev/api/v1/log/entries/2f77b399fd8a162f44c75c596fb0e5917ed2f314348e135874fae9e14eff69e3 | jq .
...
rekor-cli get --uuid=2f77b399fd8a162f44c75c596fb0e5917ed2f314348e135874fae9e14eff69e3
...
In addition to the upload we can also check presence of the record in transparency log. Above snippet uses both rekor-cli and curl to directly access the public API.
All that's left to do is upload the signature to the registry to be stored alongside container image:
# Upload to Docker Hub
cosign upload blob -f artifact.sig index.docker.io/martinheinz/sigstore-hello:new-signature.sig
Uploading file from [artifact.sig] to [index.docker.io/martinheinz/sigstore-hello:new-signature.sig]
with media type [application/octet-stream]
File [artifact.sig] is available directly at [index.docker.io/v2/martinheinz/sigstore-hello/blobs/sha256:...]
Uploaded image to:
index.docker.io/martinheinz/sigstore-hello@sha256:4c2f015295318a35b6096d6a971e42e5b5afb237dfc4fdf44364502a2e7064a1
That's it. We have signed our image and added record of it to transparency log. This approach would work, but no one probably wants to do this on a daily basis, so let's see how the proper tools can make this easy.
The Easy Way
The "hard way" wasn't really hard, but it gets much easier if we use the tools provided:
cosign generate-key-pair
Enter password for private key:
Enter again:
Private key written to cosign.key
Public key written to cosign.pub
# We already uploaded signature in previous step so upload is set to false here
cosign sign -key cosign.key --upload=false martinheinz/sigstore-hello > file.sig
Enter password for private key:
...
# You can later upload the signature
cosign attach signature -signature file.sig martinheinz/sigstore-hello
All we need to do is generate a keypair and then sign the artifact. Upon signing, cosign automatically uploads the signature to the registry where the image is located. In the above example we chose not to upload the signature and just save it to a file, because we did sign it in the previous section already. If we later decided to upload it anyway, then we can do it with cosign attach as shown above.
It's also worth pointing out, that as of right now (cosign version 1.0), the above snippet will not upload the data to rekor transparency log, for that to work, we would need to set COSIGN_EXPERIMENTAL=1, so for example COSIGN_EXPERIMENTAL=1 cosign sign -key cosign.key ....
There are also other ways to use cosign to sign artifacts depending on your use case and workflow. These are described in detail in usage page in GitHub.
"Keyless"
Even easier than the easy way is using the "keyless" method where only ephemeral keys are used, meaning you don't need to generate and maintain your own keys:
COSIGN_EXPERIMENTAL=1 cosign sign \
-oidc-issuer "https://oauth2.sigstore.dev/auth" \
-fulcio-url "https://fulcio.sigstore.dev" \
-rekor-url "https://rekor.sigstore.dev" \
docker.io/martinheinz/sigstore-hello:latest
Generating ephemeral keys...
Retrieving signed certificate...
Your browser will now be opened to:
https://oauth2.sigstore.dev/auth/auth?access_type=online&client_id=sigstore&code_challenge=...
tlog entry created with index: 33692
Pushing signature to: index.docker.io/martinheinz/sigstore-hello:sha256-af5909c54fe66d03dda...5d8d60d85491.sig
All we need to do is run cosign sign with COSIGN_EXPERIMENTAL set to 1 while at the same time omitting the -key argument. In the above example we also specified endpoints of OIDC provider, fulcio server and rekor server - these are the default values of the public-good services provided by sigstore, so they can be omitted, but are shown here for clarity and to highlight which services are being accessed/used. You could also replace those with your own instances - that would make sense if you wanted to run everything behind a firewall.
Verify Everything
Now that we signed the artifact in all the ways possible we should also try verifying it, otherwise what would be the point of signing it in the first place, right?
First let's take the outputs of signing the image digest the "hard way". For that we can use rekor-cli:
rekor-cli verify --artifact artifact --signature artifact.sig --public-key ec_public.pem --pki-format x509
rekor-cli verify --artifact artifact --signature artifact.sig --public-key sigingCertChain.pem --pki-format x509
Here we have 2 cases - if we signed the artifact with our public key, then we use that when verifying. On the other hand if we used the signing cert provided by fulcio we would use that in place of the public key.
Next up is the verification using cosign which is suitable for the basic signing with keys. All we need to do is run cosign verify providing the key and image URL:
cosign verify -key cosign.pub docker.io/martinheinz/sigstore-hello:latest | jq .
Finally, for the keyless method - we can do essentially the same as above, but we need to add the experimental flag and we can skip the key argument:
COSIGN_EXPERIMENTAL=1 cosign verify docker.io/martinheinz/sigstore-hello:latest
Verification for docker.io/martinheinz/sigstore-hello:latest --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- Any certificates were verified against the Fulcio roots.
...
Closing Thoughts
In this article I tried to outline and explain the basic use cases and approaches for signing container images using sigstore and more specifically cosign. There are however, many more options and features of cosign which might be useful to you, such as working with other types of artifacts, using hardware tokens or signing git commits, so I encourage you to mess with the tool and see what else you can use it for. A lot of these options are described in very well written usage documentation here, so make sure to check that out too.
Also, if you want to dig even deeper, you can checkout "sigstore the hard way", which is a guide to setting everything up, for scratch - including fulcio CA and rekor transparency log server.