A lot of people assume that Docker images and containers are secure by default, which - unfortunately - is not the case. There are quite a few things that effect security of your Docker images. Whether it's packages installed in the image, libraries used by your application or even the base image - all these components might introduces vulnerability into your application. A lot of these problems are easily avoidable, though...
Anchore & Clair
- Anchore Engine: Anchore is a centralized service for inspection, analysis and certification of container image. It scans images using vulnerability data (feeds) from OS vendors like Red Hat, Debian or Alpine. For non-OS data it uses NVD (National Vulnerability Database), which includes vulnerabilities for RPM, Deb, APK as well as Python (PIP), Ruby Gems, etc.
- Clair: Clair is a static analyzer developed by CoreOS for Docker and APPC containers. It uses vulnerability metadata from similar sources as Anchore - Red Hat Security Data, NVD, Ubuntu CVE Tracker, Alpine SecDB, Debian Security Bug Tracker, etc.
Now that we know the tools we want to use, it's time spin them up. Both Anchore and Clair include various integration and can be deployed to Kubernetes or OpenShift, but for the purpose of this demonstration, we will set them up using
docker-compose on local machine:
To setup Anchore run following:
mkdir ~/aevolume cd ~/aevolume docker pull docker.io/anchore/anchore-engine:latest docker create --name ae docker.io/anchore/anchore-engine:latest docker cp ae:/docker-compose.yaml ~/aevolume/docker-compose.yaml docker rm ae docker-compose pull docker-compose up -d export ANCHORE_CLI_USER=admin export ANCHORE_CLI_PASS=foobar docker run --net=host -e ANCHORE_CLI_URL=http://localhost:8228/v1/ -it anchore/engine-cli
To setup Clair run following:
# Download Clair Scanner from https://github.com/arminc/clair-scanner/releases chmod +x clair-scanner docker run -p 5432:5432 -d --name db arminc/clair-db:$(date +%F) docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.6
And with that we are ready to analyze!
Note: We will come back to Clair little later in the article.
Check Image for Vulnerabilities
Let's start with Anchore and basic Debian image. What we need to do, to get our image analyzed is to
add it and
wait for the analysis to complete:
# inside anchore-cli Docker container ~ $ anchore-cli image add docker.io/library/debian:latest Image Digest: sha256:121dd2a723be1c8aa8b116684d66157c93c801f2f5107b60287937e88c13ab89 Parent Digest: sha256:a63d0b2ecbd723da612abf0a8bdb594ee78f18f691d7dc652ac305a490c9b71a Analysis Status: analyzed Image Type: docker Analyzed At: 2020-03-07T10:46:20Z Image ID: 971452c943760ab769134f22db8d3381b09ea000a6c459fbfa3603bb99115f62 Dockerfile Mode: Guessed Distro: debian Distro Version: 10 Size: 126607360 Architecture: amd64 Layer Count: 1 Full Tag: docker.io/library/debian:latest Tag Detected At: 2020-03-07T10:45:48Z ~ $ anchore-cli image wait docker.io/library/debian:latest Status: analyzing Waiting 5.0 seconds for next retry. ... ~ $ anchore-cli image list Full Tag Image Digest Analysis Status docker.io/library/debian:latest sha256:121dd2a723be1c8aa8b116684d66157c93c801f2f5107b60287937e88c13ab89 analyzed
With the image analyzed let's see what vulnerabilities were found:
~ $ anchore-cli image vuln docker.io/library/debian:latest all Vulnerability ID Package Severity Fix CVE Refs Vulnerability URL CVE-2005-2541 tar-1.30+dfsg-6 Negligible None CVE-2005-2541 https://security-tracker.debian.org/tracker/CVE-2005-2541 CVE-2007-5686 login-1:4.5-1.1 Negligible None CVE-2007-5686 https://security-tracker.debian.org/tracker/CVE-2007-5686 ... ~ $ anchore-cli evaluate check docker.io/library/debian:latest Image Digest: sha256:121dd2a723be1c8aa8b116684d66157c93c801f2f5107b60287937e88c13ab89 Full Tag: docker.io/library/debian:latest Status: pass Last Eval: 2020-03-14T13:25:24Z Policy ID: 2c53a13c-1765-11e8-82ef-23527761d060
We first run
image vuln command with
all flag to list both OS and package vulnerabilities present in the image. All of those are
Unknown, so that's pretty good. Next, we run
evaluate check command to see whether this image passes the default policy check and as you can see above (
Status: pass), it does. These kinds of base images generally perform pretty well in this regard as they are widely used and therefore under a lot of scrutiny.
But, what about simple simple "Hello World" Python application built using Python 3 Debian Buster image? Let's look at the Dockerfile first:
# debian.Dockerfile FROM python:3-slim AS build-env ADD . /app WORKDIR /app FROM python:3-buster COPY --from=build-env /app /app WORKDIR /app CMD ["python", "hello.py", "/etc"]
Nothing that screams "vulnerable" and "insecure", right? So, let's built it and analyze it:
~ $ docker build -f debian.Dockerfile -t martinheinz/debian-python-anchore . ~ $ docker run martinheinz/debian-python-anchore:latest ~ $ docker push martinheinz/debian-python-anchore:latest # From anchore Docker CLI ~ $ anchore-cli image add martinheinz/debian-python-anchore:latest ~ $ anchore-cli image wait martinheinz/debian-python-anchore:latest ~ $ anchore-cli image list Full Tag Image Digest Analysis Status docker.io/library/debian:latest sha256:121dd2a723be1c8aa8b116684d66157c93c801f2f5107b60287937e88c13ab89 analyzed docker.io/martinheinz/debian-python-anchore:latest sha256:59dff8bdf4af5cd8e9ba0754d25a43a96dfb47b46b771549a0d79d35bc3cc1aa analyzed
I first built the image using the above
debian.Dockerfile and simple
hello.py. I then pushed it to Docker Hub, from where it was added to Anchore and finally analyzed. Now we can check out the results:
~ $ anchore-cli image vuln docker.io/martinheinz/debian-python-anchore:latest os Vulnerability ID Package Severity Fix CVE Refs Vulnerability URL CVE-2007-3476 libwmf-dev-0.2.8.4-14 Low None CVE-2007-3476 https://security-tracker.debian.org/tracker/CVE-2007-3476 CVE-2007-3476 libwmf0.2-7-0.2.8.4-14 Low None CVE-2007-3476 https://security-tracker.debian.org/tracker/CVE-2007-3476 CVE-2007-3477 libwmf-dev-0.2.8.4-14 Low None CVE-2007-3477 https://security-tracker.debian.org/tracker/CVE-2007-3477 CVE-2007-3477 libwmf0.2-7-0.2.8.4-14 Low None CVE-2007-3477 https://security-tracker.debian.org/tracker/CVE-2007-3477 CVE-2016-8660 linux-libc-dev-4.19.98-1 Low None CVE-2016-8660 https://security-tracker.debian.org/tracker/CVE-2016-8660 CVE-2007-3996 libwmf-dev-0.2.8.4-14 Medium None CVE-2007-3996 https://security-tracker.debian.org/tracker/CVE-2007-3996 CVE-2007-3996 libwmf0.2-7-0.2.8.4-14 Medium None CVE-2007-3996 https://security-tracker.debian.org/tracker/CVE-2007-3996 CVE-2009-3546 libwmf-dev-0.2.8.4-14 Medium None CVE-2009-3546 https://security-tracker.debian.org/tracker/CVE-2009-3546 CVE-2009-3546 libwmf0.2-7-0.2.8.4-14 Medium None CVE-2009-3546 https://security-tracker.debian.org/tracker/CVE-2009-3546 ... ~ $ anchore-cli image vuln docker.io/martinheinz/debian-python-anchore:latest os | wc -l 1056 ~ $ anchore-cli evaluate check docker.io/martinheinz/debian-python-anchore:latest Image Digest: sha256:59dff8bdf4af5cd8e9ba0754d25a43a96dfb47b46b771549a0d79d35bc3cc1aa Full Tag: docker.io/martinheinz/debian-python-anchore:latest Status: fail Last Eval: 2020-03-07T12:15:00Z Policy ID: 2c53a13c-1765-11e8-82ef-23527761d060
Well, well, well... Not so great anymore. All we did, was switch to official Python Debian image and we suddenly have more than 1000 vulnerabilities including some
Medium severity ones compared to just a few
Negligible ones with plain Debian image. On top of that, after running evaluation of the image, we can see that it failed (
So, even though we used official and seemingly secure image and while creating very simple application with no obvious vulnerabilities, we still got a lot of security problems. So, how can we improve this?
Finding Superior Image
When it comes to security, the best choice for a base image is
scratch - that is - empty container. It is however not very practical to build our own base images from nothing and it might bring more security issues considering that most of us are not security experts.
Second best choice after
scratch is - in my opinion - Distroless, which is set of images made by Google, that were created with intent to be secure. These images contain the bare minimum that's needed for your app, meaning that there are no shells, package managers or any other tools that would bloat the image and create signal noise for security scanners (like CVE) making it harder to establish compliance. I already went into more detail about Distroless in my previous blog post here, so in case you want to know more, go check it out.
Alright, so Distroless is now our choice for secure image, but before we analyze anything, let's first look at the new
# distroless.Dockerfile FROM python:3-slim AS build-env ADD . /app WORKDIR /app FROM gcr.io/distroless/python3 COPY --from=build-env /app /app WORKDIR /app CMD ["hello.py", "/etc"]
Not much has changed there compared to Debian version. We really just switched the Runner image to
gcr.io/distroless/python3. Now, we can go ahead and build it, push it and add it to Anchore Engine:
docker build -f distroless.Dockerfile -t martinheinz/distroless-python-anchore . docker run martinheinz/distroless-python-anchore:latest docker push martinheinz/distroless-python-anchore:latest # From anchore Docker CLI anchore-cli image add martinheinz/distroless-python-anchore:latest
Finally, time to see whether the Distroless performed any better than Debian version:
~ $ anchore-cli image vuln docker.io/martinheinz/distroless-python-anchore:latest os Vulnerability ID Package Severity Fix CVE Refs Vulnerability URL CVE-2019-16935 libpython3.5-minimal-3.5.3-1+deb9u1 Low None CVE-2019-16935 https://security-tracker.debian.org/tracker/CVE-2019-16935 CVE-2019-16935 python3.5-minimal-3.5.3-1+deb9u1 Low None CVE-2019-16935 https://security-tracker.debian.org/tracker/CVE-2019-16935 ... ~ $ anchore-cli image vuln docker.io/martinheinz/distroless-python-anchore:latest os | wc -l 53 ~ $ anchore-cli evaluate check docker.io/martinheinz/distroless-python-anchore:latest Image Digest: sha256:29b5288e934dc724377f2ff187e8a0664246c95b0e70bd9a13f6874628dc8662 Full Tag: docker.io/martinheinz/distroless-python-anchore:latest Status: pass Last Eval: 2020-03-07T12:15:20Z Policy ID: 2c53a13c-1765-11e8-82ef-23527761d060
And it sure did! Compared to first example we have only 53 vulnerabilities and just 2
Low severity ones. Also, looking at the policy evaluation - this image passed, while the Debian one failed!
So, we can safely conclude that Distroless is superior choice when it comes to security of our containerized applications. There are however, a few more things we can check and improve - for example, we can try using different evaluation policies:
~ $ anchore-cli policy hub list Name Description anchore_security_only Single policy, single whitelist bundle for performing security checks, including example blacklist known malicious packages by name. anchore_default_bundle Default policy bundle that comes installed with vanilla anchore-engine deployments. Mixture of light vulnerability checks, dockerfiles checks, and warning triggers for common best practices. anchore_cis_1.13.0_base Docker CIS 1.13.0 image content checks, from section 4 and 5. NOTE: some parameters (generally are named 'example...') must be modified as they require site-specific settings
The above command lists all available policies that we can use to check our images. By default, it uses
anchore_default_bundle which works just fine, but in case we want to see the check performed using different whitelist and rules, then we can try, for example
~ $ anchore-cli policy hub get anchore_cis_1.13.0_base Policy Bundle ID: anchore_cis_1.13.0_base Name: anchore_cis_1.13.0_base Description: Docker CIS 1.13.0 image content checks, from section 4 and 5. NOTE: some parameters (generally are named 'example...') must be modified as they require site-specific settings Policy Name: CIS File Checks Policy Description: Docker CIS section 4.8 and 4.10 checks. Policy Name: CIS Dockerfile Checks Policy Description: Docker CIS section 4.1, 4.2, 4.6, 4.7, 4.9 and 5.8 checks. Policy Name: CIS Software Checks Policy Description: Docker CIS section 4.3 and 4.4 checks. Whitelist Name: RHEL SUID Files Whitelist Description: Example whitelist with triggerIds of files that are expected to have SUID/SGID, for rhel-based images Whitelist Name: DEB SUID Files Whitelist Description: Example whitelist with triggerIds of files that are expected to have SUID/SGID, for debian-based images Mapping Name: default Mapping Rule: */*:* Mapping Policies: CIS Software Checks,CIS Dockerfile Checks,CIS File Checks Mapping Whitelists: DEB SUID Files,RHEL SUID Files ~ $ anchore-cli policy hub install anchore_cis_1.13.0_base Policy ID: anchore_cis_1.13.0_base Active: False Source: local Created: 2020-03-07T12:20:46Z Updated: 2020-03-07T12:20:46Z ~ $ anchore-cli policy list Policy ID Active Created Updated 2c53a13c-1765-11e8-82ef-23527761d060 True 2020-02-17T11:10:20Z 2020-02-17T11:10:20Z anchore_cis_1.13.0_base False 2020-03-07T12:20:46Z 2020-03-07T12:20:46Z ~ $ anchore-cli policy activate anchore_cis_1.13.0_base Success: anchore_cis_1.13.0_base activated
From the above you can see that we first looked at description of this specific policy by running
policy hub get. After that we installed it and ran
policy list command which shows that we now have 2 policies available, with the new one being inactive. So, we activate it with the last command (
Before we run the check with the new policy, let's first talk about what it includes. Considering that it's Docker CIS 1.13.0 check, it will focus on guidance provided by this document. Just a few examples of what kind of issues it looks for:
ADDcommand used in place of
- Non-whitelisted opened ports
- Using non-trusted base images
rootas effective user
For a full list of rules in this policy you can run
anchore-cli --json policy hub get anchore_cis_1.13.0_base. Now, that we know what it will look for, it's time to run it:
~ $ anchore-cli evaluate check docker.io/martinheinz/distroless-python-anchore:latest --detail ... dockerfile instruction Dockerfile directive 'HEALTHCHECK' not found, matching condition 'not_exists' check stop dockerfile instruction Dockerfile directive 'FROM' check 'not_in' matched against 'example_trusted_base1,example_trusted_base2' for line 'scratch' stop dockerfile effective_user User root found as effective user, which is explicity not allowed list stop ...
There are quite a few warnings in the output coming from some of the installed packages, but ones that are specific to this policy are the ones listed above. These are issues/vulnerabilities that make the image fail the policy evaluation and should be analyzed. One of these (second) is caused by usage of Distroless image as it is not whitelisted image, so that one can be ignored. As for the other two - these are real issues that should be fixed, but both of them have straightforward, simple solution.
This shows us, that it's a good idea to choose specific policy based on our requirements or even use multiple ones to catch as many issues as possible.
Types/Levels of vulnerabilities
Not all vulnerabilities are equal though and some of them don't even apply to our applications/containers/environments. Therefore, we should always not only look at their severity, but also the factors from which the severity is calculated. That includes attack vector, attack complexity, confidentiality impact, integrity impact, etc. These factors then create final severity score generated using CVSS Calculator. The specific brackets are -
Critical. More info about them can be found at NIST website: https://nvd.nist.gov/vuln-metrics/cvss.
Vulnerabilities in the above example were easy to fix, but that's not always the case. You might run into vulnerabilities with high severity score or ones that are problematic for your specific use case. When it comes to problems with base images or packages included in them, you shouldn't be the one fixing them, as that's responsibility of developers and/or publishers of said tool. That said though, you should prevent or at least mitigate exploitability of existing and future vulnerabilities by removing attack vectors, for example by using Distroless, which doesn't have any shell. On top of that, it's a good idea to run vulnerability checks periodically using build/deployment pipelines to find out as soon as possible when vulnerabilities get introduced, ideally using multiple scanning tools.
Anchore vs. Clair
Speaking about running multiple scanning tools... What about Clair? How does it compare to Anchore? This is how we can run it against our example Debian and Distroless images:
# Run setup steps from beginning of article... ~ $ clair-scanner --ip 192.168.1.56 martinheinz/debian-python-anchore:latest 2020/03/07 17:54:11 [INFO] ▶ Start clair-scanner 2020/03/07 17:54:18 [INFO] ▶ Server listening on port 9279 2020/03/07 17:54:18 [INFO] ▶ Analyzing 55db4dd701c0a4eea04ba231e74bd04d6c1cdbd86cf8084e988648aa7cf5af9b 2020/03/07 17:54:18 [INFO] ▶ Analyzing 1938b02d46a5b801035965c5680d11c513a679e0ad4a560276df9a8ececa0bed 2020/03/07 17:54:18 [INFO] ▶ Analyzing 90bd3e8e4e9f1dd7624ecdd1ee35c36072e582a2cd5c606f290d2b3cb4a1559c 2020/03/07 17:54:18 [INFO] ▶ Analyzing ced4f27972d40674b72da7b1a51b63a8396bbca3a4fca86dffa8031228ab88cb 2020/03/07 17:54:18 [INFO] ▶ Analyzing 7abae7ca0ef88724a59f51748b1658136bdbb1eae246550478f5939484358d94 2020/03/07 17:54:18 [INFO] ▶ Analyzing 62547b912c7f2d5d69eaca3cd5ad1cb053c39886cb7276b0cb753b3824c32358 2020/03/07 17:54:18 [INFO] ▶ Analyzing cb7076fa495a6c437ac374b1b67a62075b94db45400c10e8849b5db75ffbce21 2020/03/07 17:54:18 [INFO] ▶ Analyzing a3ffe04a8073815ecfa94620efbbccf412b85ca80b6550c029617eed51c5edb6 2020/03/07 17:54:18 [INFO] ▶ Analyzing 715fd7b0b1161fa57f74a843f4525f1ab7035c9a8051a6559c91ff360e44f20f 2020/03/07 17:54:18 [INFO] ▶ Analyzing 20519035fc6a7d347a1540a633c461ac321dc8db7cb8c163283c1d802097697b 2020/03/07 17:54:18 [WARN] ▶ Image [martinheinz/debian-python-anchore:latest] contains 356 total vulnerabilities 2020/03/07 17:54:18 [ERRO] ▶ Image [martinheinz/debian-python-anchore:latest] contains 356 unapproved vulnerabilities ... List of vulnerabilities follows ~ $ clair-scanner --ip 192.168.1.56 martinheinz/distroless-python-anchore:latest 2020/03/07 17:56:08 [INFO] ▶ Start clair-scanner 2020/03/07 17:56:09 [INFO] ▶ Server listening on port 9279 2020/03/07 17:56:09 [INFO] ▶ Analyzing 51756e4adf1ad2b5a00c36dcc9e799a69d72462d60e6f88c87f96eed832b8a2a 2020/03/07 17:56:09 [INFO] ▶ Analyzing 49905beb7372788afbefede92af761383ad2bac7f8d21b2f2d6fb165afff48e8 2020/03/07 17:56:09 [INFO] ▶ Analyzing 3d9fd80998dc9ea371366e1ef54046fbe86e6e360d17ece650ea8a3ea5023c63 2020/03/07 17:56:09 [INFO] ▶ Analyzing 83b671756e1e2cc132505e55c87c5c5b1da185665f9bb875d771568959dc05a2 2020/03/07 17:56:09 [INFO] ▶ Analyzing b80190979f279448d479a3ed0450f6d2912662e12edd1fe477a2b74d0568e1f0
Looking at output of above commands, it looks pretty similar to what we've got from Anchore - lots of issues for Debian image and nothing for Distroless. In the output of first scan, there is huge list of vulnerabilities which I omitted, but I checked all the
Medium severity ones and they are present both in Anchore and Clair scan results, but that might not always be the case. So, I would recommend running multiple scanning tools, ideally ones that use different sources for vulnerability metadata.
Some other tools you might want to have a look at include Docker Bench for Security or vulnerability scan included in IBM Container Registry.
With security related stuff, it's always preferable to be proactive and try to avoid vulnerabilities before they become an actual problem and tools shown in this article can greatly help with that. I think using and implementing this tooling is easy enough, so that anybody can make them part of daily workflow or ideally, part of their automatic pipelines/jobs. Security of applications we create is - to a certain extent - responsibility of every developer, not just security experts or pentesters, so we should all take part in making them sufficiently secure, maybe just by running vulnerability scan from time-to-time. 😉