A lot of people assume that Docker images and containers are secure by default, which - unfortunately - is not the case. There are quite a few things that effect security of your Docker images. Whether it's packages installed in the image, libraries used by your application or even the base image - all these components might introduces vulnerability into your application. A lot of these problems are easily avoidable, though...

Anchore & Clair

The easiest way to find vulnerabilities in Docker images is to run inspection against them using tools like Anchore or Clair:

Anchore Engine: Anchore is a centralized service for inspection, analysis and certification of container image. It scans images using vulnerability data (feeds) from OS vendors like Red Hat, Debian or Alpine. For non-OS data it uses NVD (National Vulnerability Database), which includes vulnerabilities for RPM, Deb, APK as well as Python (PIP), Ruby Gems, etc.
Clair: Clair is a static analyzer developed by CoreOS for Docker and APPC containers. It uses vulnerability metadata from similar sources as Anchore - Red Hat Security Data, NVD, Ubuntu CVE Tracker, Alpine SecDB, Debian Security Bug Tracker, etc.

Setting Up

Now that we know the tools we want to use, it's time spin them up. Both Anchore and Clair include various integration and can be deployed to Kubernetes or OpenShift, but for the purpose of this demonstration, we will set them up using docker-compose on local machine:

To setup Anchore run following:


mkdir ~/aevolume
cd ~/aevolume

docker pull docker.io/anchore/anchore-engine:latest
docker create --name ae docker.io/anchore/anchore-engine:latest
docker cp ae:/docker-compose.yaml ~/aevolume/docker-compose.yaml
docker rm ae

docker-compose pull
docker-compose up -d

export ANCHORE_CLI_USER=admin
export ANCHORE_CLI_PASS=foobar

docker run --net=host -e ANCHORE_CLI_URL=http://localhost:8228/v1/ -it anchore/engine-cli

To setup Clair run following:


# Download Clair Scanner from https://github.com/arminc/clair-scanner/releases
chmod +x clair-scanner

docker run -p 5432:5432 -d --name db arminc/clair-db:$(date +%F)
docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.6

And with that we are ready to analyze!

Note: We will come back to Clair little later in the article.

Check Image for Vulnerabilities

Let's start with Anchore and basic Debian image. What we need to do, to get our image analyzed is to add it and wait for the analysis to complete:


# inside anchore-cli Docker container

~ $ anchore-cli image add docker.io/library/debian:latest
Image Digest: sha256:121dd2a723be1c8aa8b116684d66157c93c801f2f5107b60287937e88c13ab89
Parent Digest: sha256:a63d0b2ecbd723da612abf0a8bdb594ee78f18f691d7dc652ac305a490c9b71a
Analysis Status: analyzed
Image Type: docker
Analyzed At: 2020-03-07T10:46:20Z
Image ID: 971452c943760ab769134f22db8d3381b09ea000a6c459fbfa3603bb99115f62
Dockerfile Mode: Guessed
Distro: debian
Distro Version: 10
Size: 126607360
Architecture: amd64
Layer Count: 1

Full Tag: docker.io/library/debian:latest
Tag Detected At: 2020-03-07T10:45:48Z

~ $ anchore-cli image wait docker.io/library/debian:latest
Status: analyzing
Waiting 5.0 seconds for next retry.
...

~ $ anchore-cli image list
Full Tag                                                      Image Digest                                                                   Analysis Status
docker.io/library/debian:latest                               sha256:121dd2a723be1c8aa8b116684d66157c93c801f2f5107b60287937e88c13ab89        analyzed

With the image analyzed let's see what vulnerabilities were found:


~ $ anchore-cli image vuln docker.io/library/debian:latest all
Vulnerability ID        Package                            Severity          Fix         CVE Refs                Vulnerability URL
CVE-2005-2541           tar-1.30+dfsg-6                    Negligible        None        CVE-2005-2541           https://security-tracker.debian.org/tracker/CVE-2005-2541
CVE-2007-5686           login-1:4.5-1.1                    Negligible        None        CVE-2007-5686           https://security-tracker.debian.org/tracker/CVE-2007-5686
...

~ $ anchore-cli evaluate check docker.io/library/debian:latest
Image Digest: sha256:121dd2a723be1c8aa8b116684d66157c93c801f2f5107b60287937e88c13ab89
Full Tag: docker.io/library/debian:latest
Status: pass
Last Eval: 2020-03-14T13:25:24Z
Policy ID: 2c53a13c-1765-11e8-82ef-23527761d060

We first run image vuln command with all flag to list both OS and package vulnerabilities present in the image. All of those are Negligible or Unknown, so that's pretty good. Next, we run evaluate check command to see whether this image passes the default policy check and as you can see above (Status: pass), it does. These kinds of base images generally perform pretty well in this regard as they are widely used and therefore under a lot of scrutiny.

But, what about simple simple "Hello World" Python application built using Python 3 Debian Buster image? Let's look at the Dockerfile first:


# debian.Dockerfile
FROM python:3-slim AS build-env
ADD . /app
WORKDIR /app

FROM python:3-buster
COPY --from=build-env /app /app
WORKDIR /app
CMD ["python", "hello.py", "/etc"]

Nothing that screams "vulnerable" and "insecure", right? So, let's built it and analyze it:


~ $ docker build -f debian.Dockerfile -t martinheinz/debian-python-anchore .
~ $ docker run martinheinz/debian-python-anchore:latest
~ $ docker push martinheinz/debian-python-anchore:latest

# From anchore Docker CLI
~ $ anchore-cli image add martinheinz/debian-python-anchore:latest
~ $ anchore-cli image wait martinheinz/debian-python-anchore:latest
~ $ anchore-cli image list

Full Tag                                                      Image Digest                                                                   Analysis Status
docker.io/library/debian:latest                               sha256:121dd2a723be1c8aa8b116684d66157c93c801f2f5107b60287937e88c13ab89        analyzed
docker.io/martinheinz/debian-python-anchore:latest            sha256:59dff8bdf4af5cd8e9ba0754d25a43a96dfb47b46b771549a0d79d35bc3cc1aa        analyzed

I first built the image using the above debian.Dockerfile and simple hello.py. I then pushed it to Docker Hub, from where it was added to Anchore and finally analyzed. Now we can check out the results:


~ $ anchore-cli image vuln docker.io/martinheinz/debian-python-anchore:latest os
Vulnerability ID        Package                                                 Severity          Fix         CVE Refs                Vulnerability URL
CVE-2007-3476           libwmf-dev-0.2.8.4-14                                   Low               None        CVE-2007-3476           https://security-tracker.debian.org/tracker/CVE-2007-3476
CVE-2007-3476           libwmf0.2-7-0.2.8.4-14                                  Low               None        CVE-2007-3476           https://security-tracker.debian.org/tracker/CVE-2007-3476
CVE-2007-3477           libwmf-dev-0.2.8.4-14                                   Low               None        CVE-2007-3477           https://security-tracker.debian.org/tracker/CVE-2007-3477
CVE-2007-3477           libwmf0.2-7-0.2.8.4-14                                  Low               None        CVE-2007-3477           https://security-tracker.debian.org/tracker/CVE-2007-3477
CVE-2016-8660           linux-libc-dev-4.19.98-1                                Low               None        CVE-2016-8660           https://security-tracker.debian.org/tracker/CVE-2016-8660
CVE-2007-3996           libwmf-dev-0.2.8.4-14                                   Medium            None        CVE-2007-3996           https://security-tracker.debian.org/tracker/CVE-2007-3996
CVE-2007-3996           libwmf0.2-7-0.2.8.4-14                                  Medium            None        CVE-2007-3996           https://security-tracker.debian.org/tracker/CVE-2007-3996
CVE-2009-3546           libwmf-dev-0.2.8.4-14                                   Medium            None        CVE-2009-3546           https://security-tracker.debian.org/tracker/CVE-2009-3546
CVE-2009-3546           libwmf0.2-7-0.2.8.4-14                                  Medium            None        CVE-2009-3546           https://security-tracker.debian.org/tracker/CVE-2009-3546
...

~ $ anchore-cli image vuln docker.io/martinheinz/debian-python-anchore:latest os | wc -l
1056

~ $ anchore-cli evaluate check docker.io/martinheinz/debian-python-anchore:latest
Image Digest: sha256:59dff8bdf4af5cd8e9ba0754d25a43a96dfb47b46b771549a0d79d35bc3cc1aa
Full Tag: docker.io/martinheinz/debian-python-anchore:latest
Status: fail
Last Eval: 2020-03-07T12:15:00Z
Policy ID: 2c53a13c-1765-11e8-82ef-23527761d060

Well, well, well... Not so great anymore. All we did, was switch to official Python Debian image and we suddenly have more than 1000 vulnerabilities including some Low and Medium severity ones compared to just a few Negligible ones with plain Debian image. On top of that, after running evaluation of the image, we can see that it failed (Status: fail).

So, even though we used official and seemingly secure image and while creating very simple application with no obvious vulnerabilities, we still got a lot of security problems. So, how can we improve this?

Finding Superior Image

When it comes to security, the best choice for a base image is scratch - that is - empty container. It is however not very practical to build our own base images from nothing and it might bring more security issues considering that most of us are not security experts.

Second best choice after scratch is - in my opinion - Distroless, which is set of images made by Google, that were created with intent to be secure. These images contain the bare minimum that's needed for your app, meaning that there are no shells, package managers or any other tools that would bloat the image and create signal noise for security scanners (like CVE) making it harder to establish compliance. I already went into more detail about Distroless in my previous blog post here, so in case you want to know more, go check it out.

Alright, so Distroless is now our choice for secure image, but before we analyze anything, let's first look at the new Dockerfile:


# distroless.Dockerfile
FROM python:3-slim AS build-env
ADD . /app
WORKDIR /app

FROM gcr.io/distroless/python3
COPY --from=build-env /app /app
WORKDIR /app
CMD ["hello.py", "/etc"]

Not much has changed there compared to Debian version. We really just switched the Runner image to gcr.io/distroless/python3. Now, we can go ahead and build it, push it and add it to Anchore Engine:


docker build -f distroless.Dockerfile -t martinheinz/distroless-python-anchore .
docker run martinheinz/distroless-python-anchore:latest
docker push martinheinz/distroless-python-anchore:latest

# From anchore Docker CLI
anchore-cli image add martinheinz/distroless-python-anchore:latest

Finally, time to see whether the Distroless performed any better than Debian version:


~ $ anchore-cli image vuln docker.io/martinheinz/distroless-python-anchore:latest os
Vulnerability ID        Package                                    Severity          Fix         CVE Refs                Vulnerability URL
CVE-2019-16935          libpython3.5-minimal-3.5.3-1+deb9u1        Low               None        CVE-2019-16935          https://security-tracker.debian.org/tracker/CVE-2019-16935
CVE-2019-16935          python3.5-minimal-3.5.3-1+deb9u1           Low               None        CVE-2019-16935          https://security-tracker.debian.org/tracker/CVE-2019-16935
...

~ $ anchore-cli image vuln docker.io/martinheinz/distroless-python-anchore:latest os | wc -l
53

~ $ anchore-cli evaluate check docker.io/martinheinz/distroless-python-anchore:latest
Image Digest: sha256:29b5288e934dc724377f2ff187e8a0664246c95b0e70bd9a13f6874628dc8662
Full Tag: docker.io/martinheinz/distroless-python-anchore:latest
Status: pass
Last Eval: 2020-03-07T12:15:20Z
Policy ID: 2c53a13c-1765-11e8-82ef-23527761d060

And it sure did! Compared to first example we have only 53 vulnerabilities and just 2 Low severity ones. Also, looking at the policy evaluation - this image passed, while the Debian one failed!

So, we can safely conclude that Distroless is superior choice when it comes to security of our containerized applications. There are however, a few more things we can check and improve - for example, we can try using different evaluation policies:


~ $ anchore-cli policy hub list
Name                           Description
anchore_security_only          Single policy, single whitelist bundle for performing
                               security checks, including example blacklist known malicious
                               packages by name.
anchore_default_bundle         Default policy bundle that comes installed with vanilla
                               anchore-engine deployments.  Mixture of light vulnerability
                               checks, dockerfiles checks, and warning triggers for common
                               best practices.
anchore_cis_1.13.0_base        Docker CIS 1.13.0 image content checks, from section 4 and
                               5. NOTE: some parameters (generally are named 'example...')
                               must be modified as they require site-specific settings

The above command lists all available policies that we can use to check our images. By default, it uses anchore_default_bundle which works just fine, but in case we want to see the check performed using different whitelist and rules, then we can try, for example anchore_cis_1.13.0_base:


~ $ anchore-cli policy hub get anchore_cis_1.13.0_base

Policy Bundle ID: anchore_cis_1.13.0_base
Name: anchore_cis_1.13.0_base
Description: Docker CIS 1.13.0 image content checks, from section 4 and 5. NOTE: some parameters (generally are named 'example...') must be modified as they require site-specific settings

Policy Name: CIS File Checks
Policy Description: Docker CIS section 4.8 and 4.10 checks.

Policy Name: CIS Dockerfile Checks
Policy Description: Docker CIS section 4.1, 4.2, 4.6, 4.7, 4.9 and 5.8 checks.

Policy Name: CIS Software Checks
Policy Description: Docker CIS section 4.3 and 4.4 checks.

Whitelist Name: RHEL SUID Files
Whitelist Description: Example whitelist with triggerIds of files that are expected to have SUID/SGID, for rhel-based images

Whitelist Name: DEB SUID Files
Whitelist Description: Example whitelist with triggerIds of files that are expected to have SUID/SGID, for debian-based images

Mapping Name: default
Mapping Rule: */*:*
Mapping Policies: CIS Software Checks,CIS Dockerfile Checks,CIS File Checks
Mapping Whitelists: DEB SUID Files,RHEL SUID Files

~ $ anchore-cli policy hub install anchore_cis_1.13.0_base
Policy ID: anchore_cis_1.13.0_base
Active: False
Source: local
Created: 2020-03-07T12:20:46Z
Updated: 2020-03-07T12:20:46Z

~ $ anchore-cli policy list
Policy ID                                   Active        Created                     Updated
2c53a13c-1765-11e8-82ef-23527761d060        True          2020-02-17T11:10:20Z        2020-02-17T11:10:20Z
anchore_cis_1.13.0_base                     False         2020-03-07T12:20:46Z        2020-03-07T12:20:46Z

~ $ anchore-cli policy activate anchore_cis_1.13.0_base
Success: anchore_cis_1.13.0_base activated

From the above you can see that we first looked at description of this specific policy by running policy hub get. After that we installed it and ran policy list command which shows that we now have 2 policies available, with the new one being inactive. So, we activate it with the last command (policy activate).

Before we run the check with the new policy, let's first talk about what it includes. Considering that it's Docker CIS 1.13.0 check, it will focus on guidance provided by this document. Just a few examples of what kind of issues it looks for:

ADD command used in place of COPY
Non-whitelisted opened ports
Missing HEALTCHECK
Using non-trusted base images
Using root as effective user

For a full list of rules in this policy you can run anchore-cli --json policy hub get anchore_cis_1.13.0_base. Now, that we know what it will look for, it's time to run it:


~ $ anchore-cli evaluate check docker.io/martinheinz/distroless-python-anchore:latest --detail
...
dockerfile      instruction         Dockerfile directive 'HEALTHCHECK' not found, matching condition 'not_exists' check                                            stop
dockerfile      instruction         Dockerfile directive 'FROM' check 'not_in' matched against 'example_trusted_base1,example_trusted_base2' for line 'scratch'    stop
dockerfile      effective_user      User root found as effective user, which is explicity not allowed list                                                         stop
...

There are quite a few warnings in the output coming from some of the installed packages, but ones that are specific to this policy are the ones listed above. These are issues/vulnerabilities that make the image fail the policy evaluation and should be analyzed. One of these (second) is caused by usage of Distroless image as it is not whitelisted image, so that one can be ignored. As for the other two - these are real issues that should be fixed, but both of them have straightforward, simple solution.

This shows us, that it's a good idea to choose specific policy based on our requirements or even use multiple ones to catch as many issues as possible.

Types/Levels of vulnerabilities

Not all vulnerabilities are equal though and some of them don't even apply to our applications/containers/environments. Therefore, we should always not only look at their severity, but also the factors from which the severity is calculated. That includes attack vector, attack complexity, confidentiality impact, integrity impact, etc. These factors then create final severity score generated using CVSS Calculator. The specific brackets are - None, Low, Medium, High and Critical. More info about them can be found at NIST website: https://nvd.nist.gov/vuln-metrics/cvss.

Fixing Vulnerabilities

Vulnerabilities in the above example were easy to fix, but that's not always the case. You might run into vulnerabilities with high severity score or ones that are problematic for your specific use case. When it comes to problems with base images or packages included in them, you shouldn't be the one fixing them, as that's responsibility of developers and/or publishers of said tool. That said though, you should prevent or at least mitigate exploitability of existing and future vulnerabilities by removing attack vectors, for example by using Distroless, which doesn't have any shell. On top of that, it's a good idea to run vulnerability checks periodically using build/deployment pipelines to find out as soon as possible when vulnerabilities get introduced, ideally using multiple scanning tools.

Anchore vs. Clair

Speaking about running multiple scanning tools... What about Clair? How does it compare to Anchore? This is how we can run it against our example Debian and Distroless images:


# Run setup steps from beginning of article...

~ $ clair-scanner --ip 192.168.1.56 martinheinz/debian-python-anchore:latest

2020/03/07 17:54:11 [INFO] ▶ Start clair-scanner
2020/03/07 17:54:18 [INFO] ▶ Server listening on port 9279
2020/03/07 17:54:18 [INFO] ▶ Analyzing 55db4dd701c0a4eea04ba231e74bd04d6c1cdbd86cf8084e988648aa7cf5af9b
2020/03/07 17:54:18 [INFO] ▶ Analyzing 1938b02d46a5b801035965c5680d11c513a679e0ad4a560276df9a8ececa0bed
2020/03/07 17:54:18 [INFO] ▶ Analyzing 90bd3e8e4e9f1dd7624ecdd1ee35c36072e582a2cd5c606f290d2b3cb4a1559c
2020/03/07 17:54:18 [INFO] ▶ Analyzing ced4f27972d40674b72da7b1a51b63a8396bbca3a4fca86dffa8031228ab88cb
2020/03/07 17:54:18 [INFO] ▶ Analyzing 7abae7ca0ef88724a59f51748b1658136bdbb1eae246550478f5939484358d94
2020/03/07 17:54:18 [INFO] ▶ Analyzing 62547b912c7f2d5d69eaca3cd5ad1cb053c39886cb7276b0cb753b3824c32358
2020/03/07 17:54:18 [INFO] ▶ Analyzing cb7076fa495a6c437ac374b1b67a62075b94db45400c10e8849b5db75ffbce21
2020/03/07 17:54:18 [INFO] ▶ Analyzing a3ffe04a8073815ecfa94620efbbccf412b85ca80b6550c029617eed51c5edb6
2020/03/07 17:54:18 [INFO] ▶ Analyzing 715fd7b0b1161fa57f74a843f4525f1ab7035c9a8051a6559c91ff360e44f20f
2020/03/07 17:54:18 [INFO] ▶ Analyzing 20519035fc6a7d347a1540a633c461ac321dc8db7cb8c163283c1d802097697b
2020/03/07 17:54:18 [WARN] ▶ Image [martinheinz/debian-python-anchore:latest] contains 356 total vulnerabilities
2020/03/07 17:54:18 [ERRO] ▶ Image [martinheinz/debian-python-anchore:latest] contains 356 unapproved vulnerabilities
... List of vulnerabilities follows

~ $ clair-scanner --ip 192.168.1.56 martinheinz/distroless-python-anchore:latest
2020/03/07 17:56:08 [INFO] ▶ Start clair-scanner
2020/03/07 17:56:09 [INFO] ▶ Server listening on port 9279
2020/03/07 17:56:09 [INFO] ▶ Analyzing 51756e4adf1ad2b5a00c36dcc9e799a69d72462d60e6f88c87f96eed832b8a2a
2020/03/07 17:56:09 [INFO] ▶ Analyzing 49905beb7372788afbefede92af761383ad2bac7f8d21b2f2d6fb165afff48e8
2020/03/07 17:56:09 [INFO] ▶ Analyzing 3d9fd80998dc9ea371366e1ef54046fbe86e6e360d17ece650ea8a3ea5023c63
2020/03/07 17:56:09 [INFO] ▶ Analyzing 83b671756e1e2cc132505e55c87c5c5b1da185665f9bb875d771568959dc05a2
2020/03/07 17:56:09 [INFO] ▶ Analyzing b80190979f279448d479a3ed0450f6d2912662e12edd1fe477a2b74d0568e1f0

Looking at output of above commands, it looks pretty similar to what we've got from Anchore - lots of issues for Debian image and nothing for Distroless. In the output of first scan, there is huge list of vulnerabilities which I omitted, but I checked all the Low and Medium severity ones and they are present both in Anchore and Clair scan results, but that might not always be the case. So, I would recommend running multiple scanning tools, ideally ones that use different sources for vulnerability metadata.

Some other tools you might want to have a look at include Docker Bench for Security or vulnerability scan included in IBM Container Registry.

Conclusion

With security related stuff, it's always preferable to be proactive and try to avoid vulnerabilities before they become an actual problem and tools shown in this article can greatly help with that. I think using and implementing this tooling is easy enough, so that anybody can make them part of daily workflow or ideally, part of their automatic pipelines/jobs. Security of applications we create is - to a certain extent - responsibility of every developer, not just security experts or pentesters, so we should all take part in making them sufficiently secure, maybe just by running vulnerability scan from time-to-time. 😉

Blog Posts Tagged #security

Blog Posts Tagged `#security`