Security and Vulnerability Scanning of Container Images

Overview

In this article we will focus on security and vulnerability strategies for scanning container images. I know, in the past security was always viewed upon as an impedance to the speed of production but hopefully these days are behind us. Having a security breach, as you probably know, is one of the most costly things an organization can endure. It takes years to build up a reputation and only seconds to tear it down completely.

I still see today, many organizations ignoring container images completely because it is often misunderstood. Exactly what is inside a container image? Who should be responsible for it? How does it map to what we have done on servers? Security teams often don't understand containers or even know what questions to ask. We need to help them and it is our duty to do so. Unfortunately there are not very many tools that can help in broad sense. Containers are new and evolving at breakneck speed. That coupled with the fact that security can negatively impact the speed of a DevOps team (if not done right), it is no wonder we are at square one, in many cases.

Before we dive into more detail, let us review important security aspects of containers.

• Containers can have various packaging formats, Docker is the most popular today
• Containers are immutable and as such are image based
• Container are never updated, any change always results in a new container
• Container images consist of layers (base, runtime, application)
• Container images require shared responsibility between dev and ops
• Containers don't contain, they are in fact, just processes

For more information I recommend reading about the 10 layers of container security.

Container images contain OS, runtime and application (everything but the Linux kernel basically). If a container is breached, there is not much in the way preventing further containers from being breached unless you have SELinux, limit kernel capabilities, run as non-priviledged user and follow many other best practices. OpenShift and using trusted, signed images by a vendor like Red Hat does all of this by default. Even still the attack surface can be large. You still think it is OK to not worry about container images and what is inside? Thankfully there are at least several solutions we will talk about from Red Hat for the container Platform OpenShift.

• Container Catalog grades images provided by Red Hat and provides history of image as security patches are applied.
• Atomic CLI scans images and uses OpenSCAP to determine security vulnerabilities.
• CloudForms scans images using OpenSCAP (same as atomic) and also adds capabilities like taking action when container images are vulnerable, automatically scanning new images and even reporting.

All of these solutions are included with OpenShift.

Container Catalog

Red Hat offers a container catalog for all images that it provides. Images are maintained by Red Hat meaning they are frequently scanned and updated. The container catalog also provides the history of an image, each time it is updated a new tag is created. The image is graded according to a scale A to F.

Health Index	Security Errata Conditions
Grade A	This image does not have any unapplied Critical or Important security errata
Grade B	This image is affected by Critical (no older than 7 days) or Important (no older than 30 days) security errata
Grade C	This image is affected by Critical (no older than 30 days) or Important (no older than 90 days) security errata
Grade D	This image is affected by Critical (no older than 90 days) or Important (no older than 12 months) security errata
Grade E	This image is affected by Critical or Important security errata no older than 12 months
Grade F	This image is affected by Critical or Important security errata older than 12 months
Unknown	This image is missing metadata required to calculate a grade and cannot be scanned

Check image (example nodejs-6).

https://access.redhat.com/containers/?tab=overview&platform=openshift#/registry.access.redhat.com/rhscl/nodejs-6-rhel7

We see the image has a health index of A and is also signed. We also see it runs as unprivileged user.

View Image History

Under tags, the history of the image can be viewed.

Image Details

By clicking on the tag name we are able to get more detail about a given image.

Any CVEs that affect the image are shown in addition to RPM advisory.

Atomic CLI Image Scanning

Atomic Cli provides image scanning using OpenSCAP. This tool can be used via Ansible or other automation in CI/CD pipeline to automatically scan images.

Install Atomic cli

If you aren't running RHEL Atomic you need to install the atomic cli. In this case we are doing so on the OpenShift master (master0) but it can be any node that has access to the container images. If image doesn't exist locally (in this case on master0) it must be pulled down.

[master0 ~]$ yum install -y atomic

Get the image that should be scanned

[master0 ~]$ oc get image |grep nodejs
sha256:947bf778fd9ee31a4cd06a702954aa0cd91e1eb81fef366a9080c0cfe3e35e43 172.30.187.230:5000/mynodejs/test@sha256:947bf778fd9ee31a4cd06a702954aa0cd91e1eb81fef366a9080c0cfe3e35e43
sha256:a9b89bb53fef405ea73f3eaff2dafa0c37c2cc988586b1a8a0e3bc19de07d4b8 registry.access.redhat.com/rhscl/nodejs-6-rhel7@sha256:a9b89bb53fef405ea73f3eaff2dafa0c37c2cc988586b1a8a0e3bc19de07d4b8
sha256:b6fee5146e6330e9890f8290746944ab1452f709f806588fbfb1ff5cadb5aaed registry.access.redhat.com/rhscl/nodejs-4-rhel7@sha256:b6fee5146e6330e9890f8290746944ab1452f709f806588fbfb1ff5cadb5aaed
sha256:c17fa1ffa8e4acf5e66e10127aea2ae5ca3eda7c34aecc005aaf3b04da48814c 172.30.187.230:5000/mynodejs/nodejs-hello-world-v2@sha256:c17fa1ffa8e4acf5e66e10127aea2ae5ca3eda7c34aecc005aaf3b04da48814c

Scan image.

Ensure openscap image is installed.

[master0 ~]$ sudo atomic install registry.access.redhat.com/rhel7/openscap

Using the image location we perform a scan.

[master0 ~]$ sudo atomic scan registry.access.redhat.com/rhscl/nodejs-6-rhel7@sha256:a9b89bb53fef405ea73f3eaff2dafa0c37c2cc988586b1a8a0e3bc19de07d4b8

registry.access.redhat.com/rhscl/nodejs-6-rhel7@sha256:a9b89bb53fef405ea73f3eaff2dafa0c37c2cc988586b1a8a0e3bc19de07d4b8 (ae9be2ffb565659)

The following issues were found:

RHSA-2018:0260: systemd security update (Moderate)
 Severity: Moderate
 RHSA URL: https://access.redhat.com/errata/RHSA-2018:0260
 RHSA ID: RHSA-2018:0260-01
 Associated CVEs:
 CVE ID: CVE-2018-1049
 CVE URL: https://access.redhat.com/security/cve/CVE-2018-1049

RHSA-2018:0180: kernel-alt security and bug fix update (Important)
 Severity: Important
 RHSA URL: https://access.redhat.com/errata/RHSA-2018:0180
 RHSA ID: RHSA-2018:0180-01
 Associated CVEs:
 CVE ID: CVE-2017-1000405
 CVE URL: https://access.redhat.com/security/cve/CVE-2017-1000405

RHSA-2017:0372: kernel-aarch64 security and bug fix update (Important)
 Severity: Important
 RHSA URL: https://access.redhat.com/errata/RHSA-2017:0372
 RHSA ID: RHSA-2017:0372-01
 Associated CVEs:
 CVE ID: CVE-2016-5195
 CVE URL: https://access.redhat.com/security/cve/CVE-2016-5195
 CVE ID: CVE-2016-7039
 CVE URL: https://access.redhat.com/security/cve/CVE-2016-7039
 CVE ID: CVE-2016-8666
 CVE URL: https://access.redhat.com/security/cve/CVE-2016-8666

Here we notice several important or high severity CVEs and a medium.

Using atomic cli may be enough if images going into registry are tightly controlled. This could work well if additional tooling such as Artifactory is used to persist container images. They can easily be scanned before being added to Artifactory as part of CI/CD process.

CloudForms Image Scanning

CloudForms provides additional capabilities for security and vulnerability scanning. You can configure policies to take action based on an vulnerable image. For example, not allowing vulnerable images to run or notifying security team. In addition, as soon as an image is created and pushed to OpenShift registry via CI/CD, it can be scanned. Reporting allows for understanding impact of vulnerable images across projects in OpenShift. Image users can be easily notified and it allows roles between development and operations to have clear delineation.

Configure CloudForms Provider for OpenShift

The first step is to configure CloudForms provider to access OpenShift environment.

Get management-admin token for management-infra project.

[master0 ~]$ oc get sa management-admin -o yaml -n management-infra
apiVersion: v1
imagePullSecrets:
- name: management-admin-dockercfg-4dfcr
kind: ServiceAccount
metadata:
 creationTimestamp: 2018-02-09T21:01:32Z
 name: management-admin
 namespace: management-infra
 resourceVersion: "1092"
 selfLink: /api/v1/namespaces/management-infra/serviceaccounts/management-admin
 uid: 6860deab-0ddc-11e8-ab6a-fa163e0b5deb
secrets:
- name: management-admin-token-q9clk
- name: management-admin-dockercfg-4dfcr

Get Secret for management-admin token.

[master0 ~]$ oc describe secret management-admin-token-q9clk -n management-infra
Name: management-admin-token-q9clk
Namespace: management-infra
Labels: 
Annotations: kubernetes.io/service-account.name=management-admin
 kubernetes.io/service-account.uid=6860deab-0ddc-11e8-ab6a-fa163e0b5deb

Type: kubernetes.io/service-account-token

Data
====
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtYW5hZ2VtZW50LWluZnJhIiwia3ViZXJuZXRlcy32ay9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1hbmFnZW1lbnQtYWRtaW4tdG9rZW4tcTljbGsiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYW55ddssaVudC5uYW1lIjoibWFuYWdlbWVudC1hZG1pbiIsImt1YmVybmV0ZXMuaW
ca.crt: 1070 bytes
namespace: 16 bytes
service-ca.crt: 2186 bytes

Add OpenShift Provider to CloudForms.

OpenShift by default creates a management-infra project with a management-admin SA and token. This project is used by CloudForms for access and image scanning.

Set Node Selector for management-infra project.

In this example we will set the region to infra. This means any nodes with region infra will run image scanning container. Infra nodes also generally run other shared platform services like router, registry, metrics and logging.

[master0 ~]$ oc edit namespace management-infra
---
metadata:
 annotations:
   openshift.io/node-selector: region=infra
---

Configure Security Scanning Policies

CloudForms ships with compliance and control policies that will automatically scan containers. If vulnerabilities are detected, containers with those vulnerabilities will be prevented from running. This is not enabled by default but I find it a bit aggressive and recommend scanning images and reporting them as non-compliant instead.

Login to CloudForms and navigate to control->explorer.

Copy OpenScap Compliance Policy.

Edit the copied OpenScap compliance policy.

Here we will remove the action to prevent container images that are non-compliant from running.

Add new container image condition for sti builder.

Under conditions accordion, create a new condition. We will want to ignore scanning the sti builder.

Add new container image condition for deployer.

Under conditions accordion, create a new condition. We will want to ignore scanning the deployer.

Copy control policy Analyse incoming container images and edit condition assignments.

Add the two newly created conditions.

Add new profile policy.

Under "all profile policies", add a new profile policy. Add the following policies:

• Copy of OpenScap (compliance)
• Copy of Analyze incoming container images (control)
• Schedule compliance after smart state analysis (control)

Enable policy profile on OpenShift provider.

Navigate to Compute->Containers->Providers.

Our policy profile will ensure the following:

• All images that change are scanned immediately
• The deployer, STI builder and image inspector are ignored
• Images with High vulnerability are marked as non-compliant

Perform Container Image Scanning

Each build creates a new image. As soon as build is pushed the image is automatically scanned. In this case a change was made which kicked off CI/CD, resulting in a build and a new image being pushed to registry.

Check pods under management-infra project.

Each image will trigger a scan. The image scanner container will mount the image and scan it using openscap.

[master0 ~]$ oc get pods -o wide -n management-infra
NAME READY STATUS RESTARTS AGE IP NODE
manageiq-img-scan-ea955 0/1 Running 0 2m 10.30.1.54 infra0

Check image scanner container logs.

[master0 ~]$ oc logs manageiq-img-scan-ea955 -n management-infra
2018/02/15 16:26:56 Pulling image 172.30.187.230:5000/mynodejs/nodejs-hello-world-v2@sha256:e5dba582855f9de07d9a00b2f2d0986b41d229112626593a4a6ba50ff53bbf49
2018/02/15 16:26:56 Authentication with Default Empty Authentication failed: unauthorized: authentication required
2018/02/15 16:27:07 Downloading Image (17Kb downloaded)
2018/02/15 16:27:09 Finished Downloading Image (17Kb downloaded)
2018/02/15 16:27:24 Extracting image 172.30.187.230:5000/mynodejs/nodejs-hello-world-v2@sha256:e5dba582855f9de07d9a00b2f2d0986b41d229112626593a4a6ba50ff53bbf49 to /var/tmp/image-inspector-545571508
2018/02/15 16:32:49 OpenSCAP scanning /var/tmp/image-inspector-545571508. Placing results in /var/tmp/image-inspector-scan-results-428093571
2018/02/15 16:33:43 Serving image content /var/tmp/image-inspector-545571508 on webdav://0.0.0.0:8080/api/v1/content/

Find image.

Once the image scanner runs we can navigate to the image in CloudForms under Compute->Containers->Container Images. We can use the search field to filter and find image by it's name. In this case it is of course the latest image.

View Image Details.

Here we can see that smart state analysis in CloudForms (container image scan) was run. Notice compliance is not-compliant. Two High severity and a medium severity rule failed. This is exactly what we also saw when running atomic image scan. Finally notice in addition to the OpenScap results, we also have inventory of all the packages and corresponding package versions, installed in the image.

Container Security and Vulnerability Reporting

Now that we are able to scan images and flag ones that have high security vulnerabilities, it is time to look into reporting. You could easily have 1000s of images so reporting becomes increasingly important to identify projects using high vulnerability images. This allows us to nicely segregate roles and responsibilities. The platform team can scan images and notify devops teams about vulnerabilities, who can in turn fix them. There are of course other models, just an idea.

Create Container Image Vulnerability Report.

Under CloudIntel->Reports->Custom add a new report.

Select Fields.

Configure Filters.

There are two types of filters: primary and secondary. Primary is used when doing select on database while secondary filters after records are returned from database. Primary filter we will set to last compliance failed. This will find only images that failed compliance check, in this case ones that have a high severity vulnerability. Secondary filter we will set to display only rules that have failed, are high severity and only in projects that have images.

Run Report.

Under Reports->Custom select the report "OpenScap High Security Vulnerabilities" and select queue from configuration dropdown. The report will show projects that have images with a high severity rule that failed.

Once projects and images are identified more detail may be obtained by looking at the OpenScap report. Under Compute->Containers->Container Images you can find the image and open it for more details. The OpenScap html report will contain details on specific rules and security violations.

Clicking the OpenScap HTML report will download it via browser. The report shows all rules and if they passed or failed.

You can dril into the rule and get more information. The summary shows the relevant CVEs.

Violations that caused the rule to fail are also shown.

Scanning Applications Inside Container Images

Until now we have been focused on mainly scanning the base OS image every container is built on. It is also possible and there are tools to allow scanning of layers above the base OS.  Red Hat provides pluggable API in Red Hat Enterprise Linux to support multiple scanners such as OpenSCAP, Aqua Security, Black Duck Hub, JFrog Xray and Twistlock. Red Hat OpenShift gives you the ability to use scanners with continuous integration and delivery (CI/CD) tools.

A best practice for application security is to integrate automated security testing directly into build or CI/CD tooling.

• Integrate with Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST) tools like HP Fortify and IBM AppScan.
• Add Scanners for real-time checking against known vulnerabilities like Aqua Security, Black Duck, JFrog and Twistlock. Tools like these catalog open source packages in your container, notify you of any known vulnerabilities and update when new vulnerabilities are discovered in previously scanned packages.
• CI/CD process should include policies that automatically open issues when security vulnerabilities are discovered in the build process so the devops team can take immediate action to resolve problems.

Finally there is an effort underway by Red Hat, Google and others to standardize auditing and policy enforcement with Kubernetes.

Summary

In this article we looked into the topic of security and vulnerability scanning of container images. Unfortunately this is a topic that doesn't get nearly enough attention in my view. We discussed the importance of why you want to keep container images updated, signed and get them from only trusted sources. We looked into several solutions provided by Red Hat. The Container Catalog, Atomic Cli and CloudForms. A guide was provided to explore each of these solutions individually, not only to understand their value, but also how to use them. In the end I think all three tools provide valuable information to ensure security standards are upheld. Choosing one of them or all of them depends on the processes an organization has defined.

Happy Secure Containers!

(c) 2018 Keith Tenzer