As container technology adoption grows, the need to provide governance and inspection of these containers and platforms also grows.

One of the nice things about container images is that they are easier to analyze than a traditional application (which may be spread across many directories and files) since everything you need to analyze exists in that container image somewhere.

Container vulnerabilities bring a converged vulnerability footprint of both application and operating system package vulnerabilities. This means your container needs to be treated like an application in some respects, but you also need to analyze the dependencies that are along side the application inside the container, which are often linux packages in the case of linux based containers.

Most of the container scanning solutions out there are fairly immature in that they still mostly treat containers like a virtual machine. They ask the container to dump out its package list (dependencies) and create a finding if they are not at the latest version. Unfortunately, this approach completely ignores the application and/or application runtime itself in many cases. As container scanning solutions mature, they are going to need to differentiate themselves by how well they can analyze the application and application runtimes that exist in containers.

One good solution due to this lack of toolset convergence is to

• Scan & analyze application artifacts before they are allowed to be layered onto a container, then
• Scan the container itself after it is built.

This way you are covering both the application and its dependencies.

Some challenges with scanning container repositories and registries.

• Huge registries and/or repositories of container images.

Some large registries may have hundreds or thousands of different repositories. Each repository could have hundreds of container images. This can easily lead to registries that have tens or hundreds of thousands of container images. I imagine we will soon see registries with millions of container images if they don’t already exist.

Most container scanners know not to rescan things they have already seen, but the first scan on large registries can take a very long time in many cases.

This huge volume of containers can cause a few challenges, and here are some ideas on how to overcome those challenges.

• Your repo/registry scanner must be designed to scale out or up to handle 10’s of thousands of containers. This usually means…..

• The container scanner backend must track track the container layer hashes and container hashes to know what it has not already scanned. It obviously shouldn’t scan layers or images it has already scanned.

• The container scanner backend must be able handle multiple concurrent scans against multiple images or repositories. It should be able to scale up if needed. This means your scanner backend design has to be able to handle multiple concurrent scanners and be able to distribute work between them properly.

• The container scanner should implement shortcuts to know if it has already scanned images from a registry without necessarily checking every layer and image hash. If you pull down a registry manifest with 10,000 images, the next time you pull the manifest, you should try to diff the manifests to determine what are “new” images and scan those first.

• A good approach is for container scanner companies to “pre-load” containers and container layers from public registries. This way you may be able to avoid even have to scan many of the layers of the containers.

• Container scanners should natively support the main container registries in cloud providers like Azure, Google, etc.. by knowing how to use their API’s enough to access the container registries and repositories they provide.

• A container scanner should usually try to scan in a LIFO approach by scanning newer images first. This can be difficult because container tags and version tags are not very structured. You can try to scan all “latest” tags first. One field I think could be valuable to be added to the docker registry manifest is the timestamp of the image. Since tags are not structured enough to be reliable, you could use the timestamp or epoch to at least know when the container was last modified or placed in a repo.

• You want to use the LIFO approach because newer containers are the ones most likely to be used, and the ones that need to be analyzed as part of CI/CD integrations

Those are my thoughts on scanning large container registries and repositories. Do you have any thoughts on optimizing container scanning for large registries? I imagine similar work has been done on different types of file or artifact scanning in the past. It seems like we always try to “reinvent the wheel” in security products for some reason.