If you use containers frequently, there's a good chance you'll want to see the file system of a running container at some point. Maybe the container isn't working properly, you want to read some logs, maybe you want to check some configuration files inside the container... Or, you might, like me, want to put some eBPF probes on the binaries in the container (more on that later).
Whatever the reason, in this article, we will cover some methods that you can use to check the files in the container.
We'll start by looking at the simple and usually recommended methods of container file systems and discuss why they don't always work. Next, we'll have a basic understanding of how the Linux kernel manages container file systems, and we'll use that understanding to examine file systems in a different but still simple way.
If you do a quick search for how to check the file system of a container, you'll find that a common solution is to use Docker commands:
This is a great place to start. If it meets all your needs, you should continue to use it.
However, one disadvantage of this approach is that it requires the presence of a shell in the container. If there is no /bin/bash, /bin/sh, or other shell in the container, then this method will not work. For example, many of the containers we built for the Pixie project are based on distroless and do not include a shell to keep the image small. In these cases, this method does not work.
Even if the shell is available, you won't have access to all the tools you're used to. Therefore, if grep is not installed in the container, then you cannot access grep either. This is another reason to find a better job.
If you dig a little deeper, you'll realize that container processes, like other processes on a Linux host, simply run in namespaces to isolate them from the rest of the system.
So you can use the nsenter command to enter the namespace of the target container, using something like this:
It enters the mounted (-m) namespace of the target process (-t $PID) and runs /bin/bash. Going into a mount namespace essentially means that we get a view of the file system that the container sees.
This approach seems more promising than docker's exec method, but it also encounters a similar problem: it requires that the target container contain /bin/bash (or other shells). If we don't enter a mount namespace, we can still access the files on the host, but because we entered the mount namespace before executing /bin/bash (or other shells), we wouldn't be lucky if there was no shell in the mount namespace.
Another way to solve this problem is to simply copy the relevant files to the host and then use the copied files.
To copy selected files from a running container, you can use:
You can also take a snapshot of the entire file system in the following ways:
These commands enable you to examine files, and when the container may not have a shell or the tools you need, these commands are a big improvement over the previous two methods.
The copy method solves many of our problems, but what if you try to monitor log files? Or, what if you're trying to deploy an eBPF probe to a file in a container? In these cases, replication does not work.
We want to access the container's file system directly from the host. The container's files should be in the host's file system, but where?
Docker's inspect command gives us a clue:
This gives us:
Let's analyze:
LowerDir: A file system that contains all layers within the container, except the last layer
UpperDir: The topmost file system of the container. This is also where any runtime modifications are reflected.
MergedDir: A combined view of all layers of a file system.
WorkDir: The internal working directory used to manage file systems.
Overlayfs-based container file system structure.
Therefore, to view the files in the container, simply look at the MergedDir path.
If you want to learn more details about how file systems work, you can check out Martin Heinz's blog post on the overlay file system: https://martinheinz.dev/blog/44.
Leaving the best for last, there's also an easier way to find the container file system from the host. Using the host PID of the in-container process, you can simply run:
Linux already gives you a view of the process mount namespace.
At this point, you might be wondering: Why don't we take this approach and turn it into a one-line blog post? But it's all about journeys, right?
Out of curiosity, all the information discussed in Method Four about the container overlay file system can also be found directly from the Linux /proc file system. If you look at /proc/<pid>/mountinfo, you'll see something like this:
Here you can see that the container has been mounted with an overwrite file system as its root. It also reports the same type of information as docker inspect reports, including LowerDir and UpperDir for container file systems. It doesn't show MergedDir directly, but you can just use UpperDir and change diff to merged so you can see the container's file system.
At the beginning of this blog, I mentioned how the Pixie project needed to place eBPF probes on containers. Why and how?
Pixie's internal Stirling module collects observable data. Because it's k8s native, much of the data collected comes from applications running in containers. Stirling also uses the eBPF probe to collect data from the processes it monitors. For example, Stirling deploys eBPF probes on OpenSSL to track encrypted messages (if you want to learn more about this, see the SSL tracing blog[1]).
Because each container is bundled with its own OpenSSL and other libraries, any eBPF probes deployed by Stirling must reside on files within the container. Therefore, Stirling uses the techniques discussed in this article to locate the libraries of interest in the K8s container and then deploys the eBPF probes from the host onto those binaries.
The following diagram outlines how deploying an eBPF probe in another container works.
Stirling deploys eBPF probes on other containers by mounting the host file system, and then locates the target container file system on the host.
The next time you need to check the files in the container, I hope you'll try these tricks. Once you experience the freedom to no longer be limited by whether a container has a shell or not, you may never go back. Just need to access /proc/<pid>/root!
[1]
SSL Tracking Blog: https://blog.px.dev/ebpf-openssl-tracing/
Original English address: https://blog.px.dev/container-filesystems/
This article is reproduced from: "CNCF", original text: https://tinyurl.com/56nu8xra, copyright belongs to the original author.