Kubernetes cheat sheet
Kubernetes: Everything you need to know
Interactive course: Getting started with OpenShift
Red Hat OpenShift and Kubernetes ... what's the difference?
Interactive course: Deploy a cluster in Red Hat OpenShift Service on AWS (ROSA)
c. Test capabilities
Podman drops Linux capabilities when it starts a container. Podman runs root processes with the following capabilities by default:
CHOWN
DAC_OVERRIDE
FOWNER
FSETID
NET_BIND_SERVICE
SETFCAP
SETGID
SETPCAP
SETUID
SYS_CHROOT
Imagine running a build with a Containerfile that attempts to create a device node:
$ cat /tmp/Containerfile
from fedora
run mknod /dev/mynull c 1 3
Running rootful
podman build
on this Containerfile fails:
# podman build /tmp
STEP 1/2: FROM fedora
STEP 2/2: run mknod /dev/mynull c 1 3
mknod: /dev/mynull: Operation not permitted
Error: error building at STEP "RUN mknod /dev/mynull c 1 3": error while running runtime: exit status 1
Since
podman build
does not even have a
--privileged
flag, you need to start diagnosing a workaround. Check whether SELinux is causing the problem:
# setenforce 0
# podman build /tmp
STEP 1/2: FROM fedora
STEP 2/2: run mknod /dev/mynull c 1 3
mknod: /dev/mynull: Operation not permitted
Error: error building at STEP "RUN mknod /dev/mynull c 1 3": error while running runtime: exit status 1
# setenforce 1
Nope. The
podman build
command still fails while in permissive mode, so the problem is not likely to be SELinux. You could try adding all capabilities. (Note: Podman running with
--privileged
mode turns on all capabilities.)
You can turn on all capabilities for running a container by executing the following command:
# podman build --cap-add all /tmp/
STEP 1/2: FROM fedora
STEP 2/2: run mknod /dev/mynull c 1 3
COMMIT
--> ee04c826eb9
Ee04c826eb9bd8726fb234c83d9cc4c9218c433f56a804e0f06bbefa43fcf586
Because the container runs fine with all capabilities, you need to figure out which capability is required.
The most powerful Linux capability is SYS_ADMIN, so attempt that one:
# podman build --no-cache --cap-add sys_admin /tmp
STEP 1/2: FROM fedora
STEP 2/2: run mknod /dev/mynull c 1 3
mknod: /dev/mynull: Operation not permitted
Error: error building at STEP "RUN mknod /dev/mynull c 1 3": error while running runtime: exit status 1
The container ran fine with one of these missing capabilities, so you know one of them is the problem.
During diagnosis, ask what the service was attempting to do when it got permission denied. If it has something to do with the network, look at the network capabilities. Then search the capabilities list for something network related. Try to add those (NET_BIND_SERVICE, NET_BROADCAST, NET_ADMIN, NET_RAW, CAP_IPC_LOCK). In this case, the build is attempting to create a device node, so check that capability:
# podman build --no-cache --cap-add mknod /tmp
STEP 1/2: FROM fedora
STEP 2/2: run mknod /dev/mynull c 1 3
COMMIT
--> e24c8234f10
E24c8234f10fc2f8284ae91253891077dc038c30587f5a4a09f8e315218e7f14
Obviously, CAP_MKNOD is the missing capability.
Sometimes users have problems with a Podman container, and they tell me that it works with Docker. One reason for this is Podman runs with tighter security and fewer Linux capabilities than Docker. Podman drops a few capabilities that Docker allows by default. If a container runs with Docker but not Podman, try adding the missing capabilities: NET_RAW, SYS_CHROOT, AUDIT_WRITE, MKNOD.
# podman build --no-cache --cap-add sys_chroot,net_raw,audit_write,mknod /tmp
STEP 1/2: FROM fedora
STEP 2/2: run mknod /dev/mynull c 1 3
COMMIT
--> 3faef5a3084
3faef5a3084427343165c3d322e410108e18ba2b86f45ccee0ab6771a654fcf
If you want to really get down and dirty, you can use
strace
to attempt to get the actual syscall that is being denied.
[ Check out this free guide to
boosting hybrid cloud security and protecting your business
. ]
d. Test SECCOMP
Podman uses SECCOMP to limit the number of system calls available within a container. The list of syscalls is shipped in the
/usr/share/containers/seccomp.json
file. Working with seccomp files is a little advanced, so I usually just tell people to see if the container runs with seccomp separation disabled. You can disable SECCOMP easily and see if the container runs:
$ podman run –security-opt seccomp=unconfined
Sometimes SECCOMP denials show up in
/var/log/audit/audit.log
. Instead of turning off SECCOMP entirely, generate a profile for the specific workload and container. Please refer to
Improving Linux container security with seccomp
to learn how to do that with Podman.
e. Test masked kernel filesystems
Podman masks over several kernel filesystems to prevent processes within the container from certain activities on the kernel filesystems. Sometimes the processes inside the container might need to access one of these masked kernel filesystems. When running in
--privileged
mode, Podman does not mask any of the kernel filesystems. You can also run containers without the masks by executing:
$ podman run --security-opt unmask=all …
Use
man podman run
to display the unmask options:
● unmask=ALL or /path/1:/path/2, or shell expanded paths (/proc/*): Paths to unmask separated by a colon. If set to ALL, it will unmask all the paths that are masked or made read only by default. The default masked paths are /proc/acpi, /proc/kcore, /proc/keys, /proc/latency_stats, /proc/sched_debug, /proc/scsi, /proc/timer_list, /proc/timer_stats, /sys/firmware, and /sys/fs/selinux.. The default paths that are read only are /proc/asound, /proc/bus, /proc/fs, /proc/irq, /proc/sys, /proc/sysrq-trigger, /sys/fs/cgroup.
2. Namespace issues
I have covered all the standard security separations. Next, I will look at namespaces.
[ Learn how to
explain orchestration in plain English
. ]
a. Is user namespace the issue?
One of the most common issues and bug reports our team gets is that the XYZ container image works fine with Docker but blows up with Podman. This is almost invariably because the user is running rootful Docker and rootless Podman. Rootless Podman uses the
user namespace
, which causes some security issues and can cause permission to be denied. You can diagnose this by telling the user to attempt to run the container as root, which would match up to the default experience with Docker:
$ sudo podman run …
For example, examine what happens if you try to run a
MariaDB
image:
$ mkdir /tmp/data
$ podman run --env MARIADB_ROOT_PASSWORD=passwd -v /tmp/data/:/var/lib/mysql mariadb
Error: open /tmp/data/mysql: permission denied
Remember from the first section of this article that SELinux blocks access to random content on disk, so you need to add the
:Z
option:
$ podman run --env MARIADB_ROOT_PASSWORD=passwd -v /tmp/data/:/var/lib/mysql:Z mariadb
Error: open /tmp/data/mysql: permission denied
Nope, still broken. You can try it as root:
$ sudo podman run --env MARIADB_ROOT_PASSWORD=passwd -v /tmp/data/:/var/lib/mysql:Z mariadb
2022-04-06 20:28:15+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.7.3+maria~focal started.
2022-04-06 20:28:15+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-04-06 20:28:15+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.7.3+maria~focal started.
Skip to the bottom of list
When it works in rootful mode but not rootless mode, there is a good chance the issue is with
user namespace
. User namespace tends to cause issues when volumes are mounted into containers, similar to the SELinux problems. Usually, the problem is a non-root user inside a container, say the MySQL UID 999, trying to access a volume mounted from the host user's home directory. By default, the UID of the host user is treated as UID 0 inside the container. The MySQL user of the MariaDB container (UID 999) is not allowed to read and write from it.
In a user namespace, this UID is not simply UID==999. It is offset by the range of UIDs in
/etc/subuid
. On my system, this UID 999 inside the container is mapped to UID 100998 outside the user namespace. For this issue, Podman makes it easy by adding a
:U
option. The
:U
tells Podman to recursively
chown
the volume to match the default user found inside the user namespaced container.
First, stop the rootful container from running, and then remove and recreate the
/tmp/data
directory since the actual root user owns the content in this directory:
$ sudo stop -f
$ sudo rm -rf /tmp/data
$ mkdir /tmp/data
Now run the container again in rootless mode, this time with the
:U
option:
$ podman run --env MARIADB_ROOT_PASSWORD=passwd -v /tmp/data/:/var/lib/mysql:Z,U mariadb
2022-04-06 16:30:53-04:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.7.3+maria~focal started.
2022-04-06 16:30:53-04:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-04-06 16:30:53-04:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.7.3+maria~focal started.
If you'd like more information, I wrote about volumes and user namespace in
Dealing with user namespaces and SELinux on rootless containers
.
Another common issue with the user namespace is using a UID that is not mapped within the user namespace. By default, rootless users only use 65537 UIDs. If you use a UID greater than that, the user namespace treats it as undefined, and it will not be allowed. You can see the user namespace mappings of the container with the
podman unshare
command:
$ podman unshare cat /proc/self/uid_map
0 3267 1
1 100000 65536
One common problem is a failure with an unmapped user. You can probably run the container as root, or you need to expand the number of UIDs mapped for the user in
/etc/subuid
and
/etc/subgid
files.
Note: If you ever modify those files, you need to run
podman system migrate
afterward to use them in a logged-in session.
b. Is network namespace the issue?
Sometimes the issue is caused by the network namespace; you can disable this and use the host's network namespace using the
--net=host
flag:
$ podman run --net=host …
This runs the container in the host's network. Note that you still won't have full access to the network; if you are running as rootless, some access is prevented even if you have added all caps. For example, rootless users are not allowed to bind to ports < 1024:
$ podman run -p 80:80 ubi8/httpd-24
Error: rootlessport cannot expose privileged port 80, you can add 'net.ipv4.ip_unprivileged_port_start=80' to /etc/sysctl.conf (currently 1024), or choose a larger port number (>= 1024): listen tcp 0.0.0.0:80: bind: permission denied
This happens so often that Podman tells the user about it, and even describes a special sysctl that can be set to allow non-root users to bind to port 80:
$ sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
net.ipv4.ip_unprivileged_port_start = 80
$ podman run -p 80:80 ubi8/httpd-24
=> sourcing 10-set-mpm.sh ...
=> sourcing 20-copy-config.sh ...
=> sourcing 40-ssl-certs.sh ...
c. Issues with PID or IPC namespaces
Similar to a network, you could have issues with containers caused by the
PID or IPC namespaces
. It is simple to turn off the separation on these by executing with the
--pid=host
and
--ipc=host
options:
$ podman run --pid=host --ipc=host …
3. Try rootful containers
Skip to the bottom of list
Cloud services
What is hybrid cloud?
eBook: Modernize your IT with managed cloud services
Get started with Red Hat OpenShift Service on AWS (ROSA)
Managed services vs. hosted services vs. cloud services: What's the difference?
Some containers just require root. Usually, very privileged containers that want to modify the system will not work in rootless mode. Luckily, these are very rare. To run a container that mounts different types of filesystems, you need to run it in rootful mode.
The bottom line is that in rootless mode, you can only change system parameters related to namespaces and can only do what a normal user can do. Podman does not add anything special to the system, but it takes advantage of the namespaces in clever ways.
Wrap up
It should rarely be necessary to run with
--privileged
mode; if you spend a small amount of time investigating which of the subsystems is failing, you should be able to run with tighter security. You might be moving away from Papa Bear, but you don't need to go all the way to Mama Bear.
Daniel Walsh has worked in the computer security field for over 30 years. Dan is a Consulting Engineer at Red Hat. He joined Red Hat in August 2001. Dan leads the Red Hat Container Engineering team since August 2013, but has been working on container technology for several years.
More about me
The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. The content published on this site are community contributions and are for informational purpose only AND ARE NOT, AND ARE NOT INTENDED TO BE, RED HAT DOCUMENTATION, SUPPORT, OR ADVICE.
Red Hat and the Red Hat logo are trademarks of Red Hat, Inc., registered in the United States and other countries.