Mullvad VPN in a NixOS systemd-nspawn container

TL;DR: NixOS is a Linux distribution that you configure via a single file that kinda looks like JSON. Mullvad VPN typically hijacks your entire network connection, so this post describes how to configure NixOS to run the Mullvad VPN in a container and SSH'ing into that container to create a socks proxy. Then you can configure apps or browsers or browser tabs to specifically connect to that socks proxy.

This is meant for NixOS laptop/desktop users, but you could also run NixOS on a VPS or server. Then create a socks proxy on your MacOS/Windows laptop via SSH to that VPS. If the VPS was for Mullvad VPN only, you could even ignore the container and run Mullvad VPN directly on the VPS.

The Mullvad VPN app, by default, routes all traffic through the VPN. This is a sensible default for most users because it ensures no traffic leaks from various apps and/or when the VPN connection drops.

But what if you don't want Mullvad to hijack your entire network connnection. Maybe you only want certain websites or apps to go through Mullvad's VPN? Or maybe you have multiple VPN connections (Mullvad allows up to 5) and/or multiple VPN providers?

You could:

configure Mullvad exit node through your Tailscale VPN - if you're a Tailscale user, and/or Tailscale app connectors
configure Wireguard - if you're comfortable maintaining Wireguard configs and keys
run Mullvad in a Docker image (Gluetun) use Shadowsocks client/server proxy - if you're comfortable managing client+server apps
run Mullvad in a Docker image and use Socks proxy - if you're using Docker
on-demand provision AWS ec2 instances with your ssh key and simply ssh -D 1337 into them - but Mullvad is cheaper and more flexible (and maybe more secure?) than using VPS for VPN

But I wanted something simpler. A socks proxy via SSH and a pasteable configuration.

Quick summary

Conceptually, the NixOS container config will look something like:

# systemd-nspawn container
containers.mullvad-vpn = {
  config =
      { pkgs, ... }:
      {
        services.mullvad-vpn.enable = true;
      };
  };
};

# create socks proxy on host's 1337 port using ssh
services.autossh.sessions = [
  {
    name = "mullvad-socks-proxy";
    user = "myuser"; # not root, but the user that holds the ssh key
    extraArguments = "-D 1337 -nNT root@localhost";
  }
]

Then, using Firefox's multi-account containers add-on (which allows you to assign a sandbox per tab including a proxy config). Click the extension icon in toolbar, manage containers, click container, click "Advanced proxy settings" (ignore Mozilla VPN), and enter "socks://localhost:1337".

Full config:

Of course, the devil is in the details and it's rarely as simple as it seems. We need a few hacks to get things working:

cgroup to fix "EPERM: Operation not permitted"
mounting /etc/mullvad-vpn to avoid generating new mullvad devices on boot
add mullvad disconnect/connect inside container to fix transient reconnect issues
modify autossh to wait for the mullvad-vpn to actually have a connection

Note I mounted from /persist, which is an idiomatic location when using NixOS impermanence. But you can create directories anywhere and mount them to the container. The objective is avoid losing your wireguard key (stored in /etc/mullvad-vpn/device.json) across reboots. The critical mount is /etc/mullvad-vpn, the other mounts are optional.

Warning!!

In this example we're storing our Mullvad account number (see "1111111111111111") in the NixOS configuration, which means it will be in the NixOS "store" and world readable.

# skim this blog post:
# https://blog.beardhatcode.be/2020/12/Declarative-Nixos-Containers.html
networking.nat.enable = true;
networking.nat.internalInterfaces = [ "ve-mullvad-vpn" ];

# change this to your actual network interface (run ifconfig or ip a)
networking.nat.externalInterface = "eth0";

# critical fix for mullvad-daemon to run in container, otherwise errors with: "EPERM: Operation not permitted"
# It seems net_cls API filesystem is deprecated as it's part of cgroup v1. So it's not available by default on hosts using cgroup v2.
# https://github.com/mullvad/mullvadvpn-app/issues/5408#issuecomment-1805189128
fileSystems."/tmp/net_cls" = {
  device = "net_cls";
  fsType = "cgroup";
  options = [ "net_cls" ];
};

containers.mullvad-vpn = {
  ephemeral = true;
  autoStart = true;
  privateNetwork = true;

  # these IP choices are arbitrary, copied from https://blog.beardhatcode.be/2020/12/Declarative-Nixos-Containers.html
  hostAddress = "192.168.100.2";
  localAddress = "192.168.100.11";

  bindMounts = {
    "/etc/mullvad-vpn" = {
      hostPath = "/persist/etc/mullvad-vpn";
      isReadOnly = false;
    };
    "/var/cache/mullvad-vpn" = {
      hostPath = "/persist/var/cache/mullvad-vpn";
      isReadOnly = false;
    };
    "/var/log/mullvad-vpn" = {
      hostPath = "/persist/var/log/mullvad-vpn";
      isReadOnly = false;
    };
  };

  config =
    { pkgs, ... }:
    {

      # apparently need this for DNS to work
      networking.useHostResolvConf = false;
      services.resolved.enable = true;

      services.openssh.enable = true;
      users.users.root.openssh.authorizedKeys.keys = [
        # replace with your actual public key
        "ssh-ed25519 ...."
      ]

      services.mullvad-vpn.enable = true;
      # each mullvad account login will generate a new "device" (wireguard key)
      # and you're limited to 5 devices per account
      # go to https://mullvad.net/en/account/devices to clear out old devices
      systemd.services."mullvad-daemon".postStart = ''
        while ! ${pkgs.mullvad}/bin/mullvad status >/dev/null; do sleep 1; done

        # REPLACE with your actual mullvad account number
        account="1111111111111111"

        # only login if we're not already logged in otherwise we'll get a new device
        current_account="$(${pkgs.mullvad}/bin/mullvad account get | grep "account:" | sed 's/.* //')"
        if [[ "$current_account" != "$account" ]]; then
          ${pkgs.mullvad}/bin/mullvad account login "$account"
        fi

        ${pkgs.mullvad}/bin/mullvad lan set allow
        ${pkgs.mullvad}/bin/mullvad relay set location us sjc
        ${pkgs.mullvad}/bin/mullvad lockdown-mode set on
        ${pkgs.mullvad}/bin/mullvad auto-connect set on

        # disconnect/reconnect is dirty hack to fix mullvad-daemon not reconnecting after a suspend
        ${pkgs.mullvad}/bin/mullvad disconnect
        sleep 0.1
        ${pkgs.mullvad}/bin/mullvad connect
      '';
    };
};

# create a socks proxy on host port 1337
services.autossh.sessions = [
  {
    name = "mullvad-socks-proxy";
    user = "ubuntu";
    extraArguments = "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -D 1337 -nNT root@192.168.100.11";
  }
];

# hack to wait for mullvad-daemon to be ready before starting autossh
systemd.services.autossh-mullvad-socks-proxy = {
  serviceConfig.ExecStartPre = lib.mkBefore [
    "+${pkgs.bash}/bin/bash -c 'until ${pkgs.openssh}/bin/ssh -i /home/ubuntu/.ssh/id_ed25519 -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@192.168.100.11 ping -c1 google.com; do ${pkgs.coreutils}/bin/sleep 1; done'"
  ];
};

Debugging and maintenance

Searching Sourcegraph/GitHub

query is: lang:Nix "bin/mullvad"

https://sourcegraph.com/search?q=context:global+lang:Nix+%22bin/mullvad%22&patternType=keyword&sm=0

Host machine (your laptop/desktop)

manually create proxy instead of autossh: ssh -D 1337 -C -N -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@192.168.100.11
logs for autossh: journalctl -u autossh-mullvad-socks-proxy.service -f
ssh into container: ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@192.168.100.11

Container (Mullvad VPN)

logs: journalctl -u mullvad-daemon.service -f
change mullvad location: mullvad relay set location us nyc
list mullvad locations: mullvad relay list

Headless-Render-API