Menu
Article homelab

🧱 Reviving a Broken LXC Container

Drew Cranmer

Drew Cranmer

• 2 min read

While it's probably not the best thing to admit, I've been neglecting my homelab for some time now. Gotify had been giving me errors for a at least a month and eventually seemed to stop working entirely. So it wasn't a complete surprise when I logged into PVE and saw the container hosting my miscellaneous utilities (Gotify, Grafana, Homepage, etc.) wasn't running.

What started as a routine pct start 100, however, turned into a deep dive through Proxmox, ZFS, UID mapping, and AppArmor. Here's how I debugged and resurrected a broken LXC container running Docker in my homelab — and the key lessons along the way. Much like this very blog post, I owe most of the credit to Chat GPT.

⚠️ The Initial Problem

When trying to start an LXC container on my Proxmox VE host, I hit this cryptic error:

run_buffer: 571 Script exited with status 1
lxc_init: 845 Failed to run lxc.hook.pre-start for container "100"
__lxc_start: 2034 Failed to initialize container "100"
TASK ERROR: startup for container '100' failed

At first glance, it’s not obvious what’s failing — especially when the container config looks valid and the root filesystem is still intact.

🔍 Step 1: Confirm the Filesystem

I verified the container’s ZFS subvolume existed:

zfs list | grep subvol-100

Then I mounted it:

pct mount 100
ls /var/lib/lxc/100/rootfs

✅ Full filesystem structure was present, which was a relief, — but…

🚨 All files were owned by 100000:100000, which is expected for unprivileged containers. I had recently switched this one to privileged, and LXC expects root:root.

🛠️ Step 2: Fix Ownership

To convert the filesystem from unprivileged to privileged:

chown -R root:root /VM/subvol-100-disk-0

But I hit another roadblock:

chown: disk quota exceeded

Turns out ZFS refquota was silently blocking metadata writes.

🧯 Step 3: Remove ZFS Refquota

zfs set refquota=none VM/subvol-100-disk-0
zfs set quota=16G VM/subvol-100-disk-0

With that resolved, chown -R root:root ... succeeded.

✅ The container started!

🐳 Step 4: Docker Compose — Round Two

Now that the container was up, I tried launching my stack:

docker compose -f docker-compose-master.yml up -d

...only to hit this:

AppArmor enabled on system but the docker-default profile could not be loaded
...
You need policy admin privileges to manage profiles.

This is a classic conflict when running Docker inside LXC: Docker tries to load AppArmor profiles, but LXC blocks that at the kernel level.

🔐 Step 5: Disable AppArmor for LXC

I edited /etc/pve/lxc/100.conf and added:

lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: a
lxc.cap.drop:
features: nesting=1,keyctl=1

Then restarted the container:

pct restart 100

🎉 Success! All services launched, and Docker stopped complaining.

✅ Takeaways

  • Switching an LXC from unprivileged to privileged requires a full chown to root:root

  • ZFS refquota can silently block chown and metadata updates

  • Docker inside LXC requires lxc.apparmor.profile=unconfined and disabling capability drops

  • Always snapshot your LXC containers before big changes:

    pct snapshot 100 pre-docker-fix

If you're running Docker-heavy services in LXC, it’s worth weighing whether a lightweight VM might offer fewer headaches — but if you're committed to containers inside containers, it's all solvable with a bit of elbow grease.

—

Want help with composing your own container stack or Proxmox network? Hit me up — or stay tuned for more homelab dispatches.

Share

💬 Join the Discussion