Incidents during implementation

Extracted from the master document on 2026-09-04. §N numbering preserved so as not to break cross-references.

44. Incidents during implementation

(a) $HOME under root — the failure that repeated three times.

Script launched with sudo or by systemd
        │
        └─► runs as root  →  $HOME = /root
                               │
                               └─► does NOT find ~/.config/rclone/rclone.conf
                                   nor ~/.ssh/... of the balsero user
                                       │
                                       └─► the backup fails at the upload
                                           step, sometimes silently

Solution applied consistently on all three machines: every credential a root/systemd script needs lives under /etc/<service>/, owned by root:root, permissions 600, and is referenced always by absolute path:

RCLONE_CONFIG=/etc/rclone/rclone.conf
BORG_RSH="ssh -i /etc/borg-backup/ssh/borg_backup ..."

(b) root permissions inside the Borg repository.

SYMPTOM   PermissionError: [Errno 13] Permission denied  (automatic backups)
CAUSE     borg-maintenance.sh ran prune and compact as root
          → left repo files owned by root
          while the producer writes as the `borg` user

sudo chown -R borg:borg /home/borg/repo

# and maintenance explicitly as borg:
sudo -u borg env BORG_PASSPHRASE="$BORG_PASSPHRASE" borg prune ...
sudo -u borg env BORG_PASSPHRASE="$BORG_PASSPHRASE" borg compact ...

The subsequent automatic run passed correctly, validating the fix.

(c) Concurrent maintenance. A manual run was still active when the 04:00 timer launched a second instance: two rclone sync runs towards the same destination. Mutual exclusion was added:

LOCK_FILE="/run/lock/borg-maintenance.lock"
exec 9>"$LOCK_FILE"
if ! flock -n 9; then
    echo "borg-maintenance ya está ejecutándose; saliendo."
    exit 0
fi

Controlled test: the second run bails out immediately without starting prune/compact/rclone.

(d) AIO passphrase lost. When recreating the mastercontainer to add NEXTCLOUD_BACKUPDIR it was necessary to log back into the AIO panel and the original passphrase had never been saved. It was recovered read-only:

docker exec nextcloud-aio-mastercontainer \
  grep password /mnt/docker-aio-config/data/configuration.json

Lesson applied on the spot: that passphrase and the one for the AIO backup were saved in Vaultwarden immediately.

(e) Crash of nextcloud-aio-apache (Exited 137). During a WebDAV upload of ~100 MB, Apache died with code 137 (SIGKILL). No evidence of the OOM-killer was found in dmesg/journalctl, so the exact cause remains open. Relevant data point: docker-host has ~3.8 GiB of RAM and at the time ~1.6 GiB remained available. Watch for a repeat.

(f) DNS failure during the first AIO backup. The first attempt stopped the 9 Nextcloud containers but could not download ghcr.io/nextcloud-releases/aio-borgbackup (never used, not cached) because the VM’s DNS —which depends on Tailscale MagicDNS (100.100.100.100)— hiccuped. Result: 9 containers stopped and “Slim Application Error” in the panel. It was resolved with the “Start containers” button in the panel itself (it reorders the startup respecting dependencies, without needing the network) and afterwards the image was pulled by hand:

docker pull ghcr.io/nextcloud-releases/aio-borgbackup:latest

That way, if there is another network hiccup, the image is already cached.

---