Home Lab

Cron and apt timers on a server that boots once a month

My Pi-hole box shut down on August 18, the day after I stripped it down to Pi-hole, and didn’t boot again until September 13. When it came back I found two jobs that had quietly stopped doing their work. Both were built for a machine that’s always on.

Pi-hole on that box is opt-in. I point a device at it when I want filtering, and it being down isn’t an outage, so the box is on when I want it, roughly once a month.


The backup that missed four Wednesdays

The backup was a root cron line:

0 2 * * 3

Cron fires at that minute or not at all. It has no idea the box was off. Four Wednesdays passed with the box powered down, and the last archive was from August 12.

There was a second bug under it. The copy of backup.sh on the box was still the May version, from before Gitea was removed. It still ran sqlite3 against Gitea’s database, and that path had been deleted on August 17, the day I removed Gitea. Under set -euo pipefail that one line aborts the whole run. So even on a Wednesday the box was up, the backup would have produced nothing.

The fix was a systemd timer with Persistent=true, which runs a missed window shortly after the next boot:

[Timer]
OnCalendar=Wed *-*-* 02:00:00
Persistent=true
RandomizedDelaySec=10m
AccuracySec=1m

The jitter stays small because the box is usually up for well under a day. A long random delay could push the run past shutdown. I deployed the trimmed script and ran it by hand: 96M archive, 26 adlists, 115 entries.


The updates that were always a month old

Unattended-upgrades had a quieter version of the same problem. Two stock timers do the work: apt-daily.timer refreshes the package lists, apt-daily-upgrade.timer installs. Both have Persistent=true, so both catch up after a boot. The order they catch up in is down to their random delays, and the stock delays are 12 hours and 60 minutes.

On September 13 the order came out backwards. The upgrade ran at 17:58, ten minutes after boot, against package lists from mid-August, and logged:

No packages found that can be upgraded unattended

The list refresh ran at 01:31, seven and a half hours after that, and the next install window was 06:00 the following day, which a box up for about eight hours often never reaches. So each boot installed what the previous boot had discovered. The box was permanently one boot behind, which on this box means a month.

The stock upgrade unit already has After=apt-daily.service, which looks like it should prevent this. It doesn’t. After= only orders units queued in the same transaction, and two timers firing on their own are two transactions. The fix pulls the refresh into the upgrade’s transaction:

# /etc/systemd/system/apt-daily-upgrade.service.d/override.conf
[Unit]
Wants=apt-daily.service
After=apt-daily.service

It’s cheap: the refresh no-ops if the lists are already fresh. Both timers’ delays went down to 10 minutes, since a 12-hour delay on a box that’s up for well under a day often pushed the refresh past shutdown.

The next 06:00 run installed 90 security packages, including openssh-server, openssl, libc6 and the kernel.

two timers, two transactionsbootupgraderefreshold lists, nothing to upgradeWants= pulls the refresh inbootrefreshupgrade

This is the second time unattended-upgrades in my lab went quiet for a reason that had nothing to do with apt. The first is in Nothing Crashed.