Running the Lab

apt-daily was OOM-killed 75 times on a NAS with no DNS

my NAS has no route to the internet and no DNS, on purpose. for nine days apt’s daily timer ran it out of memory anyway: 75 OOM kills by 2026-11-13, each one about 3 minutes at load ~6 with 5.5 GB in swap.

the first run of my status page found it: apt-daily-upgrade.service OOM-killed at 22:28 after peaking at 5.2 GB RAM + 5.5 GB swap, and at 01:45 apt-get at 5.2 GB RSS with 34 MB of memory left. that’s what the monit loadavg errors in the journal had been about.

the cause

it’s apt 3.0.3, Debian bug #1111554, fixed in 3.0.4. the download step of apt.systemd.daily fails every fetch with TmpResolveFailure, because there’s no DNS. apt counts that as transient, and the retry-delay path loops on “Tried to start delayed item, but failed”, piling warnings up in memory until the OOM killer comes.

why it started on 2026-11-04: the package lists were refreshed that day with no upgrade after, so 68 packages were pending and the offline download step finally had something to fetch.

reproduced under a cgroup cap: flat at 28 MB through the fetch timeouts, then 1.5 GB in about 20 seconds with no output. the same run with one flag stays at 30 MB and exits with Err::

apt-get -o Acquire::Retries=0 -d dist-upgrade
apt-get -d dist-upgradewith -o Acquire::Retries=01.5 GBabout 20 s28 MB, fetch timeouts30 MBErr:time, not to scale

the fix

apt 3.0.4 first and on its own, through the SSH tunnel I use for updates, then the other 59 packages. after that the same offline jobs stay under 50 MB. on 3.0.3 they reached 5.2 GB within minutes.

the cost: on 3.0.4 the offline jobs still sit in retry delays for a long time, holding the apt lock while they wait.


see also: apt through an SSH tunnel, timers that assumed an always-on box