# Triaging home lab alerts with Claude Code

My home lab sent its alerts to my phone through ntfy ([Alerting](/articles/alerting) has the setup), until I shut it down ([Shutting Down](/articles/shutting-down)). When one fired, I pasted it into Claude Code, and `incident-triage` decided what happened next. Its rule is to answer from the servers' logs, because `CLAUDE.md` only says what should be true.

It starts by taking the alert verbatim and pinning a ±10 minute window in UTC. If all I wrote was "I got an alert", it asks for the body before grepping anything. Then it checks who actually sent the alert, since a title can be wrong after a wrapper script changes, and reads that script to see which condition tripped.

Evidence comes from more than one place. My public server forwarded its syslog to the LAN server, so the logs survived a reboot of the box that was in trouble. Before escalating, it reads a list of known noise: bots probing the public server, churn in log and cache paths. Then the incident goes into one bucket, noise, operational or security-relevant, and it proposes the smallest correct action with anything destructive flagged.

One case is a hard stop. A file-integrity diff on a system binary gets the path and the hash shown to me, the baseline left alone, and a real investigation.

The output was short because I usually read it on my phone: one headline sentence, three to six quoted log lines, the commands, and what to watch for the next day.

Save it as `.claude/skills/incident-triage/SKILL.md` and fill in the table at the top. Start the known-noise file empty and add to it each time triage comes back "noise".

````markdown
---
name: incident-triage
description: >
  Use whenever the user reports an alert, a strange notification, a spike on a daily summary,
  a file-integrity diff, a service that went down, or any "what just happened" investigation.
  Triggers on "got an alert", "what's this notification", "investigate this", "triage X",
  "something looks off on <host>", or a pasted alert body.
---

# incident-triage

Drive the investigation from raw signal, to corroborated evidence, to the smallest response.
Answers come from the machines, not from what CLAUDE.md says should be true.

## Fill these in for your setup

| What | Yours |
|---|---|
| Alert channel and topics | e.g. ntfy topic `alerts` |
| Scripts and units that send alerts | e.g. `~/scripts/*.sh`, `nginx-watcher.service` |
| Where live logs are | e.g. `journalctl`, `docker compose logs nginx` on `web-1` |
| Where archived logs are | e.g. forwarded syslog in `/var/log/remote/web-1/` on `nas-1` |
| Known non-issues | a file you keep, e.g. `docs/known-noise.md` |

## Step 1: Capture the signal

Get the alert text verbatim. Pull out:

- **Source**: which topic, service or cron job.
- **Timestamp**: convert to UTC if the user quoted local time. Pin a ±10 minute window.
- **Specifics**: IP, domain, path, service name, error code. Use them as search keys.

If the user only says "I got an alert", ask for the body before running commands. Don't
blind-grep the logs.

## Step 2: Confirm the source (don't trust the title)

Titles mislead after wrapper-script changes. Find what actually sent it:

```bash
journalctl --since='30 min ago' | grep -iE 'ntfy|<your alert scripts>' | tail -40
grep CRON /var/log/syslog | tail -50
```

If it came from cron, find the exact script and read it to see what condition it tripped on.

## Step 3: Pull corroborating evidence

Run these in parallel where you can, inside the window from Step 1.

- The live host: the relevant service logs, `docker ps --format 'table {{.Names}}\t{{.Status}}'`,
  `systemctl --failed`, `uptime`, `df -h /`.
- The archive, if logs are forwarded elsewhere: it survives a reboot of the host in trouble.
- The alerting script's own log: `journalctl -u <unit> --since='1 hour ago' --no-pager`.

## Step 4: Classify

Read the known-non-issues file first. Recurring scan IPs your ban script already handles, and
integrity diffs on log, cache or tmp paths, are expected churn.

Put the incident in exactly ONE bucket:

- **Noise**: known pattern, no action. Note it and stop.
- **Operational**: degraded but not malicious (disk full, crash loop, cert near expiry).
  Propose a fix.
- **Security-relevant**: unexpected access pattern, integrity diff on a sensitive path, an
  unfamiliar IP succeeding (not just probing). Go to Step 5 carefully.

## Step 5: Propose a response

Show the smallest correct action first:

- **Disk full**: `docker system df`, find the bloat, propose a prune or log rotation. Confirm
  before running prune.
- **Crash loop**: `docker logs --tail 200 <name>`, name the error, propose the fix.
- **Unauthorized access attempt**: confirm the IP is already banned; if not, propose it.
  Never block an IP pre-emptively that hasn't actually attacked.
- **Cert expiry**: propose a forced renewal for that one cert.
- **Integrity diff on a system binary**: STOP. Show the exact path and hash diff. Do not
  update the baseline. This one gets a real investigation.

## Step 6: Document

If the incident was new and is now resolved, propose writing it down where future sessions
will read it. For a recurring false positive, propose adding it to the known-non-issues file.

## Output

1. **Headline**: one sentence with the source, the bucket and the root cause.
2. **Evidence**: 3 to 6 quoted log lines that prove the headline, with file paths.
3. **Proposed action**: exact commands, destructive ones flagged for confirmation.
4. **Follow-up**: anything to watch over the next 24 hours.

Keep it tight. The user is usually reading this on a phone right after the alert.
````

see also: [Alerting](/articles/alerting), [The Setup](/articles/the-setup)
