Part of Fourteen Automations That Run Without Me.
Once you run agents on a server you own, you have taken on a small system administration job whether you wanted one or not. There are credential files on that disk. There are ports listening. There are repositories that agents commit to, several times a day, without you reading the diff. The realistic threat is not a targeted attacker. It is entropy: a file that quietly became world-readable, a token pasted into a commit at 1 a.m., a service that started listening on a public interface after a package upgrade.
Entropy is exactly what a deterministic script is good at catching. So the security job in my fleet has no model in it at all.
Nothing in a posture check requires judgement. It requires someone to look, on a schedule, and to be unable to forget.
Why monthly, not daily
My instinct was daily. I was wrong, and the reason generalises.
These checks are stable. File permissions do not change on their own most weeks. The listening ports on a personal server change a few times a year. Running the check daily produces a report that is identical thirty times a month, and a report that is always identical stops being read — which means the one morning it differs, you skim it too. The real failure mode of a security job is not missing a check. It is alert fatigue, which is a failure of the human half of the system.
Monthly, on the first of the month, the report is an event. I read it. And because the job also sends a single line to my phone only when something is flagged, silence carries information: nothing changed.
The six checks
- Credential file permissions. Every file holding a secret — environment files, SSH keys, tool auth files — must be readable only by its owner. Anything group-readable or world-readable is flagged. This is the check that has caught the most real problems, almost always after I copied a file somewhere in a hurry.
- Repository leak scan. A regular-expression scan for key-, token-, and private-key-shaped strings over the last month of commits and the current working tree. Bounded to a month so the run stays fast; the working tree is included because the dangerous window is before the push, not after.
- Network surface. List every listening socket and compare against an allowlist kept in a small configuration file. A listener that is bound to localhost is expected; the same service appearing on a public interface is a finding. Any new public listener is filed as a real issue, not a note.
- Authentication on the web surface. Confirm the pages that are supposed to sit behind authentication still do, and that no other site configuration exposes an internal dashboard.
- Patch level. Count pending security updates by simulating an upgrade, and surface any reboot-required flag. A count, not an action.
- Remote access posture. Password authentication off, root login off, read from the effective configuration rather than the file where possible — because the file and the effective configuration disagree more often than people expect.
All six use tools that are already installed. There is no dependency to keep up to date, which is itself a security property.
Two hard rules
Read-only, always. The job never changes a permission, never edits a configuration, never installs a package, never restarts a service. It reports. I have been tempted to let it fix a bad file mode automatically — it is a one-line change with an obvious verifier, so by my own mechanical check test it would qualify as automatable. I have not done it, and the reason is worth stating plainly: a process that can repair permissions can also destroy them, and this is the one job whose failure I would not detect by other means. The check that watches everything else should be the least powerful thing on the machine.
Never write the secret down. The leak scanner reports a file and a line number. It never reports the matched value, not even partially, not in the report, not in an issue, not in a log line, not in a commit message. This is not paranoia about the report's storage. It is that a security scanner which copies secrets into its own output has built a second, less-guarded copy of every secret it finds — and that output goes into a repository, a chat message, and a log file.
Degrade, do not fail
The job's failure-mode table has three rows, and two of them are about missing capability rather than missing security.
| State | Signal | Behaviour |
|---|---|---|
| A check's tool is not installed | Run | Mark that check SKIPPED with a reason. Continue with the rest. |
| Elevated access needed, unavailable | Run | Degrade to the evidence a normal user can see, and say so in the report. |
| Kill-switch file present | Schedule fires | Exit immediately and successfully. |
SKIPPED is a first-class result, printed in the report beside PASS and FLAG. A check that silently does not run is worse than a check that does not exist, because it produces a clean report you believe. Make "I could not look" a visible outcome.
Testing a security check
The self-test runs on fixtures, with no network and no real secrets:
- A fixture file with loose permissions must be flagged; the same file with owner-only permissions must pass.
- The leak regular expression must hit a fixture token, and the resulting report must contain the file and line and must not contain the token. This is the test I care most about — it is an assertion about what is absent from the output.
- A fixture listing of listening sockets must be judged correctly against the allowlist.
- The SKIPPED path must render.
- The kill switch must short-circuit everything.
Writing an assertion that an output does not contain something is unusual in test suites and belongs in every redaction path you build.
The general lesson
There is a pull, once you have agents, to put a model into every job because you can. Resist it where parsing suffices. The model-free jobs in my fleet have these properties: they cost nothing to run, they cannot hallucinate a finding, they produce byte-identical output when nothing has changed, they work when the API is down, and their behaviour a year from now is the behaviour they have today.
Reserve the model for the jobs where judgement is genuinely the bottleneck — reading unfamiliar code, drafting prose, classifying ambiguous text. For "is this file readable by everyone", a regular expression is not a compromise. It is the better tool.
Read next: Ship the Read-Only Report First — how to start an automation that touches data you care about without being able to damage it.