Here’s a bug I wrote. It’s the most useful one I’ve produced in a year, and the kind of bug it is is sitting in your practice right now.
I have a pre-publication check for this site. Its job is narrow: before anything goes live, make sure no published article links to an article that isn’t published. A dead internal link is a small thing. It’s also exactly the kind of small thing that ought to be mechanical instead of remembered.
I wrote it as a shell loop. Roughly this:
CLEARED="alpha beta gamma"
for slug in $CLEARED; do
grep -qs "draft: false" "posts/$slug.md" \
|| echo "PROBLEM: $slug is not published"
done && echo "Safe to publish."Run it, get “Safe to publish,” push. It said that every single time.
It had never checked anything.
What actually happened
Two separate bugs, and either one alone would’ve done it.
Bug one: zsh doesn’t word-split unquoted parameter expansions.
zsh is the default macOS shell, and it isn’t bash. In bash, $CLEARED in a for list gets split on IFS and turns into three words. In zsh it doesn’t. You get one word — the whole string.
I set up a directory with alpha.md and beta.md published and gamma.md marked draft: true, then ran the identical script under both shells. Here’s what actually came out:
### zsh
PROBLEM: alpha beta gamma is not published
Safe to publish.
exit=0
### bash
PROBLEM: gamma is not published
Safe to publish.
exit=0Under zsh the loop ran once, against a file named posts/alpha beta gamma.md, which doesn’t exist. grep returned non-zero because it couldn’t open the file, not because anything in the file was wrong. The || branch fired and printed one line naming a mashed-together nonsense slug. That line looks enough like a real per-item message that you scroll right past it, especially in a script you’ve learned always prints something harmless.
gamma — the actual problem — never got looked at. Not once, in any run, ever.
Bug two: the && was decoration.
Look at the bash output. Bash split correctly, found the real problem, and printed Safe to publish. anyway.
done && echo "Safe to publish." keys off the exit status of the for loop, and a for loop’s exit status is the status of the last command in its body. The last command in the body was echo "PROBLEM: ...", which worked fine. Zero. So the “Safe” message printed because the failure message printed successfully.
Flip the grep’s sense around and it gets worse. Now the loop’s status is whatever the last grep returned, so whether “Safe” shows up depends on what’s in whatever file happened to come last. The message never had anything to do with the result.
Nothing was actually broken. I checked the whole site afterward with a tool that works, and every link resolved. That was luck. Luck isn’t verification, and that’s the whole point of this piece. For weeks I had a green light wired to a switch that wasn’t hooked up to anything, and I was steering by it.
Worse: a check that prints the passing result
The loop above printed a reassuring message. There’s a common cousin that does something worse, and it’s the default shape of every quick check anybody writes in a shell. Say you’re sweeping a couple of files for strings that must never show up on the site:
P="fileA.md fileB.md"
grep -n -E '<email regex>' $P || echo " none"
grep -n -E 'FORBIDDEN' $P || echo " none"Same cause: $P doesn’t get word-split, so it turns into one filename that doesn’t exist. But look at what this version prints:
noneThe check didn’t just fail to run. It printed the passing result.
grep couldn’t open the file, so it exited non-zero. || fired. And the message on the right side of that || was none — the exact word this script prints when it’s scanned the files and found nothing wrong. Anybody reading the output for problems sees a pass. You can’t tell it from a real pass, because the two produce byte-identical output.
And don’t count on stderr to save you. grep writes a complaint to stderr, so in a terminal there’s a warning line you might catch. But where you’d actually put a check like this — a git hook, a cron job, a launchd agent, CI — stderr gets redirected or tossed, and nobody reads the transcript. All anybody ever sees is the word none.
That’s the whole mistake in one line:
cmd || echo "clean"conflates “ran and found nothing” with “did not run.”
It’s how almost everybody writes a quick check in a shell, and it can’t tell apart the two things you most need told apart.
grep hands you the fix for free, and almost nobody uses it. Its exit status has three values, not two. Verified:
found -> 0
not found -> 1
cannot read -> 2So handle all three:
FILES=(a.md c.md missing.md)
worst=0
for f in $FILES; do
out=$(grep -n -E 'TKTK' "$f" 2>&1); rc=$? # unfinished-copy placeholder
case $rc in
0) echo "VIOLATION $f: $out"; worst=1 ;;
1) echo "clean $f" ;;
*) echo "ERROR $f: $out"; worst=2 ;;
esac
done
exit $worstRun against one clean file, one with a violation, and one that doesn’t exist:
clean a.md
VIOLATION c.md: 1:Rates start at TKTK per hour.
ERROR missing.md: grep: missing.md: No such file or directory
script exit=2Three outcomes, three messages, three exit codes. ERROR doesn’t look like clean, and the script’s own status says “I could not do my job” rather than “your files are fine.”
Here’s the bigger lesson under the small one. Knowing about a trap isn’t the same as having a check that catches it. Nobody writes cmd || echo "clean" because they think an unreadable file is the same as a clean one. They write it because it’s the shortest thing that prints something, and what it prints looks like success. That’s why I’m pushing the discipline further down, and why I’d get verification logic out of the shell altogether.
The shell fixes, and why I wouldn’t stop there
For the record, all four of these give you three iterations under zsh. I ran each one:
for s in ${=CLEARED}; do ... done # explicit split operator
CL=(alpha beta gamma); for s in $CL # an array, which is the idiomatic answer
setopt shwordsplit # make zsh behave like bash, globally
print -r -- $CLEARED | IFS=' ' read -rA arrInside the shell, the array is the right answer. setopt shwordsplit is the tempting one, and I’d stay away from it: it changes behavior for everything in the script, including code you paste in later that was written expecting zsh to act like zsh.
But the real lesson is a level up. I didn’t know zsh behaved differently. I’ve written shell for twenty years, and I’d have told you flat out that unquoted expansion splits. I didn’t fail because I didn’t know. I failed because I bet a verification step on knowing.
Here’s the rule I use now, and I think it’s the right rule for any lawyer building automation:
Shell is fine for orchestration — run this, then that, in this order. Shell is a bad place for anything whose result you will rely on. If the output of a step is going to be treated as a fact, get out of the shell.
Shell isn’t bad. The trouble is that it behaves differently by interpreter, by version, by option flags, by whether some variable happened to have a space in it, and by whether a macOS upgrade swapped your login shell out from under you. Every one of those differences is silent. None of them raise.
Exit status is the contract
The rewrite is thirty lines of Python. The language doesn’t matter. What matters is that it exits non-zero.
#!/usr/bin/env python3
"""linkcheck.py — no article may link to an article that is not published.
Exits 0 only if it checked something and found nothing wrong.
Exits 1 on a defect. Exits 2 if it could not do its job.
"""
import re, sys
from pathlib import Path
ROOT = Path(__file__).resolve().parent
COLLECTIONS = {"practice-lab": ROOT / "src/content/practice-lab",
"news": ROOT / "src/content/news"}
INTERNAL = re.compile(r"\]\(/(practice-lab|news)/([a-z0-9-]+)/?\)")
DRAFT = re.compile(r"^draft:\s*(true|false)\s*$", re.M)
def is_published(collection, slug):
"""True / False, or None if the target does not exist at all."""
path = COLLECTIONS[collection] / f"{slug}.md"
if not path.is_file():
return None
m = DRAFT.search(path.read_text(encoding="utf-8"))
return m is not None and m.group(1) == "false"
def main(argv):
for name, d in COLLECTIONS.items():
if not d.is_dir():
print(f"FATAL: {d} does not exist", file=sys.stderr)
return 2
if argv:
targets = []
for slug in argv:
hits = [d / f"{slug}.md" for d in COLLECTIONS.values()
if (d / f"{slug}.md").is_file()]
if not hits:
print(f"FATAL: no article named {slug!r}", file=sys.stderr)
return 2
targets.extend(hits)
else:
targets = sorted(p for d in COLLECTIONS.values() for p in d.glob("*.md"))
if not targets:
print("FATAL: nothing to check", file=sys.stderr)
return 2
links = defects = 0
for path in targets:
for collection, slug in INTERNAL.findall(path.read_text(encoding="utf-8")):
links += 1
state = is_published(collection, slug)
if state is None:
defects += 1
print(f"BROKEN {path.name} -> /{collection}/{slug}/ (no such article)")
elif state is False:
defects += 1
print(f"DRAFT {path.name} -> /{collection}/{slug}/ (target is draft: true)")
print(f"checked {len(targets)} file(s), {links} internal link(s), {defects} defect(s)")
if links == 0:
print("FATAL: examined 0 internal links - the check proved nothing",
file=sys.stderr)
return 2
return 1 if defects else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))Run against the live content directory:
checked 192 file(s), 432 internal link(s), 0 defect(s)
exit=0There are four design choices in there, and they’re the part you can take home.
1. Three exit codes, not two. 0 = checked and clean. 1 = checked and found a defect. 2 = couldn’t run the check. That third one is the one people leave out, and it’s the one that caused the original bug. “I could not do my job” must never be reported as “your work is fine.”
2. A missing input is fatal, not skipped. Pass a slug that doesn’t exist and you get exit 2. The tempting alternative — skip it, keep going, report on whatever you found — is exactly how a typo in an argument turns into a clean bill of health.
3. It won’t call it a success when it proved nothing. If it looked at zero internal links, it exits 2 and tells you why. That guard is four lines, and it’s the most valuable thing in the file.
4. It shows its own denominator. 192 file(s), 432 internal link(s). Those numbers are how a person notices that today the check looked at eleven links instead of four hundred. A check that only ever prints a verdict never gives you anything to be surprised by.
Because it exits the right way, you can plug it into other things:
# .git/hooks/pre-push (chmod +x)
#!/bin/bash
set -euo pipefail
python3 linkcheck.py# .github/workflows/check.yml
- run: python3 linkcheck.pyNeither of those needs to know a thing about link checking. They need one thing: a non-zero exit. A check that prints a message and always exits 0 can’t be automated at all. The only way to use it is to have a person read it carefully, and a person’s attention is exactly what you were trying to save.
Break it on purpose
This is the habit the whole piece is about.
Plant the exact problem the check is supposed to catch, and make sure it fails. Not “confirm it passes on good input” — anything passes on good input, including a script that does nothing.
Here’s the recipe, run for real:
- Create a temporary article with one link to a slug that doesn’t exist and one link to an article whose frontmatter says
draft: true. - Run the check on it.
- Confirm both the message and the exit code.
- Delete the temporary files.
BROKEN zz-temp-linkcheck.md -> /practice-lab/an-article-that-does-not-exist/ (no such article)
DRAFT zz-temp-linkcheck.md -> /practice-lab/zz-temp-draft-target/ (target is draft: true)
checked 1 file(s), 2 internal link(s), 2 defect(s)
exit=1Then the third state, on a file with no internal links at all:
FATAL: examined 0 internal links - the check proved nothing
exit=2Thirty seconds of work. It would’ve caught the original bug on day one, because that bug had exactly one visible symptom: it was incapable of printing anything but “Safe.” One planted problem would have blown it wide open.
Two tweaks make the habit stick.
Keep the broken input around. Don’t delete the poisoned file after you’ve watched it fail. Check it into a fixtures/ directory and add a test that asserts the check rejects it. Now the habit is permanent, instead of something you remembered to do once.
Write the failing test first when you can. Test-first is more than a slogan here. A test written after the code, against code you already believe works, is a test you’ve never seen fail. It documents what you believe. It doesn’t prove anything.
And for anything in a law office that’s supposed to stop you, there’s an obligation-shaped version of this question: when did this last say no? A conflicts screen that’s never returned a hit. A deadline validator that’s never rejected a date. A privilege filter that’s never held back a document. Any of them might be fine. Every one of them also looks exactly like a return True, and you’ve got no evidence which one you own.
The rest of the family
The shell bug is one member of a big family. Every one of these reports success without having done anything.
The empty loop over an empty list. The most common one. I ran this:
FILES = list(Path("posts").glob("*.markdown")) # wrong extension
def check_all():
return [f.name for f in FILES if "draft: true" in f.read_text()]
print("files considered:", len(FILES))
print("verdict:", "SAFE" if not check_all() else "STOP")files considered: 0
verdict: SAFEOne glob typo — .markdown for .md — and the function is perfectly correct and perfectly useless. A list comprehension over an empty list returns an empty list, and an empty list of problems reads as “no problems.” The guard:
if not FILES:
raise SystemExit("FATAL: matched 0 files - the check proved nothing")The grep that matches nothing. Same shape, and the second case above is one example. Here’s another: grep -q "Rule 11" brief.pdf matches nothing because a PDF isn’t text, so a check for “does this brief mention Rule 11” answers “no” with total confidence, forever. Any grep whose “not found” branch is a success message needs its exit status checked, not its output.
The try/except that swallows. except Exception: pass is the fastest way there is to turn a crash into a wrong answer. If you have to catch broadly, log the exception and re-raise or return a failure state. Never return the success state.
The filter that filters out everything. A date range with the ends reversed, a status filter for a value somebody renamed, an AND where an OR belonged. Zero rows out. Downstream, zero problems.
The assertion in code that never runs. The validation’s right there in the file, reviewed, correct — behind a condition that’s never true, or in a function nobody calls anymore. It looks like coverage. It’s decoration.
And set -euo pipefail, which you need and which isn’t enough. Put it in every script:
-eexit on error-uerror on unset variables-o pipefaila pipeline fails if any stage fails, not just the last
That last one surprises people, so here it is:
'false | cat' exit=0
with pipefail exit=1By default a pipeline only reports its final command’s status, so generate_report | tee out.txt succeeds whenever tee succeeds — including when generate_report crashed and produced nothing.
It has limits, though. I re-ran the original loop under set -euo pipefail, and it still ran once, still checked nothing, and still exited 0. Word splitting isn’t an error. || catches the failure before -e ever sees it. set -euo pipefail protects you from errors. It doesn’t protect you from a script that’s doing the wrong thing correctly.
The professional part
A verification step is worth exactly as much as its willingness to say no.
I mean that literally. A check that says “clean” on every input tells you nothing at all, and it’s worse than no check, because having it is what let you stop looking. My green light didn’t just fail to help. It took the place of the attention I’d otherwise have paid.
So quit asking every automated safeguard in your practice “is it running.” Ask it “when did it last refuse something, and have I ever watched it refuse?”
If it’s never said no, that’s not a track record. That’s an absence of evidence, and it should earn your suspicion, not your trust.
Sources
Every behavior described above was reproduced by direct execution on macOS while writing this article; the outputs shown are actual outputs, not illustrations.
- zsh 5.9 (arm64-apple-darwin25.0), the default login shell on this machine, compared against bash on identical input. zsh does not word-split unquoted parameter expansions in a
forlist; bash does. The same is true of an unquoted expansion supplied as a command’s argument list, which is how the second incident occurred. ${=VAR}, array assignment,setopt shwordsplit, andIFS=' ' read -rAeach verified to produce three iterations under zsh.for … done && echoverified to bind to the exit status of the loop’s last body command in both shells.grepexit status verified three-valued on this system:0match found,1no match,2file could not be read. The three-waycasehandler above was executed against a clean fixture, a violating fixture, and a missing file, producing the three distinct outputs and exit status 2.cmd || echo "clean"verified to print the clean message and exit 0 whencmdfailed for an operational reason rather than a substantive one.false | catverified to exit 0 by default and 1 underset -o pipefail.set -euo pipefailverified not to prevent the original defect.linkcheck.pyexecuted against this site’s live content directory (192 files, 432 internal links), against a deliberately poisoned fixture (exit 1), and against a file with no internal links (exit 2).
General commentary on engineering practice for law firm tooling. Not legal advice. All example files and slugs are invented for this article; no client information, matter, or document appears anywhere in it. Questions about anything here: Send us a message or 612-470-6529.