At 19:43 an agent working in my codebase ran bin/rails test to check one new
file. Seven tables emptied. Five point nine million episodes, three hundred and
ninety-four thousand transcripts, gone in the time it takes to read this
sentence.
I could stop there and blame the agent. It would be the wrong lesson. The command was ordinary. Everything that made it lethal was mine.
How a test run reaches production
The config file is correct. config/database.yml names mentionix_test on
localhost. Read it and you would swear the test suite cannot touch anything
real. But Rails merges DATABASE_URL into whichever environment boots, and my
transcription container sets DATABASE_URL to production. So RAILS_ENV=test
meant production, and the file that said otherwise never got a vote.
Then fixtures :all did what fixtures do. It deletes every row from every
fixtured table before it inserts. That is not a bug. That is the documented job
of a fixture loader, aimed at the wrong database.
I built that container. I put the production URL in its environment and left the test tooling installed beside it.
Nothing stopped it
Rails ships a guard for this. ActiveRecord::ProtectedEnvironmentError refuses
destructive work against production. It covers db: tasks. It does not cover
fixture loading. I knew the guard existed and let that stand in for knowing
what it catches.
My test suite had been broken for months. Every run died in before_setup. So
a destructive run and a harmless one printed the same error, and the output
read as the familiar breakage. The deletion had already happened by the time
anything reached the screen. A test suite nobody can run is not neutral. It is
camouflage, and I had been living behind it.
No alarm fired. A table went from five point nine million rows to two and every dashboard I own stayed green. I found it by opening my own homepage and seeing two episodes and zero podcasts.
What saved it
A backup from 04:30 that morning. Ten and a half gigabytes, and I had never once restored it. It worked, which was luck rather than diligence.
Restoring taught one thing worth carrying. A pg_dumpall carries
\connect mentionix_production inside it, so feeding the file to psql writes
straight back into the damaged database and destroys the evidence. Pull out the
one section you want, restore it into a new database, then swap the two with a
rename. Nothing gets dropped. Rollback stays one command away.
Two things survived that had no business surviving.
Deleted rows sit in the heap until vacuum reclaims them. Autovacuum was already
running on the episodes table when we found the damage, and it had not
committed yet. Turning autovacuum off froze six million dead rows and nine
gigabytes of TOAST in place. Worth knowing: setting autovacuum_enabled = false
does not stop a worker already running.
The search index never noticed. MeiliSearch syncs through ActiveRecord callbacks, and fixtures insert with raw SQL that skips them, so three hundred and nineteen thousand documents sat untouched while Postgres emptied. That index held the only remaining copy of the lost transcripts. It later became the control group for an unrelated experiment, which is a strange way to be repaid.
The bill
Fifteen hours, because I run no point-in-time recovery and the daily dump was the only place to land. Most of it recomputable: episodes refetch from RSS, transcripts rerun on the GPU. Eight hundred and twenty-six episodes lost their word timings for good.
What I would tell you
A config file that says mentionix_test is not evidence that you are on
mentionix_test. Environment variables outrank files and leave no mark in
them. Read the resolved value before anything runs:
bundle exec rails runner 'puts ActiveRecord::Base.connection_db_config.database'
One command, skipped, and it cost a corpus.
The guard now lives in config/environments/test.rb and refuses to boot the
test environment against any database whose name does not end in _test. It
covers rails runner and rails console too, because that is what anyone
reaches for first.
It is never the junior’s fault
When a junior drops the production database in their first week, nobody sane fires the junior. We ask who handed a first-week hire a production shell, and why one ordinary command could empty seven tables. The fault sits with the setup. It always did.
I hired a strange junior. This one forgets by morning.
A human junior carries the scar. They will check which database they are
pointed at for the rest of their career, and that flinch does more work than
any document. My junior arrives tomorrow with no memory of tonight, exactly as
sure of itself as it was at 19:43, and bin/rails test will look exactly as
harmless as it did then.
So the lesson cannot live in anyone’s head. I wrote it down, and writing it
down is worth something, but a note only helps the reader who stops to read it.
The environment file that now refuses to boot against a database not named
_test helps whether anyone reads anything.
That is the bar every safeguard here has to clear: it must work on a colleague with no memory of the last outage. It is a higher bar than a teammate who remembers, and it is the right one, because the teammate who remembered was me and I walked past this trap for months.
Point-in-time recovery comes next, and an alarm on row counts, so the next time five point nine million rows disappear the system says so before I open the homepage and find out.
Give an agent a production shell and it will eventually do the thing your setup allows. Mine allowed this. The fix is not a more careful agent. The fix is a database that refuses.