The unglamorous side of software engineering

Nobody posts about the week they spent staring at a red CI badge. This is that week.

It started with a merge. A big feature branch, weeks of work, and I merged the main line into it. Git said the conflicts were resolved and the app compiled, so I assumed all was well. It wasn’t. Two things had gone wrong without making a sound:

• One conflict had been settled by keeping the wrong side, so a screen quietly went back to an older version.
• A tiny import vanished, so one page crashed at runtime.

Nothing flagged the first one, because the code was still valid. The only way to catch it was to diff the result against both parents and read it.

The reason any of this was survivable is that none of it happened in production. We run a dev environment: every push to the dev branch deploys to a staging server, and only main reaches real users. So when the merge broke things, when CI went red, when the staging build ran out of memory, the worst outcome was a failed deploy and a rollback to the previous version. Nobody’s orders were affected. It’s easy to treat staging as a chore, but it’s the place where you’re allowed to be wrong cheaply.

The fix for “I only found out after merging” was to stop relying on me noticing. I wrote CI workflows for all three repos (the API, the web app and the admin app) that run lint, type-checks, tests and a full build on every pull request and every push to dev and main. On the backend I added a few checks aimed at exactly the things that bit me:

• a test that boots the entire app, so a broken module wiring fails in CI instead of at startup on a server
• end-to-end tests against a real database for the riskiest feature
• a check that database migrations only ever add things, never change or remove them

Then the noise began.

• Thousands of lint errors from one missing file. A fresh install had wiped the generated database client, so every type quietly became any. Regenerating it cleared the lot.
• Tests that passed on my machine and died in CI. One needed a mail service the runner didn’t have, another spent so long clearing tables that the test timed out. A local mail catcher and one bulk cleanup fixed both.
• A formatting check that flagged hundreds of files. Nobody had ever run it across the whole repo. Rather than touch every file in one rushed commit, I switched the step off until we can do a proper format-everything pass.

The worst one came at deploy time. The staging server kept crashing mid-build with an out-of-memory error. It has under 2 GB of RAM, and compiling the project wants more. I could have added swap and hoped. Instead, CI (which has plenty of memory) now builds the app and ships the compiled output to the server, which unpacks it and restarts. If something goes wrong, it rolls back to the previous version.

One thing was very clear to me. Code is cheaper to write than ever. A feature that used to take days can land in an afternoon, and it’s tempting to push it as soon as it compiles. But the checking didn’t get any cheaper. Reading a diff, thinking about the merge, noticing the quiet regression: that’s still a human job, and it’s the first thing to slip when everything else speeds up. A lot of the messy code I found wasn’t careless so much as unchecked. Nothing in the process made anyone look. The faster code gets written, the more your guardrails have to do the noticing for you.

The rest was housekeeping: stopping a hosting platform from redeploying a preview on every push, getting into a server when I didn’t have the SSH key (the cloud console’s browser terminal works fine), and resetting an admin password properly, hashed, instead of poking the database.

I took six things from the week:

1. A merge with no conflict markers isn’t a merge you’ve checked.
2. Fix the noisy checks first, or nobody trusts the ones that matter.
3. If a small server can’t build, build somewhere else.
4. Most “bugs” were configuration, so read what a setting does before changing it.
5. Have somewhere safe to break things. A staging environment that deploys automatically and rolls back when it fails turns a bad week into a boring one.
6. When AI makes code cheap to write, make checking it automatic. Speed without guardrails is just a faster way to make a mess.

None of it ships a feature. All of it is why the next feature ships.