A Django Migration Bug That Hid for a Year

Some bugs crash loudly on day one. The worst ones wait. This one lived in our Django migrations for roughly a year — invisible most days, then occasionally detonating in CI or, worse, during a deploy. This is the story of finding it and the two rules that finally made it stop.

The setup: many apps, tangled together

Our backend was split into many Django apps, the way large Django projects usually are — accounts, billing, catalog, orders, and so on. Apps are great for organisation, but they share one global thing: the migration graph.

Every migration declares what it dependencies on, and Django topologically sorts the whole graph — across apps — to decide the order to apply them. As long as those cross-app dependencies are honest, everything is fine. Ours had quietly stopped being honest.

The symptoms (that never quite added up)

For a year, three things kept happening, seemingly at random:

1. Fresh databases failed to build. On a developer's existing DB, everything worked. Spin up a clean database from scratch — CI, a new hire, a staging rebuild — and a migration in one app would blow up trying to read a field or model another app had already deleted.
2. A deploy removed a field another app was still reading. A destructive migration in catalog dropped a column that orders code still referenced. The migration ran; the reads broke.
3. Circular dependency errors. orders depended on billing, billing depended back on orders through a later migration, and Django couldn't order the graph at all — CircularDependencyError, no way to migrate.

The reason it took a year: on an already-migrated database, none of this shows. The damage only appears when migrations replay in order on a clean DB, or when a destructive change lands in production before its readers are gone.

The root cause: deletes and adds weren't ordered against their readers

Once I stopped chasing individual failures and drew the dependency graph, the pattern was obvious. Destructive changes weren't sequenced relative to the things that depended on them.

• A migration that deleted a field could be ordered before the app that still read that field had migrated off it.
• A migration that added something could be ordered after an app that already needed it.
• And because dependencies were declared ad-hoc in both directions, two apps could end up pointing at each other — the circular case.

In other words: app B was depending on whatever the latest state of app A happened to be, instead of the specific state it actually needed.

The fix: a reader rule and a delete rule

I didn't want a clever migration; I wanted a convention the whole team could follow mechanically. Two rules did it.

1. The reader rule — pin to the last stable version

If app B depends on something in app A, B's migration must depend on the last stable migration of A before any destructive change — the version that still has what B needs.

So B never floats on "whatever A is now." It pins to the exact A it was written against. The destructive change in A is then forced to come after B in the graph, because B's dependency anchors it there.

# orders/migrations/0014_use_catalog_sku.py
class Migration(migrations.Migration):
    dependencies = [
        ("orders", "0013_..."),
        # Pin to the LAST STABLE catalog migration that still has `sku`,
        # NOT to catalog's latest. This forces the delete to sequence after us.
        ("catalog", "0021_sku_stable"),
    ]

2. The delete rule — destructive changes run last

A migration that drops a field or model must depend on every app that read it, so the delete is ordered after all of them have migrated away.

This is the expand / contract (parallel-change) pattern made explicit:

1. Expand: add the new thing; readers move onto it.
2. Contract: only once nothing reads the old thing does the destructive migration run — and its dependencies name those readers so the graph can't reorder it earlier.

# catalog/migrations/0030_drop_legacy_sku.py
class Migration(migrations.Migration):
    dependencies = [
        ("catalog", "0029_..."),
        # The delete cannot run until these readers have moved off `sku`.
        ("orders", "0017_migrate_off_sku"),
        ("billing", "0009_migrate_off_sku"),
    ]
    operations = [migrations.RemoveField("Product", "sku")]

Additions get the mirror rule

New rows and fields are the same problem in reverse: a dependent must not run before the thing it needs exists. So the dependent migration depends on the addition migration — the add is guaranteed to be in place first.

Why this also kills the circular case

Circular dependencies happen when two apps point at each other's moving heads. Once every cross-app dependency points at a specific, stable, already-finished migration instead of a live one, the back-edges disappear — and the graph becomes a clean DAG that Django can always order.

What I'd tell past-me

• On an existing DB, your migrations are lying to you. Always test a build from zero in CI — that's the only place cross-app ordering bugs surface.
• Never depend on another app's latest migration. Depend on the exact one you need. "Latest" is a moving target; a named migration is a contract.
• Make destruction a two-phase ritual. Expand, let readers move, then contract — and encode "then" as a real dependency, not a hope.

A year-long bug turned out not to need a heroic fix. It needed a couple of rules that made the migration graph tell the truth.