On: Surviving Django, if you care about databases

[GlennS]

I've never worked on anything that separated data migrations from schema migrations, so I suspect I'm not quite understanding you, but...

Your data migration is just part of your chain of incremental patches?

For my example you just write:

    ALTER TABLE my_table ADD COLUMN my_column TEXT NOT NULL DEFAULT '';
    UPDATE TABLE my_table SET my_column = my_old_column_a || my_old_column_b;
    ALTER TABLE my_table ALTER COLUMN MY_COLUMN DROP DEFAULT;

[jashmatthews → GlennS]

Beyond a fairly small scale you have to separate data migrations from schema migrations because they just take too damn long.

[GlennS → jashmatthews]

I believe it and it makes sense, but I've never encountered it myself.

Sounds like your definition of "small scale" encompasses every project I've ever worked on. :D

[djrobstep → GlennS]

Right, that's exactly what you might add to a diff-generated script to do the same thing.

So it doesn't inherently seem better or worse to me.

[GlennS → djrobstep]

The difference is that - if I understood right - you're generating your diff just before you actually do the migration.

By comparison, the incremental migration scripts are written at the same time as the feature (I like to do them as step 1 at lot of the time) and by the same developer, who currently has the problem in their head.

[djrobstep → GlennS]

I don't know of anybody doing that - unless you had automated checks that it was a fast and non-destructive operation, that would be highly risky.

I'm simply talking about using the current production database as a basis for generating the required changes, rather than "here's what I think my production database currently looks like based on a version number and these 75 chained migration scripts I've run on it in the past"

[GlennS → djrobstep]

I'm still confused. When do you generate your diff then?

I think migrations ought to be done by the developer at the same time that they write their other code changes, because they have all the context at that time.

However, it doesn't seem like that could work for diffs, because the version of the production database you're diffing against may not be the same as the version you end up deploying to.

[djrobstep → GlennS]

Generate it whenever you prefer, probably with the rest of the code changes as you say.

Immediately before applying, check that the database is still in the same state as it was when you generated the diff script. If yes, apply. If no, abort.