Migrate your Django monolith to microservices. The 3rd step — migrating custom User model

Hi there!

This is the last post in a series explaining separation of existing Django project on several subapps:

• Migrate your Django monolith to microservices. The 1st step — preparation.
• Migrate your Django monolith to microservices. The 2nd step — understanding Django model and migration management.
• Migrate your Django monolith to microservices. 3rd step — migrating custom User model.

Today I am going to clarify how to deal with custom User model, if you want to transfer it to another Django app inside one project for potential further separation into another microservice. If you want to remind how Django tracks changes in models and affects database correspondingly, go to the previous posts by links presented above.

So, let’s call to mind current state of our car repairment center’s app. We’ve transferred 2 existing models, Detail and Liquid, to store app and we’ve created new app, user, to place User model there. The next goal is to technically shift custom User model to app user, because we are going to implement user-related logic (marketing and promotion mailout, mobile app authorization, etc.) here instead of mixing it with repairment process logic:

6041c45 2021–12–24 | Step #3 New app “user” created (models version 1.0 still)

We changed AUTH_USER_MODEL setting to new location, user.User, and used SeparateDatabaseAndState operation to transfer as we’ve already done with Detail and Liquid, but didn’t succeed:

3a36793 2021–12–24 | Step #3 FAIL Just try to move User to “user” app (move models to version 2.0)

Let’s take apart why User may not be transferred the same way as another custom models.

User is something special for Django

As it is explained in documentation, User is extendable model, which developer may configure himself in settings.py. If you change this settings, Django uses your custom model everywhere it deals with user-related logic — authentication system, permissions, migrations, another models referring to user with foreign keys and so on. And if we look at migration, which was created for Order model, we will see unusual dependency, migrations.swappable_dependency:

Django uses swappable dependency in migrations which related with User model, specified in AUTH_USER_MODEL setting

This is how customization of User model is implemented in Django — developer just creates his own model, sets up in settings.py, and Django uses special syntax and methods to transparently pull this model through all logic, including migrations.

But how is it connected with our error during migration process? Do you remember that Django tracks changes of models by 2 states — disk, or code-based, and database; and every time you run migration-related management command Django loads these two states and does some consistency checks. One of such checks looks at django_migrations table and disallows further execution if there are migrations which were already applied but depend on unapplied migrations. This is really important point! I’d like to make it clearer.

Let’s go step back, when we haven’t transfed User and changed USER_AUTH_MODEL setting yet. How did django_mirgations look like at this moment?

Applied migrations before User transfer. Left side visualize db table — green-bordered are our migrations, black — Django’s; arrows show how migrations depend on each other

From Django consistency check’s point of view there is everything ok, because every applied migration depends on applied migrations too. Now let’s return to our current state with transferred model and changes AUTH_USER_MODEL settings, and look at migrations’ graph:

Left side — migration graph from the previous step; right side — graph for current disk state, commit 3a36793. Dashed border — unapplied migrations transferring User; dashed arrows — dependencies on swappable User model how they were early; orange arrows — dependencies on swappable User model as they whould be after transfer.

On the right side you may see that some of already applied migrations (e.g. store.0001, admin) depended on migration repair.0001, which created our custom User model initially, but now these migrations depends on newly created user.0001 migration. And this is the reason of our exception — changing AUTH_USER_MODEL changes migrations order and Django signals that graph is inconsistent because there is at least one applied migration which depends on unapplied. Any migration-related command will not succeed until we fix this inconsistency. But how may we change migration graph if Django won’t run any related command?

Cheat Django

At glance it seems as unsolvable task — say Django about changes in migrations when it doesn’t listen our commands. But there is one small trick, I found in Django’s ticket system. If we adapt it to our case, we may fix the problem in 2 variants:

1. Manually apply migrations with --fake flag as explaind in ticket — suitable for systems without strong CI/CD. Appling it is as simple as possible.
2. Fake migrations by means of RawSQL in 2 consequent deployments — suitable for systems with high automation, with configured CI/CD, when it’s really hard or even impossible to interfere in deployment process especially only once.

The first is quite simple, just follow ticket discussion, if this variant is enough in your project. Ticket is quite old, but the core idea is still the same.

The second is more tricky. In brief, this case seems like variation of the first one, but I didn’t find anything more detailed for CI/CD, so I figured it out myself. Let’s study this variant out more precisely.

Hack Django’s migration system

What do we need is to hack Django’s consistency check — it should see that migration user.0001 is already applied, before we run final migrate command with 2 SeparateDatabaseAndState operations, transferring User model. And following the first approach we would do something like this:

1. Add db_table to User model with new location (user_user in our example);
2. Move User model to user app;
3. Recreate all migrations;
4. Apply them with --fake flag.

But how may it be implemented without --fake flag?

The first step is the same — we need to add db_table to User model with new location (commit f10a3b2).

Commit f10a3b2 | Step #3 User model table name fixed

The --fake flag internally inserts corresponding rows to django_migrations table without applying actual operations to database. This may be replaced with RawSQL operation, which inserts rows which represent migrations transferring User model:

Commit 301a8b8 | Faking further migrations where User model will be moved

At this moment user.0001, user.0002, repair.0006 migrations don’t exist in our code and disk state, we just add information about them to django_migrations table — this imitates --fake flag behaviour. Django won’t react somehow because it considers rows which are not related to migrations existed in code as trash.

Commit 301a8b8 | Step #3 Faking further migrations where User model will be moved

After applying this migration with RawSQL, database state will look like on picture above — repair.0005 and user.0001 migrations are considered as applied.

These are changes for the first deployment. Go to 301a8b8 and try run migrate command.

The last changes are actually transferring User model to new app: transfer in code, searching for all occurances; change AUTH_USER_MODEL; create migrations with SeparateDatabaseAndState operations shifting User and changing CRUD permissions:

Commit 7d68537 | Add migrations, transferring User, which were faked earlier

Be careful with naming, migrations must have the same name as you inserted in django_migrations table earlier. It is really important, otherwise Django will consider these migrations as new and will try to apply them.

These are actions for the second deployment, which are actually the same as transferring any other model, eg. Liquid or Detail. Go to commit 7d68537 and apply the rest of migrations:

Apply migrations in 2 deployment: commit 301a8b8 and 7d68537.

As you may see in logs (or on picture above) the first migrate command applied migration which fakes further ones with actual model’s transfer. And because of that the second migrate command (with --plan flag in the picture) does nothing. In the second deployment we just bring code-based state to database’s state without changes in database.

The last action is to remove custom db_table in User model, because now it’s location in code corresponds table name.

You may ask, why should we do all these actions in two separate deployments? The reason is in Django migrations’ consistency checks — without faking migration where User model will be created in new app (user in our example), Django always sees inconsistency. So, two-step deployment is tough but only way.

We successfully finished transferring User models to newly created app. You’ve deployed everything as we’ve learned. System works well. Now we are ready to implement all desires of your uncle, marketing mailout, mobile app, etc! But this is another story:)

Conclusion

Now you know how to separate your Django monolith on several subapplications even if it needs transfer models, including custom or default User model.

Let’s highlight some important points:

1. Don’t forget about database’s dump and restore!
2. For transferring custom models you need to create two migrations with SeparateDatabaseAndState operations as we discussed in the previous posts;
3. Don’t forget about such an unevident changes like CRUD and permissions for transferred model; FK, o2o and m2m links to transferred model; etc.
4. User model, custom or default, is a little bit more tricky:
   — you need two sequential deployments to shift it from one app to another, in production there musn’t be done anything between them;
   — notify system users, your colleagues and product owners that maintenance operation is proceeding before and during deployment;
   — pay attention on migrations’ names;
   — don’t take any other changes to these two deployments except User model shifting and everything connected (CRUD permissions, etc).

You’ve learned how to step to such technical tasks; how to measure and plan them; how to simplify life of your colleagues working with this potentially breaking changes.

Hope, this series of posts will help you to deal with Django’s migrations and models. Don’t hesitate ask questions. Bye!