Migrate your Django monolith to microservices. 2nd step — understanding Django model and migration management.

Hi there!
This is the second post in the series about refactoring Django project, including transferring models between apps. You may find other steps by links:

• Migrate your Django monolith to microservices. The 1st step — preparation.
• Migrate your Django monolith to microservices. The 2nd step — understanding Django model and migration management.
• Migrate your Django monolith to microservices. The 3rd step — migrating custom User model.

In the previous part we’ve done all preparation to start diving deeper in Django’s internals:

• cloned repo with project example, “Car repairment center”, started it and filled with initial data;
• fixed plan of our refactoring and shared it with teammates.

Let’s remind current state of our project — we are on 385e59 commit, new app store with one new model Order have just created, and our goal is to shift Detail and Liquid models to store app:

Commits 385e59, 7c933b5 (Fail), e4cae21b

Now we are ready to get down to business. Firstly, we need to answer, what does it mean for Django “To transfer model from one app to another”? To understand it, let’s check how Django tracks models and generates migrations.

How model relates to data in database

As it is said in documentation, if, for example, you create ordinary (not abstract, not proxy) model Post in application blog, Django creates database table blog_post — _-separated application’s and model’s names. Or, if you explicilty specified name in model Meta, this one.

Also when you create model, Django automotically creates CRUD (create-read-update-delete) permissions and ContentType instance representing new model in database.

So all what we need — is just change table name, somehow affect CRUD permissions and ContentType, which are also stored in database. How could we manage it? Let’s try simply transfer models Detail and Liquid to store app in code, rename table by means of RunSQL with ALTER TABLE and run makemigrations command. May be it would help, who knows:)

Commit 7c933b5 2021–12–24 | Step #2 FAIL Just try to shift Detail to store app (shift models to version 1.0)

Go to commit 7c933b5 in our test project. There we’ve shifted models and created 2 migrations:

• store/migrations/0002_move_detail_in_db.py— ALTERing table names;
• store/migrations/0003_move_detail_content_type.py — moving FK- M2M- links to Liquid and Detail models from another. Why do we need move links? Because Django has a lot of internal “magic” linking FK- and M2M- relations based on app_name + model_name (e.g. field’s related_name, link in ContentType to app_name).

So let’s try to apply that migrations, run manage.py migrate command.

Commit 7c933b5 2021–12–24 | Step #2 FAIL Just try to move Detail to store app (move models to version 1.0)

Oops, something went wrong. Django says that store app doesn’t have Detail model, but how is it possible? We’ve just shifted models to newly created app. Why Django didn’t recognize Detail? Let’s figure out!

How Django generates migrations

We know that when we create new model or change existing one in models.py and run manage.py makemigrations command, Django will autogenerate new migration with operations wich will affect database as we expect — creating table or altering it somehow. But how does it work internally? How does Django figure out that we changed this certain field or added this certain model?

Django uses 2 model’s states — database and code-based (or disk-based). The first one is a real state of tables at certain moment, this state is built from table django_migrations. The second is a state which is written in our python code in all migrations/ packages.

Django looks at this states every time we run management commands related to migrations (makemigrations, migrate, showmigrations, etc), compares them to each other, does some consistency checks, finds out difference and understands which operations should be applied on database to lead it to the same state as code-based model’s state.

So, when we transferred Detail and Liquid models, we assumed this are the same models, but from now they will be located in another app. From Django’s point of view they are not connected with old ones at all. It considers situation in such a way:

1. Models disappeared from repair app, but it didn’t captired in migrations;
2. There are 2 absolutely new models in store app, but their appearance didn’t captured in migartions (so Django doesn’t take them into account);
3. Also there is table renaming and changing FK- and M2M- links to nonexisting models in store app.

This is the reason of our exception, Django just doesn’t consider new models while they are captured in both, database and disk, states. To mark our models in store app as the same models, which previously were in repair, there is special operation SeparateDatabaseAndState:

What does it do? It just separates operations which Django should apply on 2 states, database and disk, allowing developers to write certain actions explaining model’s shifting to Django. Let’s try to use this operation to perform Detail and Liquid models transfer, go to commit 7c933b5b. There you may see 2 migrations, in old and new app:

• repair/migrations/0004_move_detail_liquid.py — migration in old app, telling Django two things: rename tables to lead them to new Detail and Liquid code-based state; alter code-base state with shifting models to store app;
• store/migrations/0002_move_detail_in_state.py — migration in new app, telling Django do nothing in database and just update it’s code-based state with models Detail and Liquid. Pay attention that this migration depends on 0004_move_detail_liquid.py, because we must make sure that order in which migrations will be applyed is deterministic.

Let’s display migration plan which Django will run by this migrations, run python manage.py migrate --plancommand and you will see that everything is ok, Django may apply that migrations, there is no inconsictency. Let’s apply them then, run python manage.py migrate:

Commit 4cae21b 2021–12–23 | Step #2 Detail model moved to “store” app successfully (models version 1.0)

Everything applied successfully, great! For now our project’s state is like we wanted, new store app have been created, models transferred:

You’ve tested new store features, demonstrated to your uncle, deployed it to server. Nothing crashed in database, you’re happy:) Everything works as your uncle wanted — customers buy spare parts; repairman dont’s stay waiting administrators to order spare parts and liquids, now they can order themselves; repairments took less time because all business processes accelerated. Your uncle is just happy! Business gives him so much money as he’ve never imagined!

We need more features!

Car repair center earned a lot and one day your uncle came and asked for new features which will proper business and customer’s experience to the next level. He gave an outline of new features and it includes such topics:

• mobile app to customers, where they may schedule their visit, buy spare parts, tracking repairment progress and so on;
• authentication through phone number, email and other auth services;
• marketing and promotion mailout, notifications about repairment progress;
• and so on and so forth.

Listening all this wishes and dreams you figured out that user’s logic will become overgrowth with tons of features which don’t relate to exact repair process. It looks like we need to separate out new app user, transfer User model there and start to implement this new features there in subpackages like marketing, notification end so on (remember disclaimer #1 from the first post).

As we remember, we have implemented our custom User model, so it is easy to transfer it to newly created user app as we’ve just done with Detail and Liquid. Let’s try to do it, go to commit 3a36793, there are:

• SeparateDatabaseAndState-migration in repair app, to rename table and delete model from old app’s disk state;
• SeparateDatabaseAndState-migration in user app, creating User model in new app;

Commit 3a36793 2021–12–24 | Step #3 FAIL Just try to move User to “user” app (move models to version 2.0)

And also as we used custom User model we need to change AUTH_USER_MODEL settings to new one, user.User.

Apply migrations with python manage.py migrate command.

Commit 3a36793 2021–12–24 | Step #3 FAIL Just try to move User to “user” app (move models to version 2.0)

What did happen? Django says that there is some error. But what is wrong? We’ve just done the same with another custom models and everything worked fine. Let’s answer that question in the next post, for now just apply that User is special model for Django and it manages User another way in comparison with other custom models created by developer.

Let’s remind what we’ve done today:

1. Noticed how certainly models are represented in database;
2. Learned about two project’s state, database and disk, by means of which Django tracks changes in models.py and generates migrations to alter database;
3. Used two SeparateDatabaseAndState- operations to fixed up inconsistency in migration graph and successfully applied changes in database.

If something was unclear in this part, don’t hesitate to ask in comments!

In the next post we will dive even deeper to figure out how Django works with User model, even if it is customized by developers, how it builds migrations graph and why already learned approach of transferring models is not applicable to User.

See you soon!