Migrate your Django monolith to microservices. The 1st step — preparation.
Hi there!
Guess almost every developer had thought about dividing Django monolith on several smaller services, when you refactor, move code between applications in your project. Usually this thoughts end on Django’s models management, there is plenty of problems:
- how to move models without breaking migrations graph;
- how to hold down your data in database;
- how to deal with FK- and M2M- links to transferred models;
- and so on and so fourth.
In that posts series I’d like to share approach how to deal with models while you reformat Django applications in your project to be able easily move to microservices in the future.
How this post series is organized
As it is quite huge topic with a lot of detailed information and codeblocks I separated it into several posts:
- Migrate your Django monolith to microservices. The 1st step — preparation.
- Migrate your Django monolith to microservices. The 2nd step — understanding Django model and migration management.
- Migrate your Django monolith to microservices. The 3rd step — migrating custom User model.
Also I prepared an example project where you can play with migrations changing commits. Each important point in the article will be linked with certain commit for easiness. You can go through posts commit-by-commit trying commands given in the articles.
If you find a mistake or misunderstand something, don’t hesitate to write comments, I am glad to hear remarks and help other Django developers!
Preconditions
What we have at the beginning:
1. Django project with several applications;
2. Final state’s picture of our models, which are located in different apps in that project;
3. Knowledge that there may be a lot of troubles with models transfer.
What we need to do:
1. Find a way how to move models from an initial state to the final;
2. Go through that way step by step, realizing each step in details.
Django project “Car repair center”
Let’s imagine that our business domain is car repair center. You have uncle who is owner of such a center and for now center’s routine is fully manual — customers call administrators, they sign up for repairs, writing it on papers. This papers may be lost, administrators make a mistakes, signing up different customers to one technician — plenty of dissatisfied customers and damage for business and reputation.
Your uncle knows that you are quite good developer and he asked for your help — automise his business as much as possible to prevent tons of manual work.
Initial state of project
You as a good developer desided to draw upper level architecture. Let’s think which applications and models do we need.
Disclaimer #1. This business domain is just theoretical, there is not 100% confidence that applications would be designed the same way in real world. My goal is to demonstrate how to move models between Django applications technically. If you faced a question “how to separate my monolith on apps/services/modules well?”, I recommend to google concepts such as “Bounded contexts” and “Event storming”.
So, let’s imagine we have already done event storming, asked domain experts and found bounded contexts. Our upper level architecture may look like this:
We decided to put all models in one app, repair
, because each one is connected with repairment, there is no need to put them in separate apps. Let’s explain what data each model is responsible for:
Car
— details about customer’s car, brand, model, when it was manufactured, engine type, gear shift box details, tires size, etc;Repair
— certain repairment, date, which repairman will do it, what spare parts will be changed, which operations will be applied;Repairman
— employee who will repair this certain car in that certain day;User
— personal information about repairman, payment credentials to pay salary, contact info allowing administrators connect employee, etc. And also this model is used to store personal information of car’s owners;Detail
— spare part which will be repaired or replaced;Operation
— action which will be done with car (fluid replacement, tire replacement, …).
For now we can start app on commit d380469
and fill database with some instances. Let’s do that:
- Clone repo with demo app.
- Go to commit
d380469
. - Run migrations.
- Apply fixture
initial_data.json
to fill database with data. - Run server and open http://0.0.0.0:8000/admin, with creds
admin/admin
.
You should see admin system with pre-filled users and repairman, cars, several repairs with details and operations.
You presented system’s draft to your uncle, and he was quite satisfied and approved you further work. But also he mentioned that repair includes changing of different liquids and system should track it separately from spare parts and operations. You decided that liquid is connected with repair too, so let’s just add it to existed app (go to commit 103e2f1
and apply migrations):
You started to implement automation of administrator’s manual job, to notify repairman about repairs and so on. Project became cluttered with tons of business logic.
You deployed MVP to server in repair center and uncles business started seguing from manual to automated. Customers are happy, they get notifications from your system about future repairs, so they don’t forget; time to book repair decreased a lot; administrators are happy because customers are not angry because of waiting and administator’s fails in scheduling; your uncle is happy because his business gains traction.
Next improvements
You project is running successfully. And some day uncle went to you with new request — as administrators manage ordering spare parts and liquids from wholesalers, it would be nice to start working not only as repairment center but also as shop. In that case administrators responsibility will be to respond on customer’s calls and sale booked orders, so another feature should be possibility for repairmen to order liquids and spare parts himself by means of your app without asking administrator.
You have found out details about how store business should be organized and knocked out new schema of project:
As you heard from your uncle explanation order may be connected to some repair process or no — so you’ve decided to create new Django app called store
for Order
model (go to commit 385e59
and apply migrations). As order contains some spare part or liquid you want to shift this two models to newly created app (remember disclaimer #1, this is just theoreticall example).
Roadmap of model’s shift
Before we start write some code transitioning Detail
and Liquid
model to new app let’s discuss which steps should you follow doing such a huge and potentially ruinuos refactoring.
Usually such refactoring takes place in team development when you are not the only person developing app — there are QAs, product owners, other developers, it would be great to simplify their life while you are commiting such a dangerous bursting code. To do so you need to prepare:
- Create resulting scheme “How models and apps should be related”.
This is what we have already have in car repair center example — 2 apps,store
andrepair
, whithOrder
,Liquid
,Detail
models placed instore
and other models in therepair
app. - Fix check list for developers — what must be done when you move one model to another Django app. It should contain such steps:
1) Find all places where model is used (imports, apps.get_model(), fixtures, raw SQL, etc).
It will help you and your colleagues to take in mind all use cases where you should change model’s app. Mind out that already created migrations shouldn’t be touched, because they have already applied in production database, and you musn’t change it otherwise your local and testing database will differ from production.
2) Find permissions which relates to model in the previous state of project (CRUD Django permissions, custom permissions inModel.Meta
) and rename them.
Django automatically creates permissions in database with templateapp_name.ModelName
, as model will be moved to another app we should fix permission’s name.
3) Add migrations which will transfer model from old place to newly created app.
There will be 2 migrations, each one contains instance ofSeparateDatabaseAndState
operation. We will explore migrations from example further.
4) Freeze migrations dependency — the second (in new app) must depends on the first (in old app), otherwise Django may not construct migrations dependency graph right way.
5) Recheck steps 1–4 if you merge request is waiting merging for a long time.
If another developers changed models while you were working on moving them, your code may break whole project. So when you move models your changes must be applied to main codebase as soon as possible (after high-quality testing, of course). - Whire recommendations for QA how to test your changes.
Task “move models between apps” may be unclear for QAs, clarify what they need to check after your changes, for example:
1) Apply all migrations on production database dump (if possible) and empty database.
2) Prepare template database (createdb -T
flag in PostgreSQL) before switching to commits with models shift — it helps QAs switch branches and restore database faster if your code breaks their database.
3) Prepare template for QA’s task, for example: “ModelA
was moved to appnew_app
. Recheck CRUD operations withA
and permissions for actions withA
, accessibility for different users. Check total amount ofA
in database after shift, new table name isnew_app_a
instead ofold_app_a
“. - Create tasks in your project management system (Jira, etc), which models in which task will be shifted between apps and put them into timeline.
Each task should include explanation for QAs, how to test changes and recommendations from the previous point. Such roadmap will help you and your teammates to see whole picture of future changes in your app. Ideally it whould be great if you scheduled all model’s transfers to one period of time (one sprint, one release etc). - Prepare instructions to devops or release engineers, how to deal with that changes during deploy. It’s really important to make a database dump before deploying and be able to rollback it if something goes wrong! To revert release easily, it is good idea to deploy model’s shift tasks in separate release without any other features and bug fixes.
- Share all this plans and give an outline of future refactoring to your teammates, including QAs, developers, product owners and managers, devops. Everyone in your team should know that such a potentially ruinuos work is planned and how they whould deal with it.
- Carry out your planned task!:)
So, you’ve done all that preparation and almost ready to start doing planned tasks. But let’s take some respite for now.
Today in this post we’ve:
- dipped into car repairment center business;
- got acquainted with codebase, which automise this business and prepared MVP;
- started up initial project’s state;
- realized that alive business tends to changes and it’s regular that your codebase evolving, following that changes;
- observed steps which you should go through before write the first line of code to simplify live of your teammates with such a potentially ruinuous refactoring as model’s shift.
In the next post I’m going to compartmentalize more hardcore themes and dip inside Django’s model’s and migrations system:
- How does Django manage model’s changes?
- How does Django build migrations and apply them?
- What does it mean from Django’s point of view “To transfer model to another app”?
- Are there any instruments in Django to shift model to another app without loosing data in it’s table?
In the next post tons of code is waiting for you!;) You may try to transfer Liquid
and Detail
models yourself and check your tries with mine. See you soon on my page!:)