Django: The Good, the Bad and the Ugly

04 Dec 2023 | #Django #Python #web #development #opinion

I’ve been working on the largest Django project of my career - which is Shepherd.com - in the last couple of years and have been planning to share my first-hand real-life experience in building non-trivial Django applications for a while. I finally took the plunge to write this up, below follows what I liked and did not like about Django.

The Django web framework can be used for a wide range of products of various complexity, and your mileage may vary on how good fit Django will be. My experience is focused on building a web application with a little bit of JavaScript sprinkled on top. That means not building single-page application (SPA), neither a native mobile app (I do build and optimize for mobile web though), nor a REST API. The applications I have built are monoliths (as opposed to microservices) backed by an SQL database and most of the content is managed using Django’s admin.

If you are building something drastically different from what I described above, my insights and advice here might have limited applicability. Some aspects are heavily opinionated, feel free to disagree.

To give the readers some hint on the scale of the codebase, which I consider to be a medium-sized project (for a small team or a solo developer), here are some code size stats:

The Good

I have to put forward that I generally like working with Django a lot. What did I particularly like about it? See below.

Python. I absolutely love working with Python. I am proficient in a handful of other programming languages like Java, Scala, Ruby, JavaScript and TypeScript, but for me Python comes close to the top when it comes to speed (both runtime and development cycle), developer friendliness and features (language and standard library).

The ORM. Object-relational mapping, also known as Models in Django. The ORM has a concise and very powerful way of expressing database structure and has powerful data modification and querying methods.

Migrations. Django comes with built-in database migrations, which is a key feature for an ORM. The generated migration files are expressive and easy to read, and do the job without any human intervention most of the time. It is extremely rare that I need to write a migration manually, and when that happens, it’s still an easy job.

Running and managing migrations are also very easy and straightforward using the built-in command line tools.

Management commands. Not that writing command line utilities is rocket science, but Django makes this even easier. I use these extensively for maintenance tasks, debugging and reporting and scheduled jobs.

Stability. Django is 18 years old, and that means stable. It no longer makes crazy changes in architectural direction (perhaps it never did), and it is not being rewritten to a different language. I like that! The release cycle is predictable, version upgrades are smooth. In the recent years, I’ve gone through one major version upgrade and many minor ones, all with reasonable effort.

Upgrading Django itself (excluding third party packages) usually took an hour or less even during a major version upgrade. Surprises were due mostly by not reading release notes carefully enough, or due to some deep customizations to the admin (most of the admin HTML, CSS and JavaScript structure seems to be treated as “internal API” but it is sometimes unavoidable to rely on these).

Contributing. I only have a limited experience with contributing to Django, but that is all very positive. The few issues I ever filed were responded to quickly, and the one changeset I contributed was merged into the mainline within a reasonable timeline. All in all, it seems like a friendly and active community.

The admin. Django comes with an (optional) admin user interface that allows quick CRUD operations on database records with almost no code. The admin is highly customizable, and I have successfully used it to create sophisticated content management systems (CMS) at a reasonable implementation cost and complexity.

Code structure. The framework promotes a simple, flat directory structure called apps or applications within a single project (a “project” is a website, combined from one or more “apps”). The approach may seem confusing at the beginning but it allows “clean code” structure, a folder hierarchy that follows the structure of the problem domain, not implementation details.

Django Debug Toolbar. Not part of Django, it is a third party package. An essential tool in the form of a web GUI that allows inspecting pretty much everything that happened during the rendering of a page. I mainly use it for optimizing SQL queries.

The Bad

There are a few issues that frequently and actively annoy me. I believe that solving these would lead to a significantly better experience for all Django developers.

Python’s dependency management. The horror! The horror! It is not Django to blame, but Python has a number of alternative dependency management tools or disciplines, and I found all of them to be horrible in some way. I settled with Poetry, which is somewhat heavyweight and breaks every once in a while, but does the job most of the time, seems to be the least horrible of all choices. The state of Python dependency management makes Maven or npm look amazing…

Django templates. I have always used Django templates (DTL, the framework’s own template language) for generating HTML, because the default should be fine for everyone, right? It is not the most amazing template language, but does the job just fine, the official documentation recommends sticking with it. (“If you don’t have a pressing reason to choose another backend, you should use the DTL.”)

Unfortunately, the template engine is very slow. Rendering a non-trivial, but also not insanely large or complex HTML page can take tens, sometimes hundreds of milliseconds on Standard-1X Heroku dynos.

Django’s performance optimization docs do actually mention that “heavily fragmented templates” might be slow. Given that many if not most web applications will want to use reusable HTML fragments, using {% include %}, which is documented to be slow, Django templates are not a great choice, except for trivial stuff.

Unfortunately, I realized this too late. Django does ship with an alternate template engine, Jinja2, and migrating an existing code base to a different templating engine is a risky and expensive business.

Lastly, I do not often miss tools from the JavaScript world, but I do miss Prettier, the opinionated code formatter for formatting templates. I use Black for formatting Python and DjHTML does an OK job indenting templates, but it does not come close.

Refactoring models, apps. Clean code means continuous refactoring as the application grows, which sometimes involves splitting Django apps, or moving functionality between them. Although the migration framework supports basic refactoring like renaming fields or changing types, moving models from one app to another is a complicated task, especially when model inheritance is involved.

For example I seem to be forever stuck with the poorly named List model on one project due to some weird dependency loops in the migration history or the inheritance hierarchy. I never managed to figure out why.

The Ugly

I feel a bit “meh” about a few things in Django, that are not that terrible, not deal breakers for most people; quite likely many developers will not even notice or identify them as problems. Still, my life, and maybe the life of a few others would be slightly better if these symptoms did not exist.

Lazy queries. Model property access can and often does result in on-demand database queries also known as the N+1 query problem. This is a Django feature only beneficial for absolute beginners and toy projects. Anything and anyone beyond that do not benefit from feature, and it makes certain page rendering performance issues hard.

As a solution the ORM has tools for eager loading (select_related and prefetch_related) but there is no way to force the usage of them and opt out from the default lazy behavior (although third party solutions exist).

One thing worth noting is that when using async views, lazy fetch on property access fails, so that partially solves disabling lazy fetch, but asynchronous Django has it’s own problems (see below).

Asynchronous support. Django has had asynchronous support since version 3.1. The catch is that it is still work-in-progress, therefore some parts of the framework do have asynchronous support, some do not.

That means one will end up sprinkling code with sync_to_async helpers, which quite quickly gets very ugly.

There is however a bigger catch with async Django, and that is the ORM. Even though the ORM has “asynchronous support” meaning it has async query methods like await Book.object.aget(), the underlying implementation is not truly asynchronous and does not allow concurrent database operations at all. This non-feature is not even documented clearly, in fact I had to learn it the hard way.

To add to the offense, when using an ASGI (asynchronous) server, there is no connection pooling available, which on my projects turned out to be a performance killer and therefore a deal breaker.

The lack of truly asynchronous ORM means it is hard to optimize views that require dozens or queries, no matter whether using sync or async, they will be executed one after another.

Third party Django packages. Django is a popular framework and there are third party packages for everything one can imagine. But most of these are not of “industry grade” quality. Which is OK, this is all free and open source, but one has to be prepared for eventually forking most third party packages in-use, and maintain the fork perpetually.

And even the packages that get semi-regular maintenance often break on new Django versions, sometimes taking weeks to catch up (even with community contributions).

This is of course not specific to Django, all open source communities suffer from lack of resources or interest for maintenance. I do my share of making things better by contributing fixes and filing issues, I wish everyone was doing that.

Notable exception are the packages maintained by the Jazzband community.

Limited alternatives other than ORM or raw SQL. Django’s ORM is very capable and one is able to express rather sophisticated SQL queries using Python syntax. But there are limits to this, and no ORM will likely ever cover all SQL features (standard or vendor specific), and that is fine…

But a few times I hit a wall with a more complex ORM query, which by then is rather complicated to abandon, and sometimes queries are reused elsewhere, and switching to raw SQL means no easy reusing of SQL fragments (except for manipulating strings that contain bits of SQL queries).

Ideally, there should be some low level query builder interface that allows expressing any SQL grammar using Python syntax, but also understand that it’s not something trivial to add to the ORM.

Almost everything is mutable. For those with background in functional programming and immutable data structures, Django might seem like a horrible place. In Python, as well as in Django, almost everything is mutable, object oriented programming and class based inheritance is widely used and relied on.

Not a big deal for me, but I sometimes wish somethink like Python+Django existed that is strictly immutable. I’ve probably spent too many years programming Scala…

Lack of service layer. In most non-trivial web applications one eventually want to contain “business logic” somewhere, ideally following some established patterns.

The missing piece in Django is something between models and views (n.b. what Django calls “views” are called “controllers” elsewhere).

Without further structural decisions, business logic in Django might end up in views, models, managers and querysets. Most often views end up being rather “fat”, containing a lot of business logic.

The perfect architecture can be debated, and is inndeed debated a lot, but I personally believe in “skinny” views and “skinny” models. That means implementing most of the business logic in a layer called “services”. These services are usually plain Python modules named “something_service” containing functions that execute “transaction scripts”.

This is especially useful, or even inevitable for transactions involving multiple models and/or external services (e.g. REST API calls), shared between multiple views.

When the admin app is involved, the service approach becomes messy though. There is no simple way to force the admin to always use whatever business layer we dreamed up, for good reasons (admin is truly just a CRUD GUI on top of database records).

What I ended up doing is overriding the admin’s save_model and save_related methods and call service methods from there, where it was necessary.

No zero-downtime migrations. Not that many ORM-migration tools have that included, but it is worth noting that changes to the database schema can block production applications from accessing the database for the entire duration of the change, which, for large tables, can cause a significant amount of downtime.

Outside of Django are ways around this, for example pt-online-schema-change for MySQL, but it would be nice to have something like this integrated into Django. I think it would fit Django’s role of a “heavyweight ORM”.

No European Django Foundation entity. If you use Django for business, a good way of supporting the project is with donations. As the Django Foundation is registered in the USA, making these donations tax deductible can be hard or impossible in Europe (at least in Germany, where I am tax resident).

Summary

All in all, despite the gripes above, I am very happy with Django and will keep using it on real-life projects. It is rock solid and truly lives up to it’s slogan, which is “The web framework for perfectionists with deadlines”.

I hope some day soon a truly asynchronous ORM becomes a reality. And my next project will use Jinja templates.