Thanks for your comment. I'm really interested in this topic. How do you know the web server is Django? I searched but couldn't find this.
Why would they use Django? I did some small projects in it but I assumed it wasn't very fast and wouldn't be suitable for a big app like this. I would like to know the pros and cons. Why didn't they build up something in C++ or Rust? Won't python limit the speed of responses despite being build the hard stuff in compiled languages? Sorry for being so naive, I am an amateur.
I worked at IG on a previous iteration of Threads, with some of the engineers who wrote the current Threads app. It's Django!
(Heavily modded, run on a custom Python JIT, and using an extremely custom FB-developed database, also used for IG and FB.)
It's Django because IG was originally written in Django back in the day. FB's general approach to scaling is keep the interface generally similar, and slowly replace things behind the scenes as necessary — rather than doing big rewrites. It's seemed to work pretty well.
Ultimately the language used for the web server isn't a huge deal compared to the performance of the database, for these kinds of massive social apps. Plus, well, they do have the custom JIT — but that's very new, and when I first joined IG in 2019, we were running vanilla Python in production.
They deemed getting to the market fast was more important than hardware costs. If their architecture is good, then they were probably really quick and flexible using Python to glue their architecture together which would have been the development bottle neck to getting the service up and running. All the components of the service can be done by separate teams in whatever language they deem most effective.
It is fairly normal to build web servers in a way so they can be scaled horizontally (more instances rather than larger instances). So they can just have more containers run their Django servers and distribute the load between them.
At least a few years ago, most of Instagram’s server side code was in Python. Python may be slower than C++, but it’s not about raw speed it’s about being “fast enough.”
Network latency within an AWS AZ is <1ms and throughput is in the GB/s range.
What percentage of python webapps do you think are hitting this as their latency and throughput limit?
(Assuming effective DB use of course, i.e. not doing dozens of DB roundtrips to server a single result or getting megabytes of data and filtering on the client etc.)
It turns out calling the app "Threads by Instagram" is not just a branding gimmick. The Instagram backend was always Python/Django since before the acquisition.
That wouldn't even make sense. They created hack/hhvm to scale and optimize the platform and codebase. Taking a step backwards and refactoring (hack != php now) would be a horrible idea.
It's mostly foundational work around developer tooling and infrastructure that isn't already covered by other dedicated teams (eg, there is a dedicated Cinder team). My latest work has focused around formatting and linting, and includes open source work on µsort, our import sorter [1], and fixit, our custom linter [2] that makes it easy for teams to build and deploy their own custom lint rules with autofixes.
Some of my team mates work on driving internal adoption of Cinder, rolling out new versions of Python everywhere, improvements to our build and packaging systems, supporting other Python tooling teams, etc. There's a lot of cross-functional project work, and our primary goal is to improve the Python developer experience, both internally and externally wherever we can.
Keep in mind that many/most of these tools also need to work in diff review and CI tools, not just in an editor. That said, we primarily support developers using VS Code (with internal LSP/plugins/integrations) or our internal version of Jupyter notebooks. We also have a non-negligible number of folks that prefer alternative IDEs or editors like pycharm, vim, or emacs. We try to build our tools such that they are accessible by CLI or APIs, so that they can fit into any of these systems as needed.
I'm not familiar enough with Django to say for sure, but I assume at this point it's almost entirely custom ORM and libraries on top of async django routing/response.
I believe the django app would use a python version of the internal ent ORM. You can get a sense for the style of ent/node from this external fork that was written while the author had access to the original ent https://entgo.io/docs/getting-started
How is Django now? I’m a long time python dev (data eng space) but new-ish to web dev. I started a Flask project 2 years ago and found it to be pretty full of footguns, mixed messaging on best practices for scalable apps, and the ecosystem feels overbloated with vapor ware extensions.
Is this just a Flask problem, or does Django have the same issues?
You simply can't beat Django's ORM for general stuff. It's too awesome. This alone makes it so hard to choose anything else.
I know django doesn't have that "shiny factor" to it these days - but it's very reliable.
> mixed messaging on best practices for scalable apps
The WSGI stuff can be kinda confusing and is used across a lot of python frameworks including django and I think flask?
My advice for "simple scaling" is to start with a separate Postgres instance, and then use gunicorn. Use celery immediately if you have any "long lived" tasks such as email. If you containerize your web layer, you'll be able to easily scale with that.
Finally - use redis caching, and most importantly - put Nginx in front! DO NOT serve static content with django!
> the ecosystem feels overbloated with vapor ware extensions.
This still exists to some degree for some more niche stuff, largely because of it's age. Although impressively they'll generally still work or work with minimal modifications. It's popular enough and old enough that most normal things you'd want to do have decent extensions or built in support already.
Not only can you not beat Django's modeling in Python, but I simply cannot find an in-kind solution in JS either. Nothing covers all of the bases the way Django does, or more specifically DRF/graphene-django.
The current state of the art is apparently Prisma, but it covers only a small part of the full picture.
> Use celery immediately if you have any "long lived" tasks such as email
Hey, quick question from a relative newbie who is currently trying to solve this exact problem.
Besides Celery, what are good options for handling long-running requests with Django?
I see 3 options:
- Use Celery or django Q to offload processing to worker nodes (how do you deliver results from the worker node back to the FE client?)
- Use a library called django channels that I think supports all sorts of non-trivial use cases (jobs, websockets, long polling).
- Convert sync Django to use ASGI and async views and run it using uvicorn. This option is super convoluted based on this talk [0], because you have to ensure all middleware supports ASGI, and because the ORM is sync-only, so seems like very easy to shoot yourself in the foot.
The added complication, like I mentioned, is that my long-running requests need to return data back to the client in the browser. Not sure how to make it happen yet -- using a websocket connection, or long polling?
Sorry I am ambushing you randomly in the comments like this, but it sounds like you know Django well so maybe you have some insights.
Use anything except Celery, is my vote. Even if that "anything" is something you roll yourself.
Celery is mature, but has bitten me more than anything else.
For scheduling, there are many libraries, but it's good to keep this separate from Celery IMO.
For background tasks, I think rolling your own solution (using a communication channel and method tailored to your needs) is the way to go. I really do at this point.
It definitive is not using async, I think that will bite you and not be worth the effort.
Django ORM has supported async syntax for some time now, and it can work fully async starting with Django 4.2 (and psycopg3). There are still a few rough edges (such as not being able to access deferred attributes from async contexts) but there are workarounds.
I usually use `asyncio.create_task` from async views for small, non-critical background tasks. Because they run in a thread you will lose them if the service crashes (or Kubernetes decides to restart the pod), but that's fine for some use cases. If you need persistency use Celery or something similar.
Django combined with an async-ready REST framework such as Django Ninja is very powerful these days.
True, but it's the best ORM currently available for any language. It might not be the fastest, but it is the one that's has the highest level of developer comfort.
Using Django is probably the reason why I can stand using SQLAlchemy, it's way to complicated for everything I do and it's just not a nice an experience.
> True, but it's the best ORM currently available for any language.
Such a strong assertion!
I've used multiple ORMs (not Django's), including some of Haskell's type-safe ORMs (e.g. Persistent and Beam). I could not imagine going back. What makes Django's ORM so great?
Depending on what you consider important, and I haven't used Entity since... 2010 I think. For me, those two doesn't compare, the simplicity of the Django ORM makes Entity look dated, complex, like something out of the mainframe era.
Entity is way more flexible, in the sense that you can use in any .Net project really. The Django ORM have no value outside Django, I have yet to anyone use just the ORM, without the rest of the framework.
Django is the opposite. It holds your hand and tells you how to structure your "app", templates, database, and media assets. And django-admin is godsend. It's awesome when you are on the beaten path cause everything just works. But if you step of the beaten path you will have to tweak various obscure settings and create weird hooks to override Django's defaults. Flask is the opposite. Just a very well-written web server and it's up to you to decide things like project structure, template engine, orm, asset manager, etc. Personally I prefer Django since most of my web applications are quite mundane and don't require the flexibility Flask can provide.
I've described that as Django is fantastic for writing Django apps. If the thing you want to do can be completely modeled in the defaults, it's impossible to beat.
If that's only 99% true, you might want to investigate other options.
I can't give Django enough praise. I never thought I'd be writing Python.
I've mostly used Node, Java, and Rust, with about 12yrs experience now. Only about one year with Python and Django recently. I am so much more productive than anything else I've tried. Using anything else feels like a mistake for web apps or CRUD apis. I also use type hints.
I use Django + django-ninja + django-unicorn for dynamic UIs.
Building sidewaysdata.com with django right now! Govscent is in Django too, the ORM is nice to combine with AI stuff already in python.
I'm also following the Ash framework which looks promising.
Django is fantastic. It's much bigger than flask, and has one of the best ORMs I've ever used. Also comes with a built in admin panel which is always nice. You will want to use Django REST Framework to make REST APIs, but if getting a site up and running is all you want, it's perfect.
If you want to purely make APIs though, give FastAPI a try.
I use django professionally and for personal projects. I started with Flask and then did FastAPI for a while, but I like django the most since it's the most mature.
The ORM and django admin are killer features out of the box that the other frameworks don't have. I will say though that FastAPI is really nice, especially if you need async support. However, I have found that using django ninja [1] adds a lot of nice to haves that FastAPI has to django that makes it much more fun to use again.
+1 for django ninja. I built a little side project with it a few months back and was super impressed by how easy it was to get up and running. I didn’t want to leave the Django ORM behind so was very pleased to find this project.
Having used Node/express, Rails, Flask, and Django extensively, I have been very impressed with Django. The framework itself is good, but the superpowers come from the ORM and DRF, and a few other great plugins. The stuff you can do with the ORM and DRF is nothing short of incredible. Now, you could argue that the framework shouldn't be tightly coupled to the ORM, and sqlalchemy does provide some stiff competition as an example of that, but in my experience Django ORM edges out sqlalchemy in terms of usability.
There are some footguns around N+1 queries, but they are general to all database interfaces and pretty easy to avoid IMO.
I've used Flask/SQLAlchemy and Django a bunch in my day, professionally and on the side.
It's a different strokes for different folks sort of deal.
Django is very opinionated, in a way that is "eventually" correct (i.e. they might have had some bad opinions many years ago, but they have generally drifted in a better direction). If the opinions don't align with what you're building, the escape hatches are not generally well-documented or without penalty.
Flask is minimalist and flexible. Once you find a "groove" to building with it that fits your sensibilities, it's quite, quite nice. That being said, the most recent versions of flask have excellent documentation, imo, and the tutorial is a bit more opinionated in a highly productive way. The "patterns" section of the docs is also super useful for productionizing your app.
Personally, I prefer the combo of Flask+SQLAlchemy, and eventually Alembic once you decide that's good for your app's maturity level. I respect Django a lot, I just enjoy the explicitly "less magical" aspects of a Flask stack, which is an opinion-based trade-off imo.
As a minor rebuttal, I get sick of Flask because it is bring-your-own everything. Nothing I write will ever NeedToScale(TM) or be off the beaten path from the Django Way. No interest in plugging in an alternative template language, form validation, email library etc. When I get stuck in Django, I know someone else has experienced the exact same situation. With Flask, I have to pray to deity that Miguel Grinberg has written about a similar enough situation. The majority of documentation foregoes Flask Blueprints.
I'm a self-taught noob as far as web engineering goes, but between this course (cs50.harvard.edu/web/2020/) and ChatGPT, I got several commercial Django sites live in a very short amount of time and effort.
Instagram devs have given a lot talks about python and django at conferences. As far as I know IG is still running on django so it makes sense that the dev team would stick with it for something new.
Correct me if I'm wrong, but Django is not a web server. It's a framework used in conjunction with an app server e.g. gunicorn and a web server e.g. nginx
This would partially explain why it feels so slow. Some page refreshes are taking up to a couple of seconds.
It's probably hurried along python code and fairly simple horizontal scaling. Some population of individual instances are probably pegged, and random requests are frequently landing on their backlogged request queues.
This seems to have been rushed out the door to capture a unique market opportunity, and they'll clean up the engineering once they get engagement.
