Scaling Disqus (YC S07)

[riffer]

http://www.slideshare.net/zeeg/djangocon-2010-scaling-disqus

[endlessvoid94]

This is invaluable, thanks for sharing. We're running into some really interesting problems building Djangy.com - this slideshow is a goldmine.

[kfool]

How do you deal with schema changes? Do they force you to switch into a read-only mode?

[kingkilr]

The talk was recorded, and the video will be made available (once it's ready).

[xtacy]

For such web services, what is the average/peak load on the network? Is it a bottleneck at all? Are the networks over provisioned?

[riffer → xtacy]

The article says peak is 17k req/sec ... (I'm impressed)

[xtacy → riffer]

Yes, I presume that's incoming 17k req/sec. Each incoming request can then lead to many database queries/memcached queries/etc. It would be more insightful if request/response sizes were also known.

[michaelfairley]

There was no mention of MongoDB in this presentation, but Disqus is listed on the MongoDB Production Deployments page: http://www.mongodb.org/display/DOCS/Production+Deployments

[okeumeni → michaelfairley]

I guess it’s nice to mention Disqus a using for MongoDb folks. They are not the first company to mislead on the internet.

[danielha → okeumeni]

Disqus also uses MongoDB amongst many other tools not mentioned. Those fine folks aren't misleading anyone.

[patrickaljord]

I thought they were using mongodb, any idea why they switched to pgsql?

[bdr → patrickaljord]

Disqus has used Postgres since the beginning. They also use mongodb.

[orlandu63]

Can we get a direct link to the PDF?

[ronnier → orlandu63]

This pulls out the notes: http://viewtext.org/?url=http://www.slideshare.net/zeeg/djan...

[sr3d]

How many HAProxy server is being used? Just a main one with heartbeat for failover?

[siculars]

nice starcraft2 pitch at the end ;)

[technoweenie → siculars]

I see someone playing Starcraft most of the time while I pass by their office (ours is right above theirs).

[ohashi → technoweenie]

Strategical Thinking Practice.

[amccloud → technoweenie]

Probably dcramer

[thedz → technoweenie]

It's true :/

[_3ex7]

Why is emailing an exception bad (Slide 48)? Can a small site get away with it ?

[tkaemming → _3ex7]

A possible disaster scenario: connection issues with the SMTP server are causing the initial exception, resulting in webserver connections that are open for excessively long periods of time (unless you're sending the emails asynchronously), since the process is blocked first on the failed SMTP connection, then again on the exception email attempt (and you're still not getting any notification that something is broken, unless you're doing some other kind of monitoring!)

[simonw → _3ex7]

It's fine for small sites. You'll know when your site is too big for that feature from the pain in your inbox.