How Heap's 5-line MVP evolved into a 30,000 TPS beast

[mallyvai]

http://codeinsider.us/i/2.html

[shostack]

It looks like Heap is making some progress with their UI which I'm glad to see.

That said, I'd love to see Heap pull in the rest of the web analytics picture. Event data is interesting, and they seem to have a decent solution for the issue of setting up a massive amount of event tracking (and losing all the data you haven't set events for yet). However event data in a silo without the context of things like referral URLs, search query data, campaign metrics, etc. are critical for gaining the full picture.

Perhaps I didn't tinker with the demo enough, but I didn't see those pieces of the puzzle present. For any online marketer serious about analytics, you need the complete picture under one roof.

That said, this is a great step forward in a space where this pain point around setting up event tracking has been largely unsolved for some time. I wouldn't be sad at all to see them acquired by someone like MixPanel, KissMetrics, or Google.

[matm → shostack]

We automatically annotate each event with referrer, search keyword, and all UTM properties. Full list of properties here: https://heapanalytics.com/docs#events

Would love to know if there's other important context we're missing.

[shostack → matm]

Awesome, clearly a miss on my part. Would love to see more interface elements designed around exploring the web analytics piece of it though. Google Analytics does a fantastic job of making this data intuitive (well, for someone knowledgeable of the space) and explorable. Things like cross-channel tracking and other items are also critical.

I'm playing around with the demo in our test environment right now and really love your Event Visualizer tool. I need to play a bit more, but I'm hoping that your ability to check/uncheck the components that define the event will let us work around issues of dynamic selectors generated by YUI which was our team's concern for why Heap might not work with our setup.

EDIT:

Looks like YUI is still a major issue. If you're curious, you can look up my HN username in your email list to find my account and see what I'm talking about.

[matm → shostack]

Thanks for the heads-up. As mentioned over email: if the Event Visualizer isn't providing quite the right precision, keep in mind that you can manually specify hierarchical CSS selectors within the dashboard itself. It'll work as expected.

[pdog]

> Are you still on MongoDB?

> Oh god no.

Pretty damning condemnation of MongoDB, no?

[matm → pdog]

For our use case, it just wasn't the right fit. In particular: 1) we needed more features/performance than Mongo's aggregation framework provided, 2) the mongod global lock was prohibitive for our write throughout (turning off fsync wasn't suitable), 3) schema migrations were difficult/impossible without transactional guarantees.

It seems like Mongo's addressed some of these concerns since we started building Heap.

[raverbashing → pdog]

Humm... No

They also outgrew PostgreSQL, and gave up on Redshift, is that a condemnation of them as well?

[aaronblohowiak → raverbashing]

Looks like they are still using PostgreSQL, just not a single instance. It looks like they are using Citus Data to handling sharding and merging responses.

[chris_wot → raverbashing]

He said that PostgreSQL scaled pretty well, but he got rid of MongoDB in one week. Seems pretty damning to me.

[copergi → raverbashing]

Odd that they think they outgrew postgresql, while only dealing with a tiny 30k tps.

[integraton]

So is the "massive custom data store" using Postgres or is it something totally different? Dan's answer sounds like Postgres + Citus DB is the current setup.

[mallyvai → integraton]

It's in the interview, like halfway down. They're using Citus over Postgres.

[mstump]

30k TPS is pretty damn low. That's less than the throughput of a single Cassandra node...

[nrbafna]

I am wondering what the 5 lines for MVP were. Anyone mind taking a shot?

[matm → nrbafna]

This is referring to the integration code itself. It was something like:

  var DOMEvents = ['click', 'submit', 'change'];
  for (var i = 0; i < DOMEvents.length; i++) {
    window.addEventListener(DOMEvents[i], function(e){
      $.post('/collector', {type: e.type, target_id: e.target.id, target_class: e.target.className});
    });
  }

[onion2k → matm]

That's interesting and immediately obvious how it could both be useful and valuable in the right domain. Logging what people are doing on modern event-driven websites definitely is a problem worth solving. My let's-made-a-complicated-version-first mindset would have extended jQuery's .bind or .on prototype in case the site is using event.stopPropagation(), and that's why I never finish anything. Thanks for sharing.

[shamsulbuddy → matm]

This is rocksolid basic script..one quick question..if you post on every event then dont you think so that it will generate huge traffic..how did you handle that