Looks cool but a minimum of 1 minute polling intervals? That's a pretty long time for what should be streaming data, no? If the data has no need to be real-time then why stream it?
> If the data has no need to be real-time then why stream it?
As I get it, it's basically a way to keep request rates low.
Rather than every client polling your server every minute (which can be quite heavy), only one (Heartbeat) does, and the rest just subscribe to Heartbeat streams.
Totally makes sense if you haven't planned a pubsub-based API but notice that your RESTful one is close to being hammered into a pulp - so you need something to give to your clients, fast. But I don't think it make much sense as a long-term solution.
We would like to enable a lower polling interval, but wanted to see the general response before doing that.
In fact, we would like to enable:
1. A lambda function support to transform APIs (diffing to only append new data, adding meta data, removing unnecessary fields) before they get streamed.
2. Polling frequencies of up to 1s.
3. Custom domains for the resulting streaming endpoints.
Well, I can certainly see the usefulness. I just had a need this past week, but don't mind running my own Node instance. Here's sseries-of-tubes, a Nodejs library I wrote to turn any Express/Connect route (or anything you can poll) into a Server Sent Events stream:
Why don't more web guys use MQTT for pub/sub live data? I saw the PubSubHubBub in the comments already, but most devs are still shoe-horning HTTP into solutions that really aren't request/response based
SSE makes HTTP streaming dead simple. MQTT requires a server/broker and for data to be marshaled/unmarshaled at each end. HTTP streaming can be just text.
Not to mention HTTP identifies the resource by path in the URI, and easily routes right through proxies. TCP is not nearly that accessible.
In general, the API community has consolidated around HTTP for realtime push, mainly because it's what developers know and it's super easy to use (you can just curl a stream for example).
Another thing to know about APIs is that the endpoints tend to be resource-oriented, and resources are often abstractions. For example, when you interact with Twitter's streaming API, no doubt there are some pubsub mechanisms going on behind the scenes, but pubsub concepts such as publishers, subscribers, brokers, topics, messages, store-and-forward, etc are hidden away by the API. You just fetch a resource and get tweets.
My feeling is that the most successful protocols used in APIs are going to be those that expose very little about the inner workings or topology of the server and other client entities.
This; and HTTP has a massive install base, readily available clients on every platform, casual debuggability from tools many people already possess (browser address bar, browser js console, browser addons, curl).
Specific protocols have their merits, but they can only really compete in the backend-to-backend space, and even there they have to compete against HTTP endpoints. In anything that can be construed as vaguely front-end, HTTP has a massive first mover advantage with the ecosystem it brings.
I wonder though, doesn't SSE technically violate a lot of HTTP's assumptions? From the point of view of a HTTP middlebox, a SSE response would basically be a single endless entity, transmitted at a very low bandwidth. I'm no expert at real-world deployment of SSE, but that they sounds to me like it could break a lot of proxies that don't have special handling for SSE. (Or at least cause unnecessary resource consumption for those proxies)
Also I wonder why there are not more websocket-based APIs. WS has the same widespread support and ready-made debug tools as HTTP and seemed to be designed exactly for push or non request/response use cases. So, out of couriosity, why would SSE be preferred over WS?
Streamdata.io (linked in a different post) has blogged [1] about why they chose SSE instead of WS. Additionally, WS drops you down to something very similar to raw TCP, so you have to bring your own application-level message framing, while SSE comes with its own minimalistic format. From a middle box's perspective, SSE isn't that different from streaming video or long polling, although the SSE recommendation [2] does have a line about older proxies:
"Legacy proxy servers are known to, in certain cases, drop HTTP connections after a short timeout. To protect against such proxy servers, authors can include a comment line (one starting with a ':' character) every 15 seconds or so."
As far as I know, two of the few things WS does on top of TCP are in fact: making endpoints URL-addressable - and providing message framing. You can even have text-only frames with a well-defined encoding.
In any way though, thanks for the links. I haven't given them a deep read yet but will definitely do so soon.
I wonder though, doesn't SSE technically violate a lot of HTTP's assumptions? From the point of view of a HTTP middlebox, a SSE response would basically be a single endless entity, transmitted at a very low bandwidth. I'm no expert at real-world deployment of SSE, but that they sounds to me like it could break a lot of proxies that don't have special handling for SSE. (Or at least cause unnecessary resource consumption for those proxies)
Also I wonder why there are not more websocket-based APIs. WS has the same widespread support and ready-made debug tools as HTTP and seemed to be designed exactly for push or non request/response use cases. So, out of couriosity, why would SSE be prepared over WS?
I'm sorry. That was my attempt at "editing" the post after the noprocrast mode kicked in - before an unintelligible post would sit there for ~3 hours...
Looks neat, but I can't really think of a real world application for it other than some debugging of API's. It's basically polling-as-a-service. The hashtag about the real time web is a joke, right?
Please, don't turn it into some kind of SAAS, that would ruin everything. I'd be willing to pay a license fee on a stand-alone app, not a monthly fee on a tool I'd use once every few months.
As other comments point out, there are significant benefits to having a streaming endpoint over a polling logic. It conserves resources, provides a more elegant representation with an events based interface to act on incoming data.
A latent benefit of Heartbeat (which we will start emphasizing soon) is an ability to filter the incoming stream of events by the JSON data properties, so one can build nice IFTTT style code using any REST based APIs as sources. We are leaning towards creating a monthly and a longer-term pricing plan, but if your use-case is casual, then it will always be free. We are able support the current load of requests with a single worker process.
This is very cool, thank you for sharing. I wish I could demo, but my current work is such that I really can't expose my endpoints/data to a 3rd party (financial services/PII type work). That said, pretend I could provided we had appropriate security - tell me about the security involved.
1. Source API endpoint - Heartbeat fetches the data using the authentication supported by the source API (basic auth, security token in headers, URL query strings for example).
2. Heartbeat transformed endpoint - This is accessible over an HTTPS endpoint and is secured with a basic auth username:password key. As long as you use it within a secure environment (server code, or an internal network), there should be no problems. If you use it within a web app that's distributed to other users, they can see the key. However, this key is read-only and we will allow generating a new key from within the dashboard shortly.
Might want to fix the Heartbeat icon link by the way, it currently links to https://heartbeat.appbase.io/heartbeat.appbase.io