Twitter already betrayed their entire dev base all those years ago when they effectively killed off all 3rd party clients by setting a lifetime cap on oauth tokens per developer just so they could force users to use their clients which inject ppa/ppc ads into the stream (and then didn’t even bother keeping those clients up to date, pissing both the developers and their users off in one go). Stupidest move ever on their end. All they’d have had to do is mandate the inclusion of promoted tweets in the firehouse and they’d have kept developers loyal to their platform while raking in the cash.
Also, if I’m not mistaken, retweeting and hashtags were done first by third party clients. Even the word tweet was coined in an email Blaine Cook sent discussing twitter’s auth API for 3rd party clients.
Previously (sometime last year, IIRC) Jack Dorsey asked developers for a second chance. I don't see how this lines up with that, is it that now we can pay them to access artificially capped functionality? This was never the problem for me, and in the meantime, nearly all of the devs seem to have [wisely] moved on.
I'm not interested in participating in another one-sided relationship with a company who behaves as Twitter has -
changing the functionality and rules at the drop of a hat.
Except that they didn't. There wasn't even any discussion, it was just "We're capping OAuth tokens.". Developers offered to pay for access, users offered to pay for the ability to use 3rd party apps.
If they didn't want to go down that path then Twitter could've easily required developers to show monetised tweets/ads.
Just because Twitter couldn't figure out how to monetize their APIs? Please. They could have made some premium before. They could have forced ads in the tweet stream as part of the usage requirements for the APIs etc.
They killed a community due to arrogance. And it backfired big.
It was early on to come up with any conclusions but dropping developers in the XXI century? Do you know any other company who did that?
The problem with Twitter was more serious: they never gave a concise and frictionless way to pay for the API. For example, Google innovation with AdWords was providing a self and quick service to put ads while other providers like Yahoo required a process with a lot of friction. Twitter logic was prehistoric.
I have been on tweetbot for as long as I can remember and can't stand the official apps, how does this work? Are some 3rd party clients blessed by Twitter?
They basically issued each developer n tokens and the first n people to purchase the app were let through, but no more thereafter.
What genius thought “let’s punish developers in our ecosystem by putting a cap on their lifetime earnings” should be taken out back and summarily executed.
The amount of people who can authenticate (as in, connect to the app once by linking their account) is limited to a certain amount. Developers are / were given a fixed amount of user tokens.
Fixed per app, or fixed per period of time? I.e. does it mean that when I use Tweetbot I'm either a) part of a lucky group of people that authenticated before they "ran out" of tokens, or b) that every time I log in I risk being shut out because too many people authenticated in the last 24h towards their tokens?
I keep recommending everyone to abandon the official apps and use e.g. Tweetbot because the official apps suck compared to 3rd party (commercial messages alone is a dealbreaker) - but do peolpe then take my advice and go download tweetbot only to find they cant auth? That would be bad...
Twitter then limited this API endpoint to data from the past seven days only, and now just introduces a paid version for only last 30 days data.
I find it fair to charge for elevated API access. On the other hand, the 30 days limit for the paid version is pretty ridiculous. Makes it more worth to just scrape the free online advanced search...
That's categorically not the case - the index for the search API has always been around 7 days since Twitter acquired Summize (which is where the search API came from). There was never an artificial limitation such as the one you're describing. From the page you're linking to from the archive:
"The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets."
Scraping the Twitter website is against the terms of service and may result in permanent suspension of access.
> I wish these premium APIs were available during our first few years. As we grew, we quickly ran into data limitations that prevented expansion. Ultimately, we raised a round of funding in order to accelerate our growth with the enterprise APIs. With the premium APIs, we could have bootstrapped our business longer and scaled more efficiently.
Quoting a company that is essentially saying they had to seek funding because of twitter limiting API access is an odd choice for this announcement.
I wish it were true pay-as-you-go rather than starting you in a sandbox and then jumping immediately to $150. Working on a small side project, $150/mo is a lot of money in the beginning, especially if I'm doing a relatively small number of requests. I guess I need to see what the sandbox entails, but if it's not real live twitter data, this still feels like a big step function in cost from free to premium. Just charge me per-API call or on the amount of data transferred with no minimum.
I don't know that the testimonial in the post makes me feel better about the product to be honest "they couldn't provide us what we needed so we had to give away part of our company to pay through the nose for the privilege for that feature, so it's great they've released this now, years after it'd have been useful. How's that feature you're waiting on working out for ya?".
I know it's the current thing to bash on Twitter, and I'm sure there are decent use cases out there, and glad they're finding extra revenue streams outside of ads, but the speed and level of iteration seems rather low considering the resources they have.
Of course, I'm just a username on a website and have never scaled anything to 300m users, but like a lot of people, think Twitter could still be great... but seemingly still has a lot of miles to go. And highlighting that in an announcement post seems... strange
I often see these kind of comments and find them extremely myopic.
Sure you can create a twitter clone all by yourself, but good luck scaling it to ~150k tweets per second writes[1]. All the while ensuring they're fanned out to all the respective followers some of whose tweets may need to be fanned out to tens of millions of users in a matter of seconds.
You'll need 30 engineers just to build and maintain the datastores, caching mechanisms, and service deployment infrastructure. You then have the actual services, real time streaming platforms, streaming applications, batch infra (hadoop), analytics pipelines, notifications, security, anti-spam, recommendation/machine learning pipelines, ads infrastructure etc to name a few.
Can't find it now, but there should be a circa 2011 architecture of twitter in one of the blogs somewhere. I recommend you have a look at it.
If you're comparing it with WhatsApp, note that WhatsApp messages are mostly 1->1 and it's easier to shard the servers by recipient. Also, not having to bother persisting all the messages forever simplifies the system by a lot.
I think you're making a few assertions that would be more honestly served if you conceded as assumptions.
Twitter validated it's own employee count with an absurd overhead.
It has concerns of scale, performance and user experience. So does GitHub, Shopify, Pandora, Spotify, Snapchat, LinkedIn, MailChimp, WordPress.com, etc, and yet, manage will much less resources and overhead. Oh yea, and fucking Netflix - which - as far as I can tell - has much more and difficult problems to solve at a much faster pace. They employ 4,700.
As of this comment, Twitter employs 3,583 people.
What they needed to do was hire 100 talented people, allow third-party clients, try and occasionally acknowledge the developers who implement their platform, and stopped with the asinine "we're trying so hard guys" blog posts. We get it. You're having a hard time and you want everyone off of your grass. That's fine, but don't come asking for sugar at my house 'cause you ain't Beyonce.
Netflix has less "hard problems" in terms of operations than Twitter.
With Twitter everyone's data is interconnected. With Netflix, you could theoretically have each server just be completed isolated from each other (modulo account management).
Netflix has operational problems to solve in terms of bandwidth, though. There's definitely major difficulties there, but they're of a different variety to the high coupling Twitter deals with
I'll say what I've said in another thread on this matter, which is, no one outside of this bubble buys this shit. There are global supply-chain networks that move millions of products, and they don't employ as much as a micro-blogging platform. It's hubris to think we're so special.
100% agreed that running a network on the scale of Twitter, with all the strong interconnections, is extremely hard. Probably the hardest, moreso than even things like Facebook.
I do wonder what the thousands of engineers are doing. I imagine there's a lot of fire-fighting, but is it only that?
It could be the case that twitter is constantly falling over, still.
That's the thing, any time we raise eyebrows at the size of Twitter's dev team we're met with "well YOU try to operate a business at the scale of Twitter then!". I am, like you, genuinely curious what kind of work these people are doing all day. If they're prototyping new products, optimising poorly performing DB queries and API calls, building out frameworks (like Bootstrap), maintaining some internal admin UI etc.
I've read bits and pieces, heard rumours and it'd just be nice to see what's going on. Unless I see something like this I'm just gonna think "BOY that's a lot of developers" whenever I remember Twitter's headcount ...
I got an email from a recruiter last year asking if I wanted to come build the future of live video at Twitter. I think it's plausible many of them are working on new products that end up abandoned due to constant direction changes.
They definitely attribute a lot of their early success to this choice. It seems reasonable to me that choosing a tool developed to create modern real time communications software is a pretty good idea ;)
Of these, only GitHub, Mailchimp and Auttomatic have less than 1000 employees. All have more than 500.
(That doesn't mean that Twitter isn't too big or grew too early, but it does mean that none of those nowadays qualify as "see how you can run a big product with 30 people")
You're making a stretch, here. The difference between 100 to 500 or 1000 is exponentially different than the difference of 100 to 3,700. They don't need this many people. Period. This is a "who's dick is bigger" contest over a thinly veiled (and absurd) justification of infrastructure. No one outside of this bubble buys this shit. There are global supply-chain networks that move millions of products, and they don't employ as much as a micro-blogging platform. Come on, now.
and they only need account managers because they are selling advertising. They could have chosen to monetize their platform another way but lacked the creativity to do so.
At one point like 10 years ago, I left ebay because it felt like more than 60% of everything was fraud. I think they've curtailed that a bit, but that is what Twitter feels like to me now. Like 80% of the traffic has to be fraud because they killed all the real users. They have to make money off the fraud and be sort of complicit just like Paypal and eBay used to be (to a larger degree than they are now).
No one ever held Paypal and eBay responsible for what they did because it was so esoteric. It seems that there is a government investigation that is going to lead to a nice graph of botnets used for social manipulation which is cool, but I doubt we'll ever get a proper accounting of the really inventive fraud that dominates what twitter is.
As the pricing appears to be based on # of requests and/or # of tweets consumed, does it have a `include_spambots=false` filter or do we have to pay for the pleasure of wading through spam?
I linked this elsewhere in this thread, but I reported this obvious spam account, with hundreds of spam followers, weeks ago and nothing has been done: https://twitter.com/ksfAKBARI
I feel like twitter must not be taking spambots seriously, or has malicious intent to keep them there, because out of hundreds (thousands?) of employees they can't get a team onboard to shut these obvious accounts down. I mean, it's been reported and it still exists. Either the report log is huge or it's being ignored.
Is there an available history of Twitter's APIs over the years? IIRC it was initially open, and then they locked it down so many 3rd party apps for Twitter had to shut down, and now they're sort of opening it up again?
They never "locked it down", they just rate limited some vital endpoints and never bothered to add endpoints for new, important features.
My own recommendation: disassemble any official app (for example, the Android one with apktool) and get the tokens and endpoints and do whatever the hell you want without paying a cent.
> They never "locked it down", they just rate limited some vital endpoints and never bothered to add endpoints for new, important features.
More importantly: they introduced a lifetime cap on the number of OAuth tokens that applications could create, which meant a lifetime cap on how many different users a Twitter client could support. Once you hit your Nth user, you could never have a new person authenticate that client.
That killed off basically every decent third-party Twitter client.
In the beginning you could post a tweet just by doing a http basic-authenticated post containing the tweet body, and there were unauthenticated rss feeds for most data feeds. These days you have to create app ids, tokens, etc etc etc. It really was a lot more open earlier.
Twitter was part of the reason I fell in love with development. The easy interaction with the whole platform changed my view on data. It broke my heart when they introduced v2
I feel like you could do this just watching network traffic, and judging by the fact that nobody's done it, I also feel like there's some mechanism preventing this that I'm not knowledgeable enough in the subject to be aware of.
The official twitter app is possibly using these premium endpoints, so it's token has access to those. If you "grab" that token(s) somehow, you can go ahead and use it in your app, I presume. I don't know how this will work when your app wants to connect to someone else's account, however.
Nefarious purposes, mostly. I can crawl without rate limits (well, there are rate limits, but it's much harder to hit them), and I also can spam a lot without getting my account flagged (but it eventually gets flagged, of course).
By spamming I don't mean the usual "click on this link and I'll show you my tits" spam; if you create a useful bot that sends "expected" mentions (for example in response to mentions you receive, and not just spam) it will get banned in a matter of hours. With their "secret" tokens, it won't get banned.
My experience with Twitter API has been horrible to say the least. Plenty of API errors, service over capacity, and instability. So now I have to pay to get access to an API that's stable?
Step 1 - build a product. make sure it has lots of bugs and is unstable.
Step 2 - roll out a premium product that's the same product, just stable, and less buggy.
Profit.
How is charging for an API not a finger to the developer community? Facebook for example doesn't charge for their API access. They charge for the products that the API has access to. If I have access to a product, I shouldn't need to be charged for the APIs to access that product
Interesting. I found Twitter to be vastly better than other social media companies. Far better documentation (though weird complicated auth where I get 4 pieces of info for some reason). Far more stable and understandable. Twitter returns a ton of data in each response so there's no need to go making multiple requests.
Compare to FB which is just a nightmare of trying to guess and figure things out. It's so bad. If it reflects their underlying model at all, their main codebase must be very difficult to work on.
Charging is a welcome improvement over the previous situation: Deal with Gnip or other data brokers. They're "Enterprise". Get no information, have to deal with a long sales cycle and sales team, have to wade through tons of BS to figure out what you need. It sucks.
In terms of documentation I do agree with you, although I suspect that's just due to the complexity of Facebook and its vast array of products, vs Twitter.
My bad experiences have to do with the actual API calls themselves which aren't stable and occasionally times out or errors out.
If anything I wish they would improve their API infrastructure as a whole instead of offering the improvements only on paid accounts. I can see charging for rate limits but not for system stability.
On the other hand, Facebook randomly silently gimps their APIs. I did a demo of the friendship paradox (https://alone-togetger.firebaseapp.com) for a tech demo. The paradox would have been better demonstrated by Facebook where friendship is symmetric, but their API was silently gimped years ago to only return graph information for friends who have also added the app.
Perhaps it's as simple as that. I dumped too many resources into my last effort: time designing, flying out, sitting down with their dev advocates, all a waste of my time. I shan't reward them for pulling the rug out, or make the same mistake again.
Also, it's in solidarity with developers who invested even more than myself, years of time into apps, to be betrayed.
Good for them, I honestly hope it works business-wise as they want and it makes developers satisfied. However, I'm afraid Twitter the company has got the same sort of trust issues as Google from the developers POV. They need to address this specifically, if they want people to believe their premium API move is for real.
Google is obviously a huge company with many different products, but I've had great success and satisfaction with many of their APIs. What sort of problems are you referring to?
This is a good start. Gnip seems to basically be invite only (I've tried contacted them multiple times and have not once received a response) and the public API is so limited it's near useless.
Gnip is Twitter's enterprise API business with annual subscriptions to firehose and decahose access. This is month-to-month billing for improved API access and data quality.
As someone who had a Gnip subscription for years I think "invite only" is close to a fair characterization.
It took huge amounts of effort to get them to return our inquiries. Given the huge amount of money we ended up spending that seemed ridiculous.
But I guess that just got us ready for the whole "everyone must move to Gnip 2.0, but no we won't merge the multiple pull requests to make our official client work with our new API"[1] thing. Wow, that was fun...
I think the post was quite clear that it meant "behaves like it is invite only", as others have explained. I have to say, if your job is improving how Twitter looks to developers you are IMHO not doing a very good job of it right now, a lot of your replies appear quite tone-deaf to me.
I appreciate semantics and nuance, and often get called a pedant, but you seem to be missing the argument being made. It's not technically 'invite-only' though it in practice, it is 'invite-only.' This looks like a product management shortcoming that seems all too prevalent at Twitter HQ. I say this as someone who was on Twitter in early '08, but quit late'10. I still believe in Twitter's value, which I thought was obvious, then couldn't understand the disinterest, and finally surprised it caught on with the US public. The company really has a long way to go in product management WRT user experience and developer ecosystem.
Is everything starting around $149+, or only search?
I suppose that's understandable, but I think it would be cool if they implemented something more akin to the AWS pricing model (S3, for example, as tweet streams remind me more of this than of enterprise-level pricing).
Another user suggested that a lot of searches could result in massive spam results that suck away from the rate limit and clutter analysis.
What is Twitter doing to combat this? For example, I reported this account: https://twitter.com/ksfAKBARI for being spam account about 3 weeks ago, and nothing has been done yet. I noted in my report that it has a massive follower network of hundreds of other spam accounts. Why isn't Twitter at least tackling these extremely obvious spam accounts?
Spambots aren't against Twitters ToS. That one in particular might be part of a large bot network, but hasn't actually posted anything since 2012.
Even so, automatic posting is explicitly allowed by the developer ToS, with the only restriction that you don't cause undue load (which appears to be around how often you post).
To be clear - the only things which reliably get accounts banned on Twitter are Doxing people and actual terrorist accounts.
However (based on how Gnip and the public API work), they do some level of spam filtering by default. On some APIs you can turn that filter off.
Until info about the new endpoints and queries is released, I'm not rushing to get out my wallet, but at the very least it's encouraging to see any movement regarding the API. This seems like a vote of confidence that the basic API will be around for at least a few more years.
Ecosystem lock-in. I remember shortly after Twitter launched there were many copycat microblogging services and the community had grand ideas of an "SMTP for Microblogging" so Mentions, Retweets and the like would work in a distributed fashion between service-providers, you could even run your own fully federated microblog off a Raspberry Pi in your garage if you wanted.
Twitter kept themselves in the lead - not just by delivering a superior user-experience (Nazis and trolls notwithstanding...) but also cultivating a developer community in a way that kept people off rival platforms - before stabbing them in the back in recent years (e.g. no support for polls in third-party clients).
This announcement is a continuation of the strategy: Twitter is making more data about their platform available, other platforms aren't as open, so people will stick with Twitter because their data is available - and it also benefits them directly because these "Premium" APIs do cost money to access:
> Pricing for these elevated tiers of the Search Tweets API starts at $149/month
Their search product started as a third-party integration (Summize). Their official iOS app started as a third-party app (Tweetie). They've benefited immensely.
API access to twitter is a huge business for a ton of different enterprises. Everything from customer support/happiness products to finance wants the twitter firehose. This is a totally reasonable and good business avenue for them to care about / explore.
