There are conversations that work on twitter, and ones that don't. In general, twitter conversations are quick back-and-forth affairs. If you need more than 2 tweets to express your thought, you're either using the wrong medium or thinking too far. If that conversation model doesn't fit you, that's perfectly fine, but that's not the fault of twitter. It's just a bad match-up. In that case, it's perfectly fine to respond to a tweet with a link to a medium that does support your model better.
If you do get roped into a conversation where you need more tweets to explain, here's a few simple guidelines:
- Don't split sentences over tweets. People will only read one part.
- If you really must, use "(cont.)" on both tweets (end of the first and start of the second), so it's obvious there was another part. I know that reduces your character limit, but it's worth it to be less ambiguous
- Make each continued tweet a reply to its predecessor. You can reply to your own tweets. It facilitates following the conversation on most clients.
- If it's more than 2 tweets, consider using twitlonger. In general, that should be a last resort, though.
Without trying to make it sound high-handed, there's something of an art to smooth twitter conversations. Terseness is the key. I find them rewarding because, when done right, they have an enormous density of information in a short time.
> In general, twitter conversations are quick back-and-forth affairs. If you need more than 2 tweets to express your thought, you're either using the wrong medium or thinking too far.
Hmmm. I'm not sure how to respond to the suggestion that people who can't fully express their thoughts in around 300 characters are thinking too far ... or the subtly implied pejorative of 'too-far-thinking'.
Aside: I note you used 1386 characters to describe the benefits of terseness.
No pejorative was meant at all. My point about 'thinking too far' was about the singular thought you're trying to express. It's easy to want to say too much, 'walking the chain too far' if you will. Twitter is a far more iterative medium than most, and expressing a complicated thought isn't a case of writing it out in whole. Usually, you will develop the thought together with the other party. Conversations are closer to a form of collective thinking than a long-form debate. Thought patterns are more on display.
Aside: this conversation would have taken a very different form if we had had it on twitter. The medium defines the form. So I don't mind being verbose here :)
The idea of iteration through (a) thought I think has been explained to me before, but I must have suppressed it. I can see why it's necessary within the context and medium of twitter ... however I'm unconvinced (duh :) that it's a useful constraint to apply to a communication medium. I don't expect I'm unique in voicing this opinion.
The idea of developing a thought with another party, or as you also describe it, 'collective thinking', is genuinely anathema to me. I'm happy to have a discussion that involves me adjusting my thinking, and, of course, a discussion that results in the other party(/ies) adjusting their thinking. Actually, I expect that essence of conciliation and compromise is what you're describing. However, the idea of avoiding 'long term debate' really does unnerve me.
Perhaps I've been debating for too many years with people that I really wouldn't want to share a collective thought with.
I would agree that iteration through thought is not necessarily a useful constraint on a medium by itself. In the case of twitter, however, it is merely a product of the 140-character limit for each message. If, during a stand-up debate, your opponent was allowed to interject after every sentence, you would get much the same. That sounds incredibly annoying and counter-productive at first, but the form of the debate would adapt over time.
The advantage of developing a thought in tandem with somebody is that the compromise happens more naturally. You rarely need to 'concede' a point or prove somebody wrong, since you can influence the way the point is made in the first place, pointing out flaws in their reasoning or facts as they happen. Conversely, the other can do that for your reasoning. The result is that, rather than adjust your thinking in a few big steps, you adjust it in minor ways with every sentence. It takes time and practice to get used to this, but it can be very productive and enlightening.
On the other hand, I would never avoid long-form debate either. Both have advantages and disadvantages, just as both have mediums where they work and mediums where they don't. Twitter isn't conducive to classical debate, just as hackernews isn't conducive to the iterative thought development style.
Hrm, what you are suggesting sounds like some kind of conversation back and forth between 2 people. Like a chat system or something. Issue is, you are talking in an environment where everyone can interject, and everyone has a strongly felt opinion. Just look at any news show "interview" between 2 sides of an argument. They tend to devolve into shouting matches of who can drown out the other side without getting smacked down by the host.
It is much like a conversation in a chat system, but the etiquette on twitter means your conversation partner usually won't wait for you to finish a string of messages.
It's true that it's easy for people to interject and annoy, and it happens more and more the bigger your celebrity status. However, unlike the interview example, it's also easy to ignore them. Twitter's reply helped a lot for that. Replying to anybody is optional, you don't even have to read their tweets if you don't want to. It's not perfect by any means, I don't think any platform will ever be. But it isn't bad.
Wow, you really, really nitpicked OPs argument and completely ignored things to make your point. The comment was "you're either using the wrong medium or thinking too far". Assuming the false dichotomy here (notably, of a quick comment on HN...), maybe, just maybe, it's not that people who can't fully express their thoughts in around 300 characters are thinking too far, maybe it's the other part -- that they're using the wrong medium?
The only portion of OPs comment I can find to "describe the benefits of terseness" would be the last paragraph. If I shorten that paragraph to the last two sentences, it fits in a tweet.
Where does one draw the line between cherry-picking from, and identifying the quintessential component of, a message? Rhetorical question. :)
If X or Y --> means that both are possible, right? If you then suggest that it's always and only X, then the or construct is misleading. I was picking up on the fact that there will be some Y, and that worried me.
But, to be absolutely clear, you are completely correct in saying that in some cases X (or inappropriate medium selection) is going to apply. (I wonder whether people who need more than 2 tweets to express their thoughts are capable of self-identifying which category they fit into. If not them, then who?)
Going beyond that I'd suggest that twitter is an appropriate communication medium far less frequently than people evidently believe it is.
I don't believe shortening things to fit into tweets is a healthy aspiration. I ran a character count solely to demonstrate how many tweets would have been required to get that entire message across, and most of chton's message was a guide on how to communicate in a medium that enforces brevity. An undeniably useful guide if you happen to be embracing that medium, mind.
Most people probably fall into the using the wrong medium category. If you are taking are taking over 140 characters to say what amount's to 'Who want's to go to Olive Garden?' then you are thinking too far though.
And he specifically chose a medium which was not twitter to describe it.
"If you need more than 2 tweets to express your thought, you're either using the wrong medium or thinking too far. If that conversation model doesn't fit you, that's perfectly fine, but that's not the fault of twitter."
Very true. It's like blaming a radio for not having pictures to go with an advertisement. Wrong tool. Echo chambers aren't good for in depth conversations.
God, I know. I keep hearing about conversations and such on twitter, while I have a hard time fitting a single thought into a tweet, much less a thoughtful response to someone's query.
Look! Even that sentence was longer than a tweet! I don't know how people do it.
Terseness is short, abrupt, irritated. Succinctness is short, clearly explained. They both mean short, but the tones of the words are quite different. '"Leave me be!", he replied succinctly' doesn't work, for example.
To me, this is exactly the kind of nuance that is lost by training yourself to limit ideas to 140 characters.
Terse also means short, clearly explained. Yes, 'terse' has an extra meaning, but in a technical context, and especially when contrasted with 'verbose', it should be completely clear which meaning I meant.
And for the record: I don't train myself to limit my thoughts, I train myself to use a form for expressing them appropriate to the medium I'm doing it in. Expressing something shorter and in steps is not inferior to long-winded explanations. If you must blame something for a fine nuance I failed to grasp, blame the fact that English is not my first language. I doubt I would have caught it in Dutch either, but at least that could be a valid reason.
But herein lies part of the problem - at first you said that the two meant the same thing in common parlance, then you said the same thing in technical parlance; there's confusion in the audience right there. Anyway, while it was clear what you meant, what you said was not quite correct, which is why I piped up.
Particularly in technical writing, you've got to get your words nailed down correctly. This is why there are things like the RFC defining what 'shall', 'should', and 'may' mean (intended for other RFCs, but it's also used elsewhere)[1], or the development of Simplified Technical English[2]. The latter was developed in aerospace engineering, where minor misunderstandings can literally cost lives.
Subtle differences between words are important, and they allow us to communicate more complex ideas. Twitter may have it's place, but any conversation involving nuance really isn't suited for it. A high-quality conversation includes caveats and careful framing, which can't really be fit into 140 characters. At least, in my opinion.
There are people I do know on Twitter but do not converse with them on there. I email, text, or IM them. Always use the best tools that are available...
I agree, use the best tools. But if you email or text them, there is no way for other people to join the conversation. Think of a twitter conversation as one at a party where you know people. Anybody can come join in, and can add value. It's more akin to conversing on IRC than over email.
Holy crap, now it's a multi-person conversation? I would say let's meet for drinks... How about Hangouts or what ever one uses on iPhones? Maybe this is more for the under 25 crowd who like to do things the hard way on 5" screens with no keyboards. Not me...
It can be. This conversation is a multi-person one too. That's still different than, say, hangouts because everybody is free to reply, not just people you invited. Maybe somebody neither of us know will read this next month and come up with an interesting counterpoint.
In any case, using some form of video calling is too intrusive. I can type this reply, do something else for 10 minutes, then check back. But then I was always one for text mediums rather than video or audio calls :)
Yep. Incidentally this is the exact same reason why I can't have conversations by shouting out the window of my car as I speed past people on the street.
I don't think this is due to the limit of 140 characters (even if it doesn't help) but much more due to the audience of fanboys and crow cheered during a fight.
There is the same problem on Google+ where there is not 140 characters limit. Any discussions with Linus or any other "celebrity" in it and the comments thread will be full of : "+1", "M. Linus you are so smart, you are my hero", "Well said, ripe him a new one" etc... Which drown any constructed argument and render any constructed debate impossible.
It's really unfortunate though there there are people who can't resist the urge to shame others and enlighten the world with their sharp & intelligent wit, such as the person who wrote:
"You have clearly not understood how the math works or why tail latencies matter in dist sys. I think we're done here."
> It's really unfortunate though there there are people who can't resist the urge to shame others and enlighten the world with their sharp & intelligent wit
Amusingly this has now happened in another thread in these comments.
The meaning of a post has been overshadowed by a conversation on how the author used word A when they really should have used word B, and then they were berated for trying to explain why they thought word A was acceptable in the context.
There's also the problem of "dot-@ing", where instead of starting a tweet with "@staz", people start it with ".@staz", meaning that the tweet appears in the timeline of everyone who follows you.
I rarely see this used in situations that aren't passive aggressive attempts to get followers to pile onto someone you disagree with. Totally ruins any chance of good debate.
I agree that this is not a problem that only affects Twitter.
Most of the current discussion platforms are way behind what was standard in Usenet or in your average Usenet client.
No threaded discussions, no filtering (not for the one who comments and doesn't want to read those +1 post and most often also not for the OP), not even blocking of people you don't want to read.
Oh I don't know, I remember plenty of people on usenet who would quote a fragment--deliberately eliminating relevant context--to create a straw man that they could tear into. Some people just like to argue, and the only way to deal with them in an unmoderated forum is to ignore them. Kill files were useful for persistent offenders.
Of course this did also happen on usenet. Call me old but a good flamewar beats a good shitstorm any time (and the names might even be an indicator why).
But one design element of twitter might make it easier to stumble into the realm of people who will wildly disagree with a poster's opinion.
Just replying to some tweet you can not know what kind of domain you are entering. With usenet groups the name has always been a strong indicator what people you will encounter.
If you post "Ruby sucks and is dead slow, Perl rulez!!1eleven" into comp.lang.ruby, you shouldn't be surprised if you get flamed. With twitter accounts, this is not that obvious.
Of course, this is a core design element of twitter that it is concerned with accounts and not thematical groups and both have their advantages and disadvantages but I think this is one of the reason that you can much more easily and unintentionally anger the wrong people on twitter.
Mailing lists can be ok with proper client but it's a step back. I found some subreddits useful (with RES) but even then it's hard to track new replies in threads that you are interested in.
HN has quality discussions but these are just about some links and it's hard to get back to some thread to see new replies (and preferably prioritize those from people who I value).
Every reddit thread has a separate RSS feed so you can drop it into your favorite RSS reader. I'm using a IRC bot with an RSS plugin to not spam my normal RSS feed aggregator.
But this doesn't solve the issue of being notified of replies to subthreads. Reddit threads up to 100 messages tops are manageable that way but anything larger isn't.
Compare that to usenet threads were you could easily handle thousands of messages in one threads, this is really a shame.
The average thing I publish on the Internet is thousands of words long. There was never a shortage of people taking some of those words out of their context, and arguing against what those words supposedly say. And I'm likely to have done it myself to someone else's writing. Certainly I pay attention to some things more than others when I read, especially if I come with some prior knowledge and opinions. I can easily fail to notice important things when scanning something quickly.
And the longer it is, the more likely I am to be scanning quickly, actually. On the other hand, shorter messages and less context result in things simply being omitted, even if you do read carefully.
Maybe Twitter is worse than average, but it's pretty bad with any medium.
There was never a shortage of people taking some of those words out of their context, and arguing against what those words supposedly say.
Long ago I learned that if you want to have "viral" success, even on a more informed site like HN, avoid making more than one statement or claim in a piece.
"Modern Java application servers offer surprisingly high performance" (followed by details of performance improvements, etc), for instance, is okay: everyone who agrees will forward and agree and post, while those who might not agree or care will remain ambivalent and disengaged.
Change that up slightly, though, and conversationally add "...which might allow you to deploy on fewer servers than a similar solution built in Ruby" and you'll instantly gain armies of very motivated detractors who will go forth to denounce everything you've ever said, declare everything you've written stupid and misinformed, etc. Not based on objective disagreements, but based on a knee-jerk opposition that, much as with politics, can't be on a point or issue (e.g. "agree to disagree"), but has to encompass everything.
People who might agree or yield value skip the piece after seeing endless criticisms and complaints.
The crux being that it's very hard to gain agreement, easy to gain enemies. If you state ten things, people who disagree with but one of them will often fall in the latter camp.
Not that this changes how I personally write -- where I'll load pieces with things that people can disagree with -- but I see it constantly as stories tread the bottom of the front page, the detractors who find that one tiny thing they disagree with and blow it up to be all-encompassing, and it gets flagged to oblivion.
I believe that small blogs and RSS/ATOM are a good answer here. I can't and don't want to follow high-volume streams, be it Twitter or "news" blogs. However, if you combine 20 interesting but low-volume blogs in your feed reader, you will always know when something interesting happened, without having to check all those sites by hand.
(Of course I also have high-volume sites in my feed reader, but that's more for "news" / skimming though, not for keeping in touch.)
Blogs are like an archipelago of privately owned islands. Twitter is a whole continent.
RSS and Atom allow for people to tell the people on the other islands about what they wrote but it's a one-way message. For an interaction to occur someone has to go over to the private island.
With blogs the entire process of publishing, reading and responding is privatized and fragmented.
Twitter is a centralized location that unites publishing, reading and interaction in one faux-public location. There is utility in these kinds of social networks. The issue is that they are not public places.
Twitter controls how the information is written and read. Twitter optimizes for their own private interests that have nothing to do with intelligent discourse. In the early days, Twitter was an open platform with a large number of competing clients. Early users and developers invented the core interactions that we see now see on Twitter. Hashtags and retweets were not invented by Twitter.
Twitter has since put their API on lock down. They alone have the right and ability to make clients. As a publicly traded company they're now required by law to create products that increase shareholder profits. The evolution of the platform is out of the hands of anyone but the managers of a private company.
I hope it doesn't come as a surprise that intelligent public discourse is not as profitable as the cheap and simple thrills that seem to be the core of social interactions on the Internet these days.
There were plenty of "non-techies" who used Google Reader before it shut down. And at least in my circles there are plenty of non-techies who say that they don't get the concept of twitter.
EDIT: Both, Twitter and RSS, have their own terms and concepts which need some kind of introduction for most, especially non-tech people. For Twitter its hashtag, TL, DM, etc for RSS its stuff like Feed, Reader, etc
How exactly? In the time it takes you to register an account on twitter.com you could as well register an account on wordpress.com and publish via RSS. The main difference is that longer content is possible, that you could find even easier/faster/whatever alternatives because its federated and that your communications are not dependent on the policies of a single private company - which for non-us-citizens like me is operating under foreign law, a law I don't know as well as the local one.
I don't watch much TV, but what little prime time TV I've seen recently, seems to pimp their twitter acct and semi-relevant hashtags constantly both intro and exit to the program and even believe it or not, during the program they pop hashtags on the screen. Only about 1/30th the population watches survivor so this is probably news to most HN readers.
One of the biggest things is that there is one, singular Twitter app. There are a million RSS readers, but it's difficult to know which one to get. And, of course, it's a reader, not a publisher - you need a separate app for that.
It depends a lot on how you use it. For me, it's a firehose of information (I follow around 2000 accounts). You drink from it when you feel like. If the fire hose is big enough, there's always something interesting to read. It could be the latest news, and it often is, or it could be someone just tweeting something they learned or how they think about things. Twitter enables that with the 140 character limit per tweet, it means the information is very condensed and you can ingest an enormous amount in a short time.
Twitter is terrible for meaningful conversation and debate. And I say that as an avid user.
Networking, sharing links, tidbits of insight, short messages, and Q&As all work really well. I've learned quite a bit from the curated articles and reports tweeted by those I follow. And if you follow the right people and engage appropriately, Twitter can open up a lot of opportunities. But anything beyond that is a waste of time, and can border damaging.
In fact, for all the good Twitter has facilitated (protests, revolutions, etc), I think it's having some negative impact. It is by far the most popular platform where individuals of all backgrounds, beliefs, and opinions congregate and interact, but it's built in a way where it is absolutely not conducive to meaningful debate, and so when the huge opinionated masses clash, the bite-sized arguments volleyed by either side are taken without context and nuance, further igniting and polarizing people. I'm seeing this happen with issues like feminism and Islam, where someone will tweet a very distilled version of a larger and more thorough opinion, it will be taken at face-value, someone will retweet it with a snarky comment, it snowballs into a food fight with enraged people retweeting/replying, and the original tweeter trying to add context but not being able to keep up with the reactionary domino effect. And because Twitter has almost become the official sounding board for many people, their tweets and reactions to those tweets contribute to their public image, reputation, and online presence, all damaged by death threats, accusations of sexism, bigotry, racism, etc, often thrown around unwarranted by people taking things out of context and who feel antagonized or polarized because of the way Twitter is structured.
And that's why I avoid tweeting about religion, politics, and databases.
The distributed systems community as represented on twitter is pretty shameful; discussions of any non-CA database tend to devolve into intentional misreading and jokey sarcastic pile-on bullying, with Basho folk leading the charge.
Personally I get that they've lived through some pretty gruesome use cases and wars, but jesus, glass houses and stones, guys.
BTW, this also brings to mind another thing I've come to believe over the years:
> "You have clearly not understood how the math works or why tail latencies matter in dist sys. I think we're done here."
If, when you write something, it sounds like it's coming straight out of The Simpsons' "Comic Book Guy", you should probably rethink it unless you really mean to be a bit of a jerk.
Small elements of Twitter's design really exacerbate this problem.
In a strictly time-ordered tweet view, you'd see one or another of antirez's tweets, be be curious, and go back into his & other respondents timelines to see the full conversation, w/ whatever context was originally there.
But how Twitter works right now is you get what appears to be a full conversation neatly tied together, but it's actually been selectively edited by an algorithm maximizing (I'm educated guessing) engagement, which ends up basically being an automated filter to create misunderstandings and controversy.
The more visible you are, the higher your standing, the likelier it is that people try to catch you in a mistake to improve their own visibility and standing. They are not aware that this is the motivation for their behavior.
The older I get the more I ignore. I'm not missing out on anything by not following twitter and it's really just a distraction from my goals. I sink way more time into HN than other social media, and I don't spend that much time here. My goal is to be completely off in my own world a decade from now, focusing strictly on what is important to me.
This is unfortunately a case of Twitter biasing technical discussions to highly literate, careful people.
Good tweeting requires almost Orwellian control over writing.
It needs to be extremely terse and explicit.
You need to be the kind of person that will write a tweet, read it and if it could be misunderstood in any way delete it, before rewriting it several times.
You need to do this until you are able to fit each inalienable component fact into 140 characters.
This kind of writer is often careful to form only one idea per tweet, considers using mathematical notation if it requires less characters, and will finally consider placing a [...] on the end of sentences which leave their argument hanging. Though when you have to do a lot of these things, it's often because you have not fully analysed the concept to a point where you understand it well enough to say simply.
When I want to write something over multiple tweets, I use a simple syntax to make it clear that each individual tweet is meant to be read in context with the others: I simply suffix all tweets except the last one with ">" and prefix all tweets except the first one with "<".
Isn't it a bit silly in this day and age to use that kind of ad hoc hack in order to express a more complex point?
I mean I understand that's how twitter works and probably part of what made it successful, but it baffles me that it's seemingly becoming a "serious" way of communications. Politicians are expected to have a twitter account, serious news organisations often quote twits directly in the body of their articles etc...
I don't understand why so many people chose to de-facto standardize on such a poor medium for communication, especially since you have a billion alternatives without such limitations.
Agreed; this seems like the most explicit way to do it. Of course it means you have to actually take the time to form the whole argument before firing the first tweet out there, but that's almost certainly a good thing.
Re: the original convo, they're stuck on the fact that arithmetic mean isn't terribly valid on a skewed distribution (and response/completion distributions are generally positive-skewed because of the firm/hard lower bound on task completion).
However, I do get what you're trying to do re: identify whether the problem is related to outliers. But in general, comparing to median has a lot more validity. Median isn't as sensitive to skew, and will be closer to the peak. For your purposes it probably wouldn't be a lot different but you wouldn't have pushed the "it's wrong" button.
All said, though, even comparing 50th (median) to 99th is pretty coarse. I'd probably be looking somewhere closer to 75th percentile for a comparison. Basically, you'd want to guess what percent might reasonably be affected by performance spikes and compare from there.
What is it with Postgres fans that "have to" prove they're better?
I don't want to use Postgres. I want to use Redis. Postgres IS NOT the solution for a lot of things. Yeah, feel free to call me stupid for not knowing every nook and cranny of PGSQL. I don't care
"I love to argue, but this is just a futile exercise."
Well Redis is NOT the solution for many things too. I find it is a huge problem in the DB community that people think there has to be a solve all solution. Every DB strategy has strengths and weaknesses. I use Redis and Postgres they are both excellent products. I think the days of having a single data-store are gone. I personally use different data stores depending on the type of data I am dealing with. It is like the old interview question "What platform does Google use?"
I've actually been following the "Twitter drama" around antirez, redis and everyone else - and it has little to do with "Postgres being better" and every complaining with Distrubuted Redis (Redis Sentinel/Cluster?) being terrible.
AFAIK, the arguing with Antirez with Redis has been going on for weeks now (and it hasn't been necessarily been unfounded). The exact issue the Stripe guys had, had to do with a replicated Redis setup.
There is nothing specific to replication in the Stripe issue. They noticed a well-known problem which is that, with large datasets and some categories of hardware, forking to snapshot to RDB takes a long time (can be a few seconds) and blocks the Redis process. This also happens on single-instance Redis.
EDIT: actually I am talking about the second problem mentioned in their post here: https://stripe.com/blog/game-day-exercises-at-stripe (latency spikes). The first problem (data loss) is related to replication.
I can't speak to Redis, because I don't use it, and mostly see it as being fairly distinct from Postgres, but with Mysql it's always frustrating to know that you could be using a tool that's generally better, but can't, because initially someone chose Mysql because they "heard it was fast" or "that's what the hosting company provided", and now they're stuck with it.
You are not wedded to an SQL vendor. You actually can migrate your data, as long as your code is farily vendor-agnostic. I write Django apps and use the ORM for everything. That frees my customers from any sort of raw sql specific to one DB server.
If everything you do is CRUD, sure. Often there are non-CRUD, more advanced queries that need to be run. These, and other issues, are normally what gets you. For instance, pgsql will error if you attempt to write 5 chars to a 2 chat field, mysql will silently truncate.
I'm trying for the life of me to think of queries I could make that are non-CRUD. Maybe I am just stuck in my current tools to think flexibly about it.
I presided of over a Migration from DB2 to Oracle. All of the operations were CRUD.
However the DB2 database had materialized views that were crucial to the business.
Porting those over was non-trivial. Oracle returned a different value for null fields than DB2 (empty strings for null strings vs actual null in one case) did so various portions of the code would fail seemingly randomly.
Some of the constraints didn't map quite as cleanly as the others.
And we haven't even started on the performance tuning and how your schema affects them.
Sure you may be able to port all or most of your schema and queries over to another database but you'll have a lot of work to do after that verifying that nothing is broken and your performance is still acceptable.
SQL is not even close to a guarantee of portability.
Using CTEs, including writable CTEs is not something I'd want to do in an ORM. Also, whenever I find myself having to have more than a few expression fields, I find ORMS clunky.
Trying to load more than one record from a a single row is supported in some, but not all ORMs as well.
Aggregate queries, especially more advanced ones than just min, max, sum, avg, stddev, &c, are not always supported by ORMs either, or you need to use expressions or extend parts of the ORM. It's just a mess sometimes and I'd rather query and then load.
It always annoys me to hear people talking about how slow MySQL is and have to deal with them going about modelling things poorly and sweating a few million row when they don't know how to design indexes properly and don't understand predicate order.. Or correlated sub-queries.
That said, PostgreSQL is pretty fantastic software too.
I've been friends with antirez a lot longer than Redis has been around, and have a lot of faith in his abilities as a programmer. It irritates me to see the snark directed at him on Twitter.
But for most of what I do - smallish bootstrapped things - I want to build on Postgres, because I know I can take it a lot of different directions, as needs be. Unless I'm 100% sure that it's a bad fit for SQL and a great fit for Redis or something else, I'm going to consider Redis as a later optimization, not something to build on from the start.
Like you write though, I mostly view Postgres and Redis as Apples and Oranges that could probably work together quite well in the right circumstances. Most of my ire is directed towards Mysql :-)
Moreover, I can't remember a single time PostgreSQL folks have been unfair... PostgreSQL is definitely good both as a product and as developers culture.
This is a perfect example of the problem the guy you're responding to is talking about.
You are correct, but nobody chooses databases by count-of-good-use-cases. You choose databases by applicability against the use cases you have in front of you and are likely to have in front of you.
Like any high volume channels, you have to ruthlessly curate your Twitter feed. Be picky about who you follow and block the trolls. Twitter is a pretty useful source of information after that. I usually see hints of upcoming vulnerabilities before the details get publicly released. That's pretty useful.
When trying to get through what appears to be a brick wall of myopia, the liberal use of the block feature is more productive.
Seriously, I don't even try to have arguments anymore. A sane and relatively productive one is nearly impossible with some people. I don't know if it's the brevity of speech or the even shorter attention span, but the Online Disinhibition Effect seems to be an order of magnitude greater when it comes to certain topics.
I absolutely think that you should only use twitter to post links to blog posts and mailing lists emails. That way, people who really care will read, others will not, and they likely won't comment without reading.
My 16 yr. old daughter explained to me two days ago why Twitter conversations are broken, how to fix them and that maybe we should design products together since we can do it better than Twitter or that maybe she should send them her idea.
My response was, if you want to develop products together, let's do that. Second, Twitter probably has 50 product people who have all sat around and discussed her solution in detail and come to the conclusion that, for some reason we won't understand but that is probably better for Twitter, they shouldn't implement it. Most likely it has something to do with more views.
Then I welcomed her to the shitty reality that once a company gets the market share, they stop caring about a better experience for users and more about the bottom line OR maybe they're just stupid. After all, Twitter didn't have a single backup for something like the first year or two of existence.
Not sure they are not caring or stupid. Perhaps they decided that Twitter is NOT a platform for conversations. I use Twitter as a real time platform for announcements, status updates, and just general sentiment. I do not really want to read debate there or conversations. If someone wants to discuss things there are other places to do that.
As for their engineering problems at the beginning yes it was bad but then again they grew massively very quickly and they needed to come up with novel solutions at huge scale.
If you can't have a conversation on Twitter you may be doing something wrong. Thanks to the peers on the network we've found all sorts of interesting things.
Things like the location of rockets associated with sarin attacks in Syria[0] placed squarely on Mezzeh military airbase in Damascus. The rockets at the time were thought to be Iranian Falaq but more evidence showed them to be part of their own distinct program.
The discussion did not stay entirely on Twitter but was the main venue for it. Without being able to rope millions of random peers you may not know you will converse with later into one IRC channel, Twitter works for wide broadcast fast paced discussion.
Blocking people helps sure, but I think many people's problem seems to be a perceived lack of interesting peers.
Sure sounds like you're blaming antirez for people on Twitter taking his tweet out of context and running with it. It's also completely unfair to say "Twitter was awesome in this very explicit situation so you obviously did something wrong in your situation".
No, that is on the people who did that. If anything I would blame antirez for getting flustered enough about this non-Twitter specific problem to write a blog post.
He is not to blame at all. He made a good faith effort to communicate which did not work and openly reasoned about the challenges of a relatively new platform that many people are still trying to make the best use of. I'm glad he posted.
I think the context has important role in technological issues and it needs to be explained.
Twitter is very useful when you need some quick answer, with no premises.
While a tweet with a post link is enough, one tweet is just buzz
I am quite skeptical about twitter myself but I never understood this argument: What stops you from explaining your new or controversial argument in (n * 140) chars, or multiple tweets?
The fact that people won't take time to piece together the separate parts, and will take each out of context. This is the whole point of the original submission.
I think it was a beautiful "what if." What happens when we limit a message to 140 chars? The character of conversation changes - so do people's use patterns. The most wonderful thing about twitter is that you can't get lost down the rabbit hole of a three paragraph email. And there's pressure to reply quickly to stay part of the conversation.
What happens when you set a message limit of 140 characters is you end up with a database that is perfect for training computers about semantics.
It is a genius example of design thinking, I'll grant you, but twitter's priority as a business was to build a database for analysis and that was the design constraint which resulted in a 140 character limit to messages.
It is a communication platform, but only in as much as it fulfills its main purpose of generating datasets.
It's limited to 140 characters because it depended on SMS as a transport. The idea that the main purpose of this limit was for generating datasets presumes far more foresight than the Twitter founders deserve.
> twitter's priority as a business was to build a database for analysis and that was the design constraint which resulted in a 140 character limit to messages
This is the first time I hear about this. Do you have any source or did you make it up yourself?
Twitter's original function was group SMS messaging among friends, the 140 char limit was the 160 character SMS limit + an 18 character username.
May I recommend using percentiles over the average? Reporting the 50th, 75th, 90th, 95th and 99th percentile is much better at describing the situation. Even better is to report a histogram or a kernel density plot of the the observed values.
Averages tend to be a notoriously bad indicator on their own because they don't describe anything about the distribution. This is obviously not a normal distribution, and even if it was, you need to report some kind of variance measurement as well. Otherwise it is hard to say anything about the shape of the distribution, let alone make tests for statistical significance.
Maybe if antirez doesn't want to read others simplistic responses on Twitter, he shouldn't make simplistic statements on Twitter. It's clearly a medium that doesn't convey nuance very well. There was a great deal of misinterpretation and talking past one another on all sides including his.
This was actually a continuation of a conversation that has been going on for several days, starting with an issue even more serious than 99th percentile latency - data loss due to a broken replica-repair strategy. Some very clueful people have pointed antirez at useful literature on the topic, and made specific suggestions to avoid the problem, only to be met with silence or excuses for why throwing away data was actually OK. Is it any wonder that they're frustrated with him, and ready to interpret an ambiguous statement like "the 99% percentile is bad" in a more negative way than he intended? For him to cherry-pick one exchange and present only one side only continues his own pattern of making constructive conversation almost impossible.
It's not a Twitter problem. It's a people problem.
Did you read the post? "the 99% percentile is bad" is something I never wrote, it was part of a larger sentence where it was obvious it meant if the 99th percentile figure is too big.
About pointing to relevant papers: things like pointing to timestamped replication paper, which is a CP system for replicated state machines, in response to the fact that Redis does not support slaves when the master disk persistence is turned off, is actually just another instance of why it is not possible to have decent tech conversations on Twitter.
The problem experienced at Stripe is a result of Redis replication documented behavior, regardless of using Sentinel or not. Redis replication is a very simple system where replicas will try to exactly mimic the master, and if the link breaks, will try to connect with it again and again forever. There is no builtin HA, nor failover or alike.
So before considering failover (and yet IMHO pointing to timestamped replication is not very informative even in this context, since in distributed systems the details matter, and you can't just give a random reference to a completely different system which happens to have just superficially similar issues to fix), there were different useful observations to do.
Like: Hey @antirez, what about supporting master-slaves setups where the master can be configured without persistence at all, and yet when it reboots, it will not be considered viable for reconnections? Which is the fundamental problem: stopping the basic Redis replication behavior from working as it works, with slaves that always want to replicate the current master, which is not ok if we want a system supporing the master restarted without persistence (which wipes the dataset on restart).
However the whole problem with that is that there are a lot of people like you that will regard linking to a paper as a great way to help, and as a very smart thing, while there are other that are instead trying to work to really make stuff better. Before commenting you should make the effort to understand exactly the problem domain and its subtleness: exact details or what you say is not relevant in a discussion which is all about details.
See my response to seiji. Exact mimicry might be a defined behavior, but where is it mandated who should mimic whom? Where had you said, before this, that the node with no data must be the one whose (empty) view should prevail?
As for your false dichotomy between reading the literature and getting stuff done, or your "make the effort to understand" ad hominem - go to hell. I've been doing this longer than you, I've been doing it better than you, and I've been writing about it as well. You make the effort to understand the problem before you shoot yourself in the foot yet again.
In Redis if you don't use any HA system like Sentinel, the map is fixed, it is an old-style replication system where there is the master IP address written in the configuration file.
Since the system is not supposed to lose the data on restarts, this is fine, but as soon as you want to support a different mode of operation with persistence-less masters, this must be modified, being Redis used with HA or not.
Now if you want to put Sentinel in the mix, the problem with this setup is not that the returning master gets promoted with a broken data set, but that if the restart is fast enough, the failure detection of Sentinel is not triggered at all, so the configuration remains the same. Just what is still, and was previous of the reboot, the current master of the system, restarted with a wiped data set.
In distributed systems this is the process not acting as specified, since Redis processes must reload their dataset on restart, otherwise they break everything, per design.
Now if it is a good idea to change this design: in the future yes, but so far we had not diskless replication, so for the replica to synchronize to write on disk was, anyway, needed, so why turn off persistence, and why to support it if the disk is needed anyway?
See? Arguments instead of random blabling and we can construct a reality or a model we both agree about. Then we can debate about what we think is right or not.
"it is an old-style replication system where there is the master IP address written in the configuration file."
"Old style" doesn't explain it. Even if the original master has unconditional priority when it returns, that does not preclude it gathering whatever data might still exist from the others. I've been working on replication systems since '92, and I can't recall seeing any that would make such a poor choice. Can you point to any, or is this really a "new style" idea?
"we can construct a reality or a model we both agree about."
I will concede that the behavior might be compliant with how the system was specified. I'm not 100% convinced yet, but at least - now - you've made a credible case for that.
"Then we can debate about what we think is right or not."
Not. The data-preserving mechanisms (e.g. view/epoch IDs) are so easy to implement in this case that leaving them out is unjustifiable. You even seem to be coming around to that view yourself when you say "in the future yes" but apparently you can't bring yourself to admit that it was always the right choice.
> Not. The data-preserving mechanisms (e.g. view/epoch IDs) are so easy to implement in this case that leaving them out is unjustifiable. You even seem to be coming around to that view yourself when you say "in the future yes" but apparently you can't bring yourself to admit that it was always the right choice.
It's up to you to form your opinion, but here are the facts:
1) Before diskless replication: even a master with persistence turned down, had to persist on disk, in order to support slaves.
2) Because of "1", it looked like futile to support this model of operations.
3) "1" is no longer true, I merged the diskless replication stuff just this morning.
Still I don't think you can form a fully informed idea unless you consider this: if you have persistence turned on in a setup which uses replication, like in most deployments using replication, you absolutely want the old behavior of Redis, of slaves reconnecting and replicating again on master restarts.
So when I say, in the future this could change, it is just as an opt-in option in order to support this new use case, not to say, the old behavior was crazy.
does not preclude it gathering whatever data might still exist from the others.
It absolutely does preclude that because it's not how the system works. You've described a multi-master system while comparing it against a replication-only system. Apples and racecars.
leaving them out is unjustifiable.
Feel free to submit a pull request fixing any and all deficiencies you've found. :)
No, I haven't, unless you'd say it was already a multi-master system because it allows non-masters to continue without the master being present. What I'm suggesting is just a master being smart enough to recover its own state from where it had been replicated before. How is that even controversial? In what possible use case is it preferable to discard readily available data without an explicit user request to do so?
"Feel free to submit a pull request fixing any and all deficiencies you've found."
Give me some reason to believe it won't be torpedoed by the next bad decision or failure of diligence that comes to light, and I might. Acknowledging that this needs to be fixed would help.
This whole little teapot tempest may be a people problem, but many of the issues brought up in the article - removal of context, selective re-tweets, general nastiness - are Twitter problems.
Everyone likes being right + feeling important.
Everyone is lazy.
No one wants to be called out on it.
If you are forced to keep it <140 char then no one can fault you for being lazy with your replies and critiques. Of course people are going to jump in and offer their brilliant opinions. Think about the low-cost high they can get.
This is a bit of a false dichotomy, I think. Of course there are good (and valid!) psychological explanations for the behavior patterns exhibited on Twitter.
When I say something is a "Twitter problem," I mean that these patterns are less prevalent on other platforms. I mean that Twitter has created a scaffold for communication that encourages certain bad behaviors.
One example: If I were to pluck the middle sentence from your response and criticize you for saying that "Everyone is lazy" here on HN, I'd be down voted into oblivion because I am obviously being a jackass. Twitter's structure can make it very hard to see when someone is being misquoted or their views misrepresented. It's the Fox News Soundbite version of online discussions.
I don't deny that Twitter constrains conversation in a way that can be negative, but is that what's going on in this case? Consider: antirez's blog post represents exactly the kind of one-sided un-nuanced polemic that he blames on Twitter, even though he has infinite space in his chosen medium to do better. What more could he have done to prove that the medium isn't the problem?
The problem is desire for control of the message. People who want that sort of control should just issue press releases. People who try to use the Twitter megaphone to promote their ideas, their projects, or themselves have to understand that others are doing exactly the same thing and sometimes the messages will conflict. The community into which antirez dropped this particular comment is one full of people running their own data-storage projects, academics promoting their own ideas, and others with more abstract (but no less passionate) beliefs about things like data protection or 99th percentile latency. I'm part of that community, and I've certainly had to endure pot shots against me or my project because of my presence on Twitter. It's part of the territory - just as it is on sites like this, or has been since forever on Usenet and BBSes and all the way back to the first town square. Among a thousand competing voices, yours might not be heard or understood perfectly.
data loss due to a broken replica-repair strategy.
That's the thing: Redis has no read repair strategy. Redis has a "be an exact copy of your master" strategy. It's not a secret. It's the exact design.
The complaints are like yelling at Linus when you rm -rf / your entire machine. Sure, it sucks, but it's a repercussion if your own actions, not a fault in the system. If you don't want to rm -rf / your machine, go use an OS designed for babies (cough ubuntu cough).
(Plus, there are already Redis improvements (designed within a day or two of the original problem being reported) to provide workarounds to users who _do_ want to run that exact use case. The answer to problems is solutions—not complaining and blaming endlessly.)
"the 99% percentile is bad" in a more negative way than he intended?
Text. It's only text. You can't read the intonation and people want to read absolutes. People want to read anger. Always anger. Always confrontation. It's possible the author of the text didn't mean to insult your mother. Breathe. It'll be okay.
The entire goal of the Internet is to get ALL THE ATTENTION YOURSELF. If anybody hates you online, it's because you got attention and they didn't. Nobody hates insignificant people. So, often times people with lower profiles/attention in conversations will try to increase their attention profile by arguing/hating the high-attention people.
When you say "be an exact copy of your master" is the strategy, you're kind of missing the problem here - who should be the master? In this case, Redis made a really dumb choice of which node should be master, and brought everything in sync by replicating emptiness instead of replicating data. There are very few use cases where that would be preferable. People have pointed out well known and fairly simple solutions to the selection problem, which are leading to the fixes and workarounds you've mentioned (though without the courtesy of acknowledging where the ideas came from).
Your analogy to "rm -rf /" is invalid, because that's well defined, well documented, and well known behavior. That's not true of Redis's autonomous and non-deterministic response to a failure (not to a user action). No spec or doc precluded choosing a different master and preserving data instead of discarding it. In the absence of such explicit guidance, preserving data should always be the default. How can it be user error when the user did nothing? Redis did the wrong thing because something was missed in its implementation, not because of any rational or deliberate choice.
you're kind of missing the problem here - who should be the master?
The choice of master is a static configuration set by the user.
Redis itself has no failover or promotion ability. There's an additional thing called Sentinel that can failover and promote individual Redis instances, but it is designed to recover complete instance failures (without immediately restarting), so a quick restart means no failover happens [an improvement to the "quick restart" scenario is showing up soon].
Redis made a really dumb choice of which node should be master,
(see previously; master is static, defined by the user)
pointed out well known and fairly simple solutions to the selection problem
(redis doesn't select things)
Also, this issue showed up last week. Last week. People are making it sound like this issue has been ignored for years. Nobody ran into this (and reported it) until recently. This use case is already being adapted into SOP Redis capabilities soon.
Try running into a big problem with any other DB and getting both attention and a concrete fix within two weeks. For free. The entire progress of the project has paused to address these immediate user issues.
because that's well defined, well documented, and well known behavior.
The Redis behavior is: always be a copy of a statically configured master. When the master has an empty dataset, all the replicas replicate an empty dataset. Pretty simple. :)
No spec or doc precluded choosing a different master and preserving data instead of discarding it.
Yup, specs and documentation did exactly that. Redis has no failover capability on its own.
preserving data should always be the default.
Ooops, you just re-invented the Mac trashcan.
How can it be user error when the user did nothing?
The user disabled persistence, enabled replication, restarted the process with zero data, then the replication recovered and stayed in sync with the newly zero-data master.
something was missed in its implementation, not because of any rational or deliberate choice.
nopers. more a lack of thinking it through from the user's point of view. an exact copy of nothing ends up being nothing.
"People are making it sound like this issue has been ignored for years."
Hasn't it been? How does leaving that latent in the system for years make things better? I rather think it reflects on an inability to reason about failure modes (including user failure modes), and deal with them proactively instead of after data was lost.
This isn't about about omniscience (not omnipotence BTW). This is about a far lower standard of basic diligence, expected and met by most people who work on data-storage systems. If you're given some data to store, and there's an obvious way to retain/recover that data despite and intervening failure, then failing to do that is a betrayal of the most basic trust people put in data-storage systems. Congratulations, you've implemented the distributed-system equivalent of linking fsck to mkfs. Well done. Go pat yourself on the back for conforming to your specification.
I don't think you're understanding redis or this problem correctly.
Redis lets you have slaves which mirror the master. Hundreds of thousands of redis installations use this pattern to provide read scaling and offline master-loss persistence, and in the normal case, this works great. I myself have implemented systems with hundreds of redis instances which have gracefully survived the loss of the primary.
In this particular instance, the user turned off persistence, didn't understand the ramifications, and then brought the master back up with an empty database after a hard kill without thinking things through.
Fortunately, the user was savvy enough to have kept backups off the slaves, as is the usual pattern, and so was able to continue service.
This is not a normal pattern and goes against the general practice.
I understand what you're saying, but I don't think it's a sufficient reason to throw away data. I've seen hundreds of cases where a GlusterFS user went against our advice and did something that ended up making things worse. Sometimes they even lost data. Of course, they always blame us. I'm pretty sure people who have worked on every single data-storage system ever have had similar experiences. Sometimes the user is just wrong and it's their own fault. Sometimes they're right because we made it far easier for them to make things worse than to make things better. In those cases we have to stop making excuses like "user error" or "RTFM" or "against general practice" or whatever. We need to help the user by not handing them bags of explosives. Which do you think is a better choice here?
* Default to preserving already-replicated data, provide "clean start" as an option.
* Default to throwing away data, maybe-someday implement an option to use data that's already present in the system.
Blaming the user won't prevent another user from making the same mistake with the same result. Saner defaults, and an implementation to support them, will. Who's going to complain that you saved too much of their data?
The defaults are sane, and in fact the user here had to explicitly turn them off in order to do the thing they wanted to do. Once you reach into a configuration file and change a setting, I can't think of a software system in the world that protects you from your choice. Could you maybe name a few?
The user turned off persistence. There's no reason for a normal person to suppose that also means ignoring data that's in the system when the master comes up. The fact that the two are inextricably tied to one another in the Redis implementation is not the user's mistake.
What do you think a slave should do if it is told to replace its state with empty state? How about half-empty state? There's really no answer that's satisfying for every possible use case (certainly I don't want my slaves to refuse if I tell them to clear the database completely on purpose). And indeed you haven't given any examples of databases that try to do 'better'. I think that's because there aren't any.
If you do get roped into a conversation where you need more tweets to explain, here's a few simple guidelines:
- Don't split sentences over tweets. People will only read one part.
- If you really must, use "(cont.)" on both tweets (end of the first and start of the second), so it's obvious there was another part. I know that reduces your character limit, but it's worth it to be less ambiguous
- Make each continued tweet a reply to its predecessor. You can reply to your own tweets. It facilitates following the conversation on most clients.
- If it's more than 2 tweets, consider using twitlonger. In general, that should be a last resort, though.
Without trying to make it sound high-handed, there's something of an art to smooth twitter conversations. Terseness is the key. I find them rewarding because, when done right, they have an enormous density of information in a short time.