Wow, it was quite the surprise to wake up to seeing this post near the top of HN! I wrote the post, happy to answer questions if anyone is wondering about any details.
The example at the end made me wonder if Apple's model is actually better than GPT2 for text prediction. It generated garbage, but all that garbage made somewhat sense in the context of only the word "Today".
Whereas GPT2 hallucinated random stuff about the US government. A text prediction model should predict what the user wanted to type, so if you evaluate the models based on that, GPT2 actually performed horribly, since the user showed zero intent in talking about the US.
The example at the end sounds just like the predictions you get from normal phone keyboards in the last couple of years, which presumably don't use a modern GPT-style language model. A bit disappointing.
Seriously disappointing. I was expecting that it would not produce total gibberish. It acts like it's a Markov chain, and only considers the last 1-2 words. Identical to the currently-shipping thing that we've had for the past however-many years.
People trying to draw this comparison proves making good products is harder than it seems...
The default goal everyone is assuming is spitting out the longest correct sequence possible.
But in reality the mental cost of a wildly wrong prediction is much higher than the mental cost of a slightly wrong one, so what you'd train the model for is sequences of a few words at most being with higher confidence.
Most people can/will tune out slightly wrong words especially as they get a feel for what autocorrect is good and bad at.
If you unleash the full range of tokens GPT 2 can normally output, you'll constantly be blasting out words they didn't expect.
—
The fact your long sequence prediction got better doesn't matter because the UI is autocomplete not "auto-write": they're still expecting to drive, and a smart but noisy copilot is worse than a dumb but lazy one in that case.
I wouldn't be surprised if they trained the model to an effective context window of just a few hundred tokens with that in mind
GPT-2 saw "today," and thought "this must be news copy" and generated more news copy. Given a few more words, it could have narrowed down the context. The Apple suggestions aren't even grammatically correct, seemingly no different from the shallow statistical completion we've already had for years, so it's weird that they branded it in lofty AI terms
Or it could be the contrary. The new feature doesn't suggest whole sentences because the model they are using produces gibberish. It is quite possible that if the model was better then the would allow it to suggest longer phrases.
I suspect someone (Craig even) was under some pressure from The Board to have >0 references to generative-AI in their presentation this year since every single company (even non-software) is now expected by Wall St to "be doing some AI". Even though Apple is at the top of the heap with ML in photography and many other domains, without some kind of LLM the tech news narrative will be "Apple is years behind".
It seems obvious to me that it's not, because if you asked a human to guess what comes after "today" in a text, they'd never say "probably some gibberish about a day a day".
Garbage in, garbage out? The preceding text is gibberish, so the prediction will be worse. Presumably they also only show completions with a much higher confidence threshold.
Maybe: "Today was fine. Since I've retired, I'm taking my life a day a day".
Or maybe I wanted to express myself in the timeless words of the poets:
"A day, a day of glory! A day that ends our woe! A day that tells of triumph. Against our vanquished foe!"
"Rose is a rose is a rose is a rose. Loveliness extreme.
Extra gaiters. Loveliness extreme."
"A-well now, everybody's heard about the bird, everybody's heard about the bird, About the bird, the bird, bird bird bird, Haven't you heard about the bird? Don't you know that the bird's the word?"
I don’t think you can make a fair comparison like this. The examples at the end are essentially praising GPT-2 for hallucinating - is it better when it’s suggesting completely irrelevant text to your sentences? Apple’s approach can’t generate full sentences on its own, but that’s not the goal anyway.
And interesting test. I didn’t see a mention of the temperature setting used. Temperature controls the probability to pick a token that isn’t the top prediction, which leads to more creative/less robotic results.
For actual input prediction, you probably want the temperature to be zero. But even a model as good as GPT-3 becomes very boring and repetitive with those settings.
I used greedy sampling (temperature 0) for all of them. Since I didn't have access to logits/probabilities for Apple's model, I wasn't able to do anything else in a way that would be fair.
There is an input for temperature in the CPU model. If you can find and hook the call to predict you can probably see what is being passed.
Interestingly the Neural Engine version of the model does not take a temperature input, but it does output the raw embeddings in addition to the logits.
I've been looking at these files too and have another data point for unilm.bundle being the new text prediction.
If you take an iOS simulator, turn off "Settings > General > Keyboard > Predictive", reboot it and then watch the console logs as you turn that switch back on, you'll see the "kbd" process load the models out of that bundle.
Beyond fixing iPhone’s autocorrect as widely mentioned below, I wonder if in future versions Apple will end up switching to phi-1.5 or other models that are much smaller but trained on higher quality data.
Would also be cool if they trained their own copilot for Xcode, given their obviously enturmes code base in Swift and Obj-C.
What's "high quality" refer to here? The amount of txt spk I use on my iphone keyboard, vs words that are in the dictionary, is heavily skewed in the direction of short words that a corpus consisting of English text is likely to omit.
My hypothesis is that this question, "What's 'high quality' refer to here?", is a big challenge and the answer depends upon the audience. Here are a few examples:
- kids use one level of slang
- older ages groups using another (and it might mean something quite different)
- communications for professional
wouldn't use slang and would use formal messages
- different cultures/regions will use different phrases for the same thing (and some colloquialisms are quite sophisticated like Cockney rhyming slang)
- on twitter when out of space you might then shorten words
The thing that really 'riles me up' / 'get on my wick' / 'annoys me' / 'is immensely frustrating' / 'is an opportunity for a sales feature' [trying to demonstrate the different phrasing depending upon audience] is when the auto correct 'fixes' grammar incorrectly. It needs to be clever enough to realise the correct use of there/their/they're and its/it is/it's.
You use text speak on an iPhone? How? I find it impossible due to autocorrect. Either it will correct with the full word or it will correct with a wrong word making it more efficient to just type in full.
My problem with this (I’ve tried it) is that the iOS keyboard still seems to try to guess which key you meant to type. So you hit “o” and it thinks you probably meant “p” and inserts that. Last I checked this still occurred with autocorrect off and made it even worse than using autocorrect.
I’ve had autocorrect disabled on my iPhone for years—for whatever reason it really bothers me when something I just typed magically changes into something I didn’t want to write. I like that what I type is exactly what appears, not the computer’s guess at my intent. It’s one of the first things I disable when setting up a new Mac as well.
It took a few months to get used to it, but I developed typo-correction muscle memory pretty quickly (RIP 3D Touch, it was awesome for text editing). Plus I think that the copyedit re-read that I give to things I write on my phone is helpful above and beyond just catching typos. So be it if I make the occasional typo!
I'm one of the "lucky few" that (a) really needs autocorrect because I can never seem to type correctly in mobile phone keyboards and (b) really hates autocorrect because often it either gets me wrong or worse, changes legitimate words into awkward and incorrect words - like the other day I typed something pretty mundane that got changed into a more... vulgar word.
I got a little reprieve with swiping text, but still, autocorrect should be much better.
In general so that what you write is not mutilated by software that doesn't know the words you use. If there are typos, it's not difficult for the receiver to "autocorrect" while reading, but when autocorrect miscorrects, it's not easy for the writer to notice how the message changed nor for the reader to guess what was there before the "fixes".
That assumes your typing and spelling accuracy is high and the other person reading has significant proficiency in your language.
My typing and spelling accuracy is horrendous to the point I use autocorrect on my laptop. For me, it’s an important accessibility feature. I also use dictation for single words.
I've been arguing this is the way AI should be deployed. Rather than trying to sell ai as an end to end solution, just let it do the small part that it can reliably do. It's cost effective for the host, and valuable for the user. win win engineering!
I mean, I get why all these startups are trying to sell AI as a panacea. It’s an exciting technology and someone has to figure out its limits.
That said, as a user, it is nice to see LLMs used in a small, discrete, undeniably useful way like this. No flashy promises, nothing new for me to learn- it’s just autocorrect, but better.
It makes a lot more economical sense to cloud-host big models like GPT-4 and optimize them for the hardware they run on. But for small models, sure, let the user run them locally, eliminating network latency.
It already does and I hate it. If a person you’re speaking with misspells a word, it’ll happy suggest^w forcibly autocorrect a similar word to that misspelling if it was recent enough.
I think tpowell was asking about the all-new autocorrect system. Do you have information on how that works, or were you just talking about the old system that is being deleted on Monday?
I have no evidence or reason to believe that the inputs to the corpus of valid word sources for the new system changed - most likely only the prediction model did. This misfeature only recently shipped (iOS 16, I think) and I’m sure Cupertino views it as a net gain.
Do they do that across conversations, eg if I misspell something while texting you does it recommend my misspelling when you’re talking to other people?
Yes. Ever since I went to the Maldives it wants to capitalize "male" and I have to work quite hard to get it to not correct "Yes" to "Yea", and "Thailand" to "Thailnd". Hate it.
We might ask ourselves at some point: if the point of simple text prediction is to speed up text entry, and not generate large amounts of text from scratch, then that's a sign the input interfaces are the bottleneck. If we had a way to get text from our brains to the computer faster, we wouldn't need this kind of prediction.
Does anyone know if the regular spell checker has been improved. I'm terrible at spelling and I find Apple's to be the worst, for example in the link I've shown what happens when I type "nessisary" [1] (lastest iOS 16) which is a word I often get wrong and have to Google. I feel as though this is a very clear and obvious typeo that should be picked up. If anyone is wondering what the word I'm trying to spell is, it's - necessary. Google picked this up first time.
I can only hope so. Apple’s spell checker is terrible.
My word is “bureaucrat”. No matter how many times I look it up, I can never remember how to spell it.
I always try some variation of “beurocrat”, find that spell-check is of no use whatsoever, paste it into Google, and get the proper spelling instantly.
UnilmCtrl seems to imply some dependence on Sochers CTRL models, but I wouldn't be surprised if it's just another name collision. Would be interesting if it were, as that would likely mean they have been working on releasing this for a few more years than the "chat-gpt craze". I am a bit biased, but I always have more respect if someone has worked in the NLP field for at least ~8 years, rather than jumping on after Sparrow and RLHF.
Even large language models with billions of parameters get caught in the cycle. You don't usually see it exposed to users because there are sampling tricks applied, such as repetition/frequency penalty.
That’s where you’re wrong. Raw LLM very often get stuck in loops, usually extremely small loops. There’s a big chunk of infrastructure that exists on the output end of any production LLM that exists explicitly for the purpose of preventing loops.
That post-processing infrastructure can use all kinds of mechanisms to prevent loops and induce more useful output. With the most basic simply systems simply refusing to select any output token that already in the input, to more complex stochastic process that explore the tree of possible outputs, to find branches whose overall result is improved by choosing less optimal immediate steps.
The vast majority of what make Chat GPT different to simpler GPT-3 models is this complex post-processing phase that allows designers to push and pull on the behaviour of the pre-baked static model underlying the chat interface.
Agreed 100%, and some of the "simple" approaches like repetition penalty, can harm output, because for example markdown tables repeat a lot of characters, and if you just blindly apply repeat penalty, it will just stop doing what you've asked for.
My guess is that they have some kind of state-aware sampler, and they know if they are in the table, etc. Because then you can sample in a much better way. Just like with grammars, but grammar itself is probably not enough.
Sampling and tokenization are IMHO the biggest open problems. We have something which kind of works but it's nowhere close to be perfect.
It is very common for small models, and sometimes even 7B model gets stuck.
There was an article/paper showing that GPTs (whole family) quickly get confident in looping if there is anything loop-like in the window. So basically, as soon as it loops once, it will never get out of it.
Note that sometimes loops are desired, like with docstring in the code, always starting before the function definition, and other structural things.
Such penalties would only exist for fine-tuned models. Base models have only the temperature setting. As the example at the end shows, even GPT-2 seems "smarter" than the Apple model, probably because of the number of parameters.
There's no such thing as "base models have only the temperature setting". Models do not have sampling settings (temperature, repetition penalty, etc), the sampling code does, which obviously you can use on any model.
To summarize how they work: you keep some number of previously generated tokens, and once you get logits that you want to sample a new token from, you find the logits for existing tokens and multiply them by a penalty, thus lowering the probability of the corresponding tokens.
To clarify: I meant that in general they are commonly applied, but in this case they weren't as the author confirmed. The repetition, of course, doesn't happen all the time.
Yes, you're right, I should have mentioned it in the post, but I used pure greedy sampling for the GPT-2 outputs since I couldn't do anything but that for the Apple model. So temperature was set to zero, and there was no repetition penalty.
iPhone’s auto complete and form fills drive me absolutely mad. It’s honestly just basic stuff that should have just worked a decade ago. I’m not even begging them for LLMs or some fancy ML, just learn basic words I type out all the time. For example my full email address? how about my last name? This seems like basic stuff
That doesn’t match my experience at all. I agree with you that it should “just work” and you shouldn’t have to think about it, but do you have a contact for yourself saved, and have you set it as your “me” contact in Settings>Contacts>My Info?
Yes I do. But also, my wife has a different name, do I have to type out her full name every time? There is no stored learning that I have experienced whatsoever on normal typing. The little contact auto fill that you both are referring to is extremely limited, the form field literally needs to say “email” and doesn’t work out of context. As I said, did you try to type your email out in this comment response box?
I don't want better predictive text. I need better autocorrect. Something happened about 6 years ago where the quality of the autocorrect fell off the roof, and it's been absolutely terrible since then. I spend too much of my time fighting with either mispelling, flipping to the wrong word even though I spelled the word properly, etc. It has made typing on my iPhone an unpleasant experience and I need it to change. That's just about the only thing better between the iPhone and Android for me.
I've had a similar experience with autocorrect on Android across multiple phones (lg, samsung) and keyboards (gboard, swiftkey) around the same time, I'm curious to know whether it's just my own bias being validated or something happened. All predictions have become completely stupid (switching languages mid sentence when they were robust in the past, finding least consistent suggestion first, suggesting very random terms I've never seen like "Gaylene" when I swiped to write "happened", etc).
I can vouch for this from personal experience. Back when I had my Nexus 4, the built-in keyboard surpassed the current Gboard by a long shot. I'm not sure what happened along the way, but the spell-checking capabilities of the completely offline and lightweight keyboard that Google shipped back in those days were miles ahead.
Same here. I get all sorts of nonsensical suggestions when swiping now that I never used to get. It changed a few years pre-covid, so yeah, 5-6 years ago. I assumed it was stunting to do with a list lawsuit against samsung or something
Word (or multi-word) prediction is a great starting place for an autocorrect model. If the keyboard I'm using knows the probability of all the possible next tokens I could type, then you can start making the tap targets for those keys bigger. And since the model is super cheap to run, you can replay the last n keyboard taps to simulate a different correction being made. That's at least how I'd make autocorrect better if I worked on the keyboard team.
Anecdotally, I've heard autocorrect is much better on iOS 17. But I won't update until the general release next week.
Maybe it’s better and I just need to keep typing so it can figure out what I mean.
In the last couple of weeks using the beta I’ve found myself fighting it more than ever. After years of typing by tapping at the screen, I’ve switched to swiping the words in because it seems to be more reliable.
Totally agree. I love my iPhone but autocorrect is absolutely the most painful and most infuriating part of the experience. I am constantly fighting with the autocorrect… sometimes repeating the same word 3 times because I’ve typed it correctly but iPhone really wants to change it.
Same with proper nouns. If I deliberately start a word with a capital letter it’s probably a proper noun and I know how it’s spelled, don’t correct me.
Fix this shit, Apple, it’s the one part of iPhone I really hate.
I regularly use 4 languages on my iPhone and my autocorrect is completely broken. I can’t be bothered switching between keyboards, so the English keyboard has just completely lost the plot…
Yup same, that's why I am sticking with Android as well.
I want multi-language auto-correct for my preferred languages _on the English QWERTY keyboard_.
Thank you for mentioning this! I thought Apple had eliminated this editing mode when they got rid of Force Touch. So happy to learn that it’s still available but a little frustrated I’m only learning this now.
This made me realize I was conflating these in my head, and I wholeheartedly agree. I had a 12 Pro Max 256GB until last year, and autocorrect was great/fine/never noticed issues. Fed up with heavy phones, I purchased a 14 Plus 512GB (very similar physical size) and set it up from scratch while migrating whatever is included in iCloud (Photos, iMessage etc). Autocorrect has been bad from the start on the 14 Plus, and has improved little in nearly a year. Typing feels tedious now, where it felt like a strength previously. The phone as been a joy to use otherwise.
For me stock autocorrect was always pretty much unusable, but I have no real complaints about the app I use. I'm bilingual and I type in both languages a lot. I remember when I got my first smartphone I bought SwiftKey app (later bought by Microsoft) that made language switching very easy by just swiping the space. Later the app got so good I just left it on default setting and it would still recommend the right words. Since I bought the app it has been always one if first things I installed on a new android phone (I think I had it on an iPhone 3g back in the day too, but I can't remember for sure).
I recently realised how much of a difference it makes when you type quickly as I got an extra work phone with stock android I have to use occasionally.
There is also a feature of this app I use only occasionally, but when I do its great. It let's you type by moving your finger between letters in one continously motion. When one describes it seems rather weird, but it's really one of the best features of the app. You can write really fast this way, the only difficulty it has is with very uncommon words.
I also use SwiftKey and can highly recommend. I use Polish and English, sometimes within the same message and native keyboard was an absolute shitshow for me. It would literally never correct properly, sometimes make up non existing words etc. My only pain point with SwiftKey is that since last update it crashes often, I hope they might actually fix it someday. GBoard on iOS was forgotten long time ago so that's a no go...
Same with voice to text translation. It's absolutely, utterly, worthless. I end up having to go and edit half the message. Apple can't even get basic things right
Fairly neutral British. Born and raised in Guernsey so have a complete absence of regional accent without the very forced enunciation that comes from elocution lessons / Queens English. Admittedly I probably represent the best-case subject.
> Admittedly I probably represent the best-case subject.
I don't think you could do any better unless you were from Cupertino. With an Irish accent, my experience has been... okay with Apples text to speech, it's certainly better than others, but I still have to make a conscious effort to enunciate quite differently to my normal speech.
OpenAI's whisper has really impressed me though, and transcribes almost everything perfectly, even if I throw in phrases or words from other languages part way through the conversation.
I use voice to text all the time on my iPhone and find it to be astonishingly good. I’m a 50 year old man who grew up in Southwestern Ontario so that gives you an idea of my accent.
Look at the positive side, it made you type less on your phone (strangely enough, I had to fight with my SE’s autocorrect three times in order to input “positive”)
Because it depends on the person. For me autocorrect is 99% spot on and so it’s a valid trade off for me to deal with correcting the remaining 1%. My wife on the other hand cannot live with it so she turned it off. And every time I type something on her phone I immediately feel slower because I know each of my key presses have to be much more deliberate.
In recent years mine has gone completely crazy and started autocorrecting to random words that don't exist. I think somehow over time it can get corrupted.
IMHO people don’t often enough think through what the Google search type-ahead would be if it had 100s of millis per BPE token and a demanding cost structure.
No one credible, least of all a guy like me who had been an enthusiastic amateur for a few years, is saying the stuff hadn’t gotten pretty wild recently.
It does seem absurd that the options are "completely broken keyboard features" and "literal keylogger". Meanwhile we're paying for many transistors for NN engines and getting worse results than if they were never leveraged at all.
Meanwhile my language and dozens of others with hundreds of millions of speakers still don’t get the predictions at all, don’t support multi-language typing and don’t support swipe typing. Typing on a $1000 iPhone in one of these languages is still a 2007 experience.
They do this for cycling directions in Apple Maps too.
Come on Apple, you KNOW you don’t support cycling directions in Ireland - stop getting my hopes up every time I ask for directions. Just only show the button when you’re in a geo that supports the feature.
Did you enable the Czech-English dictionary under Settings -> General -> Dictionary?
You can also add Czech as a language under Settings -> General -> Language & Region -> Add Language. This will let you switch your keyboard to Czech when typing and enable the language on websites and stuff. You can also switch the whole phone to Czech if you like, just by moving Czech to the top of the list of preferred languages.
The thing I find most interning is that typing (in English) on the microscopic Apple Watch keyboard seems to have better results than typing on my phone keyboard. I don’t know what the watch keyboard does differently but it seems to read my brain while my phone does the opposite.
Czech has around 10M speakers world wide. It sucks that they don't have solid predictions, but there are languages with massively more linguistic footprint that will end up prioritized.
Without putting too fine a point on it, I can tell you from direct experience that while it might seem that purely statistical methods are sufficient for predictive text, one of the problems is that you're building those models (whether using neural nets or pure stat models) from a body of text that is fraught with peril and the details matter.
There's really basic stuff like profanity and other offensive constructs (see all the jokes about google suggestion auto-complete where sexist, racist, and otherwise offensive suggestions get tons of attention), then there's the problem of source languages often having a clear shift in acceptable completions for entirely different reasons.
The euphemism treadmill in US English are an obvious and continuing expense, but consider whether USSR-era Czech texts, which are substantial, would be a safe bet in training a completion system.
Modern language processing relies on pretty large corpora, and to bootstrap it requires a part to be of high quality and annotated, although for spelling correction you don't need annotation. Or you can go the GPT-3 route and use really huge corpora. If that doesn't exist for your language, and for many smaller languages it doesn't, it won't get better than 2007.
Indonesian is spoken by 200M people (45M of which as first language), I bet they could find some text for it. When joined with the Malaysian family, it brings the total speakers to 290M.
Building corpora has been an ongoing research activity since the 1990s. Many countries lacked the academic programs and support to build them. Often, the material isn't even there. E.g., wikipedia is a great source, but many "national" wikis are rather empty. The Bahasa wiki has 1/10th of the articles of the English one. They also seem to be shorter.
But also, Apple doesn't do much research. MS, however, has always had an active NLP research and support program, if only because of the importance of Word. But Apple depends on the research available from third parties.
My understanding is that the languages with better support have the advantage of official bodies that have section-by-section translations of all their documents; the EU into its 24 official languages and the UN with its six, for starters.
I'm not sure about that. It doesn't require a huge corpus to learn the structural features of a language. If you have that plus an English dictionary for the language, an English language model should transfer pretty well to any language.
I’m an adult and I bought an iPhone knowing there are pros and cons. Some cons are more absurd than others and this is one of them. Otherwise I’m very happy with my Mini.
Everyone's asking tech details and "how", but I wonder about the "why". Do we want LLMs to always write for us, or whisper in our ear what to say? By design LLMs tend toward the most commonplace, mainstream ideas and ways of saying things. They're not much for originality or human idiosyncracy. Are we engineering a bland world full of pablum?
Because at work I’m typing the same bland things all the time in documents and communications. I appreciate stuff like the predictive word stuff in Google Docs. It’s helpful because business language is expected to be normalized and boring.
On the other side of that token, the average language abilities of the average American office worker are pretty low so I’m assuming they view this as an enhanced AutoCorrect and they appreciate it because it makes them look less dumb.
I agree with your point though. And to answer your last question, unfortunately I think the answer is “yes”.
> Because at work I’m typing the same bland things all the time in documents and communications.
This problem can be solved without text prediction by building a personal knowledge base with hyperlinking[0] and backlinking[1] for discovery, and transclusion[2] for automating writing the same bland things. How I org in 2023 by Nick Anderson[3] goes over a great workflow for this. The advantage of this is that all of the words that you share are actually words that you wrote, instead of sharing words that Google suggested you share.
I use macros for frequently-reused messaging but I meant business communication itself. It’s essentially a different dialect of English meant to be “inclusive”, unambiguous, and not prone to misinterpretation.
Anyway, I like and use the feature daily at work. So do lots and lots of other people.
On the other side of that token, the average language abilities of the average American office worker are pretty low so I’m assuming they view this as an enhanced AutoCorrect and they appreciate it because it makes them look less dumb.
I would also assume this to be largely true, mainly because of how language in media has gone from being formal and informative to casual and less expressive.
I agree there has been a style shift, I don't think writing has gotten any less informative. I personally don't like formal writing, but either way it has no bearing on whether or not ideas can be gotten across effectively. Substance over style!
I might say it a bit differently. The amazon writing style is to avoid flowery prose, weasel words (maybe, perhaps, etc), give precise dates, use data, etc. I'd love a model that I could hand to engineers to help them write in this style.
It's not LLM's writing for us, it's just autocomplete.
If the suggestion doesn't match what you were already planning on saying, you just ignore it.
The human desire to be original and authentic is always going to be stronger.
(It's much less effort to ignore it and keep typing your original thought, than it is to think about it, compare with what you were going to say, decide its version is better, and then accept it.)
1) there is a sort order to them. And we don't know how the 'recommendations' work. What is a recommendation? Why is Google recommending something over something else?
2) it bogs down creativity. You end up not thinking and accepting the suggestion as 'good' enough.
My job requires me to correspond with dozens of people over email, Teams, and Slack every day. We're all trying to get work done and need our communications to be as succinct as possible. Sure, occasionally I might dress it up to add some humor, but that's ~1% of cases. An AI, with access to my entire corpus of work-related communication, could likely very easily predict most of my communications, since they fall into a small set of categories.
"When can I expect to get $workproduct?"
"Here's when I can commit to getting you $workproduct"
"What's the estimated date for $milestone?"
"Here's the project plan for $initiative"
"I can't make this meeting, can you be sure to record it?"
I welcome any tool that can predict what I want to type and does it for me. I'm not sure if it's my imagination or not but Outlook and Teams seem to have gotten better in this regard. I'll take more of that.
Most writing does not need creativity. It is wrote communication. If you want to put some creativity into something or write something where you want to write with some personality, then turn off the predictions or use it just for spell check.
> The human desire to be original and authentic is always going to be stronger.
Authenticity didn't start to become a thing people cared about until about the early 1990s, and it didn't blow up and take over the mainstream culture until the 2000s and 2010s.
Prior to the 1990s, there was a lot of interest in professionalism. People spoke and wrote in overly formal, jargony ways that they perceived as being a marker for competence in some specialized professional domain. Being too honest/authentic would have been seen as unsophisticated and lower class.
Just pointing out that what people strive to emulate can change over time. Once LLM writing becomes commonplace, it will probably become trendy to write in the exact opposite way that an LLM does!
> Authenticity didn't start to become a thing people cared about until about the early 1990s…
In your lifetime, but…
Another common theme in the philosophy of both Dostoevsky and Descartes is the idea of authenticity. Dostoevsky believed that people must live authentically, in accordance with their true selves, and that this is essential for their happiness and well-being. Descartes, on the other hand, argued that the path to knowledge and certainty begins with the rejection of all previous beliefs and the adoption of a completely new and authentic perspective.¹
And of course, the U.S. counterculture movement of the 1960s was deeply preoccupied with authenticity:
You note three themes underlying the American hippie experience: authenticity, individualism and community. Why did these concepts stand out, and what did they mean in the context of the hippie/counterculture movement? / W.R.: Although hippies often disagreed about beliefs and practices, they shared a desire to be authentic. Members of the counterculture condemned mainstream society for being conformist, rule-driven and uptight. Authenticity meant “doing your own thing.” Because freaks distrusted both society and government, individual decisions were applauded as the most authentic.²
Well it's not so much about deliberately / affectedly being original and weird (which is annoying), but just leaving some space for natural idiosyncratic ways of writing.
Yes, but sometimes that's what's called for. I think I may have found my answer to where LLMs might be useful for short communications: softening something that might be interpreted as "curt" or even "rude" while not really changing the message.
As someone with Dyslexia, who struggles a lot with written communication, and frequently finds myself fighting with spellcheckers, trying to get them to provide the correct correction. I’m very excited for these types of autocomplete systems. They’re like spellcheckers, except they also use context to produce better results, and frequently remove the need to play “guess the right misspelling” to get a normal spellcheck to provide good suggestions.
I’ll accept the risk of blandness, if it means that written communication finally becomes “easy” for me to participate in.
Maybe not always (I assume an annoyed person could turn this feature off), but I think the general trend is, yes, especially for boring rote writing we have to get done for work or school that doesn’t require a tremendous amount of creativity.
I really look forward to using GPT to help me throw together RFCs, documentation, announcement letters, daily standup write ups and other artifacts like that that prevent me from getting actual work done.
One approach that I've been using with a local LLM, mostly brainstorming for my friends and my dungeons and dragons sessions, is to set up a prompt to be completed with some certain detail of storyline, character background, etc., Then running that prompt maybe 50 or 100 times and keeping informal statistics on the different results that come out. In such an application, it can be used as much for inspiration of how not to sound bland, commonplace, or derivative. You can pick the one in 100 oddity that strikes you as interesting, or simply make sure not to use any of the outcomes that the LLM came up with.
I find they're useful as a tool for achieving a particular structure or tone that isn't "my voice," including populating boilerplate from bullet points. Sometimes I'll go back and revisit everything they've produced, but they let me put something tolerable in place early. This lets me focus on the meaty parts of the text sooner than I'd otherwise find myself capable of.
I personally don't find them useful for quick / short / informal communication like email, or at least not yet.
"Bing Chat's" implementation already allows you to select more creative generation of text. It's just a radio button option. There are also different technical solutions for the LLM to select which word to generate that allow either for more interesting, or more predictable, words.
This isn't to say a human element doesn't have a ton to offer! Just to say that we aren't necessarily engineering a bland world of pablum, either.
That's just temperature, which evens out the random probability a little of the N most probable next words. It's still vastly favoring the N most common ones based on the training corpus, and will have a hard time producing uncommon ones.
E.g. try asking an LLM to name a real, non-famous person. The internet and it's training corpus is full of regular people, but you won't have much luck - they're statistically too uncommon to remember.
I was assuming those Bing Chat settings are temperature-based, those types of "creative/precise" controls usually are - but perhaps there's more to it.
Early versions of GPT would tell me about myself, but July of this year it was saying that it could not comment on a private individual. I'm not at all famous, but there's plenty of writing by and about me in the common training datasets.
I would be skeptical of the results of that experiment, just because I assume the minders of the big LLMs have attempted to make it deeply uncomfortable with discussing anything that might be "personal." For a fun time, ask ChatGPT who was executed (as in capital punishment) in the US in a certain long-past year. It responds with a bunch of rubbish about privacy and about how an LLM can't be completely sure about stuff so it wouldn't be ok to speculate for fear of tarnishing someone's reputation, as though the person executed 15 years ago is going to sue OpenAI for sharing their name and what crime they were publicly convicted of and killed for.
> Are we engineering a bland world full of pablum?
Like any other technological advancement, we're freeing ourselves from the burden of wasting mental energy manually doing those commonplace unoriginal tasks which can be easily automated, so there's more bandwidth left for human beings to focus specifically on the things which can't be
I disabled Gmail Smart Compose for this reason; I felt like it was putting words into my mouth suggesting entire sentences for the email.
I'm much more open to using transformers as a better auto-correct, where it's one word at a time and uses the first letter as a filter. Especially on a tiny phone keyboard on the go.
If I’m writing an email to my coworkers, what I really care about is the information I’m getting across and not really the presentation of it. If LLM autocomplete can speed up writing a totally functional email then how bland it is really isn’t at the top of my mind.
Just give me the option of not having autocorrect change the previous word when I am typing a new one. I don’t move on to the next word unless the current one was correct or I have already corrected it. Why does it assume I’m stupid? It even does this when I’ve picked the completion for the prior word or swiped to type and then made manual corrections. It’s painfully obvious that it shouldn’t be touched again.
In typical Apple fashion, when everyone's going bigger and stronger, they're going in a different direction... With optimising to the smallest model that can run all day without draining your battery.
I love that they're almost never first to market but they find a way to distil value than others don't. It's the almond milk of technology.
Apple has just been more methodical to the rest of the ecosystem - essentially waiting to understand use-cases before fully embracing it across their ecosystem from Apple Silicon in Mac with neural and `device=mps`, CoreML, and now more-or-less full force with their ML studio now in Xcode.
Many apps didn't wait for this and Snapchat, as one example, has been successfully taking advantage of Apple Neural since the early early days.
There are many reasons Apple has been the world's most valuable company for over a decade. One certainly is that they have a prescient ability to predict where the market is going, even if they have to use their ever-building position to drag things that way. As demonstrated by the facts and references I provided above, not to mention the iPhone essentially re-defining what a smartphone is way back when Blackberry was the current market leader.
> Apple put neural in their silicon a full four years before Google did:
You're confusing when Google started designing its own mobile SOC with when Android devices (including Google's) first started using neural network accelerators, which happened months earlier on Android than on iPhones.
> You're confusing when Google started designing its own mobile SOC with when Android devices (including Google's) first started using neural network accelerators, which happened months earlier on Android than on iPhones.
Apple (famously) doesn't announce until new functionality/devices are actually available. When they announce real people in the real world get it within a week if not less.
A press release from Qualcomm that came a few months before Apple actually got it in people's hands only further demonstrates that Apple was working on this long before Qualcomm included it in the chipsets that lag on the market from Android device manufacturers.
> As for GP's claim, Android has indeed had what the iPhone is now getting for two years
You meant to say that Pixel had a variant at that time with Gboard. I'm not going to bother to do the research on Gboard on iOS but I suspect it was also available on iOS with Gboard at the same time or soon thereafter. If not that's on the Gboard team at Google for not supporting neural on iOS and Apple hardware that came far, far sooner.
On a somewhat-negative Apple take they are infamous for being, ummm, "inspired by" successful apps and add-ons in terms of what makes it to iOS. I wouldn't be surprised in the least if Gboard vs built-in iOS keyboard is another one of these cases.
> A press release from Qualcomm that came a few months before Apple actually got it in people's hands
This press release was for a new Tensorflow release for devices that had been shipping for many months before that. It was in people's hands long before Apple even announced anything similar.
> If not that's on the Gboard team at Google for not supporting neural on iOS and Apple hardware that came far, far sooner.
Once again, Apple's hardware came later than Qualcomm's hardware. Pixel phones come with a lot of features that could be implemented on iOS, but why should Google go through the work of porting it only to promote a competitor's inferior platform? For example, Google Maps with navigation shipped years earlier on Android. Grammar check with a transformer model is now on all Android devices in Gboard (for more than a year and a half) and still no iOS devices.
All this thread has taught me is that Google/Apple religious zealotry is a very real thing.
What's interesting about this "debate" is my original links very, very, very clearly show the reality: Google being excited about launching the Pixel 6 (in 2021) with Tensor silicon - a first for them.
All anyone on this thread has done since is refuse to acknowledge how obvious and clear that is while deflecting and throwing things at the wall.
What was a first for Google was having its own silicon at all. The article was about how Google designed models specifically for it with neural architecture search, something Apple still doesn't do.
What's clear to me is you're salty over Apple continuously being behind to the point that you misread articles in order to support your fanaticism and refuse to acknowledge it even after you've been shown the evidence.
I'm pretty fatigued on constantly providing references and sources in this thread but an example of what they've made availably publicly (second result Googling "snap coreml"):
It's important to ask, because your sources are not referring to the same thing. The first is a reference to training techniques that have nothing to do with CoreML or Apple hardware. The second thing you've linked is a transformer model from 2022 that was ported to CoreML (alongside Pytorch, ONNX and 10+ other execution providers).
It's extremely unclear how any of these sources corroborate your claim, particularly the first link. It's why I asked for clarification.
Is it really so hard to believe that the technical details of Snap's implementation just may not have been made publicly available at the time they deployed it? It's not as though they're known for running a breaking engineering blog like some smaller companies... Speaking of which:
In my top comment I said "Snapchat, as one example". Feel free to search around to find others that were more transparent on implementation details, such as this random weight loss app from 2017:
If you're genuinely curious there are plenty of references of apps using CoreML in 2017 (again, Google) but at this point I'm pretty confident you're just (lazily) trying to win an argument.
Your linked source says Snapchat has been using CoreML for neural network acceleration since 2022, which is not "since the early days" as you had originally claimed.
The link was provided to refute "Snap uses CPU". The repo is from 2022 but do you really think Snap publishes breaking code and papers on new, novel, and highly competitive functionality? The snap-research GH org wasn't even created until 2020. Do you really think they weren't doing anything before that because it's not on Github?
This thread has been exhausting, I suspect due to the religious war that is Apple/Google. As I've said time and time again - if you're genuinely curious Google is your friend here and it takes seconds to find any number of apps, projects, etc using CoreML in 2017.
It's bizarre how you keep refusing to acknowledge you were wrong. You made specific claims that were specifically refuted and now you're "yeah, butting" wildly to try to somehow win an argument you started by trying to find something else that Apple might be better at.
As far as I'm concerned, the war has always (since the mid-80s) been Apple versus everybody else. Nobody else makes clearly wrong claims like the comment at the start of this thread. What yanks my chain is that Apple is against user control, and we as developers should be above falling for their marketing that they're first or best when they very clearly aren't.
For text input support, being boring feels like a feature not a bug. For writing on mobile, you'll be writing a lot of boring short messages. That's said, it'd be cool if it can precondition with the app name - You'll probably write something more interesting on Pages vs on iMessage.
Inline predictive text is an anti-feature that disrupts your flow of thought. I've tried but cannot understand how anyone could actually want this.
Edit: I was referring to writing prose where you're making creative decisions rather than code which is closer to a technical document.
In code things are different. Traditional intellisense is usually just filtering a small set of possibilities. For example, auto completion of a reference to a variable.
This is different to code prediction where for example, a variable name will be suggested.
I can see value in code prediciton ala Copilot but personally don't use it.
My favourite input app on android does real time autocorrect by displaying 3 most likely predictions on a bar above the keyboard. So it is not really inline.
With programming IDEs when I can I configure them to display suggestions below the text I type, but if there is no such setting I don't find inline autocomplete bothersome at all. (as long as it displays it's prediction in dark shade of grey or another color sufficiently different from the text I typed, also there has to be a special key to accept the suggestion, like tab, no "enter accept")
Apple's current autocorrect is skewed towards verbal, not typo, errors. Apple wants your voice. Implementing this deliberately worse change also gave Apple a path to 'improve autocorrect'. Comments are correct, T9 was based on the keyboard and common mistyped keys, simple, and effective.
Don't take my word for it, as you all blindly have not.
Take the word 'lie', often mistyped as 'loe'. This is because the o is next to the i. After accidentally typing 'loe', look at what iOS offers to replace it. low, lot, Joe. Soundalikes. This is because iOS autocorrect is skewered towards spoken vocabulary and not commonly mistyped words in english on a typical 1874 Sholes & Glidden qwerty layout. T9 worked because of this distinction. And iOS does not work as consistently because of this distinction.
My 'feeling' is that Apple were very much aware of this from the iOS outset, and not wanting to use T9, offered this poorer auto-correct, and are now riding over the hill shouting 'we've solved it'.
I have no idea whether you really don’t know what “LLM” stands for or are just trolling. Not calling it an LLM actually is more honest marketing precisely because it’s small, and “transformer-based model” is a precise technical description that everybody with even superficial knowledge of the field understood immediately.
And I’m actually onboard with most of your other examples. I’d include “all-day battery life” as another empty marketing term that doesn’t really say much.
I know that LLM stands for large language model. It is still that type of model even though that final parameter count is not very large. They are deliberately avoid "LLM" because it doesn't compare favorably with most.
This model wasn't advertised by apple, someone who is probably not even affiliated with apple wrote a piece on how he found, analyzed and activated a small neural network inside of an obscure folder. There are no misguided "apple fans" to incorrect here.
This is the one case where the technical term and Apple’s marketing actually do happen to overlap! These are transformer models. Apple didn't invent them, but their description of them is technically accurate. “LLM” is the dumbed-down term for the masses, describing only the “what” but not the “how”.
I know what a transformer model is. But Apple knows the normal technical term is LLM. They are deliberately avoiding it to try to create a new category for themselves, because their model's "intelligence" is very poor when compared to other LLMs.
Nowhere does the press release say LLM. That's because it's not very large or smart and when compared to most LLMs' "intellectual" performance, it looks bad. By avoiding that term, they are successfully sidestepping an important aspect of the evaluation of this system in the eyes of many Apple fans.
you want to compare llama2, inflection1, gpt4 et al. to apple's typing assistant model that is designed to run locally and in a tiny scope? large language model literally isn't accurate for something that's two orders of magnitude off what the term usually describes. if anything, give apple points for not hyping up "LLM technology" or whatever to boost their share price like every other tech-adjacent company is rn.
I know it's infuriating but that's how marketing works, and since they are a successful company, idk why they would stop using it. Just like some people say "RTX" instead of RayTracing now, it's quite a success for the company if it managed to replace a technical term in the mind of most consumers
I want to know if this will be used to improve all the places where Apple devices attempt to interpret what you might mean to type, including the swipe keyboard. I've been suffering for years with their terrible, unusable swipe typing. You can't even get it to swipe "I love you" because it always prioritizes "your" over you, regardless of the context. I've even experimented extremely slowly and taken screen recordings to ensure the swipes are accurate. There are a couple of other completely astonishing common words that it 100% of the time gets wrong. I guess GPT3 would probably tax the battery, but on the other hand, I'd like to have the freedom to try it, because it would let me finish tasks much easier if text entry wasn't like fighting with an insane overconfident toddler. Honestly, I don't know what happened. Text entry with iPhones 10 years ago was far less infuriating.
Seems to work fine. Though I typically use Gboard because the swipe typing is much better.
I find the most restrictive thing about Apple's autocorrect and speech to text to be the limited vocabulary. Once you start using any industry terms it completely fails.
Especially to whomever downvoted me: there are at least 64 of us with this same bug.[1] — and that’s just the ones who bothered to waste time on Apple’s official discussion board.
I reset my keyboard dictionary again on iOS 17, still exact same reproducible bug. It’s been at least 2 years.
