From Apple website: "Apple Intelligence analyzes whether [the request] can be processed on device. If it needs greater computational capacity, it can draw on Private Cloud Compute, which will send only the data that is relevant to the task to be processed on Apple silicon servers. When requests are routed to Private Cloud Compute, data is not stored or made accessible to Apple, and is only used to fulfill the user’s requests."
Are there any more details on what exactly is being sent as context to the cloud?
Do they send features extracted from an image on device or the full picture?
Is it capable of only sending current pic or will on-device model select what it thinks is needed for context and many pictures and/or texts will be sent?
There is also a technical blog post [1] on the architecture of Private Cloud Compute. I don't think that there are any details on the context being sent, just that any context is ephemeral, can't be traced back to you, and that the machine instances leverage Apple Silicon's Secure Boot and Secure Enclave.
1. Partial client side execution of the models, then main model execution in cloud. (Example: text encoder for Stable Diffusion runs local, UNet runs cloud)
2. Local LoRAs for “dynamic ML”
3. WebGPU will be new standard to unlock models in the web
Even though the article had some good calculations and overview, I am not sure word-for-word is how I would describe its relationship to the OP.
Sort of but Adapters allow for multiple weight adjustments (think loras) for specific skills so it is more like extra optimized mixture of experts or multi agent approach. They have a slide with adapters listed like summarization, prioritization, tone (happy, business, etc), editor, etc) -- this is not to be mixed up with Intents which is how on device apps publish their capabilities to the Intelligence system for real npu os level multi agent tool use.
My opinion is that the market is treating Nvidia as the shovel sellers during a gold rush. If almost every business that wants to do AI needs to buy Nvidia hardware, their sales will go through the roof.
That’s what the market is pricing into the Nvidia stock.
It is very unlikely that Apple will sell their “server” hardware to other businesses in that way. Their advantage is the vertical integration, for themselves.
While I am bullish on AAPL, I don’t expect it to have a run like Nvidia.
The one area that I think Apple might be setting themself up well for is the longer term edge device AI race. It’s reasonable to think that AI will become more and more embedded in all aspects of computation and communication, and with that, it seems plausible to think that running and executing inference at the edge on devices with low power and compute capabilities relative to what’s available in the cloud will be a desirable deployment methodology - cost, bandwidth, robustness, etc etc.
Obviously there’s nothing of the sort in the works now, but it’s not totally implausible to think that the expertise and infrastructure apple is developing now might be setting themselves up to also handle inference for things like home automation systems, industrial sensors and actuators, etc which may all need edge inference capabilities. Chips like the M Series or on the other side the NPU will presumably be coming to many devices outside of the mobile/pc market and it doesn’t seem totally insane to me to think that Apple might want to get into that world.
Again, this is all just speculation about long term pathways that might be unlikely after all.
The last time I remember this happening, the shovel-making machines lost a lot of value and got amortised out to the second hand market. When the prevailing coin still mined on GPUs switched algorithms the bottom of GPU prices fell out on eBay.
I expect at some point
a) the training and the inference will move to even more specialised hardware and the current existing silicon go to either hobbyists or some other market that requires the current cards
b) the way in which we do AI may fundamentally change in ways we can't predict, causing the required hardware to also change (move to FPGA or favouring some other aspect than VRAM)
In both scenarios NVIDIA isn't the one who win, because the sudden influx lifts a secondary market nv don't profit from directly.
I personally don't think they'll be able to sustain their current valuation past the current AI rush, and will compact back down to somewhere their levels pre-chatgpt.
That said I'm just a guy on the Internet who wouldn't mind some beefy ex-datacentre gear if I could make it work for video editing workloads.
Same here. My eBay watchlist always has a few in it, but the prices just aren't where I need them to be. For now I am focused on inference with a dual 3090 rig, and my next step will be to fit another pair of 3090s, which is the max I can easily support with my AM5 motherboard. What's slowing me down is the fact that I can't run server hardware due to noise, and I can't run an open-air setup because my cats would get into it, so I need to buy or build a case to hold everything.
This is totally not a space I understand. But it seems like there are possible threats to NVIDIA, such as
(1) MLIR, ROCm, or some other tooling that reduces CUDA's moat,
(2) AMD attracting investment to go after NVIDA
(3) ARM-based GPUs or accelerators gaining traction among cloud companies that have huge fleets of AI devices and also have the money to devote to custom chips
Can anyone who understands the industry explain why those threats (or similar ones) aren't a major issue for NVIDIA?
(1) Much development is already moving from CUDA to the LLM, so less of an issue. Nvidia is also doing more work to increase interoperability. Could be an issue I guess, but doesn't seem like it since there's nothing close to CUDA or the ecosystem.
(2) AMD has attracted significant investment looking at appreciation in its market cap, with a PE ration 3X Nvidia's. However, AMD is so far behind in so many ways, I don't believe it is an investment problem, but structural. Nvidia has just been preparing for this for so long it has a temendous head start, not to mention being more focused on this. Remember AMD also competes with Intel, etc.
(3) Hyperscalers already are building their own chips. It seems even Apple used its own chips for Apple Intelligence. It's relatively (which is doing a lot of lifting in this sentence because it's all HARD) not too hard to make custom chips for AI. The hard (near impossible) thing is making the cutting edge chips. And the cutting edge chips are what the OpenAIs of the world demand for training, but releasing the newest best model 1-3 months ahead of a competitors is worth so much.
If anything, I'd say the biggest threat to Nvidia in the next 1-3 years is an issue with TSMC or some new paradigm that makes Nvidia's approach suboptimal.
I don't think I understand your point in (1) that it's less of an issue because development is moving to the LLM. I can infer that maybe CUDA isn't a big part of the moat given your other points that the hard part is making cutting edge chips.
It's just the natural evolution of tech towards higher levels of abstraction. In the beginning most dev was on CUDA because the models had to be built and trained.
But since there are plenty of more advanced models now, the next level is getting built out as more developers start building applications that use the models (e.g. apps using GPT's API).
So where 5 years ago most AI dev was on CUDA, now most is on the LLMs that were built with CUDA to build applications.
IMO the fact that apple just made rack mountable apple silicon for this private cloud, their GPU / AI compute is equal to nvidia on a power/W basis, they are looking for their next revenue expansion territory from flat lining hardware growth and the fact that nvidia is worth more than them is enough of an indication that will probably make apple silicon server hardware in the near future just on a pure money making basis, even if it's not how they typically focus.
They kind of lucked out with apple silicon and their unified memory architecture.
> and the fact that nvidia is worth more than them
You scared me. I know the market is hyping up Nvidia hard, but if it was really worth more than Apple then things would really be crazy.
AAPL market cap is still 10x NVDA’s (3.2T vs 330B), so not yet too crazy.
Update: Nvda is at 3.24T. ddg “nvda market cap” shows the wrong amount. It probably doesn’t take the stock split into account. Djeez. That’s. Well, time to short.
The reason is because people have stopped buying iPhones every year. Apple has been priced like a growth stock for a very long time but their revenues have stopped growing! (Seriously, look at their earnings reports)
Now compare to Nvidia, their stock has been on an AI related run because their revenues have been MASSIVELY increasing over the past year, because they sell the most advanced chips for training and inferencing the most complex models.
Apple may be “doing AI” now, but it’s for consumer products, they’re not selling enterprise AI services nor are they selling the actual chips needed to train and inference.
Also, I have major doubts as to whether any of the Apple AI announcements are going to drive higher iPhone sales. There will be tech evangelists who buy a new iPhone for the AI features but the bulk of users are folk like my parents, who use their phone for calling, messaging, and taking photos. The only thing they ever noticed with new phones was either it was:
1: better camera
2: better/bigger screen
3: really obviously faster
The last few phones have been indistinguishable in terms of screen, performance, or camera, and my own personal
iPhone cycle went from every year or every other year, to now roughly every 3/4 years (and I’m an engineer).
I think there's also more subtle improvements using e.g. object recognition.
Last week I was in an art gallery and took some photos of the paintings, when I suddenly noticed that it was drawing a focus around the frame and there was a 'Look up Artwork' option, which work fairly well (not 100%). Just as well, as I couldn't remember the works myself the next day after looking at the photos.
If I hadn't taken those pictures I probably wold never have known the feature existed.
From what I understand, such gimmicks are (almost) essential utilities for many.
It also is possible that small improvements in intelligence will drive sales. For example, I don’t see much use for generating entire novels on-device, but good autocorrect and prediction of a few words may drive sales. I also like the automated tagging that iOS already does on photos.
Also, even if the new features only make people upgrade a few months earlier, that will be serious amounts of money for Apple.
Yeah. I’ve been planning to upgrade from my 12pro this fall, and I’m mostly excited to get better optical zoom and trash my lightning cables.
I do think that Siri’s Generative AI upgrade (which is how I think of this whole Apple Intelligence thing) might be significant enough to change the smart phone as a segment. There are so many things that I feel Siri should be able to do, but it just can’t. Maybe this unlocks those use cases.
If it does, that’s a major incentive for me to increase my investment/reliance on the Apple ecosystem. I’ll use the calendar and mail apps if it means Siri has all that context. I’ll upgrade devices to get the same answers with less perceptual lag.
If we’re focused on hardware sales, I’d be interested in an Apple version of the Rabbit R1. That thing is almost a modernized iPod Touch - great device for kids you don’t trust with an unsupervised connection to the mobile internet. I would be delighted to get a Real One from Apple.
The HomePod Mini is close to this already. It’s. Not clear to me how much Apple Intelligence will be available through those. Again, if it actually works I will buy more of them.
I’m still on the 11 Pro, and I feel similarly. I’ve not seen any reason to update to date, though the new AI functionality might just turn out to be it.
I’m unsure I’m willing to lock myself in to the Apple ecosystem to an even greater degree though - they make it devilishly difficult to extricate yourself, and I already find the enforced storefronts built into their apps obnoxious. The Books app went from being a library to a shop - how long until the Photos app is trying to sell me things?
If anything, I may wind up bifurcating my life to an even greater degree - iPhone for general life admin (calendar, phone book, banking etc), Linux for productivity, creativity, gaming.
> There are so many things that I feel Siri should be able to do, but it just can’t.
The competitors can and have done these things for a long time now, but all any of these things are (currently) useful for is setting a timer and playing music.
Without a rewind style access to your entire life, there is a real limit on how much useful stuff this technology can actually do.
An LLM with exclusively global context (no personal context) is already pretty powerful, but certainly not helpful as an assistant.
In theory, perfect personal context would help a lot. Certainly a smart human being who follow me around all day can be very useful, but is expensive and has human limitations.
I don’t think I want exactly that from an AI. I am not thrilled with stuff like the Humane pin. I don’t trust that it will keep all those observations secure. I don’t trust that it will sort through all the noise to identify what’s actually important.
I think a mobile OS is a good balance. It has a LOT of personal context. I already trust it to process that data securely. The context is already well-structured and actively curated. I’m optimistic that adding LLMs to this will be significantly useful.
> Siri’s Generative AI upgrade (which is how I think of this whole Apple Intelligence thing) might be significant enough to change the smart phone as a segment.
Why? Most play with generative AIs for a few days and then abandon them.
The actual gamechanger would be making an AI assistant to be, you know, an actual assistant maybe?
I think LLMs improve Natural Language Understanding enough that we can approach genuine assistance. This is especially true if (1) the AI has access to high quality, well structured context, and (2) there are some constraints on the domain of possible tasks.
I think OS integration can provide both of those things. Mobile app selection is already pretty strong, and the Shortcuts ecosystem is a good example of providing direct access to specific functions implemented by apps. AI should use the shortcuts functions exposed by the apps I’ve already selected, installed and configured as tools.
If I can use shortcuts on demand with just my voice… that’s a huge win for me.
Because nobody is going to buy iPhones in bulk just for this new AI unlike GPUs for Nvidia, it's a nice to have upgrade, not a game changer for the company.
I don’t think we’ve seen the whole story on how Apple plans to monetize what is effectively a massive, distributed AI platform.
Think of all the AI startups paying OpenAI per token for consumer app capabilities. Or having to rely on suboptimal local foundation models. The value Apple can offer here is considerable.
I’d expect Google is headed towards a similar architecture, and both platforms will include API monetization for third parties.
I wonder if Apple’s deal with OpenAI precludes the latter from attempting to build their own hybrid platform on Apple devices…
Apple isn't interested in anything outside their core interests. Even if they did make very specialized hardware for AI training they would not offer this to the public.
I'm sorry but that does not make any sense at all.
Nvidia's stock is surging because they sell GPUs and big companies by a lot of their GPUs to train AI models.
Apple's stock has not surged following the AI boom because they... don't sell GPUs used to train AI models.
AI is a profit center for Nvidia and a cost center for Apple (yes, people that weren't buying iPhone won't suddenly buy an iPhone because it has an AI assistant embedded).
> people that weren't buying iPhone won't suddenly buy an iPhone because it has an AI assistant embedded
> There's absolutely no link between the two.
These are remarkable levels of speculation. Already I've heard from Android friends who are intrigued by the recent AI announcements that they think it would be really cool to have that in their phones. I don't know if they will necessarily switch, but they certainly indicated they were very interested in the iPhone AI announcements.
I feel like this is some sort of biased perception. I suppose most people who are interested enough in technology to talk about smartphone choices are also somewhat interested in this AI craze. However, most people probably aren’t that interested in technology.
I still know more people who don’t use ChatGPT on a semi-regular basis than people who do. Most people’s phone choices come down to:
1) is it available in a color I like
2) is the camera any good
3) can I afford it
4) *optional* does my trusted techy family member recommend it
Apple Intelligence is not just about AI. Although none of us have used it yet, from the keynote, it's clear that it can benefit people who are less technically inclined, regardless of age. For example, while my mom might struggle to use Apple Photos to search for specific pictures, she would have no trouble asking Siri for all the photos of her granddaughter dressed like a princess. The goal here is not about bleeding-edge technology or AI; it's about providing more natural interfaces for users, whether they are tech-savvy or not.
>For example, while my mom might struggle to use Apple Photos to search for specific pictures, she would have no trouble asking Siri for all the photos of her granddaughter dressed like a princess.
Want to point out Google photos has been able to do this with a voice command for close to(?) ten years. Just tried it with "[person name] in a dress"
Definitely, my point is that having an easy natural language interface for all products opens up new possibilities for users of all tech levels. It's not about Apple or Google Photos being first; it's about making technology accessible to everyone. And its not about "AI" its about a different way for the consumer to utilize technology.
And you would need extreme short sight to not trust that GenAI on Android will be out in a year if not much earlier than that. Hell, it may come out BEFORE iOS 18.
That is of course without mentioning that the only two devices compatible right now (both versions of the iPhone 15 Pro) represent a total market share of 3.1% of smartphones.
AI in iPhones is essentially a distraction from the fact that Apple hasn't brought any software innovation to its devices in probably close to a decade (note I say this as someone who has only had iPhones since the 4).
>Already I've heard from Android friends who are intrigued by the recent AI announcements that they think it would be really cool to have that in their phones.
Fancy marketing tends to do that. Have they tried it?
You know the answer to your question, so the question itself does not add to the discussion. If it turns out the iPhone AI assistant lives up to the expectations, I think you will see people moving platforms.
If Apple AI succeeds, then Android AI will just look like an upgrade by comparison. Best-case-scenario, Apple creates a super-helpful model that exclusively uses local compute; then Android releases the same feature with the ability to choose uncensored models.
Especially considering how Apple's Neural Engine is borderline-pointless for LLMs, neither Android or iOS feel particularly well-poised to deal with this trend. If anything I lean more towards Android, since Android manufacturers have been shipping 8GB of RAM in phones for years, and support the Vulkan accelerators that a lot of inference apps feature.
> Already I've heard from Android friends who are intrigued by the recent AI announcements that they think it would be really cool to have that in their phones.
Most of it has been there for Samsung already and some of them like the emoji generator look suspiciously similar.
On Android, usually some large manufacturer (here Samsung) push the platform further and then some of it is integrated in stock later.
100%. And it’s also about retention too. Iphone’s speech to text has gotten so dismal compared to android that if Apple does not catch up, people would end up switching.
As an Android user, yeah this would be cool to have on my phone, but it ain't gonna make me switch to Apple lmao.
The benefit Apple has over Android (not Google) is that they're fully integrated, ie they make so much profit from their closed platform that they can toss customers bones like iMessage and "apple intelligence".
Remember, attention is all you need came from Google in the first place...
I follow up to the phone part because absolutely people will be buying ipads/mac books/phones to get access to AI Assistant. I am definitely in the camp that if AI Assistant offers some nice workflows on my phone and laptop that I may actually buy a new ipad for a seamless experience. I am sure I am not the only one in this camp. To me Apple Intelligence is the closest to a personal assistant that we have gotten so far and what Siri/Alexa/Hey Google failed at for a decade.
All of Apple's investment in AI edge capability is almost totally undermined by their huge price discrimination on RAM, shipping 8GB laptops in 2024 and charging like $300-500 for 8GB more.
I don't know why you're being downvoted. Unless Apple users want to swap a multi-gig 3B model onto their 256gb SSD and chew through it's lifetime TBW, we should be campaigning for enough RAM to do normal stuff. Our SSDs deserve a break.
Yup, idk why but Apple fans act weird, like brainwashed.
You can argue "Hey Apple shouldn't charge so much ram, it's better for me, it's better for you, it's better for everyone" and then you get attacked for it (usually by claims that they use "suuuuper" special flash and ram chips bla bla super high performance bla).
I think Apple’s approach to (more or less) private local AI, with short term reliance on OpenAI makes sense.
When I listened to the Apple Developer’s presentations this week I always filtered everything said with “they are talking about proposed product updates, and some existing working functionality.”
I have been enjoying running all of Apple’s beta iOS, iPadOS, and macOS beta releases this week, and there are nice features in Photos, Calendar, etc., but we need to wait to see what they release next fall.
I am not an Apple developer, really, I did really well selling a Mac app in 1984, and a Mac app I wrote two years ago had pathetic sales. That said, I have been trying the beta Xcode with LLM code completions, etc., and I have been experimenting with MLX and Apple Silicon for a long while. Apple is definitely an interesting player in the AI product space!
Did you watch the WWDC keynote? There are a lot of legitimately exciting features. There's a level of integration with AI and your OS/apps that is (as far as I know) unprecedented.
One thing I really want is for Siri to be able to control my house with Home Assistant with natural language, e.g. "turn off all of the lights outside except for the balcony". The updates to Siri seem like it _could_ make this possible.
I would _love_ to be proven wrong, so please do show me alternatives.
Yes - even attended. And I didn't say bad - just meh. Many promises have been made - let's see if they pan out. Many alternatives, Apple hasn't brought anything new to the table. Calculator is cool though...
> The whole thing comes down to - your request may get sent to ChatGPT for advanced features.
If that was your takeaway from the keynote/SOTU, then I would suggest you go back and watch it again. That's not even remotely close to being accurate.
If you didn’t see anything new in the announcement at all, maybe you didn’t actually see all the proposed features they are advertising? I’ve never seen a lot of stuff contained in one personal device or computer before. A single assistant that has access to finding connections throughout your life, emails, photos, etc.?
Yes I did - but in 30 years attending and watching these I've learned a lot. As for Siri - been waiting for a decade for it to do something really useful - Let's hope this time it does.
I wonder when the illusion is going to come crumbling down.
As long as there are teenagers that are susceptible to peer pressure and don't know marketing tricks, I think it could last forever. Look how many other Veblen good companies have existed for hundreds of years. With Apple's cash and media, they can have rappers and celebs always using/mentioning their stuff.
> As long as there are teenagers that are susceptible to peer pressure and don't know marketing tricks, I think it could last forever.
Consider also that you might have made a mistake in your analysis and there are factors involved other than celebrity endorsements – for example, wouldn’t it be interesting to know why Nokia, Google, Samsung, Sony, LG, etc. weren’t more successful thanks to their own endorsements? Your own peer pressure might be hindering your understanding of the market.
Apple seems to be betting that something polished which is “less than state of the art” will sell better than something less polished and more advanced.
Technical capability is rarely more important than product market fit.
The problem here is that for current AI systems, the state of the art is the one that has ironed out the most kinks. I.e. what you would expect from a polished product. That is basically ChatGPT with it's monstrous API that does tons of god knows what behind the scenes in addition to feeding text to an LLM, most of which was figured out by trial and error from user reports. There's no doubt that a modern LLM based Siri would be a much better product than the old one, but if they're not careful, they will fall into the same trap they did the last time introducing an assistant.
Privacy is a part of the equation and while Apple could gather all the context off my phone's local data and give ChatGPT the whole thing to really give it the best shot and providing a useful and helpful answer, this would not be a polished product on all axes, mainly the responsiveness and privacy ones. The tradeoff of using local models for less than SoTA inferrence but with unbeatable privacy and latency seems right to me for Apple and what Apple customers want and expect.
The best AI experience right now isn't just using SOTA models, but using them in a way that plays to their strengths and avoids their weaknesses (hallucination, limited reasoning, etc).
The way to minimize hallucination is to use LLMs to manipulate text (translate, summarize, query, reword, etc) rather than generate it out of thin air, and arguably Apple are doing this by integrating LLM use into the product and therefore controlling it's application, rather than just providing a chat interface that people are wont to use an hallucinating "search engine"/oracle.
I think this is still a tech-centric view rather than a product-oriented view.
E.g. Apple generating emojis using a diffusion model is clearly less SOTA than other image models, but could be a really neat feature from a product/user perspective. Would a better diffusion model help? Maybe, but it probably changes the product less than all the UI and tight integration work they will have done.
If I was to guess, I would guess many more emoji's will get created through this feature than images generated in Microsoft's 'Image Creator' in Paint (I could be wrong - just my gut feel).
WhatsApp has integrated AI generated stickers almost a year ago. (https://www.theverge.com/2023/8/15/23832635/whatsapp-ai-gene...) I'm not sure how much they're used globally, but there are people crazy about stickers and the quality of what I got was really decent already. There were already external apps available for this purpose for a while, so yeah, the interest is there.
Why does SOTA matter here? OpenAI has actually failed already to make something useful and mass-market. After all these years of pushing SOTA, it's still a website with a text box. Siri in its current form is more useful to me than ChatGPT 4. You simply don't need SOTA models for a lot of valuable features; you need context, annotation APIs, and the right platform and integrations to assist users where they already are (on their phones, in cars, in their IDE, etc). MSFT's GitHub copilot is a good example of getting it right.
Apple relegated ChatGPT to a 3rd tier AI capability for "google answers", and "write me a poem" party tricks, and rightfully put it behind a privacy disclaimer. It looks barely better than the various ChatGPT shortcuts so many people have cobbled together. That part of the announcement stuck out like a sore thumb and looked to me like a huge L for OpenAI. The "partnership" was a nothingburger. Like someone at Apple agreed to it early on but then late in the project realized it wasn't needed at all. So much for SOTA.
I mostly agree with you. OpenAI is reportedly still doing great in terms of revenue, but Apple's implementation is magical if it performs as shown in the keynote. In my opinion, it's the best implementation of LLM/AI in a consumer device. It's amusing to think back to all the buzz around Humane's AI pin.
> It's amusing to think back to all the buzz around Humane's AI pin
It certainly is. And the Rabbit R1. How in the world did people who supposedly know anything about AI think they could make that work as a standalone device detached from rich context. The sad thing is, rich context may not be possible outside of OS-level integration that is gate kept, but I still think they're idiots for trying.
gate kept OS-level integration is one thing, companies that won't push data to it is another. I fully expect facebook/messenger to not support intents/siri/shortcuts/etc
Same for discord and any other social media platform right now.
