Apple's chat.db has an esoteric schema owing to the fact they never designed it from the ground up and instead kept adding new columns and tables with each macOS release. This makes their queries super complicated with multiple joins.
Once you have the schema figured out, it's dead easy to build a third-party client that works better than the official one. Even search works great with a simple LIKE query but Apple re-indexes all messages leading to your CPU going over 1000%: https://twitter.com/KrauseFx/status/1396433852126670852
Source: I built a third-party desktop client for iMessage at https://texts.com and reverse engineered the complete sqlite structure.
This is basically one taken out of microsoft's .doc format playbook.
At first I thought Apple was slow and just didn't have time to get to some sort of export for iMessage chats. Mail had imap, calendar had caldav, address book had carddav. The original mac stuff did other standards like AIM and jabber.
But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
What's annoying is that I couldn't save my messages, I could only upgrade to a bigger iphone.
What's downright evil is all the people - I know several - who have lost messages + embedded photos/videos from loved ones who have died because there was no export.
>But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
In an SQLiteDB any programmer can learn to navigate, and any third party can build a tool to export them?
Sorry but the "internal emails" prove nothing about Messages. It's just a quick and dirty implementation that "works good enough" for reading the latest, but not as an archive.
Yeah I agree. The lack of a good schema/export function sounds to me like negligence not maliciousness. It’s a different issue from only allowing Apple-tech-users to use iMessage
> But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
Any source for that? I couldn't find any direct evidence for your claim.
Thanks, that's interesting, I didn't know that. On the one hand, I shouldn't be surprised since vendor lock-in is the standard way tech companies try to gain corporate advantage. On the other hand, it's kind of depressing to learn once again that customers count very little and no matter what these companies say, they will always try to abuse us in every way the law allows.
Just took a look at your product, I'm very impressed and have added myself to the waitlist. I'm almost even more impressed by the domain name itself though - how on earth did you manage to bag that domain?!
The fact so many people on this site can throw something functional together without much effort is even more disappointing apple doesn't do anything about it.
iPhone is the only apple device I own but it crashes it can't go too far back into my messaging history for any one individual too far.
Usually once I scroll to ~5 months back with people I text regularly the iMessage app just straight crashes ... it's a little absurd. To top it off, every couple pages it gives a loading icon and introduces severe latency for whatever reason. There don't seem to be any apps in their App Store that let you search and lookup old messages either.
This is actively biting me right now. Are there any solutions for accessing the iMessage/chat.db database on my phone or via iCloud from a Linux system (Ubuntu in my case)?
A little feedback on the landing page: Almost turned away because I thought „Sign up with Google“ was the only choice and I missed the little „Prefer entering email instead?“ below. I just prefer not having a Google account ;)
Maybe the styling of that link could be adjusted so that it isn’t glanced over that easily.
I don't mean to be pessimistic , but I wonder how you're going to be anything but reactionary to apples changes to the database. I've written a similar (much simpler) version of what you're proposing to sell- the database queries are hysterical.
I've been thinking about using the accessibility I framework to ETL the messages into Kinesis to facilitate event driven architecture. 'archived' Messages and attachments are stored as bsob.
Hell, I've even experimented with some goofy stuff, like FaceTime to VOIP- asterisk is just so flexible
Looking over your site, I'm curious about a few things:
- The FAQ says you'll eventually open source your SDK. I admire that as a software engineer, but from a business perspective, isn't this risky? It seems like--if you don't pass messages through an intermediary service--your value-add is mostly in this SDK + your design. All of these elements are copy-able.
- The jobs page includes a backend engineer, so what will go through your backend, if not any message data?
- Are there legal concerns about reverse-engineering these APIs and using them? Some of your supported chat services have open APIs, but others seem intentionally closed. (And a couple parent companies seem downright hostile to 3rd party use.)
Source: I built a third-party desktop client for iMessage at https://texts.com
Does the texts.com client solve the original problem? Does it make it easy to go back and view all those old text messages? I know the website says "from forever ago easily". But you didn't say that explicitly in your post.
I assume you're being modest and not "pushing" your solution. But if it simply solves the problem, maybe you should be shouting this out loud!
A piece of feedback: it would be nice to know what the price is on your site. You say "We make money by charging you a monthly subscription," but there's no pricing.
I've built an iMessage chat.db reader as well. To me the most astounding thing is that chat.db is unencrypted sqlite. Anyone who can see your home folder can read all your private chats or upload the chat.db to be read somewhere else.
Which you end up applying to random things like ruby because Emacs.app uses a ruby-based launcher and anything less than Full Disk Access breaks dired.
Why would a lack of encryption make it not reverse engineering? There's a database schema, it was engineered* and used to build a database. They went from the live data back to the schema.
*For some definitions of "engineered", apparently.
Very eager to see how your app works when it opens up to the public. Are there ways to provide assurances for your end-to-end encryption which may be digested by experts?
You also mention on your website that end-to-end only works when the platform "supports" it — does that include Windows with your currently always-on Mac solution?
> Very eager to see how your app works when it opens up to the public. Are there ways to provide assurances for your end-to-end encryption which may be digested by experts?
We recommend using a simple MITM proxy or a firewall app like Little Snitch / LuLu to inspect the network traffic.
> You also mention on your website that end-to-end only works when the platform "supports" it — does that include Windows with your currently always-on Mac solution?
Yes – there's a peer-to-peer connection between the Windows and Mac devices.
I had done something too, but I also needed a Manifest.mbdb parser to reconstruct backup to disk, then process the messages and copy all the images around to make it work. I did it mostly to make a simple-to-read archive of chats before deleting them.
(You need an unencrypted backup directory to run it over)
Note: It looks like the flat directory structure of the backup is now 256 dirs prefixed by first two characters of file name. I'm not sure, but a small change to the name "sha_file" may be needed to just path.join that with the first two chars of the sha.
Python 2 but can be ported forward without much effort. Manifest.mbdb is now Manifest.plist (there's a sqlite thing now too). timestamp sql might not work so great, would need to check query.
Yes, with a catch: it requires an always on Mac (which can be in the cloud) or a jailbroken iOS device. We plan to make it work without the catch.
I've worked with 10+ messaging platforms so far and Apple's protocol is the most obfuscated and complicated to reverse engineer. Apple has invested millions of dollars to make the iMessage protocol super hard to reverse engineer. It's how they sell tons of iPhones after all:
> However, Craig Federighi, Apple’s Senior Vice President of Software Engineering and the executive in charge of iOS, feared that “iMessage on Android would simply serve to remove [an] obstacle to iPhone families giving their kids Android phones”. (https://www.androidpolice.com/2021/04/28/apple-admits-that-i...)
Looks like a super cool project. I'm always happy to see new client-side software for communication protocols written that aims to improve upon built-in! And iMessages in particular could certainly use it. With the demise of iTunes, it now feels like iMessages is probably one of the most crufty-but-heavily-utilized user facing pieces of software Apple puts out. Maybe they have some long term plans to refactor it ala iTunes but yeesh.
This bit did make me wonder though:
>We plan to make it work without the catch.
Out of curiosity, do really think that's realistic, or even desirable long term? iMessage is ultimately an Apple service that runs heavily on Apple's infrastructure, and is directly subsidized by sales of their highly vertically integrated hardware platforms. If it turns into a cat-and-mouse fight it seems like they're always going to have the eternal upper hand, which in turn seems like it'd make for a subpar user experience (ie., breaks randomly which for an instant messaging service would be pretty bad). Also seems like they might actually be motivated to respond rather than ignore it since it'd actually be directly leeching their infra if it will work on PCs/Android without a Mac/iDevice purchase in the equation (unlike hackintoshes for example, where whatever debate there is to be had about probably very low "lost sales" it doesn't actually cost them anything).
Obviously you've probably thought this all through, but seems like just requiring an old cheap Mac or old jb'd idevice and thus avoiding Apple might be an easier path. Or alternately just stick to offering a flat out better client with the iMessage bit being Mac-only. Will be interested to see how it goes though!
> one of the most crufty-but-heavily-utilized user facing pieces of software Apple puts out. Maybe they have some long term plans to refactor it ala iTunes but yeesh.
The Messages app on Mac was rewritten for Big Sur (last year). It gained some features but also some bugs in the process.
Likely because each iMessage receiving device has a private key to decrypt the messages, and senders encrypt the iMessage against all the keys in your key bag.
To access iCloud and or services you also need to have a device security key that is tied to genuine hardware.
Author here: the gist is that it's practically impossible to scroll to the beginning of an iMessage chat, and that's the only way to get to the beginning of a conversation. These old conversations take up several gigabytes of space, yet there's no good way to search them (because search for old messages is unreliable).
This aspect of the iMessage user experience has remained untouched for a decade, despite everyone's chats growing longer. Searching for particular iMessages sucks because the results miss the broader context of the conversation (a single message is not useful when you want to reread a conversation).
(Also just noting, as the post author I created a title better-suited to HN's 80 character limit.)
I actually did try emailing Tim Cook a few months ago, but I got bounce backs. I tried tim@apple.com and timcook@apple.com but the mailer daemons said the addresses weren't registered. It sounds like a crazy thing to do, but I remembered how Steve Jobs would reply to users' emails and read an article where Tim said he set aside time to read users' emails.[0]
- I’m never filing a bug report ever again on a bug filing system where I don’t get to see the progress in a transparent manner or that I might have to pay for a developer account to see that progress (assuming that’s an option)
- I’m not wasting my time emailing a top level company executive to inform of decades old most basic of bugs because if they don’t know of it then that’s even a bigger problem.
I’m somewhat surprised that acknowledging outside emails is a thing. I know unsolicited ideas/content/etc. is commonly rejected without review in other fields. Is this successful beyond identifying technical bugs?
It would probably work better for the two Steves than it would for many others.
According to the below random site, there 218 Jobs and 10,602 Wozniaks. It claims 410,801 Steves.
Related, I emailed Tim Cook about iPhone sensor future plans, and a high-level exec of like 25+ years wrote back to me within a week. Apple is remarkably good at getting back to cold emails to top execs.
Interesting, perhaps I’ll cold email them about the pinned contact feature in iMessage being totally worthless.
If it just pinned the message thread to the top and didn’t change the icon to a giant circle that sometimes doesn’t show notifications, it’d be fine. I pinned my GF’s message thread and I ended up missing lots of messages and notifications from her and ended up unpinning her so the notifications would show up.
Not to mention the comically huge circle icon that is centered when everything else is left aligned.
I had a problem that appeared to be a corner case with Apple Pay. I wrote a paper letter to Tim Cook about it, and within a week got contacted by a staff person in his office. He worked exclusively with me to identify issues I had not considered, and followed up on all of the direct effects and side effects until they were resolved.
I emailed Craig about several controversies and he was very quick to reply to me. We last talked about the rumor that Apple was giving iPhones trust scores.
If you can remember a term, any term, that was written in the message, you can search & jump to it directly. For my girlfriend and I, it was the bar we went to on our first date. This feature only gives you small subsets (a couple dozen, maybe) messages near the query result, but then you can just search for the last message again and get the next “page,” so to speak. Of course, remembering such a term is hard, which is what I assume you mean by “search for old messages is unreliable,” but it’s at least a possible workaround.
Hm, yet I have no problem searching phrases that go back at least five years.. I would say the burden of proof is on the author here to extract the database from the phone and prove that the messages didn’t get deleted, lost in an upgrade, or otherwise cleaned up.
My sister was having a problem where she was taking all the iCloud storage for our family because she keeps sending videos to people but couldn't figure out a way to efficiently delete them. I had to write up a Python script (https://gist.github.com/saagarjha/615dee3c04e226b44828e910a5...) to scrape the iMessage database for large attachments, and then print out the text to messages around it so that you could search for it and jump to around that place in the chat history and then delete the files from there. It's kind of insane how hard it is to manage old messages :/ I feel like Apple didn't really prepare for Messages coming in from iCloud meaning that every device had every single message you've ever sent, and now it's really hard to get through them without reading the database manually…
I am convinced iMessages right now is a tool to keep your storage full, push people to buy higher storage iPhone, and pay iCloud Storage backup to increase their Services Revenue.
None of the problem listed in the blog post or what you point out are new. They have been there since day one. And it is not that they neglect it either like iTunes or Aperture. They added Memoji!
Which is why I am surprised a the popularity of iMessages in US ( and France ). I think it is mostly due to SMS usage before iMessages was a thing.
Pulled it up on my phone just to see what they considered "large", looks like 4MB is the cutoff (nothing smaller than that shows up for me). But nearly everything I'm seeing is a video or a photo taken with my real (not phone) camera. And a random PDF I apparently sent over iMessages.
However, deleting through this interface would be a pain. You select "Edit" and then each attachment is selected individually.
It seems to just show you the largest set (50? 100?). If you delete them, then you will eventually show smaller ones once you get through enough of them.
Really wish there was a "delete all" or "select all".
Thanks. That makes sense. But I didn’t feel like deleting anything. Just my wife and I on the iCloud storage and there’s plenty of it at present. My photo collection that I’ve been cleaning up is a bigger offender right now than messages. Damn cats doing cute things.
When I am on the iMessage dialogue screen with a contact, I can usually click on their name, then Info, and it displays all the video/photo attachments sent in that chat. I believe I can then delete them from there. Is that useful? I guess if there's a number of photos you'd like to keep, but not the large videos, it's a more complicated issue.
Can confirm. Shameless plug: I wrote a script to extract the pertinent information from the SQLite database into an HTML page a while back: https://github.com/kccqzy/imessage-db-extract/blob/master/im... It's open source abandonware though. Lost interest in keeping up with the schema updates because I no longer use iMessage.
That's an undocumented implementation detail. It might change without warning at any time. Are you seriously recommending this as a solution for users who want to be able to read past messages?
I’d imagine it’d be difficult for Apple to migrate to anything substantially different as it’d require rewriting a potentially multi GB file, the space for which many users would not have. It’s effectively a massive, distributed, database migration.
I save them on the expectation that Apple will someday release a better tool for browsing them. Especially the texts between me and my wife, they are a story of our life from the very first one.
I was in a similar situation with email. In 2005 or so I was migrating to new computer when I found a directory of photos I forgot to back up. Rather than burn them to DVD or upload them to my domain, I decided to just email them to myself. I sent a handful of images, then replied to that same email with the next 10, and so on, until the five dozen or so images were uploaded in a single email thread.
When I went to open the email on my new machine, the loading indicator just spun and spun until finally my browser crashed. When viewing an email conversation, Gmail will display previews of all the emails in the chain, including their attachments, which was causing my browser to crash. Next I tried connecting my Google account to Apple Mail, but Gmail's SMTP server would time out before the attachments were able to download.
After a while I forgot about the photos all together, until I randomly stumbled across them while searching my Gmail account for some other files I had sent myself around that time. At this point it was 2014 or so, and my browser was very easily able to open the messages so I could download the photos, which of course at that point I had zero interest in.
I used gmail to subscribe to lkml at some point. Then switched back to my own email server. Forgot to unsubscribe on gmail. A few years later I deleted that folder. And couldn't open Gmail anymore for weeks, everything would just time out.
On a similar note, deleting old photos to save space on iCloud is very hard. I could mass-delete photos older than a certain date in the Mac Photos app (or whatever its called these days) using a smart album, but then that mass deletion would not sync to iCloud. Or, I could log into iCloud via a web page and select a load of photos (but not more than 1000 at a time) and delete them - but then it takes ages - like about 20 minutes. Why does it take so long to delete 1000 photos from the cloud? Its unfathomable. And then after deleting them you have to go and find them in the iCloud bin and delete them again - which also takes ages.
I can only conclude that they have made it deliberately very slow and painful to clear space on iCloud. Because they'd rather you paid for more space ...
> I could mass-delete photos older than a certain date in the Mac Photos app (or whatever its called these days) using a smart album, but then that mass deletion would not sync to iCloud.
Why wouldn’t it sync? If I did this right now, my expectation is that it would sync. Is there a limitation in how this works?
We need a better communication standard. Apple made a killing selling their proprietary iMessage layer on top of SMS, but we could be actually making changes in this sector if we all used a standardized protocol. At least then we have a documented interface that we can choose to store however we want, or (perhaps wishful thinking) in a modular style. RCS is a great start, but I've already started to put my money on Matrix. While the latter won't ever see adoption by Google or Apple, it's a pretty convincing attempt at rich, decentralized communication.
Is it really "on top of SMS" though? The way I see, it's just a messaging app like WhatsApp, Telegram etc but they just combined it with the SMS app on their platform.
I am failing to see the additional value of combining them. If Google bought WhatsApp and combined it with the stock SMS app of Android, I don't think I would prefer that.
One underrated feature of iMessage is graceful degradation. If I'm unable to send/receive data for whatever reason, my message (and their replies) automatically fall back to SMS. There's been a few occasions where this was my only way of communicating important information, so I'm grateful for it.
Yes. If the message is unable to be delivered to the recipient via iMessage, then it is sent as SMS instead. So for example if you try to send a message to someone and their phone is turned off or doesn’t have signal, it will eventually fail and then send as SMS.
This is all controlled via a setting in iOS iMessage settings “Send as SMS”, which I believe is turned on by default but I’m not sure. So your mileage may vary.
The value to Apple was that if you tried to SMS someone, iPhone would check if the destination phone number was registered with Apple, and silently switch to iMessage via Apple instead.
Cynically, they intercepted millions of people's comms with hardly a murmer. Charitably they switched millions of people from country-dependent, pricey and limited SMS to internet messaging with read receipts, pictures, video, emoji, audio recordings without any hassle of logging in, changing apps, choosing clients, etc.
> I am failing to see the additional value of combining them. If Google bought WhatsApp and combined it with the stock SMS app of Android, I don't think I would prefer that.
Well, it's not useful now, but imagine in the 00s when a minority of users had iPhones. Having two clients, one for SMS and one for iMessages, would have gone against the simplicity Apple was aiming for with respect to user experience. Combining it meant that users could get a better experience (than SMS/MMS at least) when communicating with other iMessage peoples, but not have to swap out clients to chat with others.
Personally as a college student, all my friends use iPhones still. I have always wanted to switch to andriod, but one of the things that really holds me back is the fact iMessage still feels leagues above how SMS and texting works on andriod. Many subtle features of Imessage really make a difference with the overall texting experience on the iPhone over the long term.
Most of my friends do not have whatsapp, and we are constantly sharing photos and video. Regular SMS can barely send pictures let alone videos, and I don't want to go through other means to simply shoot my friend a quick video of something.
> iMessage still feels leagues above how SMS and texting works on andriod
iMessage is a messaging app that uses internet. It should be compared to WhatsApp, Telegram, Signal, WeChat, FB Messenger.
It doesn't make sense to compare iMessage to SMS. Because nobody uses SMS as their main communication method. The only people who ever use SMS are iPhone users when they text Android users.
All this confusion comes from the fact that Apple merged their messaging app with the SMS app. This creates the illusion that iMessaging isn't just a messaging app but it is actually something built on top of SMS. It's just a messaging app that is not cross-platform.
> I don't want to go through other means to simply shoot my friend a quick video of something.
By "other means", do you mean installing a new messaging app? I'm guessing this isn't about the effort of downloading an app but rather it's about getting used to a new app and convincing your friends to switch to it.
“convincing your friends to switch to it” is the real issue. Or remembering which platform various friends use and keeping that straight. And when you have multiple friends who prefer different platforms, figuring out the intersection of messaging platforms that they all use. Or when you have a friend on Facebook messenger but you don’t know their phone number, or a friend who can’t have WeChat installed on their corporate phone, or any number of other situations.
This blue coloured bubble thing I've never heard anyone outside of Silicon Valley engineers/media mention, but I've heard it extrapolated as a deciding factor to the rest of the world by those same people many times.
I don't think most people actually give a shit about the bubble colour.
For me it's the fact that you can generally see the other person typing in an iMessage, while you can't in normal SMS.
That tells me whether the person is replying right now, so it's worth keeping my phone out, or busy and therefore won't be getting back to me for a while.
No. iMessage has the low entry barrier of SMS, but has features like a real chat app. SMS is not an iMessage replacement. Instead you have to get everyone you message with to change to something like WhatsApp.
iMessage is a vestigial product from the days when people had unlimited data but also only 400 text messages a month. It made sense then. It doesn't make much sense now.
It still makes sense over SMS/MMS which are terrible protocols in comparison. Even if you get unlimited messaging now, group and media messages are still remarkably unreliable.
iMessage is not a protocol, it's an app. Nobody uses SMS because everybody uses a messaging app. It comes down to whether you choose to use a cross-platform one or an iOS-exclusive one.
I think it is less that we have more SMS allotment and more that nobody wants to actually use them anymore, because they're insecure / don't work on wifi / are slow / etc.
There were a billion people you could send messages to, and receive messages from. That’s what got them of the ground.
A system where you can’t message half your friends because they bought their phone from a different manufacturer would have had more trouble getting traction.
I think that in America most people still use SMS, but in Europe barely anyone I know does, so RCS is basically a non-starter. It's all WhatsApp and Telegram (and some Signal). Like my kids sports teams coordinate everything through WhatsApp so I'm basically forced to use it. Would love to see an alternative, but getting everyone to switch would require a lot.
It's a question of who comes first- Linux was first to market as a cloud OS, so it dominated. Email came first for long-form asynchronous messages, so it dominates.
I think people should get in the habit of deleting all their convos that are more than a month old. Most convos are 99.99% worthless, and it's very freeing to be rid of them.
One of my friends had her H1B cancelled and was given a 10 year ban from entering the US because of her text messages. When she crossed the border back into the US, the border guards requested her phone, and searched it and found messages to her ex-bf about marrying him for a green card. She was joking but it didn't matter and she had to fly back to her country of origin that she didn't live in since she was a kid. Since then, even though I don't have anything to worry about, as a habit I delete all my apps except iMessage and delete all my conversations there.
Yup auto delete your old messages, social media and downloads. Forward anything importantly email and archive that locally; or photos and self hosted blogs. That works for me.
If you have Messages set up on a Mac you can access your chat archives under ~/Library/Messages/Archive. Each conversation is stored by date in an .ichat file which can be re-opened in the Messages app.
1) I do clean installs when I update macOS, so the Library folder is created new each time. Which means that archived messages only go back to that point. (I browsed via Terminal and used the command line 'open' command to view them.)
I do retain old ~/Library folders and it's possible to use the same technique in finding messages there.
2) But, and here's the big limitation: at a quick glance, they're simply individual messages, by date. There's no way to follow entire threads. I suppose it would be possible to write your own program to organize these. But the point of the article is that Apple should make it easy. Your method isn't easy.
Edit: response to sibling comment. I'm still on Catalina and the above works. I have no idea about Big Sur.
Searching Messages on my Mac is a joke. If type the name of a friend in the search field, it returns (after unreasonable delay) various results, mostly of texts about the person I'm searching for.
How does it not realize that it should first search recipient names and present these results first? A search for "Delilah" should not show a bunch of texts about Delilah, obscuring the one thread with my friend Delilah.
So, I just tried this in Messages on Big Sur. If I search for a name, I get a list of existing-conversations that the person with that name is in at the top of the results, then the list of text results within conversations.
Messages on Big Sur is (basically) the iOS Messages app ported over with Catalyst, so it has a lot of improvements compared to the highly-stale app it replaced.
I think there’s a separate idiomatic autocomplete flow for what you’re trying to do. Try hitting “New Message”; typing your friend’s name; and selecting the result. The New Message window will transform into the existing-conversation window for that contact.
Not saying having it also work from search wouldn’t be good — but having this flow available is probably why the other issue hasn’t been fixed sooner.
> ry hitting “New Message”; typing your friend’s name; and selecting the result. The New Message window will transform into the existing-conversation window for that contact.
I think this is how MS teams chat works too. It kind of makes sense, because it has a workflow for solving the "do you want your existing chat with Alice, or are you trying to start a new group chat with Alice & Bob" .
It does make sense if you're looking to send a new message. In my case, I'm typically doing a search so I can find information/photos from that thread. In this context, it's not so intuitive, and it never occurred to me to even try.
If Apple is focusing on “services” now, as numerous tech/biz pundits have talking about for quite some time now, it doesn’t show in their tech and software offerings, except for the gloss in the presentations and the painfully obvious exaggerated wordings.
With every iOS and Mac OS release I’ve been hoping they’d fix iMessage, iCloud, App Store search, app discovery, extension ecosystem for browsers, or my iPhone taking those extra 7-8 phantom GBs (..iPhone Storage/Others) that everybody guesses could be some cache or logs or something magically else and speculates maybe one day it’ll go away but nobody knows for sure etc.
But no! It seems Apple has been saving these things so that after 10 years they get to say on stage “Now you can search apps like never before. We have named this patented revolutionary app discovery as iDiscover™. As simple as searching for apps!“
Or after another 5 years something like “Now you can read your iMessages in a more readable manner and it will truly sync now. Meet TrueSync™“ when their engineers will have a breakthrough after 17 years of backbreaking research.
I have always wondered how the engineers look in the mirror after hearing those pathetic stage ridiculousness.
It seems like a lot of iCloud storage-using services are conveniently designed to just keep eating up more and more space. Take photos for example: most people only need to archive old photos and rarely access them, but such a use case isn't really supported by iCloud photos. You either keep your entire collection in the cloud and in a state of fast (expensive) retrieval, or not at all.
For iMessages every photo or video you sent seems to take up separate storage in the iMessage portion of iCloud, and it's difficult to delete because you have to scroll through and select them.
Dropping in here to say that as a programmer I'm always super impressed with how easy it is to go back many years in WhatsApp via find and quickly scroll around it. Always makes me wonder if I'm doing something wrong in my client-server app which struggles at the tens of thousands scale and if I'd be better off re-architecting to replicate all data to the client and just using the insanely powerful CPUs that are available there. I'm sure if I did it in my high scale, highly mutable (esp access control), high user count application replicating all that state reliably would also become a nightmare, whereas whatsapp has a pretty clear replication model built in.
I wish I could have an ms outlook like interface for the long term storage of the messages. Folder per contact or group and excellent search, sort and filtering. Also could be stored offline or synced up with a mobile app.
They are priceless catalogs of entire relationships. It feels flimsy, or maybe flippant, to not be able to properly search and sort or archive.
Maybe not managed like files, maybe you’re right, but it seems like if we can make trillion dollar tech companies we should be able to completely solve this particular issue.
I’d frame texts as one of the most intimate expressions of dialog and journaling between people and families and friends. Just seem odd that this isn’t better.
Flexibits developed an app to do this search called Chatology (https://flexibits.com/chatology) but they've discontinued it since Big Sur came out. Works pretty well on my Mojave machine.
It unfortunately doesn't work well if you've enabled Messages in iCloud [1]. You can only search the messages that're in your device's local cache. Which isn't actively cleared much, that I've seen, but it does mean that it's not very helpful for any messages that predate the current install of your OS.
I'd assume this is why they discontinued it, rather than adapt it to the new from-iOS Messages in Big Sur.
Can’t you use something like iMazing (I’m not a paid shill just a satisfied user and there are many similar other products from competitors) and do an offline dump from the backup?
Article mentions this: “...there’s a whole slew of third party tools that slurp up your iMessages, and Apple should care about this, because having a strong demand for apps that process users’ messages should be seen as a privacy failure by the company.”
Many comments here are suggesting you can look at the DB on a Mac. What about people who have Windows computers? Is there a way to extract that information?
I also have a similar gripe with the browser. How can I sync my Safari tabs that are open on my iPhone with my PC?
The complete iMessage history in your backups is also not e2e encrypted, so all of it is readable by Apple. They regularly turn this information over to US federal authorities without a warrant (so, no probable cause) too.
Having such a huge trove of history of all of your communications saved that can be accessed by the police or a rogue Apple sysadmin at any time is something that I feel should even make the law-abiding nervous.
The encryption key for “Messages in iCloud” (a cross-device sync service) gets backed up in the iCloud Backup (which is, as discussed, not e2e and thus readable by Apple), however, backdooring the e2e nature of Messages in iCloud by escrowing the key to Apple in the backup.
If MIC is enabled, iCloud Backup backs up the key, breaking the e2e. If MIC is disabled, iCloud Backup backs up the plaintext message history, breaking the e2e. This is well documented in the Apple KB article 202303 on the topic.
> Messages in iCloud also uses end-to-end encryption. If you have iCloud Backup turned on, your backup includes a copy of the key protecting your Messages. This ensures you can recover your Messages if you lose access to iCloud Keychain and your trusted devices. When you turn off iCloud Backup, a new key is generated on your device to protect future messages and isn't stored by Apple.
It’s written in a misleading style, but if you read it carefully it is apparent what is happening.
Turning off iCloud Backup is insufficient, as each conversation is backed up twice: once from each side. Your conversation partners’ devices default iCloud Backup to on.
Apple has the ability to read almost every single iMessage sent or received, on a 12-24 hour lag (iCloud Backups happen overnight), from data already on their servers, without ever touching a customer phone. All of the encryption is a handwavey distraction.
arguably the worst part about imessage and scrolling is that even if you do manage to scroll up and find a previous message, you are automatically scrolled down to the bottom if a new message is received in that chat (on iphone)
yes it is really bad, the UIs for PMs was better 10 years ago, thinking about ICQ, Adium, MSN Messenger and the like. You could export history as file and easily scroll and search them
Google Voice has the same problem. The search function is useless. It does not return results that should match, and it does return results that shouldn't. Scrolling back through just a few months worth of messages can take 10-20 minutes. I wish I knew why it was so awful.
No, I am referring to Google's SMS platform. It interfaces to nearly all other carriers, and is accessible through the "Voice" app or the web page at https://voice.google.com
Tell Voice to send your messages to email, and they'll be searchable from there. Use a query to ensure that you're searching text messages from Voice based on from address or subject. It won't help with searching messages that you've sent, unfortunately, not will it help with messages from before you turned the option on; I've had it on forever as I used to use it for desktop notifications.
I've had this sort of experience with many systems, which is why I decided to use .txt files as the base data store for my current project.
It is indexed in a relational database, and it is presented in a Web 1.1 frontend, but it all flows back through the text files, and they are always available for download, optionally in zip or tar.gz archives.
SQLite is great, and I use it for caching and for providing data summaries... but it is not near-universally compatible and accessible, human-readable, the way text-files are.
I want a chat app that is: E2EE and allows me to change phone numbers.
I started migrating people to Signal when that whole WhatsApp ToS hullaballoo happened, but now I need to change my phone number and apparently this is not supported in Signal, which is just crazy to me.
With photos, I guess this could end up being around 1GB. Maybe this will push you off the free plan to the $1/mo plan, but I don't see it being that big of a needle-mover for services revenue.
I made the mistake of turning on automatic photo back up (I have no idea how to bulk copy photos off the iphone without a mac, I've given them up as lost honestly, most of them are random crap anyway) and now my icloud account is completely full and I keep getting told I need to send Apple money.
I definitely prefer my current phone that lets me just rsync ~/Pictures.
I paid $0.99 for photosync. It syncs to many cloud providers, and (most importantly to me) to ssh/sftp.
You can set it to sync whenever your phone is inside a particular location boundary (e.g. at home, where there is wifi), so it works transparently with zero ports open to the internet, and no firewall-bypassing cloud services.
Not that I'm suggesting you should send Apple money: but it's like £0.79 per month for 50gb - less than £10/year. For me at least: that's pretty cheap compared to other services.
I have an awful problem with iMessage, that also speaks IMHO about the lack of care behind it. I upgraded an iPad Mini 1st. Gen (iOS 9.3.6) to an iPad Mini 5st. Gen (iOS at the time of exit I think), restoring from a backup.
iMessage never worked since the upgrade. I can only send messages, I see them once, but once I close the app and open it again, I don't see anything else. There's absolutely no way to reset the db of an Apple-included app unless I do a full format of the device, which after my full custom-configuration, I don't want to do. It sucks.
Apps that are not have a good export story should be considered as keeping your content hostage. Either treat these as ephemeral, syndicate the content into them or switch. Your content in their app is lock in.
Who uses iMessage? I mean, I never seen one using it seriously. All my friends and family members uses WhatsApp (that has more or less the same problems, but at least you can export a conversation in a text file, or now import it in Telegram), and Telegram is nowadays used by all the young people.
This is probably due to the fact that in my country the majority of people has Android phones, and thus even who has an iPhone uses the same app that uses with people with Android to chat with other Apple users.
The AI has trouble knowing what is important in the conversation. It seems like such difficulty creates an incentive for a user to ask someone something again instead of looking it up in history (although this also may be a signal for AI that something in the history is important).
Other reason may be that it is used by services - they want as long history as possible to find out what kind of person you are in case you are being investigated.
Anyway these artificial obstacles most often have sinister motives.
Imagine buying a top of the line smartphone in 2021 and it comes with 64GB of storage. Truly Apple has done wonders with their marketing that people buy their phones.
The cynic in me says this is all part of the plan. Sure, we'll give you seemingly impossibly large amounts of cloud storage for free, learn to never delete anything that you ever talk about in your entire life.
Ten years down the road, "oh, looks like you need to start paying us monthly for more cloud storage!"
I kinda wonder how many Gmail users now pay Google for more storage space just to cover their email archive... And how many will in another ten years.
> I kinda wonder how many Gmail users now pay Google for more storage space just to cover their email archive...
Gmail’s storage space has consistently grown over the years to keep up with the average user’s demand.
Google don’t want to do anything — like charging for email storage — that would disincentivize their mass of sensors^H users from feeding them ever more data.
(If you’re an outlier with way more storage than usual, they’ll happily charge you, knowing that the data point you represent is mostly going to get pruned out of their models anyway.)
This was previously true. Free storage at Google hasn't increased from 15 GB since 2013. (I have a few promotional bonuses that bring me up to about 19 GB, but it's been that way for... over half a decade without increase.) Personally, my Google storage is mostly full, even though I haven't used Google Drive or Gmail since 2016, and have removed much of my personal content since then.
Also, now they're actually effectively decreasing everyone's storage. Previously, Google Drive document types and Google Photos didn't count towards that 15 GB limit. They will as of June 1st, 2021: https://blog.google/products/photos/storage-policy-update/
So, while cloud storage usage has exploded over the past eight years, Google's storage limits have not only not expanded to meet it, they've actually begun to tighten the belt.
They don’t need that, they can just start saving pics in HEIC at 4MB per pic, and backups start exploding.
I use Gimp and generally obtain the same quality with a 450KB JPEG, sometimes 50KB depending on the contents. Images can be remarkably small. They should try increasing the compression ratio until a change is noticeable, but do they care?
Not sure if that's available outside the EU as well, but over here you can export an archive with all data they have on you including messages and attached media. I'm sure they leave out things but I use it to backup my chats.
FB Messenger does have data takeout, it comes in some kind of JSON format. Be prepared for some surprisingly massive file downloads though -- my chat history alone was over 50gb!
I recently had to restore from an iCloud backup that contains all my messages (which I never and will never delete) and it took over 24 hours to restore them.
The restore daemon actually downloads every single image and attachment you’ve ever sent or received one file at a time.
Not that I can scroll back that far, but you never know when you need to search for something.
The comparison to Photos in TFA is apt. Apple could copy the most relevant subset of features, in particular the Years/Months/Days/All navigation views, and it would dramatically improve the utility of iMessage for old messages.
Improving the abysmal search result quality is probably more important, though.
I'm not sure if it even works anymore or if there's a better option, but I've used PhoneView <https://www.ecamm.com/mac/phoneview/> in the past to make a PDF archive of my iMessages.
my biggest problem with iOS storage is that if you pay to store a photo in iCloud and then use Messages to send it to 5 friends, you are now paying to store it in iCloud 6x. And of course it’s totally unfair that there is no priced option between 200GB and 1TB. Same with Google Photos.
Kinda dying to know if they dedup on the server? Cause technically speaking if everyone was on Apple then that 1 photo becomes 12 not to mention it getting send to another layer of people.
I've used a mess of find and rm to clear up space on my local drive, but afaik they'll still take up space in iCloud storage -- I agree that there should be a way to more efficiently discard them, and that automatically deleting old ones is not the right thing to do
Tangential: Telegram has a great search feature. It’s super fast, allows to visit messages by date and in group chats you can even see the list of people who feature in the keyword search results. The UX is great, and all chat apps should just imitate it as a start.
Try Google Messages. How do I export them? There's no chat.db. It's impossible scrape, unless you wait(5) and scrollTo() etc. I've found no good solution for extracting/backing up Google Messages (outside of Google's dominion).
The Bible contains about 3.5 million characters and an average Bible weighs about 500 grams. 11 GB of conversations would then be about 1.5 tons of Bibles.
You'd be surprised how few engineers work on apple projects relative to their size, and how apple underpays compared to other big tech companies like FB or Google unless your working on the current hot project.
> Texts has integrations for all major messaging platforms...
Any plans to include phone calls, voice mails, and email too?
Future perfect:
RSS reader-like UI. Top level is Contacts. Including groups (w/ aliases). Drill down to see missives. Show read, unread, return receipt, etc. Sparklines representing activity.
But, I'll be honest; if you start to see all of Apple's decisions through the lens of "business decisions", you'll tend to get pretty disenchanted and crestfallen (perhaps even resentful)
Once you have the schema figured out, it's dead easy to build a third-party client that works better than the official one. Even search works great with a simple LIKE query but Apple re-indexes all messages leading to your CPU going over 1000%: https://twitter.com/KrauseFx/status/1396433852126670852
Source: I built a third-party desktop client for iMessage at https://texts.com and reverse engineered the complete sqlite structure.