Apple's chat.db has an esoteric schema owing to the fact they never designed it from the ground up and instead kept adding new columns and tables with each macOS release. This makes their queries super complicated with multiple joins.
Once you have the schema figured out, it's dead easy to build a third-party client that works better than the official one. Even search works great with a simple LIKE query but Apple re-indexes all messages leading to your CPU going over 1000%: https://twitter.com/KrauseFx/status/1396433852126670852
Source: I built a third-party desktop client for iMessage at https://texts.com and reverse engineered the complete sqlite structure.
This is basically one taken out of microsoft's .doc format playbook.
At first I thought Apple was slow and just didn't have time to get to some sort of export for iMessage chats. Mail had imap, calendar had caldav, address book had carddav. The original mac stuff did other standards like AIM and jabber.
But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
What's annoying is that I couldn't save my messages, I could only upgrade to a bigger iphone.
What's downright evil is all the people - I know several - who have lost messages + embedded photos/videos from loved ones who have died because there was no export.
>But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
In an SQLiteDB any programmer can learn to navigate, and any third party can build a tool to export them?
Sorry but the "internal emails" prove nothing about Messages. It's just a quick and dirty implementation that "works good enough" for reading the latest, but not as an archive.
Yeah I agree. The lack of a good schema/export function sounds to me like negligence not maliciousness. It’s a different issue from only allowing Apple-tech-users to use iMessage
> But over time I began to suspect, then later those internal emails exposed in the lawsuits made it clear - apple was holding everyone's messages hostage.
Any source for that? I couldn't find any direct evidence for your claim.
Thanks, that's interesting, I didn't know that. On the one hand, I shouldn't be surprised since vendor lock-in is the standard way tech companies try to gain corporate advantage. On the other hand, it's kind of depressing to learn once again that customers count very little and no matter what these companies say, they will always try to abuse us in every way the law allows.
Just took a look at your product, I'm very impressed and have added myself to the waitlist. I'm almost even more impressed by the domain name itself though - how on earth did you manage to bag that domain?!
The fact so many people on this site can throw something functional together without much effort is even more disappointing apple doesn't do anything about it.
iPhone is the only apple device I own but it crashes it can't go too far back into my messaging history for any one individual too far.
Usually once I scroll to ~5 months back with people I text regularly the iMessage app just straight crashes ... it's a little absurd. To top it off, every couple pages it gives a loading icon and introduces severe latency for whatever reason. There don't seem to be any apps in their App Store that let you search and lookup old messages either.
This is actively biting me right now. Are there any solutions for accessing the iMessage/chat.db database on my phone or via iCloud from a Linux system (Ubuntu in my case)?
A little feedback on the landing page: Almost turned away because I thought „Sign up with Google“ was the only choice and I missed the little „Prefer entering email instead?“ below. I just prefer not having a Google account ;)
Maybe the styling of that link could be adjusted so that it isn’t glanced over that easily.
I don't mean to be pessimistic , but I wonder how you're going to be anything but reactionary to apples changes to the database. I've written a similar (much simpler) version of what you're proposing to sell- the database queries are hysterical.
I've been thinking about using the accessibility I framework to ETL the messages into Kinesis to facilitate event driven architecture. 'archived' Messages and attachments are stored as bsob.
Hell, I've even experimented with some goofy stuff, like FaceTime to VOIP- asterisk is just so flexible
Looking over your site, I'm curious about a few things:
- The FAQ says you'll eventually open source your SDK. I admire that as a software engineer, but from a business perspective, isn't this risky? It seems like--if you don't pass messages through an intermediary service--your value-add is mostly in this SDK + your design. All of these elements are copy-able.
- The jobs page includes a backend engineer, so what will go through your backend, if not any message data?
- Are there legal concerns about reverse-engineering these APIs and using them? Some of your supported chat services have open APIs, but others seem intentionally closed. (And a couple parent companies seem downright hostile to 3rd party use.)
Source: I built a third-party desktop client for iMessage at https://texts.com
Does the texts.com client solve the original problem? Does it make it easy to go back and view all those old text messages? I know the website says "from forever ago easily". But you didn't say that explicitly in your post.
I assume you're being modest and not "pushing" your solution. But if it simply solves the problem, maybe you should be shouting this out loud!
A piece of feedback: it would be nice to know what the price is on your site. You say "We make money by charging you a monthly subscription," but there's no pricing.
I've built an iMessage chat.db reader as well. To me the most astounding thing is that chat.db is unencrypted sqlite. Anyone who can see your home folder can read all your private chats or upload the chat.db to be read somewhere else.
Which you end up applying to random things like ruby because Emacs.app uses a ruby-based launcher and anything less than Full Disk Access breaks dired.
Why would a lack of encryption make it not reverse engineering? There's a database schema, it was engineered* and used to build a database. They went from the live data back to the schema.
*For some definitions of "engineered", apparently.
Very eager to see how your app works when it opens up to the public. Are there ways to provide assurances for your end-to-end encryption which may be digested by experts?
You also mention on your website that end-to-end only works when the platform "supports" it — does that include Windows with your currently always-on Mac solution?
> Very eager to see how your app works when it opens up to the public. Are there ways to provide assurances for your end-to-end encryption which may be digested by experts?
We recommend using a simple MITM proxy or a firewall app like Little Snitch / LuLu to inspect the network traffic.
> You also mention on your website that end-to-end only works when the platform "supports" it — does that include Windows with your currently always-on Mac solution?
Yes – there's a peer-to-peer connection between the Windows and Mac devices.
I had done something too, but I also needed a Manifest.mbdb parser to reconstruct backup to disk, then process the messages and copy all the images around to make it work. I did it mostly to make a simple-to-read archive of chats before deleting them.
(You need an unencrypted backup directory to run it over)
Note: It looks like the flat directory structure of the backup is now 256 dirs prefixed by first two characters of file name. I'm not sure, but a small change to the name "sha_file" may be needed to just path.join that with the first two chars of the sha.
Python 2 but can be ported forward without much effort. Manifest.mbdb is now Manifest.plist (there's a sqlite thing now too). timestamp sql might not work so great, would need to check query.
Yes, with a catch: it requires an always on Mac (which can be in the cloud) or a jailbroken iOS device. We plan to make it work without the catch.
I've worked with 10+ messaging platforms so far and Apple's protocol is the most obfuscated and complicated to reverse engineer. Apple has invested millions of dollars to make the iMessage protocol super hard to reverse engineer. It's how they sell tons of iPhones after all:
> However, Craig Federighi, Apple’s Senior Vice President of Software Engineering and the executive in charge of iOS, feared that “iMessage on Android would simply serve to remove [an] obstacle to iPhone families giving their kids Android phones”. (https://www.androidpolice.com/2021/04/28/apple-admits-that-i...)
Looks like a super cool project. I'm always happy to see new client-side software for communication protocols written that aims to improve upon built-in! And iMessages in particular could certainly use it. With the demise of iTunes, it now feels like iMessages is probably one of the most crufty-but-heavily-utilized user facing pieces of software Apple puts out. Maybe they have some long term plans to refactor it ala iTunes but yeesh.
This bit did make me wonder though:
>We plan to make it work without the catch.
Out of curiosity, do really think that's realistic, or even desirable long term? iMessage is ultimately an Apple service that runs heavily on Apple's infrastructure, and is directly subsidized by sales of their highly vertically integrated hardware platforms. If it turns into a cat-and-mouse fight it seems like they're always going to have the eternal upper hand, which in turn seems like it'd make for a subpar user experience (ie., breaks randomly which for an instant messaging service would be pretty bad). Also seems like they might actually be motivated to respond rather than ignore it since it'd actually be directly leeching their infra if it will work on PCs/Android without a Mac/iDevice purchase in the equation (unlike hackintoshes for example, where whatever debate there is to be had about probably very low "lost sales" it doesn't actually cost them anything).
Obviously you've probably thought this all through, but seems like just requiring an old cheap Mac or old jb'd idevice and thus avoiding Apple might be an easier path. Or alternately just stick to offering a flat out better client with the iMessage bit being Mac-only. Will be interested to see how it goes though!
> one of the most crufty-but-heavily-utilized user facing pieces of software Apple puts out. Maybe they have some long term plans to refactor it ala iTunes but yeesh.
The Messages app on Mac was rewritten for Big Sur (last year). It gained some features but also some bugs in the process.
Likely because each iMessage receiving device has a private key to decrypt the messages, and senders encrypt the iMessage against all the keys in your key bag.
To access iCloud and or services you also need to have a device security key that is tied to genuine hardware.
Once you have the schema figured out, it's dead easy to build a third-party client that works better than the official one. Even search works great with a simple LIKE query but Apple re-indexes all messages leading to your CPU going over 1000%: https://twitter.com/KrauseFx/status/1396433852126670852
Source: I built a third-party desktop client for iMessage at https://texts.com and reverse engineered the complete sqlite structure.