Time Machine backups should live on Apple Filesystems. The Ext4 or btrfs in Synology's is a timebomb for Apple's proprietary data blobs. It has something to do with the filesystem attributes.
I asked my father to just give up on it. Just manage pictures through the file system. Photos (former iPhotos) databases also get corrupt on a Synology NAS. It's a matter of time. The strange thing is that it can go well for months on end, giving you a false sense of security.
Do iCloud or local drive backups or stay away from Time Machine and Apple Databases is my advice (although my father had a lot of problems as well with an "incorrectly unplugged" external HFS+ drive). I can't find the sources right now but after my father's last drama (and there have been several) I did some intensive searching and this was my conclusion.
Filesystem likely doesn’t have anything to do with it.
What’s more likely, is Apple’s notoriously unreliable implementation of SMB causing the problem (and that’s the only option now that AFP support on Mac is dead)
I have a Synology DS220+ and connecting to it from a Windows machine vs a MacOS machine is like night and day.
On Windows, it literally feels like the NAS is an extension of my local hard drive. Browsing huge directories of thumbnails is snappy, file and folder names appear instantly. It’s a dream.
On MacOS, connecting to anything over SMB is a total nightmare. Aside from the constant mounting and unmounting (fun!), it’s just plain unreliable and slow.
And people have been complaining about this for years.
What’s even more funny, I have a friend who works for Apple and apparently they use NAS storage in some teams and deal with the exact same annoyances!
If Apple’s own employees have this problem, it’s hopeless they'll ever fix it for customers.
I’d put my money on Apple’s SMB implementation being the root cause of this file corruption issue that has been all over the Reddit Synology user forums lately.
The Time Machine over NAS situation is very frustrating.
It used to be over Apple's proprietary AFP protocol. With the exception of Apple's now-discontinued Time Capsule product line, all NAS implement AFP using the open-source Netatalk, presumably with reverse-engineered AFP protocol. And it's unreliable.
With recent versions of macOS, Time Machine switches to SMB protocol. Apple has a custom SMB implementation, and all NAS use Samba. And it's still unreliable!
I guess the only reliable solution to use Time Machine over network is to use a Mac with File Sharing over SMB enabled. At least both ends run Apple's SMB implementation.
Compound this issue with not being able to disable spotlight indexing of time machine backups on a NAS. Often I find mds is indexing for 4-6 hours at a time, using 20-40% cpu, with no way to stop or disable it (aside from disabling time machine). Its a real shame that Apple is allowing a growing list of paper cuts to fester on an otherwise pretty solid os (looking at you thunderbolt display kernel panics, unreliable external display support, etc).
Cool TIL about mdutil. I still cannot disable it as mdutil seems to be having trouble parsing the path to the share. Ive tried in bash/zsh/sh with all quoting/escaping patterns I can think of and I must be missing something.
(shows up in in `mdutil -s -a`)
/Volumes/Backups of Saucy's MacBook Pro:
Indexing enabled.
sudo mdutil -i off /Volumes/Backups\ of\ Saucy's\ MacBook\ Pro
Error: could not resolve path `/Volumes/Backups of Saucy’s MacBook Pro'.
In my experience, if you use a file share consistently, it becomes stale over time and has to be remounted.
One thing that I’ve noticed with Apple is that there is a “happy path” they they design and test. If you happen upon the magic combo, you’re good, if you’re off the path, you are on untested ground. My guess is that Apple tests against a specific Windows or samba smb version/config, and doesn’t look for any regressions outside of that.
I had a Time Capsule. It had the same issue with Time Machine rejecting the backup and requiring a re-do every year or so. (the Time Capsule ran NetBSD, I wonder what they used for a AFB/SMB stack. Did they port the macOS one over or use Netatalk/Samba?)
Time Machine over a network has always been unreliable.
Time capsule was apple’s wireless router with built in hard drive for backups (time machine)if people don’t know.
I had one too and reading this article was wondering how reliably that worked. (I think thankfully I only had to do some very minor “ get back previous file” on it. I remember it being slow..)
Time Machine's "Floating Time Tunnel" user interface for browsing backups and restoring files is such a useless pretentious piece of shit. I DO NOT CARE for it taking over the entire screen with its idiotic animation, that prevents me from browsing current Finder folders at the same time or DOING ANYTHING ELSE like looking at a list of files I want to retrieve on the same screen.
It even sadistically blacks out every other connected display, and disables Alt-Tab, as if it was so fucking important that it had to lock you out of the rest of your system while you use it.
You can't just quickly Alt-Tab to flip back to another app to check something before deciding which file to restore and then Alt-Tab back to where you were. No, that would be too easy, and you'd miss out on all that great full screen animation. It not only takes a long time to start up and play its opening animations, but when you cancel it, it SLOWLY animates and cross fades back to the starting place, so you LOSE the time and location context that you laboriously browsed to, and then you have to take even more time and effort to get back to where you just were.
It was designed by a bunch of newly graduated Trump University graphics designers on cocaine, with absolutely NO knowledge or care in the world about usability or ergonomics or usefulness, who only wanted to have something flashy and shiny to buff up their portfolios and blog about, and now we're all STUCK with it, at our peril.
Crucial system utilities should not be designed to look and operate like video games, and turn a powerful mutitasking Unix operating system interface into a single tasking Playstation game interface. ESPECIALLY not backup utilities. There is absolutely no reason it needs to take over the entire screen and lock out all other programs, and have such a ridiculously gimmicky and useless user interface.
Whatever the fuck is wrong with Apple has been very very wrong since the inception of Time Machine and is STILL very wrong. How can you "Think Different" if you're not bothering to think at all?
>Core Animation will allow programmers to give their applications flashy, animated interfaces. Some developers think Core Animation is so important, it will usher in the biggest changes to computer interfaces since the original Mac shipped three decades ago.
>"The revolution coming with Core Animation is akin to the one that came from the original Mac in 1984," says Wil Shipley, developer of the personal media-cataloging application Delicious Library. "We're going to see a whole new world of user-interface metaphors with Core Animation."
>Shipley predicts that Core Animation will kick-start a new era of interface experimentation, and may lead to an entirely new visual language for designing desktop interfaces. The traditional desktop may become a multilayered three-dimensional environment where windows flip around or zoom in and out. Double-clicks and keystrokes could give way to mouse gestures and other forms of complex user input.
>The Core Animation "revolution" is already starting to happen. Apple's iPhone at the end of the month will see people using their fingers to flip through media libraries, and pinching their fingers together to resize photos.
>The "Delicious generation" is a breed of young developers who embrace interface experimentation and brash marketing. The term "Delicious generation" was meant as an insult, but they wear it as a badge of honor.
>Image: Adam BettsShipley's initial release of Delicious Library, with its glossy, highly refined interface, gave birth to a new breed of developers dubbed the "Delicious generation." For these Mac developers, interface experimentation is one of the big appeals of programming.
[...]
>Apple has been ignoring its own HIG for some time in applications like QuickTime, and is abandoning them completely in upcoming Leopard applications like Time Machine.
>Functionality-wise, Time Machine is a banal program -- a content-version-control system that makes periodic, automated backups of a computer's hard drive.
>But Apple's take on the age-old task of incremental backups features a 3-D visual browser that allows users to move forward and backward through time using a virtual "time tunnel" reminiscent of a Doctor Who title sequence. It's completely unlike any interface currently used in Mac OS X.
[...]
>While it seems logical to speculate that interfaces like those of Time Machine and Spaces will lead to the end of the familiar "window" framework for desktop applications altogether, many Mac developers predict that the most basic elements of the current user interface forms won't disappear entirely.
Not sure if the tone gets you downvoted but I can completely feel your rage…
I thought the animation was intentionally there to keep you engaged and hide the fact that Time Machine restoring is super slow, especially over network.
Sorry for my frustrated tone, I've been grinding my gears about that for many years. Backups are not a game. The fact that I always have to "play the Time Machine Game" when under the stress of needing to retrieve something from my backup tends to make me pretty angry at the interface, yes!
Even if Time Machine were 100% reliable and didn't randomly trash your backup all the time, asking users to wait through all those gratuitous vanglorious Doctor Who animations to find out whether or not they're screwed is not very "Delicious".
If only they'd applied all that unbridled creativity to something harmless like the About This Mac box instead of the backup interface.
I have a media server I keep movies and TV shows on, and connect to Samba from my Mac. I agree, the performance is ridiculously bad. Part of it, I suspect, is Finder.
I wonder if NFS works any better? Or maybe Apple's old AFS/AFP, which I think used to be more solid than SMB on Macs? Though I read something about Apple deprecating (or removing?) AFP support recently.
From my long experience building high-end NAS servers and struggling with Macs, generally using 10GigE connections:
Using SMB, the typical Mac pro can't do much better than 150/200MB/s to/from the NAS.
The very same Mac booted on Windows via Boot Camp reads/writes at 1GB/s on the same NAS.
Back to MacOS, using either NFS or AFP the Mac easily reads/writes at 1GB/s on the NAS.
The SMB implementation of MacOS is utterly broken, and has been for ages. NFS works fine generally, but some programs such as Quicktime pro and more annoyingly, the Finder sometimes have trouble with it.
Unfortunately, the fastest and most reliable option by a large margin still is AFP, using Netatalk. If you take care of cleaning regularly the CNID database, it works like a charm. I have many customers using servers with hundreds of TB of storage with AFP and it just works.
There's no SMB on my home network at all, since I don't have any Windows machines to support. For file storage/access, I use NFS which works beautifully across my Macs and Linux boxes. Speeds are appropriate for a 1Gbps link. For Time Machine, it's netatalk pointed at a HFS+ volume:
[Time Machine]
path = /mnt/backups
time machine = yes
This setup has been rock-solid for close to a decade, and persisted across different NAS hardware. Current NAS is a Buffalo Terastation minus it's crappy built-in software, plus a barebones Debian install.
They do, under the hood. (or did at one point.) the problem is it doesn't work. it's functional, but just doesn't give the level of performance needed at the top end (where wifi isn't a good enough connection).
They used to, a very long time ago, and when they did it was an outdated version. Nowadays they have a proprietary alternative that sucks.
Modern Samba is limited by the TCP stack. You can get 10gbps to work as long as your TCP stack is performant enough with very basic tuning on the server side. The client "just works".
macOS stopped shipping Samba with 10.7 Lion. Even before Samba was outdated.
Not to get political but I always thought it was a mistake to make "modern" samba GPLv3. All it did was make all these hardware vendors either stick with the older GPLv2 stuff or... I dunno what else they can do besides write their own implementation.
Samba is too much of an "infrastructure" codebase. It should have been BSD licensed. Vendors would have strong incentives to merge changes back into the mainline... none of them want to have their own wacko implementations.
Samba operating on the application layer and working the way it does means you can totally ship a computer that runs Samba with the immediate tooling around Samba also bring GPLv3 and the rest being proprietary.
Even when they did, somewhat 10+ years ago, it is still far from perfect for whatever reason. But no one cared much because we were all taught to just use AFP.
Ten years ago Apple abandoned Samba. But Samba back then was much worse than Samba today, which is of equal quality to the Windows Server stack but harder to configure.
Yes, you can use iscsi. I used iscsi initiator on macOS to make time machine backups on an exported volume from an opensolaris zfs tank. Worked fine for many years, although I no longer use that approach.
You have to install iscsi initiator on the Mac, since macOS doesn’t come with it.
Unfortunately, despite being a nightmare, SMB seems to be still be the best option for networked filesystems on macOS. AFP is now officially deprecated/unsupported, and the NFS client seems to be even worse than Samba/SMB. One last hope was macFUSE/sshfs, however this seemed also to be more or less broken when I tried it (extremely slow speeds, issues with disconnecting, etc).
I only rarely need to work with Macs, but Apple rewrote its SMB implementation sometime in the 10.12 or 10.13 timeframe (to avoid GPL licensing of the existing open-source one, I believe) and it's been terrible ever since. Even the OSS one wasn't great but I remember reliability was much improved when forced to CIFSv1 (the original simple-but-insecure protocol) than when using any of the newer versions.
...I suppose anyone who has the time to work on this problem could find the last open-source release that Apple used, and port it to the newer versions of macOS.
It works. and it is impressive work what they've done, but let's face it, it is not even close to being as ergonomically and plug and play as it is on a windows only environment.
It is a shame that microsoft still haven't truly make SMB an open protocol.
I will have to check this out. Do you believe that this will help iOS?
I have a Synology, which I tend to run a little behind updates, since it is not directly connected to the internet. Trying to look through files on an iPad or iPhone is ... painful. The search does not work in the slightest. It loses the connection. Scanning a directory takes forever and is not cached ...
You can’t downgrade and there are some issues with upgrading (eg takes some work to properly migrate some packages like Plex over), so upgrade carefully and thoughtfully, but it’s been fine for me.
> What’s more likely, is Apple’s notoriously unreliable implementation of SMB causing the problem (and that’s the only option now that AFP support on Mac is dead)
How is AFP support on Mac dead? I’m doing Time Machine backups to synology via AFP
> What’s more likely, is Apple’s notoriously unreliable implementation of SMB causing the problem (and that’s the only option now that AFP support on Mac is dead)
Are we sure this isn't a Synology issue? I'm all-SMB for both shares and Time Machine/CCC backups on a QNAP NAS and have never had an issue. (Caveat: I moved from Lightroom to Photos this year, and am now using iSCSI APFS volume for that.)
>And people have been complaining about this for years.
And we are now coming to 2022, let me say this has been the cases for decades. Their SMB implementation has gotten better in the past 5-6 years but it is still far from the fit and finish on windows and linux.
WebDAV is also really crappy on MacOS, I was sitting next to my father and my Ubuntu laptop with Gnome Files literally got 4x his MacOS performance connecting to my Nextcloud box.
Are these applications really adding so much value that it's worth it with their proprietary catalogs? My personal solution for photo storage is a normal directory tree with files. I have Piwigo set up to catalog those so that I can browse by shooting date and tags, but the file hierarchy stays untouched. I can pull files from there to edit with any application and there's a lot of proven tools to keep the files in good shape and backed up.
Yes, definitely. In the case of Lightroom, it is preserving your original input photo along with a list of edits and workflows you've applied to it. It also manages a database of preview images, so browsing full resolution albums is fast
There are no destructive edits in Lightroom unless you really go out of your way to cause the destruction
It also has client-side face recognition / clustering which relies on a local database, indexing by geographic location for GPS-tagged images, etc.
Essentially nobody needs Lightroom until they try it, after which it easily becomes impossible to live without and there is no replacement
I've used Capture One Pro for a similar amount of time and anything else (LR included) feels like a step down. Probably mostly inertia and familiarity by this point.
Yeah, my goal is to be able to run my next photo software natively on a linux box, and to my knowledge Capture One is only available on MacOS and Windows. It does look like great software, though.
I switched from only LR to only RawTherapee for all new shots going forward about a year ago. Never looked back. Might not be suitable for a professional expecting smooth conventional workflow, but excellent for an enthusiast who enjoys full control and nerding over raw data interpretation.
You don't need the Lightroom catalogue to do most of what it does, including caching previews. I strongly prefer to use Adobe Bridge and Adobe Camera Raw because I have access to nearly all the tools in Lightroom without its wonky and cumbersome catalogue management process.
If you're in the Apple ecosystem with an iPhone, some iPads and a Macbook, you sort of roll into it. Is it worth it? Not sure, it's sure easy and just works. But you're on your own when you try to interface with your data in "non conventional" ways.
I myself use NextCloud for everything, I recently moved from Android to iOS and it's nice to see most things working... except that NextCloud has issues making previews from .heic pictures (or I should say heic picture containers containing heif images coded in hevc :s), and so the drama starts again. It's always plug and pray outside the Apple ecosystem, always ymmv.
Haha, yes it felt wrong subconsciously to call it non-conventional. Now that you mention it, I totally agree that it is Apple who in the games of doing things the non-conventional way. The opposite of the Unix philosophy. "Take all things, throw them together and make sure they only work in one, blessed, linear way."
Every once in a while I'll make an external backup of my parents' photo collection. Every single time, it's a royal pain to figure out where Apple has decided to put the iPhoto directory. There's no UI that points to it, no settings to configure the save location. It's like you're supposed to forget that these are your pictures to access as you please, and instead treat them as iPhoto's pictures, only to be accessed in the blessed interface.
I love that I can quickly click on a person and then get all the photos with that person. The same for a location.
The "memories" slideshows that iPhone or gphoto generate are sometimes also very nice to see.
The search functionality also comes handy once in a while. So that I can search for pictures of certain things.
For sharing photos the shared albums are also very easy to work with. Both to create and the receiver to import any interesting photos to my/their library.
Giving up the file system is difficult but it’s a godsend. I hated losing the Events when moving from iPhoto to Photos, but in practice no one wants to organize photos and I certainly did not enjoy it, as anal I as I was.
Nowadays we don’t have events anymore, it’s just a continuous flow or random photos that may or may not belong to a specific event. The great part is that photos are always in chronological order and I never have to deal with “files” (copies, same names, etc).
The only exception are professionals and Photos.app definitely isn’t intended for them.
I too prefer chronological order instead of some "organized" collection of directories. My point is that this is something that a photo indexing app of some sort (Piwigo in my case) should do and leave the files be.
Like many here I had the exact same issue mentioned by the poster, Time Machine on Synology just kept failing, with incredibly unhelpful error messages that basically said, let's trash and backup again.
I used AFP which was recommended back then and that worked really well for years (since 2011). But since (maybe) Catalina, issues started creeping up and it would just randomly fail. It used to be once in a while, then it became a weekly occurence before I gave up.
Samba isn't better, my mounted shares get randomly disconnected overnight fairly often too (even now on Monterey), and switching from the old Synology to a fresh dedicated NAS machine didn't change a thing.
At that point I think in general it's just "local networking" that became less reliable around that time, whether it's some power saving feature, or something else up the stack, I don't know.
The only scenario where Time Machine works flawlessly for me is using an external SSD drive for backup, formatted as APFS. At least for now.
At this stage I think Time Machine is barely fit for purpose for backing up over the network. I've lost days on this issue over the years too.
I have always been totally confused as to whether I should be using AFP or SMB (tried both). As others have said, SMB often seems very unreliable, and AFP is supposedly being deprecated...
Mounting network shares from my Synology to Mac(s) is never flawless either. As other comments have noted, this experience is very much worse than what it used to be like in Windows (not that I've mounted network shares in Windows for a while).
And same exact experience on mounting shares by the way.
When setting up my new NAS last week, I ended up booting a Windows PC to check if Samba was correctly configured because macOS kept throwing weird inscrutable errors semi randomly.
I know it's a terrible protocol but it certainly became worse on macOS on the past 3 years-ish.
I have a couple of 6tb usb disks formatted as a Btrfs RAID 1 volume plugged into a Rpi4, it's not the fastest but it has been reliable for over a year. I always make sure I stop the backup before pulling the cord on my laptop though. My wife hadn't been doing this until recently and she has had more problems with both over the network and directly connected usb disks; she has both. At some point I'm going to create a duplicate setup in my house and send Btrfs snapshots over the internet to have an offsite backup, but I haven't got round to this yet. Currently our offsite backup is just taking a laptop home...
Time Machine backups on remote drives do live inside an Apple file system. They are stored as a mutable disk image.
The critical setting for reliability is to use AFP and not SMB. To this end I have two Synology NAS devices—a multi drive unit shared as SMB for general use and a single drive unit shared as AFP for Time Machine. While I do have occasional backup trees go bad (once every two or three years) the backups themselves are still fine and so I just start a fresh one.
That's not what Apple recommends. Time Machine is pretty shit these days. With their cloud/services strategy, it simply get abandoned.
I use time machine, but I don't trust it, so I have other solutions as well
It doesn't matter what Apple recommends—the recommendations are just dead wrong. AFP works and SMB does not. I religiously check backups and I know they are working.
Slackware is actually the one distro I've succeeded deeply with AFS. One thing that made everything less buggy is to find out what your UID , User ID, was, on the OSX UNIX-compatible system e.g. 1001, let's say, and just make sure the co-responging AFS user/share on the other end shared the same. - No bugged idea why it matters but within linux' AFS implementation paired with OS X it seems to be crucial for some reason, and a lot less headaches.
So therefore: user ' osxking ' with UID ' 1001 ' connects best to user ' osxking ' UID ' 1001 ' on your Slackware AFS server. Good luck man! It will work! < 3 Happy AFS'ing, Slackware served me well!
> The critical setting for reliability is to use AFP and not SMB.
I've been using Time Machine with a QNAP SMB target (for as long as Time Machine has supported SMB) without problems. Reading this thread makes me wonder if the problem is actually Synology NASs.
We tried it with AFP as well, did not help, after some time those time machine backups got corrupted.
We suspected that maybe AFP writes failed because computers were disconnected from network before buffer was written to the server. But there are no visual indicators for that and we did not want to debug it. We just switched to 3rd party backup solution.
I did use time machine on external USB HDD and after some time it got corrupted. Happened multiple times over the years.
Then I tried Time Machine using AFP and SMB on remote NAS. Those also got corrupted multiple times.
The lesson I learned was - do not use TimeMachine.
> It has something to do with the filesystem attributes.
Synology can do all the attributes the HFS+/APFS can do. They do not use standard samba *_xattr modules though, they use their own and the result is all the @eaDir stuff. Do not delete it!
On the other hand, Time Machine is perfectly capable of damaging its archive on the local drive just fine.
omg reading this made me relive that horror I encountered too, of course with pics of the little one.
Now I have 3 backups (of which 1 is just an export again of all the pictures in raw) in 2 locations, using Lightroom, I don't trust photos anymore, and yes on apple filesystems.
My solution was for the LR database to live on local SSD, and the library/catalog on an SMB share on a ZFS server, with weekly backup to AWS Glacier. Other than having to reconnect anytime I wake up the machine from sleep, it works pretty well, and I never have worried about corruption.
Only flaw in the plan AFAICT is that if a bug in LR or the Mac introduce corruption XFS will happily store the checksummed corrupted file. I should probably add ZFS snapshots.
I suggest using Carbon Copy Cloner [0]. I have been bitten by Time Machine corrupting itself and I'm never going back. It works well and they have excellent documentation for pretty much every scenario. And USB backups are bootable. I'm mostly using it with backup to a NAS.
I have a 3x backup combo of Time Machine to local Synology (and I may as well not bother with this), Arq Backup to Arq Cloud (not flawless but I trust it more than Time Machine) and CCC to local USB-C SSD drives.
CCC is the only backup I actually trust out of the 3, but it's not automatic and relies on my plugging my backup SSDs in occasionally to clone the whole drive. (Which is more an issue with my workflow than CCC itself).
That last iPhoto update killed my album, and I've lost a shit tonne of photos and videos. And I couldn't even use my backups either because when I open then up, they too needed to read the entire album to do facial recognition etc... took a few days for each backup, but I only knew photos/videos were gone after all drives had been "updated".
One of these days, I'll plan on writing my own photo management system, with recoverable indexing, optional facial scanning, and zero phoning home.
I have all my photos prior to a certain date in an old iPhoto collection. Can anybody recommend a good way to get photos out of iPhoto without having to manually export thousands of photos?
If you‘re ok with just the raw files, right click the library and "Show Package Contents". Then sort by folder size to find the bulk of your photos or create a spotlight query that aggregates all images above a certain size from subfolders. If you edited a lot of photos, it might be messy. If you didn‘t, I think they are all in a folder called Originals, grouped by year then import session.
I asked my father to just give up on it. Just manage pictures through the file system. Photos (former iPhotos) databases also get corrupt on a Synology NAS. It's a matter of time. The strange thing is that it can go well for months on end, giving you a false sense of security.
Do iCloud or local drive backups or stay away from Time Machine and Apple Databases is my advice (although my father had a lot of problems as well with an "incorrectly unplugged" external HFS+ drive). I can't find the sources right now but after my father's last drama (and there have been several) I did some intensive searching and this was my conclusion.