Only tangentially related but I just checked the Google Photos iOS app to see if any of mine are corrupted, and it won't even allow me to view my cloud photos without giving it full permission to all current photos on my device?! Isn't this against the app store policy? You can't deny me access to one feature just because I refuse to grant permissions to a different feature.
But because it's Google I'm sure they can break whatever rules they want as long as it doesn't involve trying to avoid Apple's 30% fees.
Each passing month I am more and more convinced that I should urgently get a) my mails out of my personal (non paying) Gmail account and b) get my associated photos out of Google Photos, mainly because of baby pictures which I cannot fathom to lose.
But then I think about the convenience of my 2TB Google Drive and I can't do it.
And I think, too, of the Polaroid pictures my parents took and gave me in a box once I got married and are now fainted. Mayhaps algorithms deciding that it's time to let go of your own data is just the modern form of yellowing.
> I should urgently get a) my mails out of my personal (non paying) Gmail account and b) get my associated photos out of Google Photos, mainly because of baby pictures which I cannot fathom to lose.
I highly recommend it. It feels like pure freedom, putting a price tag on my personal data. The migration is something done in two steps: set up your alternative service, but also keep the old one (gmail/gdrive/calendar/...) running. Then start to use the new email service, while keeping the old one.
There are enough affordable options and you do not have to migrate everything at once. I started with my emails - gmail is still running and forwards all incoming emails, so I wont miss anything. I already updated all my accounts/logins to the new email address.
The next step was copying all my files from Dropbox, GDrive, and so on to {service I am using}.
The data in the other clouds are still there, but I am going to delete it all soon.
There are things that are not perfect - like accepting appointments, which I still want to show up in my Google calendar. To those mails I still have to respond in google mail. But I am getting there, one step after another. The win so far is much bigger than the loss/pain.
I'm going through the process of switching off of Gmail and onto a Fastmail account under my own domain. I'm loving the freedom of having my own domain that I can switch to another email provider whenever I want. Also, Fastmail with a custom domain has some nice features that were unavailable with a Gmail account (aliases, masked emails, automatically snoozing certain emails so I get a bunch all at once, etc.)
As for pictures, I do use Google Photos, but it's not my primary photo video store. It's more for the app that lets me lookup pictures.
For storing original versions of pictures/videos, I sync from my phone to a home computer using Syncthing, and store backups of my files on AWS S3 using Restic for encrypted backups.
If I lose Google Photos, no big deal. I still have local and cloud copies of my data.
This is definitely not a plug, but I'm actually working on an alternative to email, cloud storage, authentication/authorization/sso/etc. solutions from Google, Microsoft, etc. and one of my specific focuses is to allow free personal use and free smaller company use. It is refreshing to see I'm not building this in a vacuum. I started the project for no other reason except Google yanking out their free accounts. I absolutely do plan to commercialize it, however, I would still be happy with the result even if it failed, since under Google's pricing, I'd pay more than $1,000 a year vs. less than $10/year with my own solution.
As mancerayder says below, why can't you back them up to an external USB drive?
I rsync my 'archive' partition (photos, videos, music, programs, etc) daily to several external 4TB USB drives. Those are dirt-cheap these days. The process takes maybe 5 or 10 minutes depending on whether or not there have been any new files to transfer in the last 24 hours.
I refuse to relinquish my files to some insouciant third-party. And what recourse would you have if they lost them?? Very probably them saying "Sorry. Not our problem."
I take periodic backups of my Google data using Takeout, primarily for the GMail backup. When I took one a few months ago, I noticed that the Drive part of the backup had some mangled filenames that the backup from a year ago didn't have. I don't remember exactly what it was but it was something like long-filename.pdf got truncated to long-file~01.pdf, like back in the 8.3 DOS days. Luckily I already have a local copy of the Drive contents so I didn't need to rely on it, but anyone else should be worried.
a) was very freeing. I didn't realize there were services so much better than Gmail. Even ones I pay a small amount for and get as many custom domains as I want.
b) did this a long time ago. Glad I did. I can sync any file I want too, not just photos. Really been liking Seafile + AutoSync (android) for this.
Most writable optical media use organic dyes that aren’t stable for more than a couple years—if you can still read the contents 10 years later, consider yourself lucky.
M-DISC use a more stable media, and should last much longer, but are (much) more expensive and require more recent burners. There are dual-layer M-DISC with higher capacities, too.
Second this. Several years of use with nary an issue, plus Linux support.
Speaking of which, as an aside there's no excuse for big tech to not support the Linux desktop (I also use macOS and Windows 11 so not a die-hard fanboy). It doesn't matter if the userbase is tiny. Linux enables massive profits in the cloud so any cloud provider that supports multiple platforms but not Linux, whilst also generating huge profits from it, is doing a very bad thing. And that includes MS, Apple, and Google - full support regardless of the (comparatively small) cost as thanks for the profits they generate from it.
This is an engagement hack. Get the user to give max permissions at the start so that any future call to action will have less friction and/or not require a modal.
It’s less about Google wanting to scan your data today and more about Google’s stingy strategy for photos plus Google’s lazy engineering (too many other interesting games to solve). Google Photos was born out of Google Plus, and is under heavy revenue efficiency pressure.
Yes, I ran into this issue after I switched to iOS a couple years ago. Google demanded full permissions to even open the app. I promptly downloaded my image library on the web, and also, related to OP's post, some of my images had artifacts. Some were even corrupt. I moved everything over to onedrive, and restored the corrupted ones that I had local copies of (gradually, it took a lot of work) with local copies I have. I now mirror in "the cloud" with both iCloud and OneDrive. I also have a local copy and a couple of USB sticks with a backup handy on my keychain (solely for if a fire breaks out and I remember to grab my keys).
Institutions of that nature are incapable of following through on nearly any promise made to the public at large; every promise is a lie, whether they break it or not, because they can't be said to have the intention to uphold it.
I see no reason not to believe that Page and Brin sincerely desired Google not to be evil; but it's the institution decides to do evil, not any individual, so they did not have (or could not maintain, as the organization scaled) the power to guarantee this. No one does.
It immediately presented me with a screen with the choice to "save originals to cloud or save a compressed version to the cloud" (I don't recall the exact words) with no option to not upload to the cloud, and then immediately started uploading my library.
I scrolled back and checked some pre-2015 photos, they look fine, and then deleted the app.
Maybe they store physical photos and the albums got soaked in water. Sadly it happened to all our 90s photo albums during a flood. Some of the photos were stained really bad.
I uploaded a photo to google maps. A week later it found some random picture of a local park and offered up a notification suggesting that I upload this photo to the destination it had located in the maps app.
How on earth are they doing this without "scanning" the photo? Seriously, please tell me.
A typical feature of these cloud drives is automatic uploading of pictures from the camera roll, that’s the legitimate use case for demanding access to all pictures. Who knows when Google whoops! accidentally makes that the default. You know, it’s optional, you can just turn it off!
Nothing a little Stable Diffusion can't fix. Soon, digital memories will be just like real ones — slightly different every time we look back, but with no way of knowing what's changed
why do you assume GP is talking about general users? as you point out, this is HN: it’s reasonable to think that most comments here are written for that mostly-tech-literate HN audience, unless they state otherwise, no?
Yeah. And somebody may put up an Open Source repo for doing that 'easier'. But of course, nothing that any average user on the net could even know to exist, leave aside actually use...
Extraordinarily difficult, judging by GP's cynicism. Personally it is appalling just how little people are motivated to be nomically tech-literate in this extremely tech-driven society.
Tech is the only industry that asks this of it’s users, it’s just laziness on the developers part. People needing to know about hashing in order to save photos would be like people needing to know what refrigerant their A/C uses in order to keep cool.
Can you? Sure. Are their weird edge cases where it’s mildly helpful to know as opposed to having to ask someone? Debatably. Should you need to in order to not have things break for no reason? Absolutely not.
Lots of things are complex and confront people to deal with it directly, we just also have legions of people that we can pay to step in for us and a culture where laziness and convenience are core principles to our everyday lives. Not to mention a political establishment that touts the "virtues" of a simple, ignorant life.
Maintaining a house or yard, for example, or taxes. Driving also has a pretty high learning curve -- high enough to mandate driving lessons -- but once we learn it we're set for life. (Tech is the same way here....)
I kind of agree with you both. We do ask a lot of our users. But also, there is no A/C system out there nearly as complex as software. It is hard. For everyone. We can do better, but what does that even look like? Consumer level technology consumption is not in a great spot, but it also isn’t horrible. Power users who can get more out of computing will be a thing for a long time to come.
I think you misinterpreted the comment you're responding to; they meant that "sadly, you [the user] never thought to store a high-quality hash (e.g. SHA256) for integrity comparison with your stored data; and the hash you did decide to archive for integrity-comparison was a low-quality one (e.g. MD5) that can be trivially preimage-attacked such that a cloud provider could silently replace your data with a different one — with the same low-quality hash — without your knowledge."
Nothing about rsync.net per se, other than the general idea that any data you put on "somebody else's server" can't be trusted to stay the same if you don't have a high-quality integrity-comparison content-hash of that data kept somewhere.
To be fair, it may yet turn out that MD5 is (and has always been, we just didn't know it yet) a low-quality hash that can be trivially preimage-attacked. I think that's unlikely[0], but I can easily imagine someone reasonable being a bit more pessimistic. I certainly wouldn't willingly rely on MD5 preimage resistance for anything important.
0: I'd be willing to bet money at 1:1 odds that MD5 preimages still cost at least 2^96 bit operations (out of a nominal 2^127 hash invocations) 5, 20, or 100 years from now.
As someone who works in generative modeling, that's probably what is creating these issues in the first place. Kinda like how you can see certain GAN type of patterns on Netflix when there is upscaling or if you don't turn off the fancy features on your TVs. Of course, finding how these images are corrupted leads to better models so they will be solved. This isn't the first time Google has edited photos and made them worse and it won't be the last. But the trend is that the photos get better.
The promise of the cloud was that you don't need local storage, because Someone Else will remember things for you, and you just ask them what you remember. This is also the problem of the cloud.
"The promise of the cloud was that you don't need local storage, because Someone Else will remember things for you ..."
Agreed. And that promise has been fulfilled - although it costs more than nothing and you need to do your homework.
You know these providers are proven bad actors. You know their business model(s) are antagonistic to your goals. You also know there are stable, time-tested alternatives.
> slightly different every time we look back, but with no way of knowing what's changed
This is already the case with different image compression algorithms improving on one another. Users couldn't and shouldn't tell the difference between JPG, or slightly smaller / resized JPG, or WebP, or whatever. I personally don't think image hosts should have to alert users when making undetectable changes to their images like this.
I'm sorry, but JPEG compression is extremely noticable especially with non-photographic material such as web graphics. The compression is lossy by nature, and predictably introduces noticeable artifacts. WebP on the other hand has support for lossless encoding.
It's simply not the same thing. If I opened my file browser one day and saw that all of my FLACs had been converted to MP3, I would go on a murder spree.
Once a year I select about 100 of my favourite pictures from my phone and have them printed and bound to a book. I suspect this is likely to be the only way my descendants will be able to see them.
But at least people will notice the photo book and at least have one look through. And it might even be a great moment when family members together can look at it and discuss memories. And quite likely it will be kept by a descendant. And it can be found one day by their descendants.
The digital archive or cloud storage will likely in many cases not even be discovered. And if it is discovered, it will probably be such a load of images, good and bad, that no-one will have the time to really look it through and appreciate it.
But for sure, both options combined is probably best.
My gmail account has a deadman’s switch to grant full access to my next of kin.
If I had a gigabytes of photos of my grandparents or great grandparents I would spend hours pouring over them looking at what their life was like before I knew them. That would be such a neat keepsake.
I hate to tell you this but many people have been locked out of their Google Accounts despite knowing the correct username/password combination. Their security policies and ML algo locks accounts and requires 2FA to be used from time to time. How confident are you that in your passing, your next of kin will be able to get into that account if 2FA or backup codes are inaccessible?
Given their is no real customer service team that handles their free tier accounts, I am personally reluctant to assume my Gdrive data will be salvageable.
According to the Google Inactive Account support page, they aren't actually assuming the account, they're just getting access to the data. So, presumably, they'd just download the data and that would be it.
I imagine someone may one day do the same for me, but sadly most of my photos are clogged with mundane screenshots and repeint pictures and other junk.
My experience is that when someone dies, all their paperwork gets left in a drawer for years until the next house move and then it's all thrown out.
My experience has been the opposite, with relatives eagerly snatching up the old photographs.
My boss is young and one of those people for whom digital is the only way. Then one day she got a box of old photographs and was absolutely amazed that she could turn them over and read where they were taken, on what occasion, and who was in the photos.
She now prints out all of her photos and writes notes on the back of them and keeps them in a box, just like people have done for the last century.
I was on the subway in Toronto in 2017, two college age girls had an inch thick stack of 4x6's that they were going through. It was a strange throwback to the pre-digital era.
My family has/had a handful of generations worth of records. The problem is finding out who last had what. Some of it has been backed up digitally, but finding a thumb drive is a lot harder than spotting a man sized pile of boxes.
Nope. It will be passed to the children. But likely not the grandchildren. Even if they do, the knowledge behind the pictures will be lost and grandchildren wont know who's who in most of the pictures. (grand uncles aunts etc). And grand-grandchildren may not even know grand-grandfathers anymore...
That's why I, and every other enthusiast photographer I know, keep my archive locally, meaning on my laptops hardrive and an external one as backup. You cannot trust any cloud provider to properly maintain your data, and external storage is becoming cheaper every year.
I love having a NAS that replaces Time Machine, Dropbox, iCloud, Netflix, Spotify, Disney plus, Hulu, runs home automation, has my security system on it, etc etc.
People use so many cloud services and having it all private and self-hosted is absolutely amazing. It’s really not that hard to set up either, and there’s little to maintenance outside replacing a failed drive.
People here likely know how to do all that safely, but lots of less technically inclined people end up with their data encrypted and have to choose between starting over or paying a ransom. I have a QNAP device so I follow their subreddit and there's a steady stream of people who have lost everything to ransomware because their NAS was visible from the internet. I follow the standard advice from that subreddit - keep it behind a firewall with UPnP turned off and no ports forwarded to the NAS and never enable MyQNAPCloud (or whatever it's called).
Yup. QNAP software quality (and hardware) is complete garbage. I’ve off and on tried to incorporate them into my “system” and they always let me down. You’re putting everything at risk using one of their boxes.
I vastly prefer content be cloud based. Ever since Spotify came out, I've never downloaded music illegally again -- it's just so much better and more convenient. Can't imagine going back to what.cd days and having to download and maintain a music collection manually.
> You cannot trust any cloud provider to properly maintain your data
I trust any cloud provider a lot more than my lazy self.
> and external storage is becoming cheaper every year
Sure, but electricity isn't. At the current price in the UK it seems like running a small NAS would be 10 GBP per month, more than most online storage solutions.
That is without counting the initial investment of buying the NAS, the cost of maintaining it (if I have to spend an hour a month on maintenance, that's $1000+ a year of time wasted), the footprint of the NAS, the noise, the heat, etc.
Yeah. I have a NAS for other purposes but redundant external USB drives is the easiest solution. You can even periodically make a clone and store it somewhere else.
It sounds like you know... very little on the topic. My NAS requires no maintenance, it makes no sound while running, and costs maybe pennies to run.
Meanwhile, there hasn't been a cloud provider without massive reliability or data loss issues. My NAS has better uptime than Amazon, Microsoft, or Google.
So you took it out of the shipping box, plugged it to power, and pressed the power button, and everything was up and running?
And since then you haven't had to update any software on it, reboot it, unplug and replug an ethernet cable, shut it down and opened it to add storage, etc.?
> it makes no sound while running
Looking at a basic NAS on Synology shows that it's around 20 dB, which is admittedly very little, but is not 0.
> and costs maybe pennies to run
Again, looking at the same NAS and using my actual price per kWH, it's around $10 a month.
> there hasn't been a cloud provider without massive reliability or data loss issues
My searches haven't turned up any article about a data loss from any of the 3 big cloud providers. Even the story here, while scary, is just about compressed files, not originals.
---
I'm not denying that a NAS has advantages (the fact that it's a lot more than just a data backup), but in the context of backing up files, it's not at all convenient compared to using online solutions.
Their estimate of £10/month seems reasonable to me – that's basically 1 or 2 kWh of electricity per day (can't find current UK electricity rates). Is your NAS under 40W power draw?
My RPi 4b runs on a 18W power supply, which would put it at 432Wh per day if running at max load (It doesn't, it idles at <3W most of the time). Any arm NAS is going to have a similar power draw, and you really should pay attention to the power draw when choosing a NAS.
What about the storage devices (the S in NAS)? My external HDD (just for reference) has an AC adaptor that outputs 12V at 1.5A, so that's another 18W. And most people who care enough about storage to have a NAS have multiple disks...
I'm not saying it isn't under 40W - as you note, if it's idling properly, it's pretty easy to have a NAS including disks under 10W idle. But the electricity cost is not negligible when you are running something 24/7. Even 10W24hr30day is 7.2kWh per month, and most places cant count on 8c/kWh electricity anymore.
> My RPi 4b runs on a 18W power supply, which would put it at 432Wh per day
Which would be 13 kWh per month. If we're comparing to monthly fees for cloud storage, it's important to look at the same time frame.
I have two SSDs on it that are powered by the RPi itself, so the whole package is < 18W. Also, SSDs use way less power than HDDs.
> Which would be 13 kWh per month. If we're comparing to monthly fees for cloud storage, it's important to look at the same time frame.
Yes, running at max load. A more realistic load will be 3W/hr -> 72W/day -> 2kWh/month. That would put it at ~0.3€/month at a realistic load and ~1€/month at max load (0.15€/kWh where I live).
You also have to take into account there's other stuff running on such "NASes", like the PiHole, a git server, a plex server, which are going to cost you separately if you need them. On the other hand, my setup cost me ~300€ (RPi, charger, drives, USB-SATA adapters), which buys a whole lot of VPS and a whole lot of online storage.
Yes, SSDs would certainly help cut your power load down significantly! My system has 3 HDDs and it's a fairly small NAS compared to many - SSDs are prohibitively expensive once your storage needs get above a couple TBs.
Law of propability: I had zero dataloses due to fire, some data loss due to loosing a thumb drive backup and some data losses due to cloud storage shennenigans.
You are right, a cloud storage would make things a tad saver. Personally, I made a point in not putting anything in the cloud, especially none of the big providers, since I value my privacy and just don't trust any o those companies.
For the really important digital stuff (almost 30 years of family digital photography and video and some paperwork) I have a rotating backup schedule. Two of the drives in the rotation stay at my mother's place. Nothing fancy and it's mostly when I can bother to remember. About once a year or so. Still, some peace of mind.
Can you share some specifics on the setup? Trying to do something similar. Haven’t tried it yet, but MacOS can create encrypted “sparse images”, which can be mounted and contain files. I’d like find to a better way, especially something that could stand the test of some time (10 years seems reasonable).
My wife does a photo book every year, combining the best of both our photos. It’s a bit of a chore, but I’m so glad she does it, they are so much nicer to browse through. The kids love looking at them all from time to time.
That’s a pretty bold statement. The null hypothesis is that there’s nothing in the body that cannot in principle be emulated in silico to whatever precision necessary.
Why is that the null hypothesis? For one thing, the only experience we have is existing as bodies in a real world, so that should be the default with any other claims being supported by evidence. For another thing, simulating a bridge falling down is not a bridge falling down, and it's not proven that simulated thinking is thinking (or even, possible).
> The human experience is inseparable from the human body.
I'd say that the human body is just one very sophisticated I/O device.
The fact that we can't replicate it now doesn't mean that it wouldn't be possible after a millenia of engineering, even though this replacement device might as well be a partially organic one in the end.
That said, there are no guarantees that we don't fall into a sort of "dark age" of no more growth/research into any new directions, or don't wipe ourselves out altogether.
Sad news for people hoping for a life after death, though! (ie nearly everyone)
I think it’ll be possible to create inorganic ‘brains’ that have human-like subjective experience, because to think otherwise reduces to some form of vitalism, but I doubt we’ll be able to move specific individual consciousness across technologies.
Well, I’m assuming the majority of people currently alive (and even more of those who have ever lived) are religious, and the majority of religions followed hold with the Heaven & Hell / Cycle of Reincarnation until Enlightenment model - and that experience in the afterlife was still recognisably ’human’.
Though, saying that, OP’s point about needing a human body to have human experiences may be correct if souls get a new human body in the afterlife - I’d just assumed liberated souls, like ghosts, had no physical form.
What makes you say that? You could literally be a brain in a jar or be hooked up to god knows what "deep diving" / "full VR" experience humanity will come up with.
I'm mostly curious in what compression algorithm results in artifacts like these, and assume somebody here on HN recognizes them? But we're 225 comments in (so far) and not a single answer.
They're absolutely nothing like block-based JPEG, that's for sure. When I inspect the images Google Photos serves me in my browser, it is serving up JPEG (from the response headers). But is this an artifact that shows up in AVIF or WebP? I wonder if mobile clients are getting a different encoding.
The only objective thing I notice is that the artifact lines tend to track a line of constant brightness, so you seem them appearing perpendicular to gradients of light and shade. And that each artifact line is black/dark on one side and white/light on the other -- and that the white edge seems to be in the direction of darkening, while the black is in the direction of lightening.
Someone here who works with modern image codecs must be able to hypothesize what part of encoding/decoding must be bugging out here?
Looks to my eye like some 8-bit underflow artifacts that I've seen. And the way the line follows the gradient reminds me of when I've broken a gradient-domain Poisson solver to reconstruct an image from gradients. Maybe they're decorrelating the image data via gradients to compress better and then compressing those along with a sparse set of primal pixels?
I’ve spent a ton of time playing with codecs and colorspaces while developing PhotoStructure.
These artifacts aren’t where I’ve seen them before. Normally you’d see this sort of posterization and clamping around global highlights and shadows, especially if the incorrect colorspace profile was applied, but this seems like the affected border is around a localized area, possibly due to a buggy pseudo-HDR implementation (what you see when you move the “pop” adjustment slider, which increases localized contrast ranges). Google+ images had a mild pop adjustment applied automatically.
it seems to me from the photo of the dog, it was an face recognition, algo or eyes recognition ago, but the algo also left the changes in the photo as it proccesed it. instead of putting it all in the server side.
My first thought was that Google as recompressed everyone's images to save space and corrupted them in the the process. Hopefully everyone gets their images restored.
I was thinking something similar: it has a distinct posterization look that I tend to get when I mess around with images and reduce the bit-width of my input a little too much and my integers overflow. But those tend to result in "zebra stripes" like patterns, so you're probably right that the issue is more one of not using saturating integers while amplifying a value above the bit-width of its container.
I've been long considering switching my Google Photos library fully over to ente [0], of course alongside my existing monthly backups to local HDDs and an annual backup to a cloud server via borgbackup.
With every passing week, every Google fuck up like this and with ente's great pace of development, I draw myself nearer and nearer to the precipice of full switchover. I will soon be having my last Google Takeout.
I remember trying ente, but the mobile app had a notoriously low amount of features and low FPS (felt like 30fps on my modern, 120Hz phone - stuck out like a sore thumb). Has there been any progress on that front?
Hey! One of the founders of ente here. We have shipped a bunch of performance improvements and features[1] over the last few months. But we are far from where we want to be.
If you happen to revisit and find something lacking, please write to vishnu[at]ente.io, we'd love to make ente work better for you.
"duck" means "mine" in Malayalam (my native language), and I have a weird obsession with rubber ducks, so it just fit, and perhaps most importantly, the domain was available :)
This great Hetzner page[0] explains it really well - its pretty much exactly what I'm doing. My Google Photos are extracted from the Google takeout.zip and those folders are backed up into a borg archive.
You know how the mantra in the crypto community is "not your wallet, not your coins?" This applies to this situation. Once you trust Google with your bits, don't be surprised when they modify them without telling you.
> This is an indication that Google underestimates the value of data stored by users.
That becomes apparent if you ever had any issue with a google product. There's no way to resolve issues outside of canned answers from "AI" systems and public forums.
If you subscribe to the cypherpunk mantra that ”code is law“, then yes. Otherwise, if you stick to the law that lawyers and judges apply, then mostly no. NFT ownership does not come with any rights to the underlying asset, unless specified in an accompanying contract.
>Google underestimates the value of data stored by users.
Of course they do. How can they possibly put any value on endless PB of holiday snaps? They only care that you are now reliant on them to store your memories.
Do you have any suggestions for a secure/private option to store photos? I'm not sure what would be best. (Dang, I really want to move away from Goo now)
Your own devices, with a proper backup solution (which you should already have with or without your photos in it). Nobody can mess with your data then.
Personally, I use Syncthing to sync between devices, including a NAS as a target, which regularly backs up to Backblaze.
This sounds a bit like what I do. I r-sync two different libraries to NAS then upload them to Backblaze B2. Post r-sync success result to a muted Slack channel and failure to an unmuted alert channel.
As a starter, look for a "dumb" service that doesn't treat photos differently from any other type of binary blob. No thumbnails. No grouping by date or location. No tagging other people. The storage service should only concern itself with safekeeping the binaries you upload, not analyzing and processing them. Use your own machine to do the latter.
I stood up a pair of QNAP NAS for both my parents and I, and encrypt-then-backup their photos/documents to my NAS, and vice-versa. RAID 5 or 6 keeps hardware failure at bay, and keeping them tucked away in our respective homes makes fire or flood irrelevent (for this).
QNAP or Asustor has standalone apps my parents can use as well. Without getting your parents involved, you could either pay for remote hosting for your off-site, or trade RAID space with a friend.
That's a fairly serious accusation to level. Google's bad behavior here is not that their compression algorithm ruined users photos. Google's bad behavior is running a compression algorithm at all.
That's not the point. I doubt that the engineers who worked on the compression algorithms would store their own precious photos using the same techniques.
Also, when was the last time you heard a banker say that everyone makes mistakes all the time.
Hi, I work on the processing algorithms at Google like the one here (not sure about this particular issue) and I use Google photos for all my old photos. People at Google read Hacker News and fix things, they just don't comment on specifics for things like this.
I had noticed one of the images in my Google Photos account seemingly not showing up. It was very noticeable because it was one of the "People" images in my Photos account. It's a bit concerning that a photo storage service seemingly can't maintain the integrity of their data. Makes me wonder about what other data issues there are, lurking in the depths of Google Drive...
I personally found a corrupted file stored in online storage only, though it wasn't with Google, but with Microsoft's OneDrive.
I tried to open a Word document that I hadn't viewed for several years, but got an error that the file was damaged. It looks like this happens sometimes, after searching for the error online. Luckily I was able to recover an earlier version of the file (via version history), but it was alarming that it happened in the first place (also I'm not sure if important edits were made in the latest corrupted version).
It's unfortunate how many cloud services default to storing on the cloud only instead of also keeping a local copy, and don't even provide the option to opt out. Even if you choose the "keep offline" option for several services, for unknown reasons, this doesn't seem reliable (Maybe I'm 'using it wrong'? But in practice, I've found myself having to download files that I was sure I set to keep offline.)
Google is pretty proud of their record of not losing user data. As far as I know, they have never permanently lost even one byte of consumer data.
The same isn't true for GCP - there they lost some writes to customers persistent disks - but only ~50 megabytes worldwide, which is still rather good when you consider they store millions of terabytes.
> As far as I know, they have never permanently lost even one byte of consumer data.
It seems that every other day you learn about people being permanently locked out of of their Google account. Every byte of data stored in there is permanently lost to the person losing their account.
What if the policy is to lock accounts when data integrity errors are detected...
"We have always had better storage durability than Europa."
Note, I am not arguing that this is necessarily what happens. The thread full of contrary views on cloud storage just tickled this cynical take loose from wherever it was lodged.
It really depends what you consider lost. Thousands of users (at least) have been permanently and eternally cut off from their Google data after some AI system decided to axe someone's access to their own account. The loss is significantly over a byte. Or 50 MB.
> Google is pretty proud of their record of not losing user data.
As far as I know, they just haven't admitted to losing consumer data. Until they define that and put even a little bit of effort into checking whether it might hold true, they don't really have anything to be proud of.
There have been multiple mass-deletion events and Google has admitted to at least one of them - the infamous "only 0.02% of our users" incident, for example - but only because so many people were affected, the press picked up on it and google's PR firms couldn't keep it quiet.
They definitely had permanent data loss in App Engine around a decade ago, along with leaving any app active during the outage with copies of every database table to manually merge
For context, I know this is a display issue and not a storage issue and you can get your originals out via Takeout.
That being said my photos are the only digital files I really care about not losing. My photos and videos from my phone get backed up to iCloud, Google Photos, OneDrive and Amazon’s photo storage that comes with Prime.
My videos get backed up to all of the above except Amazon’s storage.
You should back them up locally as well even though that has its own failure points. I consider anything other than a dedicated local and/or cloud file backup system a nice supplementary place to store photos but not primary.
At one point I had them synced to my Windows computer via iTunes and that was backed up to Backblaze.
But, I don’t have a personal computer anymore and soon won’t have a personal residence. My wife and I are going to be digital nomads traveling across the country for a few years.
The local ZFS array is the “source of truth”. ZFS has built-in end-to-end checksumming. I also have a monthly scrub that should correct any untouched corrupt blocks.
Thanks, that is one way to do it. But if you don’t know about par2 (sorry, I can’t tell from what you’ve said) it’s worth chasing down. It is way, way beyond checksums in what it can do.
It’s an old project and information on the internet is getting thin but the math is sound, the tool works, and there is a wikipedia page:
I find the idea of old photo/data degradation strangely comforting. Even digital data seems to be unable to escape the law of entropy (don't mention the black holes).
On another note; if corrupting data/photos was a wide occurrence now wouldn't we have heard about it in a big way?
Yikes, this is kind of scary. Google Photos is the only place to find some of my older photos. What’s the easiest way to download my entire library for backup on a physical drive?
It works extremely well, and you can re-run it any time to sync new files locally too.
I used it like that for a few months before I finally installed syncthing on my phone and stopped using Google Photos altogether.
Now what I do is take photos on my phone, have them sync to a NAS. And on the NAS I used a modified version of this https://forum.syncthing.net/t/android-photo-sync-with-exifto... to build up a YYYY/MM folder organisation and move files older than 30 days from the syncthing folder into my archive. My archive is then in my Plex so it's still accessible to me.
In essence: 1) Take photo (implicit sync to NAS), 2) wait 30 days, 3) archive photo into long term directory naming convention, whilst making available to Plex and deleting the version from my phone (by deleting the syncthing version it will delete the one on the phone after 30 days too).
I've been putting off a Google photos back up for years, and this was my excuse - waiting for this issue to be resolved. But I'm being a fool, 60% compression is better than losing everything.
I have just compared originals I had which were uploaded to Google Photos vs the ones downloaded.
Visually there is a barely perceptible difference in a side-by-side comparison on a high quality and calibrated monitor. I wouldn't have noticed if it weren't side-by-side and zoomed in.
On the file size, it's gone from a 6MB jpeg down to about a 1.5MB jpeg.
On the smaller files that difference is far less, but jpegs above 3MB seem to see the biggest reduction in file size.
Thankfully I have all of my originals that weren't taken on the phone camera so I don't mind this so much as running my exiftool script over the original archive produces the same naming convention and I can merge those folders and still have the originals I care about.
Sheesh though... such a thing for Google to screw up on.
I have been having real problems trying to do Google takeout with my Google Photos. I keep getting an error emailed to me saying "However, we were unable to create a copy of all your files. You can still download your files until September 25, 2022, but please be aware that there will be data missing."
Apparently I am not the only one[1][2]. Other people have a mix of errors[3]. It is not the reliable data extraction I was expecting for something as valuable as my family photos. Google is losing a lot of trust from me based on this.
Google Photos used to have the option to sync all photos into Google Drive but that feature was removed. I’m guessing the raw data is still available via Google Takeout
Has there been any progress on self hosted alternatives to Google Photos in the past couple of years? Last time I checked there was great systems for individuals, but the main reason why I use Google Photos is so I can have shared albums with family & friends where anyone can add their photos.
I started using PhotoPrism a couple months ago. My only gripe with it is that you can't drag-and-drop files directly into folders or any page. You have to manually go to the upload page, change/add the folder then use the file picker to choose the files. Other than that it does the job well and even has face scanning.
IMO the best place to put photos, at least the master copies is in AWS S3. It is both reliable and redundant, and they don’t mess with your data. They charge you every month for storing it, and again for accessing it, but that is it.
Use anything else you want for day to day usage - Cloudflare, Digital Ocean, Google Drive, etc. whatever fits your budget and needs. But sleep safe knowing your data has a backup.
Same thing with YouTube - you never know when Google might decide to wipe out your channel, and if they do you’ll never know why they did it, and you’ll have no recourse after they have done it. There are plenty of stories on HN where this has happened. So the master copy of all your videos should be in AWS so you can start again if you need to. IMO.
The susceptibility of the SD card to data corruption is a big showstopper. The only digital files I've had corrupted were stored on my phone's external SD cards (after 3 to 7 years). When I looked into it, it seemed to be an issue related to lack of radiation hardening. So if you do make this service, please include some error detection/error correction metadata.
The powering on all the time is probably due to a Wake on LAN packet or other power management feature of the network hardware of the device. Most operating systems provide a way of disabling the power management features of the network hardware, but I've never used a Synology device, so I can't tell you how to do that.
One of the best ways to deal with Google customer service is to simply send in a support case and immediately dispute the credit card charge. I’ve done this with Youtube TV several times (wrong recording, live the doesn’t work, their location check locked me out) and got free months of service.
Don’t bother getting all frustrated in a big support thread. The engineers who caused these bugs DO NOT care about you. They are way way too busy solving dynamic programming problems. Chargebacks are the most effective tool here.
What have you done a chargeback for with Google? There's stories of folks doing a chargeback and then having their account closed afterwards meaning you lose Gmail etc.
I had a disaster when using DropBox way back around 2011. Ever since then I have been responsible for storing and backing up my own files. Nobody else gets to have them in their 'safe'-keeping.
This is normal. I found out that Google Photos has limited reliability. My 2007 photo was garbaged into a lower resolution. I have since recovered from the original drive. Back then Google Photos gave an option to store in lower res but still sufficient high quality or original quality. Don't trust either. Lesson: backup to your media you own or print it.
Seems unlikely Dropbox would be immune from some sort of issue like this. Probably best to have your photos in storage somewhere else (ideally both locally and off-site).
That can be ruled out. Bit rot does not appear suddenly in a widespread manner and does not cause very similar visual effects in a variety of images. Also the original images are still available, just new modified versions were shown. To me it looks like a new compression algo gone wrong.
A bad bit wouldn’t even be noticeable in an image. Changing one bit would make literally no difference. Artifacts look like something the compression algorithm has messed up. Also since the images are actually intact according to Google this is likely just the compression algorithm used to browse the images.
Update: my limited knowledge on this is proof I’m wrong.
A bad bit might have no visible effect at all (e.g. Affecting part of EXIF metadata, or a totally black pixel turning again very close to total black) or can corrupt the whole image (flipping a bit in the file header or cosine quantization table).
As a total layman when it comes to storage: Don't they use forward correcting codes to make sure a single bad bit can reliably be detected and flipped without affecting the data at all? Or are these used only for data transmission?
I've encountered older digital photos where a small corruption affects all of the remaining parts of the image as it is presumably recorded in some linear fashion as shown in this Wikipedia entry's "Visual example."
But because it's Google I'm sure they can break whatever rules they want as long as it doesn't involve trying to avoid Apple's 30% fees.