Interesting to see how this plays out in a different jurisdiction to mine. Here in the US, it seems like as long as penalties remain very light, PII retention and security is not taken very seriously.
Equifax's breach in 2017 was essentially every valuable piece of PII they had, on more than half the households in America, and they settled for $300 million in a civil suit. Their net income (after all expenses) every year is in the $500-700 million range.
They should have been fined directly, an amount intentionally set to force complete liquidation, shareholders should have been completely wiped out, and all proceeds from the liquidation after court costs should have been distributed to every victim of the breach.
If that was the case, 100% of US companies would now treat PII with the respect it deserves. As it stands, nah, it's nbd here.
Companies were told for years that their data is an asset. Instead they should treat it like a toxic substance and a liability, to get rid of it as soon as they can. But it will be a long time before this view will become more prevalent.
That’s kind of a false dichotomy; lots of things are both assets and liabilities.
I work at a megacorp, and we have an entire group that goes around auditing projects to ensure compliance with data rentention policies, which include mandatory deletion of different types of data at different timeframes.
Financial transaction records have a very long mandatory retention, IP addresses for logins have a fairly short mandatory deletion. Product telemetry has strict rules about avoiding PII and very very quick deletion (aggregate data can be kept longer).
I’m sure the policies could be improved, but I suspect every large company is intensely aware of the importance of data deletion, even if competing priorities sometimes lead to longer than technically/legally required retention.
I work on the Privacy Engineering team for one such company. Data retention is a huge topic and while there are often good guidelines, there usually aren't hard-and-fast rules. It's an exciting specialty for anyone that enjoys ambiguity (I like to joke that I enjoy the space since my dad was a philosophy professor). It also pays well and is recession-resistant.
14 days in many cases, 24 hours in some. That quick because it is very hard to ensure that app telemetry doesn't have any confidential / sensitive information. And if the purpose is troubleshooting, there's no value in old data.
This will never happen. For a long time I thought there is a an USP in treating data as toxic substance, but with the rise of LLMs I think data is the new oil.
Company A that leverages data will outcompete company B.
Lots of layers to this analogy, given the damage our use of oil is causing.
Companies are going to learn that using LLMs without paying proper consideration to data governance is a recipe for large fines and worse (such as being required to throw out entire models because their provenance violates data subjects' privacy rights).
They're only subject to large fines if those laws are in place and actually being enforced, though. Companies keep getting away with these huge data breaches in the United States with almost no real consequences.
It's getting harder and harder to ignore these laws unless you're willing to stay out of some major markets (such as Europe and California).
I think we'll see a national privacy law in the United States at some point in the next five years. There's appetite for it in both major parties (Democrats to protect bodily autonomy, Republicans to stick it to Big Tech), and I think the targets of the regulations themselves will at some point lobby for a consistent national law rather than the patchwork of state laws that we have now.
The RESTRICT Act suggests that there's political will in the US to solve the same problems that the GDPR solves. If the RESTRICT Act fails, the US might get federal-level privacy protection (subject to the PATRIOT Act, of course).
I find it helpful to think of data as uranium instead of oil. Both are valuable, but one is also a liability if you mishandle it. What you want to do is only have as little as you possibly need to make your business go toot, and dispose of it as promptly and safely as possible when you no longer need it.
Deleting data is hard work that requires a lot of preparation and has to be done without the safety net of backups. In many cases it will require changes to proprietary software. Keeping data happens automatically.
Not having data that you're supposed to have becomes obvious as soon as someone asks for it. Holding on to data for longer than necessary only becomes a problem if there's a data breach (that cannot be covered up).
isn't it both? It's valuable fuel and radioactive waste? I see 2 immediate problems though: 1. you don't know when the fuel is exhausted or if you can reuse it in the future so you stockpile it; 2. the contamination is much worse than the local radioactivity, so it's hard to manage correctly.
The added complexity is regulation saying you need to keep data on your customers. In a way the early somewhat anonymous internet was better in this regard.
An edit to add, if you encrypt user data at rest (big ask currently). You can destroy the keys past a point and then the data in backups is safe etc.
Anyway it'll take a while for this view of the world to shake through.
More like "It's a liability until proven it isn't". We've been able to finally throw away so much of old useless "WE MIGHT USE IT SOMEDAY" stuff coz of it...
It depends on the data and it depends on the way that you store it. Nobody tells you to store all your data in plaintext for seven years in the live environment on an internet accessible service, there is simply no such requirement.
It also isn't necessarily a requirement that such permanent records are digital, this depends on the country.
No. On the contrary, GDPR explicitly allows data to be stored if retention is required for law, such as all financial data for 10+ years. However, there is absolutely no need to hold all this data "hot" in the production system and not in a "cold" archive without automated connection.
Thats the whole point of the initial comment. It is necessary to store some potential PII for long periods of time due to regulations from the same people that want you to get rid of all PII.
It's unfortunate that the attempt by companies to make people associate cookie popups with GDPR has worked.
Understanding the data you collect, why you are collecting it, what you are using it for and what the risks are if the data is leaked is unsurprisingly a useful thing to do as a business.
> attempt by companies to make people associate cookie popups with GDPR
I think GDPR is generally good for individuals and the internet but if someone hates cookie banners, isn’t it fair to place the blame on GDPR?
Why can’t websites accept a special header which automatically accepts all cookies? I would enable it and handle clearing/retaining cookies myself through a browser feature/extension.
Because the GDPR already has a perfectly reasonable way to avoid the requirement for cookie banners. If you don’t collect information beyond what is strictly necessary to perform the task you are offering to users, and do not use that information other than in the performance of that task, then you don’t need a cookie banner. So Strava would not need a separate permission in order to collect location data for comparing your biking routes, but Strava would need a separate permission in order to use that location data for advertising, and Facebook would need a separate permission in order to collect the location data in the first place.
The GDPR doesn’t specify the technical means, only that permission must be explicit and freely given, with the default assumption being “no permission granted”. I think these conditions are entirely reasonable, and a header that could be set by somebody other than the user, then sent by the browser on behalf of the user, does not satisfy these conditions.
Most entrepreneurs believe that visibility over how your visitors are using your website is “strictly necessary” for running a functional/secure/performant website and surviving as a business, but GDPR disagrees. Hence, cookie banners everywhere.
Not deemed “strictly necessary” > “Statistics cookies — Also known as “performance cookies,” these cookies collect information about how you use a website, like which pages you visited and which links you clicked on. None of this information can be used to identify you. It is all aggregated and, therefore, anonymized. Their sole purpose is to improve website functions.”
> but if someone hates cookie banners, isn’t it fair to place the blame on GDPR?
No, blame companies that set cookies for merely reading a website and then bothering users about it. They have a choice, they choose to make it obnoxious.
Your plan is what what maybe 3% of the population wants. It's a good idea, but it's not a solution to the issue.
You can already handle this yourself with a browser extension to click the consent buttons.
I use those. They work on about 10% of sites, because there are about 1,000 vendors of "cookie consent modals" so nothing about them is standardized. Also, none of them that I've seen actually do anything, since "blocking all cookies set by 72 different adtech scripts loaded before and after you init" is not a real thing that Javascript snippets can even do. While most adtech snippets do have APIs to carefully pass in the user's GDPR prefs, most people don't wire them up, or even knows what order the various scripts load.
The DNT header already exists. Go figure out why it is ignored (is it too user friendly?) And since the GDPR has a large reach (it covers PII in all form, not only stored digitally), it is not the place to mandate it.
I’m thinking of the opposite of a DNT header. Websites would be very happy to respect it because it’s like auto-accepting a cookie banner. You would need to manage your own cookies.
(not OP but) we want the opposite. I'm not naïve enough to think that those cookie banners will change much about what is being stored anyway, and browsers are more than capable of dropping cookies on the ground either instantly or at end of session.
So much could have been simplified if the GDPR rules, instead of imposing burdens on a million websites, required the 3-4 browser vendors to have a toggle for preserving first-party cookies on sites where the user submits a form with a password field, and simply cleared all others at session end or periodically.
Not sure why we should want to make companies less responsible in the first place. Anyway, as I said earlier, GDPR does not imposes burdens specifically on websites, but on any kind of PII processing. It is not the place to add provisions specifically for web browsers. I understand that the next ePrivacy regulation wants to make it more user friendly, but negotiations for this bill have stalled for years.
I don't want companies to be less responsible. I just think it's a fool's errand to ever expect every single webstore that runs a Shopify shop to understand how to add their 38 different adtech "tags" in a way that truly ensures that cookie consent is captured, stored, and conveyed to each entity that could come into contact with that data.
But by regulating browser vendors, they could have made it so that it doesn't matter what cookies they sent you. If the user hadn't consented in a browser UI, the browser would forget the cookies. Easy to verify compliance.
It's just like the ol' pathetic "Do Not Track" header. Same flaw. Asking "please don't give me a cookie that I'll have to keep and send back to you anytime you see me" instead of saying nothing, and just dropping the cookies you don't need on the ground.
There seems to be some serious misunderstanding here. For one, why do you think this is about cookies at all?
This is not something that can be solved client-side other than obfuscation etc. They can track you with other means than cookies. Even worse, you might have an account on their site. Having an account and using the site (and logged in) makes it trivial to follow you, but that does not give them the right to abuse that information for other purposes. You might have an unique IP and can't reasonably expect to do anything about it.
GDPR covers all of that.
"Just delete your cookies/session" is not relevant.
GDPR requires you to request consent for any cookies the “could” be used to identify you, which makes them personal information.
So if you want to use cookies to link a user’s sessions on your own website together (without actually identifying them) so every request doesn’t look like a totally anonymous, opaque request, then you must show a cookie banner.
You could (presumably) do this through browser fingerprinting and not require consent (since you don’t actually enrich/link the browser fingerprint to be become user data) but you need a cookie banner if you do it with a cookie.
> GDPR requires you to request consent for any cookies the “could” be used to identify you, which makes them personal information.
> So if you want to use cookies to link a user’s sessions on your own website together (without actually identifying them) so every request doesn’t look like a totally anonymous, opaque request, then you must show a cookie banner.
Wrong. The ePrivacy directive has an exception for strictly necessary cookies (Article 5.3), which is applicable for user sessions.
> You could (presumably) do this through browser fingerprinting and not require consent (since you don’t actually enrich/link the browser fingerprint to be become user data) but you need a cookie banner if you do it with a cookie.
Are you able to identify someone from the fingerprint of their browser? Then the fingerprint is PII. Consent (or any other legal basis from GDPR Article 6) is therefore required if the exemption from the ePrivacy directive is not applicable.
There’s a lot of conflation between the 2003 EU Cookie Directive, and the GDPR. The cookie directives specified a technical means (“cookies”) and actions that needed to be taken in order to use them. The GDPR specifies the ends (collecting and/or processing personal information) and the conditions (explicit and freely given consent), stating that anything achieving those ends must meet the conditions. It’s a much better written law than the 2003 Cookie Directive, because it avoids the need to irritate users for legitimate use cases, while also preventing legal loopholes (e.g. “We didn’t use a cookie, just the browser’s localStorage feature.”)
This is factually incorrect. The "Cookie Directive" wasn't from 2003, it was an amendment to the ePrivacy Directive. The ePrivacy Directive came into effect in 2002, and it was amendend in 2009. That amendment is what people generally call the "Cookie Directive" because it required consent for storage of information on end user devices.
It did not specify cookies, and did not actually specify any technical means. The ePrivacy Directive requires that companies get consent from users before storing information or gaining access to information stored on end user devices. This includes every kind of cookie you can think of, including LocalStorage. There is an exception for cookies necessary for the service requested, which typically includes things like auth cookies or shopping cart cookies, so long as that data is not used for anything else.
I would argue it's not useful to the business, which is why, ignoring the societal problems caused by business is often the default and why we need the regulation.
In the UK, current news cycle is about sewage being dumped on our beaches by water companies that were previously privitised.
Especially since the old cookie banner law was abolished with GDPR. The whole cookie banner thing has technically nothing to do with GDPR and is the industries own perverted invention for "getting consent".
This is not true. The national implementation of the ePrivacy Directive are still law. You are correct that GDPR does not require cookie banners, but the law that does was not replaced by GDPR and remains in effect.
Hopefully it makes a real difference to compensate us for the never-ending, dark pattern cookie notices we have to endure as a result. Who wants a choice of 'Accept all' or 'Manage your settings' on every second webpage they land on?
These companies comply with GDPR as much as alcoholics comply with drinking driving laws by keeping a bottle on the trunk and stopping to take a swig once in a while
A law which causes the whole web to be worse for no reason deserves criticism. Managing cookies belongs in the browser, a place auditable and adjustable by the user. Not on every website in the world.
GDPR may have had good outcomes too, and I am neutral on all other aspects, but whatever part of EU and California regulation led directly to cookie banners is a colossal failure which has benefitted no one (except possibly the dozens of snake oil cookie banner products which pretend to comply).
Indeed, for future historians it is critical that every company store the address, id, and private info of all their customers, for all time, regardless of risk. Without which we may forget that I lived on Paeroa St for two years while at uni, information that will be crucial in a discussion of the habits of early 2000s university students in New Zealand, and the impact that had on the fall of western civilization and the rise of our new fish-centric ethno-religion in the year 1200 AT (After Tilapia).
Let's not forget that all this personal data should also enter public domain for historians of the world to process.
How else will future generations be able to put your residence on Paeroa St in connection with the consumption of fish in that area during that time, to reveal the political stance of you and your offspring against the ruling party of the Tilapia...
It's almost amusing to watch how companies and consultants approach personally identifiable information in 2023 compared to just a few years earlier. I've been paid a lot of money to be able to get peoples data to essentially be able to self destruct and make sure before it does that it's transported and stored securely.
If I was to walk into an ad tech space in 2016 and tell people about my new found talent of scheduling data scrambling jobs against PII and rambling on about migrating all user data to be encrypted at rest they would have called the police.
I always find it amusing that before PII got taken halfway seriously, basic protections were assumed to cause massive problems for the business, and yet life went on just fine.
This is about the bigget teeth the privacy commissioner has. Suprised to see such a heavy handed response from them given it was only 20% of the population.
Really, the NZ privacy commissioner's office is a joke. You can see this when they say things like:
> The Office of the Privacy Commissioner also encourages individuals to challenge hard why an agency needs to collect and retain their personal information.
I've talked to people who work in govt that have had full cctv surveillance at city wide (Hamilton) level Ok'd by them. Like no worries mate, they're in public so you can use casino face detection technology on them to track wherever anyone within the inner city is detected for 40 days of collection capacity.
> drivers’ licences [...] as well as people’s passports. Some of the 14 million New Zealand and Australia records taken are up to 18 years old, which isn’t okay.
A question about NZ English... How is that last clause (the final five syllables) used/interpreted?
To my ear, it sounds comical. "Our office wishes to formally advise millions of citizens that your trust has once again been violated, and in our official role we declare this to be a not okay event."
It means 'which isn't acceptable'. I am also confused as to why they used this turn of phrase, perhaps to sound more colloquial. It sounds weird to me too, and comical, coming from a govt department, but perhaps it is indicating that the privacy commission finds that to be something the population would not approve of, but which it has no actual enforcement mechanism.
ANZ governments are a little less officious than other countries (the lack of taking ourselves seriously is a defining trait). The article also mentions a "she'll be right attitude".
Two years ago they made you submit a manual request then prove with a police file or something that you have real proof that you're in real danger of identity theft. Now it seems it's more automated.
Very scary: I signed up and they know a lot of personal information such as my bank account numbers (presented completely without obfuscation), and my exact mortgage amounts, plus other information that shouldn’t be so easily available.
No actual authentication to sign up - presumably you can do so if you know a few personal details and somebodies driver’s license number or passport details (neither of which are secure information).
I believe it was nerd wallet, but I signed in a few years back and was surprised to see they had almost a decade of financial data on me. I don't recall them printing out my unobsfuscated account numbers but it was a bit unnerving regardless.
I remember being refused something (although I can't remember what) because my credit record was blank. I bought a bed on hire purchase specifically so I had a financial footprint.
I'm surprised that a standard "average" credit level can't be used as a starting point, and then every transaction afterwards is a difference against that default starting point.
That is, no evidence in favour nor evidence against you.
> I'm surprised that a standard "average" credit level can't be used as a starting point, and then every transaction afterwards is a difference against that default starting point.
> That is, no evidence in favour nor evidence against you.
It's because this isn't a court of law. You're trying to convince other people to give you an unsecured loan. The default will never be an assumption of trustworthiness.
The default assumption of a blank credit info in 2023 would have to be this person is either very young and probably a risk or a fake identity and certainly a risk. That’s why young people can only get small loans.
This seems to be unforced data retention, which is leaked/stolen, as is pretty much a given. However the privacy commissioner is saying "data retention is a problem", and at the same time we have:
Spy agencies: we need everyone to collect data to prevent terrorism!
Politicians: We need to do that to protect children!
Politicians: Also if you ever leak any of the data we forced you to collect and store data whether you want it or not, you will be liable.
I didn't get that impression. Drivers licences and passports aren't things banks generally care about. It sounds more like the "know you customer" and "anti money laundering" hoops all financial institutions are forced to jump through now.
The rest of your comment rings true, although "spy agencies" is too narrow. It's spy agencies, police, the tax office, the social welfare agencies (like Centre Link in Australia), the financial agencies like Australia's ASIC.
I don't think it's politically possible to stop the data collection. A few real terrorist plots in Australia have been caught with this data, banks have been fined billions for allowing what is effectively money washing. Regardless of what I personally think, the Law Enforcement Agencies will say not collecting this information is the equivalent of "let the terrorists/paedophiles win" making any argument to stop it a near impossible sell.
But that doesn't matter. You can insist this data is collected without creating a wide open barn door like we have currently using cryptographic protocols. As an example, the government could you an app on your phone, lets say a car rental agency requires proof you have a valid drivers licence. You present the phone to an NFC reader, the phone says "XYZ Business has an authority to verify you have a current licence and it's expiry date", you click "hand it over", the phone provides the required data to the rental company. It's signed to prove to the rental company the the government has certified you have a licence expiring on some date. It also contains a lot of details about you the police can use to chase you down if you are in a road accident - but that's all encrypted, so no one can see it, including the car rental company. So it doesn't matter if it leaks. Barn door closed.
Well sort of. The car rental is probably going to collect a lot of information about you anyway, like a name, contact phone and address. But that's because they want it, not because the government demands it. Hopefully those details won't be sufficient to pull off identity fraud. The government ID's they are collecting now definitely are ID fraud material.
This way of handling sensitive data isn't novel. We already do something like this for credit card details. When a web site offers to remember your credit card, it's likely they've never seen it, let alone remember it. The only people who have seen it is the payment processor - Stripe maybe. What they are remembering is a token Stripe gives to them, which they can use to charge your card again. But the token reveals nothing about you. What's more it is useless to everyone bar them - Stripe won't accept it from anyone else.
I initially thought that it was gov mandated data, but I felt that given the available info in the article meant I couldn't back up that as a statement so backed off.
I would guess it's probably a mix of government mandated and business purposes, at the same time I suspect there's a bunch of info they want but that happens to overlap the gov requirements.
I led the technical design team for RealMe, New Zealand’s digital identity service. Sadly, the service is woefully undervalued and poorly used. The technology is now out dated.
It was built to directly mitigate events like the Latitude breach whereby the service gives an answer to an identity question rather than spraying PII across the economy. Answers were formed by pulling data from disparate authoritative sources in real time, a set of tokens were created and an audit record created and shared with PII owner consent. No personal information was ever intended to be shared or stored. It was an elegant solution for New Zealand, though we were mindful of a potential scaling issues in larger jurisdictions.
The financial sector was the initial target to help with AML/KYC flows. The banks in particular lobbied for access to the PII rather than an answer to the question so the service was devalued from the get go. If we’d won that answer I believe that digital identity and personal information sharing would be very different today.
I miss the days when NZ was truly innovative. RealMe was very promising when first introduced.
Of course it was never going to be adopted en masse by the private sector since part of the "get approval to use RealMe on your website" was "get Parliament to pass an Order in Council adding you to the authorized users schedule".
> However, if after people have worked with Latitude their privacy harms have not been resolved to their satisfaction, we encourage them to make a complaint to our Office.
How does an organization that has leaked PI even resolve a 'privacy harm'. Once that information is out it's out. there's no taking it back.
The biggest problem is how data is stored and used - it seems it got common place to have just one for all the customer dealings, rather than a number that can synch easily - but to combat data loss, keeping small database facing the web or other attack points exposing a limited amount of information removes much of the reward a lucky hacker might gain. For most uses of online and customers, it shouldn't need date of birth and their driver's licence number, just the necessities, like customer number and secure access ... maybe a hashed credit card number to confirm. Anything more than that generally requires people to contact support ... why then have a singular database that with the right exploit ... oops sorry about that.
Data is stored in many places in your average company. First of all in the main databases, then possibly on staging servers and in some cases even on test servers (where it probably really shouldn't be except in some very special circumstances). There are third party services that the company may have purchased where some or even all of the data is stored ((sub-)processors or in some cases co-controllers), or which have been given access. Hosting providers, database service providers, search tools etc. Then there are the backups, often unencrypted even today, and often accessible to too many people and not nearly secured as well as they should be.
Yes, the more data a company has directly facing the web or exploitable path with third parties, the greater the risk. Enough risk, having a private network beings to look cost effective.
In principle. But in practice this is quite hard, especially in smaller organizations, or in large organizations that have a relatively small tech department. And that's before you get into companies that use outsourcing.
Some stats show the majority of all breaches have an insider component, either wittingly or unwittingly.
My PII was compromised. I've never had a Gem Visa card, but I have had hire purchases 10 - 15 years ago with Harvey Norman, who some time ago shifted from in-house credit to using Latitude, and obviously uploaded their entire customer DB.
> Counter terrorism legislation requires than financial services companies store customer identification.
If there's a chink in the security, it's just a matter of time ... as such, there's a question that's not really being asked, does x or any third party need all the details. Sometimes they do because which ever company had found a cheap service to do all the account processing or some other task.
But more often I don't think third parties don't need full access to the master database via web access ... if they do then surely the customer needs to be informed who the company's partner is, and what that company's policy is to guarding any personal data loss/ misuse / retention.
From what I see lately (and there has been some massive data loss here in Australia in just the last year) there's a very care free lax attitude with a few shrugs after data is lost - with the hope naughty hackers can be blamed.
“A key finding from the NZ Institute of Directors’ Director Sentiment Survey report, released late last year, was that a significant proportion of boards were not sufficiently prepared for a digital future and had an “it won’t happen to us” approach. The message from the Office of the Privacy Commissioner is “wake up to yourselves”. We talk to organisations almost every week who are counting the cost of a cyber data breach. Can you risk the impact to your customers and your reputation?”
Agencies should not be collecting or retaining personal information unless it is necessary for a lawful purpose connected with their function or activity. All agencies should have a personal information retention schedule that they review regularly. The simple discipline of deciding how long information will be retained as you collect it and acting on these decisions will save you and your customers a lot of pain.”
So many panels, surveys, commissions, and watchdog groups state the obvious and yet little action on those recommendations is ever taken. In fact, many times the actions that are taken exacerbate the problem. The recently proposed bills to try and deal with TikTok and the social media data collection practices is a great example of this: propose sweeping new laws that violate privacy and free speech in order to combat privacy issues.
On the topic of data retention, though subtly different, can anyone succinctly explain to me when 'DELETE FROM' became a 4 letter word?
Everyone wants soft delete deleted_at instead. Everywhere. Many ORMs even force this behavior.
Why? Why can't we just delete records anymore? I'm sure some have specific reasons for this, but I'm trying to find the broader reason that everyone has jumped on this bandwagon.
Maybe I'm too YAGNI, but I prefer delete by default, unless justified. Not the inverse...
Because many items don't make sense without showing old deleted data.
Imagine you run a shop with a list of products. People can order and receive products from you. View past orders and get invoices and recepts.
One day you have a product that has reached end of life, or isn't selling well. So naturally you delete it from the store.
If you actually deleted this item, all the old invoices wouldn't be able to refer to it. Old customers couldn't claim warranty on it. You couldn't create old sales reports that made sense.
Turns out that this problem extends to most tables. Old clients? Users? Products? Invoices? In most cases outright deletion makes many other things invalid that you want to keep. So it's nice when the ORM makes meeting this requirement easy for you.
All this in addition to the ability to recover from an accidental or temporary deletion.
"Don't delete stuff you are still using" has nothing to do with "delete stuff you shouldn't be using and in many jurisdictions cannot legally possess".
I understand that this may not be the point you are trying to make, but it's important to note that deleting data can result in fines. As mentioned by others, regulated entities such as banks are required to retain data for a certain period of time. For example, a bank regulated by the EU needs to store most data for 10 years, which includes all emails sent by the bank, except for sensitive customer core data that is no longer needed.
So what happens if a bank employee accidentally deletes an email in Microsoft Exchange? This is can not happen due to the policies in place. All emails are stored in a hidden folder for as long as necessary, ensuring that no data is lost.
Deleting in a relational database often involves cascading to foreign keys to ensure data integrity is maintained. Delete a company row, and the employee rows may follow; that is probably not what most want.
FK cascading and DELETE, when done poorly, can... well, cascade, and delete lots of things the end user did not foresee. Like nuking half the database levels of bad.
`deleted_at' flags avoid that problem; combine it with row-level security in your DB (like postgres) and you can hide the rows permanently as well: from orms and non-superuser roles alike with a simple policy not to show any row where `deleted_at` is not null.
Use FK etc to ensure referential integrity but don't use it for cascading deletes. Speaking of Finance industry, stuff gets seldom deleted and when it does, more often than not, it is either manually orchestrated or the same orchestration is done via code. Sure it's more work but you are in full control of what is being removed at each stage.
I like cascading, but understand that risk. Why not just restrict everything, then? It forces the code to specify everything it wants to delete, no hidden nuking...
One broader reason could be "event sourcing" at a buzzword level. The allure of having a full history of the system/data at every point in time. Combine this with ai at the buzzword level. You could then use the full history of the world as input data to large ai models. But why?
Event sourcing allows business models and access to the data to change over time without having to modify the shape of historical data.
The ability to change your interface is what's alluring, not the mere fact that you have a record of changes. Though the audit trail is nice too.
Funnily enough you don't get to keep an audit trail while also losing the data. It's a trade off and businesses prefer the model that suits their business.
I think there's a certain amount of "you must keep records pertaining to financial transactions for N years" going on combined with (more recently, I suppose) abuse/spam protection.
Because someone always changes their mind and you get blamed if they cannot change their mind. We are used to a world of Ctrl-Z and customers expect that.
Someone accidentally deletes their company profile as they thought they were deleting the duplicate. They expect to be able to get it back.
Someone wipes out all the config files to get revenge on their employer. That employer is going to get in touch with support to get them back and they expect them back, even if you have to dig through backups and will make support's life difficult if they do not.
There are a lot of regulations about data retention in many industries, but now I think people making the decision aren't really sure and its easier to soft delete than get someone to approve a real deletion. In the old days hardware was a limiting factor so more likely to delete to keep performance up.
Snowden's "Permanent Record" is an interesting read, but at the end he seemed to view a transition from the government collecting and storing data on individuals to the private sector collecting and holding that data as some kind of positive event. It's really not any different, and neither entity is 100% secure against data theft and exposure.
Corporations and governments have an obsession with tracking individuals, although probably for different reasons (profits and tax collection being the main goals, I imagine, with 'social credit score' as a newer more dystopian concept). This is well-documented in the US and no doubt applies to NZ as well:
> "As we have documented at The Privacy Issue, pacts with U.S. intelligence are a mix of cooperation with, and infiltration by, three-letter agencies. Though nearly every major Big Tech company has cooperated with the NSA's surveillance programs in some way, the Google cloud was notably backdoored via clandestine means that were likely unknown to the company, with NSA agents gleefully bragging about the infiltration in an internal presentation."
New Zealand is a member of the so-called Five Eyes Collective so the NSA likely has full access to all personal data of New Zealanders, let alone everyone else in the region:
> he seemed to view a transition from the government collecting and storing data on individuals to the private sector collecting and holding that data as some kind of positive event. It's really not any different,
Isn't it different?
At worst, companies like Walmart, Google, and Amazon are creepy in using any information they can collect to try to sell you useless products.
At worst, the government can use any data they collect about you to throw you in a metal cage for life, take your kids away from you, drone strike you, steal your money, inject you with drugs without your consent, execute you, etc.
One is a mere annoyance. The other is a direct threat to your life, liberty, and property.
> One is a mere annoyance. The other is a direct threat to your life, liberty, and property.
Yes, but these are not compartmentalized how they should be.
The government has some limits on what data they can gather/store on you (mostly it seems these limits are respected less and less, but at least they exist and you have some chance to fight for them in court on constitutional grounds).
Private industry has no such limits. So they gather everything. But there are also no limits to how they can resell your private data, so a lot of it is resold to.. the government. Which gives a clean backdoor to bypassing those pesky limits.
Unless proven otherwise, you should assume that any of your private personal data gathered by private industry is also likely being passed on to the government.
I'm getting really annoyed by this one-sided trust with companies. We're just expected to hand over all our sensitive documents, with no idea how it's stored or who can access.
I was sent a shipment recently and the sender didn't put my last name. So the courier asked me to email them a copy of my passport or driver's licence. I rang them directly and was told that this is the only way of providing my identity. I said hell no and the package was sent back (they ended up resending it with my full name).
I find the worst culprit to be healthcare providers. I've had two recently ask me to send my credit card information in the
regular mail. I drove down to one of them to do it in person, and they put my details in a filing cabinet. I sure hope that lock is good!
(Pro tip: if you ever need to send CC details via mail and you cannot arrange an alternative, use registered post with signature so you can confirm it made it to the recipient)
Just to give some context, I got the email letting me know my data was part of the breach.
I bought an iMac on interest free finance direct from Apple in 2014 via Gem (which is now lattitude). So I imagine everyone who has bought a Apple product on finance is going to be caught up in this. Also looked at the Aussie Apple site and they still use Lattitude for finance payments.
I received a letter from a movie theater I worked for as a college student for a summer job, some fifteen years after I graduated college. They let me know a data breach had occurred exposing my social security number and (out-of-date, but still relevant for answering security questions) PII including address and telephone number.
The breach vector? Legally-mandated tax records. Someone cracked the padlock on the warehouse they were stored in and made off with the hard copies.
It is entirely possible that this sort of data exposure is, in the limit, impossible to guard against. Perhaps the better approach is to ask whether there's any way to make that data less valuable.
I've worked on DoD HPCMP systems and they would handle ITAR up to and including nuclear weapons simulations but wouldn't touch PII.
But the real risk, I think, is the collection of every packet going over the wire. There are entities storing every packet in the hopes of eventually using quantum decryption to decrypt it all. The question has moved from "should it be secret" to "how long must it be secret?" because the longer you need to keep it secret, the more computationally expensive the encryption is going to be.
> The question has moved from "should it be secret" to "how long must it be secret?" because the longer you need to keep it secret, the more computationally expensive the encryption is going to be.
To be fair, encryption strenght has always been rated on the basis of how long it should stay secret.
For example a site like https://www.keylength.com/en/compare/ (which has been around a very long time) guides you to pick key strenght based on how many years you'd want the data to be protected.
Think about every little tiny mom-pop medical office that either runs their own IT or pays a medical-industry-centered consultancy that usually has an IT branch. I think you’d be surprised to see what is out there and what is being used.
You can be charged for storing more data than is actually needed. For regulated entities like banks, it is important to have processes in place and demonstrate them during your annual audit. In case of non-compliance, you may face fines or even lose your license temporarily until the issue is fixed.
And that’s why GDPR is so important in Europe. There’s education needed, both for companies and users. Well informed people will better demand their rights, and hopefully change perception at companies.
In my job (infrastructure management) I am constantly fighting against people wanting to throw data away. "We know the condition of the bridge" they say - "so we don't need the past 20 years of maintenance records. It's too expensive to keep".
What they lose is the ability to understand how things change over the long term - how bridge supports and bearings degrade, so they can take proactive steps in the future and replace things in a planned way, rather than waiting for them to break.*
*Actually, what we do is a bit more complicated than this, but hopefully you get the point.
Please, doesn't help to pull that (not personally related data) in here, this is just not comparable.
Such data can be kept forever, maybe should even be public because of public interest, and likely even publicly owned.
Personal data / PII, history and also current cases again and again show should be minimized. Germany had it pretty right with its https://de.m.wikipedia.org/wiki/Datenvermeidung_und_Datenspa... though it gets attacked and holed out recently everyday again.. for the public interest they say, but for the profit interest of some corps they mean.. if not even for spying on citizens...
Maybe they don't want record because the data also holds incriminating evidence of how they understaffed the maintenance crew and skipped upkeep to fill their pockets in the short term.
+1, I don't think this is relevant in this context. When we speak of treating data as toxic assets to minimize its collection and retention, it's about personal information with privacy concerns, not something like maintenance records.
Equifax's breach in 2017 was essentially every valuable piece of PII they had, on more than half the households in America, and they settled for $300 million in a civil suit. Their net income (after all expenses) every year is in the $500-700 million range.
They should have been fined directly, an amount intentionally set to force complete liquidation, shareholders should have been completely wiped out, and all proceeds from the liquidation after court costs should have been distributed to every victim of the breach.
If that was the case, 100% of US companies would now treat PII with the respect it deserves. As it stands, nah, it's nbd here.