SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Auto-Deleting Data


Auto-Deleting Data

Author
Message
Steve Jones
Steve Jones
SSC Guru
SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)

Group: Administrators
Points: 602530 Visits: 21101
Comments posted to this topic are about the item Auto-Deleting Data

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Knut Boehnert
Knut Boehnert
SSCrazy
SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)

Group: General Forum Members
Points: 2567 Visits: 438
Personally I think auto-deleting data is a very bad idea - kind of burning the library of Alexandria. Not all data is valuable on its own however in context and with lots of points in data this brings back a picture of a civilisation.
In thousand years the issue of PII now is irrelevant. However someone looking for ancestors might just get lucky to know his foreforeforefather if that record still exist in some retrievable form of which we today think it is irrelevant.

The Sumerians pretty much had the same approach to their clay tablets. Good for some time, then break up and use as old tiles for the roof maybe.
Only reason we know of their accounting are a few of these clay tablets which allowed us to decipher their language.
skeleton567
skeleton567
Hall of Fame
Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)

Group: General Forum Members
Points: 3720 Visits: 650
Wow, Steve, you definitely pushed my buttons with this one. It is my humble opinion ( note I don't use the vernacular there ) that the evolution of mass storage has lead to a drastic decline in the skillset to properly analyze, summarize, and archive data. It's easier to throw money at storage than to logically, properly and safely handle data volume.

First of all, there needs to be logical analysis of what level of detail and age of data actually needs to be kept online and instantaneously available. Such things as customer purchase history of product that is no longer available. Detail of price paid thirty years ago. Where the product was shipped then. This requires an analytical skill to be properly accomplished.

Secondly, proper storage of detail seems to be currently way out of control. While the above mentioned things may be required legally and for business decisions, we don't need to be paying for mass storage online. Several of my early positions in IT required the use of off-line data storage for bulk data that could be retrieved IF and WHEN needed. And by off-line, I mean that literally, not just moved to another live system. We were simply expected to design this into our systems. And this required not only proper design of systems themselves but also the physical location and safety of the data. I haven't seen the old magnetic tape storage in decades, but it did serve a huge purpose in one-time cost, ease of off-premises storage but with acceptable accessibility within a reasonable time-frame. Today a removable disk drive can contain many, many times the volume at reasonable cost.

In the 'old days', it was often the case that local business would actually reciprocate and trade off-premises storage of bulk data. If summarization is adequately designed, appropriate detail may be identified and made available within a very reasonable timeframe.

IT professionals need to get over the idea that you can just pay ongoing costs for data storage and focus on designing both logical and physical storage with appropriate detail available for appropriate timeframes. There definitely still is a place of off-line bulk data storage.
Steve Jones
Steve Jones
SSC Guru
SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)

Group: Administrators
Points: 602530 Visits: 21101
Knut Boehnert - Tuesday, May 1, 2018 2:18 AM
Personally I think auto-deleting data is a very bad idea - kind of burning the library of Alexandria. Not all data is valuable on its own however in context and with lots of points in data this brings back a picture of a civilisation.
In thousand years the issue of PII now is irrelevant. However someone looking for ancestors might just get lucky to know his foreforeforefather if that record still exist in some retrievable form of which we today think it is irrelevant.

The Sumerians pretty much had the same approach to their clay tablets. Good for some time, then break up and use as old tiles for the roof maybe.
Only reason we know of their accounting are a few of these clay tablets which allowed us to decipher their language.


However, in a few years, the issues of PII are drastically relevant. I shudder to think how many children will have issues in adulthood because their data is out there now. I would argue plenty of information about culture and about ourselves is already being saved by individuals. Not sure organizations or corporations need to be able to do so.

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Steve Jones
Steve Jones
SSC Guru
SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)SSC Guru (602K reputation)

Group: Administrators
Points: 602530 Visits: 21101
skeleton567 - Tuesday, May 1, 2018 7:02 AM
Wow, Steve, you definitely pushed my buttons with this one. It is my humble opinion ( note I don't use the vernacular there ) that the evolution of mass storage has lead to a drastic decline in the skillset to properly analyze, summarize, and archive data. It's easier to throw money at storage than to logically, properly and safely handle data volume.

...
I do agree with some offline storage, but likely some of that still should go. The price paid or shipping location for a decade ago (or any PII data)? Not sure there's any legal reason to keep this, and I'd argue that a business reason is probably outweighed by risk for liability.


Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
skeleton567
skeleton567
Hall of Fame
Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)

Group: General Forum Members
Points: 3720 Visits: 650
Another thought regarding the use of off-line storage for archival purposes is that the technology, both physically and logically has to be kept up to date along with current systems. In the old days, once you committed data to paper, it was no longer a worry for IT people. This is going to affect system redesigns and upgrades, and never should be postponed for a more convenient time.
Eric M Russell
Eric M Russell
SSC Guru
SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)SSC Guru (108K reputation)

Group: General Forum Members
Points: 108479 Visits: 14551
It makes perfect sense to retain things like sales orders, financial transactions, court records, and health records in digital format indefinitely, because there may be an essential need for this historical information at a later point. But this type of normalized transactional data doesn't consume nearly as much storage as stuff like surveillance video, IoT telemetry, click stream, and binary objects.

The fact Susan deposited $100 to Tom's bank account back in 2004 may still be relevant today or even 10 years from now. The fact that Susan liked Tom's comment on FaceBook yesterday has questionable value today and will be totally irrelevant 10 years from now.


"The universe is complicated and for the most part beyond your control, but your life is only as complicated as you choose it to be."
ZZartin
ZZartin
One Orange Chip
One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)One Orange Chip (25K reputation)

Group: General Forum Members
Points: 25523 Visits: 17232
In general I don't believe in deleting relevant data, especially after working in places where even seemingly minor records remain relevant and end up being far more important that you'd think. In general I think that records involving monetary transactions and the exchange of goods and services never stop being relevant.

In regards to email I will say that where I work now we have a 3 month retention policy on normal email with the option to archive things for far longer.
skeleton567
skeleton567
Hall of Fame
Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)Hall of Fame (3.7K reputation)

Group: General Forum Members
Points: 3720 Visits: 650
Steve Jones - SSC Editor - Tuesday, May 1, 2018 7:28 AM
skeleton567 - Tuesday, May 1, 2018 7:02 AM
Wow, Steve, you definitely pushed my buttons with this one. It is my humble opinion ( note I don't use the vernacular there ) that the evolution of mass storage has lead to a drastic decline in the skillset to properly analyze, summarize, and archive data. It's easier to throw money at storage than to logically, properly and safely handle data volume.

...
I do agree with some offline storage, but likely some of that still should go. The price paid or shipping location for a decade ago (or any PII data)? Not sure there's any legal reason to keep this, and I'd argue that a business reason is probably outweighed by risk for liability.


OK, just for instance, knowing that a decade ago a customer bought $1000 worth of a product probably was a much larger volume of the product than $1000 of the same product now, and that get's into the whole area of cost of storage, sales, transportation, and profitability, all adjusted for inflation. You can tell I'm still a detail-oriented obsessive-compulsive, right?
Rod
Rod
One Orange Chip
One Orange Chip (28K reputation)One Orange Chip (28K reputation)One Orange Chip (28K reputation)One Orange Chip (28K reputation)One Orange Chip (28K reputation)One Orange Chip (28K reputation)One Orange Chip (28K reputation)One Orange Chip (28K reputation)

Group: General Forum Members
Points: 28186 Visits: 2705
Good questions you've raised, Steve. Like you, I tend to hold onto emails for long times. At my old job we had fewer resources, so we were involved in getting rid of a lot of the older data. We used to have a huge room full out paper documents dating back 12 years or more. We were involved in converting all of that data into electronic format. Then after that we never saved any paper documents. Everything that involved a client's signature was scanned in, then shredded. But even the electronic format we couldn't keep, just didn't have the storage. I think the adult clients we were required to keep for 12 years. Adolescent clients we were required to keep for 18 years. After that, we had to archive and remove.

At my current job they have deeper pockets and a mindset to hold onto everything. So, at this point everything is stored forever, or at least practically forever. I do wonder, though, how we would handle something like GDPR, with the "right to be forgotten". It would be a huge paradigm shift here.

Kindest Regards,Rod
Connect with me on LinkedIn.
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum








































































































































































SQLServerCentral


Search