Keep Data Forever

SQLServerCentral is supported by Red Gate Software Ltd.

Popular Topics

Home

Members

Calendar

Who's On

Home » SQLServerCentral.com » Editorials » Keep Data Forever

Keep Data Forever

Rate Topic

Display Mode

Topic Options

Author

Message

Steve Jones - SSC Editor

Posted Wednesday, August 24, 2011 9:15 PM

SSC-Dedicated

Group: Administrators
Last Login: Today @ 3:19 PM
Points: 31,526, Visits: 13,863

Comments posted to this topic are about the item Keep Data Forever

Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help

Post #1165092

mar10br0

Posted Wednesday, August 24, 2011 10:05 PM

SSC-Enthusiastic

Group: General Forum Members
Last Login: Friday, October 05, 2012 3:30 AM
Points: 138, Visits: 351

The problem of readability of (very) old data from a backup is twofold: a) is the data physically readable (the bits) and b) can those bits be translated into meaningful data (ie is the file-format still supported by current software or is the original software still around and executable on current operating systems or can the OS needed to execute the software for reading the data-file installed on current hardware or can the old hardware with the old OS needed to execute the original software for reading the data-file be connected to your current network?
So, for archiving data you need to be very conservative in both physical as well as "soft" format of the data-backup. The physical problem is the easiest to solve (copy the backups to newer media every few years). A similar technique could be applied to the "soft" problem at the same time: when copying to newer media you could convert the files into a more modern file-format.
So it is not going to be a set-and-forget solution (putting the backup-media into a file-cabinet in the cellar). Because before you know it it cannot be retrieved into a meaningful form.

The end-result for me is that my portable USB-drive contains excel-files originally created with Lotus 1-2-3 in DOS on my XT (remember that one Crazy

?). Begs the question why I keep them though, I haven't touched them since I converted them into Excel about 7 years ago or so.... But I keep faithfully copying them over whenever I move to a next backup-solution...

Post #1165102

Jack Corbett

Posted Thursday, August 25, 2011 6:52 AM

SSChampion

Group: General Forum Members
Last Login: Today @ 1:41 PM
Points: 10,613, Visits: 11,952

Interesting that this editorial comes up now as my current project is to design a purge of older data. I'm not archiving any of the data, but the needed data is archived through documents that have been generated which are stored electronically in the CMS system. Now how long they'll keep the documents is not my worry. Tongue

Jack Corbett

Applications Developer

Don't let the good be the enemy of the best. -- Paul Fleming

Check out these links on how to get faster and more accurate answers:
Forum Etiquette: How to post data/code on a forum to get the best help
Need an Answer? Actually, No ... You Need a Question
How to Post Performance Problems
Crosstabs and Pivots or How to turn rows into columns Part 1
Crosstabs and Pivots or How to turn rows into columns Part 2

Post #1165299

Nadrek

Posted Thursday, August 25, 2011 7:55 AM

Say Hey Kid

Group: General Forum Members
Last Login: Monday, June 10, 2013 1:08 PM
Points: 679, Visits: 2,038

Keeping data forever?
There are a few challenges:
Physical locations fail.

Media fails catastrophically (crashes). Try to keep it on two _different_ forms of media, at least one of which should not be vulnerable to shock.

Media fails slowly (bit rot). Try to use something that is resistant to this, something that can detect this, and something that can recover afterward.
Checksums and hashes (preferably SHA512 or another 512-bit current generation hash) for larger files... like databases can detect this; but don't resist and can't help you recover.
RAID1 and RAID10 is not necessarily a good choice; does your implementation correctly figure out which of two supposedly good copies that are somehow different is correct, assuming neither drive is reporting a bad block?
RAID5 and RAID6 fit all three criteria if and only if consistency checks are performed; without this, it's entirely possible for bit rot to be hidden until one (RAID5) or two (RAID6) drive fails, after which they cannot recover (and may not be able to detect).
Reed-solomon and other ECC software solutions (DVDisaster for CD, DVD, and Blu-Ray, Par2+TBB for files in general, etc.) can both detect and recover from bit rot or partial physical media destruction.

Means to read your media no longer exist. So; you've got some reel to reel tape, likely in EBCDIC, in the closet. You're going to read this how? Or a tape cartridge from the late 80's? 8" floppies? A stack of punch cards? Even if it is good (not terribly likely except for the punch cards, which are probably fine), do you have anything left that can read it? Do you have something to plug it in to? Can that something talk to anything else anymore, and if so, how close to modern tech can you get with each jump?
You need to keep moving the archived data onto newer media, and running some kind of bit rot check every time, making sure each copy really is accurate.

Catalogs are forgotten. A much harder problem; after twenty years, if there's no one left that remembers either what file X is, or how to answer question Y... what good are your forever archives, particularly if you have eight versions of almost the same thing?

Post #1165362

michael.french 172

Posted Thursday, August 25, 2011 7:56 AM

SSC Veteran

Group: General Forum Members
Last Login: Wednesday, February 20, 2013 11:12 AM
Points: 245, Visits: 564

I tend to keep old database backups on hard drives, USB Keys, CDs and/or DVD’s. I recently had a client ask me about something we designed eight years ago. I was able to restore the backup and answer the question.

I had another client years ago who knew that I have a tendency to stash db backups all over the place. They lost a machine and had not backed up their database for 1.5 years. They found an old backup and were at least able to get something restored. If it were not for the old backup they would have had nothing.

The point is to keep as much as you can as long as you can in as many places as you can. At the end of the day we are all DBAs and one of our core responsibilities is to protect the data. You never know when something is better than nothing.

Post #1165365

tslaxar

Posted Thursday, August 25, 2011 9:12 AM

Forum Newbie

Group: General Forum Members
Last Login: Monday, August 13, 2012 12:13 PM
Points: 2, Visits: 38

The Sumerians have already successfully addressed all these issues. You want to store your data in cunieform on low-fired clay tablets.

We have and can reading Sumerian accounting data that is over 5,000 years old. Both the storage and coding solutions they used have proven track records of durability and usability.

Smile

Post #1165450

Stephanie J Brown

Posted Thursday, August 25, 2011 10:01 AM

Mr or Mrs. 500

Group: General Forum Members
Last Login: Yesterday @ 11:46 AM
Points: 512, Visits: 878

The first question really is "should you be keeping the data?" Wikipedia has a nice overview of that issue at http://en.wikipedia.org/wiki/Records_management

Too few companies pay attention to records retention standards even for paper documents. Digital data is just a different delivery system in many respects, so I believe the existing rules would apply.

Once you've determined what data to keep, the technology issues of how to maintain it comes into play. The VERS standard from Australia may come in handy: http://210.8.122.120/vers/vers/default.asp

Here there be dragons...,

Steph Brown

Post #1165512

Randy Rabin

Posted Thursday, August 25, 2011 10:45 AM

SSC-Addicted

Group: General Forum Members
Last Login: Monday, June 10, 2013 3:16 PM
Points: 498, Visits: 509

You want to store your data in cunieform on low-fired clay tablets.

Cunieform - I had to snicker at this. Yet essentially correct. Who knows if we'll be using the same language in 1000 years, much less that a glyph will be represented by eight (or sixteen - we can't even standardize on that) 1's and 0's that far ahead? As others have said, I agree the problem won't be the method of storing the data, it'll be reading/interpreting it back out again. Anyone want to try reverse-engineering table data out of an MDF file bit dump? w00t

Post #1165565

Peter Maloof

Posted Sunday, August 28, 2011 9:15 AM

SSC Veteran

Group: General Forum Members
Last Login: Friday, May 31, 2013 8:05 AM
Points: 295, Visits: 1,245

This sounds like an annual project: verify long-term backups are on supported media, and in a usable format.

Peter Maloof
Serving Data

Post #1166681

Peter Maloof

Posted Sunday, August 28, 2011 9:20 AM

SSC Veteran

Group: General Forum Members
Last Login: Friday, May 31, 2013 8:05 AM
Points: 295, Visits: 1,245

tslaxar (8/25/2011)

You want to store your data in cunieform on low-fired clay tablets.

Can someone please tell me the collation setting for Cunieform? I can't seem to find it.

Peter Maloof
Serving Data

Post #1166682

« Prev Topic | Next Topic »

Permissions