It took me 24 hours to come up with the right adjective, but I ended up with "catastrophic" data loss. During a routine upgrade of RE, our database was lost, the backup overwritten, and the manual backup failed. It fell into a Bermuda Triangle in my MIS department. I only heard "this has never happened," and "this should not have happened." But now I have to make it all happen again. The data file I have is the original file used to migrate to RE in Dec 2004. I lost everything done in the coversion process and four years of records/maitenance. I now have to rebuild. Has anyone ever written a plan for conversion or recovery/restoration that they would be willing to share. It is so overwhelming that it is hard to know where to start. Well other than with a bottle of tequila... Emotional as well as techincal support welcome.

Hi Donna:  Did you just apply the 7.84 patch for Raiser's Edge? 

Robert

 PS: Why does this say I'm posting @ 2:29AM? 

No establish plan of recovery in writing but we make it a practice to do a database restore to a different directory and attach the database on the console.  By doing so we can see if your backups are working.  It's done every four months as a standard practice.  Not much now that you lost everything but I will take you on the "ottle of tequila"

My sympathy.  Are you looking for advice on how to recover your four years of missing data, or how to avoid this to happen again ever?  Testing your back up on a regular basis as suggested above is a very basic thing to do. 

In terms of lost data, well, it depends on the volume you have.  For a small volume and if you have paper back up, you could hire someone to do some very basic gift entry, such as amount, C,A,F - forget the details, to speed up the process. I believe it is also possible to send your data outside for address correction once everything is re-entered.

Good luck though.

 Ouch. This is a painful story to hear. We tape backup every night. Additionally, I regularly copy the backup files every two weeks on two different sets of zip drives and take one set home. The most we could lose is about 2 weeks of data, and this would be horrendous. I can't imagine four years worth. I test backups less frequently. Your story has inspired me to increase my backup testing. Wishing you the best.

Not to scare anyone even further than this story already did but please please please do not simply rely on the fact that something is backed up to a tape and tapes are saved, sent off site, etc.  If you are not testing your backups you could be in big trouble.  Somewhere on the BB site I remember reading a story about an org that regularly backed up to tape for 6 years but never tested the backup.  The day they needed it when their database crashed, they found out that the tapes they were so careful about using daily and rotating and saving and moving offisite... were all blank.

There are implementation and upgrade guides and checklists, but not recovery ones. Because I am at a pre-conversion place--no codes or campaign structures. Data has no RE fields. People who were dead, are alive again. It is hard to know where to start. Manually re-entering gift data is my number one priority. But right now we are working on de-duplication and re-importing mailing lists. For weeks we have been working just to correct address information before adding gift information. When we start to enter data, where do we start? My database administrator wants to start with 2005. I want to start with 2008. Apparently, sometimes RE, although it was corrected I am told, will take last gift based on entry date not gift date. When you stand in the rubble, where do you start? What is the best way to rebuild.

 Correcting the backup and testing those procedures has already been done. But I am moving to web-hosting with Blackbaud. I don't have server access. I cannot do back ups. And our MIS dept may not be as well suited as Blackbaud to manage the data.They don't know the management console.Automatic manuals were not happening as scheduled. One of the reasons the data was overwritten is because file size of migration database and backup database were the same. The claimed this was a scientific impossibility.Until we learned it was partitioned, so file size wouldn't have changed in four years.  

 Space shuttles blow up. Data disappears. It isn't supposed to happen, but it can and it does. Denial is the first step. It took me a long time to accept it.

Donna, take my advice for what it is worth, since I have never been in such situation, but I would start by 2008.  You do not give arguments of both sides but basically, for me, 2008 is your most recent, up-to-date data.  To a limit, you could be functional for an upcoming annual mailing or a newsletter mailing very quickly, having all your most recent donors available.  Historical data is nice, especially for in-depth analysis and trends, donor walls which are based on cumulative giving, etc. but really, do you mind right now?  I would create codes as you need them.  Often our tables get clogged with inactive old codes so, ahem, I suppose that you got an unexpected chance to clean up.  :-(

Unfortunately, proceeding this way will be a disavantage for pledges. It depends how many of them you have.

Hope it helps.

This horror story has done it's job in creating fear.  We use tape backups and carry them off site.  However, we have not been testing the backup, so that's now on my list of things to do.  How do some of you backup your data?  Tape backups, external hard drives, zip drives,  outside companies doing the backups for you?  How often do you test the backup?  Do you always go through the backup procedure in the Blackbaud Management Console or do you backup through some other method on your server?  Any additional comments/suggestions/tips are welcome.  Thanks-  Nancy Williford

I must say - TEST test test test and re test backups procedures before you ACTUALLY need them :)   I cant stress enough DO not take tapes/archives home with you, this is no better.  If you drop the tape, leave it in your car, or get in an accident, not only is you data at risk of being stolen, you could lose your offsite backup!  You need to off-site them electronically, or VIA data currier.

Change the number of backups you store, and then archive them somewhere else.  We store 30 days of backups on site and offsite every backup for 6 months.  Along with this we keep our 6month backups for 3 years.

It sounds like overkill, but when your dealing with thousands of hours of data entry, you need to have a recovery plan. And you need to test it.

Get a procedure down, it does not have to be long, but it DOES have to be accurate.

Include

*how often backups should be made

*number of onsite and offsite backups to store

*encryption keys if needed

*regular testing of backups on a test workstation

*Recheck all backup schedules and procedures after ANY and EVERY software upgrade (Blackbaud, Windows, or SQL)

Donna, I'm sorry to hear about your situation! In my previous position, we cycled the backup tapes. When I was made the Database administrator, I ran a test, and yes, they weren't backing up properly and were essentially useless. We were fortunate that we hadn't needed them during that period!