Log in
::
Register
::
Not logged in
Home
Tags
Articles
Editorials
Stairways
Forums
Scripts
Videos
Blogs
QotD
Books
Ask SSC
SQL Jobs
Authors
About us
Contact us
Newsletters
Write for us
Daily SQL Articles by email:
Sign up
Back
SQLServerCentral
Register
Home
»
SQLServerCentral.com
»
Editorials
»
Disaster After Disaster
Disaster After Disaster
Post reply
Like
0
Add to Briefcase
1
2
Next
Disaster After Disaster
View
Options
Author
Message
Steve Jones
Steve Jones
Posted 6 years ago
#1395573
SSC Guru
Group: Administrators
Points: 403723
Visits: 20465
Comments posted to this topic are about the item
Disaster After Disaster
Follow me on Twitter:
@way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog:
www.voiceofthedba.com
0
Quote
Scott Arendt
Scott Arendt
Posted 6 years ago
#1395647
SSCertifiable
Group: General Forum Members
Points: 5835
Visits: 1733
The company that I work for provides services to monitor generators and we work closely with a company that provides maintenance on the generators. It is a recommended practice to start those generators at least once a month, allowing the switchgear to transfer the load from utility to the generator to make sure that all of the systems are working properly.
It is not unusual for customers to end the regular test periods because of the cost of fuel. I am aware of situations where generators have sat for years without any tests or maintenance and then, surprise, they did not function correctly when they were needed in an emergency situation.
I kind of equate this to taking regular database backups, but never restoring them to ensure that everything is working. It's easy to say that you are covered in an emergency, but you better be able to follow up.
0
Quote
GSquared
GSquared
Posted 6 years ago
#1395681
SSC Guru
Group: General Forum Members
Points: 169963
Visits: 9732
I know of a company that tested its generators every month. Then, when disaster struck, it turned out they had half an hour's worth of fuel left, because they'd never refilled the tanks after all those tests. The power outage lasted something like four hours.
Can't say what company it was, but their servers being down actually had important law enforcement implications of a very negative sort.
They also had set up replication to a co-lo. They'd never tested it. So, when the gennies went down, the co-lo failed to come up.
Worked with another company that, during a hurricane, moved all employees to a hotel in another city, moved the servers to a co-lo, then found out that the type of connection the hotel's business center had didn't allow their applications to connect to their servers. The servers were up and running, the workstations were up and could connect to websites, but the two couldn't talk to each other. (I don't know enough about internet connections to know why this was so. But it definitely was.) A couple of salespeople with laptops and cellular data plans (this was before smartphones became popular) were the only ones able to connect apps to data.
Again, lack of taking "everything" into account.
I also worked with a company that, when an earthquake took down their primary data center, they had about 1 second of downtime, and then just kept on going. They had a plan, it was complete, it was drilled (practiced), it was clearly understood by and known to everyone involved, it had as much automation to it as made sense, and so on. Worked like a dream. A few years later, another data center had a blackout (someone apparently crashed a truck into a transformer a few blocks away), and, again, no significant downtime, no measurable interruption in work.
It really does depend on the level of detail of planning, and how well everyone knows the plan and has certainty on it.
- Gus "GSquared", RSVP, OODA, MAP, NMVP, FAQ, SAT, SQL, DNA, RNA, UOI, IOU, AM, PM, AD, BC, BCE, USA, UN, CF, ROFL, LOL, ETC
Property of The Thread
"Nobody knows the age of the human race, but everyone agrees it's old enough to know better." - Anon
0
Quote
jay-h
jay-h
Posted 6 years ago
#1395702
SSCrazy Eights
Group: General Forum Members
Points: 9584
Visits: 2665
It reminds us that there are all sorts of problems that must be considered. Our particular site (in a former manufacturing site) has 3 Caterpillar diesel generators and a storage tank large enough to run for a month.
We did have a close call with the T3 lines however. We have two completely independent lines, one from a conventional telecom and one from a cable company. The cable company's system went down and stayed down for a week. Fortunately the other one did not (though it HAS failed in the past for other reasons). If that one had failed as well, we would be sunk, and it's possible that key employees could not even get to our DR site.
My neighborhood was down for 11 days (I had by pure luck purchased a generator 2 weeks before Sandy. That same cable company was out in my neighborhood, and they actually placed a portable generator by a utility pole to power their customer's phones which was considered critical because of no 911 service.)
...
-- FORTRAN manual for Xerox Computers --
0
Quote
Sigerson
Sigerson
Posted 6 years ago
#1395710
SSCrazy
Group: General Forum Members
Points: 2439
Visits: 1245
My company is in New Jersey and lost power during Hurricane Sandy. We had a meticulous business continuation plan and our colocation site had us up and running within 24 hours. The plan assumed that we would just log in from home and keep the company running. But in north central Jersey, nearly every customer had no electricity for the first few days, literally 4-5 million people with no lights and no heat, (I was one of them,) but also no Internet connections, no wifi, etc. By the time the colo site was up, most of our workers could not be contacted because their cell phones had run down and most landlines these days rely on cable or optical modems that don't run without power. No one had Internet access for the same reason.
If you have a BC plan, great, but you should examine every contingency and not just assume "that won't happen." Our assumption that the business might be down but "of course" the workers could just log in from home was, in hindsight, rather stupid on our part.
I just wanted to emphasize Steve's comment:
When a disaster gets large enough, it doesn't matter what you have contracted for.
Sigerson
"
No pressure, no diamonds.
" - Thomas Carlyle
0
Quote
Ross McMicken
Ross McMicken
Posted 6 years ago
#1395782
SSCrazy
Group: General Forum Members
Points: 2657
Visits: 2275
A few comments on disasters.
If you are critical, then you need a real landline and a phone that doens't require power to operate. This is one huge reason we have an old fashioned land line at home. It always works.
Your DR plans should be tested with a scenario where more than half your staff can't be found or can't connect. In real life, you can't count on everyone being available.
If it's available, look into running your emergency power on natural gas. Gas is safer than liquid fuels, and interruptions are rare. Many diesels can be modified to run on natural gas. If I ever get a generator for the house, it will be gas or propane powered
0
Quote
Sigerson
Sigerson
Posted 6 years ago
#1395785
SSCrazy
Group: General Forum Members
Points: 2439
Visits: 1245
Ross McMicken (12/12/2012)
If it's available, look into running your emergency power on natural gas. Gas is safer than liquid fuels, and interruptions are rare.
A good point and one that applies to having a hard landline as well.
Sigerson
"
No pressure, no diamonds.
" - Thomas Carlyle
0
Quote
jay-h
jay-h
Posted 6 years ago
#1395800
SSCrazy Eights
Group: General Forum Members
Points: 9584
Visits: 2665
Ross McMicken (12/12/2012)
A few comments on disasters.
If you are critical, then you need a real landline and a phone that doens't require power to operate. This is one huge reason we have an old fashioned land line at home. It always works.
Your DR plans should be tested with a scenario where more than half your staff can't be found or can't connect. In real life, you can't count on everyone being available.
If it's available, look into running your emergency power on natural gas. Gas is safer than liquid fuels, and interruptions are rare. Many diesels can be modified to run on natural gas. If I ever get a generator for the house, it will be gas or propane powered
Actually for many NJ residents, landlines were out for days. At least a cellphone can be charged from a car battery.
...
-- FORTRAN manual for Xerox Computers --
0
Quote
Steve Jones
Steve Jones
Posted 6 years ago
#1395815
SSC Guru
Group: Administrators
Points: 403723
Visits: 20465
Scott Arendt (12/12/2012)
The company that I work for provides services to monitor generators and we work closely with a company that provides maintenance on the generators. It is a recommended practice to start those generators at least once a month, allowing the switchgear to transfer the load from utility to the generator to make sure that all of the systems are working properly.
It is not unusual for customers to end the regular test periods because of the cost of fuel. I am aware of situations where generators have sat for years without any tests or maintenance and then, surprise, they did not function correctly when they were needed in an emergency situation.
I kind of equate this to taking regular database backups, but never restoring them to ensure that everything is working. It's easy to say that you are covered in an emergency, but you better be able to follow up.
We have a home generator that runs a self-test once a week, running for 10 minutes. I also try to check fuel before vacation because for our business (horses) this is critical.
Follow me on Twitter:
@way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog:
www.voiceofthedba.com
0
Quote
Steve Jones
Steve Jones
Posted 6 years ago
#1395816
SSC Guru
Group: Administrators
Points: 403723
Visits: 20465
Ross McMicken (12/12/2012)
Your DR plans should be tested with a scenario where more than half your staff can't be found or can't connect. In real life, you can't count on everyone being available.
I've always made this a priority. In a disaster, you can't assume that people can be contacted, or that they will come. I've told my boss that if my family is in trouble, I'm not coming to work.
If it's available, look into running your emergency power on natural gas. Gas is safer than liquid fuels, and interruptions are rare. Many diesels can be modified to run on natural gas. If I ever get a generator for the house, it will be gas or propane powered
We use Propane here, but you have to be careful with nat gas. Lots of people assume the lines will be working in a disaster, and they may get shut down. A local supply on site is needed.
Follow me on Twitter:
@way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog:
www.voiceofthedba.com
0
Quote
Go
Post reply
Like
0
Add to Briefcase
1
2
Next
Post quoted reply
Permissions
You
can't
post new topics.
You
can't
post topic replies.
You
can't
post new polls.
You
can't
post replies to polls.
You
can't
edit your own topics.
You
can't
delete your own topics.
You
can't
edit other topics.
You
can't
delete other topics.
You
can't
edit your own posts.
You
can't
edit other posts.
You
can't
delete your own posts.
You
can't
delete other posts.
You
can't
post events.
You
can't
edit your own events.
You
can't
edit other events.
You
can't
delete your own events.
You
can't
delete other events.
You
can't
send private messages.
You
can't
send emails.
You
can
read topics.
You
can't
vote in polls.
You
can't
upload attachments.
You
can
download attachments.
You
can't
post HTML code.
You
can't
edit HTML code.
You
can't
post IFCode.
You
can't
post JavaScript.
You
can
post emoticons.
You
can't
post or upload images.
Select a forum
SQL Server 2017
SQL Server 2017 - Administration
SQL Server 2017 - Development
SQL Server 2016
SQL Server 2016 - Administration
SQL Server 2016 - Development and T-SQL
SQL Server 2014
Administration - SQL Server 2014
Development - SQL Server 2014
SQL Server 2012
SQL 2012 - General
SQL Server 2012 - T-SQL
SQL Server vNext
SQL Server 15 - Administration
SQL Server 15 - Development
SQL Server 2008
SQL Server 2008 - General
T-SQL (SS2K8)
June 2007 CTP
Working with Oracle
July CTP
SQL Server Newbies
Security (SS2K8)
SQL Server 2008 High Availability
SQL Server 2008 Administration
Data Corruption (SS2K8 / SS2K8 R2)
SQL Server 2008 Performance Tuning
Cloud Computing
SQL Azure - Development
SQL Azure - Administration
Amazon AWS and other cloud vendors
General Cloud Computing Questions
CosmosDB
Azure Data Lake
Azure Machine Learning
Reporting Services
Reporting Services
Reporting Services 2005 Administration
Reporting Services 2005 Development
Reporting Services 2008/R2 Administration
Reporting Services 2008 Development
SSRS 2012
SSRS 2014
SSRS 2016
Programming
Connecting
General
SMO/RMO/DMO
XML
Service Broker
Powershell
Testing
TFS/Data Dude/DBPro
SSDT
Continuous Integration, Deployment, and Delivery
R Services and R Language
Data Warehousing
Integration Services
Strategies and Ideas
Analysis Services
Data Transformation Services (DTS)
Performance Point
Data Mining
PowerPivot
R language
Machine Learning
Database Design
Disaster Recovery
Design Ideas and Questions
Relational Theory
Hardware
Virtualization
Security and Auditing
SQLServerCentral.com
Anything that is NOT about SQL!
Contests!
Editorials
SQLServerCentral.com Announcements
SQLServerCentral.com Website Issues
Suggestions
Tag Issues with Content
Podcast Feedback
SQLServerCentral.com Test Forum
Articles Requested
SQL Server 2005
Administering
Backups
Business Intelligence
CLR Integration and Programming.
Data Corruption
Development
Working with Oracle
SQL Server 2005 Compact Edition
SQL Server 2005 General Discussion
SQL Server 2005 Security
SQL Server 2005 Strategies
SS2K5 Replication
SQL Server Express
SQL Server 2005 Performance Tuning
SQL Server 2005 Integration Services
T-SQL (SS2K5)
SQL Server Newbies
SQL Server 7,2000
Administration
Backups
Data Corruption
General
Globalization
In The Enterprise
Working with Oracle
Security
Strategies
SQL Server Newbies
Service Packs
SQL Server CE
Performance Tuning
Replication
Sarbanes-Oxley
T-SQL
SQL Server Agent
SQL Server and other platforms
MySQL
Oracle
PostgreSQL
DB2
SQL Server and Sharepoint
Older Versions of SQL (v6.5, v6.0, v4.2)
Older Versions of SQL (v6.5, v6.0, v4.2)
Career
Certification
Employers and Employees
Events
Job Postings
Resumes and Job Hunters
Presentations and Speaking
Retired Members
Testing Center
Question of the Day (QOD)
SQL Server Security Skills
Microsoft Access
Microsoft Access
Products and Books
Third Party Products
CA
SQLCentric
Extreme Technologies.
Innovartis
Embarcadero
SQL Sentry
Golden Gate Software
Idera
Lumigent
Red Gate Software
Quest Software
ApexSQL
Sonasoft
Discussions about Books
Discuss Programming Books
Discuss XML Books
Discuss Data Warehousing Books
Discuss T-SQL Books
Discuss DTS Books
Discuss SQL Server 2000 Books
Discuss SQL Server 7.0 Books
Notification Services
Administration
Article Discussions
Future Versions
SQL 12
Narrow your search by forum
Explore
Home
Latest
Popular
Calendar
Members
Who's on
Moderators
Forums
SQL Server 2017
SQL Server 2017 - Administration
SQL Server 2017 - Development
SQL Server 2016
SQL Server 2016 - Administration
SQL Server 2016 - Development and T-SQL
SQL Server 2014
Administration - SQL Server 2014
Development - SQL Server 2014
SQL Server 2012
SQL 2012 - General
SQL Server 2012 - T-SQL
SQL Server vNext
SQL Server 15 - Administration
SQL Server 15 - Development
SQL Server 2008
SQL Server 2008 - General
T-SQL (SS2K8)
June 2007 CTP
Working with Oracle
July CTP
SQL Server Newbies
Security (SS2K8)
SQL Server 2008 High Availability
SQL Server 2008 Administration
Data Corruption (SS2K8 / SS2K8 R2)
SQL Server 2008 Performance Tuning
Cloud Computing
SQL Azure - Development
SQL Azure - Administration
Amazon AWS and other cloud vendors
General Cloud Computing Questions
CosmosDB
Azure Data Lake
Azure Machine Learning
Reporting Services
Reporting Services
Reporting Services 2005 Administration
Reporting Services 2005 Development
Reporting Services 2008/R2 Administration
Reporting Services 2008 Development
SSRS 2012
SSRS 2014
SSRS 2016
Programming
Connecting
General
SMO/RMO/DMO
XML
Service Broker
Powershell
Testing
TFS/Data Dude/DBPro
SSDT
Continuous Integration, Deployment, and Delivery
R Services and R Language
Data Warehousing
Integration Services
Strategies and Ideas
Analysis Services
Data Transformation Services (DTS)
Performance Point
Data Mining
PowerPivot
R language
Machine Learning
Database Design
Disaster Recovery
Design Ideas and Questions
Relational Theory
Hardware
Virtualization
Security and Auditing
SQLServerCentral.com
Anything that is NOT about SQL!
Contests!
Editorials
SQLServerCentral.com Announcements
SQLServerCentral.com Website Issues
Suggestions
Tag Issues with Content
Podcast Feedback
SQLServerCentral.com Test Forum
Articles Requested
SQL Server 2005
Administering
Backups
Business Intelligence
CLR Integration and Programming.
Data Corruption
Development
Working with Oracle
SQL Server 2005 Compact Edition
SQL Server 2005 General Discussion
SQL Server 2005 Security
SQL Server 2005 Strategies
SS2K5 Replication
SQL Server Express
SQL Server 2005 Performance Tuning
SQL Server 2005 Integration Services
T-SQL (SS2K5)
SQL Server Newbies
SQL Server 7,2000
Administration
Backups
Data Corruption
General
Globalization
In The Enterprise
Working with Oracle
Security
Strategies
SQL Server Newbies
Service Packs
SQL Server CE
Performance Tuning
Replication
Sarbanes-Oxley
T-SQL
SQL Server Agent
SQL Server and other platforms
MySQL
Oracle
PostgreSQL
DB2
SQL Server and Sharepoint
Older Versions of SQL (v6.5, v6.0, v4.2)
Older Versions of SQL (v6.5, v6.0, v4.2)
Career
Certification
Employers and Employees
Events
Job Postings
Resumes and Job Hunters
Presentations and Speaking
Retired Members
Testing Center
Question of the Day (QOD)
SQL Server Security Skills
Microsoft Access
Microsoft Access
Products and Books
Third Party Products
CA
SQLCentric
Extreme Technologies.
Innovartis
Embarcadero
SQL Sentry
Golden Gate Software
Idera
Lumigent
Red Gate Software
Quest Software
ApexSQL
Sonasoft
Discussions about Books
Discuss Programming Books
Discuss XML Books
Discuss Data Warehousing Books
Discuss T-SQL Books
Discuss DTS Books
Discuss SQL Server 2000 Books
Discuss SQL Server 7.0 Books
Notification Services
Administration
Article Discussions
Future Versions
SQL 12
SQLServerCentral
Register
Search
Narrow your search by forum
Unthreaded, ascending
Unthreaded, descending
Subscribe to topic
Print topic
RSS feed
Go to topics forum
Jump to page
Jump to page
Copyright © 2002-2018 Redgate. All Rights Reserved.
Privacy Policy.
Terms of Use.
Report Abuse.