Donít Get Caught With Your Servers Down
by Donovan Griffin
Melissa, Olga, and Sebastien may be passing through town. Are you prepared?
Gerry Bell, lead seasonal hurricane forecaster of the National Oceanic and Atmospheric Administration (NOAA), wants to prepare people for the Atlantic hurricane season, which officially began June 1. “This year, oceanic and atmospheric conditions in the Atlantic basin are expected to produce more and stronger hurricanes,” said Bell on the NOAA blog.
In addition, researchers from the department of atmospheric science at Colorado State University are predicting an above-average probability of hurricane landfall: an estimated 18 named storms and four major (category 3–5) hurricanes. With the possibility of a bad hurricane season looming over the Atlantic and the thought of Hurricane Sandy still fresh on the mind of many on the East Coast, it’s only natural to look at emergency preparation and disaster recovery as an important aspect of information technology work.
Companies on the whipping end of the hurricane belt know the benefits of putting together a disaster plan. But that doesn’t mean that living in an area that is largely free of natural disasters should exempt a business from having a Plan B.
“There are all sorts of different disasters that can threaten a business, including fires, earthquakes, and just regular floods,” says Jack Camp, public affairs specialist with the U.S. Small Business Administration’s Office of Disaster Assistance. “Businesses need to assess their risk from all different types of disasters and consider steps that they might take to mitigate the effects of the disasters that are most likely to affect them.”
In the U.S. alone, Hurricane Sandy’s estimated damage was more than $50 billion. The vast majority of that loss was unavoidable, given the magnitude of Sandy’s destruction. But good preparation, even in the regions of the U.S. where disruptions are uncommon, is a critical facet of business, especially for information professionals. A little planning can mean the difference between surviving and going belly up for a business.
Examples of what works and what doesn’t are rife in the stories that appeared in the months since Sandy hit. PEER 1, a data center in New York City, saw its basement-level generators flooded as Sandy moved in, which forced the company to rely on backup generators that were tucked away on the 17th floor to keep the servers running. In order to feed the generator’s 40 gallons-an-hour habit, the team at PEER 1 had to get containers of fuel up the stairs at a constant pace for 2 days. The team members worked with the resources they had—a generator, a fuel supply, and a mixed group of personnel and volunteers—to create a bucket brigade that effectively managed the fuel flow.
CoreSite, another data center in New York, relied on fuel generators for nearly an entire week because of Hurricane Sandy. Finding all that fuel might have been a problem, but CoreSite had a plan in place. In an interview with Slashdot, Billie Haggard, senior vice president of CoreSite’s data centers, explained, “I’ll tell you, the best investment I ever made … was that we paid $9,000 at the beginning of the year to have a guaranteed fuel delivery within eight hours. So when everyone started scrambling, trying to find fuel, ours was already paid for.”
Hurricanes may not strike New York and New Jersey with the frequency and strength that they hit Florida, but the destruction caused by Sandy demonstrates the need for up-to-date contingency and recovery plans, even in areas where natural disasters seldom occur.
SunGard’s Mobile MetroCenter can support up to 50 workspace positions, with phones, computers, and office equipment.
Batten Down the Collections
It’s not just big business that needs to be ready for potential disasters. Libraries can equally benefit from data backup and recovery in a disaster, but they have plenty of challenges all their own. Here are a few tips from Miriam B. Kahn, founder of MBK Consulting and author of Disaster Response and Planning for Libraries, Third Edition (American Library Association, 2012), as well as from Julie Page, co-coordinator of the California Preservation Program (CPP) and the Western States and Territories Preservation Assistance Service (WESTPAS).
Close off the building (or affected areas) to the public and shut off utilities.
Following a disaster, in the absence of a checklist prioritizing the library’s collections in terms of value and relative susceptibility to water, consult a subject specialist or bibliographer to create a priority list for water-damaged materials.
If your library is shorthanded, consider contacting the local library/archives consortium, the state library, the regional library consortium, or the Regional Alliance for Preservation for help.
Contact facilities maintenance staff or a disaster response/drying company to remove any standing water in the building.
Books that have partial water exposure (only the bottom is wet from standing water, or the top or edges were affected by sprinklers) can be air-dried. Books can be distorted after they have dried and may need to be rebound.
If books are merely damp and the library has an enclosed area that isn’t compromised, staff can air-dry books. Decrease temperature in the building (less than 65 degrees Fahrenheit), keep the relative humidity low (30%–40%) with dehumidifiers, and spread the books out on clean, absorbent material. Stand the books upright and gently spread out the pages. Keep the air moving with fans but avoid direct exposure. This method is relatively inexpensive, but books may require interleaving (placing absorbent material between pages), flipping from head to tail, and monitoring so that the fans are well-placed.
If books are soaked through, freezing the volumes will buy time. Take a standard-sized box and line it with a large trash bag. Put in a single row of books, spine down, ensuring that the books are in snugly and don’t slouch. Tie up the trash bag, seal up the box, and put the box in a commercial freezer. The deterioration process is halted, and books can be sent to a vacuum freeze-dry facility.
If you can’t move undamaged items to another area, set up fans in the damaged area to lower the temperature, which decreases the chance of mold.
Offering 982 square feet of workspace per unit, SunGard’s Mobile MetroCenter can be set up for business within 1 hour after it arrives.
A 2009 study in disaster preparedness from disaster recovery company Agility Recovery Solutions found that while 94% of companies say they have formal data backup plans, only 28% have access to alternative office space, 54% could acquire temporary office equipment, and 57% have access to power generators. On top of that, 1 in 5 small and medium businesses (SMBs) do not maintain their continuity plan at all.
SMBs can find it difficult to prioritize disaster recovery. “In general small to medium-sized businesses are a little less prepared than larger organizations, and oftentimes it’s really due to a lack of budget,” says Susie Spencer, senior product marketing manager for Symantec. So a lesson in the basics of disaster planning and recovery would be useful.
For most businesses, the first line of defense against a disaster is insurance. But knowing the type of disasters likely to strike a given area and acquiring adequate coverage is only the first step.
After insurance is settled, the next issue is practical matters that relate to workers. Consider safety, “including an evacuation plan with routes and established meeting places, as well as designated staff members to do a headcount to account for all your employees,” says Camp.
Camp also stresses the importance of documentation. Someone should accurately inventory business assets before a disaster occurs, he says, and particular attention should be paid to purchase date, price, and estimated replacement cost. Beyond that, it’s a good idea to regularly photograph or create a comprehensive video inventory of your facility and the contents inside.
And then there’s IT. Perhaps one of the most important pieces of a company’s disaster strategy involves a well-measured plan to keep IT alive, or at the very least, to bring it back up to speed quickly, with minimal data loss.
Any number of actions can be taken to prepare an IT department for disaster, which largely depends on the type of equipment used in-house and the type of information professionals who work there. Both data backup and work application mirroring are important. Common solutions include cloud backup, virtual machines, and alternate work or server sites.
It’s not always easy to get the upper management to embrace the need for a well-constructed continuity plan. The same Agility Recovery disaster report shows that only about half of the respondents feel that their management team takes an effective business continuity plan as seriously as they do. But beneficial ideas can often come from those who work with the technology directly. “You’ve got to involve your employees in it; they’ve got to be aware of your plan,” says Camp.
Keep Me Updated
While a disaster plan may take into account all the aspects of your business, there’s more to it. There’s also the practical implementation.
Keep a well-stocked disaster survival kit on hand in a safe place that’s regularly checked. Common items to include are cash, a 3-day supply of nonperishable food and water, a can opener, first aid supplies, medications, radio, flashlight, batteries, blankets, a whistle, sanitation supplies, and a basic toolkit. And beyond that, you might want to consider getting some big ticket items or situational gear such as generators, fire detectors, and waterproof containers.
Add to this the continually changing nature of IT work—new hardware, software, or processes—and even for companies that put together a sound contingency plan years ago, an entire new plan might be in order. Training may be necessary, and third-party services may need to be checked periodically. “[T]he other key component of a plan is testing,” says Spencer. “And we really recommend to test on a quarterly basis to make sure that everything is recoverable when it needs to be and they can keep their business up and running.”
It’s also a good idea to make sure your plans are legible to outsiders, according to a document released by the National Institute of Standards and Technology, titled, “Contingency Planning Guide for Information Technology Systems,” compiled in 2002. “Plans should be formatted to provide quick and clear direction in the event those personnel unfamiliar with the plan or the systems are called on to perform recovery operations. … Where possible, checklists and step-by-step procedures should be used.”
While it’s feasible for larger, enterprise-level businesses to create backup and disaster services in-house, SMBs aren’t usually so lucky. Fortunately, there are a variety of outside resources to call on.
For those SMBs, it’s imperative to back up business data so that it can be reached in case of disaster, downtime, or data loss. Businesses should have a copy of the data on-site, but more importantly, they should have access to the same backup data off-site at a second location in case a local facility is compromised, says Spencer.
Symantec’s disaster recovery solution for SMBs is a product called Backup Exec. Businesses can restore virtual machines, applications, and databases in a single pass following data loss. Spencer says Backup Exec can help “even if a business needs to recover an individual file and folder, or if they do have a server failure or a natural disaster and they’d have to recover that complete system. …”
On the flip side, Symantec’s disaster recovery for enterprise-sized businesses is the NetBackup platform, which offers recovery for physical, virtual, arrays, and Big Data, with data-storage options ranging from tape to cloud.
Zetta.net, which describes itself as an enterprise cloud backup provider, is an option for companies that need data backup, first and foremost. Rather than copying data to magnetic tapes, Zetta gives customers a hybrid-cloud backup, which is composed of local backup, cloud backup, and archiving for use in disaster recovery. Westinghouse Lighting, a company with more than 30 servers, switched to Zetta for backup recovery less than a week before Sandy hit, a process that took 8 hours. The servers were backed up in Zetta’s West Coast data center while the hurricane ravaged parts of the East Coast.
TwinStrata’s CloudArray Disaster Recovery as a Service looks toward SMBs—those that can’t afford to have separate buildings and infrastructure up and running when disaster strikes. They give customers both the data they need as well as the VMware-based infrastructure necessary to call up their applications in 2–4 hours after local resources shut down.
For larger companies in need of a wide range of services, SunGard’s Availability Services division can provide the total package. SunGard offers tiered levels of service, which scale according to clients’ needs. These services range from backing up data with cloud computing to mobile-recovery hot sites, which can be launched to an area of your choice within 24 hours.
Three flavors of mobile recovery units are standard fare in the SunGard fleet, according to Ron LaPedis, workforce continuity strategist for SunGard Availability Services: the Mobile Data Center (a trailer that supports midrange and server platforms, seating up to 22), the Mobile MetroCenter (a workplace to support up to 50 workspace positions), and the Mobile MetroCenter 2 (a contiguous raised-floor series of trailers that provide a workspace between 768 and 3,070 square feet).
In a context where there might be no power and limited connectivity to the outside, SunGard Mobile units let workers focus on getting critical pieces back in place without worrying about local infrastructure going in and out; the Mobile Recovery units work as an island, each including diesel-generated power, high-speed internet, and mobile very small aperture terminals (VSATs).
Depending on the level of services a company contracts with SunGard, the mobile recovery units can be preprogrammed with a clone of the applications that workers use every day so that critical systems can be accessed without wasted setup time. “It’s your desktop, your call trees,” LaPedis says.
For the public sector, the U.S. Small Business Administration (SBA) has strategies and resources for disaster recovery. If a disaster is particularly ruinous, a state governor can call on the federal government for help. If that happens, businesses can directly apply for SBA disaster loans for help with payroll, building leases, and other expenses.
Disaster plans and recovery options aren’t just protection against once-in-a-lifetime natural disasters. Aspects of a well-constructed plan can apply to anything from the mundane (the power went out expectantly for a few hours) to the unlucky (an employee leaving a laptop with sensitive data on the train). The Federal Emergency Management Agency (FEMA) estimates that 40% of businesses never reopen after a disaster, and of those that do reopen, 25% fail within a year.
Chances are that you’re never going to account for everything that may be lost, but planning for the worst is the best chance a company has when facing a natural disaster.