A fire last night at Seattle's Fisher Plaza data center has knocked out service to some top Web sites, including Bing Travel and Authorize.net.
We first got word of the fire early this morning from the online real estate service Redfin, which suffered an outage last night. Redfin's Web site was back up this morning, but we've noticed other sites that are experiencing problems. KOMO TV and radio broadcasts also were impacted, and we've noticed that Seattle's AllRecipes.com was offline too. Reports are circulating that Verizon's FIOS service, also is down, though it was unclear whether that was tied to the Fisher Plaza fire.
A blown transformer appears to be the culprit, according to a message posted on the Web site of AdHost.
"Beginning at approximately 11:18 PM on July 2nd and continuing through the present time Fisher Plaza experienced a significant power event that required all power systems including street power, UPS, and Generator power to be completely shut down in Plaza East.
The event is ongoing and, at this point, we do not have a ETA for service restoration. Please note that Adhost's Plaza West facility is not affected by this event. The Adhost phone system is not operational as a result of this event, but we are fully staffed and responding to emails sent to support@adhost.com .
Until further notice this will be the only conduit for communicating with Adhost. Please be patient with any requests sent into the Support email. We will process all requests as quickly as we can and we promise to answer every query.
If you have specific needs for assistance for powering up your devices, please send email to Support with specific instructions. We will do our best to accommodate your requirements. We apologize for the nature and extent of this event and are doing everything possible to restore service as quickly as possible."
Here's the message from Bing Travel:
"A fire occurred at Fisher Plaza in downtown Seattle just after midnight on Friday morning. The blown transformer knocked out power to the entire building, which is home to the Bing Travel servers. We're hard at work to restore service following this unexpected event."
There has also been a steady stream of discussion of the fire on Twitter.
And this isn't the first outage at the Fisher Plaza data center, with service going offline last year after an electrical fire. The Fisher Plaza Web site also is down, and calls to the facility returned a busy signal.
However, a cached version of the site explains the facility this way:
"Fisher Plaza is the only mission-critical business community in the Northwest combining Class A office, data center, colocation and retail space with 21st century communications and media services."
We'll be tracking the situation throughout the day, so let us know if there are updates to share.
UPDATE, 10:30 a.m.: TechFlash reporter Todd Bishop has arrived on the scene, where he reports that data center tenants are carrying server boxes out of the building. We're asking tenants when the service may be restored, with some saying it could be early afternoon.
Reports from the scene are spotty. Some officials are telling tenants it may be the end of the day before service is restored.
Officials with the property management firm for Fisher Plaza declined to comment on the scene.
UPDATE, 11 a.m.: Bing Travel now is saying that it expects the travel service to be available again by 5 p.m. Pacific time today.
Does anyone have any technical information on why companies can't switch to another co-location facility in order to keep Web sites running?
UPDATE, 11:20 a.m.: Verizon Wireless has generators on the scene to power its equipment.
Update, 11:27 a.m.: Here's the alert sent out by Internap Network Services a short time ago to its data center tenants at Fisher Plaza.
At this time, there has been no significant change in status regarding the power outage at the SEF data center. Work is continuing on site to restore power as soon as possible.
The electrical vendor is onsite continuing to work with the electical switch gear at the SEF (Fisher) datacenter.
Customers in the SEF PNAP with IP connectivity continue to see their connectivity restored and stable; however, there may be further periods of degraded IP connectivity as the situation continues to develop.
Customers colocated in the SEF PNAP continue to see issues with a loss of power in some areas.
The focus of on-site personnel continues to be getting the rest of the generators online; however, we currently do not have an estimate as to when full generator capacity will be feeding the SEF facility.
We will continue to provide updates as they become available.
Update, 12:05 p.m.: Jon Davies, a spokesman for Verizon, tells TechFlash that the company's FIOS and DSL service went down for about an hour at 9:43 a.m. today after batteries in the co-location facility ran out of juice. About 50,000 customers in Oregon and Washington lost Internet connectivity, said Davies.
Service was restored at about 10:45 a.m. through replacement batteries and the company is currently transporting a large generator to the building in order to make sure that Internet service continues to operate today.
Davies -- who called the Fisher Plaza building a large "switching center" for the Northwest -- said he had not heard when full power might be restored.
Todd Bishop photo of massive generator arriving at Fisher Plaza.
Update, 2:05 p.m.: Blogger Kyle Mulka has started a list of Web sites that have experienced outages due to the fire, including Bartell Drugs and Geocaching.com.
A few that he is missing are large Seattle Internet sites such as AllRecipes.com -- which we discovered was offline early this morning as we atttempted to find a BBQ chicken recipe -- and Big Fish Games -- a Seattle company that operates a large downloadable game business. Here's the message from Big Fish:
Update, 2:25 p.m.: Here's the latest from hosting and colocation provider Internap which has a number of clients in the Fisher Plaza Building.
Two temporary generators are onsite at this time. This repair is on schedule to be completed by late this evening, but we do not have a specific time yet. We will continue to provide updates as we get further information about the implementation of this plan.
Update, 4:45 p.m.: Here's our latest report: "How one CTO avoided a Web site disaster after data center fire." We talked to Redfin CTO Michael Young to find out how the online real estate company got their site running shortly after the fire, whereas other companies are still down.
Update, 6 p.m.: TV station forced to go old school after fire at Seattle's Fisher Plaza
Update, 7 p.m: Latest on data center 'incident'
READ MORE and COMMENT
No comments:
Post a Comment