Apologies for the downtime

[Gob]

Power disruption affecting US-West Data Center (Irvine, CA)
Posted (March 19th, 2013 at 3:01 pm PST) by andrea

We are currently experiencing power disruption in our US-West Data Center (Irvine, CA). Our system engineering and data center operations teams are looking into the cause. We will update this post as soon as we have more information. We apologize for the inconvenience this is causing.

Update March 20th 12:19am: Services are still slowly buy surely coming online. We’ve had one rack of machines giving us a particularly tough time, and are working on that now. We should have more fixed in the next few moments.

Update March 19th 11:41pm: Services are slowly coming up, and addressing each one as we get the next fixed. Also, our data center has extended the UPS maintenance window to 3am PDT.

Update March 19th 10:56pm: Services are still coming up, however, some hosts require manual intervention. We’re seeing some sporadic outages as well, and are working to get those fixed ASAP!

Update March 19th 9:35pm: Ok, we see a lot more sites and servers coming online, so this is good news! We’re still monitoring things, to ensure they come back online smoothly.

Update March 19th 9:15pm: We’re still working away on this, and are now seeing some services restored. You may see some intermittent outages for a bit while we work out the kinks, but services are coming back online slowly.

Update March 19th 8:30pm: We’re having some success resurrecting part of the problem areas of our network. This does not mean hosts are up just yet, but it does mean we’re getting closer.

Update March 19th 7:55pm We are still getting the new network device in place and configured. We’re doing everything we can to work as fast as possible and restore service ASAP.

Update March 19th 7:19pm: Our data center provider is also initiating emergency maintenance on the UPS system that caused the initial power outage. This maintenance is scheduled to be complete at 11:55pm pacific time. We don’t anticipate problems to stem from this, but are bracing for issues, should they pop up.

Update March 19th 6:30pm: As of now we have two network devices, that handle traffic internal to the data center, that are down. One main one, and one backup. Our vendor has said, that BOTH devices were fried during the power outage, and are suggesting we RMA BOTH of them. We are in the process of deploying a spare to that location and we estimate that will take 1-2 hours. We will continue to update as we get more information.

Update March 19th 5:55pm: We are still working on this problem, however it’s proving fairly difficult to resolve. We do have power in our datacenter and most of it is up, however the lingering issue is a routing table we’ve been unable to restore. We’ll continue to update here.

Update March 19th 5:34pm: At this time, we’re still working on the problem, and are going through the many troubleshooting steps needed, however these steps take a bit of time to complete. We’re working as fast as possible to get things sorted out ASAP.

Update March 19th 5:10pm: While we’re continuing to work on bringing all the services back online, we’re happy to report that about 90% of DreamObjects are now available again, and we confirmed that there is no data loss. We appreciate your continued patience while we are working on the rest.

Update March 19th 4:58pm: Despite the diligent work from our technical teams, we are still not fully back up with our services in this location. We are very sorry, and will update this post again with any news we may receive.

Update March 19th 4:28pm: I’m afraid we are still not fully back up at the moment but continue working on it. We are very sorry for the inconvenience this delay is causing and appreciate your patience.

Update March 19th 4:08pm: We are slowly but surely bringing all services back online, including DreamObjects. We are very sorry for the inconvenience this delay is causing.

Update March 19th 3:47pm: We are still working on bringing back all affected services. You can check if you are affected by this power disruption by looking up where your services are hosted in the DreamHost Web Panel. Please note, DreamObjects are also located in this data center. Again, we apologize for the inconvenience.

Update March 19th 3:27pm: After the power came back up, we were able to bring back part of our services. We are slowly but surely bringing all of them back up, so your sites may take a little while longer to start working again. We appreciate your patience during this process.

Update March 19th 3:07pm: We now have power back in the data center. We are investigating the cause, as it affected the whole facility and not just us. We are also working on making sure that all services are coming back online as expected. Your sites may take a little while longer to come back up.

You can check if you are affected by this power disruption by looking up where your services are hosted in the DreamHost Web Panel. Please note, DreamObjects are also located in this data center. Again, we apologize for the inconvenience.

[Daisy]

Was just off again about an hour 11:30 GMT back on at 12:25 GMT.

[oldr_n_wsr]

Had trouble this morning, but now it seems ok.
Must be that "green" technology they use down under.

[Gob]

ha, except we're hosted in Dreamhosts US-West Data Center (Irvine, CA)

[oldr_n_wsr]

So it's Obama and his green peoples fault.