Plan B Forum

Connectivity issues with one of our datacenters
Posted (March 23rd, 2012 at 11:49 am PST) by Brandon

We are currently experiencing some connectivity issues to one of our datacenters(more specifically, there is a small subset of our network which is unresponsive, fortunately the majority of our network and customer base is not impacted). Our networking, system engineers and datacenter operations teams have been deployed and are investigating. Sorry about the inconvenience this is causing. Please check back here often, as we will be posting updates as we receive them.

This issue is still only affecting a small subset of our customers, but will be affecting mail, webmail and MySQL. Connectivity issues are specific to our internal servers being able to communicate with each other, and this doesn’t indicate an inability for the servers to communicate with machines outside of our own network. As of 12:50 pm we are currently in the process of re-routing traffic around the affected networks in order to restore communication. We don’t have a concrete time to completion just yet, but we are not anticipating much longer before there is noticeable improvement. We will continue to update this post with new information as it becomes available.

Update Mar 23rd, 4:26pm PST: A good portion of network services are back online. E-mail is now being delivered, but again there will be a delay of 4-6 hours for all e-mail affected by this issue. We’re still working on the odds and ends, and will provide further information here when it becomes available. -Brandon

Update Mar 23rd, 3:50pm PST: Services are continuing to come back up. All e-mail affected by this issue will be sent, however may be delayed for 4-6 hours. We’re still monitoring the situation and will provide more information when it becomes available. -Brandon



Update Mar 23rd, 3:30pm PST: Services are coming back online at this time. We’re continuing to monitor the situation, and we’ll update you again as we get closer to considering this issue resolved. -Brandon



Update Mar 23rd, 3:10pm PST: We’re still working on restoring services as quickly as possible. Unfortunately we’re still experiencing network instability for which we apologize. We are aware that this is causing panel service interruption in addition to impacting other customer services, including web sites and e-mail. Our networking, system engineering and datacenter operations teams are working feverishly to resolve these issues. -Brandon



Update Mar 23rd, 2:20pm PST: The uplink which was causing the initial problem came back online; however, when we attempted to integrate it again we experienced network instability which necessitated a reboot of our core routers. That is still in process after which we will be working on stabilizing the network and ensuring everything comes back up and is functioning normally again. -Brandon



Update Mar 23rd, 2:00pm PST: Upon further investigation we’ve found that one of our uplink providers is experiencing an outage that’s affecting these networks. We have redundant routes setup to prevent this type of effect when a provider issue such as this one occurs, however the routing tables in question did not self correct, and we’re now in the process of correct that, which should take care of the issue. -Brandon



Update Mar 23rd, 1:40pm PST: We’re still working on resuming communications on these internal networks by routing traffic around the problematic machines. This is taking a bit longer than previously anticipated, but we’re still working diligently toward a resolution. -Brandon





Update Mar 23rd, 1:18pm PST: Traffic is now being routed around the problematic section of our internal network, and things are beginning to improve. We’re still working diligently toward a full resolution, and will update this post when we have new information. -Brandon





Update Mar 23rd, 12:50pm PST: We are currently in the process of re-routing traffic around the affected networks in order to restore communication. We don’t have a concrete time to completion just yet, but we are not anticipating much longer before there is noticeable improvement. We will provide additional details as soon as they are available. -Brandon





Update Mar 23rd, 12:40pm PST: This issue is still only affecting a small subset of our customers, but will be affecting mail, webmail and MySQL. Connectivity issues are specific to our internal servers being able to communicate with each other, and this doesn’t indicate an inability for the servers to communicate with machines outside of our own network. Our networking, system engineering and datacenter operations teams are still hard at work, and we will update this post with further information as it becomes available. -Brandon



Update Mar 23rd, 12:20pm PST: This should only be affecting a small subset of our customers, but will be affecting mail, webmail and MySQL. Connectivity issues are specific to our internal servers being able to communicate with each other, and this doesn’t indicate an inability for the servers to communicate with machines outside of our own network. We’re still investigating the issue, and will update this post with further information as it becomes available. -Brandon


Update Mar 23rd, 12:06pm PST: This should only be affecting a small subset of our customers, but will be affecting mail, webmail and MySQL. We’re still investigating the issue, and will update this post with further information as it becomes available. -Brandon

http://www.dreamhoststatus.com/2012/03/ ... /#comments

Now how many of you thought this was due to em upgrading the software?

Lord Jim wrote:

Now how many of you thought this was due to em upgrading the software?

Should we run a poll on that?

We are excited to announce that we are at the end of our Network Improvement Series! This series of improvements started back in May is finally nearing completion. We started this project when we realized our network stability was in danger as we had ourgrown our existing hardware. We have invested heavily and painstakingly planned to make sure we can provide you with the most stable network possible. On Wednesday, February 6th we will be completely switching over to our new network infrastructure in the final data center US-West (Los Angeles, CA). This maintenance will start at approximately 8pm and the window is 4 hours. During this maintenance we will be moving public traffic to our newly installed network, putting all live traffic on our shiny new network hardware. Much like our US-West final improvement series maintenance in Irvine, we will be moving traffic in small batches which will minimize impact and allow us to move at a pace that will limit service interruption. Services impacted during the maintenance will include Web, E-Mail, MySQL and other services located in the US-West (Los Angeles) Data Center. You can check if you are affected by this maintenance by seeing where your services are hosted in the DreamHost Web Panel.

Apparently we had some downtime today, may have been due to this, or us being on the same circuit breaker as the superbowel;

Crackpot wrote:While we had som downtime this AM I got to thinking When Gar was updating the board it was referred to drinking rum and playing with crayons.

What should we call Gobs misadventures? Grand ideas and gross over-estmation of ones own skills? Nah too literal we need something more poetic.

Suggestions?