Globat hosting suffered a major downtime on February 21st 2008. The primary network switch and backup switch completely failed and took down the entire network including globat website. This site was also down for about 12 hours! From the mail I have received,
Our primary network switch in our main datacenter in downtown Los Angeles failed completely and despite tremendous efforts by our technical emergency response team could not be brought back online. This switch is a major brand name piece of equipment and as such it contains internal redundancy. We were able to switch over to the back up circuitry and begin bringing the system back online at which time, for yet unknown reasons, the entire devise, including backup system, failed completely.
After a short deliberation we decided to rush a new switch from our vendor’s nearest warehouse to our data center. While waiting for the replacement we initiated a work around to at least get email service re-established. After several attempts all email systems finally went live and back to normal at approximately 10:00am Pacific Time.
When the new switch arrived we went through the installation and testing process and as a result were able to bring all our services back online, but not before we had suffered a 5-hour outage for e-mail services and an 8-hour outage for Web related services. In Globat’s entire 6 year history, we have never experienced anything even remotely like this! The outage was so severe
that it affected our own Web sites and all but one of our telephone lines so we were unable to communicate with our customers to update anybody about the status of the incident. We decided to ask some of our staff members to post updates on well known Internet forums, which helped a little in disseminating the information.