At 12:10 PM yesterday, we began to see evidence that a number of our users were unable to get online. The technology team was able to quickly identify the source of the problem with our servers in Dallas, TX. Unfortunately the solution for the problem took much longer to develop and deploy than usual. We did not have the problem resolved until 4 PM.
This service outage affected some of our Mac users, iOS users, and a small number of Windows users who were being served from our Dallas datacenter.
On behalf of Covenant Eyes, I want to issue a sincere apology to everyone this affected. We understand that many of you were unable to get online and lost productivity at work or at school. Please know that we take the stability of our network, and thus any outages, very seriously. Outages like this are rare, but we recognize that they can be potentially catastrophic for our members, regardless of how infrequent. To that end, here is a brief explanation of what happened and what we intend to do to prevent it in the future.
What happened:
- Our datacenter in Dallas, TX suffered severe performance issues, which led to many of the network issues our customers reported. This appears to have started around 12:10 PM EST on Monday, April 7. We resolved these issues around 4:00 PM on Monday.
- The resulting increased call volume to customer service exceeded our capacity for incoming calls, causing some callers to hear an “all lines are busy” message.
- This period of high call volume also caused us to lose the ability to make calls out in order to return call customer calls.
What we’re going to do about it:
- We’re developing an improved way of scaling our services as a result of information gained during this recovery process in our Dallas datacenter. When we’re confident in it, we plan to roll it out to our other datacenters.
- We already had plans in place to add more capacity via a new datacenter in Atlanta, GA. We will accelerate deployment of this datacenter as well as increasing the capacity of Dallas to accommodate the growing needs of our users.
- We are increasing capacity of our phone system to eliminate the “all lines are busy” message.
- We are also adding a dedicated set of phone lines to make sure this level of call volume does not prevent our ability to make outgoing calls from our call center.
Thank you for your patience during the outage. Encouraging accountability and trust is important to us. After all, it’s in our mission statement. That accountability and trust applies to us as well as our users.
If you have concerns or thoughts about outages—or the inability to get online in general—we welcome your comments in our online forum.
The post Outage on Monday appeared first on Covenant Eyes.
Source: Covenant Eyes
