Yesterday's Service Disruption

Method_Greg — Fri, 26 Sep 2014 18:40:57 GMT

Dear Customers,

I am very sorry about yesterday’s service disruption. I know that being able to access your customer information is a huge component of being able to run your business effectively.

While we did our best to let you know and to resolve the problem in the quickest way possible, I know we can do better.

What caused the problem?

We spent much of the day working with Rackspace, our hosting partner, to diagnose the issue. We found there was a significant issue in the aggregate switch hardware, which essentially controls the flow of all information between many of the servers that run Method. Once the faulty hardware was identified, Rackspace “switched out the switch”, and we were back to normal.

Both finding this out and fixing this took much longer than we would like.

The actions we took to get you up and running

As soon as we found out service was slow, Method’s developers and I went to work to figure out what went wrong and how we could get around the problem. From a technical standpoint, we:

Moved you to our backup environment to get as many of you up and running as we could, while we continued to diagnose the root cause.
In parallel, began working on a “Plan B”, which was to set up a new home for the affected servers, in case the root cause or solution couldn’t be found.

While all that was going on, our Support team and Account Managers were on the phones, forums, and Twitter letting you know what was going on.

If a similar situation arises in the future, how will service disruptions be managed?

Our intention is to be up and available 100% of the time but as we experienced yesterday, our backup environment was insufficient in seamlessly providing full functionality.

To make sure this does not happen again, our first priority will be ensure that our backup environment has complete functionality with performance levels comparable to the regular environment. Our next priority will be to ensure that the move to the backup environment is as close to instantaneous as possible.

We have also learned that our communication to you needs improvement. Emails were delayed and there were gaps of time between updates. Our goal is to keep you immediately informed of service disruptions and kept in-the-know as we progress towards a quick solution.

With that said, I strongly encourage you to follow our Twitter account, @MethodCRM, and our Forums as those are the first places we will put any service disruption notices and updates.

Again, I am very sorry about what happened yesterday. As tough as it was for us to deal with the issues, we know it was even tougher for you. Thank you for your patience.

Paul

Founder and CEO

Method:Classic General Q&A

Yesterday's Service Disruption