Dear Customers,

Following the service disruption in our platforms last week, here is the detailed incident report.

Incident report
2022-03-31

Systems affected
All

Timeline
13:37 - Internal alerting
13:40 - Operations team investigate
13:42 - Issue identified
13:52 - Issue fix, monitoring
13:53 - Issue resolved

Executive Summary
Our database cluster went down unexpectedly. There was a human error while working on improving network resiliency in the data center. Unfortunately, due to this error, our SQL server node became inaccessible by the application layer, which caused downtime across our systems.

Description of the issue
While trying to add more paths in our network, our network partner forgot to enable port-channel which caused communication interruption between our SQL cluster and network switches. The problem was discovered almost immediately but it took some time for our systems to become fully functional again.

Actions taken
Our plan is to move every network work, even low risk improvement tasks in our data center to a planned night maintenance window which will be communicated to all partners.

Sincerely,
Boris
Operations Lead, Visit Group