Citybreak Online

Systems affected
Citybreak Online

Timeline
10:30 – Large increase in traffic
10:40 – Internal alerting and first external report of performance issues, investigation begins.
11:15 – Application issues ruled out, focus on allowing traffic through
11:20 – Session rate increased
11:30 – Performance returned to normal levels

Executive Summary
Several campaigns with wide reach on the same morning caused a large spike in traffic that ran into DDOS and bot protections.
Description of the issue
Several customer campaigns with wide reach landed on the same morning and caused a large spike in traffic.
The session smoothing of our load balancer (designed to help prevent DDOS) kicked in and real bookings were affected.
Because the actual number of bookings was also historically high causing an expected increase in resource use, identifying the root cause took longer while we checked for bottlenecks in our applications.
When we had ruled out application bottlenecks and confirmed the spike in traffic was not caused by DDOS or bots, increased capacity of the load balancer led to the situation normalizing.

Actions taken
The load balancer is set to be more relaxed in it’s session smoothing.