Citybreak Online

Systems affected
Citybreak Online

Timeline

10:30 – Massive increase in traffic
10:40 – Internal alerting and first external report of performance issues
10:45 – Internal investigation begins – very customer large campaigns reported by CST to Devs, ruling out issues with automated traffic
11:15 – Application issues ruled out, focus on session rate of loadbalancer
11:20 – Session rate increased
11:30 – Performance returned to normal levels

Executive Summary

Several customer campaigns with wide reach landed on the same morning and caused a large spike in traffic that ran into DDOS and bot protections that were not made ready for the influx.

Description of the issue

Several customer campaigns with wide reach landed on the same morning and caused a large spike in traffic. The campaigns were directed at booking widgets and pages rather than landing pages, increasing the amount of non-booking traffic to booking infra.
The session smoothing of our load balancer (designed to help prevent DDOS) was set too low for the number of active sessions and real bookings were affected.
Because the actual number of bookings was also historically high causing an expected increase in resource use, identifying the root cause took longer while we checked for bottlenecks in our applications.
Eventually both the changes to the load balancer and a slowdown in clickthroughs from the campaigns led to the situation normalizing.

Actions taken

The load balancer is set to be more relaxed in it’s session smoothing.
Processes around internal communication about expected upcoming spikes will be reviewed to make sure tech resources are on hand and aware in expected high traffic situations.