Visit Group - Incident report

Report date: Friday 11 July 2025
Incident occurrence: Tuesday 8 July 2025
Incident duration: 3 hours and 53 minutes
Status at report date: Resolved

Systems affected
Citybreak, WebX

Customer/end-user impact
Some customers affected, very slow response times when booking. Problem does not appear to be evenly distributed

Timeline
2025-07-08, 9:43 AM: Customers reported online booking not working effectively- Widget not loading.
2025-07-08, 09:45 AM: Engineering team alerted by internal metrics and investigation underway.
2025-07-08, 10:17 PM: Worst congestion as measured internally is over
2025-07-08, 11:20 PM: Status update Monitoring - Response times improved as reported by highest traffic customers
2025-07-08, 13:36 PM: Marked as resolved

Root Cause Analysis
Initial analysis indicated that the response time degradation was not strictly correlated with traditional peak usage but was instead exacerbated by irregular network patterns. A significant proportion of the problematic traffic was traced to automated (bot) sources, some of which were benign but misconfigured, while others appeared to be deliberately generating excess load.

Investigation/Resolution Actions
Deep investigation into traffic across our subnets.
Further limiting of automated traffic that has been an increasing factor in load

Preventive Measures & Next Steps
Added metrics on firewall and out of datacenter response time
Adding capacity to our infrastructure

Conclusion
Whilst we have planned for summer traffic and are generally well placed to handle most loads, some internal event seems to have triggered congestion in our network for the time period in question.
While internal metrics do not show a huge spike in response time issues, cumulative slowdown and uneven response time quickly lead to a degradation in the overall booking experience. We have had increasing traffic from misconfigured and malicious automated traffic and we have now implemented stricter rules to defend against these.
We have also increased the visibility of internal traffic volumes to quicker identify bottlenecks and inform us as we continue our existing plans to expand our infrastructure capabilities.