Incident report - iTicket


Incident report

Systems affected

10:05 – Received reports regarding missing availability for some products.
10:32 – Developers start investigation
10:40 – More reports of missing products, price lists and calendar availability affecting
some customers after synchronizing channels to Citybreak from iTicket.
12:30 – Problem located to database logic used when saving/approving new channel
content in combination with new caching technology.
13:05 – Developers start to work on a solution and a way to automatically find and repair
affected channels
17:44 – New solution ready from developers. Testing and repair processes started.
19:48 – Deployment and fully mitigation completed all issues confirmed to be fully resolved

Executive Summary
A new release of iTicket contained new cache technology to reduce latency. This newly
introduced technology caused a problem when making changes to the content of a
distribution channel.

Description of the issue
A new caching scheme was added to reduce latency and provide protection against high
loads. This made an existing code issue surface. After changing a channel and approving
changes, its content was inconsistent for a brief time. With the new cache this inconsistent
view of the channel could be picked up and stored in the cache which made a temporary
issue become persistent.
Since the content of the channel could missing parts of the products and price lists it could
cause a range of problems such as missing availability calendar information, removal of
agreements in Citybreak and cause archiving of products in CBIS, if channels were

Actions taken
We ended up redoing and improving the quality of the code used to store the channel
content and built code to update all channels for them to get new correct content.

Christian Holmén
ITicket Product Manager

Visit Group |  Kungsgatan 34-36 |  411 19 Göteborg |  Sweden |  +46 (0)31 38 06 000