Platform Performance Degradation
- Updated: 11:01am, 22 Sep 2021
- Experience: Slow history page loading and 503 errors
- Current Status: Resolved
09:31 A significant number of 503 pages are being received by users. Investigations commencing.
09:42 Cause identified as slow running history queries for provision histories saturating the reporting database.
09:51 Temporary remediation by narrowing the frontend load balancer flow to reduce the number of queries running at once and disabling some functionality, improving response times and reducing queue times. Fuller remediation being investigated
10:14 Root cause identified: the table query optimisation within the database index using a bad plan choice by the query optimiser. Forcing it to use one index helps in one case, but then makes another usage worse. Traditionally, our overnight re-analysing of statistics would solve the problem but in this case, a threshold for stats was reached that wasn’t solved.
10:17 Query fix identified and being tested
10:26 Query fix verified and deployed
10:31 Normal database performance resumed. Removal of temporary remediation planned over next half hour.
10:58 Performance and functionality restored in full.