We experienced a significant load increase when Freetrade launched on the 25th of April 2019. What then happened was an unfortunate combination of events that are testament to Freetrade for warming up their large community to arrive on our site at the same time, what that caused was:
- The system was working 4x harder than any previous peak effort, since the start of 2018.
- The system also had 59 of the 64 highest number of investments per second we have ever had, previous peak loads haven’t handled the investment flow this gracefully, which is what exposed a new issue.
- Finally, and this is what caused the lack of graceful failure, we hit a load failure point in the investment flow for people who need to take our risk assessment questionnaire, as first-time investors, this was hit while taking 8x the normal number of investments per second.
These 8x (investment) and 4x (logging in) events alone would have been one thing, but the inefficient code in the assessment flow running at a time of load had a knock on impact which is more like the processor utilisation going from 5% utilised to 100% (20x). This then starved everything out, a bit like trying to breathe after being dunked by a wave in the ocean, whilst being hit by another wave. As a result, the Crowdcube platform experienced issues for 50 minutes, which was followed by a period of monitoring the recovery.
The root cause of the issue was that a piece of code, which has never seen this load, was querying, inserting, and updating a database table using an unindexed field. To the technical community reading this post, that may sound very obvious, and indeed it was - once we saw the issue. Unfortunately, this piece of code had never been exposed to that load profile before, nor been spotted when tuning the system response time, which we have been working on recently.
How we fixed it
Once we found the root cause of the particular issue, we fixed it. Since then we have been load testing the system, with this updated consideration, which is an additional step on top of creating accounts, finding pitches, and investing, under load.
The graph in figure 1, shows the processor load on the system during a performance test run of the fix. The same 6 progressively harder load tests result in 100% processor utilisation in the first 6 runs (without the fix), and a peak of 9% processor utilisation after the fix. For the same number of users.
We have now completed the first round of load testing, post our fix, up to 7x the level at which we had an issue. This gives us headroom, relative to the biggest load we have ever seen. We are also not being complacent, and recognise that other parts of the system will need ongoing tuning.
This particular issue put a concentrated load on a small part of the system, it was the concentration of load, rather than general activity, which caused the problem. It was a busy time on the site, however, session activity was spread throughout the day, which isn’t a load challenge.
Once again, we’re really sorry for anyone who was affected by these issues. Please be assured that we are continuing to work on the necessary improvements to ensure the platform can deal with what was unprecedented levels of demand in Freetrade’s pitch. We understand the disappointment of those that were unable to invest in Freetrade last week. However, we’re looking forward to welcoming them back to Crowdcube in June for a bigger funding round, with an increased EIS allocation.