FCA publishes multi-firm review on implementing technology change

Introduction

Operational resilience and the relationship of financial services firms ("FS Firms") with third party service providers  (mainly technology companies) continue to attract regulators' attention. Late last year, the EU Commission published its proposed regulation intended to improve FS Firms' operational resilience and response to cyberattacks (please see our summary here). The FSB has also been consulting on regulatory and supervisory issues relating to outsourcing and third-party relationships (please see our summary here). In late 2019, The Bank of England, Financial Conduct Authority ("FCA") and Prudential Regulation Authority published a policy summary and coordinated Consultation Papers on operational resilience in financial services (please see our summary here).

In this context, the FCA has now published a multi-firm review that looks at how firms implement technology change, the challenges caused when changes fail, and steps firms can take to protect consumers from harm and disruption in the market. The review is intended to contribute to the discussion on operational resilience and help firms implement technology change in a way that will reduce operational disruption.

The Review

The FCA conducted an analysis of over 1 million changes made by FS Firms in 2019. The FCA also reviewed 20,000 incidents resulting from change over the same period. This data analysis was then supplemented through questionnaires and workshops with individuals at various FS Firms. The FCA's review is split into five parts:

1. contributing practices to change success and change failure;

2. the impact of incidents caused by technology change;

3. how FS Firms govern and manage technology change;

4. how FS Firms build and deploy technology change; and

5. how infrastructure impacts technology change.

Contributing practices to change success and change failure

The FCA found multiple similarities and trends for FS Firms that had a higher percentage of successful changes:

1. "Firms with well-established governance arrangements have a higher change success rate;

2. Relying on high levels of legacy technology is linked to more failed and emergency changes;

3. Firms that allocated a higher proportion of their technology budget to change experienced fewer change related incidents;

4. Frequent releases and agile delivery can help firms to reduce the likelihood and impact of change related incidents; and

5. Effective risk management is an important component of effective change management capabilities."

At the other end of the spectrum, the FCA found the following practices contributed to change failures:

1. "Most firms do not have complete visibility of third-party changes;

2. Firms' change management processes are heavily reliant on manual review and actions;

3. Legacy technology impacts firms' ability to implement new technologies and innovative approaches; and

4. Major changes were twice as likely to result in an incident when compared with standard changes"

These trends highlight the importance of FS Firms investing in their technology and ensuring that they have proper governance in place around change. This would allow them to retire legacy technology that may be out of date, implement more frequent updates and ultimately minimise disruption. These types of failures can be costly to FS Firms and as regulation increases it will become essential for FS Firms to manage this risk appropriately. Equally important is the necessity for strong governance and risk management.

The impact of change incidents

As a general rule, technology changes were successful. In fact, in 98.4% of cases there was no incident. However, the sheer number of changes implemented by the sample FS Firms meant this equated to over 13,767 incidents with 14% of those having customer-facing impacts. Change incidents were found to be the root cause of 1 in 4 high severity customer facing incidents:

"Our data showed that of all customer-facing incidents in 2019, 7.4% had a root cause relating to change activity. However, of all incidents classified as high severity by firms in 2019, 24% had a root cause relating to change activity, highlighting that failed technology changes can have a higher impact than incidents resulting from other root causes."

One interesting finding was that FS Firms' Change Advisory Boards ("CAB") had approved over 90% of the major changes they reviewed. The FCA hypothesizes that this may indicate that CABs perform more of a 'flight control' role rather than an assurance function.

While "major changes" were more likely to result in a failure compared to normal changes (twice as likely in fact), "emergency changes" were actually less likely to result in failure. The FCA claims the increase in failure for "major changes" is due to their inherent complexity. The FCA is unclear clear on why emergency changes are less likely to result in failure. However, it speculates that it may stem from an incident already being open or strong risk awareness for emergencies.

Governance and management

The FCA found a link between the length of time governance arrangements were in place and change success rates in the FS Firms:

"Firms that had governance arrangements in place for more than a year experienced a lower proportion of incidents resulting from change when compared to peers with newer arrangements."

Build and development

The FCA also looked at utilisation of a DevOps approach. A DevOps approach is where software development and IT operations are combined to shorten the development life cycle and provide continuous updates. However, the FCA notes that this isn't a one-size fits all solution and it comes with regulatory and operational issues. Currently 84% of FS Firms use DevOps methodologies in some form, but only 13% of FS Firms use DevOps processes for all software delivery activities.

Interestingly, the FCA also found that those FS Firms that embraced "agile" project management methodologies had fewer incidents resulting from change. Despite this, only 43% of respondents utilised these agile methodologies.

The FCA found that FS Firms were utilising a wide range of testing methods. All firms utilised regression testing, integration testing and unit testing. More than 90% of FS Firms were using sanity testing, smoke testing, interface testing and system testing. However, despite the wide range of testing methods used, the FCA found that many of these relied on manual testing and peer review. The FCA comments that these are still prone to human error. Automation allows for:

"Repeatability and consistency of processes throughout the lifecycle of a change and its deployment could reduce the testing burden, and allow for increased coverage of assurance activity."

Most FS Firms did use some automation, but challenges include the need to maintain control, and make use of deployment-specific tests. The FCA embraces automation as a way to decrease the time it takes to implement change and to reduce the possibility of human error:

"Deploying change regularly and safely usually requires firms to use automated tools and processes and to leverage modern infrastructure. Firms that deployed changes between once a week and once a day/multiple times a day made use of automated deployment tools, with 78% utilising a deployment process that was mostly or fully automated... Overall, we found that firms who used microservices, automation and deployed changes more frequently had higher change success rates."

The impact of infrastructure

Despite the FCA finding that reliance on high levels of legacy technology was linked to more failed and emergency changes, only 8% of respondents did not rely on at least some legacy technology. One third of all respondents mostly relied on legacy infrastructure and in some cases, this was as high as 70%.

FS Firms are embracing cloud infrastructure more and more. However, this presents some issues including a lack of oversight and direct control. Despite the regulatory focus on this type of outsourcing, the FCA still found that 78% of production applications are were hosted in on-premise environments, but respondents expected this number to fall.

The research demonstrates important ways that FS Firms can manage changes and mitigate disruption. Under the EBA guidelines on outsourcing arrangements, FS Firms should, at least for outsourced critical or important functions, receive an annual risk assessment and performance monitoring report from those third parties it has outsourced to. FS Firms should read the full review and consider how its findings may affect their firm and their relationship with technology companies. In particular the key trends identified above and by the FCA should be considered, when reviewing existing arrangements with service providers, considering annual monitoring reports, and procuring and implementing new technology. This research can be read in full here.