Risky Endeavours

Migrations come in many forms and, often, have data cleansing and data enrichment, consolidation or alteration mixed in with them. Cases include:

• Replacing legacy systems with a new, completely different, system, moving the data across to the new one.
• Moving from an on premise installation of a system to a cloud SaaS version of the same application.
• Updating a system to a newer release, one where the data model has moved on.
• Merging data from retiring systems into a live data set on a retained system as part of an IT estate consolidation.

These exercises are often seen as an ‘opportunity’ to undertake exercises such as:

• Deduplication of a record sets.
• Adding additional information.
• Standardising formats.

They are always complex exercises and there is a lot of opportunity to create defects in the data, one that show up immediately and ones that sit in the background, waiting to trip up a process at some future date.

Well known cases

Migrations go wrong, even apparently simple ones. Their execution can go wrong, tripped up by oversights or unexpected patterns in the data. Even where data has been ‘correctly’ dealt with the result can be issues. Other systems that consume the data can react badly to changes, the overall experience of the people and organisations the data relates to or affects can be badly damaged. Recent times have seen:

• A fixed line telecoms business that had to spend years cleaning up issues following the rapid creation of a new CRM and billing system driven by a regulatory deadline.
• A retail bank whose customers could not access their money and whose details were exposed to other customers following a migration triggered by the separation of the business from its parent bank.
• A mobile telecoms business whose consolidation of a myriad of separate billing systems onto a new strategic solution resulted in regulatory action being taken.

Inappropriate and inadequate approaches to the testing and to the evaluation of the consequences of data migration allowed these things to happen.

Specific Threats

Some key factors that cause problems for migrations included:

• The complexity of real world aged data
• The presence of data from different ages of the source systems
• Unexpected features of the data caused by bugs, some of which may now be fixed and forgotten, and by direct manipulation of the data.
• The difficulties of identifying what data relates to the same, real-world, entities.
• A tendency to ‘leave the detail’ to developers.
• The complexity of in-flight activities that can move on in the real-world whilst a migration is in-progress.

Specific Countermeasures

A disciplined, autonomous, programme of assurance is a key measure that must be taken to reduce the risk of a migration. This is not a typical test activity. It must be a holistic assurance activity looking at all aspects of the migration from all perspectives and, where necessary, going down to the bits and bytes level representation of data. Some key success factors include:

• A highly autonomous assurance team, one able to reason about the business/operational domain, yet also go down to work with tables in the database.
• Allowing the team to go where they want and explore what they need to, true autonomy.
• Use of a forensic, white box, approach to testing and reconciling the migration result back to the source data.
• Looking beyond the ‘migration specification’, an artefact that tends to focus on the ‘knowns’
• Use of a team with the technical skills that allows them to explore and manipulate the data itself and build tools to analyse and reconcile it.
• An adaptive ‘learning approach’ to assurance. Recognising it is only possible to work out what needs to be done once contact has been made with the data and with the results of migration.