When Preparation Meets Production Reality
They say failures are pillar of success. Now consider a scenario where everything in a project goes right from requirement gathering to documentation, design, implementation, testing and change management. But in the final hurdle during production deployment something goes wrong, and it fails horribly due to lack of proper planning. This kind of scenario undo whatever great job project has done earlier and makes this mistake all too glaring and visible to important stake holders. This shakes the confidence of client on project team as well. This is where a proper Project Cutover Plan comes into picture. A project cutover plan incorporates all the risk, issues and scenarios that can go wrong from past experiences and provides necessary mitigation plans and includes best practices from multiple projects.
There are numerous examples where improper planning resulted in massive failure. Let us go back to April 2018. TSB Bank, in an ambitious move, decided to upgrade its computer systems while shifting over a billion customer records. Sounds like a bold leap forward, right? Unfortunately, what was meant to be a technological triumph turned into a nightmare.
The result? A full-blown disaster of TSB’s internet and mobile banking systems. Customers were deeply impacted. Some couldn’t log in, others found their balances all wrong, and surprisingly, a few even landed up inside someone else’s account. It was chaos, the kind that shakes customer trust to the core. This incident serves as a textbook case of how critical cutover planning and risk mitigation are, especially when the stakes involve millions of end users and their financial data.
All projects generally have some kind of cutover plan depending on the type and complexity of the project. A cutover plan can vary from a new complex system implementation to something where systems are enhanced to include new features or modify existing system to fix some bugs. So now we need to know how a cutover plan looks like and what are the factors or parameters that influences a typical project cutover plan. We can analyze this from real world IT experience of numerous transformational projects from its inception to delivery and then subsequently transitioning it to Business as Usual (BAU) or maintenance activities.
Plan Early
They key to successful cutover plan is to start thinking about it early during the inception stages of the project. Ideally there should be some kind of preliminary approach to the cutover activities documented in Solution Design document e.g. type of deployment (Rolling Update, Blue-Green, Canary etc.), type of infrastructure (Cloud / On-premises) required etc. The type of deployment approach which is required as part of cutover plan depends on the type of application to be deployed and corresponding production down time that is acceptable as part of client Business Continuity Plan. These kind of broad approaches as part of conceptual solution document often requires going through Architecture review Board approval. While the detailed cutover plan or checklist will be elaborated towards the end of a project, time needs to be allocated to what cutover plan might look like much earlier. This type of thought process will lead to cutover plan being envisioned as part of Solution Design Document during early phases of project.
Involve all the stake holders
Actual cutover planning process requires involvement of all the stake holders of project delivery team, not just the architects. Different sets of people will have different perspective towards the cutover plan. Project managers will see it from perspective of resources, who is doing what and who are on standby mode. Testing team will see whether the functional and integration testing are approved by business users. Performance testing team will see whether the outcome of load testing results is approved by concerned business authority and all NFR (non-functional requirement) are met and aligned with the customer. Security team will look at the reports for security scan of code (e.g., Veracode/Prisma/Checkmarx etc.) are clean and approved by customer’s security team. Change Managers might look at a cutover plan through the lens of who needs to be notified and trained. The cutover plan should identify specific roles for each task and, ideally, nominate secondary resources should the primary ones not be available during the cutover period.
Dress Rehearsal
There is a saying that tomorrow belongs to the people who prepare for it today. The dress rehearsal of an actual cutover plan addresses just that. The actual cutover plan and checklist needs to be validated so that it is exhaustive and there is no room for new surprise. This scenario reminds us of famous scene from Apollo 13 movie where the astronauts needed to figure out a way of using carbon dioxide scrubbers meant for command module to the lunar module as well. It was a typical case of square peg in a round hole. The ground control stuff sitting in Houston command and control center worked diligently to find out a solution, rehearsed each step of the solution in the control center at earth and then provided astronauts with step-by-step process once they are confident with the results from the rehearsals at their end. The dress rehearsal or mock cutover is generally carried out in pre-production environment which closely simulates production environment in terms of infrastructure and code. The lessons learnt during dress rehearsal on pre-production environment is incorporated into production runbook and cutover plan to make it more exhaustive. We can implement some of the best practices during mock cutover in pre-prod environment itself e.g. like total code freeze, migrating / archiving data center (if applicable), involving same set of users doing both mock cutover and actual production cutover. We don’t want people doing a cutover task for the very first time in production. One of the major objectives of mock cutover is to establish time and sequence of each of the activities documented in Production Runbook and cutover plan.
Fig 1: Schematic representation of Prod and Pre-Prod activities for a bank
Stakeholder communication & Checkpoints
The communications plan is critical for banks for aligning stakeholder expectations with respect to project objectives and transition considerations. The plan considers current and targeted levels of understanding of program objectives among various stakeholder groups. The frequency of communication and checkpoints depends on the complexity and duration of the cutover process, the nature of the cutover tasks, and the number of stakeholders. Typically, high-level communications regarding the cutover will be distributed widely, while more granular communications will be tailored for project participants.
Communication plans are generally developed by the Change Managers who will work closely with Project Managers and steering committee members to define and agree with checkpoints and make stake holders aware about it. When cutover activities involve some production downtime, open and effective communication both internally and externally with business leadership and customers becomes even more important.
Fig 2: Typical Communication Plan for a bank during cut over process
Contingency / Rollback Plan
Imagine a team building a new bridge (the project) to replace a rickety old one (the legacy system). The cutover is like switching traffic to the new bridge, and the contingency / rollback is a plan to quickly switch back to the old bridge if something goes wrong during the switchover. The contingency plan may not be always about rollback plan, if something goes wrong during cutover process. The best way to handle when the things are not going on expected lines is to have an exception handling team ready who can help monitoring logs / queue for any issues. Assign issues (if any) to relevant SME (Subject Matter Expert). SME’s will then investigate and diagnose issues and identify solutions and then co-ordinate testing of the solution and authorize a work around, if possible. Rollback plan should be always kept as last resort to implement business continuity plan. A typical rollback plan might include several components or phases of rollback e.g., if cutover involves DB migration and archival for a bank then during Rollback first thing that we need to revert is DB migration related changes.
After a Rollback is done as part of contingency plan, testing team is involved to do one round of sanity testing before the Rollback is made available after cutover. At the end a detailed cutover plan is of no use unless it is followed diligently. A well-defined cutover plan, while crucial, is only as effective as it’s execution. This means not just having a plan, but actively following it, including communication, issue resolution, and stakeholder engagement.
About the Authors:
Mr. Soumya Bhattacharya is a Senior Architect working in a leading Software development/ Services company with over two decades of rich experience with expertise in Banking domain. He has led complex multi-year programs across multiple industries from vision to analysis, to implementation, to application go-live, including accommodations for business and technical requirements and impact.
Mr. Soumya Bhattacharya can be contacted at :
Mr. Raja Basu is a Senior Consulting professional in a leading MNC.
Mr. Raja Basu works as a business architect and helps global banking and financial markets clients to enable their digital transformation journey.
Mr. Raja Basu has special interest in responsible use of AI and sustainability.
Mr. Raja Basu is also pursuing his doctoral studies (PhD) from XLRI Jamshedpur.
Mr. Raja Basu is based out of Kolkata, India.
Mr. Raja Basu is an experienced leader in both technology and business, he has a proven track record of defining and implementing technology-driven transformations for clients in the global banking and financial markets.
Mr. Raja Basu focus lies in automation, particularly artificial intelligence (AI), and its impact on climate and sustainability (SCR).
Mr. Raja Basu possess a deep understanding of value-driven advisory practices, which have played a significant role in building strong client relationships. Throughout his career, he has actively contributed to numerous transformation programs involving complex applications for international clients across the United States, Canada, Europe, and Singapore.
Mr. Raja Basu is Bestowed with the following Licenses & Certifications :
https://www.linkedin.com/in/ba
Mr. Raja Basu is Volunteering in the following International Associations & Institutions :
https://www.linkedin.com/in/ba
Mr. Raja Basu is Accorded with the following Honors & Awards :
https://www.linkedin.com/in/ba
Mr. Raja Basu can be contacted at :
Also read Mr. Raja Basu’s earlier article :
