Application Deployment Strategy and Roll Back Method

page owner: tfrs devops specialist

Openshift Platform deployment strategy

• Rolling Strategy
  • When you want to take no downtime during an application update.
  • When your application supports having old code and new code running at the same time.
• Recreate Strategy
  • When you must run migrations or other data transformations before your new code starts.
  • When you do not support having new and old versions of your application code running at the same time.
  • When you want to use a RWO volume, which is not supported being shared between multiple replicas.
• TFRS chooses Recreate Strategy and add the Python data model migrations in the mid lifecycle hook.

Things to consider for the deployment method

N-1 Compatibility

Each release is backward compatible with previous release. It means two release can run at same time without any damages to the system.

• Release/Feature flag
• Graceful Termination
• A/B Deployment
• Maintenance page
• blue-green deployment strategy: implement more complex checks of the application (such as sending real user workloads to the new instance),

Blue/Green Deployment

• Need to qualify N-1 Compatibility
• Existing and running deployment config is green(on v1 image)
• Green Service uses the pods created by green config
• Existing route is using existing green service
• Create the blue deployment config uses v2 image
• Create Blue Service to use blue deployment config
• Fully test blue service before switch the route to use Blue service.
• Switch the route to use Blue service.
• Stop the green pods and delete the Green deployment config

A/B Deployment

• Need to qualify N-1 Compatibility
• Existing and running deployment config is A(on v1 image)
• Existing A Service uses the pods created by A config
• Existing route is using existing A service
• Create the B deployment config uses v2 image
• Create B Service to use blue deployment config
• Fully test B version through B service
• Add the B service to exiting Route to let the route use both service A and B.
• Update the route apply weight 90 on service A and weight 10 on service B so that the traffics can be split accordingly among A and B.
• Watch the B version to make sure it works as expected. Then update the weight to service B to 100 and lower the weight of service A to 0.
• Finally remove service A from route and shutdown A pods. note: one Route can use 2 services, one service uses pods of current running version which is A version, the other service uses pods of new release which is B version. Give weights on the two services.

TFRS option for the deployment method

TFRS chooses simple recreate strategy with N-1 Compatibility and a feature flag function (not implemented yet). When the the feature flag is turned off, the new added features will not to be rolled out. They are rolled out only when the feature flag is turned on.

Frontend/Backend deployment rollback

When frontend deployment needs to be rolled back we can choose Openshift Console to roll back the release or use oc command line

Openshift Console

Choose the deployment and choose the previous release number Click the Roll Back button and choose "deployment strategy" option. Relogin the application to verify if the roll back is successful.

oc command

• oc rollout undo dc/[deoployment config] --to-revision=[the release number]
  If the --to-revision is not specified, system will roll back to previous successful release.
• oc set triggers dc/[deoployment config] —auto During the roll back, the trigger is disabled, this command brings the trigger back so that if there is a image change, it still can be deployed.

Backend rollback

The complexity of the backend rollback is caused by data migration. First of all, this type of rollback can be successfully done and it is confirmed by a testing conducted on our dev environment. But we still need to pay additional attention when go through the process. The followings are recommended:

• Make the migration more granular. It means better to have more migration files. Don't try to do complicated migrations within one migration file.
• Try to reduce the dependencies between migration files or make it properly ordered. For example, adding the index on the column should be after addinbg the column
• Practise the rollback process on dev first to make sure it can pass. Then start the production deployment.
• Always take a full database baseline backup before apply the any new release to prod.