This last month we upgraded from MongoDB 3.6 to MongoDB 8.0 for our API engine.
Keeping up to date on core software is essential for any long-lived public facing API, both for security standards and a few performance improvements as well.
Since direct database access via pymongo was integral to our backend - 40+ usages in 15+ files - it was a software dependency with 'deep roots'. This wasn't something that could be just done in one day. At least, not safely.
Our plan and implementation steps were:
Install MongoDB V8.0.3
Create a new test branch in our code repo for MongoDBV8.
Add a layer of abstraction for which MongoDB version was running in which subdomain (dev | stage | live).
Ensure multiple versions of MongoDB could be running simultaneously on different target directories and ports in our API software stack. There were new parameters for running V8 more safely alongside a V3 version of MongoDB on the same system, a temporary situation that usually is to be avoided, but frequently required for data migrations.
Implement parallel MongoDB instances to on the above changes in our test domain.
Develop a migrator python script, to pull all records out of 3.0 to 8.0
Run the migrator in our test area.
Review the data in the new 8.0 database, ensuring the integrity of the migration process.
Run the test area version of our API on the 8.0 version, run both automated and manual tests on it. Also do deep dive integrity checks on our automated database backup framework.
Remove the older 3.x version of MongoDB from our stack framework in the test area.
Repeat the above testing on our stage release area.
Decide a good time to implement the migration on our live API and implement the change in as little time as possible. We were crossed over in a few minutes.
Re-run all tests to make sure no surprises.
Sweep through and remove any temporary code required for non-versioned MongoDB access that was required for migration to the new layer MongoDB version layer of abstraction.
Commit our MongoDB8 code changes from the MongoDB8 branch into our main branch, and re-release and re-test.
Delete our old backup archives based on MongoDB v3
Verify our ongoing backup processes are working as expected.
Repeat a much simpler version of the above for upgrading pymongo, our python module bridge to MongoDB.
In the end, our migration took about 2 weeks, just doing one or two steps every day or so. Taking it slow ensured no big surprises or panics that could arise from unexpected migration code or MongoDB behavior. Usually one hears much focus on software development practices that ensure rapid release capability for making changes. However making substantive changes with core supporting apps with a large surface area of interaction throughout the code base is a different challenge. By doing it slowly over time, with the freedom to check and verify the data was flowing as expected all the way through the migration, we made this migration a happily invisible upgrade to our clients. (Other than the benefits of software currency and speedup improvements.)
Comments