Anser is a toolkit for managing evolving data sets for applications. It focuses on on-line data transformations and providing higher-level tools to support data modeling and access.
For the Evergreen project, anser allows us to treat these routine data migrations, data back fills, and retroactively changing the schema of legacy data as part of application code rather than one-off shell scripts.
In general, anser migrations have a two-phase approach. First a generator runs with some configuration and an input query to collect input documents and creation migration jobs. Then, the output of these generators, are executed in parallel
You can define generators either directly in your own code, or you can use the configuration-file based approach for a more flexible approach.
There are three major types of migrations:
simple: these migrations perform their transformations using MongoDB's update syntax. Use these migrations for very basic migrations, particularly when you want to throttle the rate of migrations and avoid the use of larger difficult-to-index multi-updates.manual: these migrations call a user-defined function on abson.RawDocrepresentation of the document to migrate. Use these migrations for more complex transformations or those migrations that you want to write in application code.stream: these migrations are similar to manual migrations; however, they pass a database session and an iterator to all documents impacted by the migration. These jobs offer ultimate flexibility.
Internally these jobs execute using amboy infrastructure and make it possible to express dependencies between migrations. Additionally the MovingAverageRateLimitedWorkers and SimpleRateLimitingWorkers were developed to support anser migrations, as well as the adaptive ordering local queue which respects dependency-driven ordering.
While it's possible to do any kind of migration with anser, we have found the following properties to be useful to keep in mind when building migrations:
- Write your migration implementations so that they are idempotent so that it's possible to run them multiple times with the same effect.
- Ensure that generator queries are supported by indexes, otherwise the generator processes will force collection scans.
- Rate-Limiting, provided by configuring the underlying amboy infrastructure, focuses on limiting the number of migration (or generator) jobs executed, rather than limiting the jobs based on their impact.
- Use batch limits. Generators have limits to control the number of jobs that they will produce. This is particularly useful for tests, but may have adverse effects on job dependency, particularly if logical migrations are split across more than one generator function.
Anser uses grip for logging and amboy for task management, otherwise, just begin using the
Please consult the godoc for most usage. Most of the API is in the top level package; however, please do also consider the model and bsonutil package.
Additionally you can use the interfaces db package as a wrapper for mgo to access Deciduosity which allows you to use mocks as needed for testing without depending on a running database instance.
Anser is available for use under the terms of the Apache License (v2).
If you encounter a problem with anser, or would like to see a feature added, please open an issue on the GitHub project!
Future anser development will focus on supporting additional migration workflows, supporting additional MongoDB and BSON utilities, and providing tools to support easier data-life-cycle management.