PLEASE READ
Transporter no longer requires a YAML file. All configuration is in the JS file where nodes are defined and a new DSL has been developed. Run the init command to see the new changes.
- if using transporter as a library, all packages have been moved out of
pkgto the top-level evalcommand removedlistcommand removed- the
namespaceparameter now only expects a single part (the regexp filter), all adaptors have been updated to pull the "database name" from the provided URI
- NEW RabbitMQ adaptor #298
- MongoDB adaptor supports per collection query filter when needing to copy only a subset of data #301
- goja added as an option for the JavaScript VM in transformers #294
- NEW native functions
- attempted fix for mejson.S conversion to json column in postgres adaptor #314
- added RethinkDB -> PostgreSQL integration test
- fixed connection leak in PostgreSQL client
- The etcd adaptor was removed
- Integration tests are run weekly for the following pipelines:
- MongoDB -> MongoDB
- MongoDB -> Elasticsearch
- MongoDB -> RethinkDB
- RethinkDB SSL support added in #268
- RethinkDB performs bulk inserts now, #276
transporter init [source] [sink]command added in #279
- MongoDB adaptor could cause the pipeline to stop due to a concurrent flush operation, fixed via #271
- When being used as a library, several goroutines were leaking after the pipeline had stopped. Reported by @cognusion in #265 and addressed via #268 and #280
- namespace processing no longer expects their the be a "db" portion (i.e. "database.collection") but an attempt to maintain backwards compatibility is still there for the time being. #258
- #261: return a nil message to get skipped in the pipeline
This release is primarily aimed at getting the MongoDB and Elasticsearch adaptors into a stable/reliable state.
- MongoDB adaptor SSL configuration is now defined as:
nodes:
localmongo:
type: mongodb
ssl: true
cacerts: ["/path/to/cert.pem"] # optional- #211: defer bulk channel init for mongo node reuse
- #213: track mongodb _id field so we can attempt to reissue queries
- #233: update elasticsearch adaptor with better support for multiple versions of elasticsearch as well as better performance with bulk indexing for most versions. Addresses #209, #222, #167 and #159.
- properly detect oplog access when attempting to use the
tailoption on the MongoDB adaptor.
This release contains the first step to getting savable state into adaptors for the ability to resume.
- #124: Update elasticsearch adaptor
This release contains several breaking changes but is a first step towards a stable API/DSL.
- #95: add a Noop message type, and the ability to skip messages, fixes #93
- #100: replace environment variables, fixes #88
- #101: (Phase 1) Multi namespace support, addresses #78 and #23
- #94: jsonlog, fixes [#92](Comment doesnt reflect jsonlog output)
- #98: Fixes influx_test.js, thanks @ExNexu!
- #97: throw transformer initialization errors early, fixes #96
This release contains several breaking changes:
- Transporter now uses godep for vendoring depedencies.
- transformer functions now receive a document in the following format:
{
"op": "insert/update/delete",
"data": "", // what use to be passed in the doc passed to the transformer function
"ts": 1436794521, // the number of seconds elapsed since January 1, 1970 UTC
}
The same document must be returned from the function so any modifications must happen to doc["data"].
NOTE: Deletes are now passed to the transformer functions so this needs to be accounted for.
- #46: add a JsonLog.
- #64: New Source Adapter: Implements RethinkDB as a source of documents. Thanks @alindeman!
- #68: Deletes from Mongo write adaptor.
- #72: Allows RethinkDB timeout to be configured.
- #83: RethinkDB: Changes for v1 RethinkDB driver.
- #86: Use godep to vendor required libraries.
- #87: add SSL support for MongoDB, bump mgo.
- #90: NOTE BREAKING CHANGE: send more info in doc to transformers.
- #47: A few readme grammar fixes. Thanks @mm-!
- #49: fixes #48
- #51: fixes #50
- #54: import rethinkdb from gopkg.in. it's still v0, so this will change still.
- #55: remove influx until the state of the repository settles or we get proper vendoring in place.
- #63: use a smaller buffer for mongo adaptor.
- #65: Extracts only the relevant pieces of the version string. Thanks @alindeman!
- #85: ensure we're at version >=2.0 of rethink for this driver.
- #37: fixes #29
- #42: fixes #41
- #44: fix a logical error. check for transporter first.
- #45: add a bulk writer, and writeconcern options for mongo.
- #22: adaptor constructor refactor, new
transporter aboutcommand. - #26: adds the name and description to about.
- #27: Instructions for building Transporter on Windows.
- #31: add some more information in errors that we get in the javascript.
- #32: Fix typo in registry comments for doc gen.
- #7: Make list informative, stop arg panics.
- #9: beginners guide for os x. Thanks @sberryman!
- #10: Pretty print the list.
- #11: fix the config.yaml example to show the proper interval syntax.
- #13: makes the api in the config.yaml optional.
- #18: check that errors aren't nil.
- #20: log any adaptor.ERROR or adaptor.CRITICAL.
This is the initial release of Transporter.