Standalone application that filters, transforms, and indexes a stream of events from NSQ into Elasticsearch.
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.
The only prerequisites for using this repository are Docker & Node.js.
Installation links:
The configuration can be changed by editing config.yml file. This includes NSQ and ElasticSearch instance addresses, schema definition for valid messages etc.
Make sure you have Node.js installed.
git clone https://github.com/atoztoa/transformer.git # or clone your own fork
cd transformer
npm startcd transformer
docker build -t transformer .docker run -d --network host -t transformerNote: Make sure NSQ and ElasticSearch instances are running as per the configuration.
The unit tests can be run by the command
npm testNote: Make sure NSQ and ElasticSearch instances are running as per the configuration.
- Extensive app configuration.
- Docker support.
- Production ready.
- Scaleable horizontally as NSQ events are depleted on reading by any instance for the same channel.
- Overlooked the fact that in NSQ, 'Channels are created on first use by subscribing to the named channel'.
- Handling chained asynchronous HTTP requests, kudos to Promises.
- If transformer is started after NSQ has run for a while, it will start bombarding ElasticSearch which will reject the messages more than the queue size.
status: 429,
displayName: 'TooManyRequests',
message: '[es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$7@53f0ecd9 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@727c4094[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 2011]]',- Abheesh Suresh Babu
- Hat's off to Chris.