A demo to apply nsq consumer as spark consumer. Consumer receives messages from NSQd and store with spark streaming API.
Reference to dev.properties files for configuration of nsq receiver. Try playing with demo
NSQExample
Usage:
NSQExample <checkpoint-directory> <parallism>
CAUTION for nsq-client before 2.4.2
<dependency>
<groupId>com.youzan.bigdata</groupId>
<artifactId>spark-streaming-nsq_2.11</artifactId>
<version>1.0.5</version>
</dependency>
after 2.4.2
<dependency>
<groupId>com.youzan.bigdata</groupId>
<artifactId>spark-streaming-nsq_2.11</artifactId>
<version>1.0.6</version>
</dependency>
This connector consists of two version
- Unreliable
- Reliable
The difference of the two is whether "spark.streaming.receiver.writeAheadLog.enable" is set to true.
unreliable version is set to false and let the client ack messages automatically, and reliable one write messages
via WALog, and ack messages once messages are stored to disk.
One parameter needs mention is nsq.rdy, it works as throttle strategy.