Small code snippet for creating inverted index from the text files, with few small performance tweaks.
- JDK 8
- Maven
The built application tar
contains 4 directories
bin
directory contains a shell script to execute the application.conf
directory contains two files:- one to configure the number of part files (output_conf.json
) for the outputs and the another to specify spark configuration externally (spark_conf.json
).lib
directory contains ajar
of the application.
-
NOTE: Within the
spark_conf.json
, you can provide additional JVM parameter-XX:+UseStringDeduplication
through the keyspark.executor.extraJavaOptions
; all values should be space separated e.g-XX:+UseG1GC -XX:+UseStringDeduplication
-
NOTE: The output indexes are current persisted in string format. It can be stored using more efficient file format if the downstream applications are known.