Spark-Streaming-Project

Outputs

Kafka Producer

Kafka Console Consumer

Progress

Streaming UI

1. How did changing values on the SparkSession property parameters affect the throughput and latency of the data?

• processedRowsPerSecond is affected.

2. What were the 2-3 most efficient SparkSession property key/value pairs? Through testing multiple variations on values, how can you tell these were the most optimal?

• spark.default.parallelism

• spark.streaming.kafka.maxRatePerPartition

  Based on the processed rows per second.

referred: https://knowledge.udacity.com/questions/123410