Working code to demonstrate key Spark concepts!
Custom transformations and user defined functions in Spark should be tested with the spark-testing-base library.
Check out the basic package for code snippets.
DataFrame transformations can be chained with implicit classes or the Dataset#transform method.
The examples in the chaining package show how to use both these methods and how to use the Dataset#transform for custom transformations that take arguments.