Read and writable partitioned sources #969
Conversation
TypeDelimited sources.
| } | ||
| } | ||
|
|
||
| // Create the underlying scrooge-parquet scheme and explicitly set the sink fields to be only the |
There was a problem hiding this comment.
looks like these comments are out of sync.
|
This looks great! Thanks for sharing this. Once we have tests and address some minor issues we will merge. |
|
I have added some tests for writing. I'm not sure how to set up tests for reading though. I don't really know how the JobTest works for reading. Does it actually create files on disk? |
|
I think you will want to use hadoop-platform test: This actually spins up a minicluster and behaves very closely to hadoop. There are methods in HadoopPlatformJobTest to initialize the sources (and feel free to add one of two if you think it is needed). By default, JobTest does mocking of the sources and sinks, so it is only testing job logic. |
|
Actually, your test looks great. Merging this. Thanks a ton! |
Read and writable partitioned sources
This is an initial implementation of partitioned versions of the TypeDelimited and TextLine sources.
I'll add the tests next.