Refactor towards sinks and sources other than CSVs #1
Conversation
|
We need to set up CI. Also I could just make this OSS as a reference implementation of the talk I gave. Really no reason to be private. |
|
I can add CI, do you have an example in mind I can copy it from? Otherwise I can figure it out. |
|
I think this plugin basically solves all the problems: https://github.com/typelevel/sbt-typelevel But it does require a bit of setup. |
| * @param name a unique name for a given source. To run, this name has to be connected to an input path | ||
| * @param validator the validator to check each input value and extract the timestamp | ||
| */ | ||
| def genericSource[A: PipeCodec](name: String, validator: Validator[A]): Event.Source[A] = |
There was a problem hiding this comment.
nit: I think this should be called def source and the one above should be called def csvSource or something.
| * @tparam A | ||
| */ | ||
| trait PipeCodec[A] { | ||
| def pipe[F[_]: RaiseThrowable: Sync]: Pipe[F, Byte, A] |
There was a problem hiding this comment.
nit: I'd probably call this def decode or something.
| def csv[A: Row](skipHeader: Boolean = true): PipeCodec[A] = | ||
| new CsvCodec[A](skipHeader) | ||
|
|
||
| def json[A: Codec](): PipeCodec[A] = new JsonCodec[A]() |
There was a problem hiding this comment.
nit: def json[A: Codec]: PipeCodec[A] = ... no need for ()
tlazaro
force-pushed
the
tlazaro/pipe-codec
branch
from
7be1cea
to
b749b1d
Compare
|
Made the fixes and rebased to #4 |
|
looks like we need to reformat. Also the scala 2.13 version seems really old. We should probably look at that. |
tlazaro
force-pushed
the
tlazaro/pipe-codec
branch
from
b749b1d
to
d1f4633
Compare
|
Sure, after this is done I will check the Scala and dependencies versions. |
Refactor towards sinks and sources other than CSVs
I wanted to be able to read from sources that were not CSV. The Row[A] is the only conduit for encoding and decoding an A and that won't necessarily work for other sources.
Tried to do minimal changes, left CSV as the implementation in most places, didn't change the signature of
Event.sourcebut added one that can take aPipeCodec[A]instead.The
PipeCodec[A]interface is not well thought out, I hacked it so I could replace the CSV hardcoded implementation but it will need changes.