Scalding viz options #1440
Scalding viz options #1440
Conversation
| @@ -521,11 +521,11 @@ trait CounterVerification extends Job { | |||
| } | |||
| } | |||
|
|
|||
| private[scalding] object FlowStepStrategies { | |||
| private[scalding] case class FlowStepStrategies[A]() extends Semigroup[FlowStepStrategy[A]] { | |||
There was a problem hiding this comment.
Should this be a Semigroup? Looks like a side-effecting function here. Are we guaranteed associativity?
There was a problem hiding this comment.
I think strategies will be applied in a sequence so it is associative (and not commutative)? /cc @ianoc as he wrote this.
There was a problem hiding this comment.
Yep, this is a semigroup because plus is assocative
There was a problem hiding this comment.
It's not side-effecting. Nothing happens in plus. It is just creating a new strategy. Looks correct to me.
There was a problem hiding this comment.
Making an analogy to Haskell, it's equivalent to the semigroup over this type:
newtype FlowStepStrategy a =
FlowStepStrategy (Flow a -> JList (FlowStep a) -> FlowStep a -> IO ())
instance Monoid (FlowStepStrategy a) where
mempty = FlowStepStrategy (\_ _ _ -> return ())
mappend (FlowStepStrategy f1) (FlowStepStrategy f2) = FlowStepStrategy f
where
f x y z = do
f1 x y z
f2 x y z... and that's guaranteed to satisfy the Monoid laws in both Haskell and Scala.
The most common source of this thing failing in Scala specifically is if side effects are triggered prematurely by evaluating things in the wrong order but that does not appear to be the case for this code.
There was a problem hiding this comment.
Ah ok. What I was confused about was if plus(a, plus(b, c)).apply and plus(plus(a, b), c).apply will have the same effect.
| val reducerEstimatorStrategy: Seq[FlowStepStrategy[JobConf]] = config.get(Config.ReducerEstimators).toList.map(_ => ReducerEstimatorStepStrategy) | ||
|
|
||
| val otherStrategies: Seq[FlowStepStrategy[JobConf]] = config.getFlowStepStrategies.map { tTry: Try[(Mode, Config) => FlowStepStrategy[JobConf]] => | ||
| val t: (Mode, Config) => FlowStepStrategy[JobConf] = tTry.getOrElse(throw new Exception(s"Failed to decode flow step strategy $tTry when submitting job")) |
There was a problem hiding this comment.
why didn't you use the case style here?
There was a problem hiding this comment.
also, use caused-by constructor (see below).
|
Lgtm. What's up with the version number? What does exec mean here? |
|
+1 |
|
Changes Unknown when pulling 899b645 on MansurAshraf:mashraf/scalding_viz into ** on twitter:develop**. |
Continuation of #1426 with published Chill version