Add By modulators for select steps

[jeremysears]

Here's a cut at adding by modulation for select steps. Please take a look at the SelectSpec for usage of the tupled version. It accepts a tuple of StepLabel, followed by a tuple of By. The method signature for select/by could alternately be expressed as a Tuple of (StepLabel,By) pairs. I didn't spend terribly much time with it, but during my first pass at the pair-wise signature, I struggled to unzip the pairs HLists into an HList of StepLabel and an HList of Bys, because I'm not deep enough into shapeless to provide it with an implicit Unzip.Aux that the compiler could resolve. Before I hop into that rabbit hole, I thought I'd inquire with you about the method signature you would prefer to see for select/by (either one I've proposed or an alternate). Also, if you have any advice on how I could augment the following to supply an unzipper, please let me know.

    def select[StepLabelAndByTuplesAsTuples <: Product,
                 StepLabelsAndBysHList <: HList,
                 StepLabels <: HList,
                 Bys <: HList,
                 H0,
                 T0 <: HList,
                 LabelNames <: HList,
                 TupleWithValue,
                 Values <: HList,
                 Z,
                 ValueTuples](stepLabelAndByTuplesAsTuple: StepLabelAndByTuplesAsTuples)(
        implicit stepLabelAndByTuplesAsTupleToHList: ToHList.Aux[StepLabelAndByTuplesAsTuples, StepLabelsAndBysHList],
        unzipper: Unzip.Aux[StepLabelsAndBysHList, (StepLabels, Bys)],
        stepLabelToString: Mapper.Aux[GetLabelName.type, StepLabels, LabelNames],
        labelTrav: ToTraversable.Aux[LabelNames, List, String],
        byTrav: ToTraversable.Aux[Bys, List, By[_]],
        resultMapToHListFolder: RightFolder.Aux[StepLabelsAndBysHList,
                                                (HNil, JMap[String, Any]),
                                                combineModulatorWithValue.type,
                                                (Values, Z)],
        tupler: Tupler.Aux[Values, ValueTuples]
    ): GremlinScala.Aux[ValueTuples, Labels] = {
      // Select each StepLabel
      val stepLabelsAndBysHList: StepLabelsAndBysHList = stepLabelAndByTuplesAsTupleToHList(stepLabelAndByTuplesAsTuple)
      val (stepLabelsHList, bysHList) = stepLabelsAndBysHList.unzip

      val labels: List[String] = stepLabelsHList.map(GetLabelName).toList
      val label1 = labels.head
      val label2 = labels.tail.head
      val remainder = labels.tail.tail
      val selectTraversal = gremlinScala.traversal.select[Any](label1, label2, remainder: _*)

      // Apply By modulators to selected steps
      val bys: List[By[_]] = bysHList.toList
      var byTraversal = selectTraversal
      bys.foreach { by => byTraversal = by.apply(byTraversal) }

      // Extract selected values from the map of labeled values, and construct a
      // result tuple typed by Modulated from By[Modulated]
      GremlinScala(byTraversal).map { selectValues =>
        val resultTuple =
          stepLabelsAndBysHList.foldRight((HNil: HNil, selectValues))(combineModulatorWithValue)
        val values: Values = resultTuple._1
        tupler(values)
      }
    }

That fails to compile with:

Error:(201, 18) could not find implicit value for parameter unzipper: shapeless.ops.hlist.Unzip.Aux[StepLabelsAndBysHList,(StepLabels, Bys)]

Thank you for your efforts on this library!

[jeremysears]

Oh, and the calling syntax for the "pair-wise" signature would be something like this:

    val results: List[(String,Double,Double)] =
      newTraversal
        .selectBy(
          (
            (a,By(TestGraph.Name)),
            (b,By(TestGraph.Weight)),
            (c,By[Double]())
          )
        )
        .toList

[jeremysears]

It's worth mentioning that I also considered an argument list of TupleN[SelectBy*] where SelectBy is defined as something like this:

trait SelectBy[A,Modulated] {
  def label: StepLabel[A]
  def by: By[Modulated]
}

object SelectBy {
  def apply[A](label: StepLabel[A]) = new SelectBy[A,A] {
    override def label: StepLabel[A] = label
    override def by = By[A]()
  }

  def apply[A,Modulated](label: StepLabel[A], by: By[Modulated]) = new SelectBy[A,Modulated] {
    override def label = label
    override def by = by
  }
}

That would give the "identity" By() modulator a bit more type safety, as it would force the Modulated type to the StepLabel type. However, I was leery of adding more types to the public interface.

[mpollmeier]

I'll definitely get back to you on this, since I had some thoughts on this topic, but I'm super busy at the moment, so please forgive the waiting time.

[mpollmeier]

This is awesome, it's taking the already quite cool select to another level.
I agree, it would be even nicer to take tuples of (StepLabel, By). I moved it one step further by extracting the StepLabels HList and Bys HList from the Unzipped type. Now it doesn't find the implicit ToTraversable.Aux[Bys, List, By[_]] (once you uncomment it). Now sure why, in your other select it seems to work.

It's so rewarding to see this stuff actually work. And we're almost there...
Gotta stop for now, just wanted to share my intermediate results with you:

  def select2[
      StepLabelAndByTuplesAsTuples <: Product, // ((StepLabel, Key), (StepLabel, Key))
      StepLabelsAndBysHList <: HList, // (StepLabel, Key) :: (StepLabel, Key) :: HNil
      Unzipped <: Product, //(StepLabel :: StepLabel :: HNil, Key :: Key :: HNil)
      UnzippedHList <: HList, // (StepLabel :: StepLabel :: HNil) :: (Key :: Key :: HNil) :: HNil
      StepLabels <: HList, // StepLabel :: StepLabel :: HNil
      BysHListHList <: HList, // (Key :: Key :: HNil) :: HNil
      Bys <: HList, // Key :: Key :: HNil
      H0,
      T0 <: HList,
      LabelNames <: HList,
      TupleWithValue,
      Values <: HList,
      Z,
      ValueTuples](stepLabelAndByTuplesAsTuple: StepLabelAndByTuplesAsTuples)(
      implicit stepLabelAndByTuplesAsTupleToHList: ToHList.Aux[StepLabelAndByTuplesAsTuples,
                                                               StepLabelsAndBysHList],
      unzipper: Unzip.Aux[StepLabelsAndBysHList, Unzipped],
      unzippedToHList: ToHList.Aux[Unzipped, UnzippedHList],
      extractStepLabels: IsHCons.Aux[UnzippedHList, StepLabels, BysHListHList],
      extractBys: IsHCons.Aux[BysHListHList, Bys, _],
      stepLabelToString: Mapper.Aux[GetLabelName.type, StepLabels, LabelNames],
      labelTrav: ToTraversable.Aux[LabelNames, List, String]
      // byTrav: ToTraversable.Aux[Bys, List, By[_]],
      // resultMapToHListFolder: RightFolder.Aux[StepLabelsAndBysHList,
      //                                         (HNil, JMap[String, Any]),
      //                                         combineModulatorWithValue.type,
      //                                         (Values, Z)],
      // tupler: Tupler.Aux[Values, ValueTuples]
  ): Bys = ???
  // ): ValueTuples = ???

Specifying a random return type is my way of debugging the compiler, it tells me the actual return type in the error msg :)

      val traversal = newTraversal
      val x: Int = traversal.select2(
        ((StepLabel[Vertex]("v"), Key[String]("name")), (StepLabel[Edge]("e"), Key[Int]("age"))))

[mpollmeier]

I'm getting closer, not sure if the value level works yet, and need to clean up:

  lazy val select3Apply = {
    val v = StepLabel[Vertex]()
    val e = StepLabel[Edge]()
    val gs: GremlinScala.Aux[Edge, Vertex :: Edge :: HNil] = __[Vertex]().as(v).outE.as(e)
    val res: GremlinScala.Aux[(Vertex, String), _] =
      gs.select3(((v, By[Vertex]()), (e, By[String]())))
  }
  def select3[
      StepLabelAndByTuplesAsTuples <: Product, // ((StepLabel, Key), (StepLabel, Key))
      StepLabelsAndBysHList <: HList, // (StepLabel, Key) :: (StepLabel, Key) :: HNil
      Unzipped <: Product, //(StepLabel :: StepLabel :: HNil, Key :: Key :: HNil)
      UnzippedHList <: HList, // (StepLabel :: StepLabel :: HNil) :: (Key :: Key :: HNil) :: HNil
      StepLabels <: HList, // StepLabel :: StepLabel :: HNil
      BysHListHList <: HList, // (Key :: Key :: HNil) :: HNil
      Bys <: HList, // Key :: Key :: HNil
      H0,
      T0 <: HList,
      LabelNames <: HList,
      TupleWithValue,
      Values <: HList,
      Z,
      ValueTuples](stepLabelAndByTuplesAsTuple: StepLabelAndByTuplesAsTuples)(
      implicit stepLabelAndByTuplesAsTupleToHList: ToHList.Aux[StepLabelAndByTuplesAsTuples,
                                                               StepLabelsAndBysHList],
      unzipper: Unzip.Aux[StepLabelsAndBysHList, Unzipped],
      unzippedToHList: ToHList.Aux[Unzipped, UnzippedHList],
      extractStepLabels: IsHCons.Aux[UnzippedHList, StepLabels, BysHListHList],
      extractBys: IsHCons.Aux[BysHListHList, Bys, _],
      stepLabelToString: Mapper.Aux[GetLabelName.type, StepLabels, LabelNames],
      labelTrav: ToTraversable.Aux[LabelNames, List, String],
      byTrav: ToTraversable.Aux[Bys, List, By[_]],
      resultMapToHListFolder: RightFolder.Aux[StepLabelsAndBysHList,
                                              (HNil, JMap[String, Any]),
                                              combineModulatorWithValue.type,
                                              (Values, Z)],
      tupler: Tupler.Aux[Values, ValueTuples]
  ): GremlinScala.Aux[ValueTuples, Labels] = {
    // Select each StepLabel
    val stepLabelsAndBysHList: StepLabelsAndBysHList = stepLabelAndByTuplesAsTupleToHList(
      stepLabelAndByTuplesAsTuple)

    /* TODO: this is safe, but how do I avoid casting? */
    val (stepLabelsHList, bysHList) = stepLabelsAndBysHList.unzip.asInstanceOf[(StepLabels, Bys)]

    val labels: List[String] = stepLabelsHList.map(GetLabelName).toList
    val label1 = labels.head
    val label2 = labels.tail.head
    val remainder = labels.tail.tail
    val selectTraversal = traversal.select[Any](label1, label2, remainder: _*)

    // Apply By modulators to selected steps
    val bys: List[By[_]] = bysHList.toList
    var byTraversal = selectTraversal
    bys.foreach { by =>
      byTraversal = by.apply(byTraversal)
    }

    // Extract selected values from the map of labeled values, and construct a
    // result tuple typed by Modulated from By[Modulated]
    GremlinScala(byTraversal).map { selectValues =>
      val resultTuple =
        stepLabelsAndBysHList.foldRight((HNil: HNil, selectValues))(combineModulatorWithValue)
      val values: Values = resultTuple._1
      tupler(values)
    }
  }

[jeremysears]

Great progress! I haven't had a chance to get back into this since last week. Have you given any thought to using a trait for each element (like the SelectBy), instead of a (StepLabel[],By[])? It's further from the gremlin idiom, but it provides a bit more type safety. It may be possible to support both the tupled version or SelectBy in one method, by making SelectBy extend Product, or by making it a case class. My concern with the tupled version is that the identity By() requires you to supply the output type, which is effectively a user-specified cast. It doesn't do much to help By(T(oken)), other than the T.label version where we could coerce that into String. Having said that, I also like the API cleanliness of sticking with more familiar gremlin constructs.

[mpollmeier]

Yes, I'm still a bit undecided, will see how it plays out. Might also be useful for the (common) case that someone only wants to by-modulate a subset of the selected values, but maybe we can handle that with the tupled version as well.