`foldLeftM` without `Free`. #1117

TomasMikula · 2016-06-11T18:51:57Z

This is an implementation of foldLeftM without using Free. Instead, it introduces an auxiliary structure

case class Source[A](uncons: () => Option[(A, Source[A])])

and a conversion from any foldable F[A] to Source[A], which provides lazy access to elements of F[A].

It still uses foldRight, but I think there is a better separation of concerns here, because G is not involved in the foldRight at all.

~~I didn't bother to move this out of Free.scala, but Free is not used here at all.~~ I moved the implementation to Foldable.

Funny enough, if I make Source extend AnyVal, it crashes the compiler with a StackOverflowError (SI-9600).

codecov-io · 2016-06-11T19:27:51Z

Codecov Report

Merging #1117 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1117      +/-   ##
==========================================
- Coverage   93.99%   93.98%   -0.02%     
==========================================
  Files         249      249              
  Lines        4131     4140       +9     
  Branches      158      160       +2     
==========================================
+ Hits         3883     3891       +8     
- Misses        248      249       +1

Impacted Files	Coverage Δ
free/src/main/scala/cats/free/Free.scala	`84% <ø> (-2.54%)`	⬇️
core/src/main/scala/cats/Foldable.scala	`98.75% <100%> (+0.19%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ba8993...eab2053. Read the comment docs.

ceedubs · 2016-06-13T12:59:58Z

@TomasMikula thanks, this does seem quite a bit simpler than the Free approach. This week I'll try to take some time to figure out which other methods we want better stack-safe/lazy support for and whether we can do something similar there as opposed to needing to bring Free into core.

TomasMikula · 2016-06-13T17:00:00Z

This approach basically gives an Iterator-like/Stream-like interface to (any) Foldable, which could potentially be useful elsewhere. Also, I think pretty much any concrete foldable data type will have a much more efficient implementation of foldLeftM (or "toStream") than this default one using foldRight.

ceedubs · 2016-06-16T12:13:17Z

@TomasMikula I like this. I'd be interested in moving it into core somewhere where we could use it in the default implementation of Foldable.foldLeftM I think (though we could maybe leave it private for now so it doesn't become a part of the official cats API for now). I'd also be interested in some benchmarks to see whether there's any performance difference if you were to use an abstract class with an abstract def as opposed to a case class with a Function0 val. I'm happy to put these together, but I also don't want to steal your thunder if this is something that you want to see through. WDYT?

ceedubs · 2016-06-16T12:13:52Z

cc @non since he and I have talked about foldLeftM and the possibility of moving Free to core to get the functionality that this achieves.

TomasMikula · 2016-06-16T18:21:04Z

@ceedubs I made the changes. I made Source to be an abstract class. I was originally going for a value class, but hit SI-9600.

TomasMikula · 2016-06-16T18:24:45Z

Perhaps we need better naming for foldM vs. foldLeftM? They are both left folds. They differ in the ability to short-circuit.

I don't like the Eval.value calls, but I don't know of a way around them. The unit tests suggest that they aren't a problem due to the nature of `Free.runTailRec` and `FlatMapRec.tailRecM`.

ceedubs · 2016-06-18T12:21:43Z

tests/src/test/scala/cats/tests/FoldableTests.scala

+  test(".foldLeftM") {
+    // you can see .foldLeftM traversing the entire structure by
+    // changing the constant argument to .take and observing the time
+    // this test takes.


This comment should no longer be true, should it?

Right, it became obsolete earlier, though. See my comment. The whole test case is redundant in favor of the one below.

ceedubs · 2016-06-18T12:22:59Z

Perhaps we need better naming for foldM vs. foldLeftM? They are both left folds. They differ in the ability to short-circuit.

@TomasMikula I agree. I think we are going to likely end up having several methods that have a Monad form and a separate MonadRec form. Maybe to fit the names of those two type classes we could use the convention of foldLeftM and foldLeftMRec?

ceedubs · 2016-06-18T12:27:26Z

core/src/main/scala/cats/Foldable.scala

+      F.foldRight[A, Source[A]](fa, Now(Source.empty[A]))((a, evalSrc) =>
+        Now(cons(a, evalSrc))
+      ).value
+  }


This is probably me prematurely optimizing, but I'd be tempted to do a couple things:

Make empty reuse a singleton either by making Source covariant or by giving it a def widen[AA >: A]: Source[AA] = this.asInstanceOf[Source[AA]] method.

Possibly store a value of Now(Source.empty) around for reuse by fromFoldable.

TomasMikula · 2016-06-18T22:10:42Z

@ceedubs I made the changes, except renaming. I like the -Rec suffix convention. Do you mean we should rename foldM to foldLeftM? Or the current foldLeftM to foldMRec?

ceedubs · 2016-06-19T12:34:58Z

@TomasMikula I was thinking about renaming the current foldM to foldLeftM and having this new one be foldLeftMRec. I guess if we don't have a right version then there is no harm in foldM and foldMRec, but I think it makes it a little more obvious that it will be left-associated and parallels the foldLeft name better. WDYT?

TomasMikula · 2016-06-19T17:17:02Z

@ceedubs Makes sense. My concern was just whether foldM was a standard name for this operation. Probably not. scalaz has foldLeftM, Haskell has foldlM.

TomasMikula · 2016-06-19T17:23:08Z

How about now?

ceedubs · 2016-06-19T17:28:04Z

👍 thanks @TomasMikula!

It might be nice to have the foldLeftM and foldLeftMRec scaladocs reference each other to point out why people might want one or the other. Also we'll probably want to optimize the implementations in some of our instances. But I think all of that can wait for subsequent PRs. I'd rather get this merged and then iterate.

kailuowang · 2017-06-06T03:19:26Z

@TomasMikula I am sorry that this fell through the cracks, any interest in resolving the conflicts? It looks like that we are close to merging. If you don't have time I'd be happy to continue it through.

johnynek · 2017-06-06T17:40:31Z

core/src/main/scala/cats/Foldable.scala

   */
-  def foldM[G[_], A, B](fa: F[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B] =
+  def foldLeftM[G[_], A, B](fa: F[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B] =


we don't need this now, right?

johnynek · 2017-06-06T17:42:04Z

core/src/main/scala/cats/Foldable.scala

+   * have a more efficient implementation than the default one
+   * in terms of `foldRight`.
+   */
+  def foldLeftMRec[G[_], A, B](fa: F[A], z: B)(f: (B, A) => G[B])(implicit G: MonadRec[G]): G[B] = {


can we rename this foldLeftM or just stick with foldM. I'm not sure we should be changing this name this late in the game, although I agree foldLeftM is a better name.

Maybe add a deprecated final foldM that redirects to foldLeftM?

TomasMikula · 2017-06-18T00:02:14Z

How about now?

johnynek · 2017-06-18T07:23:06Z

core/src/main/scala/cats/Foldable.scala

+    val NowEmpty: Eval[Source[Nothing]] = Now(Empty)
+
+    def cons[A](a: A, src: Eval[Source[A]]): Source[A] = new Source[A] {
+      def uncons = Some((a, src.value))


I'm a little nervous about these calls to value. Can we add a comment about the stack safety here? I think the answer is that we can't recursively call .value in an Eval and we should never be in an Eval when we call uncons, is that right?

The presence of Eval in this case actually just obstructs the understanding, IMO. The src parameter of cons could just be by-name (the evaluation of .value doesn't need any trampolining built into Eval). However, since foldRight already uses Eval, I didn't want to mix both by-name and Eval.

A clearer definition of Source would be

type Source[A] = () => Option[(A, Source[A])]

except that recursive type aliases are not allowed. I guess I could make it an AnyVal, though. EDIT: Nope, because of scala/bug#9600.

johnynek · 2017-06-18T07:23:57Z

core/src/main/scala/cats/Foldable.scala

+   * in terms of `foldRight`.
+   */
+  def foldLeftM[G[_], A, B](fa: F[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B] = {
+    val src = Foldable.Source.fromFoldable(fa)(self)


Kind of a bummer that is someone has override on foldM they will have two different folds. Maybe one should be final and a pointer to the other?

TomasMikula · 2017-06-18T11:54:04Z

I realized that in case of short-circuited fold, it evaluates 1 more element than necessary. Maybe I should fix that...

EDIT: Note that because of the signature of foldM

def foldM[G[_], A, B](fa: F[A], z: B)(f: (B, A) => G[B])(implicit G: Monad[G]): G[B]

specifically the part z: B, we can never short-circuit before evaluating at least 1 element. If instead it was z: G[B], there would be a possibility to short-circuit without evaluating any elements at all.

johnynek · 2017-06-21T18:08:59Z

👍

kailuowang · 2017-06-21T18:17:22Z

merged with 3 sign-offs

kailuowang · 2017-06-23T14:48:04Z

tests/src/test/scala/cats/tests/FoldableTests.scala

+    assert(res == Left(100000))
+  }
+
+  test(".foldLeftM short-circuiting optimality") {


This test fails if we remove Foldable.iteratorFoldM, because it actually no longer tests the default foldM implementation from Foldable after #1414 which overrides foldLeftM in the instance of Stream using iteratorFoldM. This test fails when using the default foldM because, if I am not mistaken, the new foldM still have to evaluate one element after the stop. Also these two tests are now redundant with the test("Foldable[Stream]") above. I will fix this in a PR removing iteratorFoldM

Huh, I didn't notice that I was testing some other code 😃. I suppose changing the signature of

def uncons: Option[(A, Source[A])]

to

def uncons: Option[(A, Eval[Source[A]])]

and doing the necessary changes would make it pass.

right, I updated #1740

TomasMikula mentioned this pull request Jun 11, 2016

Some clean up for Free (and friends). #1085

Merged

TomasMikula force-pushed the foldLeftM branch from 436a393 to 27f2514 Compare June 16, 2016 18:18

ceedubs reviewed Jun 18, 2016
View reviewed changes

TomasMikula force-pushed the foldLeftM branch from 27f2514 to 45fcb07 Compare June 18, 2016 22:07

This was referenced Jul 14, 2016

Implement Reducible.reduceM #1193

Merged

WIP: add some missing applicative and monad ops #1216

Closed

peterneyens mentioned this pull request Jun 3, 2017

Foldable instance for List breaks substitution #1716

Closed

kailuowang added this to the 1.0.0-MF milestone Jun 6, 2017

kailuowang mentioned this pull request Jun 6, 2017

Prepare for the 1.0.0-MF release #1702

Merged

26 tasks

johnynek reviewed Jun 6, 2017

View reviewed changes

kailuowang added the Source Breaking label Jun 15, 2017

TomasMikula force-pushed the foldLeftM branch 2 times, most recently from ab7d173 to e6b28b4 Compare June 17, 2017 22:55

johnynek reviewed Jun 18, 2017

View reviewed changes

foldLeftM without Free.

eab2053

TomasMikula force-pushed the foldLeftM branch from e6b28b4 to eab2053 Compare June 18, 2017 12:28

ceedubs approved these changes Jun 21, 2017

View reviewed changes

kailuowang approved these changes Jun 21, 2017

View reviewed changes

kailuowang merged commit e3a969e into typelevel:master Jun 21, 2017

TomasMikula deleted the foldLeftM branch June 21, 2017 18:29

kailuowang reviewed Jun 23, 2017

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`foldLeftM` without `Free`. #1117

`foldLeftM` without `Free`. #1117

TomasMikula commented Jun 11, 2016 •

edited

Loading

codecov-io commented Jun 11, 2016 •

edited

Loading

ceedubs commented Jun 13, 2016

TomasMikula commented Jun 13, 2016 •

edited

Loading

ceedubs commented Jun 16, 2016

ceedubs commented Jun 16, 2016

TomasMikula commented Jun 16, 2016

TomasMikula commented Jun 16, 2016 •

edited

Loading

ceedubs Jun 18, 2016

TomasMikula Jun 18, 2016

ceedubs commented Jun 18, 2016

ceedubs Jun 18, 2016

TomasMikula commented Jun 18, 2016 •

edited

Loading

ceedubs commented Jun 19, 2016

TomasMikula commented Jun 19, 2016

TomasMikula commented Jun 19, 2016

ceedubs commented Jun 19, 2016

kailuowang commented Jun 6, 2017 •

edited

Loading

johnynek Jun 6, 2017

johnynek Jun 6, 2017

TomasMikula commented Jun 18, 2017

johnynek Jun 18, 2017

TomasMikula Jun 18, 2017 •

edited

Loading

johnynek Jun 18, 2017

TomasMikula Jun 18, 2017

TomasMikula commented Jun 18, 2017 •

edited

Loading

johnynek commented Jun 21, 2017

kailuowang commented Jun 21, 2017

kailuowang Jun 23, 2017 •

edited

Loading

TomasMikula Jun 23, 2017

kailuowang Jun 23, 2017

foldLeftM without Free. #1117

foldLeftM without Free. #1117

Conversation

TomasMikula commented Jun 11, 2016 • edited Loading

codecov-io commented Jun 11, 2016 • edited Loading

Codecov Report

ceedubs commented Jun 13, 2016

TomasMikula commented Jun 13, 2016 • edited Loading

ceedubs commented Jun 16, 2016

ceedubs commented Jun 16, 2016

TomasMikula commented Jun 16, 2016

TomasMikula commented Jun 16, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ceedubs commented Jun 18, 2016

Choose a reason for hiding this comment

TomasMikula commented Jun 18, 2016 • edited Loading

ceedubs commented Jun 19, 2016

TomasMikula commented Jun 19, 2016

TomasMikula commented Jun 19, 2016

ceedubs commented Jun 19, 2016

kailuowang commented Jun 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomasMikula commented Jun 18, 2017

Choose a reason for hiding this comment

TomasMikula Jun 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomasMikula commented Jun 18, 2017 • edited Loading

johnynek commented Jun 21, 2017

kailuowang commented Jun 21, 2017

kailuowang Jun 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

`foldLeftM` without `Free`. #1117

`foldLeftM` without `Free`. #1117

TomasMikula commented Jun 11, 2016 •

edited

Loading

codecov-io commented Jun 11, 2016 •

edited

Loading

TomasMikula commented Jun 13, 2016 •

edited

Loading

TomasMikula commented Jun 16, 2016 •

edited

Loading

TomasMikula commented Jun 18, 2016 •

edited

Loading

kailuowang commented Jun 6, 2017 •

edited

Loading

TomasMikula Jun 18, 2017 •

edited

Loading

TomasMikula commented Jun 18, 2017 •

edited

Loading

kailuowang Jun 23, 2017 •

edited

Loading