Make Show inherit from a contravariant base trait #1649

edmundnoble · 2017-05-05T03:20:03Z

Contravariance in type class parameters is well-documented to be an issue for the compiler. Specifically, the compiler has a useless version of "specificity" which forces the least specific instance to be used, if there are overlapping instances and the user does not use an explicit inheritance-based prioritization.

However, the benefit is still there: firstly there is no need for a Show[BitSet], because the existing Show[Set[Int]] is narrowed implicitly. In the same vein, Some(1).show works even if your instance only exists for Option[Int] and not Some[Int]. These benefits become more apparent when you construct values from inside a show interpolator, and need to use type ascriptions with the existing version of Show.

For examples of how users can make overlapping Show work, see https://github.com/paulp/psp-std/blob/master/std/src/main/scala/std/Show.scala.

This PR breaks the API in the following ways:
a) Not only was the BitSet show instance no longer necessary, but it actually began to conflict with the Set[Int] instance.
b) Implicit specificity no longer functions properly. Specifically, users cannot have a Show[Supertype] and Show[Subtype] and expect that implicitly[Show[Subtype]] will materialize the latter.

To be clear this is a trade-off. I am not sure how, practically, user usage of Show benefits and is hurt by this change.

kailuowang · 2017-05-05T03:33:48Z

core/src/main/scala/cats/instances/bitSet.scala

-import scala.collection.immutable.BitSet
-import cats.Show
-
-trait BitSetInstances extends cats.kernel.instances.BitSetInstances {


There are two other instances from kernel provided in cats.implicits through this.

Oh whoops. I'll add that back in.

codecov-io · 2017-05-05T04:00:51Z

Codecov Report

Merging #1649 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master    #1649   +/-   ##
=======================================
  Coverage   94.17%   94.17%           
=======================================
  Files         256      256           
  Lines        4208     4208           
  Branches       93       93           
=======================================
  Hits         3963     3963           
  Misses        245      245

Impacted Files	Coverage Δ
core/src/main/scala/cats/Show.scala	`100% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 099e459...59822c1. Read the comment docs.

kailuowang · 2017-05-08T14:39:13Z

tests/src/test/scala/cats/tests/BitSetTests.scala

-
-  test("show BitSet"){
-    BitSet(1, 1, 2, 3).show should === ("BitSet(1, 2, 3)")
-    BitSet.empty.show should === ("BitSet()")


So the new result will be "Set(1, 2, 3)" instead of "BitSet(1, 2, 3)" right?

johnynek · 2017-05-08T17:13:57Z

I'm -1 on this change. I'd rather add a contramap method and let users opt in: implicit val bitset = Show[Set[Long]].contramap(bs: BitSet => bs)

paulp · 2017-05-08T17:24:11Z

I don't think Bitset opting into something can solve anything. The issue is a continuous stream of variations on this theme:

scala> def f[A: Show](x: Some[A]) = show"$x"
<console>:17: error: type mismatch;
 found   : Some[A]
 required: cats.Show.Shown
       def f[A: Show](x: Some[A]) = show"$x"
                                          ^

scala> def f[A: Show](x: Some[A]) = show"${ x: Option[A] }"
f: [A](x: Some[A])(implicit evidence$1: cats.Show[A])String

Has anyone used the cats Show more than in a trivial test? I would like to see how they're managing that satisfactorily.

johnynek · 2017-05-08T17:36:05Z

I have used it more than trivially.

So, I think that variance is generally frowned on in the typeclasses in cats. For instance, Order or Eq could be contravariant, but they are not.

The example that @paulp gives is a pretty well known one and generally solved by not writing methods that accept subtypes of an ADT, or by writing an explicit type: show"${x: Option[Int]}" which is not great, but generally the benefit of avoiding confusing cases with variance was judged to be beneficial.

I don't actually know all the objections, but @non was telling me once how contravariant typeclasses in particular had inference issues, so I am just spreading fear.

That said, if we do this, why wouldn't we add variance all over the place in cats? I thought the idea was to use Functor or Contravariant to handle these cases, which have widen and narrow and folks felt like it creates less issues in the long run.

By the way, this is why cats ADTs generally give constructor methods that return the outer type. For instance in the old days Xor.right returned Xor not Xor.Right.

paulp · 2017-05-08T17:43:09Z

One hesitates to ever provide examples because inevitably one hears things like "not writing methods that accept subtypes of an ADT". That's not how it arises. That's how it's demonstrated in a github issue in an easy-to-reproduce way. It's usually the type of a pattern, and you can't even use a type ascription to upcast a nested binding because the pattern matcher will unify the upcast with the stronger type and use the stronger type.

That said, if we do this, why wouldn't we add variance all over the place in cats?

You should at least consider it.

paulp · 2017-05-08T17:44:33Z

@johnynek Can you point me to the more-than-trivial usage? It wasn't a rhetorical device, I would really like to see in practice. Maybe my objections would evaporate.

johnynek · 2017-05-08T17:46:29Z

@paulp I can't, it is closed source. Can you replace Show with Order, which we use all the time, and illustrate to me why Order has to be contravariant to be at all useful?

paulp · 2017-05-08T17:53:49Z

If this seems useful enough to you then I guess we have established where the difference of opinion lies.

scala> implicitly[Order[List[Int]]]
res7: cats.Order[List[Int]] = cats.kernel.instances.ListOrder@c02a72a

scala> implicitly[Order[Seq[Int]]]
<console>:18: error: could not find implicit value for parameter e: cats.Order[Seq[Int]]
       implicitly[Order[Seq[Int]]]
                 ^

scala> implicitly[Order[Vector[Int]]]
res9: cats.Order[Vector[Int]] = cats.kernel.instances.VectorOrder@17d7cf3e

scala> implicitly[Order[scala.collection.immutable.LinearSeq[Int]]]
<console>:18: error: could not find implicit value for parameter e: cats.Order[scala.collection.immutable.LinearSeq[Int]]
       implicitly[Order[scala.collection.immutable.LinearSeq[Int]]]
                 ^

So basically there is an Order if someone has decided it's a popular enough subtype of Seq. As user experiences go I think it leaves something to be desired.

erik-stripe · 2017-05-08T17:59:28Z

So, my first thought is that if we're going to do this for Show we should do it for Eq, PartialOrder, Order, etc. (i.e. any type class whose type parameters only appear in contravariant position). I don't think doing this in one case but not the others is a good idea.

In the past, we've avoided doing this due to the (IMO buggy) way contravariance and specificity interact. See SI-2509 and its related links for examples of why this can be a problem. I know @paulp had a language change that fixed these issues, but it didn't make it into the language.

Here's a restatement of the issue from the ticket:

package contra

class Dog
class Puppy extends Dog

trait Observer[-A] {
  def observe(a: A): String
}

object Observer {
  implicit val dogObserver: Observer[Dog] =
    new Observer[Dog] { def observe(d: Dog) = "saw a dog" }

  implicit val puppyObserver: Observer[Puppy] =
    new Observer[Puppy] { def observe(p: Puppy) = "saw a puppy" }
}

object Test {
  implicit class Observed[A](a: A)(implicit o: Observer[A]) {
    def observed: String = o.observe(a)
  }

  val dog = new Dog {}
  val pup = new Puppy {}

  def main(args: Array[String]): Unit = {
    println(Observer.dogObserver.observe(dog))
    println(Observer.puppyObserver.observe(pup))
    println(dog.observed)
    println(pup.observed)
    println(implicitly[Observer[Dog]].observe(dog))
    println(implicitly[Observer[Puppy]].observe(pup))
  }
  // output:
  //saw a dog
  //saw a puppy
  //saw a dog
  //saw a dog
  //saw a dog
  //saw a dog
}

erik-stripe · 2017-05-08T18:07:17Z

One side-effect of this is that if Order was contravariant and we implemented Order[Iterable[A]] then it would be impossible for any subclasses of Iterable to override that behavior for their own type. (A mitigating factor here is that Array is not a subtype of Iterable, which is the best example of a type where I would not want to be forced into an iterator-based implementation.)

paulp · 2017-05-08T18:12:14Z

No, you restructure things. Using companion scope does make it impossible to use contravariance, but if the implicits are in the package object then they can be masked by name when desired. Stacking traits lets you specify specificity rather than letting the compiler make choices.

A reasonably complete example.

paulp · 2017-05-08T18:18:27Z

There are a few more arrows in the quiver as well, which maybe could be put to good effect even if you don't want to introduce contravariance. See the last method at the above link:

implicit def showConsecutive[A, CC[X] <: Consecutive[X]]: Show[CC[_ <: A]] = by(_.in)

In other words, you can in principle print an A with a Show[_ <: A], you don't need a Show[A]. But that won't happen by accident, i.e. Vector and List would be ambiguous for Seq unless you took steps to make it otherwise.

erik-stripe · 2017-05-08T18:24:35Z

OK, right, I do agree that if we changed the way people are expected to use implicits we could mitigate these problems. It's a bigger task, but not impossible.

I personally would like to solve the issues you raise around Seq, IndexedSeq, Iterable, and so on, but I wonder if we couldn't do this without variance for two reasons:

The invariant type classes (e.g. Monoid) will still have this issue. It would be good to support Monoid[Iterable] as well as Show[Iterable] (possibly via some kind of free construction).
My experience is that one of the places people have the most trouble with is managing implicit scopes and imports. As long as it seems sustainable I'd like to maintain a consistent, global set of implicits via cats.instances.all._ which are known (or at least believed) to avoid ambiguities and conflicts with each other.

I know in the past folks have pushed back against supporting the collection interfaces which support possibly mutable collections (e.g. Seq). I've never found these criticisms particularly compelling -- as long as we can ensure that our type class instances are law-abiding, and that the resulting values work correctly when used in referentially-transparent ways, I'm willing to live and let live.

What do you think @paulp? I'm happy to try to put together a PR that demonstrates how to cover the highlights of the collections library. Does this seem like a fool's errand?

paulp · 2017-05-08T18:54:14Z

All non-final collections are possibly mutable (as are some of the final and supposedly immutable ones, like List.) Some just invite it more via their interfaces than others. It's easy to make a mutable instance of scala.collection.immutable.Seq. So I don't think there is an effective principled stance to take there.

However, Seq too was just an example - I figured it has to rub people the wrong way to have a finite list of manually created instances all doing the same thing. You get both code duplication and arbitrarily incomplete coverage of the problem space. If you find a way to make Seq and friends work uniformly which leaves you with less code instead of more, that'll be an improvement but not so much a solution from where I sit, because every ADT offers a problem isomorphic to the Option/Some issue.

Independently of contravariant Show, I think there's a lot to be said for not using companion scope in a library. People can do that at the leaves because they can change it when there's an issue. But when you ship things in companion scope, people are stuck.

milessabin · 2017-05-08T18:57:18Z

I don't know how, when or why SI-2509 was assigned to me, but it definitely looks like it would be a good candidate for a fix in Typelevel Scala ... would there be any appetite for it if it happened?

milessabin · 2017-05-08T18:58:43Z

Hmm ... apparently I reopened it and assigned it to myself about a year ago ...

edmundnoble · 2017-05-08T19:04:00Z

The user issue here, from my perspective, is the show interpolator (and syntax/ops on naturally-contravariant type classes) themselves being almost useless without contravariance. The compiler will not let you make, for example, the Shown.mat implicit materializer use a subtype of the type which actually has the Show instance and call Contravariant.narrow, because there's too much type slack for the implicit to be found by the compiler.

erik-stripe · 2017-05-08T19:05:25Z

@edmundnoble Do you think you could bridge the gap with contravariance in the interpolator (or implicits leading up to it) instead of the type class?

@milessabin Yes! Although it's a language change, so we'd definitely want to advertise it, hide it behind -Y, etc.

paulp · 2017-05-08T19:07:51Z

@edmundnoble Yes, I should have specified that it was that battle which led psp-std to where it is today. Finding something like OrderOps[A] for an arbitrary A which has an Order, when the actual Order is for a supertype of the inferred type of A, is something I never accomplished consistently until I used contravariance.

edmundnoble · 2017-05-08T19:08:09Z

@erik-stripe I tried that first; that's most of my point. It's not possible with implicit search working the way it does, and in general it doesn't work for any ops class.

@milessabin Pretty pretty please 👍 👍

ceedubs · 2017-05-12T13:42:08Z

I think that I'm 👎 on this, at least with contravariant implicit resolution being what it is in scala right now. My experience with variance (even covariance) in type classes has overwhelmingly been negative.

As @edmundnoble said, there's certainly a tradeoff. But as I see it, the two main things that you gain are:

At the user-level, saving some characters by not needing to explicitly annotate a supertype
At the library-level, not needing to set up some "forwarder" instances such as the Show[BitSet]/Show[Set[Int]] instances.

These two conveniences seem to come at a pretty high complexity cost. If you control all of the relevant instances, then you can use stack trait hierarchies to order them, but doesn't that model break down if some instances are coming from different places?

Also one could argue that the second benefit listed above can at times be an anti-feature, as you may not necessarily want BitSet to automatically pick up the Show[Set[Int]] instance.

edmundnoble · 2017-05-13T05:44:50Z

@ceedubs Two notes:

You are prohibited by contravariance with a-la-carte instances objects from even being able to provide an instance for BitSet given one for Set[A: Show] in scope. Even if you want to. Cause contravariant type classes do break down if instances come from different sources.
I don't think it's valid not to want a BitSet to be able to pick up an instance for Set[A: Show]; just ascribe the type Set[Int] (and perhaps even pass it to a method, which is completely unaware of the set's runtime type!) and you get that instance anyway.

My primary point is this: the show interpolator does not work with subtyping, and Scala's very limited type inference is based on subtyping. And implicits just plain don't work enough with type inference to change that without making Show contravariant. This also makes using the show interpolator much harder to motivate, seeing as the s interpolator does not need to worry about ascribing types. If the inability to do induction over Show instances because of broken implicit contravariance outweighs that concern, then I agree that this PR shouldn't be merged. I am interested to know users' needs and uses practically with show, so that we can make this tradeoff from an informed position.

paulp · 2017-05-13T05:58:24Z

I guess I should also point out that if Show is contravariant, you can easily have it both ways. The reverse is not true.

trait Show[-A] { def show(x: A): String }
trait IShow[A] extends Show[A]

Also, it bears repeating that you CAN reconcile matters even "if some instances are coming from different places" regardless of the implicit selection algorithm, because the names are in scope and you can manage them by name.

val someImplicitIDontLike = null 
implicit def someOtherImplicit[A <: TighterBound](x: A) = cats.someOtherImplicit(x)

which though baroque because that's all scala gives us is a better situation than you have when the library has consumed the whole companion scope and you can't influence that at all.

paulp · 2017-05-13T06:02:23Z

Oh yes, here's a real life example of using an invariant type alias on a contravariant type to make implicit resolution succeed where it would otherwise fail.

/** Scala, so aggravating.
 *  [error] could not find implicit value for parameter equiv: Eq[A => Bool]
 *  The parameter can be given explicitly, it just won't be found unless the
 *  function type is invariant. The same issue arises with intensional sets.
 */
type InvariantPredicate[A] = A => Bool

edmundnoble · 2017-05-13T06:27:36Z

Despite the fact that you can reconcile Show (and other type classes') instances manually, it's incredibly annoying and verbose and most users would never have to do this otherwise, because of the a la carte import scheme. Offering these a la carte imports doesn't seem possible using trait-based implicit prioritization; users would instead be forced to reconcile instances manually in all cases that they would otherwise use a la carte imports to avoid ambiguities. Which means shadowing a lot of implicits.

However; I am having second thoughts about cats providing any type class instances in companion objects because of the modularity problem you mentioned. Why even offer a la carte imports for some instances if the user can't control instances for datatypes from cats itself?

Edit: Can we not, then, use ContravariantShow for just the show interpolator, private it and only expose InvariantShow for actual implicit search?

paulp · 2017-05-13T07:32:24Z

I find it much less annoying than the alternatives, personally. You do it once, in your package object, and you get the same semantics everywhere in the package. You don't have to do a la carte import babysitting on a file by file basis. And there's no reason to think people wouldn't mostly be able to use the defaults anyway.

Also, I put all the implicits in a trait which can itself be mixed into a package object. Then whatever you put in your package object has the upper hand from the get-go.

You can maybe make the interpolator work with a private ContravariantShow, but I'm not sure what the motivation is to make it private. Given that the interface is not a big threat to change, it's rather paternalistic to act like the show interpolator is the only use case which anyone could ever have for contravariant Show.

edmundnoble · 2017-05-14T05:35:12Z

My motivation for making ContravariantShow private was that under cats' current import scheme instances can only be provided and manipulated for InvariantShow. Though now that I think about it, from the user perspective methods that use Show and want to work with subtyping will need to ask for ContravariantShow anyway. Making ContravariantShow sealed seems more appropriate. My only concern is letting users create both ContravariantShow and InvariantShow instances, which is unnecessarily confusing because the cats instance induction methods will use InvariantShow and not accept a plain ContravariantShow and from the instance creation standpoint ContravariantShow seems entirely useless. I'll update this PR to try it, because it seems to be better in all cases.

Also, when it comes to changing cats' import scheme, I recommend opening a new issue to make a more focused case that this can and should work with cats, as well as the API changes that would be required and the use-cases allowed and disallowed by it, seeing as it's not strictly necessary if we go with this IShow/CShow scheme.

edmundnoble · 2017-06-30T07:43:20Z

So now, this PR does two things.

It introduces a base trait ContravariantShow[-T] which trait Show[T] inherits from.
It makes the show string interpolator conversion use ContravariantShow as an implicit parameter, rather than Show.

This makes the show interpolator work conveniently (contravariantly), yet shields the rest of the related code from the complications of implicit induction using contravariant implicits.

kailuowang · 2017-06-30T17:29:34Z

it's no longer API breaking right?

johnynek · 2017-06-30T17:51:56Z

core/src/main/scala/cats/Show.scala

@@ -27,7 +32,7 @@ object Show {

  final case class Shown(override val toString: String) extends AnyVal
  object Shown {
-    implicit def mat[A](x: A)(implicit z: Show[A]): Shown = Shown(z show x)
+    implicit def mat[A](x: A)(implicit z: ContravariantShow[A]): Shown = Shown(z show x)


I wonder if we can get the test to pass by asking for Show[_ >: A] which is effectively what this trick is doing, and not having to introduce the new type. I think we can (at least on 2.12).

Unfortunately type inference looks to be just slightly shy, like it was when I asked for [A, B >: A](x: A)(implicit z: Show[B]): it never converts anything to a Shown.

[error] /Users/edmund/workspace/scala/cats/tests/src/test/scala/cats/tests/ShowTests.scala:29: type mismatch; [error] found : TimeOfDay [error] required: cats.Show.Shown [error] assertResult("Good morning, Whiskers!")(show"Good $tod, $cat!") [error] ^ [error] /Users/edmund/workspace/scala/cats/tests/src/test/scala/cats/tests/ShowTests.scala:29: type mismatch; [error] found : Cat [error] required: cats.Show.Shown [error] assertResult("Good morning, Whiskers!")(show"Good $tod, $cat!") [error] ^ [error] /Users/edmund/workspace/scala/cats/tests/src/test/scala/cats/tests/ShowTests.scala:31: type mismatch; [error] found : TimeOfDay [error] required: cats.Show.Shown [error] assertResult("Good morning, Whiskers!")(show"Good $tod, ${List(cat).head}!") [error] ^ [error] /Users/edmund/workspace/scala/cats/tests/src/test/scala/cats/tests/ShowTests.scala:31: type mismatch; [error] found : Cat [error] required: cats.Show.Shown [error] assertResult("Good morning, Whiskers!")(show"Good $tod, ${List(cat).head}!") [error] ^ [error] /Users/edmund/workspace/scala/cats/tests/src/test/scala/cats/tests/ShowTests.scala:46: type mismatch; [error] found : Cat [error] required: cats.Show.Shown [error] assertResult("Good morning, Whiskers!")(show"Good morning, $cat!") [error] ^ [error] 5 errors found

when I did a simple test in the repl this worked when I did implicitly[Foo[_ >: A]] did you try that form (not putting a specific type)? you never know with scala which minor translation of the code will work.

An implicit in the repl is like you've imported it. His is in the Shown companion object.

right. That clarifies it for me.

edmundnoble · 2017-07-01T06:56:37Z

@kailuowang I believe it is not API breaking any longer.

edmundnoble · 2017-07-05T05:27:01Z

@johnynek Any more thoughts? Thanks for the review.

@ceedubs What's your opinion on this PR as it stands now?

johnynek · 2017-07-05T19:41:13Z

I'm on board. This is an interesting trick to put in the bag of scalahacks.

👍

kailuowang · 2017-07-07T12:21:47Z

merged with 3 sign-offs

edmundnoble added the Source Breaking label May 5, 2017

edmundnoble requested review from kailuowang and mpilquist May 5, 2017 03:20

stew added the in progress label May 5, 2017

kailuowang reviewed May 5, 2017

View reviewed changes

edmundnoble force-pushed the contravariant-show branch from 17bf536 to 20add21 Compare May 5, 2017 03:40

kailuowang reviewed May 8, 2017

View reviewed changes

edmundnoble force-pushed the contravariant-show branch 2 times, most recently from 732d4d5 to e723099 Compare June 28, 2017 07:31

edmundnoble changed the title ~~Make Show contravariant~~ Make Show inherit from a contravariant base trait Jun 30, 2017

johnynek reviewed Jun 30, 2017

View reviewed changes

kailuowang added Binary Breaking and removed Source Breaking labels Jul 1, 2017

kailuowang approved these changes Jul 1, 2017

View reviewed changes

Make Show contravariant without breaking things

59822c1

edmundnoble force-pushed the contravariant-show branch from e723099 to 59822c1 Compare July 7, 2017 07:32

edmundnoble requested a review from peterneyens July 7, 2017 08:14

peterneyens approved these changes Jul 7, 2017

View reviewed changes

kailuowang merged commit cb84fd4 into typelevel:master Jul 7, 2017

kailuowang added this to the 1.0.0-MF milestone Jul 7, 2017

Make Show inherit from a contravariant base trait #1649

Make Show inherit from a contravariant base trait #1649

Conversation

edmundnoble commented May 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented May 5, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

johnynek commented May 8, 2017

paulp commented May 8, 2017

johnynek commented May 8, 2017

paulp commented May 8, 2017

paulp commented May 8, 2017

johnynek commented May 8, 2017

paulp commented May 8, 2017

erik-stripe commented May 8, 2017

erik-stripe commented May 8, 2017 • edited Loading

paulp commented May 8, 2017

paulp commented May 8, 2017

erik-stripe commented May 8, 2017 • edited Loading

paulp commented May 8, 2017

milessabin commented May 8, 2017

milessabin commented May 8, 2017

edmundnoble commented May 8, 2017

erik-stripe commented May 8, 2017 • edited Loading

paulp commented May 8, 2017

edmundnoble commented May 8, 2017 • edited Loading

ceedubs commented May 12, 2017

edmundnoble commented May 13, 2017

paulp commented May 13, 2017

paulp commented May 13, 2017

edmundnoble commented May 13, 2017 • edited Loading

paulp commented May 13, 2017

edmundnoble commented May 14, 2017

edmundnoble commented Jun 30, 2017 • edited Loading

kailuowang commented Jun 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmundnoble commented Jul 1, 2017

edmundnoble commented Jul 5, 2017

johnynek commented Jul 5, 2017

kailuowang commented Jul 7, 2017

codecov-io commented May 5, 2017 •

edited

Loading

erik-stripe commented May 8, 2017 •

edited

Loading

erik-stripe commented May 8, 2017 •

edited

Loading

erik-stripe commented May 8, 2017 •

edited

Loading

edmundnoble commented May 8, 2017 •

edited

Loading

edmundnoble commented May 13, 2017 •

edited

Loading

edmundnoble commented Jun 30, 2017 •

edited

Loading