Investigate fix/workaround for Scala import issue

[jpallas]

In #7040 I included a code snippet with the comment

// I do not understand why this import is necessary

I did some further investigation, and the answer seems to be Scala issue SI-9881. This issue was fixed in the 2.12 line but the fix was not back-ported to 2.11 (see this pull request and this issue on the status of a back port—the question is whether there will be any more 2.11 releases.)

One reason for the lack of urgency is that the Spark project did their own back-port/workaround. This is covered in SPARK-22393.

The net result is code that works in the Spark shell may not work in the Scala kernel. The Spark back-port is pretty small, but it is intimate with the REPL, so it is probably fragile. (But the Spark team is in the same boat.) It is worth looking into whether the same retrofit will work with the Scala kernel.

[scottdraves]

Thanks Joe that's really interesting. I don't see why we couldn't do the same.

[jpallas]

Urf. I tried stealing the Spark code, and it goes in cleanly. But they used a different test case than the Scala code uses, and it turns out for whatever reason that the Spark code doesn't actually cover everything the Scala test case checks.

So, if the goal is anything that works in Scala should work in the kernel, it fails. If the goal is anything that works in spark-shell should work in the kernel, it succeeds.

Also, there's a meta-question about what to do with the license/copyright headers on the files that are copied-with-minor-changes from Spark.

[jpallas]

Added a comment at SPARK-22393 documenting the problem with the retrofitted solution. I'm hoping someone who understands the import-handling code might figure out how to fix it, although I did not reopen the issue.

[jpallas]

I figured out that there was one line missing from the Spark retrofit. Adding that line makes the test pass. I attached that information to the Spark issue, in hopes of getting some attention from the author of the retrofit.

With this change, in rare instances code that works in the Scala kernel might fail in the Spark shell, but code that works in the Spark 2.3 shell should work in the Scala kernel.

[scottdraves]

We are lucky to have your attention Joe! will check this when I get back to my desk (traveling for 2 days).