You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have read through the quick start and installation sections of the README.
Info
Info
Value
Operating System
rhel7 and windows
Geni Version
0.0.38
JDK
11.0.10
Spark Version
3.1.2
Problem / Steps to reproduce
If zero-one.geni.core has been required, it creates a default spark session which impacts the behavior of calling g/create-spark-session
Specifically, geni.core loads geni.spark-context which loads geni.defaults, which creates a spark session in an atom which should probably be a delay
(defs (g/create-spark-session {:app-name “foo”}))
(g/spark-conf s)
; => {…:spark.app.name Geni app…}; which is the wrong name
if requiring zero-one.geni.spark directly instead (as g), the spark session is correctly configured.
The incorrect behaviour takes effect if core is required at any point before creating the session, so it is a bit problematic. As above, maybe replacing the default with a delay will be sufficient to avoid this.
Thanks for your work on this library!
The text was updated successfully, but these errors were encountered:
Info
Problem / Steps to reproduce
If
zero-one.geni.core
has been required, it creates a default spark session which impacts the behavior of callingg/create-spark-session
Specifically,
geni.core
loadsgeni.spark-context
which loadsgeni.defaults
, which creates a spark session in anatom
which should probably be adelay
if requiring
zero-one.geni.spark
directly instead (asg
), the spark session is correctly configured.The incorrect behaviour takes effect if core is required at any point before creating the session, so it is a bit problematic. As above, maybe replacing the default with a delay will be sufficient to avoid this.
Thanks for your work on this library!
The text was updated successfully, but these errors were encountered: