Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hi, we are facing the same issue **'module not found: io.delta#delta-core_2.12;1.0.0**' and we have **spark-3.1.2-bin-hadoop3.2** #771

Closed
kc-1891 opened this issue Aug 31, 2021 · 5 comments

Comments

@kc-1891
Copy link

kc-1891 commented Aug 31, 2021

Hi, we are facing the same issue 'module not found: io.delta#delta-core_2.12;1.0.0' and we have spark-3.1.2-bin-hadoop3.2
Any help on how do we resolve this issue and run the below command successfully?
pyspark --packages io.delta:delta-core_2.12:1.0.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

image

Originally posted by @kumaran-chandrababu in #63 (comment)

@zsxwing
Copy link
Member

zsxwing commented Aug 31, 2021

@kumaran-chandrababu Looks like it's because repo1.maven.org blocked your ip address. Could you try google mirror such as

pyspark --packages io.delta:delta-core_2.12:1.0.0 --repositories https://maven-central.storage-download.googleapis.com/maven2/ --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

@kc-1891
Copy link
Author

kc-1891 commented Sep 1, 2021

@kumaran-chandrababu Looks like it's because repo1.maven.org blocked your ip address. Could you try google mirror such as

pyspark --packages io.delta:delta-core_2.12:1.0.0 --repositories https://maven-central.storage-download.googleapis.com/maven2/ --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

Hi, Thanks for you reply, I tried that too but still not working as per below. I've attached the complete screenshot of exception below.

image
image

@zsxwing
Copy link
Member

zsxwing commented Sep 4, 2021

Does your machine have the internet access? Could you try to access the links in the logs on the same machine manually?

@kc-1891
Copy link
Author

kc-1891 commented Sep 7, 2021

Does your machine have the internet access? Could you try to access the links in the logs on the same machine manually?

Yes do we have the internet access and the links here are properly showing up in the browser. As a workaround, we now manually downloaded the Jars and Pom files locally and setting up the spark config via pySpark code, so delta works fine now for reading/ writing into delta locally., however, when we try to connect to Azure Data Lake Gen2 as per the delta documentation, we are now getting this exception as below. Kindly assist.

image

image

image

@zsxwing
Copy link
Member

zsxwing commented Oct 4, 2021

Closing this as the delta jar loading issue is resolved. The Azure issue is tracked in #777

@zsxwing zsxwing closed this as completed Oct 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants