Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deltatable does not recognise the newly created spark table #392

Closed
spachari opened this issue Apr 19, 2020 · 3 comments
Closed

Deltatable does not recognise the newly created spark table #392

spachari opened this issue Apr 19, 2020 · 3 comments

Comments

@spachari
Copy link

I created a table like this and inserted some data

spark.sql(s"create table if not exists test " +
"(key String," +
"name String," +
"address String," +
"inserted_at TIMESTAMP) " +
s" using delta LOCATION 's3://test/user/'")
I can view the table via

spark.table("test").show()
But when I do

DeltaTable.forPath(spark,"s3://test/user/" ).toDF.show(false)
I cannot see the data. But when i try this method

DeltaTable.isDeltaTable("s3://test/user/")
it is true. Can anyone please explain what I am missing?

Further, when I want to do a merge operation, I am getting this error.

[error] !
[error] java.lang.UnsupportedOperationException: null (DeltaTable.scala:639)
[error] io.delta.tables.DeltaTable$.forPath(DeltaTable.scala:639)

@rahulsmahadev
Copy link
Collaborator

Hey @spachari have you tried using DeltaTable.forName(spark, "test")

@tdas
Copy link
Contributor

tdas commented Apr 21, 2020

@spachari are you using Databricks Delta or OSS delta? I would expect this error to happen with OSS Delta because OSS does not support Hive metastore defined tables and related DDL commands yet. It's blocked on Apache Spark 3.0 release. See #85

@tdas
Copy link
Contributor

tdas commented Apr 29, 2020

@spachari any updates on this issue?

@tdas tdas closed this as completed May 1, 2020
tdas pushed a commit to tdas/delta that referenced this issue May 31, 2023
* ReadmeUpdate_Limitations - add limitations to README.md

Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com>

* ReadmeUpdate_Limitations - chanegs after code review

Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com>

* ReadmeUpdate_Limitations - changes after code review

Signed-off-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com>

* Update README.md

Co-authored-by: Krzysztof Chmielewski <krzysztof.chmielewski@getindata.com>
Co-authored-by: Scott Sandre <59617782+scottsand-db@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants