Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Files are not logged in txn-log for savemode ErrorIfExists #16

Closed
ShahSunny opened this issue Apr 28, 2019 · 1 comment
Closed

Files are not logged in txn-log for savemode ErrorIfExists #16

ShahSunny opened this issue Apr 28, 2019 · 1 comment

Comments

@ShahSunny
Copy link

Files are not getting logged in the txn-log for savemode, ErrorIfExists.

  • This can hamper the recovery from future partial writes/failed txns.
  • Partial/failed writes can also become visible in this case.
@mukulmurthy
Copy link
Collaborator

Hi Sunny, can you provide a bit more detail or a repro case?

Partial or failed writes should never be committed to Delta's transaction log, and so the files should never logically be part of the table.

LantaoJin added a commit to LantaoJin/delta that referenced this issue Mar 24, 2020
@LantaoJin
[CARMEL-2255] Add 'CONVERT TO DELTA VACUUM', 'VACUUM AUTO RUN', 'SHOW…
669ce54 
… DELTAS' commands (delta-io#16)
LantaoJin added a commit to LantaoJin/delta that referenced this issue Mar 12, 2021
@LantaoJin
[CARMEL-4270] UpdateCommand and DeleteCommand should skip to optimize…
933235e 
… subquery in their condition (delta-io#16)
tdas pushed a commit to tdas/delta that referenced this issue May 31, 2023
@zsxwing
Set 'path' in the storage properties as Spark is using it for its dat…
119e67e 
…a sources (delta-io#16)

Spark reads the `path` option from the storage properties. We should set it when creating a table in Hive so that the table can be read by Spark as well.

I also removed `delta.table.path` from the table properties since we can get it from the `location` property.
andreaschat-db added a commit to andreaschat-db/delta that referenced this issue Apr 23, 2024
@andreaschat-db
# This is a combination of 16 commits.
ad044ef 
# This is the 1st commit message:

flush

# This is the commit message delta-io#2:

flush

# This is the commit message delta-io#3:

First sane version without isRowDeleted

# This is the commit message delta-io#4:

Hack RowIndexMarkingFilters

# This is the commit message delta-io#5:

Add support for non-vectorized readers

# This is the commit message delta-io#6:

Metadata column fix

# This is the commit message delta-io#7:

Avoid non-deterministic UDF to filter deleted rows

# This is the commit message delta-io#8:

metadata with Expression ID

# This is the commit message delta-io#9:

Fix complex views issue

# This is the commit message delta-io#10:

Tests

# This is the commit message delta-io#11:

cleaning

# This is the commit message delta-io#12:

More tests and fixes

# This is the commit message delta-io#13:

Partial cleaning

# This is the commit message delta-io#14:

cleaning and improvements

# This is the commit message delta-io#15:

cleaning and improvements

# This is the commit message delta-io#16:

Clean RowIndexFilter
andreaschat-db added a commit to andreaschat-db/delta that referenced this issue Apr 26, 2024
@andreaschat-db
# This is a combination of 16 commits.
0826e53 
# This is the 1st commit message:

flush

# This is the commit message delta-io#2:

flush

# This is the commit message delta-io#3:

First sane version without isRowDeleted

# This is the commit message delta-io#4:

Hack RowIndexMarkingFilters

# This is the commit message delta-io#5:

Add support for non-vectorized readers

# This is the commit message delta-io#6:

Metadata column fix

# This is the commit message delta-io#7:

Avoid non-deterministic UDF to filter deleted rows

# This is the commit message delta-io#8:

metadata with Expression ID

# This is the commit message delta-io#9:

Fix complex views issue

# This is the commit message delta-io#10:

Tests

# This is the commit message delta-io#11:

cleaning

# This is the commit message delta-io#12:

More tests and fixes

# This is the commit message delta-io#13:

Partial cleaning

# This is the commit message delta-io#14:

cleaning and improvements

# This is the commit message delta-io#15:

cleaning and improvements

# This is the commit message delta-io#16:

Clean RowIndexFilter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ShahSunny @mukulmurthy