Skip to content

Commit

Permalink
add link to transactions page
Browse files Browse the repository at this point in the history
  • Loading branch information
avriiil committed Jul 11, 2024
1 parent fd8fd8e commit 81e15d8
Showing 1 changed file with 8 additions and 8 deletions.
16 changes: 8 additions & 8 deletions docs/how-delta-lake-works/architecture-of-delta-table.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ tmp/some-table
└── 00000000000000000000.json
```

The Parquet file stores the data that was written. The `_delta_log` directory stores metadata about the transactions. Let's inspect the `_delta_log/00000000000000000000.json` file.
The Parquet file stores the data that was written. The `_delta_log` directory stores metadata about the transactions. Let's inspect the `_delta_log/00000000000000000000.json` file.

```json
{
Expand Down Expand Up @@ -78,9 +78,9 @@ The Parquet file stores the data that was written. The `_delta_log` directory s

The transaction log file contains the following information:

* the files added to the Delta table
* schema of the files
* column level metadata including the min/max value for each file
- the files added to the Delta table
- schema of the files
- column level metadata including the min/max value for each file

Create another pandas DataFrame and append it to the Delta table to see how this transaction is recorded.

Expand Down Expand Up @@ -194,18 +194,18 @@ Here are the contents of the `_delta_log/0002.json` file:
}
```

This transaction adds a data file and marks the two exising data files for removal. Marking a file for removal in the transaction log is known as "tombstoning the file" or a "logical delete". This is different from a "physical delete" which actually removes the data file from storage.
This transaction adds a data file and marks the two exising data files for removal. Marking a file for removal in the transaction log is known as "tombstoning the file" or a "logical delete". This is different from a "physical delete" which actually removes the data file from storage.

## How Delta table operations differ from data lakes

Data lakes consist of data files persisted in storage. They don't have a transaction log that retain metadata about the transactions.
Data lakes consist of data files persisted in storage. They don't have a transaction log that retain metadata about the transactions.

Data lakes perform transactions differently than Delta tables.

When you perform an overwrite tranasction with a Delta table, you logically delete the exiting data without physically removing it.

Data lakes don't support logical deletes, so you have to physically delete the data from storage.

Logical data operations are safer because they can be rolled back if they don't complete successfully. Physically removing data from storage can be dangerous, especially if it's before a transaction is complete.
Logical data operations are safer because they can be rolled back if they don't complete successfully. Physically removing data from storage can be dangerous, especially if it's before a transaction is complete.

We're now ready to look into Delta Lake ACID transactions in more detail.
We're now ready to look into [Delta Lake ACID transactions](../how-delta-lake-works/delta-lake-acid-transactions.md) in more detail.

0 comments on commit 81e15d8

Please sign in to comment.