Skip to content

Commit

Permalink
a nod to copy destinations
Browse files Browse the repository at this point in the history
  • Loading branch information
cgardens committed Jul 21, 2021
1 parent 94208dc commit e8b9fd2
Showing 1 changed file with 11 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,17 @@
* partial success) or 0 state messages (in the case where the onClose step was never reached or did
* not complete without exception).
* </p>
*
* <p>
* When a record is "flushed" it is moved from the docker container to the destination. By
* convention, it is usually placed in some sort of temporary storage on the destination (e.g. a
* temporary database or file store). The logic in close handles committing the temporary
* representation data to the final store (e.g. final table). In the case of Copy destinations they
* often have additional temporary stores. The common pattern for copy destination is that flush
* pushes the data into cloud storage and then close copies from cloud storage to a temporary table
* AND then copies from the temporary table into the final table. This abstraction is blind to that
* detail as it implementation detail of how copy destinations implement close.
* </p>
*/
public class BufferedStreamConsumer extends FailureTrackingAirbyteMessageConsumer implements AirbyteMessageConsumer {

Expand Down

0 comments on commit e8b9fd2

Please sign in to comment.