ticdc redo log seems to use S3 in an inefficient style

[hicqu]

Before asking a question, make sure you have

• Searched existing Stack Overflow questions
• Googled your question
• Searched open and closed GitHub issues
• Read the documentation

What is your question?

The code is in cdc/redo/writer/file.go.

In Writer.Flush, Writer.writeToS3 will be finally called.

func (w *Writer) writeToS3(ctx context.Context, name string) error {
    fileData, err := os.ReadFile(name)
    return cerror.WrapError(cerror.ErrS3StorageAPI, w.storage.WriteFile(ctx, filepath.Base(name), fileData))
}

So if writeToS3 is called multiple times on one local file, same content will be put to S3 multiple times. It will waste lots of network traffic resource.

And, writeToS3 is called with a temp path. After a S3 log file is rotated, it will be renamed with Write.renameInS3(path.tmp, path). However S3 doesn't have a built-in rename API, so the rename operation is very expensive. We can check it here.

Please help me to confirm whether my investigation is correct or not, thanks!

[zhaoxinyu]

closed by #6262