GzipFile.write should be buffered

[rhpvorderman]

BPO	45387
Nosy	@marcelm, @rhpvorderman

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-10-06.08:42:34.497>
labels = ['3.8', '3.9', '3.10', 'performance', '3.7', 'library', '3.11']
title = 'GzipFile.write should be buffered'
updated_at = <Date 2021-10-06.08:59:49.938>
user = 'https://github.com/rhpvorderman'

bugs.python.org fields:

activity = <Date 2021-10-06.08:59:49.938>
actor = 'marcelm'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-10-06.08:42:34.497>
creator = 'rhpvorderman'
dependencies = []
files = []
hgrepos = []
issue_num = 45387
keywords = []
message_count = 1.0
messages = ['403289']
nosy_count = 2.0
nosy_names = ['marcelm', 'rhpvorderman']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue45387'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-89550: Buffer GzipFile.write to reduce execution time by ~15% #101251

[rhpvorderman]

Please consider the following code snippet:

    import gzip
    import sys

    with gzip.open(sys.argv[1], "rt") as in_file_h:
        with gzip.open(sys.argv[2], "wt", compresslevel=1) as out_file_h:
            for line in in_file_h:
                # Do processing on line here
                modified_line = line
                # End processing
                out_file_h.write(modified_line)

This is very slow, due to write being called for every line. This is the current implementation of write:

cpython/Lib/gzip.py

Line 272 in c379bc5

def write(self,data):

It:

• Checks if the file is not closed
• Checks if the correct mode is set
• Checks if the file is not closed (again, but in a different way)
• Checks if the data is bytes, bytearray or something that supports the buffer protocol
• Gets the length
• Compresses the data
• updates the size and offset
• updates the checksum

Doing this for every line written is very costly and creates a lot of overhead in Python calls. We spent a lot of time in Python and a lot less in the fast C zlib code that does the actual compression.

This problem is already solved on the read side. A _GzipReader object is used for reading. This is put in an io.BufferedReader which is used as the underlying buffer for GzipFile.read. This way, lines are read quite fast from a GzipFile without the checksum etc. being updated on every line read.

A similar solution should be written for write.
I volunteer (I have done some other work on gzip.py already), although I cannot give an ETA at this time.

[hugovk]

A PR has been merged, thanks for the report!