Skip to content

Commit

Permalink
Speed up ZODB.blob.BushyLayout.oid_to_path on Python 3 (#161)
Browse files Browse the repository at this point in the history
Profiling (zodb/zodbshootout#32) showed that
this method was the only blob-related method that showed up in a test
of creating blobs, other than those that actually performed IO.

With this change its total and cumulative time drops from 0.003/0.004
to 0.001/0.002 in a small benchmark. Blobs created per second shows a
small but consistent improvement.

Before:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      100    0.005    0.000    0.005    0.000 {built-in method rename}
      100    0.004    0.000    0.004    0.000 {function BlobFile.close at 0x1080d3a60}
      200    0.003    0.000    0.004    0.000 blob.py:576(oid_to_path)
      101    0.003    0.000    0.003    0.000 {built-in method mkdir}
      100    0.002    0.000    0.002    0.000 blob.py:333(__init__)
      402    0.002    0.000    0.005    0.000 {method 'dump' of '_pickle.Pickler' objects}
        1    0.002    0.002    0.034    0.034 Connection.py:553(_store_objects)
      201    0.002    0.000    0.002    0.000 {built-in method stat}
     5633    0.001    0.000    0.002    0.000 {built-in method isinstance}

After:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      100    0.005    0.000    0.005    0.000 {built-in method rename}
      101    0.005    0.000    0.005    0.000 {built-in method mkdir}
      100    0.004    0.000    0.004    0.000 {function BlobFile.close at 0x10636aa60}
      402    0.002    0.000    0.005    0.000 {method 'dump' of '_pickle.Pickler' objects}
      100    0.002    0.000    0.002    0.000 blob.py:333(__init__)
        1    0.002    0.002    0.035    0.035 Connection.py:553(_store_objects)
      201    0.002    0.000    0.002    0.000 {built-in method stat}
     4033    0.001    0.000    0.001    0.000 {built-in method isinstance}
   ....
      200    0.001    0.000    0.002    0.000 blob.py:576(oid_to_path)
  • Loading branch information
jamadden authored and jimfulton committed Apr 14, 2017
1 parent 4b500e7 commit 3580ddd
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 10 deletions.
2 changes: 1 addition & 1 deletion CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
5.2.4 (unreleased)
==================

- Nothing changed yet.
- Optimize getting the path to a blob file.


5.2.3 (2017-04-11)
Expand Down
21 changes: 12 additions & 9 deletions src/ZODB/blob.py
Original file line number Diff line number Diff line change
Expand Up @@ -574,18 +574,21 @@ class BushyLayout(object):
r'(0x[0-9a-f]{1,2}\%s){7,7}0x[0-9a-f]{1,2}$' % os.path.sep)

def oid_to_path(self, oid):
directories = []
# Create the bushy directory structure with the least significant byte
# first
for byte in ascii_bytes(oid):
if isinstance(byte,INT_TYPES): # Py3k iterates byte strings as ints
hex_segment_bytes = b'0x' + binascii.hexlify(bytes([byte]))
hex_segment_string = hex_segment_bytes.decode('ascii')
else:
hex_segment_string = '0x%s' % binascii.hexlify(byte)
directories.append(hex_segment_string)
oid_bytes = ascii_bytes(oid)
hex_bytes = binascii.hexlify(oid_bytes)
assert len(hex_bytes) == 16

directories = [b'0x' + hex_bytes[x:x+2]
for x in range(0, 16, 2)]

return os.path.sep.join(directories)
if bytes is not str: # py3
sep_bytes = os.path.sep.encode('ascii')
path_bytes = sep_bytes.join(directories)
return path_bytes.decode('ascii')
else:
return os.path.sep.join(directories)

def path_to_oid(self, path):
if self.blob_path_pattern.match(path) is None:
Expand Down

0 comments on commit 3580ddd

Please sign in to comment.