Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) #7160

Merged
merged 114 commits into from
Jun 12, 2018
Merged

bpo-33671: efficient zero-copy for shutil.copy* functions (Linux, OSX and Win) #7160

merged 114 commits into from
Jun 12, 2018
+595 −19

Commits on May 22, 2018

  1. have shutil.copyfileobj use sendfile() if possible

    @giampaolo
    giampaolo committed May 22, 2018
    Configuration menu
    Copy the full SHA
    1a72c01 View commit details
    Browse the repository at this point in the history

  2. refactoring: use ctx manager

    @giampaolo
    giampaolo committed May 22, 2018
    Configuration menu
    Copy the full SHA
    77c4bfa View commit details
    Browse the repository at this point in the history

  3. add test with non-regular file obj

    @giampaolo
    giampaolo committed May 22, 2018
    Configuration menu
    Copy the full SHA
    2afa04a View commit details
    Browse the repository at this point in the history

  4. emulate case where file size can't be determined

    @giampaolo
    giampaolo committed May 22, 2018
    Configuration menu
    Copy the full SHA
    542cd17 View commit details
    Browse the repository at this point in the history

  5. reference _copyfileobj_sendfile directly

    @giampaolo
    giampaolo committed May 22, 2018
    Configuration menu
    Copy the full SHA
    3520c6c View commit details
    Browse the repository at this point in the history

  6. add test for offset() at certain position

    @giampaolo
    giampaolo committed May 22, 2018
    Configuration menu
    Copy the full SHA
    050a722 View commit details
    Browse the repository at this point in the history

  7. add test for empty file

    @giampaolo
    giampaolo committed May 22, 2018
    Configuration menu
    Copy the full SHA
    c1fd38a View commit details
    Browse the repository at this point in the history

  8. add test for non regular file dst

    @giampaolo
    giampaolo committed May 22, 2018
    Configuration menu
    Copy the full SHA
    2ab6317 View commit details
    Browse the repository at this point in the history

  9. small refactoring

    @giampaolo
    giampaolo committed May 22, 2018
    Configuration menu
    Copy the full SHA
    dacc3b6 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2018

  1. leave copyfileobj() alone in order to not introduce any incompatibility

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    29d5881 View commit details
    Browse the repository at this point in the history

  2. minor refactoring

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    114c4de View commit details
    Browse the repository at this point in the history

  3. remove old test

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    501c0dd View commit details
    Browse the repository at this point in the history

  4. update docstring

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    41b4506 View commit details
    Browse the repository at this point in the history

  5. update docstring; rename exception class

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    fdb0973 View commit details
    Browse the repository at this point in the history

  6. detect platforms which only support file to socket zero copy

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    64d2bc5 View commit details
    Browse the repository at this point in the history

  7. don't run test on platforms where file-to-file zero copy is not suppo… 

    …rted
    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    3a3c8ef View commit details
    Browse the repository at this point in the history

  8. use tempfiles

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    7861737 View commit details
    Browse the repository at this point in the history

  9. reset verbosity

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    f3eecfd View commit details
    Browse the repository at this point in the history

  10. add test for smaller chunks

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    f67ce57 View commit details
    Browse the repository at this point in the history

  11. add big file size test

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    d457254 View commit details
    Browse the repository at this point in the history

  12. add comment

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    8eb211d View commit details
    Browse the repository at this point in the history

  13. update doc

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    a0fe703 View commit details
    Browse the repository at this point in the history

  14. update whatsnew doc

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    7296147 View commit details
    Browse the repository at this point in the history

  15. update doc

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    d0c3bba View commit details
    Browse the repository at this point in the history

  16. catch Exception

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    2cafd80 View commit details
    Browse the repository at this point in the history

  17. remove unused import

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    bb2a75f View commit details
    Browse the repository at this point in the history

  18. add test case for error on second sendfile() call

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    e5025dc View commit details
    Browse the repository at this point in the history

  19. turn docstring into comment

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    a36a534 View commit details
    Browse the repository at this point in the history

  20. add one more test

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    e9da3fa View commit details
    Browse the repository at this point in the history

  21. update comment

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    9fcc2e7 View commit details
    Browse the repository at this point in the history

  22. add Misc/NEWS entry

    @giampaolo
    giampaolo committed May 24, 2018
    Configuration menu
    Copy the full SHA
    4f32242 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2018

  1. get rid of COPY_BUFSIZE; it belongs to another PR

    @giampaolo
    giampaolo committed May 25, 2018
    Configuration menu
    Copy the full SHA
    24ad25a View commit details
    Browse the repository at this point in the history

  2. update doc

    @giampaolo
    giampaolo committed May 25, 2018
    Configuration menu
    Copy the full SHA
    24d20e6 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2018

  1. expose posix._fcopyfile() for OSX

    @giampaolo
    giampaolo committed May 27, 2018
    Configuration menu
    Copy the full SHA
    7b6e576 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into shutil-osx-copyfile

    @giampaolo
    giampaolo committed May 27, 2018
    Configuration menu
    Copy the full SHA
    b82ddc9 View commit details
    Browse the repository at this point in the history

  3. merge from linux branch

    @giampaolo
    giampaolo committed May 27, 2018
    Configuration menu
    Copy the full SHA
    b62b61e View commit details
    Browse the repository at this point in the history

  4. merge from linux branch

    @giampaolo
    giampaolo committed May 27, 2018
    Configuration menu
    Copy the full SHA
    34e9618 View commit details
    Browse the repository at this point in the history

  5. expose fcopyfile

    @giampaolo
    giampaolo committed May 27, 2018
    Configuration menu
    Copy the full SHA
    6b20902 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2018

  1. arg clinic for the win implementation

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    abf3ecb View commit details
    Browse the repository at this point in the history

  2. convert path type to path_t

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    91e492c View commit details
    Browse the repository at this point in the history

  3. expose CopyFileW

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    e02c69d View commit details
    Browse the repository at this point in the history

  4. fix windows tests

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    73837e2 View commit details
    Browse the repository at this point in the history

  5. release GIL

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    28be4c1 View commit details
    Browse the repository at this point in the history

  6. minor refactoring

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    6c59adf View commit details
    Browse the repository at this point in the history

  7. update doc

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    700629d View commit details
    Browse the repository at this point in the history

  8. update comment

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    077912e View commit details
    Browse the repository at this point in the history

  9. update docstrings

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    62c6568 View commit details
    Browse the repository at this point in the history

  10. rename functions

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    a40a755 View commit details
    Browse the repository at this point in the history

  11. rename test classes

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    7ba0085 View commit details
    Browse the repository at this point in the history

  12. update doc

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    6c96d97 View commit details
    Browse the repository at this point in the history

  13. update doc

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    80fbe6e View commit details
    Browse the repository at this point in the history

  14. update docstrings and comments

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    fdf4bcb View commit details
    Browse the repository at this point in the history

  15. avoid do import nt|posix modules if unnecessary

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    185f130 View commit details
    Browse the repository at this point in the history

  16. set nt|posix modules to None if not available

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    c8c98ae View commit details
    Browse the repository at this point in the history

  17. micro speedup

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    17bb5e6 View commit details
    Browse the repository at this point in the history

  18. update description

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    d8b9bf9 View commit details
    Browse the repository at this point in the history

  19. add doc note

    @giampaolo
    giampaolo committed May 28, 2018
    Configuration menu
    Copy the full SHA
    b59ac57 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2018

  1. use better wording in doc

    @giampaolo
    giampaolo committed May 29, 2018
    Configuration menu
    Copy the full SHA
    8eefce7 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2018

  1. Merge branch 'master' into shutil-zero-copy

    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    4fc8c6b View commit details
    Browse the repository at this point in the history

  2. rename function using 'fastcopy' prefix instead of 'zerocopy'

    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    3048e3d View commit details
    Browse the repository at this point in the history

  3. use :ref: in rst doc

    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    11102e1 View commit details
    Browse the repository at this point in the history

  4. change wording in doc

    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    7545273 View commit details
    Browse the repository at this point in the history

  5. add test to make sure sendfile() doesn't get called aymore in case it… 

    … doesn't support file to file copies
    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    3261b74 View commit details
    Browse the repository at this point in the history

  6. move CopyFileW in _winapi and actually expose CopyFileExW instead

    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    51c476d View commit details
    Browse the repository at this point in the history

  7. fix line endings

    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    729dd23 View commit details
    Browse the repository at this point in the history

  8. add tests for mode bits

    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    1823828 View commit details
    Browse the repository at this point in the history

  9. add docstring

    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    a9d6a07 View commit details
    Browse the repository at this point in the history

  10. remove test file mode class; let's keep it for later when Istart addr… 

    …essing OSX fcopyfile() specific copies
    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    e3ce917 View commit details
    Browse the repository at this point in the history

  11. update doc to reflect new changes

    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    f81a0ec View commit details
    Browse the repository at this point in the history

  12. update doc

    @giampaolo
    giampaolo committed May 30, 2018
    Configuration menu
    Copy the full SHA
    3e7475b View commit details
    Browse the repository at this point in the history

Commits on May 31, 2018

  1. adjust tests on win

    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    05dd3cf View commit details
    Browse the repository at this point in the history

  2. fix argument clinic error

    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    9b54930 View commit details
    Browse the repository at this point in the history

  3. update doc

    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    2bec11c View commit details
    Browse the repository at this point in the history

  4. OSX: expose copyfile(3) instead of fcopyfile(3); also expose flags ar… 

    …g to python
    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    c87648f View commit details
    Browse the repository at this point in the history

  5. osx / copyfile: use path_t instead of char

    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    941f740 View commit details
    Browse the repository at this point in the history

  6. do not set dst name in the OSError exception in order to remain consi… 

    …stent with platforms which cannot do that (e.g. linux)
    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    4d28c12 View commit details
    Browse the repository at this point in the history

  7. add same file test

    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    2149b8b View commit details
    Browse the repository at this point in the history

  8. add test for same file

    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    6a02a2a View commit details
    Browse the repository at this point in the history

  9. have osx copyfile() pre-emptively check if src and dst are the same, … 

    …otherwise it will return immedialtey and src file content gets deleted
    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    2287508 View commit details
    Browse the repository at this point in the history

  10. turn PermissionError into appropriate SameFileError

    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    b9da5d5 View commit details
    Browse the repository at this point in the history

  11. expose ERROR_SHARING_VIOLATION in order to raise more appropriate Sam… 

    …eFileError
    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    c921f46 View commit details
    Browse the repository at this point in the history

  12. honour follow_symlinks arg when using CopyFileEx

    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    bb24490 View commit details
    Browse the repository at this point in the history

  13. update Misc/NEWS

    @giampaolo
    giampaolo committed May 31, 2018
    Configuration menu
    Copy the full SHA
    fef8b32 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2018

  1. expose CreateDirectoryEx mock

    @giampaolo
    giampaolo committed Jun 5, 2018
    Configuration menu
    Copy the full SHA
    71be453 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2018

  1. change C type

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    6035fe2 View commit details
    Browse the repository at this point in the history

  2. CreateDirectoryExW actual implementation

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    8dc651e View commit details
    Browse the repository at this point in the history

  3. provide specific makedirs() implementation for win

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    5d0eada View commit details
    Browse the repository at this point in the history

  4. Merge branch 'shutil-zero-copy-8' of https://github.com/giampaolo/cpy… 

    …thon into shutil-zero-copy-8
    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    d67cdc5 View commit details
    Browse the repository at this point in the history

  5. fix typo

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    f65c8ae View commit details
    Browse the repository at this point in the history

  6. skeleton for SetNamedSecurityInfo

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    9c4508e View commit details
    Browse the repository at this point in the history

  7. get security info for src path

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    bb1fee6 View commit details
    Browse the repository at this point in the history

  8. finally set security attrs

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    566898a View commit details
    Browse the repository at this point in the history

  9. add unit tests

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    f435053 View commit details
    Browse the repository at this point in the history

  10. mimick os.makedirs() behavior and raise if dst dir exists

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    30c9a57 View commit details
    Browse the repository at this point in the history

  11. set 2 paths for OSError object

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    33f362f View commit details
    Browse the repository at this point in the history

  12. set 2 paths for OSError object

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    e17e729 View commit details
    Browse the repository at this point in the history

  13. expand windows test

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    bc46f75 View commit details
    Browse the repository at this point in the history

  14. in case of exception on os.sendfile() set filename and filename2 exce… 

    …ption attributes
    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    cabbc02 View commit details
    Browse the repository at this point in the history

  15. set 2 filenames (src, dst) for OSError in case copyfile() fails on OSX

    @giampaolo
    giampaolo committed Jun 6, 2018
    Configuration menu
    Copy the full SHA
    d22ee08 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2018

  1. update doc

    @giampaolo
    giampaolo committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    7a08203 View commit details
    Browse the repository at this point in the history

  2. do not use CreateDirectoryEx() in copytree() if source dir is a symli… 

    …nk (breaks test_copytree_symlink_dir); instead just create a plain dir and remain consistent with POSIX implementation
    @giampaolo
    giampaolo committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    ab284e9 View commit details
    Browse the repository at this point in the history

  3. use bytearray() and readinto()

    @giampaolo
    giampaolo committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    ac9479d View commit details
    Browse the repository at this point in the history

  4. use memoryview() with bytearray()

    @giampaolo
    giampaolo committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    fd77a7e View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2018

  1. refactoring + introduce a new _fastcopy_binfileobj() fun

    @giampaolo
    giampaolo committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    42a597e View commit details
    Browse the repository at this point in the history

  2. remove CopyFileEx and other C wrappers

    @giampaolo
    giampaolo committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    5008a8d View commit details
    Browse the repository at this point in the history

  3. remove code related to CopyFileEx

    @giampaolo
    giampaolo committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    e89dd20 View commit details
    Browse the repository at this point in the history

  4. Recognize binary files in copyfileobj() 

    ...and use fastest _fastcopy_binfileobj() when possible
    @giampaolo
    giampaolo committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    c0dc4b8 View commit details
    Browse the repository at this point in the history

  5. set 1MB copy bufsize on win; also add a global _COPY_BUFSIZE variable

    @giampaolo
    giampaolo committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    29b9730 View commit details
    Browse the repository at this point in the history

  6. use ctx manager for memoryview()

    @giampaolo
    giampaolo committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    a1bed32 View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2018

  1. update doc

    @giampaolo
    giampaolo committed Jun 9, 2018
    Configuration menu
    Copy the full SHA
    d9d27a7 View commit details
    Browse the repository at this point in the history

  2. remove outdated doc

    @giampaolo
    giampaolo committed Jun 9, 2018
    Configuration menu
    Copy the full SHA
    17bd78b View commit details
    Browse the repository at this point in the history

  3. remove last CopyFileEx remnants

    @giampaolo
    giampaolo committed Jun 9, 2018
    Configuration menu
    Copy the full SHA
    b1d4917 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2018

  1. OSX - use fcopyfile(3) instead of copyfile(3) 

    ...as an extra safety measure: in case src/dst are "exotic" files (non
regular or living on a network fs etc.) we better fail on open() instead
of copyfile(3) as we're not quite sure what's gonna happen in that
case.
    @giampaolo
    giampaolo committed Jun 12, 2018
    Configuration menu
    Copy the full SHA
    5ce94e4 View commit details
    Browse the repository at this point in the history

  2. update doc

    @giampaolo
    giampaolo committed Jun 12, 2018
    Configuration menu
    Copy the full SHA
    07bcef5 View commit details
    Browse the repository at this point in the history