Code

Buffered S3 reader#247

Code

Merged

kuenishi merged 11 commits intopfnet:masterpfnet/pfio:masterfrom belltailjp:make-s3-bufferedbelltailjp/pfio:make-s3-bufferedCopy head branch name to clipboardDec 28, 2021

+134-42

Commits

Commits on Dec 27, 2021

• Provide readinto to support buffered reading

  belltailjp

  committed
  69fb2c5

• Return S3 reader wrapped by BufferedReader

  belltailjp

  committed
  81695ed

• Reproduction test for readall behavior inconsistency to normal filesystems

  belltailjp

  committed
  939b73a

• Make _ObjectReader.readline behave consistently to the normal filesystems

  belltailjp

  committed
  689e676

Commits on Dec 28, 2021

• Test for buffering options

  belltailjp

  committed
  afca2d7

• Buffering control option

  belltailjp

  committed
  c3402af

• Add unused kwargs to each filesystem (to avoid strict filesystem dependency in open_url arguments)

  belltailjp

  committed
  ca7403d

• Run S3 read-related test cases both buffered and un-buffered

  belltailjp

  committed
  de1d0cb

• Slightly reorganized test cases

  belltailjp

  committed
  fda9ae6

• Adaptive buffer size by default

  belltailjp

  committed
  6405725

• Fix isort

  belltailjp

  committed
  af6f2f5