Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SAS7BDAT parser: Speed up RLE/RDC decompression #47405

Merged
merged 22 commits into from
Oct 3, 2022
Merged

SAS7BDAT parser: Speed up RLE/RDC decompression #47405

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0e02b8d
Speed up RLE/RDC decompression
jonashaag Jun 15, 2022
eca0db4
Update tests
jonashaag Jun 17, 2022
041a04b
ssize_t -> size_t
jonashaag Jun 20, 2022
0451c31
Merge branch 'main' into sas/decompress3
jonashaag Jun 26, 2022
f2c8b0e
Update sas.pyx
jonashaag Jun 26, 2022
17c72f8
Merge branch 'main' into sas/decompress3
jonashaag Jul 2, 2022
91f8436
Merge branch 'main' into sas/decompress3
jonashaag Jul 5, 2022
221f20c
Merge branch 'main' into sas/decompress3
jonashaag Jul 6, 2022
213b08f
Don't use null byte as except value
jonashaag Jul 7, 2022
4b24773
Nit
jonashaag Jul 7, 2022
263aea6
Simplify condition
jonashaag Jul 9, 2022
785f752
Review feedback
jonashaag Jul 9, 2022
1f36f99
Docstring -> comment
jonashaag Jul 9, 2022
26aea28
Revert "Simplify condition"
jonashaag Jul 9, 2022
afdfc1c
Merge branch 'main' into sas/decompress3
jonashaag Jul 10, 2022
6a3fd55
Merge branch 'main' into sas/decompress3
jonashaag Aug 8, 2022
fc5621b
Merge branch 'main' into sas/decompress3
jonashaag Aug 29, 2022
0d3daa8
Merge branch 'main' into sas/decompress3
jonashaag Sep 2, 2022
0588d18
Merge branch 'main' into sas/decompress3
jonashaag Sep 15, 2022
21ba0b2
Lint
jonashaag Sep 15, 2022
55cceb7
Speed up some Cython `except`
jonashaag Sep 22, 2022
ba9b019
Typo
jonashaag Sep 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 15 additions & 22 deletions asv_bench/benchmarks/io/sas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,30 +1,23 @@
import os
from pathlib import Path

from pandas import read_sas

ROOT = Path(__file__).parents[3] / "pandas" / "tests" / "io" / "sas" / "data"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC this is a style choice orthogonal to the rest of the PR? no real problem with it, but in general best to minimize these to make it easier to focus on the important bits

Copy link
Contributor Author

@jonashaag jonashaag Jul 7, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ASV file would’ve been very confusing if I left the old code because my additions can’t use the old code and then we’d end up with two almost identical but different versions.



class SAS:
def time_read_sas7bdat(self):
read_sas(ROOT / "test1.sas7bdat")

params = ["sas7bdat", "xport"]
param_names = ["format"]
def time_read_xpt(self):
read_sas(ROOT / "paxraw_d_short.xpt")

def setup(self, format):
# Read files that are located in 'pandas/tests/io/sas/data'
files = {"sas7bdat": "test1.sas7bdat", "xport": "paxraw_d_short.xpt"}
file = files[format]
paths = [
os.path.dirname(__file__),
"..",
"..",
"..",
"pandas",
"tests",
"io",
"sas",
"data",
file,
]
self.f = os.path.join(*paths)
def time_read_sas7bdat_2(self):
next(read_sas(ROOT / "0x00controlbyte.sas7bdat.bz2", chunksize=11000))

def time_read_sas(self, format):
read_sas(self.f, format=format)
def time_read_sas7bdat_2_chunked(self):
for i, _ in enumerate(
read_sas(ROOT / "0x00controlbyte.sas7bdat.bz2", chunksize=1000)
):
if i == 10:
break
Loading