Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT-#4244: Implement dataframe exchange protocol for OmniSci #4269

Merged
merged 35 commits into from
Mar 23, 2022
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
aa1cfc5
FEAT-#4244: Implement dataframe exchange protocol for OmniSci
dchigarev Feb 22, 2022
bb9b898
Implement 'from_dataframe' and other various fixes
dchigarev Feb 28, 2022
609c86f
Fix formatting
dchigarev Mar 2, 2022
be045ce
Adding more dtypes for testing
dchigarev Mar 7, 2022
7f8d7c5
Merge remote-tracking branch 'src/master' into issue-4244
dchigarev Mar 7, 2022
ad68ef6
Add more tests
dchigarev Mar 7, 2022
11f9230
Write proper doc-strings
dchigarev Mar 7, 2022
577fe37
Refactor 'from_dataframe' module
dchigarev Mar 8, 2022
7fa75ae
Add import tests
dchigarev Mar 8, 2022
17b96dc
Add zero-copy testing
dchigarev Mar 8, 2022
b551590
Fix chunking of bitmask
dchigarev Mar 8, 2022
8711a51
Minor formatting fixes
dchigarev Mar 8, 2022
80f819a
Apply suggestions from code review
dchigarev Mar 14, 2022
037c033
Apply review suggestions
dchigarev Mar 14, 2022
41d0e8e
Merge remote-tracking branch 'src/master' into issue-4244
dchigarev Mar 14, 2022
3339172
Align imports with current master branch
dchigarev Mar 14, 2022
83f40b4
Allow to access buffers of a physically chunked column
dchigarev Mar 15, 2022
854c91a
Apply review suggestions
dchigarev Mar 15, 2022
c66191b
Remove redundant 'null_count' check
dchigarev Mar 16, 2022
3e1ca03
Move 'from_dataframe' logic to core dataframe level
dchigarev Mar 16, 2022
a0219f4
Apply review suggestions
dchigarev Mar 16, 2022
e6908c6
Apply suggestions from code review
dchigarev Mar 16, 2022
e4e36e6
Move 'from_dataframe.py' from base to pandas
dchigarev Mar 16, 2022
ada742b
Apply suggestions from code review
dchigarev Mar 17, 2022
172a59a
Align var name 'nchunks' -> 'n_chunks'
dchigarev Mar 17, 2022
db9e32a
Align convertion function to 'smt_to_smt' format
dchigarev Mar 17, 2022
f035d3c
Apply formatting suggestions
dchigarev Mar 17, 2022
0029209
Add extra 'num_chunks' protocol check
dchigarev Mar 17, 2022
5104f7d
Apply review suggestions
dchigarev Mar 17, 2022
8e846d8
Add release note
dchigarev Mar 17, 2022
718414d
Apply vnlitvinov's suggestions
dchigarev Mar 22, 2022
eebcbe6
Fix date parsing
dchigarev Mar 22, 2022
bae7b11
Add more comments&doc-strings
dchigarev Mar 23, 2022
d8053a1
Apply suggestions from code review
dchigarev Mar 23, 2022
6b11631
Fix spelling 'Edianness' -> 'Endianness'
dchigarev Mar 23, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ jobs:
modin/experimental/core/execution/native/implementations/omnisci_on_native/expr.py \
modin/experimental/core/execution/native/implementations/omnisci_on_native/omnisci_worker.py \
- run: python scripts/doc_checker.py modin/experimental/core/storage_formats/omnisci
- run: python scripts/doc_checker.py modin/experimental/core/execution/native/implementations/omnisci_on_native/exchange/dataframe_protocol

lint-flake8:
name: lint (flake8)
Expand Down Expand Up @@ -339,6 +340,8 @@ jobs:
- run: MODIN_BENCHMARK_MODE=True pytest modin/pandas/test/internals/test_benchmark_mode.py
- run: pytest modin/experimental/core/execution/native/implementations/omnisci_on_native/test/test_dataframe.py
- run: pytest modin/pandas/test/test_io.py::TestCsv --verbose
- run: pytest modin/test/exchange/dataframe_protocol/test_general.py
- run: pytest modin/test/exchange/dataframe_protocol/omnisci
- uses: codecov/codecov-action@v2

test-asv-benchmarks:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/push.yml
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,8 @@ jobs:
- run: pytest modin/test/storage_formats/omnisci/test_internals.py
- run: pytest modin/experimental/core/execution/native/implementations/omnisci_on_native/test/test_dataframe.py
- run: pytest modin/pandas/test/test_io.py::TestCsv
- run: pytest modin/test/exchange/dataframe_protocol/test_general.py
- run: pytest modin/test/exchange/dataframe_protocol/omnisci
- uses: codecov/codecov-action@v2

test-all:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ class ArrowCTypes:
# - microseconds -> 'u'
# - nanoseconds -> 'n'
TIMESTAMP = "ts{resolution}:{tz}"
TIME = "tt{resolution}"
YarShev marked this conversation as resolved.
Show resolved Hide resolved


def pandas_dtype_to_arrow_c(dtype) -> str:
Expand Down
Loading