Fix partition dtypes so that `to_pandas` always has correct dtypes #1382

devin-petersohn · 2020-04-21T18:33:16Z

Resolves Incorrect dtype when read_csv reads a file big enough #1165
Add a read function to base class of all readers FileReader
Change implementation of read to _read for the children classes
Apply Categorical dtype to all partitions after the dataframe has
been read

Signed-off-by: Devin Petersohn devin.petersohn@gmail.com

What do these changes do?

passes flake8 modin
passes black --check modin
signed commit with git commit -s
Resolves Incorrect dtype when read_csv reads a file big enough #1165
tests added and passing

modin-bot · 2020-04-21T18:43:15Z

TeamCity Python test results bot

Tests PASSed

Tests Logs

-rfE --tb=native modin/pandas/test/test_io.py
============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /localdisk/tc_agent/work/199b2674cba60ed6, inifile: setup.cfg
plugins: testmon-1.0.2, cov-2.9.0, forked-1.1.3, custom-exit-code-0.3.0, xdist-1.32.0
collected 90 items

modin/pandas/test/test_io.py .................s.........s............... [ 47%]
s..............s..s.X.....s.................ss.                          [100%]


TOTAL                                                               6654   4273    36%


=========== 81 passed, 8 skipped, 1 xpassed, 117 warnings in 14.91s ============
Closing remaining open files:test_write_pandas.hdf...donetest_write_modin.hdf...done
-rfE --tb=native -n=48 modin/pandas/test/test_series.py modin/pandas/test/test_dataframe.py modin/pandas/test/test_concat.py modin/pandas/test/test_groupby.py modin/pandas/test/test_reshape.py modin/pandas/test/test_general.py
============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /localdisk/tc_agent/work/199b2674cba60ed6, inifile: setup.cfg
plugins: testmon-1.0.2, cov-2.9.0, forked-1.1.3, custom-exit-code-0.3.0, xdist-1.32.0
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I / gw32 I / gw33 I / gw34 I / gw35 I / gw36 I / gw37 I / gw38 I / gw39 I / gw40 I / gw41 I / gw42 I / gw43 I / gw44 I / gw45 I / gw46 I / gw47 I
gw0 [15286] / gw1 [15286] / gw2 [15286] / gw3 [15286] / gw4 [15286] / gw5 [15286] / gw6 [15286] / gw7 [15286] / gw8 [15286] / gw9 [15286] / gw10 [15286] / gw11 [15286] / gw12 [15286] / gw13 [15286] / gw14 [15286] / gw15 [15286] / gw16 [15286] / gw17 [15286] / gw18 [15286] / gw19 [15286] / gw20 [15286] / gw21 [15286] / gw22 [15286] / gw23 [15286] / gw24 [15286] / gw25 [15286] / gw26 [15286] / gw27 [15286] / gw28 [15286] / gw29 [15286] / gw30 [15286] / gw31 [15286] / gw32 [15286] / gw33 [15286] / gw34 [15286] / gw35 [15286] / gw36 [15286] / gw37 [15286] / gw38 [15286] / gw39 [15286] / gw40 [15286] / gw41 [15286] / gw42 [15286] / gw43 [15286] / gw44 [15286] / gw45 [15286] / gw46 [15286] / gw47 [15286]

........................................................................ [  0%]
........................................................................ [  0%]
......................................................................... [  1%]
......................................................................... [  1%]
........................................................................ [  2%]
........................................................................ [  2%]
.........................................s.............................. [  3%]
........................................................................ [  3%]
.......................................................................... [  4%]
........................................................................ [  4%]
........................................................................ [  5%]
........................................................................ [  5%]
........................................................................ [  6%]
.......................................................................... [  6%]
............................................................................ [  7%]
.........................................................s.............. [  7%]
.............................................s........................... [  8%]
........................................................................ [  8%]
........................................................................ [  8%]
...............................................................s........ [  9%]
.........................................s.............................. [  9%]
........................................................................ [ 10%]
...................................s.................................... [ 10%]
........................................................................ [ 11%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 12%]
.......................................................................... [ 13%]
......................................................................... [ 13%]
........................................................................ [ 14%]
......................................................................... [ 14%]
........................................................................ [ 15%]
.................................................................................. [ 15%]
........................................................................ [ 16%]
............................................................................ [ 16%]
............................................................................ [ 17%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 18%]
........................................................................ [ 19%]
........................................................................ [ 19%]
........................................................................ [ 19%]
......................................................................... [ 20%]
........................................................................ [ 20%]
.................................................s...................... [ 21%]
........................................................................ [ 21%]
........................................................................ [ 22%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 23%]
........................................................................ [ 24%]
........................................................................ [ 24%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 26%]
........................................................................ [ 27%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 28%]
........................................................................ [ 28%]
........................................................................ [ 29%]
.......................x................................................ [ 29%]
........................................................................ [ 30%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 32%]
...............................................ss.sssss..ssssss.sssss... [ 33%]
........................................................................ [ 33%]
........................................................................ [ 34%]
........................................................................ [ 34%]
........................................................................ [ 35%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 36%]
......................................................................... [ 36%]
........................................................................ [ 37%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 40%]
........................................................................ [ 41%]
........................................................................ [ 41%]
........................................................................ [ 42%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 44%]
........................................................................ [ 44%]
........................................................................ [ 45%]
........................................................................ [ 45%]
........................................................................ [ 46%]
........................................................................ [ 46%]
......................................................................... [ 47%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 48%]
........................................................................ [ 49%]
........................................................................ [ 49%]
........................................................................ [ 50%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 51%]
.......................................................................... [ 51%]
.......................................................................... [ 52%]
........................................................................ [ 52%]
......................................................................... [ 53%]
.......................................................................... [ 53%]
........................................................................ [ 54%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 55%]
........................................................................ [ 56%]
........................................................................ [ 56%]
........................................................................ [ 57%]
........................................................................ [ 57%]
........................................................................ [ 58%]
........................................................................ [ 58%]
........................................................................ [ 59%]
......................................................................... [ 59%]
........................................................................ [ 60%]
......................................................................... [ 60%]
........................................................................ [ 61%]
........................................................................ [ 61%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 63%]
........................................................................ [ 64%]
......................................................................... [ 64%]
........................................................................ [ 65%]
........................................................................ [ 65%]
........................................................................ [ 66%]
.......................................................................... [ 66%]
........................................................................ [ 67%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 68%]
........................................................................ [ 69%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 70%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 71%]
........................................................................ [ 72%]
......................................................................... [ 72%]
........................................................................ [ 73%]
........................................................................ [ 73%]
......................................................................... [ 74%]
........................................................................ [ 74%]
........................................................................ [ 75%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 77%]
........................................................................ [ 78%]
........................................................................ [ 78%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 79%]
........................................................................ [ 80%]
........................................................................ [ 80%]
........................................................................ [ 81%]
........................................................................ [ 81%]
......................................................................... [ 82%]
........................................................................ [ 82%]
........................................................................ [ 83%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 84%]
........................................................................ [ 85%]
........................................................................ [ 85%]
........................................................................ [ 86%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 87%]
......................................................................... [ 87%]
........................................................................ [ 88%]
......................................................................... [ 88%]
........................................................................ [ 89%]
........................................................................ [ 89%]
.......................................................................... [ 90%]
........................................................................ [ 90%]
......................................................................... [ 91%]
........................................................................ [ 91%]
...................s................X.........x........XX..x....x....... [ 92%]
.s..ssss..sx..............................X............................. [ 92%]
........................................................................ [ 93%]
........................................................................ [ 93%]
........................................................................ [ 94%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 95%]
........................................................................ [ 95%]
........................................................................ [ 96%]
........................................................................ [ 96%]
........................................................................ [ 97%]
........................................................................ [ 97%]
........................................................................ [ 98%]
........................................................................ [ 98%]
........................................................................ [ 99%]
........................................................................ [ 99%]
......................................                                   [100%]


TOTAL                                                               6654   1611    76%

= 15245 passed, 32 skipped, 5 xfailed, 4 xpassed, 23929 warnings in 507.68s (0:08:27) =

codecov · 2020-04-21T18:44:07Z

Codecov Report

Merging #1382 into master will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1382      +/-   ##
==========================================
+ Coverage   86.19%   86.21%   +0.02%     
==========================================
  Files          69       69              
  Lines        6642     6654      +12     
==========================================
+ Hits         5725     5737      +12     
  Misses        917      917

Impacted Files	Coverage Δ
modin/engines/base/io/column_stores/hdf_reader.py	`100.00% <ø> (ø)`
...in/engines/base/io/column_stores/parquet_reader.py	`97.05% <ø> (ø)`
modin/engines/base/io/text/csv_reader.py	`98.21% <ø> (ø)`
modin/engines/base/io/text/json_reader.py	`96.00% <ø> (ø)`
modin/data_management/factories.py	`83.70% <100.00%> (+0.24%)`	⬆️
...in/engines/base/io/column_stores/feather_reader.py	`100.00% <100.00%> (ø)`
modin/engines/base/io/file_reader.py	`90.69% <100.00%> (+1.22%)`	⬆️
modin/engines/base/io/sql/sql_reader.py	`95.34% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 564e6cc...23015c5. Read the comment docs.

modin-bot · 2020-04-21T18:45:03Z

TeamCity Dask test results bot

Tests PASSed

Tests Logs

-rfE --tb=native modin/pandas/test/test_io.py
============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /localdisk/tc_agent/work/199b2674cba60ed6, inifile: setup.cfg
plugins: xdist-1.32.0, custom-exit-code-0.3.0, forked-1.1.3, testmon-1.0.2, cov-2.9.0
collected 90 items

modin/pandas/test/test_io.py .................s.........s............... [ 47%]
...............s..s.X.....s.................ss.                          [100%]


TOTAL                                                               6654   3898    41%


============ 82 passed, 7 skipped, 1 xpassed, 88 warnings in 36.73s ============
Closing remaining open files:test_write_modin.hdf...donetest_write_pandas.hdf...done
-rfE --tb=native -n=48 modin/pandas/test/test_series.py modin/pandas/test/test_dataframe.py modin/pandas/test/test_concat.py modin/pandas/test/test_groupby.py modin/pandas/test/test_reshape.py modin/pandas/test/test_general.py
============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /localdisk/tc_agent/work/199b2674cba60ed6, inifile: setup.cfg
plugins: xdist-1.32.0, custom-exit-code-0.3.0, forked-1.1.3, testmon-1.0.2, cov-2.9.0
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I / gw32 I / gw33 I / gw34 I / gw35 I / gw36 I / gw37 I / gw38 I / gw39 I / gw40 I / gw41 I / gw42 I / gw43 I / gw44 I / gw45 I / gw46 I / gw47 I
gw0 [15286] / gw1 [15286] / gw2 [15286] / gw3 [15286] / gw4 [15286] / gw5 [15286] / gw6 [15286] / gw7 [15286] / gw8 [15286] / gw9 [15286] / gw10 [15286] / gw11 [15286] / gw12 [15286] / gw13 [15286] / gw14 [15286] / gw15 [15286] / gw16 [15286] / gw17 [15286] / gw18 [15286] / gw19 [15286] / gw20 [15286] / gw21 [15286] / gw22 [15286] / gw23 [15286] / gw24 [15286] / gw25 [15286] / gw26 [15286] / gw27 [15286] / gw28 [15286] / gw29 [15286] / gw30 [15286] / gw31 [15286] / gw32 [15286] / gw33 [15286] / gw34 [15286] / gw35 [15286] / gw36 [15286] / gw37 [15286] / gw38 [15286] / gw39 [15286] / gw40 [15286] / gw41 [15286] / gw42 [15286] / gw43 [15286] / gw44 [15286] / gw45 [15286] / gw46 [15286] / gw47 [15286]

........................................................................ [  0%]
........................................................................ [  0%]
........................................................................ [  1%]
........................................................................ [  1%]
......................................................................... [  2%]
........................................................................ [  2%]
........................................................................ [  3%]
........................................................................ [  3%]
......................................................................... [  4%]
.........s............................................................... [  4%]
........................................................................ [  5%]
........................................................................ [  5%]
........................................................................ [  6%]
........................................................................ [  6%]
............................................s........................... [  7%]
........................................................................ [  7%]
........................................................................ [  8%]
....s.................................................................... [  8%]
........................................................................ [  8%]
.....................s.................................................. [  9%]
.............................................................s........... [  9%]
........................................................................ [ 10%]
....s................................................................... [ 10%]
........................................................................ [ 11%]
........................................................................ [ 11%]
.......................................................................... [ 12%]
......................................................................... [ 12%]
........................................................................ [ 13%]
........................................................................ [ 13%]
........................................................................ [ 14%]
.......................................................................... [ 14%]
.......................................................................... [ 15%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 16%]
........................................................................ [ 17%]
........................................................................ [ 17%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 18%]
........................................................................ [ 19%]
........................................................................ [ 19%]
......................................................................... [ 20%]
........................................................................ [ 20%]
..........s............................................................. [ 21%]
........................................................................ [ 21%]
........................................................................ [ 22%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 23%]
........................................................................ [ 24%]
........................................................................ [ 24%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 26%]
........................................................................ [ 27%]
........................................................................ [ 27%]
......................................................................... [ 28%]
........................................................................ [ 28%]
......................................................................... [ 29%]
........................................................................ [ 29%]
........................................................................ [ 30%]
..x..................................................................... [ 30%]
........................................................................ [ 31%]
......................................................................... [ 31%]
........................................................................ [ 32%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 33%]
........................................................................ [ 34%]
..........................................ssssssssssssssssss............ [ 34%]
........................................................................ [ 34%]
........................................................................ [ 35%]
........................................................................ [ 35%]
........................................................................ [ 36%]
......................................................................... [ 36%]
........................................................................ [ 37%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 40%]
........................................................................ [ 41%]
........................................................................ [ 41%]
........................................................................ [ 42%]
........................................................................ [ 42%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 44%]
........................................................................ [ 45%]
........................................................................ [ 45%]
........................................................................ [ 46%]
........................................................................ [ 46%]
........................................................................ [ 47%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 48%]
........................................................................ [ 49%]
........................................................................ [ 49%]
........................................................................ [ 50%]
........................................................................ [ 50%]
........................................................................ [ 50%]
......................................................................... [ 51%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 52%]
........................................................................ [ 53%]
........................................................................ [ 53%]
........................................................................ [ 54%]
......................................................................... [ 54%]
........................................................................ [ 55%]
........................................................................ [ 55%]
........................................................................ [ 56%]
........................................................................ [ 56%]
........................................................................ [ 57%]
........................................................................ [ 57%]
........................................................................ [ 58%]
........................................................................ [ 58%]
........................................................................ [ 59%]
......................................................................... [ 59%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 60%]
........................................................................ [ 61%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 63%]
........................................................................ [ 64%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 65%]
........................................................................ [ 66%]
......................................................................... [ 66%]
........................................................................ [ 67%]
......................................................................... [ 67%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 68%]
........................................................................ [ 69%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 71%]
........................................................................ [ 72%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 73%]
........................................................................ [ 74%]
........................................................................ [ 74%]
........................................................................ [ 75%]
........................................................................ [ 75%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 77%]
........................................................................ [ 78%]
........................................................................ [ 78%]
......................................................................... [ 79%]
........................................................................ [ 79%]
........................................................................ [ 80%]
........................................................................ [ 80%]
........................................................................ [ 81%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 82%]
........................................................................ [ 83%]
........................................................................ [ 83%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 84%]
........................................................................ [ 85%]
........................................................................ [ 85%]
......................................................................... [ 86%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 87%]
......................................................................... [ 88%]
........................................................................ [ 88%]
........................................................................ [ 89%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 90%]
........................................................................ [ 91%]
..........s.sss......................................................... [ 91%]
........................................................................ [ 92%]
.......................................s................................ [ 92%]
...........X.........x.....x...............x.......ss..x.......X........ [ 92%]
........................................................................ [ 93%]
........................................................................ [ 93%]
........................................................................ [ 94%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 95%]
........................................................................ [ 96%]
........................................................................ [ 96%]
........................................................................ [ 97%]
.............................X.....X.................................... [ 97%]
........................................................................ [ 98%]
........................................................................ [ 98%]
........................................................................ [ 99%]
........................................................................ [ 99%]
.....................................................................    [100%]


TOTAL                                                               6654   1379    79%

= 15245 passed, 32 skipped, 5 xfailed, 4 xpassed, 24019 warnings in 725.76s (0:12:05) =

modin-bot · 2020-04-21T18:54:02Z

TeamCity Ray test results bot

Tests PASSed

Tests Logs

modin/pandas/test/test_io.py
============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /localdisk/tc_agent/work/199b2674cba60ed6, inifile: setup.cfg
plugins: xdist-1.32.0, custom-exit-code-0.3.0, forked-1.1.3, testmon-1.0.2, cov-2.9.0
collected 90 items

modin/pandas/test/test_io.py .................s.........s............... [ 47%]
...............s..s.X.....s.................ss.                          [100%]


TOTAL                                                               6654   3737    44%


============ 82 passed, 7 skipped, 1 xpassed, 78 warnings in 32.67s ============
Closing remaining open files:test_write_modin.hdf...donetest_write_pandas.hdf...done
-rfE --tb=native modin/experimental/pandas/test/test_io_exp.py
============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /localdisk/tc_agent/work/199b2674cba60ed6, inifile: setup.cfg
plugins: xdist-1.32.0, custom-exit-code-0.3.0, forked-1.1.3, testmon-1.0.2, cov-2.9.0
collected 2 items

modin/experimental/pandas/test/test_io_exp.py ..                         [100%]


TOTAL                                                               6654   3696    44%


======================== 2 passed, 5 warnings in 2.69s =========================
-rfE --tb=native -n=48 modin/pandas/test/test_series.py modin/pandas/test/test_dataframe.py modin/pandas/test/test_concat.py modin/pandas/test/test_groupby.py modin/pandas/test/test_reshape.py modin/pandas/test/test_general.py
============================= test session starts ==============================
platform linux -- Python 3.7.5, pytest-5.4.2, py-1.8.1, pluggy-0.13.1
rootdir: /localdisk/tc_agent/work/199b2674cba60ed6, inifile: setup.cfg
plugins: xdist-1.32.0, custom-exit-code-0.3.0, forked-1.1.3, testmon-1.0.2, cov-2.9.0
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I / gw32 I / gw33 I / gw34 I / gw35 I / gw36 I / gw37 I / gw38 I / gw39 I / gw40 I / gw41 I / gw42 I / gw43 I / gw44 I / gw45 I / gw46 I / gw47 I
gw0 [15286] / gw1 [15286] / gw2 [15286] / gw3 [15286] / gw4 [15286] / gw5 [15286] / gw6 [15286] / gw7 [15286] / gw8 [15286] / gw9 [15286] / gw10 [15286] / gw11 [15286] / gw12 [15286] / gw13 [15286] / gw14 [15286] / gw15 [15286] / gw16 [15286] / gw17 [15286] / gw18 [15286] / gw19 [15286] / gw20 [15286] / gw21 [15286] / gw22 [15286] / gw23 [15286] / gw24 [15286] / gw25 [15286] / gw26 [15286] / gw27 [15286] / gw28 [15286] / gw29 [15286] / gw30 [15286] / gw31 [15286] / gw32 [15286] / gw33 [15286] / gw34 [15286] / gw35 [15286] / gw36 [15286] / gw37 [15286] / gw38 [15286] / gw39 [15286] / gw40 [15286] / gw41 [15286] / gw42 [15286] / gw43 [15286] / gw44 [15286] / gw45 [15286] / gw46 [15286] / gw47 [15286]

........................................................................ [  0%]
........................................................................ [  0%]
........................................................................ [  1%]
........................................................................ [  1%]
........................................................................ [  2%]
........................................................................ [  2%]
........................................................................ [  3%]
........................................................................ [  3%]
........................................................................ [  4%]
........................................................................ [  4%]
........................................................................ [  5%]
........................................................................ [  5%]
..s....................................................................... [  6%]
........................................................................ [  6%]
.......................................s................................ [  7%]
........................................................................ [  7%]
........................................................................ [  8%]
.................................................................s...... [  8%]
.....................................................s.................. [  8%]
..................................................s..................... [  9%]
........................................................................ [  9%]
...................................................................s.... [ 10%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 12%]
........................................................................ [ 13%]
........................................................................ [ 13%]
........................................................................ [ 14%]
........................................................................ [ 14%]
......................................................................... [ 15%]
........................................................................ [ 15%]
......................................................................... [ 16%]
........................................................................ [ 16%]
......................................................................... [ 16%]
........................................................................ [ 17%]
......................................................................... [ 17%]
........................................................................ [ 18%]
........................................................................ [ 18%]
........................................................................ [ 19%]
........................................................................ [ 19%]
........................................................................ [ 20%]
......................................................................... [ 20%]
......................................s................................. [ 21%]
........................................................................ [ 21%]
........................................................................ [ 22%]
........s.sssss.sssssss.ss.sss.......................................... [ 22%]
........................................................................ [ 23%]
........................................................................ [ 23%]
........................................................................ [ 24%]
........................................................................ [ 24%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 26%]
........................................................................ [ 27%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 28%]
........................................................................ [ 29%]
........................................................................ [ 29%]
......................................................................... [ 30%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 33%]
........................................................................ [ 33%]
........................................................................ [ 34%]
........................................................................ [ 34%]
........................................................................ [ 35%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 38%]
........................................................................ [ 39%]
......................................................................... [ 39%]
........................................................................ [ 40%]
........................................................................ [ 40%]
........................................................................ [ 41%]
........................................................................ [ 41%]
........................................................................ [ 41%]
............................x............................................ [ 42%]
......................................................................... [ 42%]
........................................................................ [ 43%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 44%]
........................................................................ [ 45%]
......................................................................... [ 45%]
........................................................................ [ 46%]
......................................................................... [ 46%]
........................................................................ [ 47%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 48%]
........................................................................ [ 49%]
......................................................................... [ 49%]
........................................................................ [ 50%]
......................................................................... [ 50%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 51%]
........................................................................ [ 52%]
........................................................................ [ 52%]
........................................................................ [ 53%]
........................................................................ [ 53%]
........................................................................ [ 54%]
........................................................................ [ 54%]
........................................................................ [ 55%]
......................................................................... [ 55%]
........................................................................ [ 56%]
........................................................................ [ 56%]
......................................................................... [ 57%]
........................................................................ [ 57%]
........................................................................ [ 58%]
........................................................................ [ 58%]
........................................................................ [ 58%]
........................................................................ [ 59%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 60%]
........................................................................ [ 61%]
......................................................................... [ 61%]
........................................................................ [ 62%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 63%]
........................................................................ [ 64%]
........................................................................ [ 64%]
........................................................................ [ 65%]
......................................................................... [ 65%]
........................................................................ [ 66%]
........................................................................ [ 66%]
.......................................................................... [ 67%]
........................................................................ [ 67%]
........................................................................ [ 67%]
........................................................................ [ 68%]
........................................................................ [ 68%]
........................................................................ [ 69%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 70%]
........................................................................ [ 71%]
........................................................................ [ 71%]
........................................................................ [ 72%]
........................................................................ [ 72%]
......................................................................... [ 73%]
......................................................................... [ 73%]
........................................................................ [ 74%]
........................................................................ [ 74%]
............................................................................ [ 75%]
........................................................................ [ 75%]
........................................................................ [ 76%]
........................................................................ [ 76%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 77%]
........................................................................ [ 78%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 79%]
........................................................................ [ 80%]
........................................................................ [ 80%]
........................................................................ [ 81%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 82%]
......................................................................... [ 83%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 84%]
........................................................................ [ 84%]
........................................................................ [ 85%]
........................................................................ [ 85%]
........................................................................ [ 86%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 87%]
........................................................................ [ 88%]
........................................................................ [ 88%]
........................................................................ [ 89%]
........................................................................ [ 89%]
........................................................................ [ 90%]
........................................................................ [ 90%]
............................................................ss........... [ 91%]
......................................................................... [ 91%]
..................................................x.............X....... [ 92%]
X.X..........X..........ss...s.s.x......x.......x....................... [ 92%]
........................................................................ [ 92%]
........................................................................ [ 93%]
......................................................................... [ 93%]
........................................................................ [ 94%]
........................................................................ [ 94%]
........................................................................ [ 95%]
........................................................................ [ 95%]
........................................................................ [ 96%]
........................................................................ [ 96%]
........................................................................ [ 97%]
......................................................................... [ 97%]
..............................................................s......... [ 98%]
........................................................................ [ 98%]
........................................................................ [ 99%]
........................................................................ [ 99%]
..............................................................           [100%]


TOTAL                                                               6654   1193    82%

= 15245 passed, 32 skipped, 5 xfailed, 4 xpassed, 24020 warnings in 667.38s (0:11:07) =

* Resolves modin-project#1165 * Add a `read` function to base class of all readers `FileReader` * Change implementation of `read` to `_read` for the children classes * Apply Categorical `dtype` to all partitions after the dataframe has been read Signed-off-by: Devin Petersohn <devin.petersohn@gmail.com>

Signed-off-by: Gregory Shimansky <gregory.shimansky@intel.com>

Signed-off-by: Devin Petersohn <devin.petersohn@gmail.com>

modin/data_management/factories.py

modin/engines/base/io/file_reader.py

gshimansky mentioned this pull request May 7, 2020

DataFrame.groupby on category broken #1426

Closed

gshimansky force-pushed the issues/1165 branch from 38583b6 to 4f54b17 Compare May 27, 2020 23:28

gshimansky force-pushed the issues/1165 branch from 4f54b17 to 3b4a04e Compare May 28, 2020 14:13

gshimansky and others added 3 commits May 28, 2020 09:18

Added tests for categoricals in file io and DataFrame constructor

6e8825e

Signed-off-by: Gregory Shimansky <gregory.shimansky@intel.com>

Add a fix for non-querycompiler return types

c544e77

Signed-off-by: Devin Petersohn <devin.petersohn@gmail.com>

Apply updated API to all readers that subclass FileReader

23015c5

Signed-off-by: Devin Petersohn <devin.petersohn@gmail.com>

gshimansky force-pushed the issues/1165 branch from 3b4a04e to 23015c5 Compare May 28, 2020 14:18

gshimansky approved these changes May 28, 2020

View reviewed changes

devin-petersohn commented May 28, 2020

View reviewed changes

modin/data_management/factories.py Show resolved Hide resolved

modin/engines/base/io/file_reader.py Show resolved Hide resolved

This was referenced May 28, 2020

astype({"col": "category"}) should convert Series.values type to pandas.core.arrays.categorical.Categorical #1503

Closed

Make FileReader.read more general than current implementation for pandas kernels #1504

Open

gshimansky merged commit 94927b8 into modin-project:master May 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix partition dtypes so that `to_pandas` always has correct dtypes #1382

Fix partition dtypes so that `to_pandas` always has correct dtypes #1382

devin-petersohn commented Apr 21, 2020

modin-bot commented Apr 21, 2020 •

edited

Loading

codecov bot commented Apr 21, 2020 •

edited

Loading

modin-bot commented Apr 21, 2020 •

edited

Loading

modin-bot commented Apr 21, 2020 •

edited

Loading

Fix partition dtypes so that to_pandas always has correct dtypes #1382

Fix partition dtypes so that to_pandas always has correct dtypes #1382

Conversation

devin-petersohn commented Apr 21, 2020

What do these changes do?

modin-bot commented Apr 21, 2020 • edited Loading

TeamCity Python test results bot

Tests PASSed

codecov bot commented Apr 21, 2020 • edited Loading

Codecov Report

modin-bot commented Apr 21, 2020 • edited Loading

TeamCity Dask test results bot

Tests PASSed

modin-bot commented Apr 21, 2020 • edited Loading

TeamCity Ray test results bot

Tests PASSed

Fix partition dtypes so that `to_pandas` always has correct dtypes #1382

Fix partition dtypes so that `to_pandas` always has correct dtypes #1382

modin-bot commented Apr 21, 2020 •

edited

Loading

codecov bot commented Apr 21, 2020 •

edited

Loading

modin-bot commented Apr 21, 2020 •

edited

Loading

modin-bot commented Apr 21, 2020 •

edited

Loading