Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ReadCSV: parameter nrows cause fallback to Pandas #1523

Closed
akharche opened this issue Jun 2, 2020 · 3 comments · Fixed by #1894
Closed

ReadCSV: parameter nrows cause fallback to Pandas #1523

akharche opened this issue Jun 2, 2020 · 3 comments · Fixed by #1894
Assignees
@dchigarev
Labels
new feature/request 💬 Requests and pull requests for new features P2 Minor bugs or low-priority feature requests
Milestone
0.8.1

Comments

@akharche
Copy link

akharche commented Jun 2, 2020

Adding 'dtype' parameter to read_csv() falls back to Pandas implementation.

dtype = np.int16
df = pd.read_csv(filename, nrows=100000, dtype=dtype, header=None)

UserWarning: Parameters provided defaulting to pandas implementation.
@akharche akharche added the new feature/request 💬 Requests and pull requests for new features label Jun 2, 2020
@devin-petersohn
Copy link
Collaborator

Hi @akharche, thanks for the report!

I believe it is the nrows=100000 argument that is causing the default to pandas. dtype parameter is supported.

@akharche
Copy link
Author

akharche commented Jun 3, 2020

Hi @devin-petersohn, thanks for the clarification!

Should I reopen the issue with 'nrows' parameter?

@aregm aregm changed the title ReadCSV: parameter dtype ReadCSV: parameter nrows cause fallback to Pandas Jun 3, 2020
@devin-petersohn
Copy link
Collaborator

No we can keep the issue. @aregm has changed the title.

@devin-petersohn devin-petersohn added this to the 0.7.4 milestone Jun 3, 2020
@devin-petersohn devin-petersohn modified the milestones: 0.8.0, 0.8.1 Jul 29, 2020
@anmyachev anmyachev added the P2 Minor bugs or low-priority feature requests label Aug 4, 2020
@dchigarev dchigarev mentioned this issue Aug 5, 2020
Merged
6 tasks
dchigarev added a commit to dchigarev/modin that referenced this issue Aug 19, 2020
@dchigarev
FEAT-modin-project#1523: 'nrows' support added at 'read_csv'
953e0e4 
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
dchigarev added a commit to dchigarev/modin that referenced this issue Sep 3, 2020
@dchigarev
FEAT-modin-project#1523: 'nrows' support added at 'read_csv'
2035346 
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
dchigarev added a commit to dchigarev/modin that referenced this issue Sep 3, 2020
@dchigarev
FEAT-modin-project#1523: added docstring
6fce56c 
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
dchigarev added a commit to dchigarev/modin that referenced this issue Sep 3, 2020
@dchigarev
FEAT-modin-project#1523: solution refactoring
4efd9a8 
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
dchigarev added a commit to dchigarev/modin that referenced this issue Sep 3, 2020
@dchigarev
FEAT-modin-project#1523: moved file partitioning to 'TextFileReader'
e9dfcc0 
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
dchigarev added a commit to dchigarev/modin that referenced this issue Sep 3, 2020
@dchigarev
FEAT-modin-project#1523: doc-strings added
571de09 
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
dchigarev added a commit to dchigarev/modin that referenced this issue Sep 3, 2020
@dchigarev
FEAT-modin-project#1523: new test cases added
5139548 
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
dchigarev added a commit to dchigarev/modin that referenced this issue Sep 3, 2020
@dchigarev
FEAT-modin-project#1523: performance fix
c98501e 
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
dchigarev added a commit to dchigarev/modin that referenced this issue Sep 3, 2020
@dchigarev
FEAT-modin-project#1523: added assert statement
b981ecf 
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
devin-petersohn pushed a commit that referenced this issue Sep 4, 2020
@dchigarev
FEAT-#1523: 'nrows' support to 'read_csv' added (#1894)
c7e947b 
Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
aregm pushed a commit to aregm/modin that referenced this issue Sep 16, 2020
@dchigarev @aregm
FEAT-modin-project#1523: 'nrows' support to 'read_csv' added (modin-p…
05b014f 
…roject#1894)

Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dchigarev dchigarev

Labels
new feature/request 💬 Requests and pull requests for new features P2 Minor bugs or low-priority feature requests
Projects
None yet
Milestone
0.8.1
Development

Successfully merging a pull request may close this issue.

FEAT-#1523: 'nrows' support to 'read_csv' added dchigarev/modin
4 participants
@devin-petersohn @akharche @anmyachev @dchigarev