FEAT-#3603: add experimental read_custom_text function that can read custom line-by-line text files
#3441
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #3441 +/- ##
==========================================
+ Coverage 86.96% 89.97% +3.01%
==========================================
Files 207 208 +1
Lines 17049 17095 +46
==========================================
+ Hits 14827 15382 +555
+ Misses 2222 1713 -509
Continue to review full report at Codecov.
|
|
|
|
anmyachev
force-pushed
the
experimental-parser
branch
from
42cf3ae
to
9acee0f
Compare
This implementation allows us to work with the source data formats that are used in DIEN workload, in a broader sense - any text line-by-line format that does not fit standard parsers. P.S. added tests showing how this function works. |
anmyachev
force-pushed
the
experimental-parser
branch
4 times, most recently
from
383d52c
to
d211d9a
Compare
anmyachev
force-pushed
the
experimental-parser
branch
from
d211d9a
to
4521a98
Compare
anmyachev
changed the title
anmyachev
force-pushed
the
experimental-parser
branch
from
4521a98
to
e98afd7
Compare
|
I don't think we should delay it, my understanding is that #3540 being merged makes it much easier to implement custom parser now. |
|
I restarted CI, please let me know when it is ready for review! |
anmyachev
force-pushed
the
experimental-parser
branch
from
e98afd7
to
6c69160
Compare
|
Why cannot it re-use the same mechanism |
anmyachev
force-pushed
the
experimental-parser
branch
from
6c69160
to
cd88814
Compare
…ustom libe-by-line text files Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
anmyachev
force-pushed
the
experimental-parser
branch
from
d409c97
to
9c24116
Compare
vnlitvinov
reviewed
Mar 1, 2022
anmyachev
commented
Mar 1, 2022
| @@ -152,7 +152,7 @@ def generic_parse(fname, **kwargs): | |||
| num_splits = kwargs.pop("num_splits", None) | |||
| start = kwargs.pop("start", None) | |||
| end = kwargs.pop("end", None) | |||
| header_size = kwargs.pop("header_size", None) | |||
| header_size = kwargs.pop("header_size", 0) | |||
There was a problem hiding this comment.
otherwise, we are getting TypeError: 'NoneType' object cannot be interpreted as an integer error
anmyachev
force-pushed
the
experimental-parser
branch
from
f452341
to
ef8e4cb
Compare
Co-authored-by: Vasily Litvinov <vasilij.n.litvinov@intel.com>
anmyachev
force-pushed
the
experimental-parser
branch
from
08e723f
to
b3a5395
Compare
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
vnlitvinov
reviewed
Mar 2, 2022
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
anmyachev
commented
Mar 2, 2022
| @@ -955,7 +951,7 @@ def _read(cls, filepath_or_buffer, **kwargs): | |||
| new_query_compiler : BaseQueryCompiler | |||
| Query compiler with imported data for further processing. | |||
| """ | |||
| filepath_or_buffer_md = ( | |||
| filepath = ( | |||
There was a problem hiding this comment.
Same about filepath_or_buffer_md variable.
anmyachev
changed the title
read_custom_text function that can read custom line-by-line text files
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
anmyachev
force-pushed
the
experimental-parser
branch
from
12a16b8
to
5b9b591
Compare
This was referenced Mar 2, 2022
|
@vnlitvinov ready for review |
|
@vnlitvinov ready to merge |
vnlitvinov
approved these changes
Mar 15, 2022
YarShev
reviewed
Mar 21, 2022
Comment on lines
46
to
+47
| "PickleExperimentalDispatcher", | ||
| "CustomTextExperimentalDispatcher", |
There was a problem hiding this comment.
@anmyachev, why do we put experimental features in the base core instead of experimental one?
| from modin.config import NPartitions | ||
|
|
||
|
|
||
| class CustomTextExperimentalDispatcher(TextFileDispatcher): |
There was a problem hiding this comment.
I am in favor of ExperimentalCustomTextDispatcher. The use of Experimental in the middle of a name looks weird to me.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
flake8 modinblack --check modingit commit -sdocs/developer/architecture.rstis up-to-date