Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(WIP) Even faster .NET Parser #127

Draft
wants to merge 54 commits into
base: develop
Choose a base branch
from
Draft

(WIP) Even faster .NET Parser #127

wants to merge 54 commits into from

Conversation

philipmat
Copy link
Owner

@philipmat philipmat commented Sep 13, 2020 •
edited
Loading

File Record Count Python C# v1 C# v2
discogs_20200806_artists.xml.gz 7,046,615 6:22 2:35 0:28 - 13x / 5x
discogs_20200806_labels.xml.gz 1,571,873 1:15 0:22 0:05 - 15x / 4x
discogs_20200806_masters.xml.gz 1,734,371 3:56 1:57 0:31 - 7x / 4x
discogs_20200806_releases.xml.gz 12,867,980 1:45:16 42:38 15:17 - 7x / 3x

That's just a part of the story.
Parallel, processing is the other.
Screen Shot 2020-09-13 at 11 36 31 AM

As the screenshot above shows, the .NET version allows processing multiple files in parallel, achieving a 20x speedup from the Python version in this case.

@philipmat
uses direct identification of nodes in reader
b98b453
@philipmat
renames exporter interface and method to remove csv
38064f5
@philipmat
makes individual classes responsible for populating themselves
e02de33
@philipmat
parses artists (TODO: members)
63db4f2
@philipmat
avoids creating satellite assemblies
e785c86
@philipmat
parses members as well
864887d
@philipmat
excludes some temp python folders
c512f74
@philipmat
reverts changes by an overly eager vscode helper
7161a5e
@philipmat
parses groups
49c7f43 
in part it's done to have a more accurate parser
@philipmat
parses sublabels
b31bb3b
@philipmat
skips nodes we don't (yet) parse/export
ce984a4
@philipmat
tests artist being populated from reader and sample file
d70dec2
@philipmat
allows for both deserialize and parse to operate on the same file
9cee51c
@philipmat
removes unnecessary usings
71ca6b1
@philipmat
sublabel populate tests
3313c37
@philipmat
asserts on images
c40aeb9
@philipmat
reads sublabels
f99ab52
@philipmat
reads groups
6ba30cf
@philipmat
reduces the number of images on label
a67abfd
@philipmat
reads master, except for artist and videos
275ee56
@philipmat
parses master artists
8f46737
@philipmat
reads videos and hardens artist reading
44490f3
@philipmat
adds two artists for test
9c35723
@philipmat
implements IsValid
9427dd5
@philipmat
parser master artists and videos correctly
d36a854
@philipmat
Merge branch 'develop' into dotnet_parser_specific_xml_parsing
d004244 
# Conflicts:
#	alternatives/dotnet/discogs/Program.cs
@philipmat
updates RecordCounter with develop changes
7c84def
@philipmat
Merge branch 'develop' into dotnet_parser_specific_xml_parsing
9e46bd1
@philipmat
images provide a factory method
1667960
@philipmat
reads top level artist tags and images
ffce139
@philipmat philipmat added this to the v3.0 - .NET Parser is main milestone Sep 13, 2020
@MuleaneEve
Copy link

Very cool to go from a process that used to take almost 2 hours to barely 15 minutes :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v3.0 - .NET Parser is main
Development

Successfully merging this pull request may close these issues.
2 participants
@philipmat @MuleaneEve