Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide streaming mode #210

Closed
alexey-milovidov opened this issue Dec 17, 2020 · 4 comments
Closed

Provide streaming mode #210

alexey-milovidov opened this issue Dec 17, 2020 · 4 comments
Labels
enhancement New feature or request

Comments

@alexey-milovidov
Copy link
Contributor

Describe the bug

I cannot successfully run scc on 20 TB dump of GitHub (231 219 569 files) without OOM on a server with 128 GiB RAM,
even if I run scc with csv output format.
@alexey-milovidov
Copy link
Contributor Author

This is needed for ClickHouse/ClickHouse#15323

@boyter
Copy link
Owner

boyter commented Dec 17, 2020

Oh that's an interesting one... not one I ever considered, since I have never tried running it over that much data in one go.

I guess a stream option for CSV and SQL works in this case well enough.

@boyter boyter added the enhancement New feature or request label Dec 17, 2020
boyter added a commit that referenced this issue Jan 4, 2021
@boyter
First cut at #210
c31a33d
@boyter
Copy link
Owner

boyter commented Jan 4, 2021

@alexey-milovidov if you are able to build from source you can get from master and try the new stream mode for csv

scc --format csv-stream

It should dump the results as they come in to standard output avoiding any memory issues (in theory).

lukas-brenning added a commit to lukas-brenning/scc that referenced this issue Jan 13, 2021
@lukas-brenning @boyter
@AlDanial @arp242
Test (#1)
93de68b 
* Update README.md

* SLOCCount came before cloc

* resolve boyter#206

* First cut at boyter#210

* Add "gohtml" and "gotxt" to the list of extensions for Go templates

These are the default extensions that e.g. GoLand uses, and some other
tools as well I believe. It's useful as it disambiguates between
templates for html/template and text/template.

* partial fix for boyter#214

* resolve after feedback

* Mostly ready for boyter#211

* real fix for issue214

* few tests and typo

* Resolvehttps://github.com/boyter/issues/208

* Update to remove wrong Go version mentioned boyter#220

Co-authored-by: Ben Boyter <boyter@users.noreply.github.com>
Co-authored-by: AlDanial <al.danial@gmail.com>
Co-authored-by: Ben Boyter <ben@boyter.org>
Co-authored-by: Martin Tournoij <martin@arp242.net>
@boyter
Copy link
Owner

boyter commented Feb 22, 2021

Closing down as done and ready for release.

@boyter boyter closed this as completed Feb 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@boyter @alexey-milovidov