Out of memory crash in 0.0.6

[kachkaev]

This issue is a follow-up to igorshubovych/markdownlint-cli#108

Trying markdownlint-cli2 "**/*" in a project that has a large folder in .markdownlintignore results this output in about 5 mins:

<--- Last few GCs --->

[57132:0x102d52000]   340958 ms: Mark-sweep 2346.6 (2408.0) -> 2348.5 (2399.5) MB, 1070.8 / 0.0 ms  (average mu = 0.113, current mu = 0.028) allocation failure scavenge might not succeed
[57132:0x102d52000]   341633 ms: Mark-sweep 2348.5 (2399.5) -> 2346.6 (2391.3) MB, 675.3 / 0.0 ms  (average mu = 0.070, current mu = 0.000) allocation failure scavenge might not succeed


<--- JS stacktrace --->
Cannot get stack trace in GC.
FATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory
 1: 0x1011c2ff5 node::Abort() (.cold.1) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 2: 0x10009fbc9 node::Abort() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 3: 0x10009fd2f node::OnFatalError(char const*, char const*) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 4: 0x1001e3907 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 5: 0x1001e38a7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 6: 0x1003695e5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 7: 0x1003bd837 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 8: 0x10039b58b void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
 9: 0x10039acc8 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
10: 0x10039aa06 v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
11: 0x1003c24ce v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
12: 0x10038b7f2 v8::internal::ItemParallelJob::Task::RunInternal() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
13: 0x10038bc78 v8::internal::ItemParallelJob::Run() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
14: 0x10039dfb5 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
15: 0x10039daa5 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
16: 0x10038ff17 v8::internal::MarkCompactCollector::Evacuate() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
17: 0x10038d92e v8::internal::MarkCompactCollector::CollectGarbage() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
18: 0x10036992f v8::internal::Heap::MarkCompact() [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
19: 0x1003671b5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
20: 0x100365670 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
21: 0x10037149a v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
22: 0x100371521 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
23: 0x10033f73a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
24: 0x10068e808 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
25: 0x1009d39b9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
26: 0x1009d488d Builtins_StringAdd_CheckNone [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
27: 0x100a1060f Builtins_RegExpReplace [/Users/me/.nvm/versions/node/v12.18.3/bin/node]
error Command failed with signal "SIGABRT".

Repro steps are the same as in markdownlint-cli2 "**/*" except that the ignored folder is even bigger (354K files distributed across about 6K subfolders)

[DavidAnson]

In version 0.0.6 and later of CLI2, the .markdownlintignore file is no longer used. Because of that, the command line you are giving is telling it to lint every file (Markdown or not) in what you say is a very large directory tree. Eventually this will run out of memory. Could you please try again, either passing the ignore patterns on the command line or using the new ignores option in .markdownlint-cli2.jsonc.

If it runs out of memory with a reasonable set of Markdown files, there are some things I can do to try to fix that, but if the only issue here is that it is trying to process huge swaths of other files, that's not a use case I feel is important.

[kachkaev]

Thanks for your reply @DavidAnson, I missed the new rule around .markdownlintignore indeed. Gonna give the CLI another shot in a few days, thanks for clarifying!

Speaking of this trap I fell into, WDYT of emitting a warning of some sort if the number of files to process becomes ridiculously large? Looking at how Prettier handles formatting and linting, there's also scope for refactoring the app so that it emits to std on every processed file and does not hold too much in the memory heap. This way, the tool will be more fault-tolerant and will emit the output as it goes, which will be useful for users. If I started seeing messages like "error in file temp/abc, error in temp/xyz" in my stdout, I'd press ctrl+c without waiting till the tool crashes.

Another random thought: how about checking for .markdownlintignore and showing a warning right at the start if the file exists?

[DavidAnson]

Streaming output would not work with the current implementation of output filters. But maybe it could write some status to stdout by default. Probably just a summary of inputs, then the file count, then a count of errors.

With that in place, it should be pretty clear when too many files are being looked at. I'd rather not introduce a whole warning system just for that one scenario that should only ever happen once to people who are migrating.

[DavidAnson]

I've added progress to stdout here: bb7c4dd

It can be turned off via the noProgress option.

Example output:

Finding: **/*.md
Linting: 3 file(s)
Summary: 9 error(s)

[kachkaev]

Thanks for your reply @DavidAnson!

I tried 0.0.8 and generally managed to get a fast result without crashing. Removing .markdownlintignore and adding .markdownlint-cli2.jsonc with the ignores section helped here.

Two observations:

• When it comes to suppressing progress status, it might be useful to consider different expected behaviours for TTY and non-TTY stdout devices. When we are in TTY mode (local Terminal), it's expected to see the animated progress. When we run a tool in CI, the output that works fine locally may look pretty odd. Context:

  • https://github.com/prettier/prettier/blob/370844cc4cbe3c43b8e699e3e4bcec215bd94e08/src/cli/is-tty.js
  • Build log is too long in CI doczjs/docz#597

• I'm wondering if the format of ignores entries could support negation itself. This would allow running markdownlint-cli2 for "**/*" and ignores section would exclude all, and then re-include *.md, *.mdx and so on. At the moment, adding something like !*.md to ignores produces !!*.md and not *.md.

Since the above observations are not related to the out-of-memory crash, feel free to close this issue 🙂

[DavidAnson]

Great, thank you, I will close this! By the way, I made a couple of other refactorings intended to avoid unnecessary memory pressure. They will be in the next release.

Regarding your point about progress, note that the kind of progress used here is not the fancy kind that updates the same line in the console or anything like that. It just outputs three lines of plain text at the relevant parts of execution. And they can be turned off entirely if wanted.

Regarding double negation in the ignores property, I'm afraid that might get confusing. Remember that ignores are applied after the original globs from the command line, so they can not add back in files that were excluded there. They could add back in files that were excluded by the ignore pattern itself, but I'm afraid trying to explain and understand that might be difficult. I'm going to wait and see if this becomes an issue for users.

[DavidAnson]

Commit referenced above: 3404146