Extremely high CPU usage with -w

[max-sixty]

• Task version: v3.19.1 (h1:2KMJk6mDBacSPuFxPFvlvvHJwGZtU/hN2ENZpaFqR5s=)
• Operating System: MacOS, M1

I'm running a task with -w and seeing extremely high CPU usage every few seconds. This is the Vitals chart:

I thought it might be the underlying task, but I replaced the command with echo hello, and I'm seeing the same behavior. And the task is not re-running during this time (i.e. it's just waiting).

If anyone would like to fully reproduce, it's here; and then task -w test-rust
https://github.com/max-sixty/prql/tree/task-repro. Here's the link to the task to view it: https://github.com/max-sixty/prql/blob/701c4141441430f94932a780b32ce8de40b599d8/Taskfile.yml#L181-L208

I'd previously changed our docs to suggest task over watchexec, as that has some problems with ignoring files in PRQL/prql#1380. Zero stress if the repo is too big / doesn't work well with a large tree as cargo though.

[andreynering]

Hi @max-sixty,

You mentioned you are using v3.19.1. Some performances improvements were made on #982 by @pd93 that I believe may help you.

Can you let me know if you see the same issue on v3.20.0?

[max-sixty]

This does still have the same issue with 3.20.0.

Interestingly it only happens on task -w test-rust and not task -w test-rust-fast. The former uses (almost) the full path, which contains a huge number of files:

fd -uu . | wc -l
  208708

vs for test-rust-fast, which uses a subpath:

fd -uu . prql-compiler/ | wc -l
     123

---

Is task stat-ing the whole path on every pass rather than subscribing to changes, by any chance? If so, that seems very reasonable for small paths, less good for large paths.

[ghost]

Came here because I have that problem too, it seems to scale pretty much with the files being watched (which made me aware of checking my pattern, turns out I watched a node_modules folder for js changes ... ).

[pdf]

On this repository, with Task version: v3.39.2 (h1:Zt7KXHmMNq5xWZ1ihphDb+n2zYLCo4BdRe09AnMMIgA=):

https://github.com/delaneyj/datastar

Running task -w templ:

  templ:
    env:
      TEMPL_EXPERIMENT: rawgo
    generates:
      - "**/*_templ.go"
    sources:
      - "**/*.templ"
    cmds:
      - templ generate .

Which matches a very reasonable number of files:

find . -iname \*.templ -o -iname \*_templ.go |wc -l
64

I see ~30% CPU usage on all 16 cores from task, which is entirely unusable.

If I modify that task to reduce the scope of the watch, such that:

  templ:
    env:
      TEMPL_EXPERIMENT: rawgo
    generates:
      - "backends/**/*_templ.go"
    sources:
      - "backends/**/*.templ"
    cmds:
      - templ generate .

Which reduces the total candidate files considerably:

find . -type f |wc -l
17395
find backends -type f |wc -l
255

CPU usage drops to ~5-10% per core, which is... not great, but stops my fans from howling quite so vehemently at least.

I suspect probably not much to be done about it until #1508 can happen (#1179 (comment)).

[pdf]

Okay, weirdly, if the above templ task is modified to additionally include just files in the root directory like so:

  templ:
    env:
      TEMPL_EXPERIMENT: rawgo
    generates:
      - "backends/**/*_templ.go"
      - "./*_templ.go"
    sources:
      - "backends/**/*.templ"
      - "./*.templ"
    cmds:
      - templ generate .

Simply by adding those second globs CPU usage jumps from 5-10% per core to 10-25% per core, even though the root directory contains very few files:

find . -maxdepth 1 -type f |wc -l
16

So there's definitely something wonky going on here.

I don't use task personally, so my time for testing further is limited, but hopefully this is useful information.

[Napam]

I have also observed high CPU usage when using watch. Here is my task:

  dev-go:
    watch: true
    cmds:
      - go build -o tmp/main .
    sources:
      - "**/*.go"

My CPU usage is about 35 % for all cores. I am using a Macbook Pro with the M2 10-core Apple Silicon CPU. When doing find . -name "*.go" | wc -l I only have 9 files.

Strangely enough if I do the equivalent for my JS/TS files:

  dev-js:
    watch: true
    cmds:
      - node build.mjs
      - templ generate --notify-proxy
    sources:
      - "frontend/**/*.{ts,js}"

I barely get any higher CPU usage at all (maybe 1-2% higher than just idling, and only activity on 1 core). The number of files are the same.

EDIT:

Ok after some more testing I observe that Task seem to iterate through all files in the my directory when doing my dev-gotask. In my project root I have a node_modules directory with a lot of files, and if I simply move that out, my CPU usage is much lower. In my dev-js task, I specify that it should only look at frontend/, so it avoids the node_modules. As a double check I added the frontend/ prefix to my dev-go task sources and then I achieved equally low CPU usage. That is I change it to this:

  dev-go:
    watch: true
    cmds:
      - go build -o tmp/main .
    sources:
      - "frontend/**/*.go"

The quickest fix for me was to basically restructure my project such that I had to specify paths such that node_modules (and any other large dirs) were implicitly excluded, as the sources: [exclude: ...] thing didn't solve the issue.