Don't abort when rule throws, but continue #87

mcmire · 2024-06-03T21:55:42Z

If a project is being linted and a rule throws an error — perhaps it relies on a file to be in a certain format, or some other reason — then linting for that project will halt, and the lint report for that project will only show that error. An error like this signifies a bug in module-lint, but sometimes we can't fix that bug right away, and in the meantime this makes the report useless, since we don't know whether any of the other rules passed or not.

This commit fixes this problem by catching errors inside of rules and displaying them as such. To implement this change, we needed to convert the binary status of a rule to a trinary status ("passed", "failed", "errored"). We also updated the output of the lint report to display this error status and distinguish it from a failed status.

References

See this recent module-lint run for an example of why the output is useless when an error is thrown instead of caught.

Fixes #86.

Screenshots

Before

In this example we run yarn run-tool utils. Note how no rules are listed at all but rather a single error is shown:

In this example we run yarn run-tool . (which lints module-lint itself):

After

Running yarn run-tool utils. Note how instead of a single error being shown, the statuses of rules are shown and the error that's thrown in one of the rules is displayed:

Running yarn run-tool .:

If a project is being linted and a rule throws an error — perhaps it relies on a file to be in a certain format, or some other reason — then linting for that project will halt, and the lint report for that project will only show that error. An error like this signifies a bug in `module-lint`, but sometimes we can't fix that bug right away, and in the meantime this makes the report useless, since we don't know whether any of the other rules passed or not. This commit fixes this problem by catching errors inside of rules and displaying them as such. To implement this change, we needed to convert the binary status of a rule to a trinary status ("passed", "failed", "errored"). We also updated the output of the lint report to display this error status and distinguish it from a failed status.

mcmire · 2024-06-03T21:57:02Z

src/cli.ts

-  console.error(error);
-  process.exitCode = 1;
-});
+})


Previously, main would set process.exitCode. However, to remain testable, I'm trying to keep main free of changing global state, so that means changing process. To accommodate this constraint, I had main return true or false, which we can then use to set process.exitCode accordingly.

mcmire · 2024-06-03T22:01:29Z

src/main.test.ts

      });
    });

-    it('does not exit immediately if a project errors during linting, but shows the error and continues', async () => {
+    it('does not abort linting of a project if a lint rule throws, but shows the error and continues to the next line rule, returning false', async () => {


There are two similar tests in this file which exercise the new logic added in this PR; this is one of them.

mcmire · 2024-06-03T22:18:45Z

src/rules/all-yarn-modern-files-conform.test.ts

@@ -49,7 +49,7 @@ describe('Rule: all-yarn-modern-files-conform', () => {
      });

      expect(result).toStrictEqual({
-        passed: true,
+        status: 'passed',


All rules return a result object, and that result object contains the status. So the test file for every rule needed to be updated to check for the correct property.

kanthesha · 2024-06-06T12:18:36Z

We discussed about errors blocking the completion of module linting. And good to see this PR fixes it.

Additionally, there's one more good to have feature (definitely in a separate ticket).
Right now, for example, if we module lint this repo, there're 18 passed, 19 failed and total 37. But, if we delete package.json and run the module lint again, the resulting numbers will be 10 passed, 7 failed, 1 errored totalling 18. Because of package.json not available, couple of checks have been skipped. How about covering them as skipped?

kanthesha

Overall LGTM. Just one query on the variable renaming.

kanthesha · 2024-06-06T14:31:31Z

src/main.ts

@@ -231,23 +225,25 @@ async function lintProjects({
    projectLintResultPromiseOutcomes.filter(isPromiseRejectedResult);

  outputLogger.logToStdout('');
-  let isAllPassed = true;
+
+  let allPassed = true;


Just curious on the renaming? prefixing is seems to be fine for the boolean!

I changed it because I wanted it to read more like a part of a sentence. So I was thinking of "all passed" like how you'd ask a question ("all passed?"). But maybe that doesn't make sense here. didAllPass would probably make more sense, but it's kind of awkward. Ah... I can change it back 😂

Okay, done here: b206993

mcmire · 2024-06-06T18:00:02Z

Additionally, there's one more good to have feature (definitely in a separate ticket).
Right now, for example, if we module lint this repo, there're 18 passed, 19 failed and total 37. But, if we delete package.json and run the module lint again, the resulting numbers will be 10 passed, 7 failed, 1 errored totalling 18. Because of package.json not available, couple of checks have been skipped. How about covering them as skipped?

Hmm. All of the numbers we give at the end are totals, which means that they correspond to rules that we show above. If we show the number of skipped, that implies that we would need to list them. Is there a reason why we or someone would want to know which rules were skipped? Is it just to verify for ourselves that all rules that could have been run were at least considered? Otherwise, I'm not sure that people would think about it — they might only care about passing and failing.

kanthesha · 2024-06-07T14:30:03Z

Sure. My thought process was, for example, deleting package.json can show just one lint failed (package.json does not exist). Once the user fixes it, suddenly it'll start showing couple of more lint errors! which were hidden/didn't execute because there was no package.json. I thought this may surprise the user. But seems like, not that important. They'll know by the type of error/failed list.

src/main.ts

Co-authored-by: Kanthesha Devaramane <kanthesha.devaramane@consensys.net>

mcmire · 2024-06-07T15:31:36Z

Sure. My thought process was, for example, deleting package.json can show just one lint failed (package.json does not exist). Once the user fixes it, suddenly it'll start showing couple of more lint errors! which were hidden/didn't execute because there was no package.json. I thought this may surprise the user. But seems like, not that important. They'll know by the type of error/failed list.

Gotcha. Good point. Something to think about for sure.

mcmire · 2024-06-13T17:48:52Z

The CI failure seems to be happening because we're using a hack to import the Node-specific functions from @metamask/utils, and the newest version of this package (released just a few days ago) invalidates this hack. I've got a fix for this here: #92

kanthesha

LGTM.

mcmire requested a review from a team as a code owner June 3, 2024 21:55

mcmire force-pushed the catch-errors-in-rules branch from da45402 to cb33fd0 Compare June 3, 2024 22:00

mcmire requested a review from a team June 3, 2024 22:05

mcmire force-pushed the catch-errors-in-rules branch from cb33fd0 to ac1954b Compare June 3, 2024 22:17

mcmire commented Jun 3, 2024

View reviewed changes

mcmire mentioned this pull request Jun 3, 2024

In case of errors Project Name is not displayed. #88

Closed

mcmire added the team-wallet-framework label Jun 3, 2024

Revert output format back

b5f2b6e

kanthesha reviewed Jun 6, 2024

View reviewed changes

Change name of variable back

b206993

kanthesha reviewed Jun 7, 2024

View reviewed changes

src/main.ts Outdated Show resolved Hide resolved

Update src/main.ts

6bfe0c8

Co-authored-by: Kanthesha Devaramane <kanthesha.devaramane@consensys.net>

mcmire removed the team-wallet-framework label Jun 11, 2024

Merge branch 'main' into catch-errors-in-rules

57881bd

kanthesha approved these changes Jun 27, 2024

View reviewed changes

mcmire merged commit 54ac694 into main Jun 27, 2024
24 checks passed

mcmire deleted the catch-errors-in-rules branch June 27, 2024 23:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't abort when rule throws, but continue #87

Don't abort when rule throws, but continue #87

mcmire commented Jun 3, 2024 •

edited

Loading

mcmire Jun 3, 2024

mcmire Jun 3, 2024

mcmire Jun 3, 2024

kanthesha commented Jun 6, 2024

kanthesha left a comment

kanthesha Jun 6, 2024

mcmire Jun 6, 2024

mcmire Jun 6, 2024

mcmire commented Jun 6, 2024

kanthesha commented Jun 7, 2024

mcmire commented Jun 7, 2024

mcmire commented Jun 13, 2024 •

edited

Loading

kanthesha left a comment

Don't abort when rule throws, but continue #87

Don't abort when rule throws, but continue #87

Conversation

mcmire commented Jun 3, 2024 • edited Loading

References

Screenshots

Before

After

mcmire Jun 3, 2024

Choose a reason for hiding this comment

mcmire Jun 3, 2024

Choose a reason for hiding this comment

mcmire Jun 3, 2024

Choose a reason for hiding this comment

kanthesha commented Jun 6, 2024

kanthesha left a comment

Choose a reason for hiding this comment

kanthesha Jun 6, 2024

Choose a reason for hiding this comment

mcmire Jun 6, 2024

Choose a reason for hiding this comment

mcmire Jun 6, 2024

Choose a reason for hiding this comment

mcmire commented Jun 6, 2024

kanthesha commented Jun 7, 2024

mcmire commented Jun 7, 2024

mcmire commented Jun 13, 2024 • edited Loading

kanthesha left a comment

Choose a reason for hiding this comment

mcmire commented Jun 3, 2024 •

edited

Loading

mcmire commented Jun 13, 2024 •

edited

Loading