Improve Pylint parser #73

bpedersen2 · 2018-12-28T10:00:43Z

Adopt the parser to the standard 'parseable' output of pylint and use the symbolic message descriptor
instead of the numeric one.

This is a part of solving JENKINS-44915 and JENKINS-55340. The changes to the warnings-ng plugin will follow later.

Update regex to use Java8 named groups and adopt to the standard pylint (>2.0) "parseable" output format. Adopt test accordingly and supply the python source for generating the output here as well.

This allows to adjust the parser to custom output formats.

codecov · 2018-12-28T10:05:21Z

Codecov Report

Merging #73 into master will decrease coverage by 0.13%.
The diff coverage is 61.53%.

@@             Coverage Diff              @@
##             master      #73      +/-   ##
============================================
- Coverage     87.01%   86.87%   -0.14%     
  Complexity     1144     1144              
============================================
  Files           152      152              
  Lines          3650     3657       +7     
  Branches        385      387       +2     
============================================
+ Hits           3176     3177       +1     
- Misses          327      331       +4     
- Partials        147      149       +2

Impacted Files	Coverage Δ	Complexity Δ
...va/edu/hm/hafner/analysis/parser/PyLintParser.java	`70.37% <61.53%> (-19.63%)`	`6 <5> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7a4d7e...108e2f3. Read the comment docs.

codecov · 2018-12-28T10:05:22Z

Codecov Report

Merging #73 into master will decrease coverage by 0.08%.
The diff coverage is 77.27%.

@@             Coverage Diff              @@
##             master      #73      +/-   ##
============================================
- Coverage     86.87%   86.78%   -0.09%     
- Complexity     1151     1152       +1     
============================================
  Files           157      157              
  Lines          3741     3747       +6     
  Branches        406      408       +2     
============================================
+ Hits           3250     3252       +2     
- Misses          331      333       +2     
- Partials        160      162       +2

Impacted Files	Coverage Δ	Complexity Δ
...va/edu/hm/hafner/analysis/parser/PyLintParser.java	`76.92% <77.27%> (-13.08%)`	`7 <6> (+1)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 18bb371...e1e7f17. Read the comment docs.

uhafner · 2018-12-29T15:12:33Z

Are the files pymodule.py and init required?

uhafner · 2018-12-29T15:05:53Z

src/main/java/edu/hm/hafner/analysis/parser/PyLintParser.java

+        super(DEFAULT_PYLINT_ERROR_PATTERN);
+    }
+
+    public PyLintParser(@CheckForNull String regExp) {


What is the reason for this constructor?

To be able to make the regex user-configurable.

Now used in the legacy test, the changes for warnings-ng plugin will follow later.

uhafner · 2018-12-29T15:06:33Z

src/main/java/edu/hm/hafner/analysis/parser/PyLintParser.java

+        String message = matcher.group("message");
+        String category = matcher.group("symbol");
+        if (StringUtils.isEmpty(category)) {
+               category = UNKNOWN_CAT;


wouldn't it make more sense to set the category to - in that case?

I prefer a more verbose one and pylint-specific here,

src/main/java/edu/hm/hafner/analysis/parser/PyLintParser.java

src/test/java/edu/hm/hafner/analysis/parser/PylintParserTest.java

pom.xml

uhafner · 2018-12-29T15:14:16Z

src/main/java/edu/hm/hafner/analysis/parser/PyLintParser.java


+
+    public static final String UNKNOWN_CAT = "pylint-unknown";


package private

uhafner · 2019-01-13T23:16:40Z

Ping?

bpedersen2 · 2019-01-14T07:26:05Z

I am back to work now and try to update this the next days.

uhafner · 2019-01-14T23:10:07Z

Are the files pymodule.py and init required?

uhafner · 2019-01-14T23:14:49Z

Hasn't the new format just a new additional (optional) category? So we can have one parser without a parameter to parse the old and the new format? So the LegacyTestCase would use the same parser, just some matcher groups are empty. Or is that not possible? How would a user choose the correct parser in the UI with your approach? Typically the user selects a parser and the parser is responsible to parse the format of different versions.

bpedersen2 · 2019-01-15T05:20:33Z

Hasn't the new format just a new additional (optional) category? So we can have one parser without a parameter to parse the old and the new format? So the LegacyTestCase would use the same parser, just some matcher groups are empty. Or is that not possible? How would a user choose the correct parser in the UI with your approach? Typically the user selects a parser and the parser is responsible to parse the format of different versions.

No, the order of the fields is quite different (and it is freely user-configurable on the pylint side). The idea is to supply a default that parses the 'default' (== 'parseable') output of pylint and allow the user to supply a suitable regex for other formats.

Actually, I have never seen a pylint version in the last 5 years that produces anything resembling the old testcase ( especially the 'C' only is somthing that seems impossible to generate, as the msg_id is always [CRWE][0-9]{4}.

bpedersen2 · 2019-01-15T05:22:22Z

Are the files pymodule.py and init required?

It depends what one considers 'required' . They are not required from a java code point of view, but I think it is essential to have standard files to regenerate the test input if upstream pylint changes.

uhafner · 2019-01-15T09:32:06Z

Ok, thanks! I simplified the parser a little bit and remove the regexp in the API.

bpedersen2 added 2 commits December 28, 2018 10:28

Improve pylint parser

9dd35f6

Update regex to use Java8 named groups and adopt to the standard pylint (>2.0) "parseable" output format. Adopt test accordingly and supply the python source for generating the output here as well.

Add configurable regExp to pylint parser

108e2f3

This allows to adjust the parser to custom output formats.

uhafner requested changes Dec 29, 2018

View reviewed changes

bpedersen2 added 3 commits January 14, 2019 10:28

Restore legacy test and fix whitespace errors

95b39c2

Merge remote-tracking branch 'upstream/master'

cb6dba2

Change symbol visibility

e1e7f17

uhafner merged commit e1e7f17 into jenkinsci:master Jan 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve Pylint parser #73

Improve Pylint parser #73

bpedersen2 commented Dec 28, 2018

codecov bot commented Dec 28, 2018

codecov bot commented Dec 28, 2018 •

edited

Loading

uhafner commented Dec 29, 2018

uhafner Dec 29, 2018

bpedersen2 Jan 14, 2019

uhafner Dec 29, 2018

uhafner Dec 29, 2018

bpedersen2 Jan 14, 2019

uhafner Dec 29, 2018

uhafner commented Jan 13, 2019

bpedersen2 commented Jan 14, 2019

uhafner commented Jan 14, 2019

uhafner commented Jan 14, 2019

bpedersen2 commented Jan 15, 2019

bpedersen2 commented Jan 15, 2019

uhafner commented Jan 15, 2019

Improve Pylint parser #73

Improve Pylint parser #73

Conversation

bpedersen2 commented Dec 28, 2018

codecov bot commented Dec 28, 2018

Codecov Report

codecov bot commented Dec 28, 2018 • edited Loading

Codecov Report

uhafner commented Dec 29, 2018

uhafner Dec 29, 2018

Choose a reason for hiding this comment

bpedersen2 Jan 14, 2019

Choose a reason for hiding this comment

uhafner Dec 29, 2018

Choose a reason for hiding this comment

uhafner Dec 29, 2018

Choose a reason for hiding this comment

bpedersen2 Jan 14, 2019

Choose a reason for hiding this comment

uhafner Dec 29, 2018

Choose a reason for hiding this comment

uhafner commented Jan 13, 2019

bpedersen2 commented Jan 14, 2019

uhafner commented Jan 14, 2019

uhafner commented Jan 14, 2019

bpedersen2 commented Jan 15, 2019

bpedersen2 commented Jan 15, 2019

uhafner commented Jan 15, 2019

codecov bot commented Dec 28, 2018 •

edited

Loading