Not clear in documentation how pathblacklist works

[Zenimax-DSpeed]

scc/processor/file.go

Line 87 in 012df08

if strings.HasPrefix(filepath.Join(root, f.Name()), black) {

No mention I noticed from the example that it includes the root, and not clear why you might want to potentially blacklist files outside the root?

[boyter]

Open and closed instantly. Its a good point though. I think that the blacklist should not apply out of the root.

Any reason for closing this so quickly? I think its a good thing to include.

[montanaflynn]

I also can't figure out how to blacklist nested vendor directories. This is important for a go directory containing many packages with their own vendor directories.

[boyter]

Odd @montanaflynn because if you say do the following running against scc itself.

$ scc -co --pbl .git -wl go .
-------------------------------------------------------------------------------
Language                 Files     Lines     Code  Comments   Blanks Complexity
-------------------------------------------------------------------------------
Go                          70      9376     6670      1382     1324       1649
-------------------------------------------------------------------------------
Total                       70      9376     6670      1382     1324       1649
-------------------------------------------------------------------------------

$ scc -co --pbl .git,vendor -wl go .
-------------------------------------------------------------------------------
Language                 Files     Lines     Code  Comments   Blanks Complexity
-------------------------------------------------------------------------------
Go                          28      3517     2684       266      567        635
-------------------------------------------------------------------------------
Total                       28      3517     2684       266      567        635
-------------------------------------------------------------------------------

It will exclude the root vendor directory, and as such not even walk what is inside it. Hence any nested ones will also be missed.

I am suspecting perhaps this is not the behavior you are expecting? Could you supply a tree output showing me which directories you are expecting to be checked based on your command?

[montanaflynn]

I was referring to nested vendor directories @boyter, here's a tree:

.
├── packagea
│   ├── packagea.go
│   └── vendor
├── packageb
│   ├── packageb.go
│   └── vendor

I want to count packagea.go and packageb.go but nothing in any vendor directories. Basically I want to use scc on a path like $GOPATH/src/github.com/montanaflynn/ blacklisting any vendor directories.

[boyter]

Ah fair point. The blacklist is literally just based on the root.

I should probably add a new option that allows nested folders to be excluded. --nbl perhaps. Then keep this existing functionality.

[mcanfield]

@boyter in modern javascript/typescript land it is very common to include .spec.js | .test.js | .spec.ts | .test.ts right next to their corresponding source code files. Any suggestions on how to filter those files out since the blacklist flag is path not file oriented?

[boyter]

Currently there is none. I am thinking perhaps a regex exclusion that you can use to exclude matching folders or files might be the best approach to take here.

It would allow deprecation of the --pbl flag as well.

[boyter]

Sitting on the branch,

$ go run main.go -v -e "vendor|scripts|examples|.*\.json|cocomo\.go" -co .
 WARN 2018-09-28T06:23:09Z: skipping directory due to being in blacklist: .git
 WARN 2018-09-28T06:23:09Z: skipping file unknown extension: Gopkg.lock
 WARN 2018-09-28T06:23:09Z: skipping directory due to match exclude: examples
 WARN 2018-09-28T06:23:09Z: skipping file due to match exclude: languages.json
 WARN 2018-09-28T06:23:09Z: skipping directory due to match exclude: scripts
 WARN 2018-09-28T06:23:09Z: skipping directory due to match exclude: vendor
 WARN 2018-09-28T06:23:09Z: skipping directory due to match exclude: cocomo.go
 WARN 2018-09-28T06:23:09Z: skipping file unknown extension: scc.iml
-------------------------------------------------------------------------------
Language                 Files     Lines     Code  Comments   Blanks Complexity
-------------------------------------------------------------------------------
Go                          16      3533     2690       343      500        565
Markdown                     4       515      420         0       95          0
XML                          4       635      635         0        0          0
License                      2        45       37         0        8          0
Plain Text                   2        24       17         0        7          0
Powershell                   2       240       48       157       35          8
YAML                         1        13       13         0        0          0
nuspec                       1        22       22         0        0          0
TOML                         1        38        9        25        4          0
gitignore                    1        19        9         5        5          0
-------------------------------------------------------------------------------
Total                       34      5084     3900       530      654        573
-------------------------------------------------------------------------------

[boyter]

@montanaflynn @mcanfield I belive that the above would solve the issue for you. Let me know if you disagree and we can revisit this.

[montanaflynn]

@boyter thanks, I verified it would solve the issue for me.

 ~/D/g/s/g/b/scc  ⎇ Issue23  mkdir something                                                                                                                                  
 ~/D/g/s/g/b/scc  ⎇ Issue23  mkdir something/vendor                                                                                                                           
 ~/D/g/s/g/b/scc  ⎇ Issue23  touch something/vendor/main.go                                                                                                                   
 go run main.go -v -e "vendor" -co .                                                               
 WARN 2018-09-28T07:31:41Z: skipping directory due to being in blacklist: .git
 WARN 2018-09-28T07:31:41Z: skipping file unknown extension: Gopkg.lock
 WARN 2018-09-28T07:31:41Z: skipping directory due to match exclude: examples
 WARN 2018-09-28T07:31:41Z: skipping file due to match exclude: languages.json
 WARN 2018-09-28T07:31:41Z: skipping directory due to match exclude: scripts
 WARN 2018-09-28T07:31:41Z: skipping directory due to match exclude: vendor
 WARN 2018-09-28T07:31:41Z: skipping directory due to match exclude: vendor
 WARN 2018-09-28T07:31:41Z: skipping directory due to match exclude: cocomo.go
...

One thing I might change is listing the relative directory path in the verbose logging so you don't see just the last exluded directory name like:

 WARN 2018-09-28T07:31:41Z: skipping directory due to match exclude: vendor
 WARN 2018-09-28T07:31:41Z: skipping directory due to match exclude: vendor

[mcanfield]

@boyter Heya, yup I just gave it a try and the regex filter works great! 🎉

This tool has now become a favorite :). Thank you much!

[boyter]

@mcanfield Great to hear that.

Slight tweak to include the full path as suggested by @montanaflynn

$ go run main.go -v -e "vendor|scripts|.*\.json|cocomo\.go|cwalk" -co .
 WARN 2018-09-28T20:22:22Z: skipping directory due to being in blacklist: .git
 WARN 2018-09-28T20:22:22Z: skipping file unknown extension: Gopkg.lock
 WARN 2018-09-28T20:22:22Z: skipping file due to match exclude: languages.json
 WARN 2018-09-28T20:22:22Z: skipping directory due to match exclude: scripts
 WARN 2018-09-28T20:22:22Z: skipping directory due to match exclude: vendor
 WARN 2018-09-28T20:22:22Z: skipping directory due to match exclude: examples/cwalk
 WARN 2018-09-28T20:22:22Z: skipping file unknown extension: scc.iml
 WARN 2018-09-28T20:22:22Z: skipping file unknown extension: 1000k
 WARN 2018-09-28T20:22:22Z: skipping file unknown extension: 100k
 WARN 2018-09-28T20:22:22Z: skipping file unknown extension: 10k
 WARN 2018-09-28T20:22:22Z: skipping file unknown extension: fileread
 WARN 2018-09-28T20:22:22Z: skipping file unknown extension: linuxaverage
 WARN 2018-09-28T20:22:22Z: skipping file due to match exclude: examples/fileread/textfile.json
 WARN 2018-09-28T20:22:22Z: skipping file due to match exclude: processor/cocomo.go

[boyter]

Going to close this on the basis both @mcanfield and @montanaflynn indicate that this resolves their issues.