New test suite & linter

[weppos]

This PR adds a test suite as requested in #11 and also a linter that is supposed to check the most common errors, as proposed by @gerv in https://bugzilla.mozilla.org/show_bug.cgi?id=985003

The test suite is written in Ruby. I wanted to write it in another language (e.g. Go), but I then realized there is no way in Go to load an arbitrary list in the current x/net/suffix implementation.

Instead, this is possible in my Ruby implementation which I plan to use to check the list when a PR is submitted.

I was a little afraid about the performance of a Ruby test suite, but I'm pleasantly surprised. Only the Linter runs in less than one second.

➜  publicsuffix-list git:(testsuite) ✗ rake
/Users/weppos/.rvm/rubies/ruby-2.3.0/bin/ruby -I"lib:test" -I"/Users/weppos/.rvm/gems/ruby-2.3.0/gems/rake-10.5.0/lib" "/Users/weppos/.rvm/gems/ruby-2.3.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb" "test/lint_test.rb" "test/linter_test.rb"

# Running tests with run options --seed 19530:

......................

Finished tests in 0.045653s, 481.8928 tests/s, 1664.7205 assertions/s.

On Travis, this branch took around 12 seconds to be tested, which is still better than the existing C-based test suite that runs in around 40 seconds and doesn't test the list style.

My next step is to convert the tests in test_psl.txt into the new test suite. Actually, I'm planning to make these tests available in a sort of json format so that they can be used by other implementations to test an input and an expected output.

I see for example that Go cloned the tests in the file into their test suite. Having them defined in a standard format (can even be just a test file with a tuple of <input, expected>) can probably be helpful.

/cc @gerv @sleevi

---

Here's some failure examples:

➜  publicsuffix-list git:(testsuite) ✗ rake
/Users/weppos/.rvm/rubies/ruby-2.3.0/bin/ruby -I"lib:test" -I"/Users/weppos/.rvm/gems/ruby-2.3.0/gems/rake-10.5.0/lib" "/Users/weppos/.rvm/gems/ruby-2.3.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb" "test/lint_test.rb"

# Running tests with run options --seed 54116:

F

Finished tests in 0.003261s, 306.6248 tests/s, 306.6248 assertions/s.


Failure:
Lint#test_0001_contains only lower-case suffixes [/Users/weppos/Code/publicsuffix-list/test/lint_test.rb:15]
Minitest::Assertion: List contains non-lowercase suffix at line 28: mil.AE
.
Expected 28 to be nil.

1 tests, 1 assertions, 1 failures, 0 errors, 0 skips

➜  publicsuffix-list git:(testsuite) ✗ bundle exec rake -v
/Users/weppos/.rvm/rubies/ruby-2.3.0/bin/ruby -I"lib:test" -I"/Users/weppos/.rvm/gems/ruby-2.3.0/gems/rake-10.5.0/lib" "/Users/weppos/.rvm/gems/ruby-2.3.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb" "test/lint_test.rb"

# Running tests with run options --seed 6406:

.F

Finished tests in 0.015975s, 125.1984 tests/s, 125.1984 assertions/s.


Failure:
Lint#test_0002_does not contain leading dots [/Users/weppos/Code/publicsuffix-list/test/lint_test.rb:23]
Minitest::Assertion: List contains a leading dot at line 11079: .sk.eu.org
.
Expected 11079 to be nil.

2 tests, 2 assertions, 1 failures, 0 errors, 0 skips

➜  publicsuffix-list git:(testsuite) ✗ bundle exec rake -v
/Users/weppos/.rvm/rubies/ruby-2.3.0/bin/ruby -I"lib:test" -I"/Users/weppos/.rvm/gems/ruby-2.3.0/gems/rake-10.5.0/lib" "/Users/weppos/.rvm/gems/ruby-2.3.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb" "test/lint_test.rb"

# Running tests with run options --seed 22588:

.F.

Finished tests in 0.015931s, 188.3080 tests/s, 188.3080 assertions/s.


Failure:
Lint#test_0002_does not contain leading spaces [/Users/weppos/Code/publicsuffix-list/test/lint_test.rb:27]
Minitest::Assertion: List contains a leading space at line 23:  co.ae.
Expected 22 to be nil.

3 tests, 3 assertions, 1 failures, 0 errors, 0 skips

[rockdaboot]

On Travis, this branch took around 12 seconds to be tested, which is still better than the existing C-based test suite that runs in around 40 seconds and doesn't test the list style.

The PSL checks of libpsl are kind of a by-product when building it. Building takes a full autoreconf, ./configure, make, make check run which never can beat a script language test.

libpsl (make check) performs the tests in test_psl.txt, but also tests each entry in the PSL to be found via a checkPublicSuffix() function (or not to be found if is an exception). Wildcards are taken care of. But I think that is beyond a syntax checker / linter.

I am not Ruby affine, so can't easily see what is already implemented and what not. So please if I list some checks that already exist (just spontaneous ideas):

• Check for valid domain labels (e.g. valid UTF-8, else valid ASCII domain characters, lowercase (also UTF-8 chars))
• Check for punycode (not 'allowed' right now)
• Check for empty labels (two consecutive dots, dot as last character)
• Check for double entries
• Check for ambiguity (e.g. example.com and !example.com can't co-exist)

[weppos]

Thanks for your feedback @rockdaboot!

libpsl (make check) performs the tests in test_psl.txt, but also tests each entry in the PSL to be found via a checkPublicSuffix() function (or not to be found if is an exception).

This is interesting. I've seen the same in the Go test suite. I believe this is what caused this build to fail yesterday, I still have to properly investigate the issue. If you have any idea, feel free to share. I wonder if it's an issue with the test of the list itself.

Check for valid domain labels (e.g. valid UTF-8, else valid ASCII domain characters, lowercase (also UTF-8 chars))

It currently checks for:

• lowercase
• non spaces within a rule
• non leading spaces
• non leading dots in a rule

I'm not currently checking for UTF-8 but I can try to add it.

Check for punycode (not 'allowed' right now)

Good point, I'll add a check.

Check for empty labels (two consecutive dots, dot as last character)

I currently check only for a trailing dot, but you made a very good point here.

Check for double entries

That's reasonable.

Check for ambiguity

This is a little bit more tricky. I can definitely add it later on.

---

Thanks for your feedback @rockdaboot, that's really valuable.

One more thing. The current design is fail-fast. I run a bunch of tests, and the first entry that fails is reported. Does it make sense, or should the tool return all the issues with all the occurrences?

The current approach is a little bit faster, and it's also very test-oriented. The second approach would be closer to a linter, and potentially reusable to validate any PSL (whereas the current one is very specific to this repo).

[rockdaboot]

This is interesting. I've seen the same in the Go test suite. I believe this is what caused this build to fail yesterday, I still have to properly investigate the issue. If you have any idea, feel free to share. I wonder if it's an issue with the test of the list itself.

I am sure it is the entry '..githubcloudusercontent.com'. This is the first wildcard with two 'star' labels - libpsl code is not ready to deal with it - but it should be a quick fix (hopefully within the next days).

One more thing. The current design is fail-fast. I run a bunch of tests, and the first entry that fails is reported. Does it make sense, or should the tool return all the issues with all the occurrences?

For a test tool fail-fast makes sense, especially when the test data (PSL) evolves slowly - as it does.
But as a the mid-term goal, a linter-like behavior is a useable feature.

[weppos]

I am sure it is the entry '..githubcloudusercontent.com'. This is the first wildcard with two 'star' labels - libpsl code is not ready to deal with it - but it should be a quick fix (hopefully within the next days).

I supposed that was the cause. Apparently, we never had a double wildcard so far in the list. However, according to the PSL definition, it seems to be a valid rule.

From this page:

..foo is a valid rule: *bar.foo is not.

I'll open a separate ticket for the double wildcard rule, so that we can get feedback from @gerv and @sleevi if it doesn't break any existing implementation, and we can keep track of the related changes in the various libs.

For a test tool fail-fast makes sense, especially when the test data (PSL) evolves slowly - as it does.
But as a the mid-term goal, a linter-like behavior is a useable feature.

Thanks for your feedback. I hope to collect more feedback in the next days. I'm already working to think about a possible design for a linter, as I agree that it can be a desirable feature.

[rockdaboot]

From this page:

..foo is a valid rule: *bar.foo is not.

@sleevi asked me to use a DAFSA implementation as in Chromium (libpsl issue #35). This is a dictionary lookup that takes a domain as input and returns a 'flags' value. One of these flags is 'WILDCARD'. My conclusion is, that the Chromium implementation just works with one single kind of wildcard, namely '.foo', not '.*.foo'. It seems to me that a change is non-trivial. So IMO, using a double (or multiple) wildcard breaks existing implementations and should be avoided.

[weppos]

@rockdaboot I collected your proposals here https://github.com/weppos/pslint/wiki/Rules

That repo is a Linter prototype I'm working on. I realized that it will probably take a little extra work upfront to create a proper linter, but in the long run it's better rather than doing it as a test and then converting it into a linter in the next months.

The linter will provide --fail-first and --fail-fast options that can be used when running from a test suite.

If you have more ideas for the rules, feel free to edit the wiki. Thanks again for the feedback!

[rockdaboot]

@weppos I can't edit the wiki :-)

Not sure, are going to use Go or Ruby ?

A few things needs to be made clear regarding PSL rules:

• Is xyz.*.example.com a valid PSL rule (why not ?)
• Is !xyz.*.example.com a valid PSL rule ? If you have a ..example.com this could make sense

I guess, we are all waiting for DBOUND.
How much work should go into a linter when we assume the PSL being kicked in the (near ?) future.

[rockdaboot]

@weppos
Just wrote a PSL linter in Python: https://github.com/rockdaboot/libpsl/blob/develop/src/pslint.py

It is pretty complete regarding your list of tests (on your wiki page).
WDYT ?

And please don't hesitate to comment the code - I am C and Java coder and this is my first self-written Python code (had to ask the web a lot this afternoon).

[weppos]

@rockdaboot I apologize for the silence, I had a couple of intense weeks.

I opened #145 to discuss the wildcard issues. Thanks for pointing it out.

As for DBOUND, it's still enough far in the future to make this list reasonably relevant today. Moreover, the time currently required to manage the list justifies (to me) a little effort to get some tool to help with it.

I appreciate your help with the Python script. My main concern with it is that it's not tested (indeed it's not a lack of trust in your coding skills 😉 ), which reduces my confidence in any tool especially when it has to deal with a variety of possible bad combinations.

This is essentially the same issue I had while working on an initial linter embedded in the test suite itself. See Test the tests. That's why I decided to work on a separate tool, with the goal to have a general purpose, well tested linter we can reuse if needed.

I must admit I'm not familiar with Python either, which makes me a little bit worried for the maintenance and for the result of any commit I could make on the script. Moreover, and this is a practical issue, if we should switch Travis to run the build with the Python container, we'll have to write the tests with a Python implementation of the list (as we can't have two different languages on the same container).

I was able to make progress on the pslint tool, and yesterday I replaced the Ruby linter prototype with this tool. This PR is now ready for a review.

The current state is:

1. A commit triggers a Travis run
2. Travis downloads a binary version of the pslint and run it against the new list (I can't compile it on Travis, because of no double language support). If the test fails, it reports the lint errors and quit the build.
3. Travis runs the PSL tests against the new list

Dependencies:

• The lint process is run with the the pslint. It's a Go app, but it can be compiled in a standalone binary with no dependencies. I compiled a linux version and made it available to Travis.
• The test are run via Ruby. I had no specific preference, but the requirement was to be able to load an arbitrary list at runtime and test against it. I have no knowledge of other libs that can do it as of today (Go certainly not, as it embeds the list in the code itself).

Linter status:

The linter currently covers these rules. Ideally, all "Proposed" rules will be converted into a check. I made the code as simple as possible to encourage contributions, the check definition is in a separate file. Each test method has a resonable test suite that covers all the various scenario.

func TestLinter_CheckRuleLowercase_Cases(t *testing.T) {
    linter := &Linter{}
    var inputs []string

    inputs = []string{
        "",
        "foo",
        "// a comment",

        ".bad.suffix",
        "// A comment", // ignore comments even if mixed case
    }
    for _, input := range inputs {
        line := &line{source: input, number: 2}
        problem, err := linter.checkRuleLowercase(line)
        if err != nil {
            t.Errorf("checkRuleLowercase('%v') returned error: %v", input, err)
        }
        if problem != nil {
            t.Errorf("checkRuleLowercase('%v') should pass", input)
        }
    }

    inputs = []string{
        "mixedCase",
        "mixed.Case",
        "mixed.caSe",
    }
    for _, input := range inputs {
        line := &line{source: input, number: 2}
        problem, err := linter.checkRuleLowercase(line)
        if err != nil {
            t.Errorf("checkRuleLowercase('%v') returned error: %v", input, err)
        }
        if problem == nil {
            t.Errorf("checkRuleLowercase('%v') should NOT pass", input)
        }
    }
}

Long story short, the PR is ready to be merged. @gerv @sleevi what's your position here?

[weppos]

@rockdaboot I forgot to mention I'd be more than happy to add you as a collab to the linter, if you feel you want to play with some Go. I really appreciate all your help and feedback on this PR and I'm sad we end up working in parallel to two different solutions.

I'm also open if we want to move it elsewhere (but I'll leave the decision to @gerv, for now I just created it under my account as it is where I was most comfortable with given it was an independent idea to start working on it).

[rockdaboot]

@weppos The python linter was just an exercise (me learning python), it took me less than a day - and I basically spent this day on learning python. So nothing wasted. Don't be 'sad' - see it positively: a second solution always gives a benchmark. Also, some of the ideas I put on your wiki came when I was working on the linter implementation.
I am pretty busy these days, but I'll have a closer look at Go anyways - I'll drop you a note when I feel that I could be of any help.

Just some quick notes:

• Self-testing the python linter just needs some test PSL files (of course I used such for testing, but more randomly). Just not sure if they should go into publicsuffix/list. BTW, the python linter just has 145 lines of code (used the 'cloc' tool) to implement all. Beat this with any other language.
• Python needs no compilation/linkage which you originally pointed out as slow and clumsy. I agree and would add 'clumsy'.
• You are wrong regarding Travis. Python is also available in the C container. If it weren't, simply install it via apt-get.
• The linter and the 'implementation' (however you define that) are two different things and should be kept seperate, IMO.
• Downloading an executable (Go program compiled) sounds like a very hackish approach. I would prefer a script language program for inclusion into publicsuffix/list, so you wouldn't need a second repository/project.
• "The test are run via Ruby". That seems to add unneeded dependencies. To me it seems pretty easy to implement loading PSL + lookups in Python.

And last but not least... give yourself a kick and have a closer look into python. I completely disliked it after the first few glimpses. I already missed it... but on the second look I discovered a really helpful tool. And I looked at dozens of languages in the last 30 years - and until now only three convinced my: Assembler, C, and Java. Now there is Python (the first script language in my list)... I now time is always a limited resource, but IMHO it is worth spending some on Python.

[sleevi]

@weppos - Are there specific questions/thoughts for @gerv and I? I think I'd agree with @rockdaboot that downloading a pre-compiled binary is non-ideal; I thought we would just be able to use Travis w/ the Go compiler/runtime and run it as part of the lint step.

With regards to the current Ruby setup, I'm a little nervous with the test file and testing against the real PSL (or perhaps I've misread). At least the Chrome approach to testing our PSL implementation has been to supply a 'dummy PSL' and then test known answers against the algorithm (to make sure we're implementing the algorithm correctly). Perhaps I've misread tests/test.txt

My thought on the approach would be something like:

• Linter script (that implements the checks @rockdaboot suggested and which you had)
• Test input file for each thing to be checked
• Expected test results

For example, tests/trailing_whitespace, tests/trailing_whitespace_golden, tests/double_wildcard, tests/double_wildcard_golden, tests/utf8, etc, where the _golden files contain just the expected stdout.

I have no opinions on Ruby vs Python; personally, I find Python easier, but that's a taste thing, and I think if you're willing to do the work in Ruby, there's no reason I can't learn and adapt :)

[weppos]

I thought we would just be able to use Travis w/ the Go compiler/runtime and run it as part of the lint step.

The problem is that in order to do that I have to switch to use the Travis Go container, which doesn't have the Ruby interpreter. Vice-versa, the current Ruby container doesn't have the Go binary.

I'll be working on a Go public suffix library in the future in order to deal with some issues, but it won't be ready in the next days and I'd we can at least have a working implementation of the tests sooner rather than later.

With regards to the current Ruby setup, I'm a little nervous with the test file and testing against the real PSL (or perhaps I've misread). At least the Chrome approach to testing our PSL implementation has been to supply a 'dummy PSL' and then test known answers against the algorithm (to make sure we're implementing the algorithm correctly). Perhaps I've misread tests/test.txt

The test file has been there for ages, so I just reused the conventions we had so far. I can test using a dummy file, but at that point I guess the test will essentially be testing the library that deals with the PSL, rather than the PSL itself. If I recall correctly, one of the goal to test against the real PSL was to make sure that the PSL was not corrupted. I guess this goal is accomplished by the linter now.

At that point, we can probably totally remove the test files, but I'd still like to have a sort of real test cases. As for the Ruby thing, as I mention, it will eventually be replaced by a pure Go lib (I think), the thing is that I can't commit on a timing for that and I really wish we have something today, rather than keep trying to get the best solution that actually resulted (as of today) to not have anything.

Test input file for each thing to be checked

Can you clarify which kind of tests should be expected at this stage?

For example, tests/trailing_whitespace, tests/trailing_whitespace_golden, tests/double_wildcard, tests/double_wildcard_golden, tests/utf8, etc, where the _golden files contain just the expected stdout.

If I understood correctly, this should be tested by the linter as they are potential formatting issues, not really rules acceptance issues. Am I correct?

Also, as for the "Ruby" issue, I wonder if you guys actually checked what's inside this PR. It doesn't really require any further maintenance, as it's just loading the expectations from the existing test file (I converted the current tests/test_psl.txt into a file without named functions) and it's running it. As I said, I expect this to be replaced once I have a Go lib that will provide the same flexibility in terms of being able to load ab arbitrary list (at this time I use the PSL itself) to test against.

[rockdaboot]

@weppos I am sorry, but I don't understand what your Go implementation will have that libpsl doesn't provide. The library comes with a tool 'psl' that allows you to load arbitrary PS lists and do any checks controlled by command line options. Library and tool are packaged by Debian, Redhat, Arch (maybe more, libpsl belongs to the basic install of Debian).
What you are going to do is reinventing the wheel... why not using 'psl' for your functionality tests ? If anything is missing, just drop me a note.
Or is there any specific reason why you need a Go implementation ?

The linter needs test files (PSL files), good ones and bad ones to perform self-tests. They should reflect all those test cases that are listed in your wiki. I guess that's what @sleevi is talking about.

Example bad test case 'test_group_order.psl':

// ===BEGIN ICANN DOMAINS===

// ck : http://en.wikipedia.org/wiki/.ck
!www.ck
*.ck
abc.www.ck
*.a.www.ck

// ===END ICANN DOMAINS===

Possible linter output:

8: warning: Incorrectly sorted group of domains
  [['ck', '!www'], ['ck', '*'], ['ck', 'www', 'abc'], ['ck', 'www', 'a', '*']]
  [['ck', '*'], ['ck', '!www'], ['ck', 'www', 'abc'], ['ck', 'www', 'a', '*']]
Correct sorting would be
*.ck
!www.ck
abc.www.ck
*.a.www.ck

[weppos]

Or is there any specific reason why you need a Go implementation ?

Yes, I need a Go implementation for other uses. This is not specific to this case, I was just considering reusing it as a way to ditch the Ruby dependency (given the linter is in Go).

As you probably understood, I'm not familiar with C. I know we can use all the languages of this world to solve this problem, but frankly speaking nobody took care of #11 in these years (plus all the various connected issues still open in Bugzilla), hence I went ahead and used tools and languages I'm familiar with.

The linter needs test files (PSL files), good ones and bad ones to perform self-tests.

I already have specs for each test that is included today.

---

I will keep working on the linter according to my availability.

Regarding this specific PR, my desire is to have a test suite that covers all the tests cases defined as of today (and possible further updates for e.g. based on the wildcard discussion), plus the linter. This PR does it, any other solutions we had so far don't. If any of you want to work on a different solution that covers all these cases and is willing to maintain it for the foreseeable future, I'll be happy to hand off the work. Otherwise, I suggest we should merge this as a starting point and evaluate further improvements later on.

[rockdaboot]

@weppos I would like to join forces.

I am not against Go, I am just against reinventing the wheel. Go is able to call C library routines (cgo) - is this is an option for you ? (see http://stackoverflow.com/questions/1760468/interface-go-with-c-libraries).

Regarding the python linter... we could move it into your repo, else I would like to keep it as part of libpsl source package and would add all those test files. Also I will maintain libpsl and the linter - see this as an opportunity to get more free time for other things.

[rockdaboot]

Have to amend my writing about the linter: As a script, I would like to see it being part of publicsuffix/list repo, including the test files. So easy use with Travis, the self-checks shouldn't take longer than a glimpse. Anybody forking the repo to generate a PR automatically has the linter available to test his/her changes before committing.

[weppos]

FYI the liter I have been working on is both a bin and a lib. I am working
on some tools to simplify/automate the process of approving/validating PRs,
and I may need to call the linter from the other tools. Moreover, the list
can be considered a specification (given that you could create other
arbitrary lists using the same format and feed them to apps) hence it may
be handy to have some tool outside the main repo that anybody can pull and
reuse.

I will continue to work on that in any case. I honestly don't feel the need
to duplicate the work in Python as well. The intent is broader to simply
validate this list. The goal is to offer a tool to validate any string that
can be interpreted as PSL rules.

As for the tests, I can look into loading the libpsl into go and use a
single container type. It's an interesting option. But it will take me
probably at least until the next week.

My question is whether this is a blocking issue for this PR or an
improvement. To me it seems the latter, not the former. I am currently
traveling back after a business trip. I'll look into what it takes to use
the libpsl in go when I'll be back.

On Thursday, 18 February 2016, Tim Rühsen notifications@github.com wrote:

Have to amend my writing about the linter: As a script, I would like to
see it being part of publicsuffix/list repo, including the test files. So
easy use with Travis, the self-checks shouldn't take longer than a glimpse.
Anybody forking the repo to generate a PR automatically has the linter
available to test his/her changes before committing.

—
Reply to this email directly or view it on GitHub
#123 (comment).

Simone Carletti
Passionate programmer and dive instructor

https://simonecarletti.com/
Twitter: @weppos https://twitter.com/weppos

[rockdaboot]

@weppos Despite from any discussion about this PR I made some improvements regarding test PSL files to test (any) PSL linter with see here.

There is pslint_selftest.sh, a shell script that calls the linter with each PSL file (.input) and compares it's output with the expected output (.expected). The input files should cover all the test cases from your wiki.

Feel free to use anything from there for your own tests - the *.expected files have to be amended to match the output of your linter, of course. Most input files cover more than one test case.

BTW, I took a look at Go syntax today... well it seems pretty close to Python (and even C). It should be pretty straight forward to convert code Go<->Python. I'll give it a try in the next days.

[weppos]

@rockdaboot this discussion is sucking all my remaining energies, and I need something sooner rather than later. I don't want to continue a this-language or that-language discussion, and we are clearly doubling the work as of today.

So let's move in this way. Open a PR with your proposal to replace this PR with:

1. a full test suite that tests the PSL being submitted against the test list (possibly in this format because the old one we had is too focused on specific languages)
2. a tested linter that works and stops the build if the submitted PSL is invalid

If the PR is better than this one (and ideally I'd love to be able to make small maintenance tasks on it, but that's a plus), then I'll be happy to merge it and move forward. Otherwise, I will merge this one (remaining open to further improvements or replacements later on).

I don't want to appear rude, but I've dozens of PRs waiting to be merged, and I need to adopt a solution that works.

As I said, I also need a reusable linter and also a Go library for the PSL in any case, hence I'll continue working on it, but in that case I will be free to proceed without being constrained by the PSL.

[rockdaboot]

...but in that case I will be free to proceed without being constrained by the PSL

That was my intention. Maybe I wasn't clear enough.

1. a full test suite that tests the PSL being submitted against the test list

We already have that right now on Travis CI. When building libpsl and performing 'make check', those test are made against the current PSL. Also a few more tests, like testing each entry in the PSL against the PSL with special care taken for exceptions and wildcards. Also the test.txt file will be linted (as a by-product).
I don't see any win in rewriting libpsl in any other language right now. It will compile with any C89 compiler and just needs an IDN library. E.g. regularly tested on a OpenSolaris build farm (Sparc and X86).

1. a tested linter that works and stops the build if the submitted PSL is invalid

That is PR #130. Right now, it replaces the libpsl test suite. But if you agree, I amend the PR to first run the linter and on success run the libpsl test suite.

[weppos]

We already have that right now on Travis CI.

As I mention, I'd like the test to be pulled from a simpler text file (see the one in this PR) that reflects a list of possible input/output, rather the test_psl file (assuming libpsl uses that file).

We should be able to add more entries to the file to simply append more tests.

That is PR #130. Right now, it replaces the libpsl test suite. But if you agree, I amend the PR to first run the linter and on success run the libpsl test suite.

Yes, the tests should not be replaced. Also, I need a confidence that the linter is properly tested. That was not in #130 as the linter was a script added with no corresponding tests.

[rockdaboot]

... be pulled from a simpler text file ...

ACK. I don't like test_psl.txt format either. libpsl just strips the checkPublicSuffix away. I'll update libpsl test suite to accept both formats. Any naming preference ?

Yes, the tests should not be replaced.

I'll amend this tomorrow.

Also, I need a confidence that the linter is properly tested.

The linter self test is included since today and executed, see .travis.yml (./pslint_selftest).
It's very easy to extend it, just small PSL files with test cases.

[rockdaboot]

@weppos It's all in PR #130 now.

I added tools/convert_tests, a one-line shell script to convert test_psl.txt to tests.txt - just in case the sed command is needed again. In tests.txt the test cases are separated by a single space (' ') to make parsing as easy as possible.

The Travis YML does linting + further checking via libpsl.

Libpsl is prepared to take the new tests.txt format in it's test suite. I will prepare a new release as soon as you pull the PR into master.

[weppos]

Closing in favor of #130