Sync to linguist 7.2.0: heuristics.yml support

[bzz]

Fixes part of the #155 - generate heuristics from heuristics.yml instead of parsing heuristics.rb.

Major code changes include:

• update code generator for heuristics strategy in ./internal/code-generator/heuristics.go to consume heuristics.yml instead of heuristics.rb and produce matchable rule tree
• re-generated all datastructures in ./data/
• re-generated gold test results in ./internal/code-generator/test_files/*.gold

TODOs:

• update linguist version & re-generate data
• adapt content heuristics generator for consuming new heuristics.yml
• update doc in README
• match language Alias resolution to github linguist one
• fix new Classifier strategy failures -> moved to Bayesian classifier cann't distinguish "SQL" vs "PLpgSQL" #194
• fix new ByExtension strategy failures
• re-generate .gold test fixtures
• make CI fail on failing tests 2348fd2
• verify godoc presence for new things (cleanup \w docs everything else Ensure exported names in Go packages are properly documented #195)

[bzz]

Got back from vacation and keep debugging the case of failing Bayesian classifier for content on samples/SQL/drop_stuff.sql - SQL (linguist) vs PLpgSQL (enry).

Seems like this has to do with difference in how linguist and enry tokenize the content.

Linguist

Adding to test/test_classifier.rb

  def test_classify_sql
    results = Classifier.classify(Samples.cache, fixture("SQL/drop_stuff.sql"), ["PLpgSQL", "SQL", "PLSQL", "SQLPL"])
    assert_equal "SQL", results.first[0]
  end

$ LINGUIST_DEBUG=2 bundle exec rake test TEST=test/test_classifier.rb TESTOPTS="--name=test_classify_sql -v"

                    #     PLSQL   PLpgSQL       SQL     SQLPL
               ;   10     19.914    22.974         -    21.265
            body    1      7.934         -     5.638         -
         cascade    1          -         -     5.638     8.044
            drop   10          -   102.321    79.408         -
        function    1      9.032    10.969     5.638         -
   functionname1    1          -         -     5.638         -
linguist_package    2          -         -    12.663         -
         package    2     17.254         -    12.663         -
       procedure    1      8.850         -     5.638     9.431
           table    2     16.678    15.632    14.049         -
            type    2      3.569     1.810         -     1.593
       typename1    1          -         -     5.638         -
       typename2    1          -         -     5.638         -
            view    2          -         -    13.474         -
       viewname1    1          -         -     5.638         -
       viewname2    1          -         -     5.638         -
   who_called_me    1      7.934         -     5.638         -
               x    1          -         -     6.331         -
               y    1          -         -     6.331         -

   PLpgSQL =   -330.972 +  -5.697 =   -336.668
       SQL =   -283.377 +  -5.158 =   -288.535
     PLSQL =   -393.512 +  -5.563 =   -399.075
     SQLPL =   -444.345 +  -5.851 =   -450.196

Enry

$ ENRY_DEBUG=1 ENRY_TEST_REPO="$PWD/.linguist" go test -run '^TestRegexpEdgeCases$'

                    #     PLSQL   PLpgSQL       SQL     SQLPL
               ;   10     34.270    37.049         -    35.420
            body    1      7.990         -     4.405         -
         cascade    1          -         -     4.405     8.217
            drop   10          -   103.600    67.077         -
        function    1      9.936    11.097     4.405         -
    functionname    1          -         -     4.405         -
linguist_package    2          -         -    10.197         -
         package    2     17.366         -    10.197         -
       procedure    1      8.906         -     4.405     9.603
           table    2     17.366    15.888    11.583         -
            type    2      7.169     5.554         -     5.426
        typename    2          -         -    10.197         -
            view    2          -         -    11.008         -
        viewname    2          -         -    10.197         -
   who_called_me    1      7.990         -     4.405         -
               x    1          -         -     5.098         -
               y    1          -         -     5.098         -

   PLpgSQL =   -334.671 +  -5.693 =   -340.364
       SQL =   -340.778 +  -5.154 =   -345.932
     SQLPL =   -449.195 +  -5.847 =   -455.042
     PLSQL =   -396.868 +  -5.559 =   -402.427

As seen above, Bayesian classifier token weights for SQL are very different for the same language disambiguation case.

Resolution: fixing this is tacked under #194

[bzz]

CI passes, although there clearly are failing tests :/

--- FAIL: TestRegexpEdgeCases (0.00s)
	require.go:794: 
			Error Trace:	common_test.go:45
			Error:      	Received unexpected error:
			            	open .linguist/samples/ActionScript/FooBar.as: no such file or directory
			Test:       	TestRegexpEdgeCases
=== RUN   Test_EnryTestSuite
Cloning Linguist repo to '/tmp/linguist-735567248' as ENRY_TEST_REPO was not set

scope in PR description updated.

[bzz]

@creachadair @juanjux thank you for taking a look while it's still WIP - all initial feedback addressed in 5fbadc8

[bzz]

What is super annoying is that make test does fail on my local env but not on CI

2019/01/28 20:23:37 exit status 128
FAIL	gopkg.in/src-d/enry.v1	0.062s

with a different output, then on CI.

Allthough it clearly should fail on CI as well, as test do not pass

=== RUN   TestRegexpEdgeCases
--- FAIL: TestRegexpEdgeCases (0.00s)
	require.go:794: 
			Error Trace:	common_test.go:45
			Error:      	Received unexpected error:
			            	open .linguist/samples/ActionScript/FooBar.as: no such file or directory
			Test:       	TestRegexpEdgeCases

[bzz]

@creachadair @juanjux all feedback addressed, tests pass, ready to be merged.

Sorry for such a long set of changes but scope of this PR was already limited to only a single part of the original #152 (see it's description for updated full scope of the github<->linguist sync)

[bzz]

@creachadair feedback addressed in c57bc4a and c4f3dbe

[bzz]

Thank you for prompt reviews @juanjux, @creachadair 🚀

Also thanks for kind explanations and rising the concerns about public API structure, @creachadair !
Will address those first thing tomorrow and let you know.

[bzz]

All feedback addressed, @creachadair it's ready for another round 🙏

[bzz]

@creachadair thank you for your kind and useful feedback, I belive it all has been addressed and is ready for another pass.

[bzz]

New v7.2.0 of linguist has been released, it includes a fix to one of the issues that affected us so will bump to e4560984058b4726010ca4b8f03ed9d0f8f464db