Put the github.com/BurntSushi/toml-test in internal/ (#313)

The problem is as follows:

- BurntSushi/toml is a TOML library.

- BurntSushi/toml-test is a generic language-agnostic TOML testing
  framework that uses this toml library.

- toml-test is based on a CLI interface. This works well, but for this
  library (since it's written in Go) it's actually quite a bit more
  convenient to integrate this in "go test"; it's more flexible, easier,
  and gives us stuff like code coverage.

- This introduces a bit of a weird dependency scenario:

  1. toml depends on toml-test
  2. toml-test depends on toml
  3. Because toml-test is only referred to from the toml_test package,
     this should actually be fine (similar to how you can use the _test
     package to work around other cyclic dependency issues).

- For v0.4.0 the dependencies aren't quite "correct"; it depends on
  toml-test facb9eccd4da, which in turn depends on toml 20a94d6.
  This isn't *necessarily* a problem as such as this can be resolved,
  but commit 20a94d6 on toml is not referenced at all and orphaned:
  I updated the tests in sync with the feature I was working on, and
  then later rebased the lot it never ended up in the master branch

  All of this happens to work when using the standard proxy.golang.org
  GOPROXY because it has that commit in the cache, but right not it
  doesn't work with GOPROXY=direct as that doesn't have any cache (and a
  private GOPROXY won't either).

This should be fixed by just tagging the current master as v0.4.1, as
that now refers to toml-test v1.0.0, which refers to toml v0.4.0. I'm
not super-happpy with that solution as such, because I can see it break
in the future.

So instead, just copy the toml-test package to internal/toml-test here
to get rid of any cyclic module dependency.

More details can be found in toml-lang/toml-test#74

One downside is that updating this is a bit awkward now. That's okay for
the time being and only affects me, and this doesn't need updating all
that often anyway.

Another downside is:

	[~c/toml](master)% du -hd1 internal/
	1.7M    internal/toml-test
	12K     internal/tag
	1.7M    internal/

1.7M is kind of a lot. But then again, it's required for running the
tests, and all of it is actual test cases. They would be in *_test.go
files otherwise anyway.

The version in the go.mod is updated because without it:

	[~c/toml](dep)% go test ./...
	# github.com/BurntSushi/toml/internal/toml-test [github.com/BurntSushi/toml.test]
	internal/toml-test/runner.go:27:3: go:embed requires go1.16 or later (-lang was set to go1.13; check go.mod)
	# github.com/BurntSushi/toml/internal/toml-test
	internal/toml-test/runner.go:27:3: go:embed requires go1.16 or later (-lang was set to go1.13; check go.mod)
	FAIL    github.com/BurntSushi/toml [build failed]
	?       github.com/BurntSushi/toml/cmd/toml-test-decoder        [no test files]
	?       github.com/BurntSushi/toml/cmd/toml-test-encoder        [no test files]
	?       github.com/BurntSushi/toml/cmd/tomlv    [no test files]
	?       github.com/BurntSushi/toml/internal     [no test files]
	?       github.com/BurntSushi/toml/internal/tag [no test files]

It doesn't "see" that these files are protected by a go1.16 build tag.

It should still work for older versions of Go though, just running these
tests won't, but that was already the case (toml_test.go has a go1.16
build tag). I also had to add a build tag to the Go files in
toml-test.go, since it won't be able to find the embed and io/fs imports
on older versions of Go.

This also adds GOPROXY=direct in the CI. There aren't any dependencies
in go.mod now, but this avoids depending on the peculiarities of
proxy.golang.org, and is probably a good idea for most Go CIs.

.github/workflows/test.yml

@@ -10,6 +10,7 @@
         }
       },
       "runs-on": "${{ matrix.os }}",
+      "env": {"GOPROXY": "direct"},
       "steps": [
         {
           "name": "Install Go",

go.mod

@@ -1,5 +1,3 @@
 module github.com/BurntSushi/toml
 
-go 1.13
-
-require github.com/BurntSushi/toml-test v1.0.0
+go 1.16

go.sum

@@ -1,12 +0,0 @@
-github.com/BurntSushi/toml v0.3.2-0.20210614224209-34d990aa228d/go.mod h1:2QZjSXA5e+XyFeCAxxtL8Z4StYUsTquL8ODGPR3C3MA=
-github.com/BurntSushi/toml v0.3.2-0.20210621044154-20a94d639b8e/go.mod h1:t4zg8TkHfP16Vb3x4WKIw7zVYMit5QFtPEO8lOWxzTg=
-github.com/BurntSushi/toml v0.3.2-0.20210624061728-01bfc69d1057/go.mod h1:NMj2lD5LfMqcE0w8tnqOsH6944oaqpI1974lrIwerfE=
-github.com/BurntSushi/toml v0.3.2-0.20210704081116-ccff24ee4463/go.mod h1:EkRrMiQQmfxK6kIldz3QbPlhmVkrjW1RDJUnbDqGYvc=
-github.com/BurntSushi/toml v0.4.0/go.mod h1:wtejDu7Q0FhCWAo2aXkywSJyYFg01EDTKozLNCz2JBA=
-github.com/BurntSushi/toml-test v0.1.1-0.20210620192437-de01089bbf76/go.mod h1:P/PrhmZ37t5llHfDuiouWXtFgqOoQ12SAh9j6EjrBR4=
-github.com/BurntSushi/toml-test v0.1.1-0.20210624055653-1f6389604dc6/go.mod h1:UAIt+Eo8itMZAAgImXkPGDMYsT1SsJkVdB5TuONl86A=
-github.com/BurntSushi/toml-test v0.1.1-0.20210704062846-269931e74e3f/go.mod h1:fnFWrIwqgHsEjVsW3RYCJmDo86oq9eiJ9u6bnqhtm2g=
-github.com/BurntSushi/toml-test v0.1.1-0.20210723065233-facb9eccd4da/go.mod h1:ve9Q/RRu2vHi42LocPLNvagxuUJh993/95b18bw/Nws=
-github.com/BurntSushi/toml-test v1.0.0 h1:auSA806b0nBApu9BNK0waBUzaydmHd9TpLIM1UFkvKg=
-github.com/BurntSushi/toml-test v1.0.0/go.mod h1:XDBCig4BwVX4bO0Ei0EU51EJ6DamtW+xOoH7+Vkhlz0=
-zgo.at/zli v0.0.0-20210619044753-e7020a328e59/go.mod h1:HLAc12TjNGT+VRXr76JnsNE3pbooQtwKWhX+RlDjQ2Y=

internal/toml-test/.gitignore

@@ -0,0 +1,2 @@
+/dist
+/toml-test

internal/toml-test/.gogo-release

@@ -0,0 +1 @@
+build_flags="$build_flags ./cmd/toml-test"

internal/toml-test/COPYING

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2018 TOML authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

internal/toml-test/README.md

@@ -0,0 +1,258 @@
+`toml-test` is a language-agnostic test suite to verify the correctness of
+[TOML][t] parsers and writers.
+
+Tests are divided into two groups: "invalid" and "valid". Decoders or encoders
+that reject "invalid" tests pass the tests, and decoders that accept "valid"
+tests and output precisely what is expected pass the tests. The output format is
+JSON, described below.
+
+Both encoders and decoders share valid tests, except an encoder accepts JSON and
+outputs TOML rather than the reverse. The TOML representations are read with a
+blessed decoder and is compared. Encoders have their own set of invalid tests in
+the invalid-encoder directory. The JSON given to a TOML encoder is in the same
+format as the JSON that a TOML decoder should output.
+
+Compatible with TOML version [v1.0.0][v1].
+
+[t]: https://toml.io
+[v1]: https://toml.io/en/v1.0.0
+
+Installation
+------------
+There are binaries on the [release page][r]; these are statically compiled and
+should run in most environments. It's recommended you use a binary, or a tagged
+release if you build from source especially in CI environments. This prevents
+your tests from breaking on changes to tests in this tool.
+
+To compile from source you will need Go 1.16 or newer (older versions will *not*
+work):
+
+    $ git clone https://github.com/BurntSushi/toml-test.git
+    $ cd toml-test
+    $ go build ./cmd/toml-test
+
+This will build a `./toml-test` binary.
+
+[r]: https://github.com/BurntSushi/toml-test/releases
+
+Usage
+-----
+`toml-test` accepts an encoder or decoder as the first positional argument, for
+example:
+
+    $ toml-test my-toml-decoder
+    $ toml-test my-toml-encoder -encoder
+
+The `-encoder` flag is used to signal that this is an encoder rather than a
+decoder.
+
+For example, to run the tests against the Go TOML library:
+
+    # Install my parser
+    $ go install github.com/BurntSushi/toml/cmd/toml-test-decoder@master
+    $ go install github.com/BurntSushi/toml/cmd/toml-test-encoder@master
+
+    $ toml-test toml-test-decoder
+    toml-test [toml-test-decoder]: using embeded tests: 278 passed
+
+    $ toml-test -encoder toml-test-encoder
+    toml-test [toml-test-encoder]: using embeded tests:  94 passed,  0 failed
+
+The default is to use the tests compiled in the binary; you can use `-testdir`
+to load tests from the filesystem. You can use `-run [name]` or `-skip [name]`
+to run or skip specific tests. Both flags can be given more than once and accept
+glob patterns: `-run 'valid/string/*'`.
+
+See `toml-test -help` for detailed usage.
+
+### Implementing a decoder
+For your decoder to be compatible with `toml-test` it **must** satisfy the
+expected interface:
+
+- Your decoder **must** accept TOML data on `stdin` until EOF.
+- If the TOML data is invalid, your decoder **must** return with a non-zero
+  exit, code indicating an error.
+- If the TOML data is valid, your decoder **must** output a JSON encoding of
+  that data on `stdout` and return with a zero exit code indicating success.
+
+An example in pseudocode:
+
+    toml_data = read_stdin()
+
+    parsed_toml = decode_toml(toml_data)
+
+    if error_parsing_toml():
+        print_error_to_stderr()
+        exit(1)
+
+    print_as_tagged_json(parsed_toml)
+    exit(0)
+
+Details on the tagged JSON is explained below in "JSON encoding".
+
+### Implementing an encoder
+For your encoder to be compatible with `toml-test`, it **must** satisfy the
+expected interface:
+
+- Your encoder **must** accept JSON data on `stdin` until EOF.
+- If the JSON data cannot be converted to a valid TOML representation, your
+  encoder **must** return with a non-zero exit code indicating an error.
+- If the JSON data can be converted to a valid TOML representation, your encoder
+  **must** output a TOML encoding of that data on `stdout` and return with a
+  zero exit code indicating success.
+
+An example in pseudocode:
+
+    json_data = read_stdin()
+
+    parsed_json_with_tags = decode_json(json_data)
+
+    if error_parsing_json():
+        print_error_to_stderr()
+        exit(1)
+
+    print_as_toml(parsed_json_with_tags)
+    exit(0)
+
+JSON encoding
+-------------
+The following JSON encoding applies equally to both encoders and decoders:
+
+- TOML tables correspond to JSON objects.
+- TOML table arrays correspond to JSON arrays.
+- TOML values correspond to a special JSON object of the form:
+  `{"type": "{TTYPE}", "value": {TVALUE}}`
+
+In the above, `TTYPE` may be one of:
+
+- string
+- integer
+- float
+- bool
+- datetime
+- datetime-local
+- date-local
+- time-local
+
+`TVALUE` is always a JSON string.
+
+Empty hashes correspond to empty JSON objects (`{}`) and empty arrays correspond
+to empty JSON arrays (`[]`).
+
+Offset datetimes should be encoded in RFC 3339; Local datetimes should be
+encoded following RFC 3339 without the offset part. Local dates should be
+encoded as the date part of RFC 3339 and Local times as the time part.
+
+Examples:
+
+    TOML                JSON
+
+    a = 42              {"type": "integer": "value": "42}
+
+---
+
+    [tbl]               {"tbl": {
+    a = 42                  "a": {"type": "integer": "value": "42}
+                        }}
+
+---
+
+    a = ["a", 2]        {"a": [
+                            {"type": "string", "value": "1"},
+                            {"type: "integer": "value": "2"}
+                        ]}
+
+Or a more complex example:
+
+```toml
+best-day-ever = 1987-07-05T17:45:00Z
+
+[numtheory]
+boring     = false
+perfection = [6, 28, 496]
+```
+
+And the JSON encoding expected by `toml-test` is:
+
+```json
+{
+  "best-day-ever": {"type": "datetime", "value": "1987-07-05T17:45:00Z"},
+  "numtheory": {
+    "boring": {"type": "bool", "value": "false"},
+    "perfection": [
+      {"type": "integer", "value": "6"},
+      {"type": "integer", "value": "28"},
+      {"type": "integer", "value": "496"}
+    ]
+  }
+}
+```
+
+Note that the only JSON values ever used are objects, arrays and strings.
+
+An example implementation can be found in the BurnSushi/toml:
+
+- [Add tags](https://github.com/BurntSushi/toml/blob/master/internal/tag/add.go)
+- [Remove tags](https://github.com/BurntSushi/toml/blob/master/internal/tag/rm.go)
+
+Assumptions of Truth
+--------------------
+The following are taken as ground truths by `toml-test`:
+
+- All tests classified as `invalid` **are** invalid.
+- All tests classified as `valid` **are** valid.
+- All expected outputs in `valid/test-name.json` are exactly correct.
+- The Go standard library package `encoding/json` decodes JSON correctly.
+- When testing encoders, the TOML decoder at
+  [BurntSushi/toml](https://github.com/BurntSushi/toml) is assumed to be 
+  correct. (Note that this assumption is not made when testing decoders!)
+
+Of particular note is that **no TOML decoder** is taken as ground truth when
+testing decoders. This means that most changes to the spec will only require an
+update of the tests in `toml-test`. (Bigger changes may require an adjustment of
+how two things are considered equal. Particularly if a new type of data is
+added.) Obviously, this advantage does not apply to testing TOML encoders since
+there must exist a TOML decoder that conforms to the specification in order to
+read the output of a TOML encoder.
+
+Adding tests
+------------
+`toml-test` was designed so that tests can be easily added and removed. As
+mentioned above, tests are split into two groups: invalid and valid tests. 
+
+Invalid tests **only check if a decoder rejects invalid TOML data**. Or, in the
+case of testing encoders, invalid tests **only check if an encoder rejects an
+invalid representation of TOML** (e.g., a hetergeneous array). Therefore, all
+invalid tests should try to **test one thing and one thing only**. Invalid tests
+should be named after the fault it is trying to expose. Invalid tests for
+decoders are in the `tests/invalid` directory while invalid tests for encoders
+are in the `tests/invalid-encoder` directory.
+
+Valid tests check that a decoder accepts valid TOML data **and** that the parser
+has the correct representation of the TOML data. Therefore, valid tests need a
+JSON encoding in addition to the TOML data. The tests should be small enough
+that writing the JSON encoding by hand will not give you brain damage. The exact
+reverse is true when testing encoders.
+
+A valid test without either a `.json` or `.toml` file will automatically fail.
+
+If you have tests that you'd like to add, please submit a pull request.
+
+Why JSON?
+---------
+In order for a language agnostic test suite to work, we need some kind of data
+exchange format. TOML cannot be used, as it would imply that a particular parser
+has a blessing of correctness.
+
+My decision to use JSON was not a careful one. It was based on expediency. The
+Go standard library has an excellent `encoding/json` package built in, which
+made it easy to compare JSON data.
+
+The problem with JSON is that the types in TOML are not in one-to-one
+correspondence with JSON. This is why every TOML value represented in JSON is
+tagged with a type annotation, as described above.
+
+YAML may be closer in correspondence with TOML, but I don't believe we should
+rely on that correspondence. Making things explicit with JSON means that writing
+tests is a little more cumbersome, but it also reduces the number of assumptions
+we need to make.

internal/toml-test/gen-multi.py

@@ -0,0 +1,14 @@
+#!/usr/bin/env python3
+
+import glob
+import os.path
+
+for f in glob.glob('tests/invalid/*/*.multi'):
+    base = os.path.dirname(f[:-6])
+    for l in open(f, 'rb').readlines():
+        name = l.split(b'=')[0].strip().decode()
+        if name == '' or name[0] == '#':
+            continue
+        path = base + "/" + name + '.toml'
+        with open(path, 'wb+') as fp:
+            fp.write(l)