Use yaml-test-suite

[Paalon]

There is https://github.com/yaml/yaml-test-suite and it is useful for testing.

[GunnarFarneback]

Yes, it would be great if we could incorporate that in our tests.

[GunnarFarneback]

With the script below you can run the test suite in a limited capacity, which only checks whether the input YAML can be read at all, or in the case of invalid input, that it does throw an exception. I'm currently getting the results

Test Summary:   | Pass  Fail  Error  Total  Time
YAML test suite |  300    23     83    406  7.4s

Some of the fails are cases where we do throw, but not an Exception, or accept invalid input. Quite a few errors are related to TABs and to unusual mapping keys, e.g. an empty key. Some tests mention the not really existing YAML 1.3, but I'm unsure whether those are expected to be readable with older versions.

Script:

import YAML
using Test

function run_all_tests(yaml_test_suite_clone_dir)
    tests_dir = joinpath(yaml_test_suite_clone_dir, "src")
    for filename in readdir(tests_dir, join = true)
        run_test(filename)
    end
end

function read_test(filename)
    data = YAML.load_file(filename)
    for test in data
        test["yaml"] = recode(test["yaml"])
    end
    return data
end

function recode(s)
    return replace(s, "␣" => " ",
                   "———»" => "\t",
                   "——»" => "\t",
                   "—»" => "\t",
                   "»" => "\t",
                   "↵\n" => "\n",
                   "∎\n" => "",
                   "←" => "\r",
                   "⇔" => "\ufeff")    
end

function run_test(filename)
    for (i, data) in enumerate(read_test(filename))
        if get(data, "fail", false)
            @test_throws Exception YAML.load(data["yaml"])
        else
            @test (YAML.load(data["yaml"]); true)
        end
    end
end

@testset "YAML test suite" begin
    run_all_tests(only(ARGS))
end