TOML: improvements

[kodebach]

Fixes #3925

Basics

These points need to be fulfilled for every PR:

• Short descriptions of your changes are in the release notes
  (added as entry in doc/news/_preparation_next_release.md which
  contains _(my name)_)
  Please always add something to the release notes.
• Details of what you changed are in commit messages
  (first line should have module: short statement syntax)
• References to issues, e.g. close #X, are in the commit messages.
• The buildservers are happy. If not, fix in this order:
  • add a line in doc/news/_preparation_next_release.md
  • reformat the code with scripts/dev/reformat-all
  • make all unit tests pass
  • fix all memleaks
• The PR is rebased with current master.

If you have any troubles fulfilling these criteria, please write
about the trouble as comment in the PR. We will help you.
But we cannot accept PRs that do not fulfill the basics.

Checklist

Check relevant points but please do not remove entries.
For docu fixes, spell checking, and similar none of these points below
need to be checked.

• I added unit tests for my code
• I fully described what my PR does in the documentation
  (not in the PR description)
• I fixed all affected documentation
• I added code comments, logging, and assertions as appropriate (see Coding Guidelines)
• I updated all meta data (e.g. README.md of plugins and METADATA.ini)
• I mentioned every code not directly written by me in THIRD-PARTY-LICENSES

Review

Reviewers will usually check the following:

• Documentation is introductory, concise, good to read and describes everything what the PR does
• Examples are well chosen and understandable
• Code is conforming to our Coding Guidelines
• APIs are conforming to our Design Guidelines
• Code is consistent to our Design Decisions

[kodebach]

@markus2330 I have now reverted the comment handling to the state that is described in METADATA.ini. The string handling should be much better now. However, we are still not 100% TOML 1.0.0 compatible. In particular, we cannot parse these valid strings:

str1 = """"This," she said, "is just a pointless statement.""""
str2 = ''''That,' she said, 'is still pointless.''''

Other things like the following do work however:

str4 = """Here are two quotation marks: "". Simple enough."""
str5 = """Here are three quotation marks: ""\"."""
str6 = """Here are fifteen quotation marks: ""\"""\"""\"""\"""\"."""
quot15 = '''Here are fifteen quotation marks: """""""""""""""'''

We should probably add some more tests, but I didn't have time to add many. I also don't really like the way the tests in testmod_toml are written. It is very hard to see what e.g. this is doing:

	TEST_WR_HEAD;

	WRITE_KEY ("table");
	SET_ORDER (0);
	SET_TOML_TYPE ("simpletable");
	DUP_EXPECTED;


	WRITE_KV ("table/b", "123");
	SET_ORDER (1);
	DUP_EXPECTED;
	SET_TYPE ("long_long");

	WRITE_KEY ("ta");
	SET_ORDER (2);
	SET_TOML_TYPE ("tablearray");
	DUP_EXPECTED;
	SET_ARRAY ("#0");

	WRITE_KV ("ta/#0/a", "1337");
	SET_ORDER (3);
	DUP_EXPECTED;
	SET_TYPE ("long_long");

	TEST_WR_FOOT;

[kodebach]

@markus2330 As far as I'm concerned, this PR can be merged now. You may want to review it first, but the tests suggest that I didn't break anything. I would merge this ASAP and leave any further improvements for follow-up PRs.

PS. The failing link-check build job is unrelated to the changes.

[mpranj]

Can you please rebase?

[mpranj]

Thank you, amazing work! 💯

[markus2330]

In particular, we cannot parse these valid strings:

These examples would be perfect for TOML readme. Please always add information within the code repository. People will not read what we discussed in PRs or issues.

[kodebach]

These examples would be perfect for TOML readme.

Agreed. I first thought we would immediately start a follow-up PR, but now that priorities have shifted, we should probably add this to the README.

[markus2330]

TOML is still priority for 1.0 but these cases are probably not keeping us from using it as default?

[kodebach]

I don't think these cases should be a big issue.