Decode: validate dates

[jidicula]

Issue: #613

Constructs a datestring with RFC3339 formatting using the parsed date
values, then attempts to call time.Parse() on it, which returns a
non-nil error if datetime string was invalid or time's parsing of
the string failed.

Fixes go test -tags testsuite -run TestTOMLTest_Invalid_Datetime_ImpossibleDate failure.

[jidicula]

I have some local changes adding similar calls to time.Parse() for parseLocalDate(), parseLocalTime(), and parseLocalDateTime() but I'm struggling a bit on where to add tests that adequately cover these changes – should I just be adding cases in TestUnmarshalDecodeErrors()?

[jidicula]

@pelletier how should I add test cases for covering 9d45398? I tried adding an additional case in TestUnmarshalDecodeErrors but adding the case

{
	desc: `invalid local date-time`,
	data: `A = "2021-03-40T23:59:00"`,
	msg:  `invalid local date-time`,
},

doesn't seem to return the expected error.

[pelletier]

Sorry for the late reply! TestUnmarshalDecodeErrors should be a good place to put those tests. What error are you getting instead?

Commenting on the general approach taken: this looks like a very expensive approach. Given we have year/month/day already parsed, can we just have a validation step that does not perform allocations?

[jidicula]

Sorry for the late reply! TestUnmarshalDecodeErrors should be a good place to put those tests. What error are you getting instead?

--- FAIL: TestUnmarshalDecodeErrors (0.00s)
    --- FAIL: TestUnmarshalDecodeErrors/invalid_local_date-time (0.00s)
        unmarshaler_test.go:2003: 
            	Error Trace:	unmarshaler_test.go:2003
            	Error:      	An error is expected but got nil.
            	Test:       	TestUnmarshalDecodeErrors/invalid_local_date-time
FAIL
exit status 1
FAIL	github.com/pelletier/go-toml/v2	0.016s

As I understand it, this error tester is expecting an error to be returned, but one isn't being returned even though I'm feeding it a test case with an invalid date.

[pelletier]

Given the test makes a call to Unmarshal it means that the code path is not hit or the error is lost somewhere.

[jidicula]

Commenting on the general approach taken: this looks like a very expensive approach. Given we have year/month/day already parsed, can we just have a validation step that does not perform allocations?

I was wondering about that, it seemed to me that validating day counts for each month reinvents the wheel a bit. It's definitely easier for time and month validation, though. What approach do you suggest for validating days – perhaps a map that maps month-number to its day count? Or, as I'm typing this I'm thinking this might be an appropriate use for a switch statement.

[pelletier]

In that kind of scenario I usually defer to the Go stdlib implementation:

https://github.com/golang/go/blob/24e798e2876f05d628f1e9a32ce8c7f4a3ed3268/src/time/format.go#L1272-L1275

https://github.com/golang/go/blob/24e798e2876f05d628f1e9a32ce8c7f4a3ed3268/src/time/time.go#L1002-L1026

They use a simple table lookup. As far as I know those methods are not publicly accessible. We could use unsafe to access them directly, or we could copy that table and couple functions over. In this case I prefer the latter approach because unsafe exposes us to breakage in future versions of Go, and it's not like the number of days per month is about to change any time soon. The drawback to copyying it over is extra 52 bytes in the resulting binary because of that duplicated table, but I think for usages of go-toml it is an acceptable trade-off.

[jidicula]

Woops, looks like there are some new tests upstream for handling what constitutes a valid time. @pelletier do you think it's better to keep changes introduced in #614 (here) where each element of a time datum is checked as it's parsed, or is it better to check the whole time datum with the isValidTime() function I add here?

[pelletier]

I have no preference on the approach. However the existing tests provide more precise error messages to the user, which is a good property to have. As long as validation makes it clear to the user what the problem is, I don’t have a preference.

[jidicula]

I have no preference on the approach. However the existing tests provide more precise error messages to the user, which is a good property to have. As long as validation makes it clear to the user what the problem is, I don’t have a preference.

Ok, in that case I've removed my redundant time validation 🙂

[jidicula]

Ready for review now

[jidicula]

@pelletier how should I add test cases for covering 9d45398? I tried adding an additional case in TestUnmarshalDecodeErrors but adding the case

{
	desc: `invalid local date-time`,
	data: `A = "2021-03-40T23:59:00"`,
	msg:  `invalid local date-time`,
},

doesn't seem to return the expected error.

Also the resolution for this was that I was mistakenly quoting the date value

[pelletier]

Thanks for working on this! Added two small comments on the change. Seems like the coverage went down, so it should mean that some code paths are not covered by the tests -- we really try to not bring down test coverage percentage. It's not a perfect metric, but I found it to mostly accomplish its goal.

[jidicula]

Resolved in 602925b – thanks for the feedback, I'm still relatively new to Go so getting these tips is really helping my learning 😁.

re: code coverage, are the testgen cases run in CI too now that you've added the skip statements to the still-breaking cases?

[jidicula]

re: code coverage, are the testgen cases run in CI too now that you've added the skip statements to the still-breaking cases?

nvm, I checked and it is. I would guess that the coverage drop is because of the size of the stuff copied over from time. I'll add test cases for those.

[jidicula]

Good call with the coverage comment, I looked deeper and found some unreachable code in a redundant date validity check 🙂. Should be good for re-review now @pelletier.

[pelletier]

Looking good, merged! Thank you!

[jidicula]

Looking good, merged! Thank you!

My pleasure! If it's not too much trouble, could you put a hacktoberfest-accepted label on this PR? They don't seem to be tracking review acceptance and merges correctly.

[pelletier]

No problem, done. I bet they only look at PRs merged in the main branch, which v2 is not quite ready for that yet.

[jidicula]

No problem, done. I bet they only look at PRs merged in the main branch, which v2 is not quite ready for that yet.

Ah, good point. Thanks!