TG2-AMENDMENT_DATEIDENTIFIED_STANDARDIZED

[iDigBioBot]

TestField	Value
GUID	39bb2280-1215-447b-9221-fd13bc990641
Label	AMENDMENT_DATEIDENTIFIED_STANDARDIZED
Description	Proposes an amendment to the value of dwc:dateIdentified to a valid ISO date.
TestType	Amendment
Darwin Core Class	dwc:Identification
Information Elements ActedUpon	dwc:dateIdentified
Information Elements Consulted
Expected Response	INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is bdq:Empty; AMENDED if the value of dwc:dateIdentified is not a properly formatted ISO 8601 date but is unambiguous and altered to be a valid ISO 8601 date; otherwise NOT_AMENDED.
Data Quality Dimension	Conformance
Term-Actions	DATEIDENTIFIED_STANDARDIZED
Parameter(s)
Source Authority
Specification Last Updated	2024-09-16
Examples	[dwc:dateIdentified="2021-28-10": Response.status=AMENDED, Response.result=dwc:dateIdentified="2021-10-28", Response.comment="dwc:dateIdentified assuming dwc:year, dwc:day and dwc:month"]
	[dwc:dateIdentified="21-10-28": Response.status=NOT_AMENDED, Response.result=, Response.comment="dwc:dateIdentified contains ambiguous values. It could be dd-mm-yy or yy-mm-dd"]
Source	Kurator
References	ISO (2019) ISO 8601-1:2019(en) Date and time — Representations for information interchange — Part 1: Basic rules. https://www.iso.org/obp/ui/ Wikipedia (2020) ISO 8601. https://en.wikipedia.org/wiki/ISO_8601 Library of Congress (2019) Extended Date/Time Format (EDTF). https://www.loc.gov/standards/datetime/
Example Implementations (Mechanisms)	Kurator:event_date_qc
Link to Specification Source Code	https://github.com/FilteredPush/event_date_qc/blob/1abbd3f02eb6c28129764defab78f72156972864/src/main/java/org/filteredpush/qc/date/DwCEventDQ.java#L489
Notes	We reference Wikipedia for the ISO standard because the standard documents are not free.

[iDigBioBot]

Comment by Paul Morris (@chicoreus) migrated from spreadsheet:
Corresponding ammendment for IDENTIFIED_DATE_INVALID (VALIDATION_DATEIDENTIFIED_INVALID)

[chicoreus]

There appears to be a copy/paste error in the specification,

EXTERNAL_PREREQUESITES_NOT_MET if the specified target source authority was not found;

This test does not involve a query on any external source authority such as a list of taxonomic names.

[ArthurChapman]

@chicoreus This probably refers to the ISO 8601-1:2019 that is needed. But you probably don't need to go to the actual Standard to use the format, so probably unnecessary. This may apply to other date tests?

[Tasilee]

If it doesn't have a Parameter, then there is no "specified target authority"?

[chicoreus]

EXTERNAL{PREREQUISITES_NOT_MET should signal that the test must connect to an external resource in order to perform the test, this test would be expected to be implemented strictly with local code that doesn't need to do a network lookup each time the test is run, so no. need for this assertion in the specification.

@Tasilee A test might have a parameter and not need to connect to an external source (such as earliest date as a parameter), or a test might specify some authority to be used by implementors (such as the ISO date format specified here), but not need to connect to an external source on each run, and it might do so with or without taking the source authority as a parameter (though I expect our likely taxonomic cases that would take a taxonmic authority as a source would typically need to do a lookup for each test run - the relationship is logical, but isn't necessaraly present).

[ArthurChapman]

@chicoreus your comment would apply to all that are tagged ISO/DCMI Standard. To date we have only used the Parameter field to define default minimum and maximums and source Authorities (the later using bdq:sourceAuthority (defalt=xxxxx)) in Paramaterized tests. Perhaps we should extend that to include standard source authorities such as ISO and DCMI Standards using bdq:sourceAuthority=ISO 8601-1:2019 etc. under the Parameter field ?? What do others think?

[Tasilee]

I agree with @chicoreus: If there is no (EXTERNAL) source authority, then we remove that part of the Expected Responses. Other comments?

@ArthurChapman: Hopefully quoting @chicoreus - If there is only one obvious bdq:sourceAuthority or maybe value, it is not a Parameter?

Then it is a demarcation between a bdq:sourceAuthority Parameter API-type lookup vs a Reference to a standard that the test relies on (as the latter is here). The ISO/DCMI STANDARD flag is visible evidence of the use of a particular standard but is not needed as a Parameter.

[chicoreus]

@ArthurChapman Parameter should not be specified for cases like this where there is a standardized source authority that implementors would be expected to embed in their code. Parameter should only be used when different use cases would call for different limits (such as a national data set wishing to use elevation/depth limits that apply to their country and a national taxonomic authority file or species list).

Issue looking good, removing the needs work label.

[chicoreus]

Actually, making one change (adding unambigous) to conform with language in #61

From:

INTERNAL_PREREQUISITES_NOT_MET if the field dwc:dateIdentified was not present or is EMPTY; AMENDED if the value of dwc:dateIdentified was altered to conform with the ISO 8601-1:2019 date format; otherwise NOT_CHANGED

To:

INTERNAL_PREREQUISITES_NOT_MET if the field dwc:dateIdentified was not present or is EMPTY; AMENDED if the value of dwc:dateIdentified was altered to unambiguously conform with the ISO 8601-1:2019 date format; otherwise NOT_CHANGED

[Tasilee]

Due to recent discussions, changed

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it can be unambiguously interpreted as valid using bdq:sourceAuthority; otherwise NOT_AMENDED

to

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if can be unambiguously interpreted as valid ISO 8601-1 date; otherwise NOT_AMENDED

...and also removed bdq:sourceAuthority

[tucotuco]

I agree with Arthur. Sufficient that the user knows where to go to see what we're talking about while minimizing standard maintenance. I think this would be a good thing to recommend in DwC term change requests as well.

[Tasilee]

I have amended the Reference and corrected a typo in the Expected response. Please check.

[Tasilee]

I've edited the Expected Response according to @tucotuco suggestion:

From

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it can be unambiguously formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

to

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was unambiguously formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

[ArthurChapman]

Sorry - I am not sure that the new wording reads well. How about:

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it could be unambiguously formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

[ArthurChapman]

After discussion with @Tasilee - I am not sure, but can accept was. It just doesn't seem to read well. I am not sure with "was", if unambiguously is in the right place - my thinking is that if it is unambiguous (i.e. 3 April rather than 03-04) then you format it there is then nothing ambiguous or unambiguous about the formatting - if it is ambiguous (i.e. 03-04) then it is NOT_AMENDED

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was unambiguous by formatting as a valid ISO 8601-1 date; otherwise NOT_AMENDED

[tucotuco]

Counter offer...

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was unambiguous and formatted as a valid ISO 8601-1 date; otherwise NOT_AMENDED

[Tasilee]

I have updated the Expected Response (as suggested) and the ISO reference (and while it looks odd, it works).

[ArthurChapman]

What is the thumbs down symbol in the middle of the reference?

[Tasilee]

Some form of markdown I presume, but as I said, it works as a link.

[ArthurChapman]

It appears that ":-" appears as a thumbs down sign in GitHub

[chicoreus]

Current phrasing is not readily iterpretable by implementors. It implies AMENDED if the value already conforms, not if it has been changed to conform. Amendments should be explicit about changing values. The text "if it was
unambiguous and formatted" means to an implementor that the text was, past tense, prior to any action, unambigous and formatted (noun) in the expected form. We should be using explicit "altered to conform" not "formatted, which could be read as either the data having had that that state in the past or having been changed to that state. Other amendments need review to make sure that this form of ambiguity has not crept into them.

Very strongly recommend that we return to the previous phrasing.

Changing back from:

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is
EMPTY; AMENDED the value of dwc:dateIdentified if it was
unambiguous and formatted as a valid ISO 8601-1 date; otherwise
NOT_AMENDED

to:

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is
EMPTY; AMENDED if the value of dwc:dateIdentified was altered
to unambiguously conform with the ISO 8601-1:2019 date format;
otherwise NOT_AMENDED

[chicoreus]

@Tasilee my feeling is that we should reference ISO 8601-1:2019, not the latest version, as the latest version could change the nature of the specifications, with the potential of making test cases and implementations diverge. Referencing a standard set of data values (as in a list taxon names) carries with it the expectation that the data will change, and thus that test cases that test implementations are validated against may need to be changed over time without either the specification for a test or implementations of the test changing, but referencing a standard for the format of data carries the expectation that the format won't change, and that neither test cases nor implementation should change over time without a change in the specification for the test.

[chicoreus]

Whether that is a practical issue for latest version of ISO 8601-1 vs ISO 8601-1:2019 is another question. @ArthurChapman 's note about introduction of validity of 24:00 in a subsequent amendment is relevant....

[chicoreus]

Paralleling the proposal in #61

Propose changing to:

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was not a properly formatted ISO 8601-1 date but was unambiguous and was altered to be a valid ISO 8601-1 date;
otherwise NOT_AMENDED.

Both here and in #61 we want to amend if:

1. the existing value was not a correctly formatted data
2. the existing value was unambiguous (e.g. not a 2 digit year, not nn-nn-yyyy where nn are both below 12).
3. the value could be transformed into an ISO date.

[Tasilee]

Thanks @chicoreus. While your logic is good, there is repetition and therefore ambiguity. Suggest changing

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was not a properly formatted ISO 8601-1 date but was unambiguous and was altered to be a valid ISO 8601-1 date;
otherwise NOT_AMENDED.

to

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was not a properly formatted ISO 8601-1 date but was unambiguously interpreted as a valid ISO 8601-1 date; otherwise NOT_AMENDED.

[chicoreus]

On Sat, 10 Jun 2023 17:59:32 -0700 Lee Belbin ***@***.***> wrote: Thanks @chicoreus. While your logic is good, there is repetition and therefore ambiguity. Suggest changing INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was not a properly formatted ISO 8601-1 date but was unambiguous and was altered to be a valid ISO 8601-1 date; otherwise NOT_AMENDED. to INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY; AMENDED the value of dwc:dateIdentified if it was not a properly formatted ISO 8601-1 date but was unambiguously interpreted as a valid ISO 8601-1 date; otherwise NOT_AMENDED.

For clarity for implementors, I think we need to be explicit both here and in #61 that there are three concerns: 1. the date identified was not properly formatted 2. the date identified was unambigouus 3. the date identified was transformed to a valid ISO 8601-1 format. if all three concerns are met, Response.status is AMENDED, response.result contains the list of amended terms. Reading "AMENDED" as a verb rather than a constant is helping confuse the issue....

[chicoreus]

How about:

INTERNAL_PREREQUISITES_NOT_MET if dwc:dateIdentified is EMPTY;
AMENDED if the value of dwc:dateIdentified was not a properly
formatted ISO 8601-1 date but was unambiguous and was altered to be a
valid ISO 8601-1 date; otherwise NOT_AMENDED.

[Tasilee]

Expected response, and Specification last updated amended accordingly. I will search for other tests that may require a change.

[Tasilee]

Add bdq:souceAuthority value "bdq:sourceAuthority is "ISO 8601-1:2019" [https://www.iso.org/obp/ui/]" to align with related tests.

[Tasilee]

Due to recent discussions, changed

bdq:sourceAuthority is "ISO 8601-1:2019" [https://www.iso.org/obp/ui/]

to

bdq:sourceAuthority = "ISO 8601-1:2019" {[https://www.iso.org/iso-8601-date-and-time-format.html]}

I don't see a great need to change the references in the Expected response to bdq:sourceAuthority.

[Tasilee]

Splitting bdqffdq:Information Elements into "Information Elements ActedUpon" and "Information Elements Consulted"

[Tasilee]

Changed reference in Expected Response from ISO 8601-1 to ISO 8601 and removed bdq:sourceAuthority entry (which is covered in References).