Feature dtcenter/METplus-Internal#32 add timestamp to final conf and file_lists

[georgemccabe]

I can't link directly under Development, but the relevant GitHub issue is: https://github.com/dtcenter/METplus-Internal/issues/32

This PR changes the default value for METPLUS_CONF to include the log timestamp to the final conf file that is generated, i.e. metplus_final.conf.YYYYMMDDHHMMSS. It also causes the default file_lists directory where text files that contain lists of files to pass into a MET tool are created to also include the timestamp (i.e. {OUTPUT_BASE}/staging/file_lists.YYYYMMDDHHMMSS) so that it is clear which file lists were created for a given METplus run. These changes prevent some issues if parallel METplus runs are using the same output directory and make it more clear which settings were used for a given run for debugging.

While I think the new default for the final conf file should definitely include the timestamp to ensure that the file sent to us for debugging corresponding to the log file, I am torn if the new default for the file_lists dir should include the timestamp. This will create a new directory for each run and while the files typically are small text files, it will clutter up the staging directory. The alternative to making this the default is to add instructions in the METplus User's Guide to describe steps that can be taken to be able to run parallel runs of METplus.

Note that we have not tested running METplus in parallel to know if other issues may still be present. These recent changes, as well as the changes in #1756, only address known issues that have come up.

Pull Request Testing

• Describe testing already performed for these changes:
  

Ran a few use cases locally and examined the files that were created.

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

• Ensure all automated tests pass
• Run a few use cases locally that generate file lists (MTD and UserScript met_tool_wrapper use cases are good examples) and examine the output that is produced

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [No]

I'd like to get feedback on this PR and make any adjustments before adding documentation since we may change how things are handled.

• Do these changes include sufficient testing updates? [Yes]

• Will this PR result in changes to the test suite? [No]
  

There are conf files used for the use case tests and pytest unit tests that override the defaults for METPLUS_CONF and FILE_LISTS_DIR so that they do not include timestamps, so the results should not change in the automated tests.

• Please complete this pull request review by 2022/09/01.
  

Pull Request Checklist

See the METplus Workflow for details.

• Add any new Python packages to the METplus Components Python Requirements table.
• Review the source issue metadata (required labels, projects, and milestone).
• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s)
  Select: Organization level software support Project or Repository level development cycle Project
  Select: Milestone as the version that will include these changes
• After submitting the PR, select Development with the original issue number: https://github.com/dtcenter/METplus-Internal/issues/32
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[j-opatz]

Per the PR request, all checks were passed and some simple use cases that create file lists were tested.

While the PR does as it indicates, I still believe that the best option is to add the new timestamp feature to the final conf, but do NOT add individual directories for file lists based on the time stamps. As I voiced in the METplus Engineering meeting (8/30) I understand the benefit of creating time run-specific directories for troubleshooting, but I think the cost of creating an ever-increasing directory list of files is too much clutter.

Perhaps there's a middle ground, where these directories are created but the cleanup option (SCRUB_STAGING_DIR I think) is set "on" by default? That way, troubleshooting is not the default option and users can expand their help options as needed. But as it stands, this PR does what it intends to do and I approve.

[JohnHalleyGotway]

I like the idea of appending timestamps to the file lists created by METplus. I can see how being able to differentiate between file lists for multiple runs is useful, esp when debugging unexpected behavior.

I approve.