Restructure the contents of the MET repository so that it matches the existing release tarfiles.

[JohnHalleyGotway]

NOTE: This will be a major change to all existing development branches!

Describe the Task

Motivation:

• Make it easier to compile MET directly from GitHub.
• Utilizing GitHub's release creation rather than manually building/attaching a custom release tarfile.
• Supports the METplus use of Manage Externals.

Recommend making the following changes:

• Store the output of running bootstrap (Makefile.in, configure, etc) directly in the repo.
• Store the output of running enum_to_string anf ylwrap directly in the repo.
• Modify enum_to_string logic to NOT include the timestamp of the runtime in its output. Otherwise we'd end up with lots of diffs in the future where the only real diff is that timestamp.
• Move the contents of the "met" directory up to the top-level of the repo.
• Create a top-level directory for "internal".
• Move existing top-level "scripts" and "test" directories into the new "internal" directory. Consider renaming "test" to something else, perhaps "unit_tests"?
• Prune any existing scripts that are no longer needed after this change.
• Update and test all existing scripts to make sure they still work.
• Update the nightly build script in particular.
• Test the compile_all_MET.sh script and update as needed.
• Update the DockerHub build rules since the location of the Dockerfile will have changed.
• Test/fix the RTD automation logic, if needed.
• Update the logic for $MET_DEVELOPMENT. Commit the files created by it directly to the repository. If that env var is set, recreate them as necessary. If not, use them as is from the repo. Also, only compile ‘internal’ code if set.
• Update the ‘make dist’ logic. It still is useful to build a tar file in this way. But it will likely need to do fewer things.
• Update the METplus Release Guide instructions... they should be simpler now.
• Create a branch and PR in the hpc-stack repo to update the code to install MET (Julie)
• Communicate with Will Hatheway about the impact on his scripts (Julie)

The main goal is minimizing impacts on existing scripts and installation instructions so that any that worked previously with a released tarfile will continue to work with GitHub's tarfile.

Be sure to go a git move instead of delete/add so that it can keep tracks of these changes.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the task down into sub-issues.
No sub-issues needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Split 2702691, 2792541

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: @JohnHalleyGotway (and @georgemccabe is willing to help if needed)
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy
• Fix build components scripts to install MET and other METplus components METplus#1068 updates manage externals which depends on this work.

Task Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

Discussed the timing of this work during the 9/23/21 MET development meeting. This is a large change and we recommend moving it from the minor version 10.1.0 release over to the next major version 11.0.0 release. After completing the work for 10.1.0 we'll have a good opportunity to make this large structural change before moving into new development for 11.0.0.

So this should be first thing done for 11.0.0. Will create a beta1 development project for 11.0.0 and add this as "required" for that development release.

[georgemccabe]

@JohnHalleyGotway I agree that this major change should be done for a major release.

Something to keep in mind is that when you move a file in GitHub, you cannot follow the history of that file using the GitHub web interface. Here is a relevant discussion: community/community#3272
There are browser extensions that can add this functionality. The history can be viewed using git on the command line with the --follow argument.

[JohnHalleyGotway]

Review the shell scripts that live in the repo and look for needed updates:

There are 54 of them in total.

find ./ -name "*.sh" | wc -l
      54

1. No updates needed in the 16 in scripts/examples.

• I ran make clean install test which exercises them and it runs fine.

2. 12 in internal/test_unit/bin:

• Deleted 3 stale ones for Stat-Analysis.
• Found no reference to '/met/' in the other 9. If these do fail, that'll be apparent in the regression test.
• Did strip out stale default settings for MET_TEST_BASE from the scripts.

3. 1 in internal/scripts/docker/build_met_docker.sh.

• Removed met from docker-clean
• Reviewed logic in the script and Dockerfiles.

4. 2 in internal/scripts/fortify.

• Deleted the fortify directory that is no longer needed.

5. 1 in internal/scripts/installation/compile_MET_all.sh

• No changes needed since this is already setup to compile code from a tarfile.

6. 2 in internal/scripts/build

• Removed call to bootstrap
• Tested running met_checkout_and_build.sh branch feature_1920_repo

8. 3 in internal/scripts/regression

• Updated paths in all 3 files.

9. 2 in internal/scripts/sonarqube

• Updated paths in 2 files.

Misc changes:

• Deleted 11 of 12 sample data files from internal/test_unit/data/mnc that are no longer used in the unit tests.
• Recommend getting rid of the last one too by changing the unit tests.

[jprestop]

@JohnHalleyGotway I really appreciate all of the detail in your comments above. Regarding:

5. 1 in internal/scripts/installation/compile_MET_all.sh
   No changes needed since this is already setup to compile code from a tarfile.

I believe there are two changes that are needed on lines 796 and 799.

I believe those will need to be changed to be capitalized (e.g. MET) because I believe the tarball made by GitHub will be MET-x.y.x.tar.gz and will unpack to MET-x.y.z. In its current state, I believe the script would not remove a MET-x.y.z directory with "rm -rf met*" or change directories to MET-x.y.z with "cd met*". Although, perhaps we don't want to change it to "MET" if there is someway to make it case insensitive. Thoughts?

[JohnHalleyGotway]

@jprestop I made your recommended change to compile_MET_all.sh with this commit.

Note that I'm still keeping the shell scripts that we've used in the past to create release tarfiles in the repo. We may eventually be able to remove those if/when we decide they aren't useful anymore.