Improve filter installation process to avoid use of an extra shell script

[DennisHeimbigner]

re: #2338
re: #2294

In issue #2338, Ed Hartnett suggested a better way to install filters to a user defined location -- for Automake, anyway.

This PR implements that suggestion. It turns out to be more
complicated than it appears, so there are fair number of changes;
mostly to shell scripts. Most of the change is in plugins/Makefile.am.

NOTE: this PR still does NOT address the use of HDF5_PLUGIN_PATH
as the default; this turns out to be complex when dealing with NCZarr.
So this will be addressed in a subsequent post 4.9.0 PR.

Misc. Changes

1. Record the occurrences of incomplete codecs in libnczarr so that
   they can be included in _Codecs attribute correctly. This allows
   users to see what missing filters are referenced in the Zarr file.
   Primarily affects libnczarr/zfilter.[ch]. Also required creating a
   new no-effect filter: H5Zunknown.c.
2. Move the unknown filter test to a separate test file.
3. Incorporates PR Fix distcheck problems #2343

[edwardhartnett]

Does this instead the plugins so zstandard compression works?

If not, then that needs to be fixed before release. It's quite simple and there is no reason to release with non functional compression.

[edwardhartnett]

When I build and test this branch works OK, but when I do make install:

make[1]: Entering directory '/home/ed/nd/plugins'
make  install-am
make[2]: Entering directory '/home/ed/nd/plugins'
make[3]: Entering directory '/home/ed/nd/plugins'
make  install-exec-hook
make[4]: Entering directory '/home/ed/nd/plugins'
sh ./stdinstall.sh
sh: 0: Can't open ./stdinstall.sh
make[4]: *** [Makefile:1106: install-exec-hook] Error 127
make[4]: Leaving directory '/home/ed/nd/plugins'
make[3]: *** [Makefile:1008: install-exec-am] Error 2
make[3]: Leaving directory '/home/ed/nd/plugins'
make[2]: *** [Makefile:910: install-am] Error 2
make[2]: Leaving directory '/home/ed/nd/plugins'
make[1]: *** [Makefile:904: install] Error 2
make[1]: Leaving directory '/home/ed/nd/plugins'
make: *** [Makefile:754: install-recursive] Error 1
(base) ed@koko:~/nd$ 

[DennisHeimbigner]

I am confused; I thought I removed stdinstall.sh and I cannot find a reference to it
in any Makefile.am in this PR.

[edwardhartnett]

Whoops! I was not trying the correct branch. Let me try again and I will report back...

[edwardhartnett]

OK, sorry about the confusion. When I use the correct branch, make install and make distcheck work fine. ;-)

In order to install the plugins where HDF5 will find them, only one line needs to be added to configure.ac:

with_plugin_dir=${HDF5_PLUGIN_PATH-'/usr/local/hdf5/lib/plugin'}
This needs to be added just before:

AC_MSG_CHECKING([whether and where we should install plugins])

This tells configure to install the plugins in the default plugin path, unless the user specifies a different location with the --with-plugin-dir option, or by having environment variable HDF5_PLUGIN_PATH set.

Once this PR is merged, I can do a PR adding this one line to configure.ac, and adding a test for the nc_def_var_zstandard()/nc_inq_var_zstandard() functions, which are currently untested.

For the CMake build, the Windows default plugin path is C:\ProgramData\hdf5\lib\plugin.

If this is not fixed before release, there will be a lot of confused users and support questions, because the zstandard functions will not work.

[DennisHeimbigner]

I perceive this as more complex than your fix allows for.

If the user sets HDF5_PLUGIN_PATH as an env variable, then the fix needs to parse that path to extract the first directory in that path to use.

If you default to /usr/local/hdf5/lib/plugin, and subsequently, the user runs with a different HDF5_PLUGIN_PATH, then I do not know if HDF5 searches the PATH and also /usr/local/hdf5/lib/plugin. I know that NCZarr does not do so at the moment. I am going to check on what HDF5 does.

You should not conflate the builder/installer with the user. You need to tell me what each is expected to know about filters. If they are the same, then presumably the user understands the role of HDF5_PLUGIN_PATH and can set --with-plugin-dir appropriately. If the user is different from the builder, then presumably the user was given some set of instructions *by the builder) about using HDF5_PLUGIN_PATH (or not using it).

My interpretation of your solution is the builder and user are required to use the same HDF5_PLUGIN_PATH or the default install location is used and the user never sets HDF5_PLUGIN_PATH. Is my intepretation correct? What is the norm at NOAA? Have you talked to any users about their knowledge of HDF5_PLUGIN_PATH?

[DennisHeimbigner]

I checked the HDF5 1.12.0 code. If HDF5_PLUGIN_PATH environment variable
is set, then it uses that exclusively. If it is not set, then it used /usr/local/lib/plugin
exclusively. So, both user and builder must set same HDF5_PLUGIN_PATH -- as a single directory --
or neither must set HDF5_PLUGIN_PATH.

[edwardhartnett]

I expect installers and builders to know nothing about filters. That's why it's important for the default to work.

If the builder knows enough about filters to be setting HDF5_PLUGIN_PATH, then they must also set it for the users. Once they start setting their own plugin path, obviously they are responsible for seeing that it works for the users of their system.

It's important to allow installers to override the default, but I suspect that very few will ever do so. That's why it's important that netcdf works out of the box without anyone explicitly setting any plugin path.

If we install to the default plugin directory the user can just use zstandard compression without knowing anything about filters.

[DennisHeimbigner]

Good points. This seems a little strong to me.

I expect installers and builders to know nothing about filters. That's why it's important for the default to work.

Anyone else care to way in on this.

[edwardhartnett]

To answer your question about NOAA, we have installed CCR on many NOAA HPCs, always using the default directory. It works well.

It's necessary for HDF5_PLUGIN_PATH to supercede the default as we use this in tests. (So we can test filter code before it is installed.) But, in general, I would be very surprised if any netcdf user or installer knows much about HDF5 plugins. Why would they?

It makes sense that, by default, we should use the HDF5 default. Otherwise there will be much confusion and support requests.

[DennisHeimbigner]

Via support questions, it is clear that e.g. IDL and R both tell uses
about filters and about how to set HDF5_PLUGIN_PATH.
Another issue is that of current workflows that do their own filter installation
(essentially everyone who uses filters now). Presumably they would not be happy
about having to modify those workflows to add something like "--disable-plugin-dir".
But that would be the case if plugin installation is automatic. This is why I prefer
an explicit --with-plug-dir or equivalent.

[edwardhartnett]

I do not understand the objection to using the default plugin directory by default and allowing users to override it if they need to.

[DennisHeimbigner]

Now I am confused. Tell me again precisely what options and values the user can or must put
on the ./configure command

[DennisHeimbigner]

What I have at the moment is this:

1. The builder specifies "./configure ... --with-plugin-dir=<directory>" or "./configure ... --with-plugin-dir" (i.e. with no argument).
2. In the no argument case, the default is the last directory in the list of directories specified by HDF5_PLUGIN_PATH.
3. If HDF5_PLUGIN_PATH is not defined, then it effectively defaults to /usr/local/hdf5/lib/plugin.
4. If --with-plugin-dir is not used, then filters are not installed.

The rationale is that this gives most of what Ed is suggesting, but still does not break existing workflows because of point 4.

[edwardhartnett]

I don't understand how any existing workflows can be broken by using the default plugin directory.

[DennisHeimbigner]

Suppose that you are a user who understands filters and HDF5_PLUGIN_PATH.
So you have already installed a set of filters for your use.
But if auto-install is in place, then suddenly, some of the filters you had already
set up get overwritten by our versions via auto-install.

For example, suppose they had installed a Blosc filter that supported more sub-filters than does
our Blosc filter. Suddenly, their workflows stop working.

This is because two filters having the same name does not mean they have exactly the same
semantics.

[edwardhartnett]

The problem of installing on top of existing libraries is a known problem with all installs. In the exact same way, if someone installs a newer version of libnetcdf.so on top of an old version, and the newer version was built with --disable-dap, then all existing netCDF applications that use dap will break. This is just part of the landscape of installing software, and is no different for plugins than for any other binary object. Users and sysadmins have to keep this in mind whenever installing new versions of any software. So this is not a reason not to install plugins in the default directory.

Furthermore, no proposed solution can solve this problem. Making the user specify the directory does not change the result in any way - if the user installs new versions of the plugins in their custom plugin directory, they have the exact same problems as if they were using the default plugin directory.

Let's consider the installation workflows of interest, from most common to least common.

1 - Most common - used by NOAA everywhere, NCAR everywhere I've seen, and almost all new users:

./configure --prefix=/some/install/point && make && sudo make install

This is the most important and common workflow, and we should ensure it continues to work. But it will not work for zstandard with this PR. Instead, filters will be turned off(?) That's not standard or the usual way I would expect software to work. Filters are a key feature that should be turned on by default. (Incidentally, if filters are off, does that mean zlib will also no longer work? That is certainly unacceptable.)

2 - Much less common - used by those who are knowledgeable about filters:

export HDF5_PLUGIN_PATH=/plugin/install/point
./configure --prefix=/some/install/point && make && sudo make install

This allows the users and sysadmins to set system-wide HDF5_PLUGIN_PATH for all users. In this case, our install should respect the HDF5_PLUGIN_PATH environment variable and install the plugins there. (I actually don't think parsing HDF5_PLUGIN_PATH is good. If it contains more than one directory, then how do we know which one to install plugins at? Users who want that level of complexity must resort to workflow 3 below, and take full responsibility for their non-default plugin choices.)

3 - Least common, used by those who know about filters, and have some special filter configurations they want to support - perhaps different HDF5_PLUGIN_PATH settings for different applications or versions:

./configure --prefix=/some/install/point --with-plugin-dir=/plugin/install/point && make && sudo make install

This workflow gives the builder 100% control - plugins can be installed anywhere, and it is up to the installer and the user to understand what setting is needed for HDF5_PLUGIN_PATH. It seems like this would support all current users of filters with netcdf-c.

The number of users now using plugins, and installing them in non-default places, is tiny. The number of users using workflow 1 is massive - almost every user in fact. With your proposed solution, zstandard will not work for most users and they will have to either contact support, or figure out what to do and rebuild netcdf-c with the --with-plugin-dir option. This will lead to a lot of confusion. NOAA sysadmins will be able to handle this (because I will tell them about it), but users of NOAA's zstandard-compressed data will find files that they can't open until they reinstall netcdf-c, and put the fitters where they can be found by HDF5.

So I suggest we use the default plugin install location for workflow 1, and give all these poor users a working zstandard compression, right out of the box, without making them contact support to find out how to get it. ;-)

Workflows 2 will work for users/builders with HDF5_PLUGIN_PATH set, and that's good.

Finally, workflow 3 will work for any user/builder who needs complete control of plugin location. It's fair that users with these needs will have to read the documentation and figure out what to do.

All of this has been worked out and well-tested with CCR for the past several years.

[DennisHeimbigner]

I just realized last night the real danger. You can have two HDF5 filters that use exactly the same compression library, but assume a different order for the parameters. Since the parameters are stored in the file, you can write the data using one wrapper and try to read the data later using a different assumed parameter order. This will fail and the user will have no idea why.

[edwardhartnett]

Ok but can we get the zeroth order case to work?

[DennisHeimbigner]

There is a difference between overwriting our own libraries (in /usr/local/lib, for example) with no warning and overwriting other peoples code with no warning. I think that there needs to be some kind of specific option that must be used so the builder has a chance to avoid the problem. Silent overwriting of other peoples code seems problematic.

[edwardhartnett]

Releasing a version that is broken for almost all users seems problematic to me.

[DennisHeimbigner]

"Broken" is a pretty strong word.

[jswhit]

Could netcdf-c install the plugins in a different default directory (say /usr/local/netcdf/lib/plugin) and then tell HDF5 to look in a different directory by default if HDF5_PLUGIN_PATH is not set?

[DennisHeimbigner]

In HDF5 version 1.13.x, it is possible to programmatically set the plugin path, so it could
be done with that version. In earlier versions, I do not think this is possible.

[WardF]

@DennisHeimbigner As is so often the case, the compilation symptom has disappeared today. I'm still seeing an issue with make check, instead of make. See the attached test-suite.log, which reflects a 'filter not available' error when libbz2-dev is not installed. This issue goes away when I install the libbz2-dev package.

It feels like the easiest thing to do as is outlined above, as long as @edwardhartnett is correct and there's no subtle reason not to fencepost these tests around libbz2 being found.

Anyways, I've double-checked that the attached file actually reflects the error I'm observing in nczarr_test/
test-suite.log.zip

[DennisHeimbigner]

Well, until I can solve the problem, it is certainly ok to conditionalize the relevant test.
As near as I can tell, they are as follows:

nc_test4

• tst_filter.sh tst_filter.sh tst_specific_filters.sh

nczarr_test

• run_specific_filters.sh
• run_filter.sh

[DennisHeimbigner]

I looked at the test-suite log and was surprised to see those
seg faults. This is ubuntu correct?

[WardF]

@DennisHeimbigner correct, where the package libbz2-dev seems to be the deciding factor, when using autotools-based builds. CMake-based builds work regardless of whether or not libbz2-dev is installed.

[DennisHeimbigner]

I believe I have fixed the problem. Turns out just changing the value of have_bz2
isn't enough to test the local bzip2 case. One needs to actually uninstall libbz2-dev.
Will add a commit to this PR after the tests complete.

[DennisHeimbigner]

Looks like the github actions queue is screwed up. Needs restarting.

[WardF]

Re-started the job queue.

[WardF]

make distcheck is failling now because it doesn't have access to the default plugin directory in /usr/lib. I anticipate issues around that from users who regularly run make distcheck. Taking a look at workaround.

[edwardhartnett]

@WardF set environment variable DISTCHECK_CONFIGURE_FLAGS to '--with-plugin-dir-/what/you/want' before running make distcheck.

[WardF]

@edwardhartnett yeah, I've found that fix, thanks :), I'm just thinking about the "default user experience" and how to minimize "Hey, make distcheck is failing" esupport questions. Thanks!

[DennisHeimbigner]

It should be the case that all checks will run even if no default plugin dir is specified.
So looks like I screwed that up.

[WardF]

@DennisHeimbigner I'm seeing the following at the end of make distcheck

make[3]: Entering directory '/home/wfisher/Desktop/gitprojects/netcdf-c/netcdf-c-4.8.2-development/_build/sub/plugins'
make[4]: Entering directory '/home/wfisher/Desktop/gitprojects/netcdf-c/netcdf-c-4.8.2-development/_build/sub/plugins'
make[4]: Nothing to be done for 'install-exec-am'.
 /usr/bin/mkdir -p '/usr/local/hdf5/lib/plugin'
/usr/bin/mkdir: cannot create directory ‘/usr/local/hdf5’: Permission denied
make[4]: *** [Makefile:768: install-pluginLTLIBRARIES] Error 1
make[4]: Leaving directory '/home/wfisher/Desktop/gitprojects/netcdf-c/netcdf-c-4.8.2-development/_build/sub/plugins'
make[3]: *** [Makefile:1049: install-am] Error 2

[edwardhartnett]

I believe the DISTCHECK_CONFIGURE_FLAGS must be set for make distcheck to work in this case.

However, I doubt many users will be running make distcheck. Most will be running make install, so that is the one to make work by default.

[DennisHeimbigner]

OK, I see. We would need to do the distcheck under sudo for it to work.
I think it is easier to specifiy the DISTCHECK_FLAG "--without-plugin-dir"

[WardF]

@DennisHeimbigner As long as we can set that as part of the default make distcheck without the user needing to, that should be fine. I'll take a look.

[edwardhartnett]

Make check will not encounter this problem, it will only happen with the make install that distcheck does.

When the user does make install, he or she must be sudo to install in default location, but that is typically the case so not a problem.

If the user has no sudo powers he must set --prefix and --with-plugin-dir to directories he has write permissions.

[WardF]

Added AM_DISTCHECK_CONFIGURE_FLAGS = --without-plugin-dir to the top-level Makefile.am file in a forked copy of this branch. Everything else looks good, so I will merge this PR as part of an additional PR. Thanks @DennisHeimbigner and @edwardhartnett !