Enhance Bufr data support including multi-category messages

[Yaqiang]

Enhance Bufr data support including multi-category messages in a sigle data file, so that prepbufr data file is supported. Also CMA bufr tables were added for reading CMA released bufr data files.

Description of Changes

_ Add CMA bufr tables (table B and D) as resource tables for reading the bufr data files released from CMA.

_ prepbufr data file may containes multiple category messages such as ADPUPA, ADPSFC, etc. BufrIosp2 only support single category messages before. So BufrIosp2 and related classes (BufrIospBuilder, EmbeddedTable, TableA, ...) was revised to support more root variables (not just 'obs').

_ 'reallyRead' function in ucar.nc2.Variable class has problem when a Structure variable read result is ArrayObject with ArraySequence as elements, so it was revised to solve the issue.

• 'extractMemberArrayFromIteration' in ucar.ma2.ArraySequence class was revised for SEQUENCE data type case.

I test them using MeteoInfoLab (version 3.9.7) bufr reading scripts with netcdf library behand.

PR Checklist

• Link to any issues that the PR addresses
• Add labels
• Open as a draft PR
  until ready for review
• Make sure GitHub tests pass
• Mark PR as "Ready for Review"

[lesserwhirls]

Greetings @Yaqiang!

Thank you for this contribution! There are two issues at play with the github checks. This first is minor, in that there are some issues detected with the code style. These can be seen by running ./gradlew spotlessCheck, and in most cases can be fixed by running ./gradlew spotlessApply.

The second is there are failing tests. It looks like there is a test failing due to an error trying to reach a remote server (unrelated to your pull requests). I ran our full testsuite against your PR and did find some failures related to the changes, however. For example, when trying to read a sequence from the attached .bufr file using the old API (NetcdfFile.open):

try (NetcdfFile nc = NetcdfFile.open("ecmwf-wmo16-hasOptionalSection.bufr")) {
    Variable var = nc.findVariable("obs.seq1");
    Array data = var.read();
}

we get the following error:

java.lang.NullPointerException
	at ucar.nc2.iosp.bufr.BufrIosp2.findRootSequence(BufrIosp2.java:227)
	at ucar.nc2.iosp.bufr.BufrIosp2.readData(BufrIosp2.java:210)
	at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:2122)
	at ucar.nc2.Variable.reallyRead(Variable.java:854)
	at ucar.nc2.Variable._read(Variable.java:736)
	at ucar.nc2.Variable.read(Variable.java:614)
	at ucar.nc2.Variable.reallyRead(Variable.java:795)
	at ucar.nc2.Variable._read(Variable.java:736)
	at ucar.nc2.Variable.read(Variable.java:614)

I'm going to open a PR here shortly to disable the failing tests when running on GitHub (we've seen connectivity issues with this server before when running on GitHub), so that should help clear up the GitHub checks.

[Yaqiang]

@lesserwhirls Thanks for you review and helpful information on this PR! I mentioned that the check is failling but unfortunatelly I am very busy in these days to response quickly. Can you comment me how to do in next step? Also you mentioned the file "ecmwf-wmo16-hasOptionalSection.bufr" which I didn't find here. Otherwise I will debug the problem on this file.

[lesserwhirls]

Ah, yes, I forgot to attach the file! Here it is:

ecmwf-wmo16-hasOptionalSection.bufr.zip

These would be my suggestion for the next steps:

1. rebase or merge in the changes I've made to the maint-5.x branch into your develop branch, which will allow the unit tests to further test your changes. I suspect the unit tests will pass once you've picked up the changes to the maint-5.x branch.
2. once you have those changes, you can run ./gradlew spotlessApply to fix the codestyle check and commit those changes.
3. at that point, the bufr file above can be used to debug the issue uncovered by the larger test suite.

Thank you again!

[lesserwhirls]

I've ran the full test suite against your changes and all tests are pass! The current github failure is due to a code formatting issue:

> The following files had format violations:
32 actionable tasks: 29 executed, 3 up-to-date
      bufr/src/main/java/ucar/nc2/iosp/bufr/BufrIosp2.java
          @@ -44,10 +44,10 @@
           ····debugIter·=·debugFlag.isSet("Bufr/iter");
           ··}
           
          -··//private·Structure·obsStructure;
          -··//private·Message·protoMessage;·//·prototypical·message:·all·messages·in·the·file·must·be·the·same.
          +··//·private·Structure·obsStructure;
          +··//·private·Message·protoMessage;·//·prototypical·message:·all·messages·in·the·file·must·be·the·same.
           ··private·MessageScanner·scanner;
          -··private·List<Message>·protoMessages;··//·prototypical·messages:·the·messages·with·different·category.
          +··private·List<Message>·protoMessages;·//·prototypical·messages:·the·messages·with·different·category.
           ··private·List<RootVariable>·rootVariables;
           ··private·HashSet<Integer>·messHash;
           ··private·boolean·isSingle;
      bufr/src/main/java/ucar/nc2/iosp/bufr/tables/TableA.java
          @@ -153,8 +153,7 @@
           ····}
           
           ····public·String·toString()·{
          -······return·String.valueOf(code)·+·"·"·+·getName()·+·"·"·+
          -··············this.description;
          +······return·String.valueOf(code)·+·"·"·+·getName()·+·"·"·+·this.description;
           ····}
           
           ····@Override
  Run './gradlew :bufr:spotlessApply' to fix these violations.

If you follow the instructions of the last line and run ./gradlew :bufr:spotlessApply again and commit those changes, then everything should pass.

One question I have at this point is if you have a small sample file that can be used to test your changes? I found some prepbufr files that could work, but they are too large to be included with the source code to be part of a unit test. We could add a larger one to the datasets used by the full test suite, but it would be nice if it was a few megabytes or less in size (I have not found an example like that at this point).

[Yaqiang]

Just finished codetyle check. Unfortunately, I can not find a small size prepbufr data file also.

[Yaqiang]

Also this PR can solve the issue: #1184

[lesserwhirls]

I'll see if I can track down a smaller file for our full tests suite, or see if I can convince ecCodes to slim down one of the files that I do have (so far a lot of segfaults using bufr_filter). Thank you again for your contribution!