Thredds server return me a extreme big value among multiple thredds environment

[kensinzl]

We are using the Thredds server 5.4 docker image(unidata/thredds-docker:5.4) and deploying it as a docker container.

Now I have faced one issue and makes me annoying how to fix

• I have a Java App which is using multiple threads to fetch several NetCDF files from the above Thredds server to boostup the response time.

• These requests are among a defined lat&lon range, which means each request with different lat&lon value but with the same time range.

• Majority of response is normal, but some random NetCDF files' response would give me an extremely big value than expected.

• But if I make a single HTTP request for the same params, the extremely big value will back to normal.

eg:
https://thredds/ncss/grid/NetCDF_1?var=Var_1&latitude=-37&longitude=174&time_start=2023-08-13T12%3A00%3A00Z&time_end=2023-08-19T12%3A00%3A00Z&timeStride=&vertCoord=&accept=csv
https://thredds/ncss/grid/NetCDF_1?var=Var_1&latitude=-36&longitude=176&time_start=2023-08-13T12%3A00%3A00Z&time_end=2023-08-19T12%3A00%3A00Z&timeStride=&vertCoord=&accept=csv
https://thredds/ncss/grid/NetCDF_2?var=Var_2&latitude=-37&longitude=174&time_start=2023-08-13T12%3A00%3A00Z&time_end=2023-08-19T12%3A00%3A00Z&timeStride=&vertCoord=&accept=csv
https://thredds/ncss/grid/NetCDF_2?var=Var_2&latitude=-36&longitude=176&time_start=2023-08-13T12%3A00%3A00Z&time_end=2023-08-19T12%3A00%3A00Z&timeStride=&vertCoord=&accept=csv
......

if I found the following request response at log showing an extremely big value among multiple threads, then I hit the exactly same HTTP request again by itself, and then the value will be back to normal.

https://thredds/ncss/grid/NetCDF_2?var=Var_2&latitude=-36&longitude=176&time_start=2023-08-13T12%3A00%3A00Z&time_end=2023-08-19T12%3A00%3A00Z&timeStride=&vertCoord=&accept=csv

Any idea, please

[haileyajohnson]

Is there a reason you need each lat/lon as a separate request? We do have an API for subsetting the data on a single time and a range of lat/lons.

[kensinzl]

@mnlerman

Thanks for the reply.

• for the concurrent requests, it seems to be a known issue, and other people have already raised it, eg: Fix race condition in the h5iosps #1122.
  I am also waiting for the Thredds server image 5.5 at the docker hub to officially be out because this version seems to fix the race condition issue.
  If we do multiple requests syn, the response time is not good if faces thousands of requests.

• anyway, back to the lat/lon bounding box request, thanks for the info, but current business logic needs the grid point feature.
  even with the bounding box, the Thredds(ServerA) may concurrently receive two requests for the same NetCDF file fetching from Server B and Server C, something like this.

[mnlerman]

We are working to get the official release of thredds-5.5 out. Meanwhile, we do have thredds-5.5-SNAPSHOT available on Docker Hub which does have the changes that you're looking at. Have you had a chance to test that out?

Another possible solution you might try is to accessing multiple files asynchronously but staggering points in the same file so that only one request is being sent per file at a time. This would still allow you to get the grid as point features asynchronously without trying to read the same file when it's already open

[kensinzl]

@mnlerman

Thank you for the reply

• Yes, I am aware that there is a 5.5-SNAPSHOT version, but consider it is SNAPSHOT, so better for me to wait until the official version is launched because I use Thredds in the prod environment, just in case.

• could you please share more details for your second sentence for the solution, I want to make sure that I am understanding your meaning for the correct page.

1. accessing multiple files asynchronously but staggering points in the same file so that only one request is being sent per file at a time
   => eg, I have 10 requests from Server B to Thredds(ServerA), these 10 requests are fetching 5 NetCDF files with the same time range and every 2 requests for the same NetCDF file but with different point(lat/lon)

do you mean I should send requests order by NetCDF files rather than without order?

=> without trying to read the same file when it's already open

do you mean when Thredds receive a request, it will open the related NetCDF file, if the next coming request among a 'Thredds configured time range' is still the same NetCDF file, the Thredds will not open the file again? if so, I guess this is the reason you hope for me to order the request by NetCDF name if my understanding is correct for your first sentence.

[mnlerman]

I mean the you can make two requests to two separate files asynchronously, but you can't make two requests to one file asynchronously

From the example requests that you posted, I mean that these two requests can run at the same time:

https://thredds/ncss/grid/NetCDF_1?var=Var_1&latitude=-37&longitude=174&time_start=2023-08-13T12%3A00%3A00Z&time_end=2023-08-19T12%3A00%3A00Z&timeStride=&vertCoord=&accept=csv
https://thredds/ncss/grid/NetCDF_2?var=Var_2&latitude=-37&longitude=174&time_start=2023-08-13T12%3A00%3A00Z&time_end=2023-08-19T12%3A00%3A00Z&timeStride=&vertCoord=&accept=csv

And these two requests can run at the same time:

https://thredds/ncss/grid/NetCDF_1?var=Var_1&latitude=-36&longitude=176&time_start=2023-08-13T12%3A00%3A00Z&time_end=2023-08-19T12%3A00%3A00Z&timeStride=&vertCoord=&accept=csv
https://thredds/ncss/grid/NetCDF_2?var=Var_2&latitude=-36&longitude=176&time_start=2023-08-13T12%3A00%3A00Z&time_end=2023-08-19T12%3A00%3A00Z&timeStride=&vertCoord=&accept=csv

because they are not reading from the same file. I think you're encountering problems when you're reading NetCDF_1 on thread 1 and then thread 2 tries to read NetCDF_1 and resets the cursor. If you wait for the response from NetCDF_1, then send the next request for NetCDF_1, you shouldn't encounter the same problems. One way to achieve this might be to send all of the requests for for a file on a single thread so that there can't be multiple concurrent requests to the same file.

[kensinzl]

@mnlerman

Thanks for the details information.

em.... seems if the Thredds can not deal with multiple requests concurrently well for one NetCDF, that will be inconvenient here.

So far, there are two isolated servers to hit the Thredds server.

Considering that more and more users come to use it in the future, it is hard to avoid there being no two requests for the same NetCDF file concurrently. Meanwhile if do the syn logic, the performance will be slow for business logic which needs a huge amount of requests.

[haileyajohnson]

Hi @kensinzl , yes I expect you are right that the issue is fixed by #1122
What @mnlerman was trying to offer was just a workaround to minimize the chance of hitting the race condition you're seeing. The ideal solution would be to use the 5.5-SNAPSHOT. We are working to get the official release out, but at the moment the latest SNAPSHOT release IS the recommended version to be used in production (5.4 has a few security exploits that are fixed in 5.5-SNAPSHOT. It would also help us if you could just test out 5.5 and let us know is the issue is resolved.

Thanks!

[kensinzl]

@mnlerman @mnlerman

Thanks for the reply. Now I will do the following several things to have a try.

• try the 5.5-SNAPSHOT at the UAT version
• Considering this issue may bring from the concurrent scenario, I will have a re-try HTTP request when I I receive the extremely big value
• last thing, I had a quick try at the range of lan/lat for each request, and it seems did not give me the JSON response, and it auto downloads the subset(smaller) NetCDF file. I may miss something. if possible, could you please share a URL, then I can have a hit.

This is my current grid point URL, then it works
/thredds/ncss/grid/?var=uv02&latitude=-37&longitude=174&time_start=2023-08-17T18%3A00%3A00Z&time_end=2023-08-19T18%3A00%3A00Z&timeStride=&vertCoord=&accept=csv

I changed it with the North, West, East and South, but not work? any idea
/thredds/ncss/grid/?var=uv02&north=-33.941&west=166.093&east=179.703&south=-47.677&time_start=2023-08-17T18%3A00%3A00Z&time_end=2023-08-19T18%3A00%3A00Z&timeStride=&vertCoord=&accept=csv

Thanks so much, hope you also have a great weekend. thanks this whole week help.

[kensinzl]

Hey, @haileyajohnson for the reply.

Currently we are using this DAP4 API to fetch one attribute from the response(XML style) at the 5.4, which is suitable for us one particular business case.

I just found oh... same URL for 5.5-snapshot is throwing the exception for me, so I made this post to let you know, so your team will fix this issue at the later commit, right?

once again, thank you so much

[haileyajohnson]

@kensinzl, we are working on DAP4 support (you can follow along with pull requests such as this one) and we will re-enable DAP4 requests once it is operational.

[kensinzl]

@haileyajohnson Thanks for the reply and I will keep an eye on this pull request.

On the other hand, could you please have a look at the previous message, I just copied the following.

Please help me double confirm, from my test, I think the range lat/lon for TDS will let you download a subset NetCDF file rather than give you a json response. Once again, thank you so much

---

Thanks for the reply. I list what I have tried, and please have a look.

/thredds/ncss/grid/<NetCDF_File>?var=uv02&north=-33.941north=-33.941&west=166.093&east=179.703&south=-47.677&horizStride=1&time_start=2023-08-21T12:00:00Z&time_end=2023-08-21T13:00:00Z&&&accept=netcdf3

=> Tried the above URL for TDS from csv into netcdf3, and it made me download a subset(smaller) NetCDF file, but I hope it can give me the JSON(CSV) response like Grid Single Point because now my client needs the JSON(CSV) file, something like this pattern "2023-02-13T12:00:00Z, GridPointRequestedAt[37.000S_174.000E],-36.996,174.000,291.5084"

Thank you so much

[haileyajohnson]

That is correct, csv is not supported for grid requests.

[kensinzl]

@haileyajohnson

Thanks for the reply. Yes, I know csv is not supported.

now I have changed to netcdf3, and the response is to let me download a subset of NetCDF rather than give me the JSON response.

So my question is 1. Does a range lon/lat request behaviour like this, e.g.: download a subset NetCDF file? 2. can a range lon/lat request return a JSON response?

thank you so much