pds-api-56: crawl the hierarchical tree #29
Conversation
The POM needs to request the correct version of the api JAR file. Update the 3 API interfaces with the new methods. Currently they are blank place holders and the appropriate branches will be created to fill in each of the bodies. The branches will be created from here then merged back.
Basically use the search capabilities to find where the collection lid shows up in the ref_lid_collection variable. Pretty straight forward.
Pretty straight foward finding all collections that hold a this product then all bundles that reference those collections.
|
All sub items of PDS API 56 are implemented and tested. Please review. |
There was a problem hiding this comment.
This request returns observational_products:
curl --location --request GET 'http://localhost:8080/products/urn:nasa:pds:orex.ovirs:data_raw:20191117t030457s365_ovr_scil0.fits::1.0/bundles'
I am expecting to only see bundles in the result.
I don't have a result with this request:
curl --location --request GET 'http://localhost:8080/products/urn:nasa:pds:orex.ovirs:data_raw:20191117t030457s365_ovr_scil0.fits::1.0/collections'
I am testing with the following bundle archive loaded in my registry:
dph.tar.gz
The other request work for me, I will close the tickets so that we can focus on the remaining one.
src/main/java/gov/nasa/pds/api/engineering/controllers/MyProductsApiController.java
Outdated
Show resolved
Hide resolved
|
|
||
| if (getCollectionResponse.isExists()) | ||
| { | ||
| MyBundlesApiController.log.info("get ref response " + getCollectionResponse.toString()); |
There was a problem hiding this comment.
The iteratable class available on https://github.com/NASA-PDS/registry-api-service/tree/master/src/main/java/gov/nasa/pds/api/engineering/elasticsearch/business can be used to find the list of products of a collection between start and start+limit without having to load all products in memory.
This method could also be implemented as an iterator.
That will be convenient to re-use the code since we want also to optimize the code which access the products of the collection by doing a single request for all the products instead of one per product as you do in line 277.
Since that works anyway here, we can work on the code re-useability in the optimization ticket #13
There was a problem hiding this comment.
It cannot. In order to complete the search, first have to get the bundle then all collections then all products. The limits and pagination are over all products not bundles and collections, therefore bundle and collections have to be read to build the products list before pagination can take place. Ugly, but that is part of one of the various emails with @jordanpadams were the punch line was pagination is products not collections in this case.
As I stated in that email also, the iterator also reads all the data. It is not possible to do fractional searches unless state is saved and all these interfaces are REST.
What are observational_products? How are they different than bundles? Good find on /products/x/collections. I was returning the contents from registry-ref not registry. Added content to pull actual collection so it should work correctly now. |
|
Please post the registry-ref JSON object that you have here. |
|
@al-niessner actually the first request (products/x/bunldes) works when the lidivid is available in registry-refs. I made a mistake by using a lidvid which had no bundle. In this case something weird is happening since we have some responses with your code. Maybe an unexpected elasticSearch request behavior is causing that ? |
@tloubrieu-jpl Did the lidvid not exist at all or did it identify something not a bundle? I find this behavior very intriguing. |
If the lidvid is wrong, make sure that nothing is returned. This was added because it was found that the double search of /products/x/bundles returned multiple responses instead of 0. Because the second search had no criteria it returned the entire database. Added code to prevent this then added negative checks for all URLs.
|
Identified problem and fixed it. |
|
Thanks @al-niessner I will look at that now. Per your question on bundles/obserational products you can look at the representation of the PDS4 model in section 4. Scope https://pds.nasa.gov/datastandards/documents/im/v1/index_1F00.html We can discuss that during the breakout. |
@al-niessner the lidvid exists but it does not belong to a collection. So it is available in the registry index but not in the registry-refs index. |
Summary*
Work was done in branches off this branch that match this epic's dependents. All now work according to ./verify/issue_56.py.
Test Data and/or Report
./verify/issue_56.py
Related Issues
Fixed PDS-API issues
And sub-tickets: