Issue 13.1 #45

al-niessner · 2021-06-22T22:58:40Z

Summary*
Brief summary of changes if not sufficiently described by commit messages.

Test Data and/or Report
One of the following should be included here:

Reference to regression test included in code (preferred wherever reasonable)
Attach test data here + outputs of tests

Related Issues

As a developer, I want to utilize ElasticSearch performance robustness for API response time requirements. #13 is fixed
The end point /bundles/{lidvid}/products does not fully work #36 is also fixed in this pull request

While bringing the patch across from branch issue_13, refactored it a bit and now have a re-usable bit of code with elastic search stuff in the ElasticSearchUtil class and some PDS product stuff in buisness objects. There is a mix of stuff that uses the elastic seaarch stuff and pushes it to buisness objects that was left in the controllers. May refactor some that as it is fairly repitive too.

Moved stuff common to bundles, collections, and products to the base class.

al-niessner · 2021-06-22T23:01:05Z

@tloubrieu-jpl @jordanpadams

I reworked bundles and now moving onto collections and products. Please take a look and comment please. Bundles is tested and it returns only requested fields and never loads the blob. It is highly efficient in the sense it only pulls what is needed even during the dereferencing of all the lidvid pointers.

al-niessner · 2021-06-22T23:02:10Z

@tloubrieu-jpl @jordanpadams

Oh, I will need a tutorial on why the integration is failing and how to fix it. Looks like it still wants master over main. I just do not know how to fix it.

jordanpadams · 2021-06-22T23:23:15Z

@al-niessner you can ignore those for now. once you get back to main and use the latest code that should be fine

al-niessner · 2021-06-23T16:57:02Z

@tloubrieu-jpl

Did gov.nasa.pds.model.PropertyArrayValues; not get committed? Have it in imports but cannot be resolved.

The code changes are basically for the walking forward/backward in the product tree more efficiently so used verify/issue_56.py to show that they still function correctly.

al-niessner · 2021-06-24T00:03:04Z

@tloubrieu-jpl @jordanpadams

Ready for review with 3 caveats:

verify/issue_4.py now has some failures -- getting a 200 return instead of 404 -- but not because of these changes. All of the verify/issue_56.py negative checks (bad lidvid) return 200 with no results.
Can update POM to api-0.4.0 if you like.
Could not verify performance improvement due to lack of testing ability. No way to run notebook page from As a developer, I want to utilize ElasticSearch performance robustness for API response time requirements. #13 locally.

Both collections and products from a bundle lidvid should not work through all of the available matching data.

Removed ElasticSearchUitl.collate() and updated much of the looping so that it would find all items. It was the bulk of the work. When looking child to parent, the start and limit are used as part of the search request for which collate worked well and thus put back.

src/main/java/gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchHitIterator.java

src/main/java/gov/nasa/pds/api/engineering/controllers/MyBundlesApiController.java

src/main/java/gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchHitIterator.java

Seems that asking for too much from ES causes it to really slow down. For that reason, changed some of the outer paging stuff back to a fixed default 10 items.

...va/gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchRegistrySearchRequestBuilder.java

al-niessner · 2021-07-08T01:14:57Z

@jordanpadams @tloubrieu-jpl

Still working on this problem. Got somewhat vexed today as I had the third rewrite in place and it was 10x slower. Thought I found the problem but no it was just a problem -- changing lids to lidvids is apparently expensive.

Now I have some debug messages in there and need to add a couple of more to do a complete analysis (next week since I toasted 2 days this week to get here) but the current look up is now milliseconds. However, the overall time is still the same as before this optimization. Now I am going to dig deeper to find out why. Basically it takes 2 seconds to find, build, and generate 500 products. It takes about 100 ms to find the 500 products to get. I am going to look for the 1.9 seconds that I cannot account for next week. From what I have learned, it may require optimizing the lid to lidvid function.

tloubrieu-jpl · 2021-07-08T20:33:55Z

@al-niessner that sounds good, that is getting us closer to the goal anyway. Thanks for your feedback.

al-niessner · 2021-07-13T20:02:37Z

@tloubrieu-jpl @jordanpadams

Philosophical question: self protective coding or not? The bulk of that 1.3 seconds is turning lids to lidvids. However it may not be necessary. From the registry_ref index, product_lidvids is retrieved. Using a protective coding style out of habit, the values from product_lidvids are converted to lidvids using this.productBO.getLatestLidVidFromLid() just in case a lid made it into product_lidvids. This is 1 second of the 1.3 seconds for 500 lidvids (expensive).

Now, depending on the development philosophy you want to use, there are two solutions to speed things up.

simply trust all values in product_lidvids is a lidvid and use them as is
modify this.productBO.getLatestLidVidFromLid() to short-cut if value is already a lidvid instead of checking the DB to see if it exists

Preference?

Confirmed: removing lidvid call changed from 1.3 seconds to .3 seconds which is the expected 4x speed up.

tloubrieu-jpl · 2021-07-15T18:41:27Z

Hi @al-niessner ,

I am having the following error message when I try to start the service on my latptop where an elasticsearch is on localhost with no password.
Can you look at that ?
Thanks,

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'myBundlesApiController': Unsatisfied dependency expressed through field 'esRegistryConnection'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'esRegistryConnection' defined in class path resource [gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [gov.nasa.pds.api.engineering.elasticsearch.ElasticSearchRegistryConnection]: Factory method 'ElasticSearchRegistryConnection' threw exception; nested exception is java.lang.RuntimeException: ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [GET], host [https://search-pds-dev-esext-kcq7xxa4lsrakjw33lywpjdyfy.us-west-2.es.amazonaws.com:443], URI [/_cluster/settings?master_timeout=30s], status line [HTTP/1.1 403 Forbidden]
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}];
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1422) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:893) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.3.1.RELEASE.jar:2.3.1.RELEASE]
	at gov.nasa.pds.api.engineering.SpringBootMain.main(SpringBootMain.java:27) [classes/:na]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'esRegistryConnection' defined in class path resource [gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [gov.nasa.pds.api.engineering.elasticsearch.ElasticSearchRegistryConnection]: Factory method 'ElasticSearchRegistryConnection' threw exception; nested exception is java.lang.RuntimeException: ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [GET], host [https://search-pds-dev-esext-kcq7xxa4lsrakjw33lywpjdyfy.us-west-2.es.amazonaws.com:443], URI [/_cluster/settings?master_timeout=30s], status line [HTTP/1.1 403 Forbidden]
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}];
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:655) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:483) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1304) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1224) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	... 18 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [gov.nasa.pds.api.engineering.elasticsearch.ElasticSearchRegistryConnection]: Factory method 'ElasticSearchRegistryConnection' threw exception; nested exception is java.lang.RuntimeException: ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [GET], host [https://search-pds-dev-esext-kcq7xxa4lsrakjw33lywpjdyfy.us-west-2.es.amazonaws.com:443], URI [/_cluster/settings?master_timeout=30s], status line [HTTP/1.1 403 Forbidden]
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}];
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:650) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	... 31 common frames omitted
Caused by: java.lang.RuntimeException: ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [GET], host [https://search-pds-dev-esext-kcq7xxa4lsrakjw33lywpjdyfy.us-west-2.es.amazonaws.com:443], URI [/_cluster/settings?master_timeout=30s], status line [HTTP/1.1 403 Forbidden]
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}];
	at gov.nasa.pds.api.engineering.elasticsearch.ElasticSearchRegistryConnectionImpl.checkCCSConfig(ElasticSearchRegistryConnectionImpl.java:166) ~[classes/:na]
	at gov.nasa.pds.api.engineering.elasticsearch.ElasticSearchRegistryConnectionImpl.<init>(ElasticSearchRegistryConnectionImpl.java:111) ~[classes/:na]
	at gov.nasa.pds.api.engineering.elasticsearch.ElasticSearchConfig.ElasticSearchRegistryConnection(ElasticSearchConfig.java:79) ~[classes/:na]
	at gov.nasa.pds.api.engineering.elasticsearch.ElasticSearchConfig$$EnhancerBySpringCGLIB$$f238e449.CGLIB$ElasticSearchRegistryConnection$8(<generated>) ~[classes/:na]
	at gov.nasa.pds.api.engineering.elasticsearch.ElasticSearchConfig$$EnhancerBySpringCGLIB$$f238e449$$FastClassBySpringCGLIB$$90c9893b.invoke(<generated>) ~[classes/:na]
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at gov.nasa.pds.api.engineering.elasticsearch.ElasticSearchConfig$$EnhancerBySpringCGLIB$$f238e449.ElasticSearchRegistryConnection(<generated>) ~[classes/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_241]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_241]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_241]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_241]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	... 32 common frames omitted
Caused by: org.elasticsearch.ElasticsearchStatusException: Unable to parse response body
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1880) ~[elasticsearch-rest-high-level-client-7.11.2.jar:7.11.2]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1634) ~[elasticsearch-rest-high-level-client-7.11.2.jar:7.11.2]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1591) ~[elasticsearch-rest-high-level-client-7.11.2.jar:7.11.2]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1561) ~[elasticsearch-rest-high-level-client-7.11.2.jar:7.11.2]
	at org.elasticsearch.client.ClusterClient.getSettings(ClusterClient.java:87) ~[elasticsearch-rest-high-level-client-7.11.2.jar:7.11.2]
	at gov.nasa.pds.api.engineering.elasticsearch.ElasticSearchRegistryConnectionImpl.checkCCSConfig(ElasticSearchRegistryConnectionImpl.java:155) ~[classes/:na]
	... 44 common frames omitted
	Suppressed: org.elasticsearch.common.ParsingException: Failed to parse object: expecting field with name [error] but found [Message]
		at org.elasticsearch.common.xcontent.XContentParserUtils.ensureFieldName(XContentParserUtils.java:38) ~[elasticsearch-7.11.2.jar:7.11.2]
		at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:581) ~[elasticsearch-7.11.2.jar:7.11.2]
		at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:168) ~[elasticsearch-7.11.2.jar:7.11.2]
		at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1900) ~[elasticsearch-rest-high-level-client-7.11.2.jar:7.11.2]
		at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1877) ~[elasticsearch-rest-high-level-client-7.11.2.jar:7.11.2]
		... 49 common frames omitted
Caused by: org.elasticsearch.client.ResponseException: method [GET], host [https://search-pds-dev-esext-kcq7xxa4lsrakjw33lywpjdyfy.us-west-2.es.amazonaws.com:443], URI [/_cluster/settings?master_timeout=30s], status line [HTTP/1.1 403 Forbidden]
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}
	at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326) ~[elasticsearch-rest-client-7.11.2.jar:7.11.2]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296) ~[elasticsearch-rest-client-7.11.2.jar:7.11.2]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.11.2.jar:7.11.2]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1621) ~[elasticsearch-rest-high-level-client-7.11.2.jar:7.11.2]
	... 48 common frames omitted

al-niessner · 2021-07-20T17:32:00Z

@tloubrieu-jpl

Sorry but I cannot reproduce the problem. Pulled the branch from github -- already up to date so no changes -- and built the docker containers. Ran the tests verify/issue_56.py which does a products of bundle with no errors. Scrolled down through the logs and found no errors like the one you have. It all works on pds-gamma as well.

Maybe your configuration like the application.properties or something?

tloubrieu-jpl

Thanks @al-niessner , I've seen 2 minor bug that we will look at (or not) after, it is not blocking the merge.
The performances are improved by 1/3 on my test (15 minutes --> 11 minutes) so that is good !

I have just a request for comments in a part of the code that I suspect will be difficult to maintain otherwise.

Thanks, sorry for the delay in the review

src/main/java/gov/nasa/pds/api/engineering/controllers/MyBundlesApiController.java

tloubrieu-jpl · 2021-07-22T23:25:36Z

src/main/java/gov/nasa/pds/api/engineering/controllers/MyBundlesApiController.java

    	HashSet<String> uniqueProperties = new HashSet<String>();
-    	List<String> productLidvids = new ArrayList<String>();
+    	List<String> clidvids = this.getRefLidCollection(lidvid, limit);


it is unclear to me here as well: we are asking for product children of bundle from start to start+limit, I don't see how that can match with the collection pagination, so we can not use the limit parameter here

As state above, it is used a random number more than anything else. It could be fixed to 10 or 3 or 50 or 100. Much bigger than 100 and elasticsearch returns an error regularly.

It is an interesting question of what to set the limit of the collection of a bundle to when looking for the product.

The idea is that if the user wants to wait for bigger chunks like 100 products then also getting 100 collections is better than 10 collections then having to ask again for the other 90 in the case of one product per collection.

This code no longer exists (outdated) so not sure what comments are required.

src/main/java/gov/nasa/pds/api/engineering/controllers/MyCollectionsApiController.java

src/main/java/gov/nasa/pds/api/engineering/controllers/MyBundlesApiController.java

src/main/java/gov/nasa/pds/api/engineering/controllers/MyProductsApiController.java

tloubrieu-jpl

@al-niessner , @jordanpadams

I retested the branch in pds-gamma with the notebokk and the performance are not getting better:
branch issue13.1

retrieved 279500 products in 25.9 minutes

branch main

retrieved 279500 products in 14.2 minutes

So I suggest to table this pull request for now. We can discuss that tomorrow during the breakout.

i will retest now in case there are some caching effects on elasticsearch which create a bias in the peasured performances. I'll let you know.

src/main/java/gov/nasa/pds/api/engineering/controllers/MyBundlesApiController.java

src/main/java/gov/nasa/pds/api/engineering/controllers/MyCollectionsApiController.java

al-niessner · 2021-08-04T21:37:58Z

@tloubrieu-jpl

When testing, just make sure network is the same. It is currently the biggest variable and causes timing to really, really swim around. I use 3 networks regularly and they all vary in how long it takes by 50% or so. If using VPN, then rate will vary minute by minute.

I have not noticed an order problem (run master or 13.1 first then followed by other) in timing. Not to say that it does not exist, but I have not noticed it.

tloubrieu-jpl · 2021-08-04T22:15:51Z

@tloubrieu-jpl

When testing, just make sure network is the same. It is currently the biggest variable and causes timing to really, really swim around. I use 3 networks regularly and they all vary in how long it takes by 50% or so. If using VPN, then rate will vary minute by minute.

I have not noticed an order problem (run master or 13.1 first then followed by other) in timing. Not to say that it does not exist, but I have not noticed it.

I did re-test the new branch and had the same duration 25 minutes. So the order does not change you are right.

Otherwise, per the network question, I had rather stable duration in the past and still now although I don't have a big series of tests (2 !)

al-niessner · 2021-08-05T17:15:29Z

@tloubrieu-jpl

Had to undo one of your requests (see last commit) to make it fast again.

Al Niessner added 2 commits June 22, 2021 15:47

refactored some more

439a308

Moved stuff common to bundles, collections, and products to the base class.

Al Niessner and others added 3 commits June 22, 2021 16:51

update collections controller

b85e78a

update product

208a95b

Merge branch 'main' into issue_13.1

4d2f2cf

al-niessner and others added 3 commits June 23, 2021 11:55

Merge branch 'main' into issue_13.1

0731a41

merge of main repair

dee3ff3

make all fo issue_56 checks work

d85ce47

The code changes are basically for the walking forward/backward in the product tree more efficiently so used verify/issue_56.py to show that they still function correctly.

Al Niessner added 10 commits July 1, 2021 11:15

Merge branch 'main' into issue_13.1

f50daa9

also collect default fields

508bcf1

request maximum data

b3bfb7d

request maximum data

2fdfda2

request maximum data

d12d817

request maximum data

fa58a54

revert and find another way

cb9f030

updated the bundle endpoint

2e2e52b

Both collections and products from a bundle lidvid should not work through all of the available matching data.

second rework

47e54e5

Removed ElasticSearchUitl.collate() and updated much of the looping so that it would find all items. It was the bulk of the work. When looking child to parent, the start and limit are used as part of the search request for which collate worked well and thus put back.

++ operator behaves oddly

78b561c

sonatype-lift bot reviewed Jul 7, 2021

View reviewed changes

src/main/java/gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchHitIterator.java Show resolved Hide resolved

sonatype-lift bot reviewed Jul 7, 2021

View reviewed changes

src/main/java/gov/nasa/pds/api/engineering/controllers/MyBundlesApiController.java Show resolved Hide resolved

sonatype-lift bot reviewed Jul 7, 2021

View reviewed changes

src/main/java/gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchHitIterator.java Outdated Show resolved Hide resolved

Al Niessner added 4 commits July 7, 2021 11:31

fix auto checks

b8bc666

gamma debug messages

ea50daf

too much

3819ca7

Seems that asking for too much from ES causes it to really slow down. For that reason, changed some of the outer paging stuff back to a fixed default 10 items.

making a lidvid is expensive??

3289191

sonatype-lift bot reviewed Jul 7, 2021

View reviewed changes

...va/gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchRegistrySearchRequestBuilder.java Outdated Show resolved Hide resolved

Al Niessner added 3 commits July 7, 2021 15:42

have to jump forward at times

0d94733

delay is aback to large gets from ES

339488f

more auto check fixes

b480f0c

more timing data

5ba30ee

Al Niessner added 2 commits July 13, 2021 13:14

test no lid to lidvid safety

6c50437

option 1

f1634c4

tloubrieu-jpl suggested changes Jul 22, 2021

View reviewed changes

Al Niessner added 3 commits July 23, 2021 13:34

requested updates

a22f69a

add comments

fc86d5e

remove extra use of limit

5ba6c93

sonatype-lift bot reviewed Jul 30, 2021

View reviewed changes

src/main/java/gov/nasa/pds/api/engineering/controllers/MyBundlesApiController.java Outdated Show resolved Hide resolved

sonatype-lift bot reviewed Jul 30, 2021

View reviewed changes

src/main/java/gov/nasa/pds/api/engineering/controllers/MyProductsApiController.java Outdated Show resolved Hide resolved

remove extra use of limit

bd66f5d

tloubrieu-jpl reviewed Aug 4, 2021

View reviewed changes

get full list of products if ready

40b9239

tloubrieu-jpl approved these changes Aug 10, 2021

View reviewed changes

tloubrieu-jpl merged commit 7461140 into main Aug 10, 2021

jordanpadams mentioned this pull request Aug 27, 2021

As a developer, I want to utilize ElasticSearch performance robustness for API response time requirements. #13

Closed

2 tasks

jordanpadams mentioned this pull request Sep 24, 2021

The end point /bundles/{lidvid}/products does not fully work #36

Closed

tdddblog deleted the issue_13.1 branch September 27, 2021 22:17

alexdunnjpl mentioned this pull request Feb 14, 2023

/classes/collections/<lidvid>/members (and deprecated equivalent) hangs NASA-PDS/registry-api#231

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 13.1 #45

Issue 13.1 #45

al-niessner commented Jun 22, 2021 •

edited by tloubrieu-jpl

Loading

al-niessner commented Jun 22, 2021

al-niessner commented Jun 22, 2021

jordanpadams commented Jun 22, 2021

al-niessner commented Jun 23, 2021

al-niessner commented Jun 24, 2021

al-niessner commented Jul 8, 2021

tloubrieu-jpl commented Jul 8, 2021

al-niessner commented Jul 13, 2021 •

edited

Loading

tloubrieu-jpl commented Jul 15, 2021

al-niessner commented Jul 20, 2021

tloubrieu-jpl left a comment •

edited

Loading

tloubrieu-jpl Jul 22, 2021

al-niessner Jul 23, 2021

al-niessner Aug 10, 2021

tloubrieu-jpl left a comment

al-niessner commented Aug 4, 2021

tloubrieu-jpl commented Aug 4, 2021

al-niessner commented Aug 5, 2021

Issue 13.1 #45

Issue 13.1 #45

Conversation

al-niessner commented Jun 22, 2021 • edited by tloubrieu-jpl Loading

al-niessner commented Jun 22, 2021

al-niessner commented Jun 22, 2021

jordanpadams commented Jun 22, 2021

al-niessner commented Jun 23, 2021

al-niessner commented Jun 24, 2021

al-niessner commented Jul 8, 2021

tloubrieu-jpl commented Jul 8, 2021

al-niessner commented Jul 13, 2021 • edited Loading

tloubrieu-jpl commented Jul 15, 2021

al-niessner commented Jul 20, 2021

tloubrieu-jpl left a comment • edited Loading

Choose a reason for hiding this comment

tloubrieu-jpl Jul 22, 2021

Choose a reason for hiding this comment

al-niessner Jul 23, 2021

Choose a reason for hiding this comment

al-niessner Aug 10, 2021

Choose a reason for hiding this comment

tloubrieu-jpl left a comment

Choose a reason for hiding this comment

al-niessner commented Aug 4, 2021

tloubrieu-jpl commented Aug 4, 2021

al-niessner commented Aug 5, 2021

al-niessner commented Jun 22, 2021 •

edited by tloubrieu-jpl

Loading

al-niessner commented Jul 13, 2021 •

edited

Loading

tloubrieu-jpl left a comment •

edited

Loading