-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Orion-LD entity size limitations and configuration (-inReqPayloadMaxSize) #1356
Comments
ok, I actually implemented that myself for Orion (but, that was after forking out Orion's repo to start building Orion-LD, that's why it's not in Orion-LD). |
Should be fixed now. Same CLI option names as Orion. |
Hello, thank you very much for the quick reply & fix. I have been able to test it with docker build fiware/orion-ld:latest (8 hours ago) at my test infrastructure. Orion-LD is able to start with the new parameter now. I will need to test in our plant once I am there again to see if large entities can also be processed in production now. I'll keep you up to date. Two further things I noticed:
Greetings |
ok, noted. The second nuisance is all me. |
Hello @kzangeli, regarding the second issue it would be great if Orion-LD is able to use a self-hosted core context URL like Mintaka does for airgapped environments in the future. That way there is no need for dump and update of the dump each time you change something bigger anymore and deployment in airgapped environments would be simplefied. But for now it is also working with the Mongo-DB dump as a workaround. I had the chance to test the fix at our productive location. Therefore I used the version compiled 04/27/2023 and have been able to start Orion-LD with the new parameter (I experienced that orion throws an error in older versions if you start it with a non-existing parameter, so I assume orion knows the parameter). Unfortunately we have two issues with it:
To make sure that I am testing correctly can you confirm following?
I can continue testing next week. Thank you in advance! Greetings |
Hi Inga, I'm out travelling today (going home from Berlin), but I will take a closer look once I'm home. I'll let you know |
Hi again, the built-in core context is (hopefully) taken care of: #1378. About the "too large payload", the logfile seems to be from an iot-agent ... And, about postgres ... About the error in itself, it complains about not finding any connection in the connection pool. |
So, I ran some tests with "postgres server not there" but it all worked just as it should:
So, not able to reproduce the problem you're seeing. |
Hello @kzangeli, thank you very much for your response and sorry for my late reaction (I have been traveling as well). I will do the test again at our power plant on Wednisday. Therefore I will use latest image by today (1.3.0-PRE-1382-debug -> 30.05.23) and test large entities + postgres issues. Regarding the postgres issue: I don't understand why it throws the error, because I can query Mintaka for STH values without issue, persist new data to time-series-DB via Orion-LD and so on. I would say it is working fine, if there would be no error in log. But maybe we can see something with traces in the new image. So I will send an update on Wednisday or Thursday after my tests. |
Hello @kzangeli, sorry for late reply - it took some more time until I have been able to test again. Unfortunately Orion-LD still throws the "Entity too large issue". I have been using image orion-ld_1.3.0-PRE-1382 (no debug version because the image is huge). Test with large entity (~ 54 entities with ~ 3000 Tags)
Same environment, but only with 6 out of the 54 entities
I am not sure what is the issue with that datasource. I have two further OPC UA datasources in the same environment and both are working perfectly (IoT-Agent reads data, sends to Orion-LD, uses Mintaka to store to TSDB, visualize with Grafana -> I can also see no errors in Oiron-LD logs for this datasources). Earlier I also have been successful to persist some values for the datasource with the issue (07.06.) - I guess the second postgres issue might be related to my new Mintaka version and is another issue I might reference in a new ticket if I am not able to debug & solve it myself. Regarding the actual "Entity too large" issue I would say the second example with only a few entities shows that the new parameter -inReqPayloadMaxSize isn't working at least in my setup => Somehow Orion-LD still thinks that the entity is too large. Here is also the IoT-Agent configuration from the second testcase as it is not so huge than the first. Can you see any issue with the entities or attribute names that might make a difference and cause the issues at Orion-LD / Mintaka?
|
Hello @kzangeli, I just wanted to update you: The feature inReqPayloadMaxSize is working on Orion-LD size. I can now consume a large amount of entities and parameters from IoT-Agent OPC UA with Orion-LD without any issue (~ 50 entities, 4000 parameters). The size limitation has been on the IoT-Agent OPC UA side, because of the underlying iotagent-node-lib library. With the latest release I have now been able to increase the size limitation at IoT-Agent and Orion-LD side and the large amount of entities and parameters can now be persisted correctly. Thank you very much for your support and the implementation of the feature inReqPayloadMaxSize at Orion-LD 👏 Greetings Inga |
Hello,
we are using FIWARE with quite large entities and seem to run into an Orion-LD Context Broker limitation. In our scenario we collect around 4.000 tags from about 50 entities with OPC UA IoT-Agent at once and try to send them to Orion-LD Context Broker via Agent. Unfortunately Orion-LD Context Broker doesn't allow 'too large entities' and denies the request. You can find a screenshot, more details and kind of a solution for the issue here: Engineering-Research-and-Development/iotagent-opcua#110
The solution provided is accoring to Orion (not Orion-LD). It seems that Orion has a build in limitation for incoming requests of 1 MB to prevent denial of service attacks. You can configure this limitation via
-inReqPayloadMaxSize
parameter and also allow larger entities: https://github.com/telefonicaid/fiware-orion/blob/0dc77185b8f98274840abc28364e2bd071c1d7e4/doc/manuals/admin/perf_tuning.md#payload-and-message-size-and-performanceAs we are using Orion-LD we'd like to ask if there is already a setting like that for Orion-LD? I tried to use it in my installation (Orion-LD 1.0.1), but the parameter doesn't seem to exist. Moreover I searched in this repository, but could not find anything about request limitation sizes for Orion-LD.
Thank you in advance and greetings,
Inga
The text was updated successfully, but these errors were encountered: