Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

integration test for stopping target service of proxy feature broken #772

Closed
engelmi opened this issue Mar 4, 2024 · 2 comments · Fixed by #820
Closed

integration test for stopping target service of proxy feature broken #772

engelmi opened this issue Mar 4, 2024 · 2 comments · Fixed by #820
Labels
bug Something isn't working testing This issue adds or improves the testing

Comments

@engelmi
Copy link
Member

engelmi commented Mar 4, 2024

Describe the bug

It seems that the proxy-service-stop-service feature is broken in the GH CI:
https://github.com/eclipse-bluechi/bluechi/actions/runs/8138365284/job/22240289714?pr=770#step:8:175

Running tests locally in the contianer setup doesn't lead to a failure.

It receives a timeout while waiting for the requesting.service to start (which resolves the proxy dependency).

Note:
Its currently a bit hard to debug since no journal logs and other artifacts are collected due to the pytest timeout (which can't be caught and its not intended to do so, apparently). Therefore, a small and simple custom implementation of a signal-based timeout might be better in the future.

To Reproduce

Running integration tests in the CI

Expected behavior

Test passes
@engelmi engelmi added the bug Something isn't working label Mar 4, 2024
engelmi added a commit to engelmi/bluechi that referenced this issue Mar 4, 2024
@engelmi
replaced pytest-timeout with custom timeout
913785f 
Relates to: eclipse-bluechi#772

When pytest-timeout triggers a timeout is triggered, the
test will be exited with pytest.fail and test artifacts
might not be collected - so debugging issues with timeouts
become hard to debug.
Therefore, replace pytest-timeout with a slim, custom
implementation that throws an error when timeout is reached.
In addition, lets distinguish between timeouts for setup,
test execution and artifact collection.

Signed-off-by: Michael Engel <mengel@redhat.com>
@engelmi engelmi mentioned this issue Mar 4, 2024
Merged
engelmi added a commit to engelmi/bluechi that referenced this issue Mar 4, 2024
@engelmi
replaced pytest-timeout with custom timeout
ddddd8b 
Relates to: eclipse-bluechi#772

When pytest-timeout triggers a timeout is triggered, the
test will be exited with pytest.fail and test artifacts
might not be collected - so debugging issues with timeouts
become hard to debug.
Therefore, replace pytest-timeout with a slim, custom
implementation that throws an error when timeout is reached.
In addition, lets distinguish between timeouts for setup,
test execution and artifact collection.

Signed-off-by: Michael Engel <mengel@redhat.com>
@engelmi
Copy link
Member Author

engelmi commented Mar 4, 2024 •
edited
Loading

It seems there are logs about too many open files:

11:18:34                 out: 2024-03-04 11:18:34+0000,097 DEBUG   [bluechi_test.test] Stopping all BlueChi components in all container... (test:99)                                                                                                          
11:18:34                 out: 2024-03-04 11:18:34+0000,148 DEBUG   [bluechi_test.client] Executed command 'systemctl stop bluechi-agent' with result '0' and output 'b''' (client:84)                                                                         
11:18:34                 out: 2024-03-04 11:18:34+0000,222 DEBUG   [bluechi_test.client] Executed command 'systemctl show --property="Result" bluechi-agent' with result '0' and output 'Result=success' (client:84)                                          
11:18:34                 out: 2024-03-04 11:18:34+0000,293 DEBUG   [bluechi_test.client] Executed command 'systemctl stop bluechi-agent' with result '0' and output 'Failed to allocate directory watch: Too many open files' (client:84)                     
11:18:34                 out: 2024-03-04 11:18:34+0000,368 DEBUG   [bluechi_test.client] Executed command 'systemctl show --property="Result" bluechi-agent' with result '0' and output 'Result=success' (client:84)

So this might be related to podman and the GH hosts in the CI.

Update:
It seems that if this issue occurs, it'll persist quite a while but eventually disappears after restarting the pipeline.

This was referenced Mar 4, 2024
Merged
Merged
engelmi added a commit to engelmi/bluechi that referenced this issue Mar 4, 2024
@engelmi
replaced pytest-timeout with custom timeout
9fcca39 
Relates to: eclipse-bluechi#772

When pytest-timeout triggers a timeout is triggered, the
test will be exited with pytest.fail and test artifacts
might not be collected - so debugging issues with timeouts
become hard to debug.
Therefore, replace pytest-timeout with a slim, custom
implementation that throws an error when timeout is reached.
In addition, lets distinguish between timeouts for setup,
test execution and artifact collection.

Signed-off-by: Michael Engel <mengel@redhat.com>
engelmi added a commit that referenced this issue Mar 4, 2024
@engelmi
replaced pytest-timeout with custom timeout
b2bee53 
Relates to: #772

When pytest-timeout triggers a timeout is triggered, the
test will be exited with pytest.fail and test artifacts
might not be collected - so debugging issues with timeouts
become hard to debug.
Therefore, replace pytest-timeout with a slim, custom
implementation that throws an error when timeout is reached.
In addition, lets distinguish between timeouts for setup,
test execution and artifact collection.

Signed-off-by: Michael Engel <mengel@redhat.com>
@engelmi engelmi added the testing This issue adds or improves the testing label Mar 6, 2024
@engelmi engelmi mentioned this issue Mar 7, 2024
Merged
@mwperina mwperina mentioned this issue Mar 12, 2024
Merged
@mwperina
Copy link
Member

Should be fixed by #820 feel free to reopen if appears again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working testing This issue adds or improves the testing
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Increase inotify limits for integration tests in GH CI mwperina/bluechi
2 participants
@mwperina @engelmi