unstable tests #525

RomanBelozerov · 2023-09-28T09:55:45Z

Tempesta - ff0f02145172b92794ef3cd086a12af737caa793
Tempesta-test - ff0860655187b159cd981399961f9f1345e09b58

Any config:

ws tests receive warning, but can be successful.
t_frang.test_http_resp_code_block - AssertionError: 0 != 1 : Expected nums of warnings in journalctl: 1, but got 0
tls.test_tls_integrity.ManyClientsH2 with body greater than 64 KB receives incorrect hash.

FAIL: test_various_req_resp_sizes (tls.test_tls_integrity.ManyClientsH2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/_PR_tempesta-test_PR-509/tls/test_tls_integrity.py", line 251, in test_various_req_resp_sizes
    self.common_check(65536, 65536)
  File "/home/jenkins/workspace/_PR_tempesta-test_PR-509/tls/test_tls_integrity.py", line 328, in common_check
    self.assertTrue(hash1 == hash2, "Bad response checksum")
AssertionError: False is not true : Bad response checksum

sessions.test_h2_sticky_scheduler.H2StickySchedulerTestCase.test_h2_cookie_scheduler - doesn't receive a warning in dmesg sometimes.

Local config:

t_long_body.test_long_response.LongBodyInResponse - these tests do not work correctly after DeproxyServer reworking. Now tests work for me with 100 MB body. Maybe Nginx should be used for these tests.
access_log.test_access_log.AccessLogFrang and access_log.test_access_log.AccessLogTest sometimes don't receive a access log message in dmesg, but actually the message is present;

Local config with TCP seg 1:

~~http2_general.test_h2_hpack.TestFramePayloadLength.test_large_frame_payload_length~~ fixed in #2021

FAIL: test_large_frame_payload_length (http2_general.test_h2_hpack.TestFramePayloadLength)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/_PR_tempesta_PR-1981/tempesta-test/http2_general/test_h2_hpack.py", line 792, in test_large_frame_payload_length
    self.assertIn(ErrorCodes.PROTOCOL_ERROR, client.error_codes)
AssertionError: <ErrorCodes.PROTOCOL_ERROR: 1> not found in [<class 'ssl.SSLWantReadError'>]

http2_general/test_h2_streams.TestH2Stream.test_max_concurrent_stream - test fail when runnig all tests in http2_general -T 1 http2_general. I cannot reproduce fail with -R option. client.error_codes when test fail - [<class 'ssl.SSLWantReadError'>, <class 'ssl.SSLWantReadError'>]

Traceback (most recent call last):
  File "/home/jenkins/workspace/Daily_remote_tests/tempesta-test/http2_general/test_h2_streams.py", line 38, in test_max_concurrent_stream
    self.assertTrue(client.wait_for_reset_stream(stream_id=client.stream_id - 2))
AssertionError: False is not true

Remote config:

access_log - remote tests sometimes freeze on CI. Last test - access_log.test_access_log_h2.Response404Test.test_tempesta.
t_frang.test_connection_rate_burst.FrangTcpVsBoth

FAIL: test_rate (t_frang.test_connection_rate_burst.FrangTcpVsBoth)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/Run_remote_tests/tempesta-test/t_frang/test_connection_rate_burst.py", line 351, in test_rate
    self.check_connections(
  File "/home/jenkins/workspace/Run_remote_tests/tempesta-test/t_frang/frang_test_case.py", line 112, in check_connections
    warns_occured = self.assertFrangWarning(warning, resets_expected)
  File "/home/jenkins/workspace/Run_remote_tests/tempesta-test/t_frang/frang_test_case.py", line 132, in assertFrangWarning
    self.assertTrue(self.klog.find(warning, cond=dmesg.amount_equals(expected)), expected)
AssertionError: False is not true : 1

Please remove these tests after fix from tests_disabled*.json files and from tests_retry.

The text was updated successfully, but these errors were encountered:

voodam · 2023-10-04T12:59:14Z

Also see these:

voodam · 2023-10-04T12:59:35Z

./run_tests.py -T 1:

FAIL: test_large_frame_payload_length (http2_general.test_h2_hpack.TestFramePayloadLength)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/jenkins/workspace/_PR_tempesta_PR-1981/tempesta-test/http2_general/test_h2_hpack.py", line 792, in test_large_frame_payload_length

    self.assertIn(ErrorCodes.PROTOCOL_ERROR, client.error_codes)

AssertionError: <ErrorCodes.PROTOCOL_ERROR: 1> not found in [<class 'ssl.SSLWantReadError'>]

Flaky test on TCP segmentation (pretty much, about half of tries). I tried to find the place where it was introduced, but didn't find, probably it was enabied by:

commit cac7c14f8b3e93b4be7ad0a872ec2d079a984820 (HEAD)
Author: EvgeniiMekhanik <em@tempesta-tech.com>
Date:   Mon Jul 24 10:48:05 2023 +0300

    Enabled tests blocked by PR 1926

EvgeniiMekhanik · 2024-02-20T15:53:55Z

test_block_action_error_reply_with_conn_close_garbage (http2_general.test_h2_block_action.BlockActionH2ReplyFramesAfterShutdownWithCustomErrorPageSmallWindow)
is also unstable with TCP segmentation These tests were reworked

EvgeniiMekhanik · 2024-03-05T07:22:54Z

tls.test_tls_cert.WrkTestsMultipleVhosts.test_wrk
is also unstable with TCP segmentation

======================================================================
ERROR: test_wrk (tls.test_tls_cert.WrkTestsMultipleVhosts)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/helpers/remote.py", line 107, in run_cmd
    assert p.returncode == 0, "Cmd: '%s' return code is not 0 (%d)." % (
AssertionError: Cmd: '/root/tempesta/scripts/tempesta.sh --reload' return code is not 0 (1).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/tls/test_tls_cert.py", line 875, in test_wrk
    self.config_changer(10)
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/tls/test_tls_cert.py", line 861, in config_changer
    self.get_tempesta().reload()
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/helpers/control.py", line 334, in reload
    self._do_run(f"{self.srcdir}/scripts/tempesta.sh --reload")
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/helpers/control.py", line 346, in _do_run
    self.node.run_cmd(cmd, timeout=30, env=env, err_msg=(self.err_msg % "start"))
  File "/home/jenkins/workspace/_PR_tempesta_PR-2072/tempesta-test/helpers/remote.py", line 121, in run_cmd
    raise CmdError(
helpers.remote.CmdError: ('Сmd /root/tempesta/scripts/tempesta.sh --reload exited with return code 1', b'...compile html templates for JS challenge\nRunning live reconfiguration of Tempesta...\nERROR: cannot reconfigure Tempesta FW (sysctl message: \n0), please check dmesg\n', b'', 1)

----------------------------------------------------------------------

RomanBelozerov · 2024-08-29T09:47:41Z

Tempesta - 95f63a83a16d1a2e41131c364261e0749110697e
tempesta-test - 875741e

Any setups:

t_frang.test_concurrent_connections.ConcurrentConnections - work unstable on CI. I cannot reproduce on my VM. Most likely the clients in tests establish the connections in different order (unexpected).
~~t_stress.test_header_leak - work unstable on CI, I cannot reproduce in my VM. Probably VMs on CI requires more time.~~ disable by 2181 issue from TempestaFW repo
t_frang.test_ip_block.FrangIpBlockConn - like for t_frang.test_concurrent_connections.ConcurrentConnections. These tests should rework as test_request_rate_burst.py or without rate limits

EvgeniiMekhanik · 2024-09-26T16:05:51Z

Tempesta - 487bf2c9f4750d5eed94d1b0c34bf7763d44e511
tempesta-test - 7ef6681

access_log.test_access_log.AccessLogTest local setup
selftests.test_deproxy.DeproxyTestH2 wait_for_headers_frame returned True, but client did not add a new response to buffer.

EvgeniiMekhanik · 2024-10-03T07:17:52Z

Tempesta - 4647212ef7143d4b88c20dc4c7d1dc20574962d1
tempesta-test - f826adb

~~test_bad_host (tls.test_tls_handshake.TlsVhostHandshakeTest) remote setup~~ fixed in fix tls.test_tls_handshake.TlsVhostHandshakeTest.test_bad_host #720

RomanBelozerov · 2024-10-14T12:31:25Z

Tempesta - fa7cd2d
tempesta-test - 80fd04d

The tests does not work on CI with TCP segmentation. You must run all tests suite before reproducing an exception. First ./run_tests.py -v 1 and than ./run_tests.py -v 1 cache.test_purge.TestPurgeAcl.test_purge_acl_ipv6
Traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
  File "/home/jenkins/workspace/Daily_remote_tests/tempesta-test/cache/test_purge.py", line 89, in test_purge_acl
    self.start_all_services()
  File "/home/jenkins/workspace/Daily_remote_tests/tempesta-test/test_suite/tester.py", line 475, in start_all_services
    self.start_all_clients()
  File "/home/jenkins/workspace/Daily_remote_tests/tempesta-test/test_suite/tester.py", line 324, in start_all_clients
    client.start()
  File "/home/jenkins/workspace/Daily_remote_tests/tempesta-test/framework/stateful.py", line 64, in start
    self.run_start()
  File "/home/jenkins/workspace/Daily_remote_tests/tempesta-test/framework/deproxy_client.py", line 148, in run_start
    raise e
  File "/home/jenkins/workspace/Daily_remote_tests/tempesta-test/framework/deproxy_client.py", line 138, in run_start
    deproxy.Client.run_start(self)
  File "/home/jenkins/workspace/Daily_remote_tests/tempesta-test/helpers/deproxy.py", line 1028, in run_start
    self.bind((self.bind_addr, 0))
  File "/usr/lib/python3.10/asyncore.py", line 333, in bind
    return self.socket.bind(addr)
OSError: [Errno 99] Cannot assign requested address

cache.test_purge.TestPurgeAcl.test_purge_acl_ipv6
cache.test_purge.TestPurgeAcl.test_purge_acl_ipv6_with_mask
http2_general.test_h2_headers.TestIPv6.test_request_with_some_data

EvgeniiMekhanik · 2024-10-17T10:07:51Z

Tempesta 81531bf8f41c29160c36e9dfb7acfe20252fd809
tempesta-test f78542b

test_stream_priority_stress (http2_general.test_h2_stream_priority.TestStreamPriorityStress)

RomanBelozerov added enhancement New feature or request CI stability labels Sep 28, 2023

krizhanovsky added this to the 1.0 - GA milestone Sep 28, 2023

EvgeniiMekhanik added a commit that referenced this issue Nov 10, 2023

Enabled some tests blocked by #525

b59ae2e

EvgeniiMekhanik added a commit that referenced this issue Nov 23, 2023

Enabled some tests blocked by #525

65b1660

EvgeniiMekhanik added a commit that referenced this issue Dec 12, 2023

Enabled some tests blocked by #525

9cd484d

EvgeniiMekhanik added a commit that referenced this issue Dec 14, 2023

Enabled some tests blocked by #525

f5961ed

EvgeniiMekhanik added a commit that referenced this issue Dec 25, 2023

Enabled some tests blocked by #525

31093db

EvgeniiMekhanik added a commit that referenced this issue Dec 25, 2023

Enabled some tests blocked by #525

243fb3b

EvgeniiMekhanik added a commit that referenced this issue Jan 29, 2024

Enabled some tests blocked by #525

637ae2e

EvgeniiMekhanik added a commit that referenced this issue Jan 29, 2024

Enabled some tests blocked by #525

f4571c2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unstable tests #525

unstable tests #525

RomanBelozerov commented Sep 28, 2023 •

edited

Loading

voodam commented Oct 4, 2023 •

edited by RomanBelozerov

Loading

voodam commented Oct 4, 2023

EvgeniiMekhanik commented Feb 20, 2024 •

edited by RomanBelozerov

Loading

EvgeniiMekhanik commented Mar 5, 2024 •

edited by RomanBelozerov

Loading

RomanBelozerov commented Aug 29, 2024 •

edited

Loading

EvgeniiMekhanik commented Sep 26, 2024

EvgeniiMekhanik commented Oct 3, 2024 •

edited by RomanBelozerov

Loading

RomanBelozerov commented Oct 14, 2024

EvgeniiMekhanik commented Oct 17, 2024 •

edited

Loading

unstable tests #525

unstable tests #525

Comments

RomanBelozerov commented Sep 28, 2023 • edited Loading

Any config:

Local config:

Local config with TCP seg 1:

Remote config:

voodam commented Oct 4, 2023 • edited by RomanBelozerov Loading

voodam commented Oct 4, 2023

EvgeniiMekhanik commented Feb 20, 2024 • edited by RomanBelozerov Loading

EvgeniiMekhanik commented Mar 5, 2024 • edited by RomanBelozerov Loading

RomanBelozerov commented Aug 29, 2024 • edited Loading

EvgeniiMekhanik commented Sep 26, 2024

EvgeniiMekhanik commented Oct 3, 2024 • edited by RomanBelozerov Loading

RomanBelozerov commented Oct 14, 2024

EvgeniiMekhanik commented Oct 17, 2024 • edited Loading

RomanBelozerov commented Sep 28, 2023 •

edited

Loading

voodam commented Oct 4, 2023 •

edited by RomanBelozerov

Loading

EvgeniiMekhanik commented Feb 20, 2024 •

edited by RomanBelozerov

Loading

EvgeniiMekhanik commented Mar 5, 2024 •

edited by RomanBelozerov

Loading

RomanBelozerov commented Aug 29, 2024 •

edited

Loading

EvgeniiMekhanik commented Oct 3, 2024 •

edited by RomanBelozerov

Loading

EvgeniiMekhanik commented Oct 17, 2024 •

edited

Loading