Python errors in benchmark toolset

[bdw429s]

Hi all, I'm running the benchmarks for a single framework on my Windows machine. I have the latest master branch pulled and I'm running the tests via Docker Machine like so

$ docker run -it --rm --network tfb -v /var/run/docker.sock:/var/run/docker.sock -v /D_DRIVE/docs/GitHub/FrameworkBenchmarks:/FrameworkBenchmarks techempower/tfb --mode verify --test cfml

The test verifications themselves run fine, but this Python error halts execution at the end

...
No problems to report
wrk: Build time: 1s
techempower/postgres:latest: Build time: 1s
cfml: Build time: 1s
cfml: Time starting database: 1s
cfml: Time until accepting requests: 3s
cfml: Verify time: 57s
A fatal error has occurred
Traceback (most recent call last):
  File "/FrameworkBenchmarks/toolset/run-tests.py", line 244, in main
    any_failed = benchmarker.run()
  File "/FrameworkBenchmarks/toolset/benchmark/benchmarker.py", line 67, in run
    if not self.__run_test(test, benchmark_log):
  File "/FrameworkBenchmarks/toolset/benchmark/benchmarker.py", line 247, in __run_test
    self.docker_helper.stop()
  File "/FrameworkBenchmarks/toolset/utils/docker_helper.py", line 284, in stop
    self.database.containers.prune()
  File "/usr/local/lib/python2.7/dist-packages/docker/models/containers.py", line 951, in prune
    return self.client.api.prune_containers(filters=filters)
  File "/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py", line 34, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker/api/container.py", line 971, in prune_containers
    return self._result(self._post(url, params=params), True)
  File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 267, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
APIError: 409 Client Error: Conflict ("a prune operation is already running")
Shutting down (may take a moment)

Furthermore, if I run the tests themselves (without the --mode verify flags) like so...

$ docker run -it --rm --network tfb -v /var/run/docker.sock:/var/run/docker.sock -v /D_DRIVE/docs/GitHub/FrameworkBenchmarks:/FrameworkBenchmarks techempower/tfb --test cfml

The tests run ok but I get the same error as above plus this one below:

...
STARTTIME 1612477645
ENDTIME 1612477661
cfml: Traceback (most recent call last):
cfml:   File "/FrameworkBenchmarks/toolset/benchmark/benchmarker.py", line 209, in __run_test
cfml:     self.__benchmark(test, benchmark_log)
cfml:   File "/FrameworkBenchmarks/toolset/benchmark/benchmarker.py", line 295, in __benchmark
cfml:     benchmark_type(test_type)
cfml:   File "/FrameworkBenchmarks/toolset/benchmark/benchmarker.py", line 285, in benchmark_type
cfml:     results = self.results.parse_test(framework_test, test_type)
cfml:   File "/FrameworkBenchmarks/toolset/utils/results.py", line 149, in parse_test
cfml:     rawData["startTime"], rawData["endTime"], 1)
cfml:   File "/FrameworkBenchmarks/toolset/utils/results.py", line 471, in __parse_stats
cfml:     if len(main_header[item_num]) != 0:
cfml: IndexError: list index out of range
cfml: Error during test: cfml

In the examples above, I'm testing the new cfml framework, but I get the same Python errors even if I run the commands against an existing framework such as gemini so it doesn't appear to be related to the framework being tested.

[bdw429s]

Any ideas on this?

[msmith-techempower]

This is a strange one... it looks like the Docker daemon is already attempting to perform a prune operation while the toolset is requesting it.

Can you try restarting the Docker daemon and testing again?

[bdw429s]

Thanks for the reply @msmith-techempower Restarting my Docker machine does seem to have cleared up the "prune" error. However, the list index out of range error still persists.

[bdw429s]

The list index out of range Python error is still a blocker preventing me from effectively run the benchmarks locally.

[bdw429s]

@msmith-techempower Actually, I see something that may be the issue. The code appears to be looping over all the items in the rows and then accessing the corresponding header name from the headers. This assumes that the header list is as long (or longer) than the number of items in the row. You mentioned above that the headers were present, but even if they were present, if they contained fewer items than the row, that would also cause this error. I opened the stats file (minus the first 4 rows) in open office and look what I see:

The header row is two columns shorter than the data rows. I confirmed that zram0 is the last header. I think your issue is that there are fewer headers than items in the rows and the code assumes there will be a header for each row.

[bdw429s]

@msmith-techempower Does the findings I posted here last week regarding the CSV file having mismatched header and data rows help you understand the error?

[bdw429s]

@msmith-techempower Would you like me to create a ticket for my finding above where the column headers in the CSV don't match value rows?

[bdw429s]

@msmith-techempower Would you like a ticket for this finding?

[bdw429s]

It's been a month since I've gotten a reply here, so I created a ticket here

#6464