analyze --capture-analysis-output writes corrupt zipfiles

[jimis]

Describe the bug
At the end of analyze, I see hundreds of the following warnings:

/usr/lib/python3.8/zipfile.py:1517: UserWarning: Duplicate name: 'sources-root/home/cc-runs/src/qt/qt5/qtbase/src/printsupport/kernel/qprint_p.h'
  return self._open_to_write(zinfo, force_zip64=force_zip64)

The zipfiles stored under results_dir/run_name/failed are troublesome to unzip. On linux with the unzip command I am asked countless of times whether I want to overwrite certain files.

CodeChecker version is today's latest HEAD at d78f1d9.

[jimis]

This still happens. Here is an example when I try to unzip one of the files:

$ unzip ../qdatetime.cpp_clangsa_fbaaa3c8986c74ae8f6ae431391de560.plist_CTU_unknown.zip 
Archive:  ../qdatetime.cpp_clangsa_fbaaa3c8986c74ae8f6ae431391de560.plist_CTU_unknown.zip
  inflating: sources-root/usr/lib/llvm-11/include/c++/v1/cstring  
  inflating: sources-root/usr/lib/llvm-11/include/c++/v1/__threading_support  
  inflating: sources-root/home/cc-runs/src/qt/qt5/qtbase/build/include/QtCore/qglobalstatic.h  
[ ... ]
  inflating: sources-root/home/cc-runs/src/qt/qt5/qtbase/build/include/QtCore/qtmetamacros.h  
replace sources-root/home/cc-runs/src/qt/qt5/qtbase/src/corelib/text/qstringlist.h? [y]es, [n]o, [A]ll, [N]one, [r]ename: 

And this message appears many times. Answering A helps it move on, but I'm not sure if this corrupts any files.

[csordasmarton]

@jimis I need more information to debug this problem:

• Could you please list the zip files in your results_dir/run_name/failed directory and attach the list?
• Could you please attach your compilation database (compile_commands.json)?
• Could you please attach the compilation error for this file: qdatetime.cpp?
• How is your CodeChecker analyze command look like?

[jimis]

Could you please list the zip files in your results_dir/run_name/failed directory and attach the list?

Attached: unzil_-l.log (EDIT: wrong data provided, see next comment)

Could you please attach your compilation database (compile_commands.json)?

Attached: codechecker_log.json.gz

Could you please attach the compilation error for this file: qdatetime.cpp?

The error is because of CTU failing (see #3012) and shouldn't be relevant, the file is analyzed fine without CTU.

How is your CodeChecker analyze command look like?

CodeChecker.py analyze /home/cc-runs/codechecker_results/qtbase-dev-20201210-238f466d49/codechecker_log.json -j 18 -i SKIPFILE --capture-analysis-output --ctu --ctu-reanalyze-on-failure --ctu-ast-mode load-from-pch -d bugprone-sizeof-expression -o /home/cc-runs/codechecker_results/qtbase-dev-20201210-238f466d49

[jimis]

Could you please list the zip files in your results_dir/run_name/failed directory and attach the list?

Sorry, I attached the contents of the zip (you see the multiple same files). Regarding the "failed" contents, it's too long, it's all the files that failed with CTU analysis. Is that relevant?

[csordasmarton]

Do you have the standard output and error logs related to the analysis of qdatetime.cpp file?

Because in your zip file name I can see the unknown keyword which comes from this code:

codechecker/analyzer/codechecker_analyzer/analysis_manager.py

Lines 536 to 540 in 792bd81

	failure_type = "_unknown"
	if rh.analyzer_returncode == 1:
	failure_type = "_compile_error"
	elif rh.analyzer_returncode == 254:
	failure_type = "_crash"

So the problem is not a compiler crash or a compiler error. I am curious what it can be.

[csordasmarton]

Maybe if you can run the analysis with the --verbose debug option and attach the logs would be useful. It is enough to attach the logs which is related only to qdatetime.cpp file.

[jimis]

@csordasmarton how can I re-run the codechecker command, but only for a file (i.e. skip everything else)?

[csordasmarton]

You can use the following option of the CodeChecker analyze command:

--file FILE [FILE ...]
                        Analyze only the given file(s) not the whole
                        compilation database. Absolute directory paths should
                        start with '/', relative directory paths should start
                        with '*' and it can contain path glob pattern.
                        Example: '/path/to/main.cpp', 'lib/*.cpp', */test*'.

[jimis]

Attaching the combined stdout+stderr output from codechecker. Almost same command as previous plus --file '*/qdatetime.cpp' --verbose debug, on a somewhat newer source tree. The bug is in the log, search for "Duplicate name" (it's printed in stderr, so I had to re-run to get the combined output).

cc-stdout_stderr.log.gz

[jimis]

Do you have the standard output and error logs related to the analysis of qdatetime.cpp file?

Because in your zip file name I can see the unknown keyword which comes from this code:

codechecker/analyzer/codechecker_analyzer/analysis_manager.py

Lines 536 to 540 in 792bd81

	failure_type = "_unknown"
	if rh.analyzer_returncode == 1:
	failure_type = "_compile_error"
	elif rh.analyzer_returncode == 254:
	failure_type = "_crash"

So the problem is not a compiler crash or a compiler error. I am curious what it can be.

So as you can see in the log, the error is:

/usr/include/c++/9/array:245:28: error: cannot import unsupported AST node CXXFoldExpr
      -> array<enable_if_t<(is_same_v<_Tp, _Up> && ...), _Tp>,

The return code cannot be seen in the log, but in the file return-code I see -11.

As mentioned earlier, this has been reported at #3012 (see 2nd point in description). It is blocking CTU analysis for C++17 projects.

But how is it related to the corrupted zipfiles issue?