codeql does not use lgtm.yml file to create database

[singleghost]

I create a lgtm.yml file in my project like this.

extraction:
  javascript:
    index:
      include: ""
      exclude:
      - "benchmarks"
      - "test"
      - "testing"
      filters:
      - exclude: "**/*.js"
      - exclude: "**/*.html"
      - exclude: "**/*.ts"
  python:
    index:
      include: "aaaa"
      filters:
        exclude: "**/*.py"
  cpp:
    after_prepare: "mkdir -p ${LGTM_SRC}/build"
    configure:
      command: "(cd ${LGTM_SRC}/build; cmake .. -DCMAKE_C_FLAGS='-no-pie')"
    index:
      build_command: "(cd ${LGTM_SRC}/build; make)"

Then I use codeql to create a database, codeql database create -s ~/softwares/my-codeql-project --language=cpp codeql -j=20, but the output of the codeql shows that it does not use the lgtm.yml as configuration file and build failed. Part of the output:

[2019-11-22 13:22:53] [build] Makefile:132: recipe for target 'all' failed
[2019-11-22 13:22:53] [build-err] + '[' -f build.ninja ']'
[2019-11-22 13:22:53] [build-err] + '[' -d ../_lgtm_build_dir ']'
[2019-11-22 13:22:53] [build-err] + cd ..
[2019-11-22 13:22:53] [build] Semmle autobuild: no supported build system detected.
[2019-11-22 13:22:53] [build-err] + for f in build build.sh
[2019-11-22 13:22:53] [build-err] + '[' -x build ']'
[2019-11-22 13:22:53] [build-err] + '[' -f build ']'
[2019-11-22 13:22:53] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/home/dddong/softwares/codeql-home/codeql/tools/linux64/preload_tracer, /home/dddong/softwares/codeql-home/codeql/cpp/tools/autobuild.sh])
[2019-11-22 13:22:53] [build-err] + for f in build build.sh
[2019-11-22 13:22:53] [build-err] + '[' -x build.sh ']'
[2019-11-22 13:22:53] [build-err] + '[' -f setup.py ']'
[2019-11-22 13:22:53] [build-err] + echo 'Semmle autobuild: no supported build system detected.'
[2019-11-22 13:22:53] [build-err] + exit 1
A fatal error occurred: Exit status 1 from command [/home/dddong/softwares/codeql-home/codeql/cpp/tools/autobuild.sh]

[hmakholm]

It is true that the CodeQL CLI does not use lgtm.yml files when creating databases. There are good arguments that it should, even though some of the possible parts of an lgtm.yml (especially configuring how the LGTM builders get hold of dependencies) are firmly outside the scope of the CodeQL CLI.

We're having some internal discussions to figure out what the behavior should be.

For now, you will need to carry out the configuration manually and give the index command as a --command argument to codeql database create.

[singleghost]

So is it correct to pass multiple --command argument like this? codeql database create --language=cpp codeql -c="cmake . -DCMAKE_C_FLAGS='-no-pie'" -c="make" -j=20 It doesn't generate any errors but I'm not sure if the behaviour inside codeql is correct. When I query the codebase in vscode, the results seem to be incomplete.

[hmakholm]

Yes, you can pass multiple --command arguments.

However, as long as it's only the last of them that actually invokes the C compiler you can also just run the cmake command manually before you start codeql database create and then only run the actual make invocation as part of database create.

[singleghost]

Thank you very much!

[mostlycached]

Hey @hmakholm - Could you recommend an alternative for the xml_mode config available in lgtm.yml? I can't quite think of a way to use the --command option to work around it.

[p0]

Hey @hspan -- CodeQL 2.1.0 added XML indexing support, which hadn't been reimplemented in the CodeQL CLI before that. What value for xml_mode would you like to emulate? The Java and C# extractors for CodeQL should now behave in the same way as the default LGTM extractors.

[xsser]

Hey @hspan -- CodeQL 2.1.0 added XML indexing support, which hadn't been reimplemented in the CodeQL CLI before that. What value for xml_mode would you like to emulate? The Java and C# extractors for CodeQL should now behave in the same way as the default LGTM extractors.

When will it be released? I am looking forward the wonderful feature! lmao

[p0]

When will it be released? I am looking forward the wonderful feature! lmao

CodeQL 2.1.0 was released around 10 months ago.

[xsser]

When will it be released? I am looking forward the wonderful feature! lmao

CodeQL 2.1.0 was released around 10 months ago.

Thanks a lot for your heolp.
My codeql cli version is 2.3.3, and my codeql plugin for vscode version is 1.3.10.
I can not scan mybatis Mapper.xml in my .ql file(in my rules)

What i ran command :
codeql database create xxxx --language=java
According to what you said, mapper.xml documents should be included

[p0]

By default, only some commonly required XML files are included -- you have to explicitly ask for files that are not covered in the standard set. One way of doing that is shown here -- could you try this to see if it works?

We don't have out-of-the-box queries that look at MyBatis, and so mapper.xml files are not picked up by default. Is that file name fixed, and the only one that's required? We could easily add it to the default behaviour if it would be generally useful.

[xsser]

By default, only some commonly required XML files are included -- you have to explicitly ask for files that are not covered in the standard set. One way of doing that is shown here -- could you try this to see if it works?

We don't have out-of-the-box queries that look at MyBatis, and so mapper.xml files are not picked up by default. Is that file name fixed, and the only one that's required? We could easily add it to the default behaviour if it would be generally useful.

Yes, if an application uses the mybatis framework to query SQL. Its sink is the mapped statement in the XML file.

[xsser]

By default, only some commonly required XML files are included -- you have to explicitly ask for files that are not covered in the standard set. One way of doing that is shown here -- could you try this to see if it works?

We don't have out-of-the-box queries that look at MyBatis, and so mapper.xml files are not picked up by default. Is that file name fixed, and the only one that's required? We could easily add it to the default behaviour if it would be generally useful.

github/codeql#3887 (comment)
This method you provided is very useful! It completely solved my problem