Gauge validate does not work together with gauge_data_dir feature introduced in 1.4.0

[BrudiBanani]

Describe the bug
In Gauge v1.4.0 #2047 and #2057 were introduced, which brings the ability to use gauge_data_dir in properties, so that one is able to define environment specific .csv's for data driven executions.
When using gauge run -e xxx the spec parser seems to respect the used environment. With gauge validate always default environment is used. And there is no possibility to define it.

To Reproduce
Steps (or project) to reproduce the behavior: (alternatively clone/fork/download this https://github.com/BrudiBanani/gauge_validate_behaviour_example)

1. Initialise a gauge java project
2. Add 2 extra environments (e.g. test1 & test2)
3. Create a default.properties file in both environments with gauge_data_dir=gauge_data/<env> (<env> resolves to test1 & test2)
4. On root level create a folder gauge_data with subfolders test1 & test2. Inside test1 folder create a test.csv file with dummy data (atleast one column with one entry). Leave test2 empty.
5. Duplicate example.spec and name it example_2.spec.
6. Add table:test.csv in example.spec before first scenario
7. Add tags: test1 to example.spec and tags: test2 to example_2.spec
8. Run gauge validate
9. Run gauge run -e test1
10. Run gauge run -e test2

Logs

For 8.

[ParseError] /gauge_test/specs/example.spec:12 Could not resolve table. File /gauge_test/test.csv doesn't exist. => 'table:test.csv'

For 9.

# Specification Heading
  ## Vowel counts in single word         ✔ ✔ ✔ ✔
  ## Vowel counts in multiple word       ✔ ✔

Successfully generated json-report to => /gauge_test/reports/json-report

Successfully generated html-report to => /gauge_test/reports/html-report/index.html

Specifications: 1 executed      1 passed        0 failed        0 skipped
Scenarios:      2 executed      2 passed        0 failed        0 skipped

Total time taken: 286ms

For 10.

[ParseError] /gauge_test/specs/example.spec:12 Could not resolve table. File /gauge_test/gauge_data/test2/test.csv doesn't exist. GAUGE_DATA_DIR property is set to 'gauge_data/test2', Gauge will look for data files in this location. => 'table:test.csv'
# Specification Heading
  ## Vowel counts in single word         ✔ ✔ ✔ ✔
  ## Vowel counts in multiple word       ✔ ✔

Successfully generated json-report to => /gauge_test/reports/json-report

Successfully generated html-report to => /gauge_test/reports/html-report/index.html

Specifications: 1 executed      1 passed        0 failed        0 skipped
Scenarios:      2 executed      2 passed        0 failed        0 skipped

Total time taken: 201ms

Workaround
For 8. I can just place a dummy test.csv in my root directory or define a gauge_data_dir in default environment and place a dummy test.csv there. Than gauge validate works again and i can built my project.

Expected behavior
I would expect that gauge validate somehow respects all environments, so that I do not need to create dummy files like described in workaround

Versions:

Gauge version: 1.4.3
Plugins
-------
html-report (4.1.4)
java (0.9.0)
json-report (0.3.6)
screenshot (0.1.0)
spectacle (0.1.4)
xml-report (0.2.3

• macOS Monterey 12.4
• java 11.0.13
• VS Code and IntelliJ (no difference in this case)

Additional context
Another issue I have with this behaviour is that gauge run respects environments but not tags.
For example:
gauge run -e test2 -t test2 also returns a parse error, because it seems like every .spec is parsed even I do not want to execute it. Please extend the specParser so that gauge run will just parse what gets executed.

One last issue:
Please document exit codes. (Only documentation found "If either one of the user-created environment is not found, Gauge ends with a non-zero exit code.")
Executing step 10 gauge run -e test2 throws an exit code 2, which is not documented.
From what I know gauge exits with 0 if all tests pass, with 1 if some test fails. And every other issue is exit code 2.

Thanks :)

[dnl-re]

+1
we have the same problem here and would love to use the table feature if gauge validate would work.

[stefan-spiess]

+1
The environment specific "gauge_data_dir" is a very cool feature and within our growing project, we'd like to use it more often. Would be very nice to handle the missing CSVs, if they are not relevant for the provided environment(s).

[zabil]

Right now gauge validate does not use environment flags. Will having the option of passing an environment to validate help? For example.

gauge validate -e test1

[BrudiBanani]

@zabil
In my opinion there are two problems to solve. I think I did not make this clear while raising this issue. At first we need to differentiate between gauge validate during compilation (1) of a project and gauge validate during runtime (2) (execution of .specs). Atleast for gauge java projects. Not sure how this works in e.g. python.

1. Problem: While compilation gauge validate does not know for which environments we want to use given test data (e.g. table:test.csv), because at this time it is simply unclear how the user (tester) is executing the gauge run.

• Possible solution (1.1): Enable gauge validate to use tags to recognize in which environments specs/scenarios will be executed (and ofc if no tags are given defaultenvironment is used). This would also require to analyze gauge.properties inside the env/<env> directory to obtain the correct gauge_data_dir path.
• Possible solution (1.2): Much simpler solution would be to exlude given testdata (e.g. table:test.csv) from validation.
• Possible solution (1.3): Something in between of 1.1 and 1.2. Just validate if there is any file with given name. That would at least ensure there exists an e.g. test.csv file inside the project.

2. Problem: During runtime gauge validation is performed for all tests inside the project, even if i just execute a small bit of it.

• Solution: Independent from this issue gauge validate should not be performed for all specs & steps if i use gauge run -e test1 or gauge run /path/to/spec or gauge run -t tag1 or anything else to execute just a part of all tests. It should always just validate those specs/scenarios/steps which will be executed. That would also safe quite a bit of time in big projects. Therefore gauge validate should be extended to respect same parameters which are invoked by using gauge run as you already have proposed (atleast the -e part of it). So to answer your question: yes passing an environment variable to gauge validate would help :)

If something is unclear please feel free to ask me. I can also extend my example project to a more real life project for better reference.

[zabil]

@BrudiBanani thanks for taking the time to explain this.

Problem: While compilation gauge validate does not know for which environments we want to use given test data (e.g. table:test.csv), because at this time it is simply unclear how the user (tester) is executing the gauge run.

Agree

Possible solution (1.2): Much simpler solution would be to exlude given testdata (e.g. table:test.csv) from validation.

This is tricky as validations include parsing the csv and checking if dynamic parameters are right.

To understand the use case better can you explain why you run validate outside run? Is it to quickly check if the specs don't have any parse errors?

[BrudiBanani]

This is tricky as validations include parsing the csv and checking if dynamic parameters are right.

Thanks, i did not think about it.

To understand the use case better can you explain why you run validate outside run? Is it to quickly check if the specs don't have any parse errors?

I am using gauge in a big java maven project. Working together with a lot of people on it. It is best practice to ensure that a new version does not have negative impact on other tests (or the whole project) and therefore a part of our building process is the gauge validation (triggered by the gauge maven plugin).

[coding-lama]

Hi guys, I am facing the same issue with validation when trying to use the environment specific input files. Do we have some ETA when this issue could be resolved? We also use validation as a separate step in our project to verify the specs. We need to enable this new environment specific input files functionality while keeping the validation functionality.

[zabil]

No one is actively working on adding features to Gauge at the moment. However any pull request to add running gauge validate with -e and -t flags to apply environment and tags filter is welcome and will be merged.