Test Cases Created From OpenAPI spec not adhering to specifications

[AnirudhChaudhary]

Hello Kerry,

I am trying to create test cases from my OpenAPI spec but am running into troubles because I am getting test cases that do not adhere to the openapi specifications.

This is happening particularly for a variable that is coming as null when I have set nullable to false.

This is happening in a test called getAPI_XisDefined_Is_Yes(). I have clearly stated that X should not be nulled out in the openapi spec under the "schema" section. However, in the test I stated above, X is nulled out. I am curious about what is happening in this test and if its a mistake in the specification document or something that TCases is not recognizing.

Thanks.

[kerrykimbrough]

Can you attach the OpenAPI file so I can reproduce this?

Also, have you looked at messages in the log file? This is usually a file named tcases-api-test.log.

[kerrykimbrough]

Please open a new issue. Be sure to attach a copy of the OpenAPI file and an explanation of why the results you are seeing are wrong.

[kerrykimbrough]

Also attach a copy of the tcases-api-test.log file and include the name of the OpenAPI file.

[AnirudhChaudhary]

Hi Kerry, I am currently running this project with the maven plugin. Usually when I ran TCases, I would see the cases-api-test.log file but while running it through the maven plugin, I do not see that file. Does it have a different name?

Also I apologize, but I do not currently have the ability to send the entire OpenAPI file, simply only a screenshot. Will this significantly inhibit your ability to diagnose and solve the problem?

[kerrykimbrough]

Let me guess: you are seeing a test case for GET /api/StudentPerf?StudentIDS= that expects an isSuccess() response. But by default nullable is false for this parameter. So there is another test case that sends the same request but expects an isBadRequest() response.

Is that the problem you are seeing?

[AnirudhChaudhary]

The StudentID variable in this test case is nulled out but it should not be because it is a required parameter for the request. My problem: Why is this test getting created when the parameter should not be nulled out from the spec? And if it is getting created, why is it expecting a Success?

[kerrykimbrough]

This problem happens when TFOA generates inconsistent test cases for null vs. empty values -- see issue #201.

In the case shown above, you can work around the problem by adding "minItems": 1 to the schema for StudentIDs. This tells TFOA that an empty array (which in this case looks that same as a null array) is not a valid value. This is not a perfect solution, because it will also result in redundant failure test cases -- one for an invalid null value and another for an invalid empty value.

[kerrykimbrough]

Now fixed in release 3.7.1 -- see the release notes for details.

[kerrykimbrough]

BTW, it turns out that the correct encoding for an empty StudentIDs array actually is StudentIDs=, i.e. an empty value. For a null value, the encoding would be different: just StudentIDs (no =). So even though issue #201 was a real problem that is now fixed, you will see the same result in 3.7.1.