Remove Authorization Header in Integration Tests

[jpjpjp]

Hi.

A few weeks back we had a discussion that resulted in a PR to allow this type of configuration in a variation test:

        - name: noAuthHeader
          overwrites:
            - overwriteRequestHeaders:
                - key: Authorization
                  remove: true

to generate a request which is has no authorization header.

Can Portman have the same behavior for a configuration in an Integration test as well?

In my project, I have been writing a bunch of variation tests for each endpoint to validate that the APIs handle various invalid input scenarios. As I began work on my 3rd endpoint, I realized that the configuration file was getting a bit monolithic, so I had hoped to use the $ref directive to create separate variation tests for each endpoint.

My original idea was to create a unique configuration file for each endpoint, including the array of operationIds to perform various tests on. Basically I'd have one configuration file for each of the folders that Portman generates under the "Variation Tests" folder.
Unfortunately, the variationTests directive doesn't have an attribute that maps to these folder names. Instead it wants an array of operationIds. For example, when I tried this configuration:

tests:
  contractTests:
    $ref: portman/tests/contracts/generic-contract-tests.yaml
  variationTests:
      $ref: portman/tests/variations/me-variations.yaml
      $ref: portman/tests/variations/categories-variations.yaml

Only the tests in the final referenced configuration file are generated.

I discovered that the integrationTests directive does have a name attribute which does align to the folder, so this configuration works:

tests:
  contractTests:
    $ref: portman/tests/contracts/generic-contract-tests.yaml
  # variationTests:
  #     $ref: portman/tests/variations/me-variations.yaml
  #     $ref: portman/tests/variations/categories-variations.yaml
  integrationTests:
    - name: /me tests
      $ref: portman/tests/variations/me-variations.yaml
    - name: /categories tests
      $ref: portman/tests/variations/categories-variations.yaml
    - name: /transactions tests
      $ref: portman/tests/variations/transactions-variations.yaml

The only thing that doesn't work in this case is the test that removes the "authorization" header.

Is this an appropriate use of integrationTests? They are discussed only lightly in the documentation.

Thanks!
--jp

[thim81]

hi @jpjpjp

The integration tests re-use the regular directives, so normally it should work.
Another case that I'll investigate more in detail.

[thim81]

hi @jpjpjp

Any chance you can zip your files and config, so I can reproduce it locally?

[jpjpjp]

Hi @thim81

Thanks for asking for this. I started with the same sample that I provided for this discussion: #580

And I was able to validate that this test:

  variationTests:
    - openApiOperationId: getMe
      openApiResponse: '401'
      variations:
        - name: noAuthHeaderAsVariationTest
          overwrites:
            - overwriteRequestHeaders:
                - key: Authorization
                  remove: true
          tests:
            contentTests:
              - responseBodyTests:
                  - key: message
                    value: Unauthorized
                  - key: errors
                    minLength: 1
                  - key: errors[0].errMsg
                    value: Missing authorization header

which is designed to fail due to the lack of an Authorization header still works (i.e., it fails because the generated request correctly has no Authorization header). You fixed this issue as an outcome of that discussion.

I thought the issue in this discussion was that when I try to run the same test under the integrationTests directive, the generated test still includes the Authorization header. However, I'm finding that simply having the integrationTests directive in the configuration also causes the variationTest, shown above to fail.

In the attached zip, I've included a simple API spec and two portman configs:

• portman-config.yaml has the test above and the integrationTests section commented out. It generates the tests as expected
• portman-config-fails has the integrationTests section uncommented. It generates a bad variationTest.

I hope this makes sense. If it would be helpful to include screenshots of the generated Postman, or the generated JSON itself, just let me know.

yaml-configs.zip

[thim81]

hi @jpjpjp,

I used your config with indeed the unexpected result, that the "Authorisation" is kept in both the "Integration" AND "Variation" requests.

Keep you posted off any findings.

[thim81]

hi @jpjpjp

I found the reason and created a PR that solves the issue, which contains the BEFORE/AFTER results.

[thim81]

FYI Portman v1.28.0 is just released, which fixes the unexpected authorisation header behaviour with integration tests.

[jpjpjp]

Just confirmed that 1.28.0 resolves this issue for me.

Thank you!