using App.create(<url>) with HTTPS, auth, and custom headers?

[rbcollins123]

I see in #60 that there was an addition to allow a Client to pass additional parameters to requests, but at 1st glance, it looks like the App.create() method uses a urllib getter that doesn't allow any manipulation of the session used to get access the URL that contains the OpenAPI JSON definition. Is that correct, or did I miss an intended pattern to allow dealing with cert issues, auth methods, and potentially required headers when pulling the JSON definitions via remote URL?

Thanks!

[mission-liao]

In short:

• https is allowed when loading OpenAPI JSON/YAML definitions
• auth, custom headers is not customizable.

The issue described a way to make custom request against an API service, which is not for the case to load the OpenAPI definitions and discover what functionality that service provide. Usually speaking, the should not be a need to provide auth info when requesting the OpenAPI definitions.

It's not very hard to write one client to fulfill what you need by referencing pyswagger.contrib.client.requests.

By the way, can I know the use case to manipulate the session used to load OpenAPI JSON definition?

[rbcollins123]

I was hoping to leverage the Swagger 1.2 definitions within another vendor's product, and to access their definitions, they require you to 1st make a POST call to an API to get a token, then all subsequent calls must have the X-Auth-Token header set with that value. Their api-docs are protected and require the auth token to be set to access them. Also, in some cases customer installations of this product may use self-signed certificates which may fail SSL cert validation, so the ability to skip verification, or point to custom certs for validation is required also.

My 1st approach was to create a custom Getter, and then use App.load(getter=CustomGetter) and App.prepare() manually instead, but this failed during the App.prepare() step and the logging wasn't intuitive as to why.

Here is a quick example of the custom Getter I used (this test was done in Python 3.6.0):

# coding=utf-8
import json
import logging
import requests
from pyswagger import App
from pyswagger.getter import Getter

APIC_FQDN = 'sandboxapic.cisco.com'
APIC_USERNAME = '******'
APIC_PASSWORD = '*****'
API_VERSION = 'v1'
APIC_BASE_API_URL = 'https://' + APIC_FQDN + '/api/' + API_VERSION
APIC_BASE_APIDOCS_URL = 'https://' + APIC_FQDN + '/apic/api/' \
                        + API_VERSION + '/api-docs/'

logging.basicConfig()
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logging.getLogger('pyswagger').setLevel(logging.DEBUG)

class ApicEmGetter(Getter):
    def __init__(self, path):
        super().__init__(path)
        if self.base_path.endswith('/'):
            self.base_path = self.base_path[:-1]
        self.urls = [path]

    def load(self, url):
        # as a work around, fore-go caching auth token in the app and
        # issue an request to get a token before every GET,
        # then set the token in the headers, then GET the url.
        # In the future, the ability to use a requests.Session would be better
        logger.debug(f"Getter requests for {url}")
        data = json.dumps(
                {"username": APIC_USERNAME, "password": APIC_PASSWORD})
        apic_headers = { "Content-Type": "application/json"}
        response = requests.post(url=APIC_BASE_API_URL + "/ticket",
                                data=data, headers=apic_headers,verify=False)
        logger.debug(f"Response Code: {response.status_code}")
        logger.debug(f"Received from authentication request: {response.text}")
        result_json = json.loads(response.text)
        auth_ticket = "" # default to empty token if not in the response
        if 'serviceTicket' in result_json['response'].keys():
            auth_ticket = result_json["response"]["serviceTicket"]
        logger.debug(f"Received auth ticket {auth_ticket}")
        apic_headers["X-Auth-Token"] = auth_ticket
        apic_headers["X-CSRF-Token"] = "soon-enabled"
        logger.debug(f"Request headers: {apic_headers}")
        response = requests.get(url, headers=apic_headers)
        logger.debug(f"Result from GET of {url}: {response.text}")
        return response.json()

app = App.load(APIC_BASE_APIDOCS_URL+'file-service', getter=ApicEmGetter)
app.prepare()

And here is the result from that approach so far:

INFO:pyswagger.core:load with [https://sandboxapic.cisco.com/apic/api/v1/api-docs/file-service]
INFO:pyswagger.core:init with url: https://sandboxapic.cisco.com/apic/api/v1/api-docs/file-service
INFO:pyswagger.resolve:https://sandboxapic.cisco.com/apic/api/v1/api-docs/file-service patch to https://sandboxapic.cisco.com/apic/api/v1/api-docs/file-service
DEBUG:__main__:Getter requests for https://sandboxapic.cisco.com/apic/api/v1/api-docs/file-service
/home/netmagus/.pyenv/versions/apicem-testing/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
DEBUG:__main__:Response Code: 200
DEBUG:__main__:Received from authentication request: {"response":{"serviceTicket":"ST-3045-oQQE6fQZQeSeModag6nB-cas","idleTimeout":1800,"sessionTimeout":21600},"version":"1.0"}
DEBUG:__main__:Received auth ticket ST-3045-oQQE6fQZQeSeModag6nB-cas
DEBUG:__main__:Request headers: {'Content-Type': 'application/json', 'X-Auth-Token': 'ST-3045-oQQE6fQZQeSeModag6nB-cas', 'X-CSRF-Token': 'soon-enabled'}
DEBUG:__main__:Result from GET of https://sandboxapic.cisco.com/apic/api/v1/api-docs/file-service: {"apiVersion":"1.0","swaggerVersion":"1.2","apis":[{"path":"/default/fileservice","description":"File Service API"}],"info":{"title":"File","description":"APIC-EM Service API based on the Swagger™ 1.2 specification","termsOfServiceUrl":"http://www.cisco.com/web/siteassets/legal/terms_condition.html","license":"Cisco DevNet","licenseUrl":"https://developer.cisco.com"}}
INFO:pyswagger.resolve:https://sandboxapic.cisco.com/apic/api/v1/api-docs/file-service/default/fileservice patch to https://sandboxapic.cisco.com/apic/api/v1/api-docs/file-service/default/fileservice
DEBUG:__main__:Getter requests for https://sandboxapic.cisco.com/apic/api/v1/api-docs/file-service/default/fileservice
/home/netmagus/.pyenv/versions/apicem-testing/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
DEBUG:__main__:Response Code: 200
DEBUG:__main__:Received from authentication request: {"response":{"serviceTicket":"ST-3046-cfhYTj5G1lu7av7bLJVn-cas","idleTimeout":1800,"sessionTimeout":21600},"version":"1.0"}
DEBUG:__main__:Received auth ticket ST-3046-cfhYTj5G1lu7av7bLJVn-cas
DEBUG:__main__:Request headers: {'Content-Type': 'application/json', 'X-Auth-Token': 'ST-3046-cfhYTj5G1lu7av7bLJVn-cas', 'X-CSRF-Token': 'soon-enabled'}
DEBUG:__main__:Result from GET of https://sandboxapic.cisco.com/apic/api/v1/api-docs/file-service/default/fileservice: {"apiVersion":"1.0","swaggerVersion":"1.2","basePath":"/","resourcePath":"/file","produces":["application/json"],"consumes":["multipart/form-data"],"apis":[{"path":"/file/namespace","description":"getNameSpaceList","operations":[{"method":"GET","summary":"Returns list of available namespaces","notes":"This method is used to obtain a list of available namespaces","type":"NameSpaceListResult","nickname":"getNameSpaceList","produces":["application/json"],"parameters":[{"name":"scope","description":"Authorization Scope for RBAC","defaultValue":"ALL","required":true,"type":"List","paramType":"header","allowMultiple":false}],"responseMessages":[{"code":200,"message":"This Request is OK","responseModel":"NameSpaceListResult"},{"code":403,"message":"This user is Forbidden Access to this Resource"},{"code":401,"message":"Not Authorized Yet, Credentials to be supplied"},{"code":404,"message":"No Resource Found"}],"deprecated":"false"}]},{"path":"/file/namespace/{nameSpace}","description":"getFilesByNamespace","operations":[{"method":"GET","summary":"Returns list of files under a specific namespace","notes":"This method is used to obtain a list of files under a specific namespace","type":"FileObjectListResult","nickname":"getFilesByNamespace","parameters":[{"name":"nameSpace","description":"A listing of fileId's","defaultValue":"","required":true,"type":"string","paramType":"path","allowMultiple":false},{"name":"scope","description":"Authorization Scope for RBAC","defaultValue":"ALL","required":true,"type":"List","paramType":"header","allowMultiple":false}],"responseMessages":[{"code":200,"message":"This Request is OK","responseModel":"FileObjectListResult"},{"code":403,"message":"This user is Forbidden Access to this Resource"},{"code":401,"message":"Not Authorized Yet, Credentials to be supplied"},{"code":404,"message":"No Resource Found"}],"deprecated":"false"}]},{"path":"/file/{fileId}","description":"downLoadFile","operations":[{"method":"GET","summary":"Downloads a file referred by the fileId","notes":"This method is used to download a file referred by the fileId","type":"void","nickname":"downLoadFile","parameters":[{"name":"fileId","description":"File Identification number","defaultValue":"","required":true,"type":"string","paramType":"path","allowMultiple":false},{"name":"scope","description":"Authorization Scope for RBAC","defaultValue":"ALL","required":true,"type":"List","paramType":"header","allowMultiple":false}],"responseMessages":[{"code":200,"message":"This Request is OK"},{"code":403,"message":"This user is Forbidden Access to this Resource"},{"code":401,"message":"Not Authorized Yet, Credentials to be supplied"},{"code":404,"message":"No Resource Found"}],"deprecated":"false"}]},{"path":"/file/{fileId}","description":"deleteFile","operations":[{"method":"DELETE","summary":"Deletes a file with the specified fileId","notes":"This method is used to delete a file associated with the specified fileId","type":"SuccessResult","nickname":"deleteFile","produces":["application/json"],"parameters":[{"name":"fileId","description":"File Identification number","defaultValue":"","required":true,"type":"string","paramType":"path","allowMultiple":false},{"name":"scope","description":"Authorization Scope for RBAC","defaultValue":"ALL","required":true,"type":"List","paramType":"header","allowMultiple":false}],"responseMessages":[{"code":200,"message":"This Request is OK","responseModel":"SuccessResult"},{"code":403,"message":"This user is Forbidden Access to this Resource"},{"code":401,"message":"Not Authorized Yet, Credentials to be supplied"},{"code":404,"message":"No Resource Found"}],"deprecated":"false"}]},{"path":"/file/{fileId}/checksum","description":"getChecksumOfFile","operations":[{"method":"GET","summary":"Retrieves checksum for the file referred to by the fileId","notes":"This method is used to obtain checksum for the file referred to by the fileId","type":"SuccessResult","nickname":"getChecksumOfFile","produces":["application/json"],"parameters":[{"name":"fileId","description":"File Identification number","defaultValue":"","required":true,"type":"string","paramType":"path","allowMultiple":false},{"name":"scope","description":"Authorization Scope for RBAC","defaultValue":"ALL","required":true,"type":"List","paramType":"header","allowMultiple":false}],"responseMessages":[{"code":200,"message":"This Request is OK","responseModel":"SuccessResult"},{"code":403,"message":"This user is Forbidden Access to this Resource"},{"code":401,"message":"Not Authorized Yet, Credentials to be supplied"},{"code":404,"message":"No Resource Found"}],"deprecated":"false"}]},{"path":"/file/{nameSpace}","description":"uploadFile","operations":[{"method":"POST","summary":"Uploads a new file within a specific nameSpace","notes":"This method is used to upload a new file within a specific nameSpace","type":"FileObjectResult","nickname":"uploadFile","produces":["application/json"],"consumes":["multipart/form-data"],"parameters":[{"name":"nameSpace","description":"Specify File's namespace,namespace is a grouping of multiple files","defaultValue":"","required":true,"type":"string","paramType":"path","allowMultiple":false},{"name":"toEncrypt","description":"toEncrypt","defaultValue":"","required":false,"type":"boolean","paramType":"query","allowMultiple":false},{"name":"scope","description":"Authorization Scope for RBAC","defaultValue":"ALL","required":true,"type":"List","paramType":"header","allowMultiple":false},{"name":"fileUpload","description":"","defaultValue":"","required":false,"type":"file","paramType":"body","allowMultiple":false,"paramAccess":""}],"responseMessages":[{"code":200,"message":"This Request is OK","responseModel":"FileObjectResult"},{"code":202,"message":"This Request is Accepted"},{"code":403,"message":"This user is Forbidden Access to this Resource"},{"code":401,"message":"Not Authorized Yet, Credentials to be supplied"},{"code":404,"message":"No Resource Found"}],"deprecated":"false"}]},{"path":"/file/{nameSpace}/{fileId}","description":"updateFile","operations":[{"method":"PUT","summary":"Updates an existing file within a specific nameSpace","notes":"This method is used to update an existing file within a specific nameSpace","type":"FileObjectResult","nickname":"updateFile","produces":["application/json"],"parameters":[{"name":"nameSpace","description":"Specify File's namespace,namespace is a grouping of multiple files","defaultValue":"","required":true,"type":"string","paramType":"path","allowMultiple":false},{"name":"fileId","description":"Specify File Identification number","defaultValue":"","required":true,"type":"string","paramType":"path","allowMultiple":false},{"name":"scope","description":"Authorization Scope for RBAC","defaultValue":"ALL","required":true,"type":"List","paramType":"header","allowMultiple":false},{"name":"fileUpload","description":"","defaultValue":"","required":false,"type":"file","paramType":"body","allowMultiple":false,"paramAccess":""}],"responseMessages":[{"code":200,"message":"This Request is OK","responseModel":"FileObjectResult"},{"code":403,"message":"This user is Forbidden Access to this Resource"},{"code":401,"message":"Not Authorized Yet, Credentials to be supplied"},{"code":404,"message":"No Resource Found"}],"deprecated":"false"}]}],"models":{"NameSpaceListResult":{"id":"NameSpaceListResult","description":"","extends":"","properties":{"version":{"type":"string"},"response":{"type":"array","items":{"type":"string"}}}},"FileObject":{"id":"FileObject","description":"","required":["name","id","downloadPath","nameSpace","fileFormat","fileSize","md5Checksum","sha1Checksum"],"extends":"","properties":{"name":{"type":"string","description":"Name of the file"},"id":{"type":"string","description":"file indentification number"},"downloadPath":{"type":"string","description":"Absolute path of the file"},"nameSpace":{"type":"string","description":"A group of file IDs contained in a common nameSpace"},"encrypted":{"type":"boolean","description":"isEncrypted of the file"},"fileFormat":{"type":"string","description":"MIME Type of the File. e.g. text/plain, application/xml, audio/mpeg"},"fileSize":{"type":"string","description":"Size of the file in bytes"},"md5Checksum":{"type":"string","description":"md5Checksum of the file"},"sha1Checksum":{"type":"string","description":"sha1Checksum of the file"},"attributeInfo":{"$ref":"object"}}},"FileObjectListResult":{"id":"FileObjectListResult","description":"","extends":"","properties":{"version":{"type":"string"},"response":{"type":"array","items":{"$ref":"FileObject"}}}},"FileObjectResult":{"id":"FileObjectResult","description":"","extends":"","properties":{"version":{"type":"string"},"response":{"$ref":"FileObject"}}},"SuccessResult":{"id":"SuccessResult","description":"","extends":"","properties":{"version":{"type":"string"},"response":{"type":"string"}}},"Void":{"id":"Void","description":"","extends":"","properties":{}}}}
INFO:pyswagger.core:version: 1.2
Traceback (most recent call last):
  File "/home/netmagus/apic_em_testing/swaggerpy_test.py", line 100, in <module>
    app.prepare()
  File "/home/netmagus/.pyenv/versions/apicem-testing/lib/python3.6/site-packages/pyswagger/core.py", line 317, in prepare
    self.validate(strict=strict)
  File "/home/netmagus/.pyenv/versions/apicem-testing/lib/python3.6/site-packages/pyswagger/core.py", line 307, in validate
    raise errs.ValidationError('this Swagger App contains error: {0}.'.format(len(result)))
pyswagger.errs.ValidationError: this Swagger App contains error: 2.

The App appears to follow and pull the 2 corresponding JSON definitions as you can see from the two Result from GET log lines. But without digging further into the codebase I was not sure yet how the validate/resolve process worked to troubleshoot further, or if simply altering the Getter would be enough to suffice for the downstream logic to function properly.

[rbcollins123]

So I'll leave one more comment before you respond 😉

I believe there may be errors in the vendor's Swagger 1.2 definitions after evaluating it's content in another tool. I see now where they have the "type" of several parameters and refs set to non-existent models ☹️

I would still be curious to know if the above approach with the custom getter should work though, if I fix their models...

[mission-liao]

Things you did so far is just correct, I've loaded the json content in your comment and logging the errors:

2017-02-01 07:54:05,787 - pyswagger.core - ERROR - ((u'#/apis/default/apis/updateFile/parameters/3', 'Parameter'), 'body parameter with invalid name: fileUpload')
2017-02-01 07:54:05,787 - pyswagger.core - ERROR - ((u'#/apis/default/apis/uploadFile/parameters/3', 'Parameter'), 'body parameter with invalid name: fileUpload')

In short, the provide name "fileUpload" for body parameters, which is meaningless and prohibited by OpenAPI 1.2 spec (ref). But it should be just fine to use this spec by skipping the validation step of pyswagger, by this:

# instead of calling App.prepare
App.prepare(strict=False)

Once you skip the validation, you'll fail with this error, which means pyswagger failed to resolve some '$ref'

...
KeyError: u'default!##!object'

This $ref is from definitions.FileObject.properties.attributeInfo, things you can try:

• simply remove it and it should work.
• ask for the definition
• It's just my guess that "object" is a generic map, however, you have no way to declare a generic map in OpenAPI 1.2 (as fas as I know), one way you can do it is
  • remove "attributeInfo", you can still access that property by loading from pyswagger.Response.raw.
  • upgrade this document to OpenAPI 2.0, then use "additionProperties" to declare a generic map object for "attributeInfo"

---

My action items:

• show the validation errors (I have no idea why I didn't log them, it's not reasonable)
• provide another getter to accept a callback to use your prepared session.

[mission-liao]

those actions items (and test cases) are included in the latest build: v0.8.27, please feel free to reopen it if it's not what you expected.

[wryfi]

Usually speaking, the should not be a need to provide auth info when requesting the OpenAPI definitions.

For what it's worth, I disagree with this statement. I can think of many swagger APIs I use that are part of private systems, that absolutely require authentication to access the API schema. It's kind of ridiculous that I have to write a custom getter for this. Why not just use requests everywhere?

[mission-liao]

@wryfi Indeed, I agree with you now, it's the usual case that auth is required when making requests.

But I still didn't see a way to embed this part (especially OAuth2) in this library that could provide cleaner code than using oauthlib(or any oauth-x library directly in python) directly. If you can have some proposal / pseudo code / thoughts on this, maybe I can help to implement it.

[mission-liao]

@wryfi I just go through the document of requests to handle OAuth, the point that confuse me on providing similar functionality in pyswagger is step#1 of Web Application Flow: how do I provide a way for caller to handle the redirection to authorization_url ?