Import document with $ref not able to resolve

[michaelpro1]

Describe the bug

When trying to import a document with references, those references are not resolved against the original document. For example using the example:
https://raw.githubusercontent.com/asyncapi/avro-schema-parser/master/tests/asyncapi-avro-1.9.0.yaml
It is unable to resolve the schemas/Person-1.9.0.avsc and looks locally on the system.

I have also tried putting all the async.yaml and related artifacts onto file system of the app however the app is not able to resolve them. Similarly placing them into public/ folder resolves references at the wrong spot instead looking at /usr/src/app instead of /usr/src/app/public

How to Reproduce

Steps to reproduce the issue. Attach all resources that can help us understand the issue:

Scenario 1 HTTP

1. Go to https://playground.asyncapi.io/?embed=1
2. Put https://raw.githubusercontent.com/asyncapi/avro-schema-parser/master/tests/asyncapi-avro-1.9.0.yaml click import - error appears

Scenario 2 local file

1. Clone playground repo
2. Clone https://github.com/asyncapi/avro-schema-parser/tree/master/tests into public folder
3. Perform docker build and run
4. Import http://$DOCKERHOST:83/asyncapi-avro-1.9.0.yaml - error appears

Expected behavior

1. $ref should be resolved relative to original resource if possible - by client if required
2. file:/// handling should be clarified so that something like a volume mount can work, any chance of using local files?

[github-actions]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue.

Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[derberg]

Hi @michaelpro1

It is currently not possible to resolve references relative to the URL of the spec location. What I mean is that if you want to reference schema from the remote, you need to do this:

asyncapi: 2.0.0
info:
  title: My API
  version: '1.0.0'
channels:
  mychannel:
    publish:
      message:
        schemaFormat: application/vnd.apache.avro;version=1.9.0
        payload:
          $ref: 'https://raw.githubusercontent.com/asyncapi/avro-schema-parser/master/tests/schemas/Person-1.9.0.avsc'

[michaelpro1]

@derberg Would you consider taking a PR for this feature? I was thinking to simply do it on the client and qualify all relative links

[derberg]

@michaelpro1 I'm not 100% sure what you mean by qualify all relative links? What would you do exactly? Playground uses https://github.com/asyncapi/parser-js/ to parse AsyncAPI file, and among other things, it is responsible also for dereferencing.

[michaelpro1]

For example interface https://github.com/asyncapi/avro-schema-parser/blob/9180ea497b11bd52d4027060251edca3b82fbd1b/tests/asyncapi-avro-1.9.0.yaml#L11 has a reference to a separate schema

I import this document via https://github.com/asyncapi/avro-schema-parser/blob/master/tests/asyncapi-avro-1.9.0.yaml URL

During the /generate call documents get modified
$ref: schemas/Person-1.9.0.avsc => $ref: https://github.com/asyncapi/avro-schema-parser/blob/master/tests/schemas/Person-1.9.0.avsc

This allows the backend service to resolve all relative references to the imported document seamless from client side. It allows playground editor to work with imported documents without modification such as when they are loaded from source control, or wiki etc which is how typically interface docs are stored including Swagger, OpenAPI etc

[derberg]

The problem with such an approach is that this would be supported in the playground only and would confuse users. The user would then take such a schema that works fine in the playground and would get errors in the generator for example. Best would be to have something like this in the parser. same as we have with the path option that you need to pass to make it easy to resolve references to external files, we could have the same for the base url. The best would be if this would be supported natively by https://github.com/APIDevTools/json-schema-ref-parser

I dug a bit and looks like it is supported

• How to specify $ref base url? APIDevTools/json-schema-ref-parser#20
• Add possibility to specify location when you pass object APIDevTools/json-schema-ref-parser#13

But using it in the https://github.com/asyncapi/parser-js/ is not so easy

🤔

[michaelpro1]

That was the first thing I tried actually, in version.js to supply parserOptions.path with the value of req.header('x-asyncapi-base-url') however it would always error out with an error where it was still looking on file system.

I tried passing the sourceFileUrl and just sourceFolder without any luck, but couldn't figure out why it was failing. Tried it with latest libs without any luck.

This was the only way to get it working, however it will not work with nested $refs which would still fail resolution.

[michaelpro1]

@derberg The #31 seems to resolve it by simply passing the path, I think I was looking at the wrong spot - should have updated html.js/markdown.js. Since file resolution is currently turned off I didn't test it.

[derberg]

awesome that it is possible without any magic

[tanujazz3]

Can anyone please help why I am not able to run and execute the generator for the remote using
ag ./asyncapi.yaml @asyncapi/java-spring-template -o output -p user="username"-p password="password"

I have seen some examples where they are providing the schema registry url under Url-->Binding

Here is how my asyncapi.yaml looks like

asyncapi: 2.4.0
info:
title: Sample Service
version: 1.0.0
description: This service reads the schema from remote confluent schema registry
servers:
development:
url: "url:port"
description: Development server
protocol: kafka
protocolVersion: 1.0.0
bindings:
kafka:
schemaRegistryUrl: "schemaRegistryUrl"
schemaRegistryVendor: "confluent"
schemaRegistryAvailable: true
channels:
Sample-output-topic:
publish:
bindings:
kafka:
message:
name: TempratureReading
schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
contentType: 'application/octet-stream'
payload:
$ref: 'Schema url'
title: ConsumerRecord
Sample-output-topic::
subscribe:
bindings:
kafka: {}
message:
name: TempratureReading
schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
contentType: 'application/octet-stream'
payload:
$ref: 'Schema url'
title: Customer