Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDK: Improve error message if an input JSON cannot be deserialized #18946

Closed
sherifnada opened this issue Nov 4, 2022 · 3 comments
Closed

CDK: Improve error message if an input JSON cannot be deserialized #18946

sherifnada opened this issue Nov 4, 2022 · 3 comments
Assignees
Labels
accepting-contributions Feel free to contribute to them team/extensibility type/enhancement New feature or request

Comments

@sherifnada
Copy link
Contributor

sherifnada commented Nov 4, 2022

Tell us about the problem you're trying to solve

If I provide an invalid JSON to the low-code CDK (e.g python --config secrets/config.json --catalog integration_tests/configured_catalog.json --state integration_tests/state.json I get a very unhelpful error message e.g: cannot decode bla bla bla.

As a user I have no idea what the problem is in this case

Describe the solution you’d like

I would prefer to get a clear error message e.g: (Cannot read the input JSON file xyz.json . Please ensure it is valid JSON). We should be able to catch these errors within a try... except block and re-throw a more user friendly error.

There are four inputs that we should clean up the error messages for. For scoping we'll limit this to inputs in entrypoint.py:

  • The json specified by --config
  • The configured catalog specified by --catalog
  • The stream state specified by --state
  • spec.json / spec.yaml (Not technically an command input, but worth validating)

Acceptance Criteria:

For an existing source connector (greenhouse, freshdesk, etc), after pulling the latest version of airbyte-cdk, run a read against the connector using the following command in the connector's top directory:
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json

For each of the following cases, modify the file to have invalid json and we should expect to see the improved error message:

  • secrets/config.json
  • integration_tests/configured_catalog.json
  • source_<name>/spec.<json | yaml>
  • integration_tests/state.json (also append --state integration_tests/state.json to the command)
@ryjabe
Copy link

ryjabe commented Nov 4, 2022

Here is an example of this (extra comma in line 11)

{
  "streams": [
    {
      "stream": {
        "name": "recall",
        "json_schema": {},
        "supported_sync_modes": ["full_refresh"]
      },
      "sync_mode": "full_refresh",
      "destination_sync_mode": "overwrite"
    },
  ]
}

This is the error it throws:

rbernstein@Ryans-MacBook-Pro source-recall % python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug | jq
{
  "type": "DEBUG",
  "message": "Debug logs enabled",
  "data": {}
}
{
  "type": "LOG",
  "log": {
    "level": "FATAL",
    "message": "Expecting value: line 12 column 3 (char 239)\nTraceback (most recent call last):\n  File \"/Users/rbernstein/airbyte/airbyte-integrations/connectors/source-recall/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/Users/rbernstein/airbyte/airbyte-integrations/connectors/source-recall/.venv/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 131, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/Users/rbernstein/airbyte/airbyte-integrations/connectors/source-recall/.venv/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 119, in run\n    config_catalog = self.source.read_catalog(parsed_args.catalog)\n  File \"/Users/rbernstein/airbyte/airbyte-integrations/connectors/source-recall/.venv/lib/python3.10/site-packages/airbyte_cdk/sources/source.py\", line 90, in read_catalog\n    return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\n  File \"/Users/rbernstein/airbyte/airbyte-integrations/connectors/source-recall/.venv/lib/python3.10/site-packages/airbyte_cdk/connector.py\", line 53, in read_config\n    return json.loads(contents)\n  File \"/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py\", line 346, in loads\n    return _default_decoder.decode(s)\n  File \"/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py\", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File \"/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py\", line 355, in raw_decode\n    raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 12 column 3 (char 239)"
  }
}
{
  "type": "TRACE",
  "trace": {
    "type": "ERROR",
    "emitted_at": 1667598781100.925,
    "error": {
      "message": "Something went wrong in the connector. See the logs for more details.",
      "internal_message": "Expecting value: line 12 column 3 (char 239)",
      "stack_trace": "Traceback (most recent call last):\n  File \"/Users/rbernstein/airbyte/airbyte-integrations/connectors/source-recall/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/Users/rbernstein/airbyte/airbyte-integrations/connectors/source-recall/.venv/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 131, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/Users/rbernstein/airbyte/airbyte-integrations/connectors/source-recall/.venv/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 119, in run\n    config_catalog = self.source.read_catalog(parsed_args.catalog)\n  File \"/Users/rbernstein/airbyte/airbyte-integrations/connectors/source-recall/.venv/lib/python3.10/site-packages/airbyte_cdk/sources/source.py\", line 90, in read_catalog\n    return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\n  File \"/Users/rbernstein/airbyte/airbyte-integrations/connectors/source-recall/.venv/lib/python3.10/site-packages/airbyte_cdk/connector.py\", line 53, in read_config\n    return json.loads(contents)\n  File \"/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py\", line 346, in loads\n    return _default_decoder.decode(s)\n  File \"/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py\", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File \"/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py\", line 355, in raw_decode\n    raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 12 column 3 (char 239)\n",
      "failure_type": "system_error"
    }
  }
}

@brianjlai
Copy link
Contributor

grooming notes:

  • break this into two tickets of renaming the config and deduplicating
  • catching and writing the prettier error message

@maxi297
Copy link
Contributor

maxi297 commented Dec 7, 2022

This has been solved as part of #20019

@maxi297 maxi297 closed this as completed Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepting-contributions Feel free to contribute to them team/extensibility type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants