🚨🚨 Source Microsoft Teams: Update Schemas

[ChristoGrab]

What

This PR is part of the larger effort to update the CDK versions on some of our older Community connectors. It also resolves #10243.

• Updated all stream field names to use "camelCase" naming convention rather than "snake_case", to match the Microsoft Graph API.
• The "team_device_usage_report" stream was experiencing consistent fatal sync errors during tests. This stream extracts CSV data from the API and maps it to the declared field names before expressing the record, so the mismatched names were rendering it unusable. It seems to be working properly with the updated field names.
• Removed all instances of the anyOf keyword being used to make array fields nullable, in favor of:

"type": ["null", "array"]

• Added date and timestamp_without_timezone airbyte-types where applicable.
• Added some undeclared columns.
• Flagged the tenant_id field as an airbyte_secret in the connector's spec
• Removed explicit schema declarations from configured_catalog.json
• Unpinned airbyte-cdk version

Note on CAT failure

There is one outstanding CAT test failure due to the explicit declaration of additionalProperties: false in the connector's spec. We enforce that our specs should never explicitly disallow additional properties for backwards compatibility purposes, but it seems in this connector the flag is being purposefully used to delineate two authentication options (OAuth and Token) which are almost identical in nature:

OAuth: ['tenant_id', 'client_id', 'client_secret', 'refresh_token']
Token: ['tenant_id', 'client_id', 'client_secret']

Removing the false flag causes the OAuth check to fail, due to the credentials being considered valid for either authentication method (since refresh_token is treated as a valid undeclared key for Token auth). While this reliance on an anti-pattern is not ideal, it seems to work, so I think a fix is outside the scope of this update and best saved for a separate PR. I have created an issue and am hoping this PR can be merged in spite of the test failure.

Suggested Reading Order

1. source_microsoft_teams/client.py && source_microsoft_teams/schemas/*.json
2. docs/integrations/sources/microsoft-teams-migration.md
3. anything else

User Impact

This is a breaking change due to the removal/renaming of multiple schema properties. Although Microsoft Teams does not support incremental sync modes, customers will still need to refresh their schemas, and should be encouraged to reset their data if they are using Full Refresh | Append mode.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 10, 2024 6:38pm

[github-actions]

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

• PR name follows PR naming conventions
• Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
• Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
• You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
• Secrets in the connector's spec are annotated with airbyte_secret
• All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
• Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
• Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
• If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

1. Check for hidden checklists in your PR description

2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

[ChristoGrab]

Did not add a "date" format to the "deletedDate" field; it is expressed as an empty string rather than a null value when empty, which will throw a validation error if the format is enforced.

[evantahler]

@alafanechere - what do you think about this CAT test failure? Should we allow this case?

[alafanechere]

@alafanechere - what do you think about this CAT test failure? Should we allow this case?

I indeed think that the additionalProperties field should be removed from the credentials object in the spec, it will default to true.
If we're not allowing additionalProperties it means we'd have to perform a config migration on the next iteration on this credentials field if we remove a field and a rollback to the previous config state needs to be supported in case of a version downgrade.
So we enforce additionalProperties: true on spec field to keep flexibility and maximize backward compatiblity.

Looking at this PR, this is a test that was likely previously failing. And to make it pass we should just remove additionalProperties: false and the credentials oneOf (not the advancedAuth section.

For example, on source-google-sheets, we have a similar credentials field on which additionalProperties is not set.

[ChristoGrab]

@alafanechere Thanks for taking a look at this! Apologies for the slow turnaround, took me a while to figure out what was causing the CAT failures when removing the false flag.

It turns out the root cause was just a slight misconfiguration in our GSM credentials which was preventing the connector from validating the correct auth method being used when additional properties were allowed. I've updated the creds, and we are now fully green with additionalProps allowed 👍

[alafanechere]

🟢 CI = LGTM 😄