1s1t CatalogParser should handle raw table name collisions

[edgao]

In 1s1t, two streams may want to write to the same raw table. E.g. public.users_foo and public_users.foo would both naively want to write to airbyte.public_users_foo. CatalogParserTest#rawNameCollision captures this, but is currently disabled.

Decide how we want to resolve these collisions, then implement that decision in CatalogParser#parseCatalog. Maybe options:

• use a more interesting delimiter between the namespace+name (e.g. write to airbyte.public__airbyte_raw__users_foo + airbyte.public_users__airbyte_raw__foo)
• add some more characters to colliding names (incrementing counter, more underscores, etc)
• something else?

[aaronsteers]

Related:

• airbytehq/airbyte-internal-issues#1877

[rodireich]

Grooming meeting 6/28:
Sources team to investigate whether this usage of '_' is generated in source.

[edgao]

uhhhhhh I may have screwed up a zendesk thing. This is definitely a destinations issue. Not sure why I tagged it as db sources, that's my bad.

[rodireich]

Thanks Ed!

[jbfbell]

@edgao did I make a duplicate of this in #27817 ? or is this something else

[edgao]

I think they're slightly different and worth having separate issues. This issue is: within a single connection, user has two streams public.users_foo and public_users.foo. This is solvable purely by making CatalogParser resolve those conflicts, because we have both streams in our catalog.

Your issue is: across multiple connections (or even outside of airbyte entirely) the user has an airbyte schema with tables in it already. That's not solvable in CatalogParser, and presumably requires a lot more thought.

[jbfbell]

When working on this ticket, consider cleaning up the parseCatalog method in CatalogParser per this discussion

[evantahler]

Grooming:

• we should do something, but something small. To start, let's use __ as a delimiter and we can rev on this as we see bugs