Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐞 Postgres source: fix azure database catalog retrieval #12834

Merged
merged 5 commits into from
May 13, 2022
Merged

🐞 Postgres source: fix azure database catalog retrieval #12834

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
47816e1
Remove host in azure username
tuliren May 13, 2022
80714f2
Bump version
tuliren May 13, 2022
90d9c07
Fix test
tuliren May 13, 2022
53e8262
Replace host with jdbc_url
tuliren May 13, 2022
977f6f1
auto-bump connector version
octavia-squidington-iii May 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -715,7 +715,7 @@
- name: Postgres
sourceDefinitionId: decd338e-5647-4c0b-adf4-da0e75f5a750
dockerRepository: airbyte/source-postgres
dockerImageTag: 0.4.14
dockerImageTag: 0.4.15
documentationUrl: https://docs.airbyte.io/integrations/sources/postgres
icon: postgresql.svg
sourceType: database
Expand Down
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/source_specs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6540,7 +6540,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-postgres:0.4.14"
- dockerImage: "airbyte/source-postgres:0.4.15"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/postgres"
connectionSpecification:
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.4.14
LABEL io.airbyte.version=0.4.15
LABEL io.airbyte.name=airbyte/source-postgres-strict-encrypt
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-postgres/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.4.14
LABEL io.airbyte.version=0.4.15
LABEL io.airbyte.name=airbyte/source-postgres
20 changes: 19 additions & 1 deletion ...source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,7 +332,7 @@ public Set<JdbcPrivilegeDto> getPrivilegesTableForCurrentUser(final JdbcDatabase
WHERE t.grpid = m.oid AND t.userid = r.oid)
AND privilege_type = 'SELECT';
""");
final String username = database.getDatabaseConfig().get("username").asText();
final String username = getUsername(database.getDatabaseConfig());
ps.setString(1, username);
ps.setString(2, username);
ps.setString(3, username);
Expand All @@ -349,6 +349,24 @@ public Set<JdbcPrivilegeDto> getPrivilegesTableForCurrentUser(final JdbcDatabase
.collect(toSet());
}

@VisibleForTesting
static String getUsername(final JsonNode databaseConfig) {
final String jdbcUrl = databaseConfig.get("jdbc_url").asText();
final String username = databaseConfig.get("username").asText();

// Azure Postgres server has this username pattern: <username>@<host>.
// Inside Postgres, the true username is just <username>.
// The jdbc_url is constructed in the toDatabaseConfigStatic method.
if (username.contains("@") && jdbcUrl.contains("azure.com:")) {
final String[] tokens = username.split("@");
final String postgresUsername = tokens[0];
LOGGER.info("Azure username \"{}\" is detected; use \"{}\" to check permission", username, postgresUsername);
return postgresUsername;
}

return username;
}

@Override
protected boolean isNotInternalSchema(final JsonNode jsonNode, final Set<String> internalSchemas) {
return false;
Expand Down
18 changes: 18 additions & 0 deletions ...ce-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresSourceTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
import io.airbyte.test.utils.PostgreSQLContainerHelper;
import java.math.BigDecimal;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
Expand Down Expand Up @@ -359,4 +360,21 @@ void testIsCdc() {
assertTrue(PostgresSource.isCdc(config));
}

@Test
void testGetUsername() {
final String username = "airbyte-user";

// normal host
final JsonNode normalConfig = Jsons.jsonNode(Map.of(
"username", username,
"jdbc_url", "jdbc:postgresql://airbyte.database.com:5432:airbyte"));
assertEquals(username, PostgresSource.getUsername(normalConfig));

// azure host
final JsonNode azureConfig = Jsons.jsonNode(Map.of(
"username", username + "@airbyte",
"jdbc_url", "jdbc:postgresql://airbyte.azure.com:5432:airbyte"));
assertEquals(username, PostgresSource.getUsername(azureConfig));
}

}
3 changes: 2 additions & 1 deletion docs/integrations/sources/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,7 @@ According to Postgres [documentation](https://www.postgresql.org/docs/14/datatyp

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------|
| 0.4.15 | 2022-05-13 | [12834](https://github.com/airbytehq/airbyte/pull/12834) | Fix the bug that the connector returns empty catalog for Azure Postgres database |
| 0.4.14 | 2022-05-08 | [12689](https://github.com/airbytehq/airbyte/pull/12689) | Add table retrieval according to role-based `SELECT` privilege |
| 0.4.13 | 2022-05-05 | [10230](https://github.com/airbytehq/airbyte/pull/10230) | Explicitly set null value for field in json |
| 0.4.12 | 2022-04-29 | [12480](https://github.com/airbytehq/airbyte/pull/12480) | Query tables with adaptive fetch size to optimize JDBC memory consumption |
Expand All @@ -278,7 +279,7 @@ According to Postgres [documentation](https://www.postgresql.org/docs/14/datatyp
| 0.4.8 | 2022-02-21 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats |
| 0.4.7 | 2022-02-18 | [10242](https://github.com/airbytehq/airbyte/pull/10242) | Updated timestamp transformation with microseconds |
| 0.4.6 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | (unpublished) Add `-XX:+ExitOnOutOfMemoryError` JVM option |
| 0.4.5 | 2022-02-08 | [10173](https://github.com/airbytehq/airbyte/pull/10173) | Improved discovering tables in case if user does not have permissions to any table |
| 0.4.5 | 2022-02-08 | [10173](https://github.com/airbytehq/airbyte/pull/10173) | Improved discovering tables in case if user does not have permissions to any table |
| 0.4.4 | 2022-01-26 | [9807](https://github.com/airbytehq/airbyte/pull/9807) | Update connector fields title/description |
| 0.4.3 | 2022-01-24 | [9554](https://github.com/airbytehq/airbyte/pull/9554) | Allow handling of java sql date in CDC |
| 0.4.2 | 2022-01-13 | [9360](https://github.com/airbytehq/airbyte/pull/9360) | Added schema selection |
Expand Down