Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 New Destination: Pulsar #7315

Merged
merged 20 commits into from
Nov 5, 2021
Merged

🎉 New Destination: Pulsar #7315

merged 20 commits into from
Nov 5, 2021

Conversation

mmolimar
Copy link
Contributor

What

Destination for Apache Pulsar.
Related with this issue. Part of the hacktoberfest

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/SUMMARY.md
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions
  • Connector added to connector index like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions
  • Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • If adding a new generator, add it to the list of scaffold modules being tested
  • The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • Documentation which references the generator is updated as needed.

@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 24, 2021
@marcosmarxm
Copy link
Member

Awesome @mmolimar I requested the team to review your contribution

@VitaliiMaltsev
Copy link
Contributor

@mmolimar seems like you have an issue with acceptance tests hanging
pulsar

@mmolimar
Copy link
Contributor Author

Let me check this and I'll get back to you. Thanks @VitaliiMaltsev !

VitaliiMaltsev
VitaliiMaltsev reviewed Oct 28, 2021
View reviewed changes
...ulsar/src/test/java/io/airbyte/integrations/destination/pulsar/PulsarRecordConsumerTest.java Outdated
final PulsarRecordConsumer recordConsumer = new PulsarRecordConsumer(config, CATALOG, mock(Consumer.class), NAMING_RESOLVER);

final Map<AirbyteStreamNameNamespacePair, Producer<JsonNode>> producerMap = recordConsumer.buildProducerMap();
assertEquals(1, producerMap.size());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add also unit test to build ProducerMap with more then 1 stream

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still needed unit test for producerMap.size() > 1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@mmolimar
Copy link
Contributor Author

Hey @VitaliiMaltsev

I did a refactor and added some configurations.
I think I've addressed all your comments :-)

Let me know what you think.

@mmolimar
Copy link
Contributor Author

mmolimar commented Nov 2, 2021

I think everything should be fine now @VitaliiMaltsev

VitaliiMaltsev
VitaliiMaltsev approved these changes Nov 3, 2021
View reviewed changes
Copy link
Contributor

@VitaliiMaltsev VitaliiMaltsev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CLAassistant
Copy link

CLAassistant commented Nov 5, 2021 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@marcosmarxm marcosmarxm merged commit 16d9d15 into airbytehq:master Nov 5, 2021
@jrhizor jrhizor mentioned this pull request Nov 6, 2021
Merged
schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022
@mmolimar @marcosmarxm @schlattk
🎉 New Destination: Pulsar (airbytehq#7315)
4525cee 
* Adding Pulsar destination

* Add enable_chunking property

* Add dependency for DNS resolve for Mac

* Updating build.gradle

* Adding definition

* Refactor and fix tests

* Adding new config properties

* Reformat code

* Close resources when checking connection

* Updating doc

* Adding more test to build the producer map

* Rename brokers property

* Avoid emitting null states to the output collector

* fix config file

* run seed source/destination

Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>
@sajarin sajarin added internal and removed bounty labels Oct 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@VitaliiMaltsev VitaliiMaltsev VitaliiMaltsev approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation community connectors/destination/pulsar gl internal
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@mmolimar @marcosmarxm @VitaliiMaltsev @CLAassistant @sajarin @octavia-squidington-iii @octavia-squidington-iv