Connector-builder server stub #18410

girarda · 2022-10-25T00:32:19Z

What

Add the skeleton for the connector-builder server without including it the the deploys.

The connector-builder server will server as a bridge between the connector builder UI and the CDK.

The updates to the deploy files were commented out so we can minimize the drift between master and our dev branch without impacting the users.
The module is added to gradle so we still build it.

The server is a FastAPI server running on port 8003 exposing the following endpoints:

"/"
"/items/{item_id}"

A follow up PR includes server generation for the connector-builder server.

How

Create airbyte-connector-builder module which contains a Dockerfile running a FastAPI dummy server
Move the openAPI spec file to the airbyte-connector-builder module (and update the webapp to point to it)
Add airbyte-connector-builder to the docker-compose and kube charts (commented out)
Install python on the CI runners

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

This reverts commit aca3534.

This reverts commit f9258a7.

girarda · 2022-11-04T04:12:09Z

.bumpversion.cfg

@@ -61,3 +61,11 @@ serialize =
 [bumpversion:file:octavia-cli/install.sh]

 [bumpversion:file:octavia-cli/setup.py]
+
+[bumpversion:file:connector-builder/Dockerfile]


add the module to so its version gets bumped with the others

does this mean that to get a connector builder update, I have to upgrade my whole airbyte instance? in that case we should probably recommend that people develop lowcode connectors in a separate instance from their prod one. Just a note for the docs in the future

girarda · 2022-11-04T04:13:37Z

airbyte-connector-builder/build.gradle

+
+
+task generateApiServer(type: GenerateTask) {
+    inputSpec = "$rootDir.absolutePath/airbyte-connector-builder/src/main/openapi/openapi.yaml"


this is the new path to the openAPI spec

girarda · 2022-11-04T04:14:39Z

airbyte-connector-builder/connector_builder/entrypoint.py

+
+
+@app.get("/")
+def read_root():


this is a dummy server. the openAPI template will come in a follow up PR

girarda · 2022-11-04T04:18:10Z

airbyte-webapp/orval.config.ts

@@ -23,7 +23,7 @@ export default defineConfig({
    },
  },
  connectorBuilder: {
-    input: "../connector-builder-server/src/main/openapi/openapi.yaml",
+    input: "../airbyte-connector-builder/src/main/openapi/openapi.yaml",


moved the openapi spec to the airbyte-connector-builder module

girarda · 2022-11-04T04:18:40Z

airbyte-webapp/src/config/defaultConfig.ts

@@ -5,7 +5,7 @@ const defaultConfig: Config = {
  healthCheckInterval: 20000,
  version: "dev",
  apiUrl: `${window.location.protocol}//${window.location.hostname}:8001/api`,
-  connectorBuilderApiUrl: `${window.location.protocol}//${window.location.hostname}:8080/`,
+  connectorBuilderApiUrl: `${window.location.protocol}//${window.location.hostname}:8003/`,


changed the port to 8003 as discussed

girarda · 2022-11-04T04:19:27Z

docker-compose.build.yaml

@@ -29,6 +29,13 @@ services:
      context: airbyte-workers
      labels:
        io.airbyte.git-revision: ${GIT_REVISION}
+    #connector-builder:


(commented out)
this adds the connector-builder server to the docker-compose deploy

girarda · 2022-11-04T04:19:37Z

docker-compose.yaml

+  #  container_name: airbyte-connector-builder
+  #  restart: unless-stopped
+  #  ports:
+  #    - 8003


port is not exposed to localhost

girarda · 2022-11-04T04:19:50Z

docker-compose.yaml

  airbyte-proxy:
    image: airbyte/proxy:${VERSION}
    container_name: airbyte-proxy
    ports:
      - 8000:8000
      - 8001:8001
+      #- 8003:8003 FIXME: Uncomment this when enabling airbyte-connector-builder


only expose the server through airbyte-proxy

sherifnada

LGTM, will add back the commented out parts in #18886

sherifnada · 2022-10-29T16:52:43Z

airbyte-connector-builder/Dockerfile

+
+RUN pip install --no-cache-dir .
+
+ENTRYPOINT ["uvicorn", "connector_builder.entrypoint:app", "--host", "0.0.0.0", "--port", "80"]


@girarda do you remember why this is explicitly specified as 0.0.0.0? 127.0.0.1 didn't work for me, was curious if you understood why

I'm not entirely sure why 127.0.0.1 does not work. the doc says

Use --host 0.0.0.0 to make the application available on your local network

127.0.0.1 is the loopback address which will only take traffic from the same host, if you need any to listen to traffic coming from outside the container use 0.0.0.0 (means effectively "from anywhere")

sherifnada · 2022-11-04T14:54:17Z

.bumpversion.cfg

@@ -61,3 +61,11 @@ serialize =
 [bumpversion:file:octavia-cli/install.sh]

 [bumpversion:file:octavia-cli/setup.py]
+
+[bumpversion:file:connector-builder/Dockerfile]


does this mean that to get a connector builder update, I have to upgrade my whole airbyte instance? in that case we should probably recommend that people develop lowcode connectors in a separate instance from their prod one. Just a note for the docs in the future

girarda · 2022-11-04T15:50:52Z

@sherifnada

to get a connector builder update, I have to upgrade my whole airbyte instance?
Yes, this is true for the FE too

lmossman · 2022-11-07T21:45:34Z

kube/overlays/stable/kustomization.yaml

+  #- name: airbyte/connector-builder #FIXME: Uncomment this block when enabling airbyte-connector-builder
+  #  newTag: 0.40.18


If it is not a huge lift to also add this server to our airbyte-cloud deployment, it would be pretty nice to do that so people could just access the Connector Builder through a URL instead of having to spin up Airbyte themselves.

Alternatively, I believe the demo.airbyte.io instance is actually an OSS deployment that is just made publicly accessible. So people could access the Connector Builder there too, which has the added benefit of not requiring any authentication to use

there are a few open questions to answer like whether we need to enforce rate limits to protect the backend against a bad user, but using the demo instance is a good idea for making the connector public without putting prod at risk

davinchia · 2022-11-08T01:53:53Z

.env.dev

@@ -23,6 +23,7 @@ HACK_LOCAL_ROOT_PARENT=/tmp
 WEBAPP_URL=http://localhost:8000/
 API_URL=/api/v1/
 INTERNAL_API_HOST=airbyte-server:8001
+#CONNECTOR_BUILDER_API_HOST=airbyte-connector-builder:8003 #FIXME: Uncomment this when enabling the connector-builder


this is commented out for now because this is stubbed?

yes. I doubt there would be any harm to exposing the env variable, but we can just turn this on at the same time as we enable the server

davinchia

I had a short conversation with Sherif offline and although I understand the reasons for choosing to host the Connector Builder UI in the Webapp, I think we now have leaked this complexity into the rest of the deploy tooling.

We should properly decide how to handle this sooner rather than later to reduce both product and technical complexity.

davinchia · 2022-11-08T23:45:48Z

@airbytehq/production-engineering can someone take a quick look here?

I'm tagging someone from the team because I think it's useful for the PE to understand bits of the OSS deploy stack for future work.

Here, what we are doing is:

setting the stage so the connector builder server can be part of the general Airbyte deployment, OSS and Kube.
the connector builder server is the backend for the low-code cdk UI - this is the low-code framework that allows devs to build apis without writing code.
ideally this is decoupled from airbyte - i.e. a low-code dev doesn't have to deploy Airbyte to use the low-code cdk. However, the FE doesn't really support this today, and we've made the choice to embed the low-code UI with the current Webapp for velocity sake.
In the long term, we will remove this. In the short term, the server will be deployed with Airbyte to unblock us for now.

perangel · 2022-11-09T00:59:25Z

airbyte-connector-builder/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Airbyte, Inc.


s/2020/2022/

lmossman · 2022-11-09T17:18:27Z

docker-compose.yaml

@@ -209,12 +210,24 @@ services:
      - workspace:${WORKSPACE_ROOT}
    networks:
      - airbyte_internal
+  #airbyte-connector-builder: #FIXME: Uncomment this block when enabling airbyte-connector-builder


@girarda I wonder if we should call this airbyte-connector-builder-server, to make it clear that this is only the backing server for the connector builder, and not the connector builder webapp?

@lmossman do you think we should rename the module too?

update: I renamed the module too for consistency

* init * bad copy/paste * move to top level * Revert "move to top level" This reverts commit aca3534. * attempt to wire up connector builder frontend to server * copy from octaviacli * fix connection to builder server * update * delete * Update * delete python-version * Revert "delete python-version" This reverts commit f9258a7. * setup python * install python * rename * kube stuff * Install python * missing kube file * rename * Update files * Update bumpversion * install python * try with different entrypoint * rename container * point to docker-compose.yaml file * derp * copy acceptance_test.sh * copy from acceptance tests * delete cruft * update * remove application env * reset * reset to master * update * skip comprehensive incremental tests * Revert "skip comprehensive incremental tests" This reverts commit 9cee657. * reset to master * remove cruft * delete superfluous steps * update port to 8003 * reset to master * Update publish docker * move openapi spec to airbyte-connector-builder * point to openapi spec * dont expose the connector builder to localhost * reset FE components to master * Don't deploy the connector-builder * Revert "Don't deploy the connector-builder" This reverts commit 3d15749. * Revert "Revert "Don't deploy the connector-builder"" This reverts commit beac3d4. * comment out more things related to connector builder server * more attempts at removing the connector builder * comment out more things * Apply suggestions from code review Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/config/configProviders.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * update * rename * indent * Revert "move openapi spec to airbyte-connector-builder" This reverts commit 57dda04. * Revert "rename" This reverts commit b2d802b. * Revert "Revert "rename"" This reverts commit 91db24f. * point to wrong file in case it fixes the build * point to right openapi file * Revert "Revert "move openapi spec to airbyte-connector-builder"" This reverts commit e46a837. * point to moved file * fix path * Update from master * newline * Add failing test * Revert "Add failing test" This reverts commit ed9fea0. * comment * update commented requires * Add a comment * 2022 * rename to connector-builder-server * typo Co-authored-by: lmossman <lake@airbyte.io>

girarda and others added 30 commits October 18, 2022 15:38

init

7736109

bad copy/paste

ad5abf0

move to top level

aca3534

Revert "move to top level"

75ad65c

This reverts commit aca3534.

attempt to wire up connector builder frontend to server

77197a1

copy from octaviacli

b05daa8

fix connection to builder server

8543034

update

7fad84f

Merge branch 'master' into alex/connector_builder

4cad0d5

delete

f745d28

Update

795523b

delete python-version

f9258a7

Revert "delete python-version"

2419f54

This reverts commit f9258a7.

setup python

5022710

install python

3360dad

rename

faf6eae

kube stuff

39efd07

Install python

33e895b

missing kube file

88d846a

rename

e8da375

Update files

1172f14

Update bumpversion

82e98c7

install python

0e16c33

try with different entrypoint

e1e01b7

Merge branch 'master' into alex/connector_builder

0820e52

rename container

71516c8

point to docker-compose.yaml file

a10d7ba

derp

742913b

copy acceptance_test.sh

66fa528

merge

c7cf7e5

girarda added 2 commits November 3, 2022 21:22

update commented requires

0f048d3

Add a comment

460720a

girarda temporarily deployed to more-secrets November 4, 2022 04:25 Inactive

girarda commented Nov 4, 2022

View reviewed changes

sherifnada approved these changes Nov 4, 2022

View reviewed changes

lmossman reviewed Nov 7, 2022

View reviewed changes

lmossman approved these changes Nov 7, 2022

View reviewed changes

davinchia reviewed Nov 8, 2022

View reviewed changes

davinchia approved these changes Nov 8, 2022

View reviewed changes

perangel reviewed Nov 9, 2022

View reviewed changes

airbyte-connector-builder/LICENSE Outdated

@@ -0,0 +1,21 @@

MIT License

Copyright (c) 2020 Airbyte, Inc.

Copy link

Contributor

perangel Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/2020/2022/

perangel approved these changes Nov 9, 2022

View reviewed changes

2022

c683e83

girarda temporarily deployed to more-secrets November 9, 2022 06:50 Inactive

Merge branch 'master' into alex/connector_builder_fastapi

75fd366

girarda temporarily deployed to more-secrets November 9, 2022 15:29 Inactive

lmossman reviewed Nov 9, 2022

View reviewed changes

rename to connector-builder-server

6e7019d

girarda temporarily deployed to more-secrets November 9, 2022 19:29 Inactive

typo

78d50ca

girarda temporarily deployed to more-secrets November 9, 2022 19:38 Inactive

Merge branch 'master' into alex/connector_builder_fastapi

2a439af

girarda temporarily deployed to more-secrets November 9, 2022 20:22 Inactive

girarda merged commit 64b50ad into master Nov 9, 2022

girarda deleted the alex/connector_builder_fastapi branch November 9, 2022 21:07

girarda mentioned this pull request Nov 10, 2022

Use nginx to route requests from port 8003 to connector builder #19264

Closed

37 tasks

github-actions bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connector-builder server stub #18410

Connector-builder server stub #18410

girarda commented Oct 25, 2022 •

edited

Loading

girarda Nov 4, 2022

sherifnada Nov 4, 2022

girarda Nov 4, 2022

girarda Nov 4, 2022

girarda Nov 4, 2022

girarda Nov 4, 2022

girarda Nov 4, 2022

girarda Nov 4, 2022

girarda Nov 4, 2022

sherifnada left a comment

sherifnada Oct 29, 2022

girarda Nov 4, 2022

perangel Nov 9, 2022

sherifnada Nov 4, 2022

girarda commented Nov 4, 2022 •

edited

Loading

lmossman Nov 7, 2022

girarda Nov 9, 2022

davinchia Nov 8, 2022

girarda Nov 9, 2022

davinchia left a comment

davinchia commented Nov 8, 2022

perangel Nov 9, 2022

lmossman Nov 9, 2022

girarda Nov 9, 2022 •

edited

Loading



		task generateApiServer(type: GenerateTask) {
		inputSpec = "$rootDir.absolutePath/airbyte-connector-builder/src/main/openapi/openapi.yaml"


		RUN pip install --no-cache-dir .

		ENTRYPOINT ["uvicorn", "connector_builder.entrypoint:app", "--host", "0.0.0.0", "--port", "80"]

		#- name: airbyte/connector-builder #FIXME: Uncomment this block when enabling airbyte-connector-builder
		# newTag: 0.40.18



		@app.get("/")
		def read_root():

Connector-builder server stub #18410

Connector-builder server stub #18410

Conversation

girarda commented Oct 25, 2022 • edited Loading

What

How

Recommended reading order