🪟🎉 Connector builder UI: User inputs, request options, authentication…

…, pagination, slicing (#20809) * improve some types * improve further * clean up a bit more * refactor loading state * move loading state up * remove isLoading references * remove unused props and make fetch connector error work * remove special component for name * remove top level state for unifinished flows * start removing uiwidget * Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss Co-authored-by: Tim Roes <tim@airbyte.io> * remove undefined option for selected id * remove unused prop * fix types * remove uiwidget state * clean up * adjust comment * handle errors in a nice way * do not respect default on oneOf fields * rename to formblock * reduce re-renders * pass error to secure inputs * simplify and improve styling * align top * code review * remove comment * review comments * rename file * be strict about boolean values * add example * track form error in error boundary * review comments * handle unexpected cases better * enrich error with connector id * 🪟🎉 Add copy stream button (#20577) * add copy stream button * review comments * rename prop * 🪟🎉 Connector builder: Integrate connector form for test input (#20385) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * fix small stuff * add warning label * review comments * adjust translation Co-authored-by: lmossman <lake@airbyte.io> * use request_body_json instead of request_body_data * 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699) * move add button into line * add stories for empty with control, and content + control * change button name to Control * 🪟🎉 Connector builder: Allow defining inputs (#20431) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * handle stored form values that don't contain new fields properly * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * 🪟🎉 Connector builder authentication (#20645) * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * fix keys * 🪟🎉 Connector builder: Session token and oauth authentication (#20712) * session token and oauth authentication * fill in session token variable * typos * make sure validation error does not go away * 🪟🎉 Connector builder: Always validate inputs form (#20664) * validate user input outside of form * review comments Co-authored-by: lmossman <lake@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io> * fix merge conflict with dropdown prop being renamed to control * [Connector Builder] Add paginator (#20698) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly Co-authored-by: Joe Reuter <joe@airbyte.io> * [Connector Builder] Add stream slicer (#20748) * move connector builder components into the same shared components/connectorBuilder directory * move diff over from poc branch * save current progress * add modal for adding streams * focus stream after adding and reset button style * add reset confirm modal and select view on add * style global config and streams buttons * styling improvements * handle long stream names better * pull in connector manifest schema directly * add box shadows to resizable panels * upgrade orval and use connector manifest schema directly * remove airbyte protocol from connector builder api spec * generate python models from openapi change * fix position of yaml toggle * handle no stream case with better looking message * group global fields into single object and fix console error * confirmation modal on toggling dirty form + cleanup * fix connector name display * undo change to manifest schema * remove commented code * remove unnecessary change * fix spacing * use shadow mixin for connector img * add comment about connector img * change onSubmit to no-op * remove console log * clean up styling * simplify sidebar to remove StreamSelectButton component * swap colors of toggle * move FormikPatch to src/core/form * move types up to connectorBuilder/ level * use grid display for ui yaml toggle button * use spread instead of setting array index directly * add intl in missing places * pull connector manifest schema in through separate openapi spec * use correct intl string id * throttle setting json manifest in yaml editor * use button prop instead of manually styling * consolidate AddStreamButton styles * fix sidebar flex styles * use specific flex properties instead of flex * clean up download and reset button styles * use row-reverse for yaml editor download button * fix stream selector styles to remove margins * give connector setup guide panel same corner and shadow styles * remove blur from page display * set view to stream when selected in test panel * add placeholder when stream name is empty * switch to index-based stream selection to preserve testing panel selected stream on rename * handle empty name in stream selector * make connector form work in connector builder * wip * fix small stuff * add basic input UI * user inputs * make most of inputs configuration work * fix a bunch of stuff * handle unknown config types * add warning label * fix label * fix some styling * review comments * improve state management and error handling * allow auth configuration * check for conflicts with the inferred inputs * fix invisible inputs * handle stored form values that don't contain new fields properly * session token and oauth authentication * fill in session token variable * fix merge of default values * add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts * add cursor and primary key fields to ui * save * add page size and token option inputs * fixes after rebase * add pagination * fix pagination types * handle empty field_name better * Update airbyte-webapp/src/locales/en.json Co-authored-by: Lake Mossman <lake@airbyte.io> * Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx Co-authored-by: Lake Mossman <lake@airbyte.io> * inputs editing weirdness * input form reset * using the Label component * reduce redundancy and hide advanced input options for inferred inputs * unnecessary validation * typo * unnecessary effect hook * build spec even for invalid forms but do not update stream list * typos * make sure validation error does not go away * make primary key and cursor optional, and reorder * save toggle group progress * fix style of toggle label * handle empty values better * fix page size/token option field validation and rendering * handle cursor pagination page size option correctly * save stream slicer progress * finish stream slicer * fix stream slicer fields and validation Co-authored-by: Joe Reuter <joe@airbyte.io> * debounce form builder values update to reduce load * 🪟🔧 Connector builder: use new lowcode manifest (#20715) * use new manifest yaml * Update airbyte-webapp/src/components/connectorBuilder/types.ts Co-authored-by: Lake Mossman <lake@airbyte.io> * use updated manifest types Co-authored-by: Lake Mossman <lake@airbyte.io> * debounce validation as well * akways show stream test button in error state if there are errors * fix type of oauth input * add validation schema for add stream form * validate all views on test click * add type to prevent console warning Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: lmossman <lake@airbyte.io>
airbytehq · Jan 4, 2023 · 0f26915 · 0f26915
1 parent 6a68d1f
commit 0f26915
Show file tree

Hide file tree

Showing 58 changed files with 2,531 additions and 300 deletions.
diff --git a/airbyte-webapp/src/components/GroupControls/GroupControls.module.scss b/airbyte-webapp/src/components/GroupControls/GroupControls.module.scss
@@ -36,11 +36,10 @@ $border-width: variables.$border-thick;
   white-space: nowrap;
 }
 
-.dropdown {
+.control {
   margin-left: auto;
   padding: 0 variables.$spacing-xs;
   background-color: colors.$white;
-  min-width: calc(50% - 100px);
 }
 
 .content {

diff --git a/airbyte-webapp/src/components/GroupControls/GroupControls.tsx b/airbyte-webapp/src/components/GroupControls/GroupControls.tsx
@@ -1,21 +1,29 @@
+import classNames from "classnames";
 import React from "react";
 
 import styles from "./GroupControls.module.scss";
 
 interface GroupControlsProps {
   label: React.ReactNode;
-  dropdown?: React.ReactNode;
+  control?: React.ReactNode;
+  controlClassName?: string;
   name?: string;
 }
 
-const GroupControls: React.FC<React.PropsWithChildren<GroupControlsProps>> = ({ label, dropdown, children, name }) => {
+const GroupControls: React.FC<React.PropsWithChildren<GroupControlsProps>> = ({
+  label,
+  control,
+  children,
+  name,
+  controlClassName,
+}) => {
   return (
     // This outer div is necessary for .content > :first-child padding to be properly applied in the case of nested GroupControls
     <div>
       <div className={styles.container}>
         <div className={styles.title}>
           <div className={styles.label}>{label}</div>
-          <div className={styles.dropdown}>{dropdown}</div>
+          <div className={classNames(styles.control, controlClassName)}>{control}</div>
         </div>
         <div className={styles.content} data-testid={name}>
           {children}

diff --git a/airbyte-webapp/src/components/GroupControls/index.stories.tsx b/airbyte-webapp/src/components/GroupControls/index.stories.tsx
@@ -1,5 +1,6 @@
 import { ComponentStory, ComponentMeta } from "@storybook/react";
 
+import { Button } from "components/ui/Button";
 import { Card } from "components/ui/Card";
 
 import { FormBlock, FormConditionItem } from "core/form/types";
@@ -73,3 +74,21 @@ WithContent.args = {
     </>
   ),
 };
+
+export const EmptyWithControl = Template.bind({});
+EmptyWithControl.args = {
+  label,
+  control: <Button variant="secondary">Control</Button>,
+};
+
+export const ControlAndContent = Template.bind({});
+ControlAndContent.args = {
+  label,
+  control: <Button variant="secondary">Control</Button>,
+  children: (
+    <>
+      <SectionContainer>Content part 1</SectionContainer>
+      <SectionContainer>Content part 2</SectionContainer>
+    </>
+  ),
+};
diff --git a/airbyte-webapp/src/components/connectorBuilder/Builder/AddStreamButton.tsx b/airbyte-webapp/src/components/connectorBuilder/Builder/AddStreamButton.tsx
@@ -1,14 +1,17 @@
 import { Form, Formik, useField } from "formik";
+import merge from "lodash/merge";
 import { useState } from "react";
+import React from "react";
 import { FormattedMessage, useIntl } from "react-intl";
+import * as yup from "yup";
 
 import { Button } from "components/ui/Button";
 import { Modal, ModalBody, ModalFooter } from "components/ui/Modal";
 
 import { FormikPatch } from "core/form/FormikPatch";
 
 import { ReactComponent as PlusIcon } from "../../connection/ConnectionOnboarding/plusIcon.svg";
-import { BuilderStream } from "../types";
+import { BuilderStream, DEFAULT_BUILDER_STREAM_VALUES } from "../types";
 import styles from "./AddStreamButton.module.scss";
 import { BuilderField } from "./BuilderField";
 
@@ -19,44 +22,48 @@ interface AddStreamValues {
 
 interface AddStreamButtonProps {
   onAddStream: (addedStreamNum: number) => void;
+  button?: React.ReactElement;
+  initialValues?: Partial<BuilderStream>;
 }
 
-export const AddStreamButton: React.FC<AddStreamButtonProps> = ({ onAddStream }) => {
+export const AddStreamButton: React.FC<AddStreamButtonProps> = ({ onAddStream, button, initialValues }) => {
   const { formatMessage } = useIntl();
   const [isOpen, setIsOpen] = useState(false);
   const [streamsField, , helpers] = useField<BuilderStream[]>("streams");
   const numStreams = streamsField.value.length;
 
+  const buttonClickHandler = () => {
+    setIsOpen(true);
+  };
+
   return (
     <>
-      <Button
-        className={styles.addButton}
-        onClick={() => {
-          setIsOpen(true);
-        }}
-        icon={<PlusIcon />}
-      />
+      {button ? (
+        React.cloneElement(button, {
+          onClick: buttonClickHandler,
+        })
+      ) : (
+        <Button className={styles.addButton} onClick={buttonClickHandler} icon={<PlusIcon />} />
+      )}
       {isOpen && (
         <Formik
           initialValues={{ streamName: "", urlPath: "" }}
           onSubmit={(values: AddStreamValues) => {
             helpers.setValue([
               ...streamsField.value,
-              {
+              merge({}, DEFAULT_BUILDER_STREAM_VALUES, {
+                ...initialValues,
                 name: values.streamName,
                 urlPath: values.urlPath,
-                fieldPointer: [],
-                httpMethod: "GET",
-                requestOptions: {
-                  requestParameters: [],
-                  requestHeaders: [],
-                  requestBody: [],
-                },
-              },
+              }),
             ]);
             setIsOpen(false);
             onAddStream(numStreams);
           }}
+          validationSchema={yup.object().shape({
+            streamName: yup.string().required("form.empty.error"),
+            urlPath: yup.string().required("form.empty.error"),
+          })}
         >
           <>
             <FormikPatch />
@@ -71,13 +78,13 @@ export const AddStreamButton: React.FC<AddStreamButtonProps> = ({ onAddStream })
                 <ModalBody className={styles.body}>
                   <BuilderField
                     path="streamName"
-                    type="text"
+                    type="string"
                     label={formatMessage({ id: "connectorBuilder.addStreamModal.streamNameLabel" })}
                     tooltip={formatMessage({ id: "connectorBuilder.addStreamModal.streamNameTooltip" })}
                   />
                   <BuilderField
                     path="urlPath"
-                    type="text"
+                    type="string"
                     label={formatMessage({ id: "connectorBuilder.addStreamModal.urlPathLabel" })}
                     tooltip={formatMessage({ id: "connectorBuilder.addStreamModal.urlPathTooltip" })}
                   />

diff --git a/airbyte-webapp/src/components/connectorBuilder/Builder/AuthenticationSection.tsx b/airbyte-webapp/src/components/connectorBuilder/Builder/AuthenticationSection.tsx
@@ -0,0 +1,179 @@
+import { BuilderCard } from "./BuilderCard";
+import { BuilderField } from "./BuilderField";
+import { BuilderOneOf } from "./BuilderOneOf";
+import { BuilderOptional } from "./BuilderOptional";
+import { KeyValueListField } from "./KeyValueListField";
+import { UserInputField } from "./UserInputField";
+
+export const AuthenticationSection: React.FC = () => {
+  return (
+    <BuilderCard>
+      <BuilderOneOf
+        path="global.authenticator"
+        label="Authentication"
+        tooltip="Authentication method to use for requests sent to the API"
+        options={[
+          { label: "No Auth", typeValue: "NoAuth" },
+          {
+            label: "API Key",
+            typeValue: "ApiKeyAuthenticator",
+            default: {
+              api_token: "{{ config['api_key'] }}",
+              header: "",
+            },
+            children: (
+              <>
+                <BuilderField
+                  type="string"
+                  path="global.authenticator.header"
+                  label="Header"
+                  tooltip="HTTP header which should be set to the API Key"
+                />
+                <UserInputField
+                  label="API Key"
+                  tooltip="The API key issued by the service. Fill it in in the user inputs"
+                />
+              </>
+            ),
+          },
+          {
+            label: "Bearer",
+            typeValue: "BearerAuthenticator",
+            default: {
+              api_token: "{{ config['api_key'] }}",
+            },
+            children: (
+              <UserInputField
+                label="API Key"
+                tooltip="The API key issued by the service. Fill it in in the user inputs"
+              />
+            ),
+          },
+          {
+            label: "Basic HTTP",
+            typeValue: "BasicHttpAuthenticator",
+            default: {
+              username: "{{ config['username'] }}",
+              password: "{{ config['password'] }}",
+            },
+            children: (
+              <>
+                <UserInputField label="Username" tooltip="The username for the login. Fill it in in the user inputs" />
+                <UserInputField label="Password" tooltip="The password for the login. Fill it in in the user inputs" />
+              </>
+            ),
+          },
+          {
+            label: "OAuth",
+            typeValue: "OAuthAuthenticator",
+            default: {
+              client_id: "{{ config['client_id'] }}",
+              client_secret: "{{ config['client_secret'] }}",
+              refresh_token: "{{ config['client_refresh_token'] }}",
+              refresh_request_body: [],
+              token_refresh_endpoint: "",
+            },
+            children: (
+              <>
+                <BuilderField
+                  type="string"
+                  path="global.authenticator.token_refresh_endpoint"
+                  label="Token refresh endpoint"
+                  tooltip="The URL to call to obtain a new access token"
+                />
+                <UserInputField label="Client ID" tooltip="The OAuth client ID" />
+                <UserInputField label="Client secret" tooltip="The OAuth client secret" />
+                <UserInputField label="Refresh token" tooltip="The OAuth refresh token" />
+                <BuilderOptional>
+                  <BuilderField
+                    type="array"
+                    path="global.authenticator.scopes"
+                    optional
+                    label="Scopes"
+                    tooltip="Scopes to request"
+                  />
+                  <BuilderField
+                    type="string"
+                    path="global.authenticator.token_expiry_date_format"
+                    optional
+                    label="Token expiry date format"
+                    tooltip="The format of the expiry date of the access token as obtained from the refresh endpoint"
+                  />
+                  <BuilderField
+                    type="string"
+                    path="global.authenticator.expires_in_name"
+                    optional
+                    label="Token expiry property name"
+                    tooltip="The name of the property which contains the token exipiry date in the response from the token refresh endpoint"
+                  />
+                  <BuilderField
+                    type="string"
+                    path="global.authenticator.access_token_name"
+                    optional
+                    label="Access token property name"
+                    tooltip="The name of the property which contains the access token in the response from the token refresh endpoint"
+                  />
+                  <BuilderField
+                    type="string"
+                    path="global.authenticator.grant_type"
+                    optional
+                    label="Grant type"
+                    tooltip="The grant type to request for access_token"
+                  />
+                  <KeyValueListField
+                    path="global.authenticator.refresh_request_body"
+                    label="Request Parameters"
+                    tooltip="The request body to send in the refresh request"
+                  />
+                </BuilderOptional>
+              </>
+            ),
+          },
+          {
+            label: "Session token",
+            typeValue: "SessionTokenAuthenticator",
+            default: {
+              username: "{{ config['username'] }}",
+              password: "{{ config['password'] }}",
+              session_token: "{{ config['session_token'] }}",
+            },
+            children: (
+              <>
+                <BuilderField
+                  type="string"
+                  path="global.authenticator.header"
+                  label="Header"
+                  tooltip="Specific HTTP header of source API for providing session token"
+                />
+                <BuilderField
+                  type="string"
+                  path="global.authenticator.session_token_response_key"
+                  label="Session token response key"
+                  tooltip="Key for retrieving session token from api response"
+                />
+                <BuilderField
+                  type="string"
+                  path="global.authenticator.login_url"
+                  label="Login url"
+                  tooltip="Url for getting a specific session token"
+                />
+                <BuilderField
+                  type="string"
+                  path="global.authenticator.validate_session_url"
+                  label="Validate session url"
+                  tooltip="Url to validate passed session token"
+                />
+                <UserInputField label="Username" tooltip="The username" />
+                <UserInputField label="Password" tooltip="The password" />
+                <UserInputField
+                  label="Session token"
+                  tooltip="Session token generated by user (if provided username and password are not required)"
+                />
+              </>
+            ),
+          },
+        ]}
+      />
+    </BuilderCard>
+  );
+};
diff --git a/airbyte-webapp/src/components/connectorBuilder/Builder/Builder.module.scss b/airbyte-webapp/src/components/connectorBuilder/Builder/Builder.module.scss
@@ -18,4 +18,5 @@
 .form {
   flex: 1;
   padding: variables.$spacing-xl;
+  overflow: auto;
 }