Skip to content

Commit

Permalink
🪟🎉 Connector builder: Substream slicer and cartesian slicer (#20861)
Browse files Browse the repository at this point in the history
* improve some types

* improve further

* clean up a bit more

* refactor loading state

* move loading state up

* remove isLoading references

* remove unused props and make fetch connector error work

* remove special component for name

* remove top level state for unifinished flows

* start removing uiwidget

* Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard.module.scss

Co-authored-by: Tim Roes <tim@airbyte.io>

* remove undefined option for selected id

* remove unused prop

* fix types

* remove uiwidget state

* clean up

* adjust comment

* handle errors in a nice way

* do not respect default on oneOf fields

* rename to formblock

* reduce re-renders

* pass error to secure inputs

* simplify and improve styling

* align top

* code review

* remove comment

* review comments

* rename file

* be strict about boolean values

* add example

* track form error in error boundary

* review comments

* handle unexpected cases better

* enrich error with connector id

* 🪟🎉 Add copy stream button (#20577)

* add copy stream button

* review comments

* rename prop

* 🪟🎉 Connector builder: Integrate connector form for test input (#20385)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* fix small stuff

* add warning label

* review comments

* adjust translation

Co-authored-by: lmossman <lake@airbyte.io>

* use request_body_json instead of request_body_data

* 🪟 🎨 Move `Add` button into the line of Connector Builder key value list fields (#20699)

* move add button into line

* add stories for empty with control, and content + control

* change button name to Control

* 🪟🎉 Connector builder: Allow defining inputs (#20431)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* handle stored form values that don't contain new fields properly

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <lake@airbyte.io>

* inputs editing weirdness

* input form reset

* using the Label component

* 🪟🎉 Connector builder authentication (#20645)

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* fix keys

* 🪟🎉 Connector builder: Session token and oauth authentication (#20712)

* session token and oauth authentication

* fill in session token variable

* typos

* make sure validation error does not go away

* 🪟🎉 Connector builder: Always validate inputs form (#20664)

* validate user input outside of form

* review comments

Co-authored-by: lmossman <lake@airbyte.io>

Co-authored-by: lmossman <lake@airbyte.io>

* fix merge conflict with dropdown prop being renamed to control

* [Connector Builder] Add paginator (#20698)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* handle stored form values that don't contain new fields properly

* session token and oauth authentication

* fill in session token variable

* fix merge of default values

* add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts

* add cursor and primary key fields to ui

* save

* add page size and token option inputs

* fixes after rebase

* add pagination

* fix pagination types

* handle empty field_name better

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <lake@airbyte.io>

* inputs editing weirdness

* input form reset

* using the Label component

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* typos

* make sure validation error does not go away

* make primary key and cursor optional, and reorder

* save toggle group progress

* fix style of toggle label

* handle empty values better

* fix page size/token option field validation and rendering

* handle cursor pagination page size option correctly

Co-authored-by: Joe Reuter <joe@airbyte.io>

* [Connector Builder] Add stream slicer (#20748)

* move connector builder components into the same shared components/connectorBuilder directory

* move diff over from poc branch

* save current progress

* add modal for adding streams

* focus stream after adding and reset button style

* add reset confirm modal and select view on add

* style global config and streams buttons

* styling improvements

* handle long stream names better

* pull in connector manifest schema directly

* add box shadows to resizable panels

* upgrade orval and use connector manifest schema directly

* remove airbyte protocol from connector builder api spec

* generate python models from openapi change

* fix position of yaml toggle

* handle no stream case with better looking message

* group global fields into single object and fix console error

* confirmation modal on toggling dirty form + cleanup

* fix connector name display

* undo change to manifest schema

* remove commented code

* remove unnecessary change

* fix spacing

* use shadow mixin for connector img

* add comment about connector img

* change onSubmit to no-op

* remove console log

* clean up styling

* simplify sidebar to remove StreamSelectButton component

* swap colors of toggle

* move FormikPatch to src/core/form

* move types up to connectorBuilder/ level

* use grid display for ui yaml toggle button

* use spread instead of setting array index directly

* add intl in missing places

* pull connector manifest schema in through separate openapi spec

* use correct intl string id

* throttle setting json manifest in yaml editor

* use  button prop instead of manually styling

* consolidate AddStreamButton styles

* fix sidebar flex styles

* use specific flex properties instead of flex

* clean up download and reset button styles

* use row-reverse for yaml editor download button

* fix stream selector styles to remove margins

* give connector setup guide panel same corner and shadow styles

* remove blur from page display

* set view to stream when selected in test panel

* add placeholder when stream name is empty

* switch to index-based stream selection to preserve testing panel selected stream on rename

* handle empty name in stream selector

* make connector form work in connector builder

* wip

* fix small stuff

* add basic input UI

* user inputs

* make most of inputs configuration work

* fix a bunch of stuff

* handle unknown config types

* add warning label

* fix label

* fix some styling

* review comments

* improve state management and error handling

* allow auth configuration

* check for conflicts with the inferred inputs

* fix invisible inputs

* handle stored form values that don't contain new fields properly

* session token and oauth authentication

* fill in session token variable

* fix merge of default values

* add primaryKey and cursorField to builder types, and consolidate default valeues to types.ts

* add cursor and primary key fields to ui

* save

* add page size and token option inputs

* fixes after rebase

* add pagination

* fix pagination types

* handle empty field_name better

* Update airbyte-webapp/src/locales/en.json

Co-authored-by: Lake Mossman <lake@airbyte.io>

* Update airbyte-webapp/src/components/connectorBuilder/Builder/InputsView.tsx

Co-authored-by: Lake Mossman <lake@airbyte.io>

* inputs editing weirdness

* input form reset

* using the Label component

* reduce redundancy and hide advanced input options for inferred inputs

* unnecessary validation

* typo

* unnecessary effect hook

* build spec even for invalid forms but do not update stream list

* typos

* make sure validation error does not go away

* make primary key and cursor optional, and reorder

* save toggle group progress

* fix style of toggle label

* handle empty values better

* fix page size/token option field validation and rendering

* handle cursor pagination page size option correctly

* save stream slicer progress

* finish stream slicer

* fix stream slicer fields and validation

Co-authored-by: Joe Reuter <joe@airbyte.io>

* debounce form builder values update to reduce load

* 🪟🔧  Connector builder: use new lowcode manifest (#20715)

* use new manifest yaml

* Update airbyte-webapp/src/components/connectorBuilder/types.ts

Co-authored-by: Lake Mossman <lake@airbyte.io>

* use updated manifest types

Co-authored-by: Lake Mossman <lake@airbyte.io>

* add substream slicer

* add substream and cartesian slicer

* debounce validation as well

* akways show stream test button in error state if there are errors

* fix type of oauth input

* add validation schema for add stream form

* validate all views on test click

* add type to prevent console warning

* do not allow path for substream slicer request option

* do not show request option for substream slicer

* rewrite stream slice field tooltip

Co-authored-by: Tim Roes <tim@airbyte.io>
Co-authored-by: lmossman <lake@airbyte.io>
  • Loading branch information
3 people authored Jan 5, 2023
1 parent eeedd64 commit db3d398
Show file tree
Hide file tree
Showing 14 changed files with 602 additions and 257 deletions.
41 changes: 33 additions & 8 deletions airbyte-webapp/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions airbyte-webapp/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
"@sentry/tracing": "^6.19.6",
"@tanstack/react-table": "^8.7.0",
"@types/segment-analytics": "^0.0.34",
"@types/uuid": "^9.0.0",
"classnames": "^2.3.1",
"dayjs": "^1.11.3",
"firebase": "^9.8.2",
Expand Down Expand Up @@ -80,6 +81,7 @@
"styled-components": "^5.3.5",
"typesafe-actions": "^5.1.0",
"unist-util-visit": "^4.1.0",
"uuid": "^9.0.0",
"yup": "^0.32.11"
},
"devDependencies": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import merge from "lodash/merge";
import { useState } from "react";
import React from "react";
import { FormattedMessage, useIntl } from "react-intl";
import { v4 as uuid } from "uuid";
import * as yup from "yup";

import { Button } from "components/ui/Button";
Expand Down Expand Up @@ -55,6 +56,7 @@ export const AddStreamButton: React.FC<AddStreamButtonProps> = ({ onAddStream, b
...initialValues,
name: values.streamName,
urlPath: values.urlPath,
id: uuid(),
}),
]);
setIsOpen(false);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
@use "scss/variables";
@use "scss/colors";

$removeButtonWidth: 20px;

.itemWrapper {
display: flex;
gap: variables.$spacing-md;
}

.itemContainer {
flex-grow: 1;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import { faPlus } from "@fortawesome/free-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { useField } from "formik";
import React, { ReactElement, useMemo } from "react";
import { FormattedMessage } from "react-intl";

import { Button } from "components/ui/Button";

import styles from "./BuilderList.module.scss";
import { RemoveButton } from "./RemoveButton";

interface BuilderListProps {
children: (props: { buildPath: (path: string) => string }) => ReactElement;
basePath: string;
emptyItem: object;
}

export const BuilderList: React.FC<BuilderListProps> = ({ children, emptyItem, basePath }) => {
const [list, , helpers] = useField<object[]>(basePath);

const buildPathFunctions = useMemo(
() =>
new Array(list.value.length).fill(undefined).map((_value, index) => {
return (path: string) => `${basePath}[${index}]${path !== "" ? "." : ""}${path}`;
}),
[basePath, list.value.length]
);

return (
<>
{buildPathFunctions.map((buildPath, currentItemIndex) => (
<div className={styles.itemWrapper} key={currentItemIndex}>
<div className={styles.itemContainer}>{children({ buildPath })}</div>
<RemoveButton
onClick={() => {
const updatedItems = list.value.filter((_, index) => index !== currentItemIndex);
helpers.setValue(updatedItems);
}}
/>
</div>
))}
<div>
<Button
variant="secondary"
icon={<FontAwesomeIcon icon={faPlus} />}
onClick={() => {
helpers.setValue([...list.value, { ...emptyItem }]);
}}
>
<FormattedMessage id="connectorBuilder.addNewSlicer" />
</Button>
</div>
</>
);
};
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ interface Option {
default?: object;
}

interface OneOfOption {
export interface OneOfOption {
label: string; // label shown in the dropdown menu
typeValue: string; // value to set on the `type` field for this component - should match the oneOf type definition
default?: object; // default values for the path
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,3 @@ $removeButtonWidth: 20px;
.kvLabel {
color: colors.$grey-400;
}

.removeButton {
border: none;
background-color: transparent;
color: colors.$grey;
cursor: pointer;
padding: 0;
width: $removeButtonWidth;
height: $removeButtonWidth;
font-size: 18px;
transition: variables.$transition;

&:hover {
color: colors.$red;
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import { faXmark } from "@fortawesome/free-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { useField } from "formik";
import { FormattedMessage } from "react-intl";

Expand All @@ -10,6 +8,7 @@ import { Input } from "components/ui/Input";
import { Text } from "components/ui/Text";

import styles from "./KeyValueListField.module.scss";
import { RemoveButton } from "./RemoveButton";

interface KeyValueInputProps {
keyValue: [string, string];
Expand All @@ -32,9 +31,7 @@ const KeyValueInput: React.FC<KeyValueInputProps> = ({ keyValue, onChange, onRem
</Text>
<Input value={keyValue[1]} onChange={(e) => onChange([keyValue[0], e.target.value])} />
</div>
<button type="button" className={styles.removeButton} onClick={onRemove}>
<FontAwesomeIcon icon={faXmark} size="1x" />
</button>
<RemoveButton onClick={onRemove} />
</div>
);
};
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
@use "scss/variables";
@use "scss/colors";

$removeButtonWidth: 20px;

.removeButton {
border: none;
background-color: transparent;
color: colors.$grey;
cursor: pointer;
padding: 0;
width: $removeButtonWidth;
height: $removeButtonWidth;
font-size: 18px;
transition: variables.$transition;

&:hover {
color: colors.$red;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import { faXmark } from "@fortawesome/free-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";

import styles from "./RemoveButton.module.scss";

export const RemoveButton = ({ onClick }: { onClick: () => void }) => {
return (
<button className={styles.removeButton} onClick={onClick}>
<FontAwesomeIcon icon={faXmark} />
</button>
);
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
import { useField } from "formik";
import { useMemo } from "react";
import { FormattedMessage } from "react-intl";

import { ControlLabels } from "components/LabeledControl";
import { DropDown } from "components/ui/DropDown";
import { Text } from "components/ui/Text";

import { BuilderStream } from "../types";
import styles from "./BuilderField.module.scss";

interface StreamReferenceFieldProps {
// path to the location in the Connector Manifest schema which should be set by this component
path: string;
label: string;
tooltip?: string;
optional?: boolean;
currentStreamIndex: number;
}

export const StreamReferenceField: React.FC<StreamReferenceFieldProps> = ({
path,
label,
tooltip,
optional,
currentStreamIndex,
...props
}) => {
const [streams] = useField<BuilderStream[]>("streams");
const [field, meta, helpers] = useField(path);
const hasError = !!meta.error && meta.touched;

const options = useMemo(() => {
return streams.value
.filter((_value, index) => index !== currentStreamIndex)
.map((stream) => ({
value: stream.id,
label: stream.name,
}));
}, [currentStreamIndex, streams.value]);

return (
<ControlLabels className={styles.container} label={label} infoTooltipContent={tooltip} optional={optional}>
<DropDown
{...props}
options={options}
onChange={(selected) => selected && helpers.setValue(selected.value)}
value={field.value}
error={hasError}
/>
{hasError && (
<Text className={styles.error}>
<FormattedMessage id={meta.error} />
</Text>
)}
</ControlLabels>
);
};
Loading

0 comments on commit db3d398

Please sign in to comment.