Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🪟🧹 Connector builder: Fast fields performance improvements #20957

Merged
merged 118 commits into from
Jan 6, 2023
Merged
Show file tree
Hide file tree
Changes from 109 commits
Commits
Show all changes
118 commits
Select commit Hold shift + click to select a range
d2ed762
improve some types
Dec 5, 2022
f474a1c
Merge remote-tracking branch 'upstream/master' into chore-improve-con…
Dec 5, 2022
5b0521f
improve further
Dec 5, 2022
af659fd
Merge remote-tracking branch 'upstream/master' into chore-improve-con…
Dec 6, 2022
1cd3bbb
clean up a bit more
Dec 6, 2022
807c436
refactor loading state
Dec 6, 2022
6c76e0f
Merge remote-tracking branch 'upstream/master' into flash1293/connect…
Dec 6, 2022
05c3107
move loading state up
Dec 6, 2022
eee4303
remove isLoading references
Dec 6, 2022
7d6f977
remove unused props and make fetch connector error work
Dec 6, 2022
0fe11f0
remove special component for name
Dec 6, 2022
34d4814
remove top level state for unifinished flows
Dec 6, 2022
67d62c5
Merge remote-tracking branch 'upstream/master' into flash1293/connect…
Dec 6, 2022
afea882
Merge branch 'flash1293/connector-form-name-override' into flash1293/…
Dec 6, 2022
c7e7475
Merge branch 'master' into flash1293/connector-form-loading-state
Dec 6, 2022
cc7f65f
Merge branch 'chore-improve-connector-form-types' into flash1293/conn…
Dec 6, 2022
a1c6cbd
start removing uiwidget
Dec 6, 2022
9be8337
Update airbyte-webapp/src/views/Connector/ConnectorCard/ConnectorCard…
Dec 6, 2022
37cee8f
Merge remote-tracking branch 'upstream/master' into flash1293/connect…
Dec 6, 2022
5686241
Merge branch 'flash1293/connector-form-loading-state' of github.com:a…
Dec 6, 2022
7b73364
remove undefined option for selected id
Dec 6, 2022
7058d26
Merge branch 'flash1293/connector-form-loading-state' into flash1293/…
Dec 6, 2022
dd7e82a
Merge branch 'flash1293/connector-form-name-override' into flash1293/…
Dec 6, 2022
6d4c7d3
Merge remote-tracking branch 'upstream/master' into flash1293/connect…
Dec 7, 2022
d624792
Merge branch 'flash1293/connector-form-loading-state' into flash1293/…
Dec 7, 2022
36cf941
Merge branch 'flash1293/connector-form-name-override' into flash1293/…
Dec 7, 2022
b7a6322
Merge remote-tracking branch 'upstream/master' into flash1293/connect…
Dec 7, 2022
72b602a
Merge branch 'flash1293/connector-form-loading-state' into flash1293/…
Dec 7, 2022
5e47a4a
Merge branch 'flash1293/connector-form-name-override' into flash1293/…
Dec 7, 2022
6c71028
Merge remote-tracking branch 'origin/master' into flash1293/connector…
Dec 8, 2022
809625f
remove unused prop
Dec 8, 2022
10446eb
Merge branch 'flash1293/connector-form-loading-state' into flash1293/…
Dec 8, 2022
4a7a885
Merge branch 'flash1293/connector-form-name-override' into flash1293/…
Dec 8, 2022
2024b60
Merge branch 'flash1293/connector-form-unifinished-flows' into flash1…
Dec 8, 2022
ee30092
fix types
Dec 8, 2022
d91df38
remove uiwidget state
Dec 8, 2022
93fe558
Merge remote-tracking branch 'origin/master' into flash1293/connector…
Dec 9, 2022
cbc023a
Merge branch 'flash1293/connector-form-name-override' into flash1293/…
Dec 9, 2022
ec54d38
Merge branch 'flash1293/connector-form-unifinished-flows' into flash1…
Dec 9, 2022
47152ea
clean up
Dec 9, 2022
6ec0e10
adjust comment
Dec 9, 2022
1aa7829
handle errors in a nice way
Dec 9, 2022
7b90088
do not respect default on oneOf fields
Dec 9, 2022
fbefbdd
rename to formblock
Dec 9, 2022
6b3e5e2
reduce re-renders
Dec 9, 2022
9aa3f80
Merge remote-tracking branch 'origin/master' into flash1293/connector…
Dec 12, 2022
c765003
Merge branch 'flash1293/connector-form-unifinished-flows' into flash1…
Dec 12, 2022
5cd77ec
pass error to secure inputs
Dec 12, 2022
5dfbc22
Merge remote-tracking branch 'origin/master' into flash1293/connector…
Dec 13, 2022
a6aa741
simplify and improve styling
Dec 13, 2022
95717d2
align top
Dec 13, 2022
40d162d
Merge branch 'flash1293/connector-form-unifinished-flows' into flash1…
Dec 13, 2022
b9384ee
Merge remote-tracking branch 'origin/master' into flash1293/connector…
Dec 13, 2022
582a5f2
code review
Dec 13, 2022
18d6ce8
remove comment
Dec 13, 2022
60c1f1c
Merge branch 'master' into flash1293/connector-form-remove-ui-widget-…
Dec 14, 2022
9cb595d
Merge remote-tracking branch 'origin/master' into flash1293/connector…
Dec 15, 2022
fe74a45
review comments
Dec 15, 2022
5c00ad0
rename file
Dec 15, 2022
bd0dffc
be strict about boolean values
Dec 15, 2022
1ff8c13
add example
Dec 15, 2022
c6ee526
track form error in error boundary
Dec 15, 2022
05bbd6f
Merge branch 'flash1293/connector-form-remove-ui-widget-state' of git…
Dec 15, 2022
fff4703
Merge remote-tracking branch 'origin/master' into flash1293/connector…
Dec 16, 2022
94ba866
review comments
Dec 16, 2022
97bb0a7
handle unexpected cases better
Dec 16, 2022
633b9b2
Merge branch 'master' into flash1293/connector-form-remove-ui-widget-…
Dec 16, 2022
e47517f
Merge branch 'master' into flash1293/connector-form-remove-ui-widget-…
Dec 16, 2022
e931956
Merge remote-tracking branch 'origin/master' into feature/connector-b…
Dec 16, 2022
219fcc1
speed up some bits
Dec 17, 2022
8cb6873
more changes
Dec 17, 2022
6889f66
enrich error with connector id
Dec 19, 2022
7b60406
Merge branch 'flash1293/connector-form-remove-ui-widget-state' into f…
Dec 19, 2022
d5e56eb
Merge branch 'master' into flash1293/connector-form-remove-ui-widget-…
Dec 19, 2022
35a74ba
Merge branch 'flash1293/connector-form-remove-ui-widget-state' into f…
Dec 19, 2022
02c8096
🪟🎉 Add copy stream button (#20577)
Dec 19, 2022
526adfc
rename prop
Dec 20, 2022
7d49233
🪟🎉 Connector builder: Integrate connector form for test input (#20385)
Dec 20, 2022
c5da764
Merge remote-tracking branch 'origin/master' into flash1293/connector…
Dec 20, 2022
6c0bf9d
Merge branch 'flash1293/connector-form-remove-ui-widget-state' into f…
Dec 20, 2022
477977f
use request_body_json instead of request_body_data
lmossman Dec 20, 2022
200cfe7
:window: :art: Move `Add` button into the line of Connector Builder k…
lmossman Dec 21, 2022
c7fd310
🪟🎉 Connector builder: Allow defining inputs (#20431)
Dec 22, 2022
e0e3796
fix merge conflict with dropdown prop being renamed to control
Dec 22, 2022
e23a6d8
[Connector Builder] Add paginator (#20698)
lmossman Dec 22, 2022
610e668
[Connector Builder] Add stream slicer (#20748)
lmossman Dec 22, 2022
e68ff49
debounce form builder values update to reduce load
Dec 22, 2022
9ccf294
Merge branch 'master' into flash1293/connector-form-remove-ui-widget-…
Dec 22, 2022
ae9a229
Merge branch 'flash1293/connector-form-remove-ui-widget-state' into f…
Dec 22, 2022
a9e1d09
🪟🔧 Connector builder: use new lowcode manifest (#20715)
Dec 22, 2022
e4f7bba
debounce validation as well
Dec 27, 2022
16c71ae
Merge branch 'master' into flash1293/connector-form-remove-ui-widget-…
Jan 2, 2023
54c070d
Merge branch 'flash1293/connector-form-remove-ui-widget-state' into f…
Jan 2, 2023
2e20fa6
akways show stream test button in error state if there are errors
Jan 2, 2023
65199d0
fix type of oauth input
Jan 2, 2023
6b71161
Merge branch 'feature/connector-builder' into flash1293/performance-work
Jan 2, 2023
81c12b2
review comments
Jan 2, 2023
b447b24
fix more
Jan 2, 2023
92722e0
Merge branch 'master' into flash1293/connector-form-remove-ui-widget-…
Jan 3, 2023
ef86690
start implementing fast fields
Jan 3, 2023
718c233
Merge branch 'flash1293/connector-form-remove-ui-widget-state' into f…
Jan 3, 2023
26acc32
Merge branch 'feature/connector-builder' into flash1293/performance-work
Jan 3, 2023
6a8be0b
Merge remote-tracking branch 'origin/master' into feature/connector-b…
Jan 3, 2023
e166b03
Merge branch 'feature/connector-builder' into flash1293/performance-work
Jan 3, 2023
4b4eb7e
Merge branch 'flash1293/performance-work' into flash1293/fast-fields
Jan 3, 2023
a4d35be
a few more fastfields
Jan 3, 2023
54d744b
Merge remote-tracking branch 'origin/master' into flash1293/fast-fields
Jan 6, 2023
f5818e6
fix some stuff
Jan 6, 2023
5dc707b
complete fast fields work
Jan 6, 2023
742538e
memoize a bit more
Jan 6, 2023
b2b03bf
remove irrelevant change
Jan 6, 2023
4ce7811
fix types
Jan 6, 2023
0d5fcc5
some more improvements
Jan 6, 2023
98a33cf
prevent all out-of-line validation calls
Jan 6, 2023
7e08ea7
Merge branch 'master' into flash1293/fast-fields
Jan 6, 2023
80ec5c6
Merge branch 'master' into flash1293/fast-fields
Jan 6, 2023
8c33c36
Merge branch 'master' into flash1293/fast-fields
Jan 6, 2023
f996b6d
use derivedJsonManifest when in UI view
lmossman Jan 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { useField } from "formik";
import { FastField } from "formik";
import { ReactNode } from "react";
import { FormattedMessage } from "react-intl";

Expand Down Expand Up @@ -64,17 +64,19 @@ const ArrayField: React.FC<ArrayFieldProps> = ({ name, value, setValue, error })
return <TagInput name={name} fieldValue={value} onChange={(value) => setValue(value)} error={error} />;
};

export const BuilderField: React.FC<BuilderFieldProps> = ({
const InnerBuilderField: React.FC<BuilderFieldProps & { field: any; helpers: any; meta: any }> = ({
path,
label,
tooltip,
optional = false,
readOnly,
pattern,
field,
meta,
helpers,
adornment,
...props
}) => {
const [field, meta, helpers] = useField(path);
const hasError = !!meta.error && meta.touched;

if (props.type === "boolean") {
Expand Down Expand Up @@ -133,3 +135,11 @@ export const BuilderField: React.FC<BuilderFieldProps> = ({
</ControlLabels>
);
};

export const BuilderField: React.FC<BuilderFieldProps> = (props) => {
return (
<FastField name={props.path}>
{({ field, helpers, meta }: any) => <InnerBuilderField {...props} field={field} helpers={helpers} meta={meta} />}
</FastField>
);
};
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { faPlus, faUser } from "@fortawesome/free-solid-svg-icons";
import { FontAwesomeIcon } from "@fortawesome/react-fontawesome";
import { useField } from "formik";
import { useMemo, useState } from "react";
import React from "react";
import { FormattedMessage, useIntl } from "react-intl";

import { ListBox, ListBoxControlButtonProps, Option } from "components/ui/ListBox";
Expand All @@ -26,20 +27,18 @@ export const BuilderFieldWithInputs: React.FC<BuilderFieldProps> = (props) => {
);
};

export const UserInputHelper = ({
setValue,
currentValue,
}: {
interface UserInputHelperProps {
setValue: (value: string) => void;
currentValue: string;
}) => {
}

export const UserInputHelper = (props: UserInputHelperProps) => {
const { formatMessage } = useIntl();
const [modalOpen, setModalOpen] = useState(false);
const { builderFormValues } = useConnectorBuilderFormState();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't get rid of subscribing to the builder form state, but we can memoize building the actual options list and memoize the actual listbox component based on that.

const listOptions = useMemo(() => {
const options: Array<Option<string | undefined>> = [
...builderFormValues.inputs,
...getInferredInputs(builderFormValues),
...getInferredInputs(builderFormValues.global, builderFormValues.inferredInputOverrides),
].map((input) => ({
label: input.definition.title || input.key,
value: input.key,
Expand All @@ -50,45 +49,56 @@ export const UserInputHelper = ({
icon: <FontAwesomeIcon icon={faPlus} />,
});
return options;
}, [builderFormValues, formatMessage]);
return (
<>
<ListBox<string | undefined>
buttonClassName={styles.button}
optionClassName={styles.option}
className={styles.container}
selectedOptionClassName={styles.selectedOption}
controlButton={UserInputHelperControlButton}
selectedValue={undefined}
onSelect={(selectedValue) => {
if (selectedValue) {
setValue(`${currentValue || ""}{{ config['${selectedValue}'] }}`);
} else {
// This hack is necessary because listbox will put the focus back when the option list gets hidden, which conflicts with the auto-focus setting of the modal.
// As it's not possible to prevent listbox from forcing the focus back on the button component, this will wait until the focus went to the button, then opens the modal
// so it can move it to the first input
setTimeout(() => {
setModalOpen(true);
}, 50);
}
}}
options={listOptions}
/>
{modalOpen && (
<InputForm
inputInEditing={newInputInEditing()}
onClose={(newInput) => {
setModalOpen(false);
if (!newInput) {
return;
}, [builderFormValues.global, builderFormValues.inferredInputOverrides, builderFormValues.inputs, formatMessage]);
return <InnerUserInputHelper {...props} listOptions={listOptions} />;
};

const InnerUserInputHelper = React.memo(
({
setValue,
currentValue,
listOptions,
}: UserInputHelperProps & { listOptions: Array<Option<string | undefined>> }) => {
const [modalOpen, setModalOpen] = useState(false);
return (
<>
<ListBox<string | undefined>
buttonClassName={styles.button}
optionClassName={styles.option}
className={styles.container}
selectedOptionClassName={styles.selectedOption}
controlButton={UserInputHelperControlButton}
selectedValue={undefined}
onSelect={(selectedValue) => {
if (selectedValue) {
setValue(`${currentValue || ""}{{ config['${selectedValue}'] }}`);
} else {
// This hack is necessary because listbox will put the focus back when the option list gets hidden, which conflicts with the auto-focus setting of the modal.
// As it's not possible to prevent listbox from forcing the focus back on the button component, this will wait until the focus went to the button, then opens the modal
// so it can move it to the first input
setTimeout(() => {
setModalOpen(true);
}, 50);
}
setValue(`${currentValue}{{ config['${newInput.key}'] }}`);
}}
options={listOptions}
/>
)}
</>
);
};
{modalOpen && (
<InputForm
inputInEditing={newInputInEditing()}
onClose={(newInput) => {
setModalOpen(false);
if (!newInput) {
return;
}
setValue(`${currentValue}{{ config['${newInput.key}'] }}`);
}}
/>
)}
</>
);
}
);

const UserInputHelperControlButton: React.FC<ListBoxControlButtonProps<string | undefined>> = () => {
return (
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { FastField } from "formik";
import { useField } from "formik";
import React from "react";

Expand Down Expand Up @@ -25,38 +26,46 @@ interface BuilderOneOfProps {
tooltip: string;
}

export const BuilderOneOf: React.FC<BuilderOneOfProps> = ({ options, path, label, tooltip }) => {
const [, , oneOfPathHelpers] = useField(path);
const typePath = `${path}.type`;
const [typePathField] = useField(typePath);
const value = typePathField.value;
const InnerBuilderOneOf: React.FC<BuilderOneOfProps & { field: any; helpers: any }> = React.memo(
({ options, label, tooltip, field: typePathField, helpers: oneOfPathHelpers }) => {
const value = typePathField.value;

const selectedOption = options.find((option) => option.typeValue === value);
const selectedOption = options.find((option) => option.typeValue === value);

return (
<GroupControls
label={<ControlLabels label={label} infoTooltipContent={tooltip} />}
control={
<DropDown
{...typePathField}
options={options.map((option) => {
return { label: option.label, value: option.typeValue, default: option.default };
})}
value={value ?? options[0].typeValue}
onChange={(selectedOption: Option) => {
if (selectedOption.value === value) {
return;
}
// clear all values for this oneOf and set selected option and default values
oneOfPathHelpers.setValue({
type: selectedOption.value,
...selectedOption.default,
});
}}
/>
}
>
{selectedOption?.children}
</GroupControls>
);
}
);

export const BuilderOneOf: React.FC<BuilderOneOfProps> = (props) => {
const [, , helpers] = useField(props.path);
return (
<GroupControls
label={<ControlLabels label={label} infoTooltipContent={tooltip} />}
control={
<DropDown
{...typePathField}
options={options.map((option) => {
return { label: option.label, value: option.typeValue, default: option.default };
})}
value={value ?? options[0].typeValue}
onChange={(selectedOption: Option) => {
if (selectedOption.value === value) {
return;
}
// clear all values for this oneOf and set selected option and default values
oneOfPathHelpers.setValue({
type: selectedOption.value,
...selectedOption.default,
});
}}
/>
}
>
{selectedOption?.children}
</GroupControls>
<FastField name={`${props.path}.type`}>
{({ field }: any) => <InnerBuilderOneOf {...props} field={field} helpers={helpers} />}
</FastField>
);
};
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,9 @@ export const BuilderSidebar: React.FC<BuilderSidebarProps> = React.memo(({ class
<FontAwesomeIcon icon={faUser} />
<FormattedMessage
id="connectorBuilder.userInputs"
values={{ number: values.inputs.length + getInferredInputs(values).length }}
values={{
number: values.inputs.length + getInferredInputs(values.global, values.inferredInputOverrides).length,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason the getInferredInputs result is not being memoized in this component?

}}
/>
</ViewSelectButton>

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import React from "react";
import { useIntl } from "react-intl";

import { AuthenticationSection } from "./AuthenticationSection";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,10 @@ export const InputForm = ({
}) => {
const { values, setFieldValue } = useFormikContext<BuilderFormValues>();
const [inputs, , helpers] = useField<BuilderFormInput[]>("inputs");
const inferredInputs = useMemo(() => getInferredInputs(values), [values]);
const inferredInputs = useMemo(
() => getInferredInputs(values.global, values.inferredInputOverrides),
[values.global, values.inferredInputOverrides]
);
const usedKeys = useMemo(
() => [...inputs.value, ...inferredInputs].map((input) => input.key),
[inputs.value, inferredInputs]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@ export const InputsView: React.FC = () => {
const { formatMessage } = useIntl();
const { values } = useFormikContext<BuilderFormValues>();
const [inputInEditing, setInputInEditing] = useState<InputInEditing | undefined>(undefined);
const inferredInputs = useMemo(() => getInferredInputs(values), [values]);
const inferredInputs = useMemo(
() => getInferredInputs(values.global, values.inferredInputOverrides),
[values.global, values.inferredInputOverrides]
);

return (
<BuilderConfigView heading={formatMessage({ id: "connectorBuilder.inputsTitle" })}>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { useField } from "formik";
import React, { useRef } from "react";
import { FormattedMessage } from "react-intl";

import GroupControls from "components/GroupControls";
Expand Down Expand Up @@ -60,6 +61,10 @@ interface KeyValueListFieldProps {
export const KeyValueListField: React.FC<KeyValueListFieldProps> = ({ path, label, tooltip }) => {
const [{ value: keyValueList }, , { setValue: setKeyValueList }] = useField<Array<[string, string]>>(path);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't get rid of the "useField" here, but we can memoize everything happening within the component if the key value list didn't actually change

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't we use a FastField here?


// need to wrap the setter into a ref because it will be a new function on every formik state update
const setKeyValueListRef = useRef(setKeyValueList);
setKeyValueListRef.current = setKeyValueList;

return (
<GroupControls
label={<ControlLabels label={label} infoTooltipContent={tooltip} />}
Expand All @@ -69,20 +74,36 @@ export const KeyValueListField: React.FC<KeyValueListFieldProps> = ({ path, labe
</Button>
}
>
{keyValueList.map((keyValue, keyValueIndex) => (
<KeyValueInput
key={keyValueIndex}
keyValue={keyValue}
onChange={(newKeyValue) => {
const updatedList = keyValueList.map((entry, index) => (index === keyValueIndex ? newKeyValue : entry));
setKeyValueList(updatedList);
}}
onRemove={() => {
const updatedList = keyValueList.filter((_, index) => index !== keyValueIndex);
setKeyValueList(updatedList);
}}
/>
))}
<KeyValueList keyValueList={keyValueList} setKeyValueList={setKeyValueListRef} />
</GroupControls>
);
};

const KeyValueList = React.memo(
({
keyValueList,
setKeyValueList,
}: {
keyValueList: Array<[string, string]>;
setKeyValueList: React.MutableRefObject<(val: Array<[string, string]>) => void>;
}) => {
return (
<>
{keyValueList.map((keyValue, keyValueIndex) => (
<KeyValueInput
key={keyValueIndex}
keyValue={keyValue}
onChange={(newKeyValue) => {
const updatedList = keyValueList.map((entry, index) => (index === keyValueIndex ? newKeyValue : entry));
setKeyValueList.current(updatedList);
}}
onRemove={() => {
const updatedList = keyValueList.filter((_, index) => index !== keyValueIndex);
setKeyValueList.current(updatedList);
}}
/>
))}
</>
);
}
);
Loading