Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): Improvements/overhaul for nodes working with binary data #7651

Merged
merged 65 commits into from
Jan 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
d1e68b2
:zap: FilesReadWrite setup, support for input items loop when reading…
michael-radency Nov 8, 2023
1e49662
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 8, 2023
6a48c58
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 8, 2023
f3a3360
:zap: filesReadWrite ui update
michael-radency Nov 9, 2023
ff4cc88
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 9, 2023
e9695b5
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 13, 2023
8fe8eb9
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 13, 2023
3ab796a
:zap: Binary Data => Binary File, aliases update
michael-radency Nov 14, 2023
c36ee23
:zap: Binary Property => File Property
michael-radency Nov 14, 2023
34c74c6
:zap: convertToFile, extractFromFile nodes setup
michael-radency Nov 14, 2023
fab7c9e
:zap: extracted convertJsonToSpreadsheetBinary helper
michael-radency Nov 14, 2023
92a6078
:zap: ConvertToFile node, WIP
michael-radency Nov 14, 2023
38f2829
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 14, 2023
74ae802
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 14, 2023
373451f
:zap: ConvertToFile moveValueToBinar, iCall operations
michael-radency Nov 15, 2023
310c663
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 15, 2023
d0071f0
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 15, 2023
c794d63
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 15, 2023
3295498
:zap: Spreadsheet refactoring, integration fromFile into ExtractFrom …
michael-radency Nov 16, 2023
8acb5ec
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 16, 2023
2b2e777
:zap: binaryToPropery and fromJson operations
michael-radency Nov 16, 2023
7572168
:zap: extract from pdf, extract from text file
michael-radency Nov 17, 2023
9d57814
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 17, 2023
f046bcd
:zap: missing actions fix
michael-radency Nov 17, 2023
a9526de
:zap: added continueOnFail handling
michael-radency Nov 17, 2023
010adef
:zap: refactored ExtractFromFile
michael-radency Nov 17, 2023
244ed36
:zap: refactored ReadWriteFile, ConvertToFile
michael-radency Nov 17, 2023
0c5473c
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Nov 17, 2023
a33a6d8
Compress node copy tweaks and added alias.
gandreini Nov 28, 2023
412591e
FTP node added placeholder to fields and an alias.
gandreini Nov 28, 2023
18e8318
Read/write copy tweaks and temporary icon.
gandreini Nov 28, 2023
013ee60
Various copy tweaks and icons updates.
gandreini Dec 4, 2023
a4cdbe8
Various copy updates. Alphabetic sorting.
gandreini Dec 5, 2023
072798d
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Dec 6, 2023
90d4ac8
:zap: linter fixes
michael-radency Dec 6, 2023
53c0454
:zap: file property displayName and hint update
michael-radency Dec 6, 2023
c5bc6b9
:zap: convert to file node fixes
michael-radency Dec 6, 2023
d36326e
:zap: extract from file updates
michael-radency Dec 6, 2023
23ac6e7
:zap: readWriteiles update, added extract and convert files nodes to …
michael-radency Dec 6, 2023
e60e535
:zap: compress gaip fixes and UI update, readWriteFiles read operatio…
michael-radency Dec 6, 2023
4fa063a
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Dec 6, 2023
1b754a7
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Dec 7, 2023
22c6354
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Dec 7, 2023
bb82100
:zap: errors mapping, fixes
michael-radency Dec 7, 2023
f93764f
:zap: icall fix, error messages update
michael-radency Dec 7, 2023
d344083
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Dec 7, 2023
a03b0d8
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Dec 8, 2023
aa38d31
:zap: extract from ics operation, fixes for icall operation, other fixes
michael-radency Dec 8, 2023
7783040
Merge branch 'node-840-file-nodes' of https://github.com/n8n-io/n8n i…
michael-radency Dec 8, 2023
3c1ec88
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Dec 8, 2023
0be51e5
:zap: csv parser reverted change
michael-radency Dec 8, 2023
83d1cd9
Icons update and minor copy tweaks.
gandreini Dec 13, 2023
90c648a
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Dec 14, 2023
d753d42
:zap: decompress gz fixes
michael-radency Dec 14, 2023
66bc1c5
:zap: review updates
michael-radency Dec 14, 2023
2a17c08
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Dec 18, 2023
498f34d
Merge branch 'master' into node-840-file-nodes
maspio Dec 21, 2023
c355586
updated pnpm-lock.yaml after merge
maspio Dec 21, 2023
6d603f6
adding ReadWriteFile tests
maspio Dec 21, 2023
bba1cf0
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Jan 1, 2024
6922dba
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Jan 2, 2024
2a78527
:zap: cypress tests fix
michael-radency Jan 3, 2024
68467ff
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Jan 3, 2024
f32ced8
:zap: convert to / extract from tests
michael-radency Jan 3, 2024
b13a52b
Merge branch 'master' of https://github.com/n8n-io/n8n into node-840-…
michael-radency Jan 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions cypress/e2e/16-webhook-node.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -187,12 +187,14 @@ describe('Webhook Trigger node', async () => {

ndv.getters.backToCanvas().click();

workflowPage.actions.addNodeToCanvas('Convert to/from binary data');
workflowPage.actions.addNodeToCanvas('Convert to File');
workflowPage.actions.zoomToFit();

workflowPage.actions.openNode('Convert to/from binary data');
workflowPage.actions.openNode('Convert to File');
cy.getByTestId('parameter-input-operation').click();
getVisibleSelect().find('.option-headline').contains('Convert to JSON').click();
cy.getByTestId('parameter-input-mode').click();
getVisibleSelect().find('.option-headline').contains('JSON to Binary').click();
getVisibleSelect().find('.option-headline').contains('Each Item to Separate File').click();
ndv.getters.backToCanvas().click();

workflowPage.actions.executeWorkflow();
Expand Down
7 changes: 4 additions & 3 deletions cypress/e2e/4-node-creator.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ describe('Node Creator', () => {
it('should not show actions for single action nodes', () => {
const singleActionNodes = [
'DHL',
'iCalendar',
'Edit Fields',
'LingvaNex',
'Mailcheck',
'MSG91',
Expand Down Expand Up @@ -484,8 +484,9 @@ describe('Node Creator', () => {
nodeCreatorFeature.getters.nodeItemName().first().should('have.text', 'Wait');

nodeCreatorFeature.getters.searchBar().find('input').clear().type('spreadsheet');
nodeCreatorFeature.getters.nodeItemName().first().should('have.text', 'Spreadsheet File');
nodeCreatorFeature.getters.nodeItemName().eq(1).should('have.text', 'Google Sheets');
nodeCreatorFeature.getters.nodeItemName().first().should('have.text', 'Convert to File');
nodeCreatorFeature.getters.nodeItemName().eq(1).should('have.text', 'Extract From File');
nodeCreatorFeature.getters.nodeItemName().eq(2).should('have.text', 'Google Sheets');

nodeCreatorFeature.getters.searchBar().find('input').clear().type('sheets');
nodeCreatorFeature.getters.nodeItemName().first().should('have.text', 'Google Sheets');
Expand Down
23 changes: 17 additions & 6 deletions packages/core/src/NodeExecuteFunctions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -987,16 +987,27 @@ export function assertBinaryData(
): IBinaryData {
const binaryKeyData = inputData.main[inputIndex]![itemIndex]!.binary;
if (binaryKeyData === undefined) {
throw new NodeOperationError(node, 'No binary data exists on item!', {
itemIndex,
});
throw new NodeOperationError(
node,
`This operation expects the node's input data to contain a binary file '${propertyName}', but none was found [item ${itemIndex}]`,
{
itemIndex,
description: 'Make sure that the previous node outputs a binary file',
},
);
}

const binaryPropertyData = binaryKeyData[propertyName];
if (binaryPropertyData === undefined) {
throw new NodeOperationError(node, `Item has no binary property called "${propertyName}"`, {
itemIndex,
});
throw new NodeOperationError(
node,
`The item has no binary field '${propertyName}' [item ${itemIndex}]`,
{
itemIndex,
description:
'Check that the parameter where you specified the input binary field name is correct, and that it matches a field in the binary input',
},
);
}

return binaryPropertyData;
Expand Down
22 changes: 21 additions & 1 deletion packages/editor-ui/src/components/Node/NodeCreator/viewsData.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ import {
AI_CATEGORY_EMBEDDING,
AI_OTHERS_NODE_CREATOR_VIEW,
AI_UNCATEGORIZED_CATEGORY,
CONVERT_TO_FILE_NODE_TYPE,
EXTRACT_FROM_FILE_NODE_TYPE,
SET_NODE_TYPE,
CODE_NODE_TYPE,
DATETIME_NODE_TYPE,
Expand All @@ -48,6 +50,8 @@ import {
HELPERS_SUBCATEGORY,
RSS_READ_NODE_TYPE,
EMAIL_SEND_NODE_TYPE,
EDIT_IMAGE_NODE_TYPE,
COMPRESSION_NODE_TYPE,
} from '@/constants';
import { useI18n } from '@/composables/useI18n';
import { useNodeTypesStore } from '@/stores/nodeTypes.store';
Expand Down Expand Up @@ -394,7 +398,16 @@ export function RegularView(nodes: SimplifiedNodeType[]) {
{
key: 'convert',
title: i18n.baseText('nodeCreator.sectionNames.transform.convert'),
items: [HTML_NODE_TYPE, MARKDOWN_NODE_TYPE, XML_NODE_TYPE, CRYPTO_NODE_TYPE],
items: [
HTML_NODE_TYPE,
MARKDOWN_NODE_TYPE,
XML_NODE_TYPE,
CRYPTO_NODE_TYPE,
EXTRACT_FROM_FILE_NODE_TYPE,
CONVERT_TO_FILE_NODE_TYPE,
COMPRESSION_NODE_TYPE,
EDIT_IMAGE_NODE_TYPE,
],
},
],
},
Expand Down Expand Up @@ -422,6 +435,13 @@ export function RegularView(nodes: SimplifiedNodeType[]) {
properties: {
title: FILES_SUBCATEGORY,
icon: 'file-alt',
sections: [
{
key: 'popular',
title: i18n.baseText('nodeCreator.sectionNames.popular'),
items: [CONVERT_TO_FILE_NODE_TYPE, EXTRACT_FROM_FILE_NODE_TYPE],
},
],
},
},
{
Expand Down
4 changes: 4 additions & 0 deletions packages/editor-ui/src/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,8 @@ export const XERO_NODE_TYPE = 'n8n-nodes-base.xero';
export const ZENDESK_NODE_TYPE = 'n8n-nodes-base.zendesk';
export const ZENDESK_TRIGGER_NODE_TYPE = 'n8n-nodes-base.zendeskTrigger';
export const DISCORD_NODE_TYPE = 'n8n-nodes-base.discord';
export const EXTRACT_FROM_FILE_NODE_TYPE = 'n8n-nodes-base.extractFromFile';
export const CONVERT_TO_FILE_NODE_TYPE = 'n8n-nodes-base.convertToFile';
export const DATETIME_NODE_TYPE = 'n8n-nodes-base.dateTime';
export const REMOVE_DUPLICATES_NODE_TYPE = 'n8n-nodes-base.removeDuplicates';
export const SPLIT_OUT_NODE_TYPE = 'n8n-nodes-base.splitOut';
Expand All @@ -172,6 +174,8 @@ export const MARKDOWN_NODE_TYPE = 'n8n-nodes-base.markdown';
export const XML_NODE_TYPE = 'n8n-nodes-base.xml';
export const CRYPTO_NODE_TYPE = 'n8n-nodes-base.crypto';
export const RSS_READ_NODE_TYPE = 'n8n-nodes-base.rssFeedRead';
export const COMPRESSION_NODE_TYPE = 'n8n-nodes-base.compression';
export const EDIT_IMAGE_NODE_TYPE = 'n8n-nodes-base.editImage';

export const CREDENTIAL_ONLY_NODE_PREFIX = 'n8n-creds-base';
export const CREDENTIAL_ONLY_HTTP_NODE_VERSION = 4.1;
Expand Down
4 changes: 2 additions & 2 deletions packages/nodes-base/nodes/ApiTemplateIo/ApiTemplateIo.node.ts
Original file line number Diff line number Diff line change
Expand Up @@ -181,12 +181,12 @@ export class ApiTemplateIo implements INodeType {
description: 'Name of the binary property to which to write the data of the read file',
},
{
displayName: 'Binary Property',
displayName: 'Put Output File in Field',
name: 'binaryProperty',
type: 'string',
required: true,
default: 'data',
description: 'Name of the binary property to which to write to',
hint: 'The name of the output binary field to put the file in',
displayOptions: {
show: {
resource: ['pdf', 'image'],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ export class AwsRekognition implements INodeType {
default: 'detectFaces',
},
{
displayName: 'Binary Data',
displayName: 'Binary File',
name: 'binaryData',
type: 'boolean',
default: false,
Expand All @@ -104,7 +104,7 @@ export class AwsRekognition implements INodeType {
description: 'Whether the image to analyze should be taken from binary field',
},
{
displayName: 'Binary Property',
displayName: 'Input Binary Field',
displayOptions: {
show: {
operation: ['analyze'],
Expand All @@ -115,7 +115,7 @@ export class AwsRekognition implements INodeType {
name: 'binaryPropertyName',
type: 'string',
default: 'data',
description: 'Object property name which holds binary data',
hint: 'The name of the input binary field containing the file to be written',
required: true,
},
{
Expand Down
10 changes: 5 additions & 5 deletions packages/nodes-base/nodes/Aws/S3/V1/FileDescription.ts
Original file line number Diff line number Diff line change
Expand Up @@ -373,7 +373,7 @@ export const fileFields: INodeProperties[] = [
description: 'If not set the binary data filename will be used',
},
{
displayName: 'Binary Data',
displayName: 'Binary File',
name: 'binaryData',
type: 'boolean',
default: true,
Expand Down Expand Up @@ -401,7 +401,7 @@ export const fileFields: INodeProperties[] = [
description: 'The text content of the file to upload',
},
{
displayName: 'Binary Property',
displayName: 'Input Binary Field',
name: 'binaryPropertyName',
type: 'string',
default: 'data',
Expand All @@ -414,7 +414,7 @@ export const fileFields: INodeProperties[] = [
},
},
placeholder: '',
description: 'Name of the binary property which contains the data for the file to be uploaded',
hint: 'The name of the input binary field containing the file to be uploaded',
},
{
displayName: 'Additional Fields',
Expand Down Expand Up @@ -698,7 +698,7 @@ export const fileFields: INodeProperties[] = [
},
},
{
displayName: 'Binary Property',
displayName: 'Put Output File in Field',
name: 'binaryPropertyName',
type: 'string',
required: true,
Expand All @@ -709,7 +709,7 @@ export const fileFields: INodeProperties[] = [
resource: ['file'],
},
},
description: 'Name of the binary property to which to write the data of the read file',
hint: 'The name of the output binary field to put the file in',
},
/* -------------------------------------------------------------------------- */
/* file:delete */
Expand Down
10 changes: 5 additions & 5 deletions packages/nodes-base/nodes/Aws/S3/V2/FileDescription.ts
Original file line number Diff line number Diff line change
Expand Up @@ -373,7 +373,7 @@ export const fileFields: INodeProperties[] = [
description: 'If not set the binary data filename will be used',
},
{
displayName: 'Binary Data',
displayName: 'Binary File',
name: 'binaryData',
type: 'boolean',
default: true,
Expand Down Expand Up @@ -401,7 +401,7 @@ export const fileFields: INodeProperties[] = [
description: 'The text content of the file to upload',
},
{
displayName: 'Binary Property',
displayName: 'Input Binary Field',
name: 'binaryPropertyName',
type: 'string',
default: 'data',
Expand All @@ -414,7 +414,7 @@ export const fileFields: INodeProperties[] = [
},
},
placeholder: '',
description: 'Name of the binary property which contains the data for the file to be uploaded',
hint: 'The name of the input binary field containing the file to be uploaded',
},
{
displayName: 'Additional Fields',
Expand Down Expand Up @@ -698,7 +698,7 @@ export const fileFields: INodeProperties[] = [
},
},
{
displayName: 'Binary Property',
displayName: 'Put Output File in Field',
name: 'binaryPropertyName',
type: 'string',
required: true,
Expand All @@ -709,7 +709,7 @@ export const fileFields: INodeProperties[] = [
resource: ['file'],
},
},
description: 'Name of the binary property to which to write the data of the read file',
hint: 'The name of the output binary field to put the file in',
},
/* -------------------------------------------------------------------------- */
/* file:delete */
Expand Down
10 changes: 5 additions & 5 deletions packages/nodes-base/nodes/Box/FileDescription.ts
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ export const fileFields: INodeProperties[] = [
default: '',
},
{
displayName: 'Binary Property',
displayName: 'Put Output File in Field',
name: 'binaryPropertyName',
type: 'string',
required: true,
Expand All @@ -172,7 +172,7 @@ export const fileFields: INodeProperties[] = [
resource: ['file'],
},
},
description: 'Name of the binary property to which to write the data of the read file',
hint: 'The name of the output binary field to put the file in',
},

/* -------------------------------------------------------------------------- */
Expand Down Expand Up @@ -671,7 +671,7 @@ export const fileFields: INodeProperties[] = [
description: 'The name the file should be saved as',
},
{
displayName: 'Binary Data',
displayName: 'Binary File',
name: 'binaryData',
type: 'boolean',
default: false,
Expand Down Expand Up @@ -700,7 +700,7 @@ export const fileFields: INodeProperties[] = [
description: 'The text content of the file',
},
{
displayName: 'Binary Property',
displayName: 'Input Binary Field',
name: 'binaryPropertyName',
type: 'string',
default: 'data',
Expand All @@ -712,7 +712,7 @@ export const fileFields: INodeProperties[] = [
resource: ['file'],
},
},
description: 'Name of the binary property which contains the data for the file',
hint: 'The name of the input binary field containing the file to be uploaded',
},
{
displayName: 'Parent ID',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -327,7 +327,7 @@ export const messageFields: INodeProperties[] = [
value: 'url',
},
{
name: 'Binary Data',
name: 'Binary File',
value: 'binaryData',
},
],
Expand Down
15 changes: 13 additions & 2 deletions packages/nodes-base/nodes/Compression/Compression.node.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,19 @@
}
]
},
"alias": ["Zip", "Gzip", "uncompress"],
"alias": [
"Zip",
"Gzip",
"uncompress",
"compress",
"decompress",
"archive",
"unarchive",
"Binary",
"Files",
"File"
],
"subcategories": {
"Core Nodes": ["Files"]
"Core Nodes": ["Files", "Data Transformation"]
}
}
Loading
Loading