Skip to content

Commit

Permalink
fix: Stop OOM crashes in Execution Data pruning (#5095)
Browse files Browse the repository at this point in the history
* fix: Stop OOM crashed in Execution Data pruning

Currently while pruning execution data, we are loading all the data in memory. For instances where there are thousands of executions, this causes the container to run out of memory.
Since ids is all we need, we should only query for ids.

* query for Executions only when ids are actually needed for pruning binary data

in default mode the binary data is in the database, and will get pruned along with the executions.
  • Loading branch information
netroy authored Jan 6, 2023
1 parent f4140d0 commit c4df204
Showing 1 changed file with 13 additions and 8 deletions.
21 changes: 13 additions & 8 deletions packages/cli/src/WorkflowExecuteAdditionalData.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ import * as WorkflowHelpers from '@/WorkflowHelpers';
import { getWorkflowOwner } from '@/UserManagement/UserManagementHelper';
import { findSubworkflowStart } from '@/utils';
import { PermissionChecker } from './UserManagement/PermissionChecker';
import { eventBus } from './eventbus';
import { WorkflowsService } from './workflows/workflows.services';

const ERROR_TRIGGER_TYPE = config.getEnv('nodes.errorTriggerType');
Expand Down Expand Up @@ -204,18 +203,24 @@ async function pruneExecutionData(this: WorkflowHooks): Promise<void> {
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
const utcDate = DateUtils.mixedDateToUtcDatetimeString(date);

const toPrune = { stoppedAt: LessThanOrEqual(utcDate) };
const isBinaryModeDefaultMode = config.getEnv('binaryDataManager.mode') === 'default';
try {
const executions = await Db.collections.Execution.find({
stoppedAt: LessThanOrEqual(utcDate),
});
await Db.collections.Execution.delete({ stoppedAt: LessThanOrEqual(utcDate) });
const executions = isBinaryModeDefaultMode
? []
: await Db.collections.Execution.find({
select: ['id'],
where: toPrune,
});
await Db.collections.Execution.delete(toPrune);
setTimeout(() => {
throttling = false;
}, timeout * 1000);
// Mark binary data for deletion for all executions
await BinaryDataManager.getInstance().markDataForDeletionByExecutionIds(
executions.map(({ id }) => id),
);
if (!isBinaryModeDefaultMode)
await BinaryDataManager.getInstance().markDataForDeletionByExecutionIds(
executions.map(({ id }) => id),
);
} catch (error) {
ErrorReporter.error(error);
throttling = false;
Expand Down

0 comments on commit c4df204

Please sign in to comment.