Enhance WAIT task to support time-based wait scenarios #2986

v1r3n · 2022-05-12T19:00:58Z

Pull Request type

Bugfix
Feature
Refactoring (no functional changes, no api changes)
Build related changes (Please run ./gradlew generateLock saveLock to refresh dependencies)
WHOSUSING.md
Other (please describe):

Changes in this PR

Enhance wait task to be able to wait for a specific amount of time.
One commonly asked use case is to allow a task to wait for specific duration of time (e.g. 15 minutes) or until certain date/time (Dec 31, 2022).

These updates allows users to configure Wait tasks with two inputs:

Wait For time duration

Format duration as "XhYmZs"

{
  "type": "WAIT",
   "inputParameters": {
     "duration": "10m20s",
   }
}

Wait until specific date/time

Specify the date as one of the formats:

"yyyy-MM-dd HH:mm",
"yyyy-MM-dd HH:mm z",
"yyyy-MM-dd"

{
  "type": "WAIT",
   "inputParameters": {
     "until": "2022-12-31 11:59",
   }
}

aravindanr · 2022-05-16T19:58:20Z

core/src/main/java/com/netflix/conductor/core/execution/tasks/Wait.java

+        String until =
+                Optional.ofNullable(task.getInputData().get(UNTIL_INPUT)).orElse("").toString();
+
+        if (Strings.isNotBlank(duration) && Strings.isNotBlank(until)) {


Strings is from log4j. Please use StringUtils from commons-lang3.

aravindanr · 2022-05-16T20:59:21Z

core/src/main/java/com/netflix/conductor/core/execution/tasks/Wait.java

+        if (Strings.isNotBlank(duration) && Strings.isNotBlank(until)) {
+            task.setReasonForIncompletion(
+                    "Both 'duration' and 'until' specified. Please provide only one input");
+            task.setStatus(FAILED);


Can the status be FAILED_WITH_TERMINAL_ERROR? This error is not recoverable.

aravindanr · 2022-05-16T21:18:11Z

core/src/main/java/com/netflix/conductor/core/execution/tasks/Wait.java

+        String until =
+                Optional.ofNullable(task.getInputData().get(UNTIL_INPUT)).orElse("").toString();
+
+        if (Strings.isNotBlank(duration) && Strings.isNotBlank(until)) {


Task validations are typically done when the workflow is registered.

added this as part of workflow creation.

aravindanr · 2022-05-16T21:31:51Z

core/src/main/java/com/netflix/conductor/core/execution/tasks/Wait.java

+        } else if (Strings.isNotBlank(until)) {
+            try {
+                Date expiryDate = parseDate(until);
+                System.out.println("expiryDate: " + expiryDate);


Remove SOP?

aravindanr · 2022-05-16T21:32:27Z

core/src/main/java/com/netflix/conductor/core/execution/tasks/Wait.java

+            } catch (ParseException parseException) {
+                task.setReasonForIncompletion(
+                        "Invalid/Unsupported Wait Until format.  Provided: " + until);
+                task.setStatus(FAILED);


I think this could also be FAILED_WITH_TERMINAL_ERROR.

aravindanr · 2022-05-16T21:37:08Z

core/src/main/java/com/netflix/conductor/core/execution/tasks/Wait.java

+        if (Strings.isNotBlank(duration)) {
+
+            Duration timeDuration = parseDuration(duration);
+            task.getOutputData()


Instead of adding the timeout to the output data, we suggest that its added as a field to the TaskModel. We think its a better fit because,

TaskModel is internal only and fields in it are not exposed to the user. Adding it to outputData makes the timeout field part of the task contract.

We have plans to create custom TaskModels which only contain fields for a particular task type.

updated to use TaskModel

aravindanr · 2022-05-16T21:39:13Z

core/src/main/java/com/netflix/conductor/core/execution/tasks/Wait.java

+            task.getOutputData()
+                    .put(TIMEOUT, System.currentTimeMillis() + (timeDuration.getSeconds() * 1000));
+            long seconds = timeDuration.getSeconds();
+            task.setCallbackAfterSeconds(seconds);


The callbackAfterSeconds is moot since WAIT is a synchronous task and is not added to a queue.

updated and removed.

aravindanr · 2022-05-16T21:42:27Z

@v1r3n can you update the documentation section for WAIT task? cc: @dougsillars

1. Added validations at the time of workflow creation 2. Using commons lang StringUtils 3. Using TaskModel to store timeout 4. Update documentation

apanicker-nflx · 2022-05-23T21:58:20Z

polyglot-clients/README.md

+# SDKs for other languages
+
+Language specific client SDKs are maintained at a dedicated [conductor-sdk](https://github.com/conductor-sdk) repository. 
+All the 


Incomplete?

apanicker-nflx · 2022-05-23T21:59:51Z

core/src/main/java/com/netflix/conductor/core/execution/tasks/Wait.java

    public Wait() {
        super(TASK_TYPE_WAIT);
    }

+    public static Duration parseDuration(String text) {


Consider moving these static methods to Utils?

apanicker-nflx · 2022-05-26T21:04:36Z

core/src/main/java/com/netflix/conductor/utils/DateTimeUtils.java

+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ */
+package com.netflix.conductor.utils;


There is already a Utils package here. Can you move this class under the same package?

* Update Wait.java * Add support for waiting for time duration or until a specific date/time * add unit tests and SDK methods * formatting * Remove Guava dependency * Fix formatting * Wait task updates 1. Added validations at the time of workflow creation 2. Using commons lang StringUtils 3. Using TaskModel to store timeout 4. Update documentation * remove callbackafter * add the break * Fix readme * Move the date parsing to a util class * move DateTimeUtils to the common utils package

v1r3n added 7 commits April 21, 2022 16:25

Update Wait.java

a249563

Add support for waiting for time duration or until a specific date/time

cf40979

Merge branch 'Netflix:main' into wait_improvements

7586801

add unit tests and SDK methods

e1bbbc4

formatting

9a11791

Remove Guava dependency

8c8fe55

Fix formatting

5cec41b

aravindanr suggested changes May 16, 2022

View reviewed changes

apanicker-nflx added the type: feature New feature label May 16, 2022

v1r3n added 3 commits May 18, 2022 23:00

Wait task updates

f1c2ecd

1. Added validations at the time of workflow creation 2. Using commons lang StringUtils 3. Using TaskModel to store timeout 4. Update documentation

remove callbackafter

ae8b874

add the break

2876f3d

apanicker-nflx reviewed May 23, 2022

View reviewed changes

v1r3n added 2 commits May 23, 2022 17:17

Fix readme

be6194c

Move the date parsing to a util class

03ec26f

apanicker-nflx reviewed May 26, 2022

View reviewed changes

v1r3n added 2 commits May 30, 2022 09:09

move DateTimeUtils to the common utils package

f9679d9

Merge branch 'main' into wait_improvements

739c615

aravindanr approved these changes May 31, 2022

View reviewed changes

apanicker-nflx approved these changes May 31, 2022

View reviewed changes

apanicker-nflx changed the title ~~Wait improvements~~ Enhance WAIT task to support Timed-based wait scenarios May 31, 2022

apanicker-nflx changed the title ~~Enhance WAIT task to support Timed-based wait scenarios~~ Enhance WAIT task to support timed-based wait scenarios May 31, 2022

apanicker-nflx changed the title ~~Enhance WAIT task to support timed-based wait scenarios~~ Enhance WAIT task to support time-based wait scenarios May 31, 2022

apanicker-nflx merged commit 3b5aed1 into Netflix:main May 31, 2022

v1r3n deleted the wait_improvements branch June 15, 2022 06:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance WAIT task to support time-based wait scenarios #2986

Enhance WAIT task to support time-based wait scenarios #2986

v1r3n commented May 12, 2022

aravindanr May 16, 2022

v1r3n May 19, 2022

aravindanr May 16, 2022

v1r3n May 30, 2022

aravindanr May 16, 2022

v1r3n May 30, 2022

aravindanr May 16, 2022

v1r3n May 19, 2022

aravindanr May 16, 2022

v1r3n May 19, 2022

aravindanr May 16, 2022

v1r3n May 19, 2022

aravindanr May 16, 2022

v1r3n May 19, 2022

aravindanr commented May 16, 2022

apanicker-nflx May 23, 2022

v1r3n May 30, 2022

apanicker-nflx May 23, 2022

v1r3n May 30, 2022

apanicker-nflx May 26, 2022

v1r3n May 30, 2022

Enhance WAIT task to support time-based wait scenarios #2986

Enhance WAIT task to support time-based wait scenarios #2986

Conversation

v1r3n commented May 12, 2022

Pull Request type

Changes in this PR

Wait For time duration

Wait until specific date/time

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aravindanr commented May 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment