-
-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extend Execution Stages with a new Level - Execution Batches #72
Comments
Hi @mrpaulandrew, I too had a similar requirement to support multiple parallel executions. I approached it similarly except that I called the higher-level concept as "Job". A Job is comprised of one or more stages. Metadatadb changes include Examples of Jobs:
I didn't have much use for Grand Parent at my work setup. So, my idea was to have the ParentPipeline to have different triggers (schedule or tumbling) and to pass the JobId as a parameter to the pipeline When a worker pipeline is reused in a stage within the same job or across jobs, for example, LaunchDatabricksJob is a pipeline that gets reused a lot at my work where I need to pass different invocation params, class name and jar name as well to submit different apps/jars on to Databricks cluster. Examples of Jobs: DailyTrigger-SalesIngestion => Scheduled At 3 AM Daily in AUS Time Zone => Parameter: JobId = 1 DailyTrigger-MarketingEgression => TumblingWindow trigger Scheduled at 4 AM Daily UTC => Parameter: JobId = 2 WeeklyTrigger-CampaignIngestion => Scheduled At 3 AM On Mondays in AUS Time Zone => Parameter: JobId = 3 I had this as a work-in-progress project in my local for some time, but after looking at recent conversations around this issue, I have pulled recent changes from the master branch, applied my changes (only metadata database changes) and pushed them to my fork. Planning to work on ADF Pipeline changes tonight and give a test end to end Please have a look at the changes and let me know if it helps - Thanks, |
@mrpaulandrew I have pushed my implementation of this onto my fork here: There are a few outstanding bits to fix:
Let me know if you have questions / any issues testing it |
Building on community feedback in the following issues:
#61
#62
Allow the framework to support multiple executions of the parent pipeline using a new higher level concept called Batches.
Examples of Batch names:
Finally, it is expected that a Trigger hitting the parent pipeline will have the batch name included.
The text was updated successfully, but these errors were encountered: