Auto tuning feature enhancements #379

arpang · 2018-05-10T13:01:16Z

Auto tuning feature enhancements:

Tuning auto switch off when:
- parameter converges
- the maximum number of tuning iterations is reached
- no gain in cost function
Remembers and returns the best parameter set when:
- tuning is switched off
- no new parameter suggestion exists for the job
- execution has failed and a retry is attempted
Parameters being tune now attain only discrete values defined by the step size
Bug fixes

File-wise summary of changes:

exception_class parameter in Scheduler.conf renamed to workflow_client
Removing APIFitnessComputeUtil.java
Remembering the best parameter set
Changes in FitnessComputeUtil.java:
- Checking if tuning can be switched off. The qualifying scenarios are:
  - If parameters converge
  - If median gain is negative
  - Max #executions are reached
- Removing the penalty applied in case of failed executions
- Metric violation normalized by input size
- Check and update if the best parameter set is found
Changes in JobTuningInfo.java, PSOParamGenerator.java, ParamGenerator.java, pso_param_generation.py:
- Added jobType information
Added the missing java docs
Removed unused variables from models
Changes in tunein-test1.sql, test-init.sql: Added the column names in insert statements

Support for spark

akshayrai

Looks good to merge after addressing the minor comments!

Thanks for cleaning up the code as well.

akshayrai · 2018-05-11T13:09:16Z

app/com/linkedin/drelephant/tuning/AutoTuningAPIHelper.java

+   * @param jobExecId String jobExecId of the execution to which penalty has to be applied
+   */
+  private void applyPenalty(String jobExecId) {
+    Integer penaltyConstant = 3;


Add a comment in the code on why 3 was chosen?

akshayrai · 2018-05-11T13:23:25Z

scripts/pso/pso_param_generation.py

        elif param_name[i] == PARAM_PIG_MAX_COMBINED_SPLIT_SIZE:
-            max_combined_split_size_index = i
+            pig_max_combined_split_size_index = i
+        elif param_name[i] == PARAM_SPARK_EXECUTOR_MEMORY:


As you mentioned offline, it would make sense to send this together with the Spark changes.

akshayrai · 2018-05-11T13:25:37Z

conf/evolutions/default/6.sql

+) ENGINE=InnoDB;
+
+
+# --- !Downs


Can you write the downs section for the alter and insert statements as well?

akshayrai · 2018-05-11T13:26:47Z

conf/evolutions/default/6.sql

+INSERT INTO tuning_parameter VALUES (11,'spark.memory.fraction',2, 0.6, 0.1, 0.9, 0.1, 0, current_timestamp(0), current_timestamp(0));
+INSERT INTO tuning_parameter VALUES (12,'spark.memory.storageFraction', 2, 0.5, 0.1, 0.9, 0.1, 0, current_timestamp(0), current_timestamp(0));
+INSERT INTO tuning_parameter VALUES (13,'spark.executor.cores', 2, 1 , 1, 1, 1, 0, current_timestamp(0), current_timestamp(0));
+INSERT INTO tuning_parameter VALUES (14,'spark.yarn.executor.memoryOverhead', 2, 384, 384, 1024, 100, 0, current_timestamp(0), current_timestamp(0));


Can you add a comment on the background behind these constants?

This was actually not needed now (it was needed for spark tuning) so I removed it altogether.

akshayrai · 2018-05-11T13:27:48Z

scripts/pso/pso_param_generation.py

@@ -38,8 +39,14 @@
 PARAM_MAPREDUCE_MAP_JAVA_OPTS = 'mapreduce.map.java.opts'
 PARAM_MAPREDUCE_REDUCE_JAVA_OPTS = 'mapreduce.reduce.java.opts'

+PARAM_SPARK_EXECUTOR_MEMORY = "spark.executor.memory"


You can include these with the spark changes.

akshayrai · 2018-05-11T13:30:12Z

app/com/linkedin/drelephant/tuning/ParamGenerator.java

+          logger.info("Constraint violated: Sort memory > 60% of map memory");
+          violations++;
+        }
+        if (mrMapMemory - mrSortMemory < 768) {


It would be better to define variables for these constants like 768.

akshayrai · 2018-05-11T13:33:36Z

app/com/linkedin/drelephant/tuning/FitnessComputeUtil.java

-  protected void updateExecutionMetrics(List<TuningJobExecution> completedExecutions) {
-    for (TuningJobExecution tuningJobExecution : completedExecutions) {
+  private void updateExecutionMetrics(List<TuningJobExecution> completedExecutions) {
+    Integer penaltyConstant = 3;


akshayrai · 2018-05-11T13:35:14Z

app/com/linkedin/drelephant/tuning/FitnessComputeUtil.java

+        .eq(TuningJobDefinition.TABLE.job + '.' + JobDefinition.TABLE.id, jobDefinition.id)
+        .findUnique();
+    if (tuningJobDefinition.tuningEnabled == 1) {
+      tuningJobDefinition.tuningEnabled = 0;


You might want to log this event

Good point. Done.

akshayrai · 2018-05-11T13:36:09Z

app/com/linkedin/drelephant/tuning/FitnessComputeUtil.java

+   * @return true if the median gain is negative, else false
+   */
+  private boolean isMedianGainNegative(List<JobExecution> jobExecutions) {
+    int num_fitness_for_median = 6;


explanation added in java docs.

This reverts commit d3fb6ba.

This reverts commit 38c4d8f.

Auto tuning feature enhancements: Tuning auto switch off when: * parameter converges * the maximum number of tuning iterations is reached * no gain in cost function Remembers and returns the best parameter set when: * tuning is switched off * no new parameter suggestion exists for the job * execution has failed and a retry is attempted Parameters being tune now attain only discrete values defined by the step size Bug fixes

This reverts commit d3fb6ba.

This reverts commit 1b36348.

This reverts commit d3fb6ba.

arpang added 3 commits May 9, 2018 23:34

In progress: Support for Spark

f921094

Support for spark

top commit

b63e908

Removing Spark implementation

55ad08a

akshayrai approved these changes May 11, 2018

View reviewed changes

arpang added 3 commits May 15, 2018 08:26

Addressing review comments

b4f7bfa

Addressed review comments

8a2f53a

Bug fix

f46ca13

akshayrai merged commit d3fb6ba into linkedin:master May 21, 2018

varunsaxena added a commit that referenced this pull request Aug 30, 2018

Revert "Auto tuning feature enhancements (#379)"

38c4d8f

This reverts commit d3fb6ba.

varunsaxena added a commit that referenced this pull request Aug 31, 2018

Revert "Revert "Auto tuning feature enhancements (#379)""

d28659b

This reverts commit 38c4d8f.

varunsaxena added a commit that referenced this pull request Oct 16, 2018

Revert "Auto tuning feature enhancements (#379)"

1b36348

This reverts commit d3fb6ba.

varunsaxena added a commit that referenced this pull request Oct 16, 2018

Revert "Revert "Auto tuning feature enhancements (#379)""

9296cb2

This reverts commit 1b36348.

edwinalu pushed a commit to edwinalu/dr-elephant that referenced this pull request Oct 23, 2018

Revert "Auto tuning feature enhancements (linkedin#379)"

d9ffee6

This reverts commit d3fb6ba.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto tuning feature enhancements #379

Auto tuning feature enhancements #379

arpang commented May 10, 2018 •

edited

Loading

akshayrai left a comment

akshayrai May 11, 2018

arpang May 15, 2018

akshayrai May 11, 2018

arpang May 15, 2018

akshayrai May 11, 2018

arpang May 16, 2018

akshayrai May 11, 2018

arpang May 15, 2018 •

edited

Loading

akshayrai May 11, 2018

arpang May 15, 2018

akshayrai May 11, 2018

akshayrai May 11, 2018

arpang May 15, 2018

akshayrai May 11, 2018

arpang May 15, 2018

akshayrai May 11, 2018

arpang May 15, 2018

Auto tuning feature enhancements #379

Auto tuning feature enhancements #379

Conversation

arpang commented May 10, 2018 • edited Loading

akshayrai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arpang May 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arpang commented May 10, 2018 •

edited

Loading

arpang May 15, 2018 •

edited

Loading