Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fuzzing for sep-CMA-ES #105

Merged
merged 1 commit into from
Apr 10, 2021
Merged

Fuzzing for sep-CMA-ES #105

merged 1 commit into from
Apr 10, 2021

Conversation

c-bata
Copy link
Collaborator

@c-bata c-bata commented Apr 10, 2021

refs #104

@github-actions
Copy link

Tuning LightGBM for Kaggle Toxic Challenge (Surrogate)

plot curve image

  • Kurobako Version: 0.2.9
  • Number of Solvers: 2
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
cmaes 0 0
ws-cmaes 1 1

Individual Results

(1) Problem: wscmaes-toxic-target with warm starting

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 ws-cmaes (study) 0.017000 +- 0.000126 0.511 +- 0.007 0.699 +- 0.141
2 cmaes (study) 0.018301 +- 0.000472 0.571 +- 0.012 0.224 +- 0.059

Solvers

ID: 1388a6de65200d045a60b5a1d1913437b57ea1d2a9eb6e07656e13729503bdee

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: ef6eefd4e69f619390fab64533530b7badd4391fc6709167a0d7144c94df45d4

recipe:

{
  "name": "ws-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "ws-cmaes",
      "--warm-starting-trials",
      "100"
    ]
  }
}

specification:

{
  "name": "ws-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 4b8c2bfbea634e8b32601e81057749fa9185971643acf6d1cf36906867e163ce

recipe:

{
  "warm_starting": {
    "source": {
      "surrogate": {
        "model": "./toxic-lightgbm-warm-start/wscmaes-toxic-source/"
      }
    },
    "target": {
      "surrogate": {
        "model": "./toxic-lightgbm-warm-start/wscmaes-toxic-target/"
      }
    }
  }
}

specification:

{
  "name": "wscmaes-toxic-target with warm starting",
  "attrs": {},
  "params_domain": [
    {
      "name": "bagging_fraction",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.1,
        "high": 0.9
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "feature_fraction",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.1,
        "high": 0.9
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "lambda_l1",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.1,
        "high": 10.0
      },
      "distribution": "LOG_UNIFORM"
    },
    {
      "name": "lambda_l2",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.1,
        "high": 10.0
      },
      "distribution": "LOG_UNIFORM"
    },
    {
      "name": "learning_rate",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.001,
        "high": 1.0
      },
      "distribution": "LOG_UNIFORM"
    },
    {
      "name": "num_leaves",
      "range": {
        "type": "DISCRETE",
        "low": 8,
        "high": 129
      },
      "distribution": "UNIFORM"
    }
  ],
  "values_domain": [
    {
      "name": "value",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM"
    }
  ],
  "steps": [
    0,
    1
  ]
}

Studies

ID: 69024f37aa0af2fe15a01e67fa10ab7ecff875a5ffa8e75e2bea7f6fdb1e32bb

ID: ccbbaa37867579491ed3bb37921689f8e429cda0d133b064397e28a0e5c0ac52

@github-actions
Copy link

Benchmark of Six-Hump Camel function

plot curve image

  • Kurobako Version: 0.2.9
  • Number of Solvers: 4
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Random 0 1
cmaes 0 1
pycma 0 1
sep-cmaes 0 1

Individual Results

(1) Problem: Six-Hump Camel Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 cmaes (study) -1.031628 +- 0.000001 17707.279 +- 33403.962 1.081 +- 0.050
1 Random (study) -0.246249 +- 0.411201 26043.434 +- 41591.883 0.000 +- 0.000
1 pycma (study) -1.031628 +- 0.000000 17680.699 +- 34535.049 33.570 +- 6.947
1 sep-cmaes (study) -0.953502 +- 0.069965 17922.753 +- 33305.250 1.383 +- 0.127

Solvers

ID: a939db22692860645c92c8d357743db4cecbd97108212d5bc1d04b594aa830a9

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 1388a6de65200d045a60b5a1d1913437b57ea1d2a9eb6e07656e13729503bdee

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: e826da9b1a6e098f9b9d3ab4aeb3ed55546b179e8a3cf7bc59493c7a37cc15d7

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: f93f4880bcd7c97253921a95feff5726f9224363eae3684d639511f4a77a436e

recipe:

{
  "name": "sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "sep-cmaes"
    ]
  }
}

specification:

{
  "name": "sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 2c63bdd44db7d921185285d58afb2eae15d3eb87414c3923a59bde0c615175de

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_six_hump_camel.py"
    ]
  }
}

specification:

{
  "name": "Six-Hump Camel Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM"
    }
  ],
  "values_domain": [
    {
      "name": "Six-Hump Camel",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM"
    }
  ],
  "steps": 1
}

Studies

ID: 937df77d401862d742f5cfd9ae768c94307c01a1898d4665f8fcc55b00fd958f

ID: 221cde865b208c8d7e515b243219fd596396c4c4167d8b0ff8e97d9baaa7183f

ID: 5f6f4881d6c431d88acd8aab8e11cc8e7591e25b97d13a24512fde0437dff095

ID: 9210a1c5d00ae58fbde9c72038de4359f818639d62c3f18eff07d8b011504e34

@github-actions
Copy link

Benchmark of Rastrigin function

plot curve image

  • Kurobako Version: 0.2.9
  • Number of Solvers: 5
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Random 0 0
cmaes 0 1
ipop-cmaes 1 1
ipop-sep-cmaes 1 1
sep-cmaes 1 1

Individual Results

(1) Problem: Rastrigin (dim=2)

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 cmaes (study) 1.282304 +- 1.061057 3658.082 +- 2517.779 19.998 +- 1.323
1 ipop-cmaes (study) 0.303060 +- 0.449983 2682.230 +- 1367.767 19.079 +- 1.531
1 ipop-sep-cmaes (study) 0.410046 +- 0.280275 2153.686 +- 883.489 39.919 +- 3.211
1 sep-cmaes (study) 0.410046 +- 0.280275 2153.686 +- 883.489 37.228 +- 5.834
4 Random (study) 1.502151 +- 0.428810 6051.881 +- 1391.946 0.003 +- 0.000

Solvers

ID: a939db22692860645c92c8d357743db4cecbd97108212d5bc1d04b594aa830a9

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 1388a6de65200d045a60b5a1d1913437b57ea1d2a9eb6e07656e13729503bdee

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 5a57a82da232a4ce528dd9a9b6511d01feb022bd94e873b585213e78e61d4bc0

recipe:

{
  "name": "ipop-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "ipop-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: abc48ac81d021098a930e7f7cb2dcc3d8b67a846c88ff1f991f90f881bd34e51

recipe:

{
  "name": "ipop-sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "ipop-sep-cmaes"
    ]
  }
}

specification:

{
  "name": "ipop-sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: f93f4880bcd7c97253921a95feff5726f9224363eae3684d639511f4a77a436e

recipe:

{
  "name": "sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "sep-cmaes"
    ]
  }
}

specification:

{
  "name": "sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 313e939a7e591d92e5be006977daa358d2be4730efd89f70732984f4edb495e4

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_rastrigin.py",
      "2"
    ]
  }
}

specification:

{
  "name": "Rastrigin (dim=2)",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM"
    }
  ],
  "values_domain": [
    {
      "name": "Rastrigin",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM"
    }
  ],
  "steps": 1
}

Studies

ID: c785cfe8e5f2ad8fd010b3efeead86ae7378974fbc42ce7ceae0688fe486c813

ID: c07d8181cf55ecd92131bbd9241cf925a045b28d861fc6a3ae9b45314f732a03

ID: 9ae8d021c54d33ef43b0e38f98ac3a38333b1415bb6f0ac030aadc2c595639c5

ID: d6ab63ab5f33e13302f45ca28ee9294be403819cbbe3672f243cb8dc5b2a43dc

ID: 006e847a4161a177605eb5069a7eb6a50778b87e2a5bf932590ecbf01ff3e6f3

@c-bata c-bata merged commit 51853b1 into main Apr 10, 2021
@c-bata c-bata deleted the hypothesis-for-sepcmaes branch April 10, 2021 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@c-bata