Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix dimensions of Warm starting CMA-ES #98

Merged
merged 1 commit into from
Feb 19, 2021
Merged

Fix dimensions of Warm starting CMA-ES #98

merged 1 commit into from
Feb 19, 2021

Conversation

c-bata
Copy link
Collaborator

@c-bata c-bata commented Feb 19, 2021

Fix #97.

@c-bata c-bata mentioned this pull request Feb 19, 2021
Closed
@c-bata c-bata added the bug Something isn't working label Feb 19, 2021
@github-actions
Copy link

Benchmark of Six-Hump Camel function

plot curve image

  • Kurobako Version: 0.2.8
  • Number of Solvers: 4
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Random 0 1
cmaes 0 1
pycma 0 1
sep-cmaes 0 1

Individual Results

(1) Problem: Six-Hump Camel Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 sep-cmaes (study) -1.031628 +- 0.000000 57511.033 +- 53896.077 0.978 +- 0.038
1 pycma (study) -1.031628 +- 0.000000 66767.817 +- 53781.414 33.564 +- 7.393
1 Random (study) -0.479527 +- 0.615616 45706.409 +- 44149.296 0.000 +- 0.000
1 cmaes (study) -1.031628 +- 0.000000 57520.176 +- 53897.363 1.089 +- 0.042

Solvers

ID: adba9490e43fe66d939f067852b174c90815eb5f7d736b45282d07bd324579d4

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.0"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: b2a339c8c86f5a3004528633a6a7b7181a7942c167b847a7695c5cffa2a1f32d

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.6.0.dev0, kurobako-py=0.1.9"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 39bc9da0bd1de93a0ff3ac66c3c066a8373f86fc7defb1eb744c5acb88f700ea

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.6.0.dev0, kurobako-py=0.1.9"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 02c97960d6e708b3fa4359e2594042564bf54ce72a351d9956a7f0b3da7d6970

recipe:

{
  "name": "sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "sep-cmaes"
    ]
  }
}

specification:

{
  "name": "sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.6.0.dev0, kurobako-py=0.1.9"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 5769b3adfa0de72663fca388a0968622a3d3e268a876cda34270c7cc10898254

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_six_hump_camel.py"
    ]
  }
}

specification:

{
  "name": "Six-Hump Camel Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Six-Hump Camel",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1,
  "reference_point": null
}

Studies

ID: 1c1c73e73172104744f6ce9c41fd2f29623862f54eb32dc121defc2d4da558d8

ID: 4151163d714650db2579699be1bd9e5efb62f93ed68d58ccbdf50d40b7fd2ea0

ID: c7daa80a05374fb9bf75c5d8144d70a87af4f728d404ca16971bbaf2eaab3d3a

ID: d4140fa68ef12667353e1b3da5f023c5e5c52762a2377bbf07899ca36aa0332a

@github-actions
Copy link

Benchmark of Rastrigin function

plot curve image

  • Kurobako Version: 0.2.8
  • Number of Solvers: 5
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Random 0 0
cmaes 0 0
ipop-cmaes 3 1
ipop-sep-cmaes 1 1
sep-cmaes 0 0

Individual Results

(1) Problem: Rastrigin (dim=2)

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 ipop-cmaes (study) 0.026032 +- 0.065817 1463.436 +- 960.272 18.688 +- 2.956
1 ipop-sep-cmaes (study) 0.181123 +- 0.346026 2310.735 +- 1138.421 18.166 +- 1.351
2 sep-cmaes (study) 1.403603 +- 1.018936 3953.814 +- 2548.705 17.787 +- 2.530
2 cmaes (study) 0.824359 +- 0.676621 2448.751 +- 1720.725 20.017 +- 1.570
3 Random (study) 1.392878 +- 0.512078 5576.501 +- 1551.925 0.004 +- 0.000

Solvers

ID: adba9490e43fe66d939f067852b174c90815eb5f7d736b45282d07bd324579d4

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.0"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: b2a339c8c86f5a3004528633a6a7b7181a7942c167b847a7695c5cffa2a1f32d

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.6.0.dev0, kurobako-py=0.1.9"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 7d78449387eca11639d0147d938c071c0471270917c6a945c83c2adb6a5f4678

recipe:

{
  "name": "ipop-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "ipop-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.6.0.dev0, kurobako-py=0.1.9"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: d573b1ed7db9bd4385f51bbcf3c2117b97e31ca341f55f5df068ee9e0d3aa16d

recipe:

{
  "name": "ipop-sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "ipop-sep-cmaes"
    ]
  }
}

specification:

{
  "name": "ipop-sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.6.0.dev0, kurobako-py=0.1.9"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 02c97960d6e708b3fa4359e2594042564bf54ce72a351d9956a7f0b3da7d6970

recipe:

{
  "name": "sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "sep-cmaes"
    ]
  }
}

specification:

{
  "name": "sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.6.0.dev0, kurobako-py=0.1.9"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: f7401d817c92517b8a9513f31de65be8bada24717215e8bf3c844b12911b569b

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_rastrigin.py",
      "2"
    ]
  }
}

specification:

{
  "name": "Rastrigin (dim=2)",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rastrigin",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1,
  "reference_point": null
}

Studies

ID: f6c55bb604503521061db9435a057025ba3d21378eb5d3ceb7eec3305af1d0b6

ID: 95cd4825e594a3c699c48c94ea352da85200b8ab2022f6b721d774f38841c128

ID: 901b247df4364a2e87e2d26798b420734c908aa7e2822316a7caaebbf057d4d3

ID: ab7cb49ca5d7bfe3dce6dee526c7a73d42fe72fd1690f1c58ec0e25877a914b2

ID: 8474adcd396e9e1ed3729771ad1ced4a605725148a03caf7ebbbe701172c479d

@c-bata c-bata merged commit 05ef85c into main Feb 19, 2021
@c-bata c-bata deleted the fix-ws-cma-es-dim branch February 19, 2021 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Wrong definition of dim in get_warm_start_mgd
1 participant
@c-bata