Add GitHub action setting for continuous benchmark

[c-bata]

No description provided.

[github-actions]

Benchmark of Sphere function

• Kurobako Version: 0.2.10
• Number of Solvers: 3
• Number of Problems: 1
• Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

1. Overall Results
2. Individual Results
3. Solvers
4. Problems
5. Studies

Overall Results

Solver	Borda	Firsts
Random	0	1
cmaes	0	1
pycma	0	1

Individual Results

(1) Problem: Sphere (dim=5)

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	cmaes (study)	0.000001 +- 0.000000	45.615 +- 14.082	2.170 +- 0.113
1	pycma (study)	0.000001 +- 0.000000	44.814 +- 3.402	63.207 +- 1.212
1	Random (study)	0.764246 +- 0.320599	599.472 +- 214.537	0.001 +- 0.000

Solvers

ID: f37905b5a31d6ea50b24dbb4d99e62c68d91a6eb83aad459be8edbfc8930bacd

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.2"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 0394b39d80ea73f7488aac85a2874a9835736db800590ac13532f244a43366bb

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=3.6.0.dev0, kurobako-py=0.2.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 2522ff71f79237de0b1f64e3204b2a6d69598a28699cda697e8fb89cb1989526

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=3.6.0.dev0, kurobako-py=0.2.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: deb9f36377d2b29273eb06e59cc3ed7585ae85e85accf9d60a308ccaddc9aadc

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_sphere.py",
      "5"
    ]
  }
}

specification:

{
  "name": "Sphere (dim=5)",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "x3",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "x4",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "x5",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM"
    }
  ],
  "values_domain": [
    {
      "name": "Sphere",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM"
    }
  ],
  "steps": 1
}

Studies

ID: 7cfc4a70103a32b021f065981cd6f747f1dced5386af3f7658dad9b77ccc39ae

• problem: Sphere (dim=5)
• solver: Random
• budget: 500
• repeats: 5
• concurrency: 1

ID: d049b22711ea0f79eb31047af67da9036a14728876bf6f8c4a47ff0c6845c1d4

• problem: Sphere (dim=5)
• solver: cmaes
• budget: 500
• repeats: 5
• concurrency: 1

ID: 1232fa34bc706f4151604b3641416ea0e37e5a78f58ed35b9ac9e70e59c129bb

• problem: Sphere (dim=5)
• solver: pycma
• budget: 500
• repeats: 5
• concurrency: 1