Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a process to scrape EC2 CPU information out of AWS API #7830

Closed
shoenig opened this issue Apr 29, 2020 · 3 comments · Fixed by #9038
Closed

Create a process to scrape EC2 CPU information out of AWS API #7830

shoenig opened this issue Apr 29, 2020 · 3 comments · Fixed by #9038
Assignees
@shoenig
Labels
theme/environment-aws theme/platform-arm

Comments

@shoenig
Copy link
Member

shoenig commented Apr 29, 2020

#7828 introduces a "best-effort" lookup table of CPU information per EC2 instance type.

This table is going to bitrot over time as Amazon introduces and upgrades instance types.

There is an EC2 API available that looks to have the information we need

processor-info.sustained-clock-speed-in-ghz - The CPU clock speed, in GHz.
vcpu-info.default-cores - The default number of cores for the instance type.
vcpu-info.default-threads-per-core - The default number of threads per core for the instance type.
vcpu-info.default-vcpus - The default number of vCPUs for the instance type.

Using the API requires AWS secrets, which I think precludes the possibility of having Nomad Client lookup this data directly.

Instead, we could establish some kind of process for generating a JSON version (or direct Go source file?) of the table using this API. E.g. part of the Nomad release process could be to push a button that triggers a pre-configured CI job that queries the API, generates the metadata table, and commits it to the nomad repo. Then aws_env.go would be redesigned to consume that generated content.
@shoenig shoenig mentioned this issue Apr 29, 2020
Merged
@shoenig
Copy link
Member Author

shoenig commented May 12, 2020

Some new instances types added today
https://aws.amazon.com/blogs/aws/new-m6g-ec2-instances-powered-by-arm-based-aws-graviton2/

@shoenig
Copy link
Member Author

shoenig commented Jun 5, 2020

More new instances (c5a -> epyc rome, which people might actually want)
https://aws.amazon.com/blogs/aws/new-amazon-ec2-c5a-instances-powered-by-2nd-gen-amd-epyc-processors/

@shoenig shoenig self-assigned this Oct 6, 2020
shoenig added a commit that referenced this issue Oct 6, 2020
@shoenig
client: get ec2 cpu perf data from AWS API
209a751 
Previously, Nomad was using a hand-made lookup table for looking
up EC2 CPU performance characteristics (core count + speed = ticks).

This data was incomplete and incorrect depending on region. The AWS
API has the correct data but requires API keys to use (i.e. should not
be queried directly from Nomad).

This change introduces a lookup table generated by a docker image based
on the Amazon cli tool.

https://hub.docker.com/r/hashicorpnomad/ec2info

The docker image is created from https://github.com/hashicorp/nomad-ec2info.

Going forward, Nomad can incorporate regeneration of the lookup table
somewhere in the CI pipeline so that we remain up-to-date on the latest
offerings from EC2.

Fixes #7830
shoenig added a commit that referenced this issue Oct 6, 2020
@shoenig
env_aws: get ec2 cpu perf data from AWS API
fa2bee0 
Previously, Nomad was using a hand-made lookup table for looking
up EC2 CPU performance characteristics (core count + speed = ticks).

This data was incomplete and incorrect depending on region. The AWS
API has the correct data but requires API keys to use (i.e. should not
be queried directly from Nomad).

This change introduces a lookup table generated by a docker image based
on the Amazon cli tool.

https://hub.docker.com/r/hashicorpnomad/ec2info

The docker image is created from https://github.com/hashicorp/nomad-ec2info.

Going forward, Nomad can incorporate regeneration of the lookup table
somewhere in the CI pipeline so that we remain up-to-date on the latest
offerings from EC2.

Fixes #7830
@shoenig shoenig mentioned this issue Oct 6, 2020
Merged
shoenig added a commit that referenced this issue Oct 6, 2020
@shoenig
env_aws: get ec2 cpu perf data from AWS API
df04f18 
Previously, Nomad was using a hand-made lookup table for looking
up EC2 CPU performance characteristics (core count + speed = ticks).

This data was incomplete and incorrect depending on region. The AWS
API has the correct data but requires API keys to use (i.e. should not
be queried directly from Nomad).

This change introduces a lookup table generated by a docker image based
on the Amazon cli tool.

https://hub.docker.com/r/hashicorpnomad/ec2info

The docker image is created from https://github.com/hashicorp/nomad-ec2info.

Going forward, Nomad can incorporate regeneration of the lookup table
somewhere in the CI pipeline so that we remain up-to-date on the latest
offerings from EC2.

Fixes #7830
shoenig added a commit that referenced this issue Oct 6, 2020
@shoenig
env_aws: get ec2 cpu perf data from AWS API
59dc5d4 
Previously, Nomad was using a hand-made lookup table for looking
up EC2 CPU performance characteristics (core count + speed = ticks).

This data was incomplete and incorrect depending on region. The AWS
API has the correct data but requires API keys to use (i.e. should not
be queried directly from Nomad).

This change introduces a lookup table generated by a docker image based
on the Amazon cli tool.

https://hub.docker.com/r/hashicorpnomad/ec2info

The docker image is created from https://github.com/hashicorp/nomad-ec2info.

Going forward, Nomad can incorporate regeneration of the lookup table
somewhere in the CI pipeline so that we remain up-to-date on the latest
offerings from EC2.

Fixes #7830
shoenig added a commit that referenced this issue Oct 6, 2020
@shoenig
env_aws: get ec2 cpu perf data from AWS API
8bd6bab 
Previously, Nomad was using a hand-made lookup table for looking
up EC2 CPU performance characteristics (core count + speed = ticks).

This data was incomplete and incorrect depending on region. The AWS
API has the correct data but requires API keys to use (i.e. should not
be queried directly from Nomad).

This change introduces a lookup table generated by a docker image based
on the Amazon cli tool.

https://hub.docker.com/r/hashicorpnomad/ec2info

The docker image is created from https://github.com/hashicorp/nomad-ec2info.

Going forward, Nomad can incorporate regeneration of the lookup table
somewhere in the CI pipeline so that we remain up-to-date on the latest
offerings from EC2.

Fixes #7830
shoenig added a commit that referenced this issue Oct 8, 2020
@shoenig
env_aws: get ec2 cpu perf data from AWS API
53ab308 
Previously, Nomad was using a hand-made lookup table for looking
up EC2 CPU performance characteristics (core count + speed = ticks).

This data was incomplete and incorrect depending on region. The AWS
API has the correct data but requires API keys to use (i.e. should not
be queried directly from Nomad).

This change introduces a lookup table generated by a small command line
tool in Nomad's tools module which uses the Amazon AWS API.

Running the tool requires AWS_* environment variables set.
  $ # in nomad/tools/cpuinfo
  $ go run .

Going forward, Nomad can incorporate regeneration of the lookup table
somewhere in the CI pipeline so that we remain up-to-date on the latest
offerings from EC2.

Fixes #7830
fredrikhgrelland pushed a commit to fredrikhgrelland/nomad that referenced this issue Oct 22, 2020
@shoenig @fredrikhgrelland
env_aws: get ec2 cpu perf data from AWS API
0dd474b 
Previously, Nomad was using a hand-made lookup table for looking
up EC2 CPU performance characteristics (core count + speed = ticks).

This data was incomplete and incorrect depending on region. The AWS
API has the correct data but requires API keys to use (i.e. should not
be queried directly from Nomad).

This change introduces a lookup table generated by a small command line
tool in Nomad's tools module which uses the Amazon AWS API.

Running the tool requires AWS_* environment variables set.
  $ # in nomad/tools/cpuinfo
  $ go run .

Going forward, Nomad can incorporate regeneration of the lookup table
somewhere in the CI pipeline so that we remain up-to-date on the latest
offerings from EC2.

Fixes hashicorp#7830
@github-actions
Copy link

github-actions bot commented Nov 1, 2022

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 1, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@shoenig shoenig

Labels
theme/environment-aws theme/platform-arm
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

env_aws: get ec2 cpu perf data from AWS API hashicorp/nomad
2 participants
@schmichael @shoenig