-
Notifications
You must be signed in to change notification settings - Fork 588
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AllowedValues from botocore during --update-specs #1682
Conversation
#50 AWS::WorkSpaces::Workspace.ComputeTypeName should pick up POWERPRO and GRAPHICSPRO: https://github.com/boto/botocore/blob/ae2d0855ac3162520e584a24f5522b633de4e2a9/botocore/data/workspaces/2015-04-08/service-2.json#L822-L823 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-workspaces-workspace-workspaceproperties.html#cfn-workspaces-workspace-workspaceproperties-computetypename AWS::CodeBuild::Project.Environment.Type should pick up WINDOWS_SERVER_2019_CONTAINER: https://github.com/boto/botocore/blob/ae2d0855ac3162520e584a24f5522b633de4e2a9/botocore/data/codebuild/2016-10-06/service-2.json#L1873 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codebuild-project-environment.html#cfn-codebuild-project-environment-type AWS::S3::Bucket.TopicConfiguration.Event should pick up s3:ObjectRestore:* and s3:Replication:*: https://github.com/boto/botocore/blob/ae2d0855ac3162520e584a24f5522b633de4e2a9/botocore/data/s3/2006-03-01/service-2.json#L2887-L2890 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-notificationconfig-topicconfig.html#cfn-s3-bucket-notificationconfig-topicconfig-event https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#notification-how-to-event-types-and-destinations EbsVolumeType should pick up io2: https://aws.amazon.com/blogs/aws/new-ebs-volume-type-io2-more-iops-gib-higher-durability/ https://raw.githubusercontent.com/boto/botocore/master/botocore/data/ec2/2016-11-15/service-2.json https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-blockdev-template.html#cfn-ec2-blockdev-template-volumetype https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-ebs-volume.html#cfn-ec2-ebs-volume-volumetype AWS::GuardDuty::IPSet.Format should pick up ALIEN_VAULT and PROOF_POINT and FIRE_EYE: https://github.com/boto/botocore/blob/ae2d0855ac3162520e584a24f5522b633de4e2a9/botocore/data/guardduty/2017-11-28/service-2.json#L3130-L3132 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-guardduty-ipset.html#cfn-guardduty-ipset-format AWS::CodeBuild::Project.Environment.ComputeType should pick up BUILD_GENERAL1_2XLARGE: https://github.com/boto/botocore/blob/16958153b1e01dcbf5d4334394a14715438b53f1/botocore/data/codebuild/2016-10-06/service-2.json#L1413 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codebuild-project-environment.html#cfn-codebuild-project-environment-computetype https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-compute-types.html AWS::CloudFront::Distribution.SslSupportMethod should pick up static-ip: https://github.com/boto/botocore/blob/ae2d0855ac3162520e584a24f5522b633de4e2a9/botocore/data/cloudfront/2020-05-31/service-2.json#L6101 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cloudfront-distribution-viewercertificate.html#cfn-cloudfront-distribution-viewercertificate-sslsupportmethod AWS::WAFv2::RuleGroup.Rate.AggregateKeyType should pick up FORWARDED_IP: https://github.com/boto/botocore/blob/b9b494aac2a0dc03433a8cda711d3dfabb4e5b45/botocore/data/wafv2/2019-07-29/service-2.json#L2715 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-wafv2-rulegroup-ratebasedstatementone.html#cfn-wafv2-rulegroup-ratebasedstatementone-aggregatekeytype https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-wafv2-rulegroup-ratebasedstatementtwo.html#cfn-wafv2-rulegroup-ratebasedstatementtwo-aggregatekeytype https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-wafv2-webacl-ratebasedstatementone.html#cfn-wafv2-webacl-ratebasedstatementone-aggregatekeytype https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-wafv2-webacl-ratebasedstatementtwo.html#cfn-wafv2-webacl-ratebasedstatementtwo-aggregatekeytype AWS::EC2::LaunchTemplate.TagSpecification.ResourceType should pick up 42 values: https://raw.githubusercontent.com/boto/botocore/master/botocore/data/ec2/2016-11-15/service-2.json https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-launchtemplate-tagspecification.html#cfn-ec2-launchtemplate-tagspecification-resourcetype AWS::Glue::Connection.ConnectionInput.ConnectionType picks up NETWORK: https://github.com/boto/botocore/blob/ae2d0855ac3162520e584a24f5522b633de4e2a9/botocore/data/glue/2017-03-31/service-2.json#L3372 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-connection-connectioninput.html#cfn-glue-connection-connectioninput-connectiontype
6bba8c0
to
a03242f
Compare
0456cf4
to
6bf11af
Compare
6bf11af
to
553e7e1
Compare
there are 145 types from 37 services, so this should reduce requests ~4x $ time cfn-lint --update-specs # before real 10m3.694s user 2m12.121s sys 0m27.602s $ time cfn-lint --update-specs # after real 2m15.260s user 0m52.673s sys 0m9.445s
src/cfnlint/maintenance.py
Outdated
service = '/'.join(service_and_type.split('/')[:-1]) | ||
botocore_type = service_and_type.split('/')[-1] | ||
if service not in botocore_cache: | ||
botocore_cache[service] = json.loads(get_url_content('https://raw.githubusercontent.com/boto/botocore/master/botocore/data/' + service + '/service-2.json')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another option here is
botocore_cache[service] = json.loads(get_url_content('https://raw.githubusercontent.com/boto/botocore/master/botocore/data/' + service + '/service-2.json')) | |
botocore_cache[service] = json.loads(pkgutil.get_data('botocore', os.path.join('data', service, 'service-2.json'))) |
While we don't directly rely on botocore but it is getting loaded because of the SAM dependencies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the biggest decision I've been debating
This would speed --update-specs
up significantly, but would grab outdated information if botocore
is outdated
$ time cfn-lint --update-specs # pulling from botocore latest on Github
real 2m15.260s
$ time pip3 install botocore --upgrade
real 0m15.649s
$ time cfn-lint --update-specs # pulling directly from local copy of botocore
real 0m24.975s
# might be able to pull latest data faster than these individual requests though
$ time git clone https://github.com/boto/botocore/ --depth 1
real 0m7.271s
$ time curl -s -L https://codeload.github.com/boto/botocore/zip/master > botocore.zip
real 0m2.605s
$ time unzip botocore.zip
real 0m2.074s
45e973e
to
74b5e74
Compare
have you ensured that there is 0 difference in the json files before this PR and with this PR? |
the difference in |
…lumeType AllowedValues expansion (io2) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-imagebuilder-imagerecipe-ebsinstanceblockdevicespecification.html#cfn-imagebuilder-imagerecipe-ebsinstanceblockdevicespecification-volumetype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
…lumeType AllowedValues expansion (io2) (#1824) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-imagebuilder-imagerecipe-ebsinstanceblockdevicespecification.html#cfn-imagebuilder-imagerecipe-ebsinstanceblockdevicespecification-volumetype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
…DOWS_SERVER_2019_CONTAINER) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codebuild-project-environment.html#cfn-codebuild-project-environment-type could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
…DOWS_SERVER_2019_CONTAINER) (#1825) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codebuild-project-environment.html#cfn-codebuild-project-environment-type could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
…pansion (NETWORK) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-connection-connectioninput.html#cfn-glue-connection-connectioninput-connectiontype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
…pansion (NETWORK) (#1826) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-connection-connectioninput.html#cfn-glue-connection-connectioninput-connectiontype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
#50 one of the few AllowedValues lists left to actively maintain: #1682 #1680 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-amazonmq-broker.html#cfn-amazonmq-broker-engineversion https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-engine.html https://aws.amazon.com/blogs/aws/amazon-mq-update-new-rabbitmq-message-broker-service/
#50 one of the few AllowedValues lists left to actively maintain: #1682 #1680 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-amazonmq-broker.html#cfn-amazonmq-broker-engineversion https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-engine.html https://aws.amazon.com/blogs/aws/amazon-mq-update-new-rabbitmq-message-broker-service/
…n (GENEVE) #649 #983 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticloadbalancingv2-listener.html#cfn-elasticloadbalancingv2-listener-protocol could be sourced automatically from botocore: #1682 https://github.com/boto/botocore/blob/ccdc98aa50929fab77cb29863301e90718f4a5ad/botocore/data/elbv2/2015-12-01/service-2.json#L2425
…yLoadBalancer) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpcendpoint.html#cfn-ec2-vpcendpoint-vpcendpointtype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
…yLoadBalancer) (#1833) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpcendpoint.html#cfn-ec2-vpcendpoint-vpcendpointtype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
…n (GENEVE) (#1784) #649 #983 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticloadbalancingv2-listener.html#cfn-elasticloadbalancingv2-listener-protocol could be sourced automatically from botocore: #1682 https://github.com/boto/botocore/blob/ccdc98aa50929fab77cb29863301e90718f4a5ad/botocore/data/elbv2/2015-12-01/service-2.json#L2425 Co-authored-by: Kevin DeJong <kddejong@amazon.com>
… expansion (FLINK-1_11) #1584 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesisanalyticsv2-application.html#cfn-kinesisanalyticsv2-application-runtimeenvironment could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
#50 one of the few AllowedValues lists left to actively maintain: #1682 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-amazonmq-broker.html#cfn-amazonmq-broker-engineversion https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-engine.html
…#1841) #50 one of the few AllowedValues lists left to actively maintain: #1682 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-amazonmq-broker.html#cfn-amazonmq-broker-engineversion https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-engine.html
…pansion https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-connection-connectioninput.html#cfn-glue-connection-connectioninput-connectiontype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
…pansion (#1852) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-connection-connectioninput.html#cfn-glue-connection-connectioninput-connectiontype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682
…aws-cloudformation#1841) aws-cloudformation#50 one of the few AllowedValues lists left to actively maintain: aws-cloudformation#1682 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-amazonmq-broker.html#cfn-amazonmq-broker-engineversion https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-engine.html
most commonly maintained concise constraint format last time I checked:
#50 (comment), terraform-linters/tflint#274, terraform-linters/tflint#421
CloudSpecs
is autogenerated bycfn-lint --update-specs
AllowedValues
were hardcoded inExtendedSpecs/all
and patched into all the Resource Specifications inCloudSpecs
whencfn-lint --update-specs
was run. Now they're references to botocore types inExtendedSpecs/all
andcfn-lint --update-specs
pulls from botocore directly to write that part ofCloudSpecs
AWS::WorkSpaces::Workspace.ComputeTypeName
should pick upPOWERPRO
andGRAPHICSPRO
AWS::CodeBuild::Project.Environment.Type
should pick upWINDOWS_SERVER_2019_CONTAINER
AWS::S3::Bucket.TopicConfiguration.Event
should pick ups3:ObjectRestore:*
ands3:Replication:*
EbsVolumeType
should pick upio2
from hereAWS::GuardDuty::IPSet.Format
should pick upALIEN_VAULT
andPROOF_POINT
andFIRE_EYE
AWS::CodeBuild::Project.Environment.ComputeType
should pick upBUILD_GENERAL1_2XLARGE
AWS::CloudFront::Distribution.SslSupportMethod
should pick upstatic-ip
AWS::WAFv2::RuleGroup.Rate.AggregateKeyType
should pick upFORWARDED_IP
AWS::Glue::Connection.ConnectionInput.ConnectionType
picks upNETWORK
AWS::EC2::LaunchTemplate.TagSpecification.ResourceType
should pick up 42 valuesOf the decent sized hardcoded
AllowedValues
lists left,AWS::SecretsManager::SecretTargetAttachment.TargetType
,,AWS::DMS::Endpoint.EngineName
AWS::Glue::Trigger.Condition.State
, andare the only ones that changed in 2020. Will watch their documentation with http://followthatpage.com/AWS::AmazonMQ::Broker.EngineVersion