AllowedValues from botocore during --update-specs #1682

PatMyron · 2020-09-05T19:58:31Z

most commonly maintained concise constraint format last time I checked:

src/cfnlint/data/ExtendedSpecs/all $ git grep 'AllowedValues":' | wc -l
AllowedValues: 187 # only 42 hardcoded AllowedValues lists remaining after this PR
NumberMin/NumberMax: 46
AllowedPatternRegex: 12
StringMin/StringMax: 8
ListMin/ListMax: 4
JsonMax: 2

#50 (comment), terraform-linters/tflint#274, terraform-linters/tflint#421

CloudSpecs is autogenerated by cfn-lint --update-specs

AllowedValues were hardcoded in ExtendedSpecs/all and patched into all the Resource Specifications in CloudSpecs when cfn-lint --update-specs was run. Now they're references to botocore types in ExtendedSpecs/all and cfn-lint --update-specs pulls from botocore directly to write that part of CloudSpecs

AWS::WorkSpaces::Workspace.ComputeTypeName should pick up POWERPRO and GRAPHICSPRO

AWS::CodeBuild::Project.Environment.Type should pick up WINDOWS_SERVER_2019_CONTAINER

AWS::S3::Bucket.TopicConfiguration.Event should pick up s3:ObjectRestore:* and s3:Replication:*

EbsVolumeType should pick up io2 from here

AWS::GuardDuty::IPSet.Format should pick up ALIEN_VAULT and PROOF_POINT and FIRE_EYE

AWS::CodeBuild::Project.Environment.ComputeType should pick up BUILD_GENERAL1_2XLARGE

AWS::CloudFront::Distribution.SslSupportMethod should pick up static-ip

AWS::WAFv2::RuleGroup.Rate.AggregateKeyType should pick up FORWARDED_IP

AWS::Glue::Connection.ConnectionInput.ConnectionType picks up NETWORK

AWS::EC2::LaunchTemplate.TagSpecification.ResourceType should pick up 42 values

Of the decent sized hardcoded AllowedValues lists left, AWS::SecretsManager::SecretTargetAttachment.TargetType, ~~AWS::DMS::Endpoint.EngineName~~, AWS::Glue::Trigger.Condition.State, and ~~AWS::AmazonMQ::Broker.EngineVersion~~ are the only ones that changed in 2020. Will watch their documentation with http://followthatpage.com/

src/cfnlint/maintenance.py

there are 145 types from 37 services, so this should reduce requests ~4x $ time cfn-lint --update-specs # before real 10m3.694s user 2m12.121s sys 0m27.602s $ time cfn-lint --update-specs # after real 2m15.260s user 0m52.673s sys 0m9.445s

src/cfnlint/maintenance.py

kddejong · 2020-09-23T15:21:18Z

src/cfnlint/maintenance.py

+                service = '/'.join(service_and_type.split('/')[:-1])
+                botocore_type = service_and_type.split('/')[-1]
+                if service not in botocore_cache:
+                    botocore_cache[service] = json.loads(get_url_content('https://raw.githubusercontent.com/boto/botocore/master/botocore/data/' + service + '/service-2.json'))


Another option here is

Suggested change

botocore_cache[service] = json.loads(get_url_content('https://raw.githubusercontent.com/boto/botocore/master/botocore/data/' + service + '/service-2.json'))

botocore_cache[service] = json.loads(pkgutil.get_data('botocore', os.path.join('data', service, 'service-2.json')))

While we don't directly rely on botocore but it is getting loaded because of the SAM dependencies.

This is the biggest decision I've been debating
This would speed --update-specs up significantly, but would grab outdated information if botocore is outdated

$ time cfn-lint --update-specs # pulling from botocore latest on Github real 2m15.260s $ time pip3 install botocore --upgrade real 0m15.649s $ time cfn-lint --update-specs # pulling directly from local copy of botocore real 0m24.975s # might be able to pull latest data faster than these individual requests though $ time git clone https://github.com/boto/botocore/ --depth 1 real 0m7.271s $ time curl -s -L https://codeload.github.com/boto/botocore/zip/master > botocore.zip real 0m2.605s $ time unzip botocore.zip real 0m2.074s

https://stackoverflow.com/questions/56170380/downloading-files-raw-githubusercontent-com-is-immensely-slow

#1682 (comment)

miparnisari · 2020-09-25T02:19:42Z

have you ensured that there is 0 difference in the json files before this PR and with this PR?

PatMyron · 2020-09-25T02:30:02Z

have you ensured that there is 0 difference in the json files before this PR and with this PR?

the difference in CloudSpecs is that all of these AllowedValues are now sorted and all the additional AllowedValues from botocore we were missing listed in the description

…lumeType AllowedValues expansion (io2) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-imagebuilder-imagerecipe-ebsinstanceblockdevicespecification.html#cfn-imagebuilder-imagerecipe-ebsinstanceblockdevicespecification-volumetype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

…lumeType AllowedValues expansion (io2) (#1824) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-imagebuilder-imagerecipe-ebsinstanceblockdevicespecification.html#cfn-imagebuilder-imagerecipe-ebsinstanceblockdevicespecification-volumetype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

…DOWS_SERVER_2019_CONTAINER) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codebuild-project-environment.html#cfn-codebuild-project-environment-type could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

…DOWS_SERVER_2019_CONTAINER) (#1825) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codebuild-project-environment.html#cfn-codebuild-project-environment-type could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

…pansion (NETWORK) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-connection-connectioninput.html#cfn-glue-connection-connectioninput-connectiontype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

…pansion (NETWORK) (#1826) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-connection-connectioninput.html#cfn-glue-connection-connectioninput-connectiontype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

#50 one of the few AllowedValues lists left to actively maintain: #1682 #1680 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-amazonmq-broker.html#cfn-amazonmq-broker-engineversion https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-engine.html https://aws.amazon.com/blogs/aws/amazon-mq-update-new-rabbitmq-message-broker-service/

…n (GENEVE) #649 #983 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticloadbalancingv2-listener.html#cfn-elasticloadbalancingv2-listener-protocol could be sourced automatically from botocore: #1682 https://github.com/boto/botocore/blob/ccdc98aa50929fab77cb29863301e90718f4a5ad/botocore/data/elbv2/2015-12-01/service-2.json#L2425

…yLoadBalancer) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpcendpoint.html#cfn-ec2-vpcendpoint-vpcendpointtype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

…yLoadBalancer) (#1833) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpcendpoint.html#cfn-ec2-vpcendpoint-vpcendpointtype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

…n (GENEVE) (#1784) #649 #983 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticloadbalancingv2-listener.html#cfn-elasticloadbalancingv2-listener-protocol could be sourced automatically from botocore: #1682 https://github.com/boto/botocore/blob/ccdc98aa50929fab77cb29863301e90718f4a5ad/botocore/data/elbv2/2015-12-01/service-2.json#L2425 Co-authored-by: Kevin DeJong <kddejong@amazon.com>

… expansion (FLINK-1_11) #1584 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesisanalyticsv2-application.html#cfn-kinesisanalyticsv2-application-runtimeenvironment could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

#50 one of the few AllowedValues lists left to actively maintain: #1682 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-amazonmq-broker.html#cfn-amazonmq-broker-engineversion https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-engine.html

…#1841) #50 one of the few AllowedValues lists left to actively maintain: #1682 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-amazonmq-broker.html#cfn-amazonmq-broker-engineversion https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-engine.html

…pansion https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-connection-connectioninput.html#cfn-glue-connection-connectioninput-connectiontype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

…pansion (#1852) https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-connection-connectioninput.html#cfn-glue-connection-connectioninput-connectiontype could be sourced automatically from botocore: aws-cloudformation/cfn-lint#1682

…aws-cloudformation#1841) aws-cloudformation#50 one of the few AllowedValues lists left to actively maintain: aws-cloudformation#1682 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-amazonmq-broker.html#cfn-amazonmq-broker-engineversion https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker-engine.html

PatMyron force-pushed the AllowedValues branch from 6bba8c0 to a03242f Compare September 22, 2020 00:55

sorting existing AllowedValues

3689083

PatMyron mentioned this pull request Sep 22, 2020

tests fail after 2020-12-31 #1705

Closed

new AllowedValues from botocore

3565486

PatMyron force-pushed the AllowedValues branch from 0456cf4 to 6bf11af Compare September 22, 2020 18:27

retrieving AllowedValues from botocore during --update-specs

553e7e1

PatMyron force-pushed the AllowedValues branch from 6bf11af to 553e7e1 Compare September 22, 2020 18:30

PatMyron commented Sep 22, 2020

View reviewed changes

src/cfnlint/maintenance.py Outdated Show resolved Hide resolved

PatMyron requested review from kddejong, miparnisari, fatbasstard, johnttompkins and omkhegde September 22, 2020 18:47

PatMyron marked this pull request as ready for review September 22, 2020 18:47

PatMyron changed the title ~~AllowedValues from botocore~~ AllowedValues from botocore during --update-specs Sep 22, 2020

PatMyron added 2 commits September 22, 2020 14:30

removing requests dependency

e770491

PatMyron mentioned this pull request Sep 23, 2020

Adding NETWORK Connection Type awsdocs/aws-cloudformation-user-guide#793

Closed

kddejong reviewed Sep 23, 2020

View reviewed changes

PatMyron added a commit that referenced this pull request Sep 24, 2020

directly handling object instead of text processing

45e973e

#1682 (comment)

directly handling object instead of text processing

74b5e74

#1682 (comment)

PatMyron force-pushed the AllowedValues branch from 45e973e to 74b5e74 Compare September 25, 2020 01:00

Merge branch 'master' into AllowedValues

cfec2eb

miparnisari approved these changes Sep 25, 2020

View reviewed changes

PatMyron merged commit 1f39305 into master Sep 25, 2020

PatMyron deleted the AllowedValues branch September 25, 2020 19:08

PatMyron mentioned this pull request Sep 25, 2020

Description of cfn-lint development environment and build steps #903

Closed

PatMyron mentioned this pull request Nov 3, 2020

AWS::ImageBuilder::ImageRecipe.EbsInstanceBlockDeviceSpecification.VolumeType AllowedValues expansion (io2) cloudtools/troposphere#1824

Merged

PatMyron mentioned this pull request Nov 3, 2020

AWS::CodeBuild::Project.Environment.Type AllowedValues expansion (WINDOWS_SERVER_2019_CONTAINER) cloudtools/troposphere#1825

Merged

PatMyron mentioned this pull request Nov 3, 2020

AWS::Glue::Connection.ConnectionInput.ConnectionType AllowedValues expansion (NETWORK) cloudtools/troposphere#1826

Merged

PatMyron mentioned this pull request Nov 12, 2020

AWS::AmazonMQ::Broker.EngineVersion AllowedValues expansion #1778

Merged

PatMyron mentioned this pull request Nov 14, 2020

AWS::ElasticLoadBalancingV2::Listener.Protocol AllowedValues expansion (GENEVE) #1784

Merged

PatMyron mentioned this pull request Nov 14, 2020

AWS::EC2::VPCEndpoint.VpcEndpointType AllowedValues expansion (GatewayLoadBalancer) cloudtools/troposphere#1833

Merged

PatMyron mentioned this pull request Dec 22, 2020

AWS::AmazonMQ::Broker.EngineVersion AllowedValues expansion (5.15.14) #1841

Merged

PatMyron mentioned this pull request Dec 30, 2020

AWS::Glue::Connection.ConnectionInput.ConnectionType AllowedValues expansion cloudtools/troposphere#1852

Merged

PatMyron mentioned this pull request Jan 12, 2021

Get value constraints from AWS registry types #1867

Merged

PatMyron mentioned this pull request Apr 8, 2021

removing AWS::AmazonMQ::Broker.EngineVersion AllowedValues #1975

Merged

PatMyron mentioned this pull request Sep 29, 2021

removing AWS::DMS::Endpoint.EngineName AllowedValues #2128

Merged

PatMyron mentioned this pull request Mar 4, 2022

[cfn-lint] E3030 for config resources #2228

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AllowedValues from botocore during --update-specs #1682

AllowedValues from botocore during --update-specs #1682

PatMyron commented Sep 5, 2020 •

edited

Loading

kddejong Sep 23, 2020 •

edited by PatMyron

Loading

PatMyron Sep 23, 2020 •

edited

Loading

miparnisari commented Sep 25, 2020

PatMyron commented Sep 25, 2020 •

edited

Loading

	botocore_cache[service] = json.loads(get_url_content('https://raw.githubusercontent.com/boto/botocore/master/botocore/data/' + service + '/service-2.json'))
	botocore_cache[service] = json.loads(pkgutil.get_data('botocore', os.path.join('data', service, 'service-2.json')))

AllowedValues from botocore during --update-specs #1682

AllowedValues from botocore during --update-specs #1682

Conversation

PatMyron commented Sep 5, 2020 • edited Loading

kddejong Sep 23, 2020 • edited by PatMyron Loading

Choose a reason for hiding this comment

PatMyron Sep 23, 2020 • edited Loading

Choose a reason for hiding this comment

miparnisari commented Sep 25, 2020

PatMyron commented Sep 25, 2020 • edited Loading

PatMyron commented Sep 5, 2020 •

edited

Loading

kddejong Sep 23, 2020 •

edited by PatMyron

Loading

PatMyron Sep 23, 2020 •

edited

Loading

PatMyron commented Sep 25, 2020 •

edited

Loading