Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update glue.py with SNOWFLAKE validate type #2247

Merged
merged 1 commit into from
May 30, 2024

Conversation

evilensky
Copy link
Contributor

@evilensky evilensky commented May 28, 2024

The SNOWFLAKE validation type allows an AWS::Glue::Connection with the ConnectionType of SNOWFLAKE to be created.

This is not yet in any documentation nor is the underlying API documented :(

aws-cloudformation/cloudformation-coverage-roadmap#1810

This is not yet in any documentation nor is the underlying API documented :( 

aws-cloudformation/cloudformation-coverage-roadmap#1810
@ewascent
Copy link
Contributor

I think you need to add a target class in glue.py (e.g. JDBCTarget or MongoDBTarget) and add that to the list of targets in the Targets class dict.

For this connection I do not think this needs a default port in constants.py. Snowflake has a default port of 443. We both have a default https_port value, and the default of the Snowflake connection properties assume the port is 443 here

@markpeek
Copy link
Member

Thank you for the PR. As @ewascent mentions, there are likely more changes needed to fully support Snowflake. With this change were you able to bring up a valid CloudFormation stack with Snowflake?

@evilensky
Copy link
Contributor Author

evilensky commented May 30, 2024

Thank you for considering the PR.

We aren't currently using Crawler or Target so I'm unable to verify at this time.

and just like the SNOWFLAKE connection type, I cannot find documentation for SNOWFLAKE support in these objects either in Cloudformation or underlying API documentation, only this time there is no Roadmap card for them.

Right now I'm monkey patching the Validator with this change, and yes, it creates a snowflake Connection successfully.

image

There are some interesting internal validators in the Glue API:

SparkProperties.sfUrl: does not match the regex pattern .+[.]snowflakecomputing[.]com (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException; Request ID: 79f731f2-89f7-4f04-b32f-9eda6b6f3e02; Proxy: null)

I'm curious if there is a pattern for troposphere to validate this since the API type is String but the string must be of a Json of a particular shape, and some validation rules apply to the values.

@markpeek
Copy link
Member

I'm going to pull in this change since it doesn't really hurt and your screenshot shows it seems to work.

To your validation question, for validation of interconnected properties you would want to create a "class validator" for the object in validators/glue.py. I'll highlight this easy example of the EC2 Subnet class validator here that ensures Ipv6CidrBlock is set if AssignIpv6AddressOnCreation is used in the object.

@markpeek markpeek merged commit 91f8487 into cloudtools:main May 30, 2024
3 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants