Added in Identity Center Nodes + relationship mapping #1380

SecPrez · 2024-11-10T14:17:56Z

Summary
Mapped in AWS Identity Center and the access it provides to AWS accounts.
New Nodes: (AWSIdentityCenter), (AWSPermissionSet), (AWSSSOUser)
New Relationships:
(AWSAccount)-[RESOURCE]->(AWSIdentityCenter)
(AWSIdentityCenter)-[HAS_PERMISSION_SET]->(AWSPermissionSet)
(AWSSSOUser)<-[ALLOWED_BY]-(AWSRole)
(OktaUser)<-[CAN_ASSUME_IDENTITY]-(AWSSSOUser)
(AWSPermissionSet)-[ASSIGNED_TO_ROLE]->(AWSRole)

Console Trace
INFO:cartography.intel.aws.identitycenter:Syncing Identity Center instances for region us-east-1 INFO:cartography.intel.aws.identitycenter:Loading 1 Identity Center instances for region us-east-1 INFO:cartography.intel.aws.identitycenter:Loading 32 permission sets for instance arn:aws:sso:::instance/ssoins-72237a0dcb8c6df7 in region us-east-1 INFO:cartography.intel.aws.identitycenter:Loading 777 permission set role assignments INFO:cartography.intel.aws.identitycenter:Loading 803 SSO users for identity store d-906747a0b9 in region us-east-1 INFO:cartography.intel.aws.identitycenter:Getting role assignments for 803 users INFO:cartography.intel.aws.identitycenter:Loading 24292 role assignments INFO:cartography.intel.aws.identitycenter:Syncing Identity Center instances for region us-east-2 INFO:cartography.intel.aws.identitycenter:Loading 0 Identity Center instances for region us-east-2 INFO:cartography.intel.aws.identitycenter:Syncing Identity Center instances for region us-west-1 INFO:cartography.intel.aws.identitycenter:Loading 0 Identity Center instances for region us-west-1 INFO:cartography.intel.aws.identitycenter:Syncing Identity Center instances for region us-west-2 INFO:cartography.intel.aws.identitycenter:Loading 0 Identity Center instances for region us-west-2 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement #1 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement #2 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement #3 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement #4 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement #5 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement #6

Related issues or links

Fixes - #990

Checklist
Provide proof that this works (this makes reviews move faster). Please perform one or more of the following:

[ x ] Update/add unit or integration tests.
[ X ] Include a screenshot showing what the graph looked like before and after your changes.
[ X ] Include console log trace showing what happened before and after your changes.
If you are changing a node or relationship:

[ x ] Update the schema and readme.
If you are implementing a new intel module:

[ X ] Use the NodeSchema data model.

SecPrez · 2024-11-11T12:55:41Z

@achantavy Hey Alex ... how are you today?

achantavy

Thank you! Gave it a first pass and left comments about the data model. If something doesn't make sense let's get on a call. I can also push to your branch to show you what I mean.

achantavy · 2024-11-12T04:45:02Z

cartography/intel/aws/identitycenter.py

+    """
+    logger.info(f"Loading {len(role_assignments)} role assignments")
+    if role_assignments:
+        neo4j_session.run(


Let's use the data model here. I think we can do this by creating a SSOUserToRole relationship.

We can have a transform step that shapes the data returned from get_sso_users() and get_role_assignments() into what we need.

This way we don't have to use neo4j_session.run() (since this can fail and does not retry automatically), and the data model does batching and automatic cleanups and all that.

Let's get on a call if you're stuck with this. I'm also happy to push changes up to your branch if that's easier.

Same issue as above. There is no easy transform as it depends on relationships based on data already in the graph. I have to find the specific role by doing

(acc:AWSAccount{id:ra.AccountId}) -[:RESOURCE]-> (role:AWSRole)<-[:ASSIGNED_TO_ROLE]- (permset:AWSPermissionSet {id: ra.PermissionSetArn})

I am looking for a role in the middle of the AWSAccount -> AWSPermissionSet path

achantavy · 2024-11-12T04:50:08Z

cartography/data/jobs/cleanup/aws_import_identity_center_cleanup.json

+{
+  "statements": [
+    {
+      "query": "MATCH (n:AWSIdentityCenterInstance) WHERE n.lastupdated <> $UPDATE_TAG WITH n LIMIT $LIMIT_SIZE DETACH DELETE (n) RETURN COUNT(*) as TotalDeleted",


Since the sync job runs one account at a time, this will currently delete all other AWSIdentityCenterInstance nodes in other accounts when we actually want to only delete the stale ones in the current account.

So since you've already built a data mode for this, you don't need to have a manually written cleanup job anymore and you can just do something like this:

cartography/cartography/intel/aws/ec2/instances.py

Lines 307 to 310 in bf8d667

def cleanup(neo4j_session: neo4j.Session, common_job_parameters: Dict[str, Any]) -> None:

logger.debug("Running EC2 instance cleanup")

GraphJob.from_node_schema(EC2ReservationSchema(), common_job_parameters).run(neo4j_session)

GraphJob.from_node_schema(EC2InstanceSchema(), common_job_parameters).run(neo4j_session)

It will automatically generate and run the cleanup queries for you so you don't have to worry about any mistakes.

Smart move. I will update

cartography/data/indexes.cypher

cartography/intel/aws/identitycenter.py

achantavy · 2024-11-12T04:55:36Z

cartography/models/aws/identitycenter/ssouser.py

+
+
+@dataclass(frozen=True)
+class SSOUserSchema(CartographyNodeSchema):


Add in a sub_resource_relationship with label RESOURCE to the AWSAccount. This makes it so that the autocleanup queries are easy to run.

cartography/models/aws/identitycenter/permissionsets.py

achantavy · 2024-11-12T04:57:54Z

cartography/models/aws/identitycenter/identitycenter.py

+
+
+@dataclass(frozen=True)
+class IdentiyCenterToAwsAccountRelProperties(CartographyRelProperties):


Typo: should be IdentityCenterToAwsAccountRelProperties.

Pending fix

### Summary > Describe your changes. We often need to connect 2 nodes based on a fuzzy match like in #1380 (comment). This PR adds a `fuzzy_and_ignore_case` option to the `PropertyRef` so that the final rendered query performs a `CONTAINS` in a case-insensitive way instead of an exact match. ### Related issues or links > Include links to relevant issues or other pages. #1380 ### Checklist Provide proof that this works (this makes reviews move faster). Please perform one or more of the following: - [x] Update/add unit or integration tests. - [ ] Include a screenshot showing what the graph looked like before and after your changes. - [ ] Include console log trace showing what happened before and after your changes. --------- Signed-off-by: Alex Chantavy <achantavy@lyft.com>

into identitycenter

SecPrez · 2024-11-19T15:57:57Z

All of the fixes have been resolved. Added Fuzzy matching and removed try-catch blocks around boto3

achantavy

Thanks for this! looks good. I need to figure out how to support some of the weird role mapping stuff with the model but that can come in a follow-up

achantavy · 2024-11-18T06:04:52Z

cartography/intel/aws/identitycenter.py

+        for page in paginator.paginate():
+            instances.extend(page.get('Instances', []))
+    except client.exceptions.ClientError as e:
+        logger.warning(f"Failed to get Identity Center instances in region {region}: {e}")


Can we remove the exception handling so that it looks like this instead?

cartography/cartography/intel/aws/ec2/network_interfaces.py

Lines 35 to 41 in c86e388

def get_network_interface_data(boto3_session: boto3.session.Session, region: str) -> List[Dict[str, Any]]:

client = boto3_session.client('ec2', region_name=region, config=get_botocore_config())

paginator = client.get_paginator('describe_network_interfaces')

subnets: List[Dict] = []

for page in paginator.paginate():

subnets.extend(page['NetworkInterfaces'])

return subnets

If we don't allow the sync to crash here, it will continue on to the cleanup job, and then the cleanup job will incorrectly believe that all previously synced identity center instances are stale and then delete them.

achantavy · 2024-11-18T23:53:40Z

cartography/intel/aws/identitycenter.py

+        MATCH (ps:AWSPermissionSet {arn: ra.PermissionSetArn})
+        MATCH (role:AWSRole)
+        WHERE role.arn STARTS WITH ra.RoleArn
+        MERGE (ps)-[r:ASSIGNED_TO_ROLE]->(role)


Fuzzy match

achantavy · 2024-11-18T23:55:25Z

cartography/intel/aws/identitycenter.py

+            MATCH (acc:AWSAccount{id:ra.AccountId}) -[:RESOURCE]->
+            (role:AWSRole)<-[:ASSIGNED_TO_ROLE]-
+            (permset:AWSPermissionSet {id: ra.PermissionSetArn})
+            MATCH (sso:AWSSSOUser {id: ra.UserId})


(notes to self) Per our convo: AWS identitycenter is weird. SSO users are associated with perm sets, and can be assigned to roles, but this assignment only takes place in one account.

…phy-cncf#1383) ### Summary > Describe your changes. We often need to connect 2 nodes based on a fuzzy match like in cartography-cncf#1380 (comment). This PR adds a `fuzzy_and_ignore_case` option to the `PropertyRef` so that the final rendered query performs a `CONTAINS` in a case-insensitive way instead of an exact match. ### Related issues or links > Include links to relevant issues or other pages. cartography-cncf#1380 ### Checklist Provide proof that this works (this makes reviews move faster). Please perform one or more of the following: - [x] Update/add unit or integration tests. - [ ] Include a screenshot showing what the graph looked like before and after your changes. - [ ] Include console log trace showing what happened before and after your changes. --------- Signed-off-by: Alex Chantavy <achantavy@lyft.com> Signed-off-by: chandanchowdhury <chandan.chowdhury@hotmail.com>

…cf#1380) **Summary** Mapped in [AWS Identity Center](https://aws.amazon.com/iam/identity-center/) and the access it provides to AWS accounts. New Nodes: (AWSIdentityCenter), (AWSPermissionSet), (AWSSSOUser) New Relationships: (AWSAccount)-[RESOURCE]->(AWSIdentityCenter) (AWSIdentityCenter)-[HAS_PERMISSION_SET]->(AWSPermissionSet) (AWSSSOUser)<-[ALLOWED_BY]-(AWSRole) (OktaUser)<-[CAN_ASSUME_IDENTITY]-(AWSSSOUser) (AWSPermissionSet)-[ASSIGNED_TO_ROLE]->(AWSRole) ![image](https://github.com/user-attachments/assets/e0e6c746-8ef6-4c89-b08a-d5192277fbda) ![image](https://github.com/user-attachments/assets/6ec645b8-6157-4001-b6f6-f44dbc3df2cc) **Console Trace** INFO:cartography.intel.aws.identitycenter:Syncing Identity Center instances for region us-east-1 INFO:cartography.intel.aws.identitycenter:Loading 1 Identity Center instances for region us-east-1 INFO:cartography.intel.aws.identitycenter:Loading 32 permission sets for instance arn:aws:sso:::instance/ssoins-72237a0dcb8c6df7 in region us-east-1 INFO:cartography.intel.aws.identitycenter:Loading 777 permission set role assignments INFO:cartography.intel.aws.identitycenter:Loading 803 SSO users for identity store d-906747a0b9 in region us-east-1 INFO:cartography.intel.aws.identitycenter:Getting role assignments for 803 users INFO:cartography.intel.aws.identitycenter:Loading 24292 role assignments INFO:cartography.intel.aws.identitycenter:Syncing Identity Center instances for region us-east-2 INFO:cartography.intel.aws.identitycenter:Loading 0 Identity Center instances for region us-east-2 INFO:cartography.intel.aws.identitycenter:Syncing Identity Center instances for region us-west-1 INFO:cartography.intel.aws.identitycenter:Loading 0 Identity Center instances for region us-west-1 INFO:cartography.intel.aws.identitycenter:Syncing Identity Center instances for region us-west-2 INFO:cartography.intel.aws.identitycenter:Loading 0 Identity Center instances for region us-west-2 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement #1 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement cartography-cncf#2 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement cartography-cncf#3 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement cartography-cncf#4 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement cartography-cncf#5 INFO:cartography.graph.statement:Completed aws_import_identity_center_cleanup statement cartography-cncf#6 **Related issues or links** Fixes - cartography-cncf#990 Checklist Provide proof that this works (this makes reviews move faster). Please perform one or more of the following: [ x ] Update/add unit or integration tests. [ X ] Include a screenshot showing what the graph looked like before and after your changes. [ X ] Include console log trace showing what happened before and after your changes. If you are changing a node or relationship: [ x ] Update the [schema](https://github.com/lyft/cartography/tree/master/docs/root/modules) and [readme](https://github.com/lyft/cartography/blob/master/docs/schema/README.md). If you are implementing a new intel module: [ X ] Use the NodeSchema [data model](https://cartography-cncf.github.io/cartography/dev/writing-intel-modules.html#defining-a-node). --------- Signed-off-by: chandanchowdhury <chandan.chowdhury@hotmail.com>

= added 4 commits November 10, 2024 15:15

Added in Identity Center Nodes + relationship mapping

98bafdc

adding cleanup job

d5ee38f

Change from linking to Okta user to linking to any UserAccount object

7c7a0fa

fix lint issue

33ba7ec

achantavy requested changes Nov 12, 2024

View reviewed changes

fixed code review issues

762d066

achantavy mentioned this pull request Nov 13, 2024

Add support for fuzzy case-insensitive match in PropertyRef #1383

Merged

3 tasks

= added 4 commits November 18, 2024 16:07

Merge branch 'master' of https://github.com/cartography-cncf/cartography

1f0d702

into identitycenter

Fixed code review feedback

c7d8bcf

Fixup the cleanup job

3a97a35

removed try catch blocks for the boto3 calls

c442b83

achantavy approved these changes Nov 19, 2024

View reviewed changes

achantavy merged commit 3ac1727 into cartography-cncf:master Nov 19, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added in Identity Center Nodes + relationship mapping #1380

Added in Identity Center Nodes + relationship mapping #1380

SecPrez commented Nov 10, 2024

SecPrez commented Nov 11, 2024

achantavy left a comment

achantavy Nov 12, 2024

SecPrez Nov 12, 2024

achantavy Nov 12, 2024

SecPrez Nov 12, 2024

achantavy Nov 12, 2024

SecPrez Nov 12, 2024

achantavy Nov 12, 2024

SecPrez Nov 12, 2024

SecPrez commented Nov 19, 2024

achantavy left a comment

achantavy Nov 18, 2024

achantavy Nov 18, 2024

achantavy Nov 18, 2024

	def cleanup(neo4j_session: neo4j.Session, common_job_parameters: Dict[str, Any]) -> None:
	logger.debug("Running EC2 instance cleanup")
	GraphJob.from_node_schema(EC2ReservationSchema(), common_job_parameters).run(neo4j_session)
	GraphJob.from_node_schema(EC2InstanceSchema(), common_job_parameters).run(neo4j_session)



		@dataclass(frozen=True)
		class SSOUserSchema(CartographyNodeSchema):



		@dataclass(frozen=True)
		class IdentiyCenterToAwsAccountRelProperties(CartographyRelProperties):

	def get_network_interface_data(boto3_session: boto3.session.Session, region: str) -> List[Dict[str, Any]]:
	client = boto3_session.client('ec2', region_name=region, config=get_botocore_config())
	paginator = client.get_paginator('describe_network_interfaces')
	subnets: List[Dict] = []
	for page in paginator.paginate():
	subnets.extend(page['NetworkInterfaces'])
	return subnets

Added in Identity Center Nodes + relationship mapping #1380

Added in Identity Center Nodes + relationship mapping #1380

Conversation

SecPrez commented Nov 10, 2024

SecPrez commented Nov 11, 2024

achantavy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SecPrez commented Nov 19, 2024

achantavy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment