Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support For Concurrent Account-Based Resource Searches When Running With AWS Org Support #141

Closed
magneticstain opened this issue Sep 9, 2023 · 2 comments · Fixed by #213
Assignees
Labels
enhancement New feature or request epic A group of related stories or requirements
Milestone

Comments

@magneticstain
Copy link
Owner

magneticstain commented Sep 9, 2023

IP2Cr supports searching through all AWS Orgs accounts. For large organizations, that can mean a lot of accounts. In order to keep with the IP2CR tenancy of maximum speed, and ultimately to improve the user experience, we should add support for running concurrent resource searches, each working on a different account.


Architecture options:

  1. Start a pool of X amount of threads. Each thread is assigned an account, cycling through each if number of accounts > X
  2. Start a pool of X amount of threads. For each account, send the result set from AWS to be processed by each thread
  3. Start a pool of X amount of threads. Each thread is assigned an account, cycling through each if number of accounts > X. For each account thread, Y number of threads is created to process the resource set in batches. Essentially 1 + 2
@magneticstain magneticstain added the enhancement New feature or request label Sep 9, 2023
@magneticstain magneticstain transferred this issue from magneticstain/tf-ip2cr Sep 9, 2023
@magneticstain magneticstain changed the title Add Support For Concurrent Account-Based ResourceSsearches When Running With AWS Org Support Add Support For Concurrent Account-Based Resource Searches When Running With AWS Org Support Oct 1, 2023
@magneticstain magneticstain moved this from 📋 Backlog to 🔖 Ready in IP-2-CloudResource Oct 10, 2023
@magneticstain magneticstain self-assigned this Oct 10, 2023
@magneticstain magneticstain added this to the v1.0.3 milestone Oct 10, 2023
@magneticstain magneticstain added the epic A group of related stories or requirements label Oct 12, 2023
@magneticstain magneticstain moved this from 🔖 Ready to 🏗 In progress in IP-2-CloudResource Oct 22, 2023
@magneticstain
Copy link
Owner Author

I think we should start with implementing option 1, as it's the most simplest and straightforward option. Then, we can look into implementing 3 at a later date to potentially increase processing speed, at the cost of additional complexity.

@magneticstain
Copy link
Owner Author

Additional architecture details:

  1. Create a buffered receive channel set to receive 1 object
  2. Create a go routine for each account search, sending the channel object as a param
  3. In the secondary function, send the matching resource object via the channel, if found
  4. In the primary function, listen on the channel for a return value
  5. If value is returned, proceed. Otherwise, wait until all go routines are complete

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request epic A group of related stories or requirements
Projects
Archived in project
1 participant