MSP dashboard: API endpoint to get all software available for install for all teams

[noahtalerman]

Goal

User story
As an engineer working at a managed service provider (MSP) that's building a white-label MDM solution on top of Fleet,
I want to hit an API endpoint to list all the software that's available to install across all my teams in Fleet
so that I can render this list in a dropdown in Retool that my help desk uses to click to add software.

Key result

Deliver customer promises and prioritized requests

Original requests

• API endpoint for MSP help desk dashboard, which allows listing software for all teams #22403

Context

• Product designer: @rachaelshaw

Changes

Product

• MSP dashboard: New API endpoint: MSP dashboard: API endpoint to get all software available for install for all teams #23824 (comment)
• UI changes: No changes
• CLI (fleetctl) usage changes: No changes
• YAML changes: No changes
• REST API changes: No changes
• Fleet's agent (fleetd) changes: No changes
• Activity changes: No changes
• Permissions changes: No changes
• Changes to paid features or tiers: No changes
• Other reference documentation changes: No changes
• Once shipped, requester has been notified
• Once shipped, dogfooding issue has been filed

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Manual testing steps

• Set up:
  • Create an MSP dashboard that is connected to a Fleet instance that has self-service software assigned to teams.
  • Add at least one Linux, one macOS, and one Windows software to two different teams on the Fleet instance
• Send a GET request to the /api/v1/list-software with no authorization header.
  • It should respond with a 401 Unauthorized response.
• Go to your Fleet instance and copy your user's API token and send another request to the /api/v1/list-software with the authorization header set to Bearer ${apiToken}
  • It should return a list of software on the Fleet instance.
• Add the platform query parameter to filter the results and confirm that
  • GET /api/v1/list-software?platform=darwin returns a list of macOS software on the Fleet instance
    • Confirm that the nested teams array in the response contains the name and ID of each team the software is assigned to.
  • GET /api/v1/list-software?platofrm=windows Returns a list of Windows software on the Fleet instance
    • Confirm that the nested teams array in the response contains the name and ID of each team the software is assigned to.
  • GET /api/v1/list-software?platofrm=linux Returns a list of Linux software on the Fleet instance
    • Confirm that the nested teams array in the response contains the name and ID of each team the software is assigned to.
  • GET /api/v1/list-software?platofrm=chrome Returns a 400 bad request response.

Testing notes

Confirmation

1. Engineer (@____): Added comment to user story confirming successful completion of QA.
2. QA (@____): Added comment to user story confirming successful completion of QA.

[eashaw]

@rachaelshaw

Specification:

• Add a new endpoint: /list-software that:
  1. Send requests to the Fleet instance to collect information about installable software for all teams.
  2. Accept an optional platform parameter to filter the results to a specified platform. It can be one of darwin, windows, or linux.
  3. Build a JSON response that contains an array of all software installers on the Fleet instance
     1. Each item in the software array should have:
        • software_title_name - (string) The name of this software in the Fleet UI
        • installer_name - (string) The filename of this software's installer
        • software_title_id - (number) The Fleet API ID of this software title
        • installer_version - (string) The version of the software installer.
        • platform - (string) The platform the software is for. Either darwin, windows, or linux
        • teams - An array of the team specific information for a software installer
          • Each item in this array will have:
            • team_id - The Fleet API ID of the team this software is assigned to.
            • team_name - (string) The name of the team this software is assigned to.
• For authenticating requests:
  • Add an API key attribute to the User model.
  • Disable CSRF on /list-software endpoint
  • /list-software endpoint should return an unauthorized response if an API key is not provided via http header or doesn't match the API key of an existing user account.
  • Update the account settings page to display the user's API key.

Example API response

[
	{
		"software_title_name": "zoom.us.app",
		"installer_name": "Zoom.pkg",
		"software_title_id": 849,
		"installer_version": "6.2.3.40682",
		"teams": [
			{
				"id": 3,
				"team_name": "macOS Workstations"
			},
			{
				"id": 9,
				"team_name": "Team 2"
			}
			...
		]
	},
	{
		...
	}
]

[noahtalerman]

Hey @eashaw I think let's go w/ installer_version instead of version. That way, we're explicit that the version is for the installer (package).

What do you think?

cc @rachaelshaw

[eashaw]

@noahtalerman Makes sense! I updated the example API response.

[jmwatts]

@eashaw @noahtalerman I am trying to use https://fleetdm.com/guides/building-an-effective-dashboard-with-fleet-rest-api-flask-and-plotly to set up an MSP Dashboard but are there different instructions I should be following? Assistance would be appreciated. Thanks!

[eashaw]

@jmwatts Sorry about this, I met with Mike to review this PR last night and I made some changes that affected the QA steps in this issue. I updated the steps to reflect the new behavior.

The MSP dashboard is a separate node app in the ee/bulk-operations-dashboard/ folder. To QA this change, you should be able to follow the steps in this section of the app's README to start the dashboard. I can also help you get started if you want to schedule some time with me.

[jmwatts]

QA Notes:

Verified new /api/v1/list-software endpoint is available and returns results based on the specs above.

@noahtalerman @eashaw One thing to note:
Reponse time is generally 1-3 minutes per request with 70 packages uploaded to 9 teams. If that is acceptable then this ticket is good to go.

bulk-operations-dashboard-1  | About a minute later...
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | <- GET /api/v1/list-software                  (58195ms 200)
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 3 minutes later...
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | <- GET /api/v1/list-software                  (59334ms 200)
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 2 minutes later...
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | <- GET /api/v1/list-software                  (57760ms 200)
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 3 minutes later...
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | <- GET /api/v1/list-software                  (57713ms 200)
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | 3 minutes later...
bulk-operations-dashboard-1  | 
bulk-operations-dashboard-1  | <- GET /api/v1/list-software                  (57090ms 200)

[noahtalerman]

Response time is generally 1-3 minutes per request with 70 packages uploaded to 9 teams. If that is acceptable then this ticket is good to go.

@jmwatts thanks! I don't think that's accetable. @pintomi1989 I don't think this improvement is ready for customer-deebradel to use in production.

@lukeheath, for when you're back, can you please work with Scott and Eric on improvements that could speed this up?

[jmwatts]

@eashaw or @noahtalerman can this be moved back to "Waiting" or "In Progress" as there isn't further QA action to be taken at this time?

[lukeheath]

@jmwatts Issues can be moved back to "In Progress" if they don't clear QA. I've updated.

@sgress454 If you are online before next sprint kick off could you chat with @eashaw to brainstorm how we could improve response time? If it's straight-forward you can go ahead and implement, but if it's a heavier lift we should discuss the architectural changes. Thanks!

[eashaw]

@jmwatts I updated the PR to have the new endpoint send multiple requests to the Fleet instance at once to improve the response time.