Repository REST API

[denelon]

I'd like to be able to have a repository accessible via REST APIs rather than a signed MSIX package.

The API will need a full design. This will also have an impact on client behavior. The MSIX index package can become a backup mechanism if the APIs are offline, or the client is unable to connect.

Notes:
The client needs to be able to get the list of packages to enable "search"
The client needs to be able to get the manifest so it can "install"
This implies that the client will build a local cache of packages after retrieving the list so they can perform a local search.
In the future we may create a web based search API (TBD).
The contents of the Microsoft Community Package manager should be available via this API once it's built.

[svmastersamurai]

This aspect is super interesting to those of us in orgs that build and deploy internally built software.

As someone who has to manage chocolatey infrastructure: please please please keep this design simple.

As an operator I would love to be able to run a tool over "packages" to generate a static website. Assuming the API is of trivial complexity, the server might be about 20 or 30 lines of code in Go or I could use something like apache or nginx to serve the content. I could more easily leverage a CI/CD system to handle rebuilding the repo metadata when there is a change.

For example we looked into sleet but chocolatey speaks an older dialect of nuget and that ended up being non-viable.

I think other package managers like RPM and apt are great examples to draw inspiration from.

The tools for repo management should ideally be cross-platform so we could build or manage changes to repo metadata on either POSIX-based OSes or Windows.

[denelon]

@svmastersamurai That's great feedback. I like the idea of running the tool over the packages. Would you think in terms of UNC style path's to the binaries or would HTTP still be OK?

I'm thinking about a concept where you might have the installers in a structure that looks like the manifests directory in the repository here, and then having the manifests able to co-locate in the same structure. If the installers are MSIX, we can pull nearly all of the meta-data to generate the manifest based on what is found.

[svmastersamurai]

@denelon I think to generate a static website that makes perfect sense. This assumes that there will be adequate saturation of MSIX usage of course! There are a lot of legacy installers out there, so would this be able to also handle the usual suspects we know about?

Our team manages multiple operating systems so we tend to prefer HTTP since it is usable cross platform. We use the same webservers to serve installers for Windows that we do for Munki packages on MacOS and RPMs for our fedora fleet.

One thing we do for redundancy is have a few locations to retrieve the installers from. It appears the package spec supports this, so from a tooling perspective if I have 5 mirrors with different hostnames it would be nice to be able to have the tool aware of the mirrors (in a simplistic sense, I can just give it a list in a file for example) and be able to append those sources to the specs if they don't exist. From the winget-cli's perspective it could cycle through the mirrors until it finds one that is accessible. This would probably lead to templating within the package spec, which I admittedly haven't dived too much into this project to know if that's supported or if there was a plan for that already 😄.

[majkinetor]

As an operator I would love to be able to run a tool over "packages" to generate a static website.

This deserves a separate ticket IMO and has nothing to do with REST interface to repository. Such tool would indeed be great for multiple reasons, one of which is that it allows anybody to keep their public list of packages and host it anywhere (i.e. github pages and friends).

Such tool might not be practical in producing all desired functionality of REST backend such as versioning and dependencies although it could go long way into that direction.

[denelon]

We are actively working on the REST API and our initial implementation in the client. The schemas for validating the new YAML syntax have been checked in, but may very well churn a bit. https://github.com/microsoft/winget-cli/tree/master/schemas/JSON/manifests/v1.0.0

The REST API will have it's own object structure, but is expected to be essentially conforming to the same meta-data.

[denelon]

We have a prototype with install and search. The Windows Package Manager v0.2.10771 Preview has an experimental implementation for install and search. As soon as we round out the other commands, we will share the swagger YAML.

[jedieaston]

I was wondering, in this first pass of the API spec, is there anything about authentication? Something that would be a dream feature is if there is a way it can use Azure AD SSO (or another SSO provider that uses the Windows Credential Provider API) to authenticate, so that we don't have to have the repo open to everyone on the network (or the internet). For now, of course, we could probably use the custom headers stuff to send a secret we get some other way, but at some point it would be nice to have something a bit more robust.

[denelon]

@jedieaston we don't currently have anything for authentication in the API yet. We've had several different suggestions for which mechanisms to use. I expect to create an "Issue-Feature" as soon as we open up the reference implementation repository. This is clearly a high-value use case. We do have a "getInformation" endpoint to query a source for the protocol versions supported. This may be a reasonable place to provide authentication mechanisms as well.

[ghost1372]

Hi @denelon
Is there any documentation to use Rest API?

[denelon]

It's coming soon [TM] with a reference implementation.

[ghost1372]

Thanks, where can we find the documentation when it was published?

[denelon]

It's another GitHub repository. I'll have copious documentation updates to make as soon as we flip the repository to public, Think in terms of days not weeks. It does include Swagger documentation.

[denelon]

@ghost1372 https://github.com/microsoft/winget-cli-restsource is where the reference implementation is.

[ghost1372]

Thank you

[mherrmann]

Hi @denelon, may I ask out of curiosity what the motivation for the REST API is vs. a UNC source of offline manifest files (#160)? What I don't understand is that it seems to me that a UNC source will always be easier to create and deploy than a REST API. What does the REST API give us in return, in terms of use cases that can be implemented?

Thank you!
Michael

[denelon]

@mherrmann we had more asks for a REST API during early feedback cycles with Microsoft MVPs and ISVs than a UNC implementation. The two primary use cases at the time were hosting repositories for software distribution by ISVs (to the public), and enterprise customers hosting internal repositories for a distributed workforce.

[mherrmann]

@denelon I see, thank you very much for the prompt response. I am working in a tangential space and would love to get a better feel for the industry opinion. Is there any chance the feedback cycles you mentioned are public or documented somewhere? If not, can you recommend people (eg. Microsoft MVPs) I might reach out to to find out more?

Thank you again!
Best,
Michael

[denelon]

@mhermann those early discussions were under NDA before the preview launched. I'd be willing to reach out via Twitter if you have some specific questions to get some feedback.

[mherrmann]

@denelon I understand, and thank you. I'd like to understand why people asked for REST APIs, when static files would be easier to host. I would love to understand which features people want that require a dynamic approach such as REST and cannot be implemented with static files.

My twitter is @m_herrmann if you want to include it. Thank you 🙏

[matsmcp]

@denelon I understand, and thank you. I'd like to understand why people asked for REST APIs, when static files would be easier to host. I would love to understand which features people want that require a dynamic approach such as REST and cannot be implemented with static files.

My twitter is @m_herrmann if you want to include it. Thank you 🙏

I also would like to upvote a more static approach on a Webserver.
The scenario is both for the MSP use case where multiple Customers would use the MSP:s server for MSP created packages and for larger organisations with multiple domains where a fileshare is unpractical. Having a webserver provide a static site with the packages and a contentfile (ie packages.tgz in a Unix world) would be very simple to implement and also be completly plattform independant.

This also needs a Winget client that is usable over SSH/winrm to be usefull

[denelon]

@mherrmann I think I must have missed an e-mail notification from GitHub on this. A couple of other features in the works related to Delivery Optimization are likely to have an impact on the API and subsequently a UNC driven approach. I believe it's still possible to generate a static site with the appropriate values to support the client search, install, upgrade, and list behaviors. We just went down the dynamic path due to the scope/scale of duplicating the Windows Package Manager Community App Repository. We have some active development in the area currently, and we're also looking at possibly refactoring the search interface.

[mherrmann]

@denelon I see. Thank you for clarifying :-)