Switch Python /simple and /<project>/ APIs to using the JSON-based format (PEP-691)

[jeffwidman]

Code improvement description

This is mostly a brain dump of a bunch of research I did this evening around the current state of PyPI APIs as part of #5723 and whether any changes can/should be made in :

Warehouse / PyPI exposes several JSON-based APIs:

/simple

• Provides an index of all packages.
• Defaults to HTML
• Can be requested in JSON format, as codified in PEP-691.
• The HTML version is currently used by Dependabot for fetching available versions:

  dependabot-core/python/lib/dependabot/python/update_checker/latest_version_finder.rb

  Lines 141 to 231 in efc538c

  	# See https://www.python.org/dev/peps/pep-0503/ for details of the
  	# Simple Repository API we use here.
  	def available_versions
  	@available_versions ||=
  	index_urls.flat_map do |index_url|
  	sanitized_url = index_url.gsub(%r{(?<=//).*(?=@)}, "redacted")
  	index_response = registry_response_for_dependency(index_url)
  	if index_response.status == 401 || index_response.status == 403
  	registry_index_response = registry_index_response(index_url)
  	if registry_index_response.status == 401 || registry_index_response.status == 403
  	raise PrivateSourceAuthenticationFailure, sanitized_url
  	end
  	end
  	version_links = []
  	index_response.body.scan(%r{<a\s.*?>.*?</a>}m) do
  	details = version_details_from_link(Regexp.last_match.to_s)
  	version_links << details if details
  	end
  	version_links.compact
  	rescue Excon::Error::Timeout, Excon::Error::Socket
  	raise if MAIN_PYPI_INDEXES.include?(index_url)
  	raise PrivateSourceTimedOut, sanitized_url
  	rescue URI::InvalidURIError
  	raise DependencyFileNotResolvable, "Invalid URL: #{sanitized_url}"
  	end
  	end
  	# rubocop:disable Metrics/PerceivedComplexity
  	def version_details_from_link(link)
  	doc = Nokogiri::XML(link)
  	filename = doc.at_css("a")&.content
  	url = doc.at_css("a")&.attributes&.fetch("href", nil)&.value
  	return unless filename&.match?(name_regex) || url&.match?(name_regex)
  	version = get_version_from_filename(filename)
  	return unless version_class.correct?(version)
  	{
  	version: version_class.new(version),
  	python_requirement: build_python_requirement_from_link(link),
  	yanked: link&.include?("data-yanked")
  	}
  	end
  	# rubocop:enable Metrics/PerceivedComplexity
  	def get_version_from_filename(filename)
  	filename.
  	gsub(/#{name_regex}-/i, "").
  	split(/-|\.tar\.|\.zip|\.whl/).
  	first
  	end
  	def build_python_requirement_from_link(link)
  	req_string = Nokogiri::XML(link).
  	at_css("a")&.
  	attribute("data-requires-python")&.
  	content
  	return unless req_string
  	requirement_class.new(CGI.unescapeHTML(req_string))
  	rescue Gem::Requirement::BadRequirementError
  	nil
  	end
  	def index_urls
  	@index_urls ||=
  	IndexFinder.new(
  	dependency_files: dependency_files,
  	credentials: credentials
  	).index_urls
  	end
  	def registry_response_for_dependency(index_url)
  	Dependabot::RegistryClient.get(
  	url: index_url + normalised_name + "/",
  	headers: { "Accept" => "text/html" }
  	)
  	end
  	def registry_index_response(index_url)
  	Dependabot::RegistryClient.get(
  	url: index_url,
  	headers: { "Accept" => "text/html" }
  	)
  	end

• Would be nice to migrate to the JSON variant as parsing static HTML is never fun
• Probably blocked by lack of support in private registry implementations:
  • Please add support for PEP-691: support JSON for the /simple API endpoint pypiserver/pypiserver#508
  • Support PEP-691: JSON response to the /simple API endpoint devpi/devpi#986
  • Artifactory: unknown
  • GitLab's pypi implementation: unknown
  • Cloudsmith: unknown
  • GemFury: unknown
  • Sonatype Nexus: unknown
  • Others??

/<package-name>/

• Provides some version details about the package
• Can be requested in JSON format, as codified in PEP-691.
• Used by Dependabot here:

  dependabot-core/python/lib/dependabot/python/update_checker/latest_version_finder.rb

  Lines 219 to 224 in efc538c

  	def registry_response_for_dependency(index_url)
  	Dependabot::RegistryClient.get(
  	url: index_url + normalised_name + "/",
  	headers: { "Accept" => "text/html" }
  	)
  	end

• Like /simple probably we can't migrate this to using the JSON API until/unless private registries support this.

/pypi/<package-name>/json

• A per-project JSON API: https://warehouse.pypa.io/api-reference/json.html#project.
• This has not yet been codified into a standard and per the comments on Request: PEP to describe current Warehouse JSON API pypa/packaging-problems#367 is heavily affected by PyPI implementation details, so may never get codified into a standard.
• Today we use this in Dependabot for Metadata fetching for Python packages:

  dependabot-core/python/lib/dependabot/python/metadata_finder.rb

  Line 165 in efc538c

  base_url.gsub(%r{/$}, "") + "/#{normalised_dependency_name}/json"

• Not supported by most private registry implementations:
  • support the PyPI JSON api devpi/devpi#801
  • Support pypi API for fetching package hashes/digests pypiserver/pypiserver#437
  • I'd be surprised if any of the hosted providers support this endpoint

Conclusion:

1. The one JSON API that's non-standard is the one we use in Dependabot, because that's the only way to retrieve the metadata information.
2. The other APIs for fetching available versions now have a PEP standardizing how they should expose their data via JSON in addition to static HTML.
3. Today Dependabot fetches these via static HTML.
4. We are probably blocked for the foreseeable future from migrating those APIs to use JSON because it'd break all the private registries.
5. And running both JSON and HTML parsing paths doesn't make a lot of sense, at least right now, because it adds complexity with no real benefit.

[jeffwidman]

This will likely have to sit on backlog for several years until PEP-691 (adopted last year) sees more widespread adoption.

[jeffwidman]

Related ticket with more technical info:

• Support PEP 658 / PEP 714 devpi/devpi#1018

[bewinsnw]

FWIW, dependabot is already broken on simple-only indexes. We're currently trying to use dependabot with an internal artifactory (with a simple index, no json) and dependabot tries to use json. I think the request is coming from here:

https://github.com/dependabot/dependabot-core/blob/e5ec7e979/python/lib/dependabot/python/update_checker.rb#L270

But with config like this:

version: 2
registries:
  python-artifactory:
    type: python-index
    url: https://redacted-internal-server-name/artifactory/api/pypi/pypi/simple/
    replaces-base: true
updates:
  - package-ecosystem: "pip"
    # dependabot will not run without this
    # https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#insecure-external-code-execution
    insecure-external-code-execution: allow
    directory: "/"
    registries:
      - python-artifactory
    schedule:
      interval: "daily"

We get errors like this in the dependabot on internal runners log:

  proxy | 2024/05/09 11***17***05 [021] GET https***//redacted-internal-server-name***443/artifactory/api/pypi/pypi/simple/smart-open/
2024/05/09 11***17***05 [021] 200 https***//redacted-internal-server-name***443/artifactory/api/pypi/pypi/simple/smart-open/
updater | 2024/05/09 11***17***05 INFO <job_826040134> Filtered out 2 pre-release versions
updater | 2024/05/09 11***17***05 INFO <job_826040134> Requirements to unlock own
2024/05/09 11***17***05 INFO <job_826040134> Requirements update strategy bump_versions
updater | 2024/05/09 11***17***05 INFO <job_826040134> Updating smart-open from 5.2.1 to 7.0.4
  proxy | 2024/05/09 11***17***06 [023] GET https***//redacted-internal-server-name***443/pypi/smart-open/json
  proxy | 2024/05/09 11***17***06 [023] 404 https***//redacted-internal-server-name***443/pypi/smart-open/json

Note those last two urls. dependabot is not respecting the config where we've told it the main url to use, but is inventing the path prefix to look for the index. However, since this index is a simple index, even if dependabot was using the correct path prefix, that would 404.