Add API v1 (#2062, #2066, #759, #394)

This is a major rework of the API with a lot of breaking changes. Compared the the old API v0, API v1:

- is generated using a Jekyll Generator (see https://jekyllrb.com/docs/plugins/generators/),
- is versioned using the api/v1 prefix (#2066). This will make it easier to implement non-backward-compatible changes in API,
- feels more "Restful".

The API v0 is still generated to give time to users to migrate to API v1.

Changes in the "All products" endpoint:

- Path has been changed from api/all.json to api/v1/products/
- Response has been changed from a simple array to a JSON document. This made it possible to add endoflife-level data, such as the number of products.
- Array elements have been changed from a simple string to a full JSON document. This made it possible to expose new data, such as product category and tags (#2062).

Changes in the "Product" endpoint:

- Path has been changed from api/<product>.json to api/v1/products/<product>/.
- Response has been changed from a simple array to a JSON document. This made it possible to expose product-level data, such as product category and tags (#2062).
- Cycles data now always contain all the release cycles properties, even if they are null (example: discontinued, latest, latestReleaseDate, support...).

Changes in the "Cycle" endpoint:

- Path has been changed from api/<product>/<cycle>.json to api/v1/products/<product>/cycles/<cycle>/.
- Cycles data now always contain all the release cycles properties, even if they are null (example: discontinued, latest, latestReleaseDate, support...).
- A special /api/v1/products/<product>/cycles/latest/ cycle, containing the same data as the latest cycle, has been added (#2078).

New endpoints :

- /api/v1/categories/ - list categories used on endoflife.date
- /api/v1/tags/<tag> - list products having the given category
- /api/v1/tags/ - list tags used on endoflife.date
- /api/v1/tags/<tag> - list products having the given tag

Note that we thought of disabling API generation in development (using JEKYLL_ENV like the Jekyll Feed plugin - see https://github.com/jekyll/jekyll-feed/blob/master/lib/jekyll-feed/generator.rb#L145), but it was finally reverted. It does not work well with netlify preview, and generate production URL (i.e. https://endoflife.date URLs) in development which makes it difficult to use.

_layouts/json.json

@@ -0,0 +1 @@
+{{ page.data | jsonify }}

_plugins/generate-api-v1.rb

@@ -1,89 +1,226 @@
-#!/usr/bin/env ruby
-
 # This script creates API files for version 1 of the endoflife.date API.
 #
-# There are three kind of generated files :
-# - all.json: contains the list of all the product names.
-# - <product>.json: contains a given product data ()including releases data).
-# - <product>/<release>.json: contains a given product release data.
+# There are two kind of generated files :
+# - all.json: contains endoflife metadata, such as the list of all the products and the product count.
+# - <product>.json: contains a given product data (including releases data).
 
-require 'fileutils'
-require 'json'
-require 'yaml'
-require 'date'
+require 'jekyll'
 
 module ApiV1
 
-  # This API path
-  DIR = 'api/v1'
+  STRIP_HTML_BLOCKS = Regexp.union(
+    /<script.*?<\/script>/m,
+    /<!--.*?-->/m,
+    /<style.*?<\/style>/m
+  )
+  STRIP_HTML_TAGS = /<.*?>/m
 
-  # Returns the path of a file inside the API namespace.
-  def self.file(name, *args)
-    File.join(DIR, name, *args)
+  # Remove HTML from a string (such as an LTS label).
+  # This is the equivalent of Liquid::StandardFilters.strip_html, which cannot be used
+  # unfortunately.
+  def self.strip_html(input)
+    empty = ''.freeze
+    result = input.to_s.gsub(STRIP_HTML_BLOCKS, empty)
+    result.gsub!(STRIP_HTML_TAGS, empty)
+    result
   end
 
-  # Holds information about a product.
-  Product = Class.new do
-    attr_accessor :data
-
-    # Initializes the product with the given product's markdown file.
-    # The markdown file is expected to contain a YAML front-matter with the appropriate properties.
-    #
-    # Copying the data makes it easier to process it.
-    def initialize(data)
-      @data = Hash.new
-      # The product name is derived from the product's permalink (ex. /debian => debian).
-      @data["name"] = data['permalink'][1..data['permalink'].length]
-      @data["title"] = data['title']
-      @data["category"] = data['category']
-      @data["iconSlug"] = data['iconSlug']
-      @data["permalink"] = data['permalink']
-      @data["versionCommand"] = data['versionCommand']
-      @data["auto"] = data.has_key? 'auto'
-      @data["releasePolicyLink"] = data['releasePolicyLink']
-      @data["releases"] = data['releases'].map do |release|
-        release_data = Hash.new
-        release_data["name"] = release['releaseCycle']
-        release_data["codename"] = release['codename']
-        release_data["releaseDate"] = release['releaseDate']
-        release_data["support"] = release['support']
-        release_data["eol"] = release['eol']
-        release_data["discontinued"] = release['discontinued']
-        release_data["lts"] = release['lts'] || false # lts is optional, make sure it always has a value
-        release_data["latest"] = release['latest']
-        release_data["latestReleaseDate"] = release['latestReleaseDate']
-        release_data
+  class ApiGenerator < Jekyll::Generator
+    safe true
+    priority :lowest
+
+    TOPIC = "EndOfLife API v1:"
+
+    def generate(site)
+      @site = site
+      Jekyll.logger.info TOPIC, "Generating API for products"
+
+      add_index_page(site)
+      product_pages = add_product_pages(site)
+      add_all_products_page(site, product_pages)
+
+      add_category_pages(site, product_pages)
+      add_all_categories_page(site, product_pages)
+
+      add_tag_pages(site, product_pages)
+      add_all_tags_page(site, product_pages)
+    end
+
+    private
+
+    def add_index_page(site)
+      site_url = site.config['url']
+      site.pages << JsonPage.new(site, '/', 'index', {
+        result: [
+          { name: "products", uri: "#{site_url}/api/v1/products/" },
+          { name: "categories", uri: "#{site_url}/api/v1/categories/" },
+          { name: "tags", uri: "#{site_url}/api/v1/tags/" },
+        ]
+      })
+    end
+
+    def add_product_pages(site)
+      product_pages = []
+
+      site.pages.each do |page|
+        if page.data['layout'] == 'product'
+          product_pages << page
+          site.pages << ProductJsonPage.new(site, page)
+
+          site.pages << ProductCycleJsonPage.new(site, page, page.data['releases'][0], 'latest')
+          page.data['releases'].each do |cycle|
+            site.pages << ProductCycleJsonPage.new(site, page, cycle)
+          end
+        end
+      end
+
+      return product_pages
+    end
+
+    def add_all_products_page(site, products)
+      site.pages << ProductsJsonPage.new(site, '/products/', 'index', products)
+    end
+
+    def add_category_pages(site, products)
+      pages_by_category = {}
+
+      products.each do |product|
+        category = product.data['category'] || 'unknown'
+        add_to_map(pages_by_category, category, product)
+      end
+
+      pages_by_category.each do |category, products|
+        site.pages << ProductsJsonPage.new(site, "/categories/#{category}", 'index', products)
       end
     end
 
-    def name
-      data["name"]
+    def add_all_categories_page(site, products)
+      site.pages << JsonPage.new(site, '/categories/', 'index', {
+        result: products
+                  .map { |product| product.data['category'] }
+                  .uniq
+                  .sort
+                  .map { |category| { name: category, uri: "#{site.config['url']}/api/v1/categories/#{category}/" } }
+      })
+    end
+
+    def add_tag_pages(site, products)
+      products_by_tag = {}
+
+      products.each do |product|
+        product.data['tags'].each { |tag| add_to_map(products_by_tag, tag, product) }
+      end
+
+      products_by_tag.each do |tag, products|
+        site.pages << ProductsJsonPage.new(site, "/tags/#{tag}", 'index', products)
+      end
+    end
+
+    def add_all_tags_page(site, products)
+      site.pages << JsonPage.new(site, '/tags/', 'index', {
+        result: products
+                  .flat_map { |product| product.data['tags'] }
+                  .uniq
+                  .sort
+                  .map { |tag| { name: tag, uri: "#{site.config['url']}/api/v1/tags/#{tag}/" } }
+      })
+    end
+
+    def add_to_map(map, key, page)
+      if map.has_key? key
+        map[key] << page
+      else
+        map[key] = [page]
+      end
     end
   end
-end
 
-product_names = []
-FileUtils.mkdir_p(ApiV1::file('.'))
-
-Dir['products/*.md'].each do |file|
-  # Load and prepare data
-  raw_data = YAML.safe_load(File.open(file), permitted_classes: [Date])
-  product = ApiV1::Product.new(raw_data)
-  product_names.append(product.name)
-
-  # Write /<product>.json
-  product_file = ApiV1::file("#{product.name}.json")
-  File.open(product_file, 'w') { |f| f.puts product.data.to_json }
-
-  # Write all /<product>/<release>.json
-  FileUtils.mkdir_p(ApiV1::file(product.name))
-  product.data["releases"].each do |release|
-    # Any / characters in the name are replaced with - to avoid file errors.
-    release_file = ApiV1::file(product.name, "#{release['name'].to_s.tr('/', '-')}.json")
-    File.open(release_file, 'w') { |f| f.puts release.to_json }
+  class JsonPage < Jekyll::Page
+    def initialize(site, path, name, data)
+      @site = site
+      @base = site.source
+      @dir = "api/v1#{path}"
+      @name = "#{name}.json"
+      @data = {}
+      @data['layout'] = 'json'
+      @data['data'] = data
+
+      self.process(@name)
+    end
+  end
+
+  class ProductsJsonPage < JsonPage
+    def initialize(site, path, name, products)
+      super(site, path, name, {
+        total: products.size(),
+        products: products.map { |product| {
+          name: product.data['id'],
+          title: product.data['title'],
+          category: product.data['category'],
+          tags: product.data['tags'],
+          uri: "#{site.config['url']}/api/v1/products/#{product.data['id']}/",
+        } }
+      })
+    end
   end
-end
 
-# Write /all.json
-all_products_file = ApiV1::file('all.json')
-File.open(all_products_file, 'w') { |f| f.puts product_names.sort.to_json }
+  class ProductJsonPage < JsonPage
+    def initialize(site, product)
+      id = product.data['id']
+      super(site, "/products/#{id}", 'index', {
+        name: id,
+        label: product.data['title'],
+        category: product.data['category'],
+        tags: product.data['tags'],
+        links: {
+          icon: product.data['iconUrl'],
+          html: "#{site.config['url']}/#{id}",
+          releasePolicyLink: product.data['releasePolicyLink'],
+        },
+        versionCommand: product.data['versionCommand'],
+        lastModified: product.data['lastModified'],
+        auto: product.data.has_key?('auto'),
+        cycles: [
+          product.data['releases']
+                 .map { |cycle| {
+                   name: cycle['releaseCycle'],
+                   codename: cycle['codename'],
+                   label: ApiV1.strip_html(cycle['label']),
+                   date: cycle['releaseDate'],
+                   support: cycle['support'],
+                   lts: cycle['lts'],
+                   eol: cycle['eol'],
+                   discontinued: cycle['discontinued'],
+                   latest: {
+                     version: cycle['latest'],
+                     date: cycle['latestReleaseDate'],
+                     link: cycle['link'],
+                   }
+                 } }
+        ]
+      })
+    end
+  end
+
+  class ProductCycleJsonPage < JsonPage
+    def initialize(site, product, cycle, identifier = nil)
+      name = identifier ? identifier : cycle['id']
+
+      super(site, "/products/#{product.data['id']}/cycles/#{name}", 'index', {
+        name: cycle['releaseCycle'],
+        codename: cycle['codename'],
+        label: ApiV1.strip_html(cycle['label']),
+        date: cycle['releaseDate'],
+        support: cycle['support'],
+        lts: cycle['lts'],
+        eol: cycle['eol'],
+        discontinued: cycle['discontinued'],
+        latest: {
+          version: cycle['latest'],
+          date: cycle['latestReleaseDate'],
+          link: cycle['link'],
+        }
+      })
+    end
+  end
+end