Skip to content

Latest commit

 

History

History
477 lines (392 loc) · 18.8 KB

rep-0137.rst

File metadata and controls

477 lines (392 loc) · 18.8 KB
Raw

REP: 137 Title: ROS distribution files Author: Tully Foote, Dirk Thomas, Paul Mathieu Status: Accepted Type: Standards Track Content-Type: text/x-rst Created: 04-Feb-2013 Post-History: 19-April-2013

Abstract

This REP specifies a set of files which define ROS distributions and facilitate the building, packaging, testing and documenting process. The intention is to formalize the existing infrastructure and simplify hosting of (potentially customized) ROS buildfarms.

Motivation

A ROS distribution consists of numerous software packages maintained by various contributors forming a coherent set. Until now there was only one buildfarm responsible for running unit tests and generating Debian packages for various Ubuntu versions and no formal description of the semantics of the files used in that process was given so far. Expanding the current capabilities to support other platforms and architectures will be important extensions.

This REP aims to specify all information and file formats necessary to release and package ROS distributions for various platforms and architectures. By formalizing these data structures it should be enabled to:

  • release packages using bloom [1] for all targeted platforms
  • build binary packages for selected platforms and architectures
  • run unit tests on selected platforms and architectures
  • generation documentation

The use cases this REP is designed to address are:

  1. simplify the setup of a buildfarm which builds, tests and documents a ROS distribution, i.e. building, testing and documenting individual ROS distributions on separate buildfarms
  2. enable building binary packages for (experimental) platforms and architectures
  3. enable building binary packages of custom packages on top of the existing ROS distributions for publicly or privately hosted projects

Design requirements

The information formalized in this REP is used in four separate processes:

  • releasing a package This is the process of exporting upstream source and generating platform specific build files specific meta-information)
  • building binary packages for a specific platform and architecture
  • running automated tests
  • running documentation jobs

There should be a single configuration which contains (or references) all information required to run the above processes. The configuration must be easily readable and editable by both humans as well as machines and be extendable with future functionality.

The set of packages for which to build binary packages must be configurable for each platform and architecture.

The release process should stay independent of the build process. The testing can be performed on either the upstream source or on the released tarball (depending on if the upstream source contains enough information to run the test suite). The build process can only work on released packages.

Each configuration file must contain a label which identifies the semantics of the file as well as a version number to enable future format updates.

Rationale

A single configuration containing or referencing all required information allows for a single entry point for automated processing in the above mentioned processes.

Specifying individual subsets of packages per platform and architecture is necessary since not all released packages can and should be build on all platforms and architectures.

The information for the various processes should be separated from each other to ease customizing individual processes. This is especially necessary since different processes could be run by separate entities.

Specification

This REP gives a formal specification for the files that are used for that. Currently the information is stored in: * distribution files (groovy.yaml) * targets.yaml * individual rosinstall files for documentation, and depends files

The changes to the distribution file are minimal, such as restructuring and adding type and version fields. The content of the targets.yaml file will be split and merged into the release, source and doc files and their corresponding building files.

The content of the individual rosinstall files is merged into a single file. The dependencies for documentation are replaced by referencing other repositories specified in the same file.

All current information can be converted into the new format using a script in the reference implementation.

File format

Files use YAML 1.1 for the sake of human and machine readability and write-ability. As a good practice, the files should contain a header such as:

%YAML 1.1
# ROS index|release|release-build|source|source-build|doc|doc-build file
# see REP 137: https://ros.org/reps/rep-0137.html
---

Index file

The index file acts as a single configuration entrypoint which lists all available ROS distributions, along with references pointing to further process specific information. The references can be either relative to the index file or absolute.

The information stored in the index file is:

  • distributions: a list of ROS distributions

    For each distribution further information are referenced:

    • release: reference to the release file
    • release_builds: list of references to the release-build files used to build the binary packages
    • release_cache: reference to a release cache. Whether this field is a dictionary, a list or a scalar is left as an implementation detail. The following examples will assume that the implementation necessitates an url.
    • source: reference to the source file
    • source_builds: list of references to the source-build files used to run the tests
    • doc: reference to the doc file
    • doc_builds: list of references to the doc-build files used to run the documentation

  • type: must be 'index'

  • version: version number, this REP describes version 1

Example

An index file referencing multiple distribution with their release, source and doc files and their corresponding build files.

%YAML 1.1
# ROS index file
# see REP 137: https://ros.org/reps/rep-0137.html
---
distributions:
  groovy:
    release: groovy/release.yaml
    release_builds: [groovy/release-build-ubuntu.yaml, releases/release-build-arm.yaml]
    release_cache: http://www.example.com/groovy-cache
    source: groovy/source.yaml
    source_builds: [groovy/source-build.yaml]
    doc: groovy/doc.yaml
    doc_builds: [groovy/doc-build.yaml]
  hydro:
    ...
type: index
version: 1

Release file

A release is identified by the code name of the ROS distribution in the index file. Each release file contains the following information:

  • repositories: a list of GBP repositories which are identified by unique names

    • url: the URL of the release git repository The URL must be anonymously readable and (with the appropriate credentials) writable.

    • version: version number for which packages are released. This can be a tag, branch or hash.

    • status: defines the status of the packages of the repository. Can be one of the following: developed, maintained, unmaintained, end-of-life

    • status_description: an optional field describing in a short sentence the current status of the repository. For example detailing the reason for EOL and the recommended upgrade path.

    • packages: an optional list of package names. If no package is specified, one package with the name of the repository is assumed in the root of the repository. For each package name a dictionary with the following optional keys is possible:

      • subfolder: the relative path to the package from the repository base (default: package name)
      • status: overrides the repository-wide status
      • status_description: overrides the repository-wide status description

    • tags: a dict of tags which can be used by tools like bloom and the rosinstall generator, preventing the need for those tools to guess about tags in release repositories, e.g. 'release/package/1.2.3' vs 'release/groovy/package/1.2.3-4'. This provides a useful future proofing mechanism for tools which use tags in the release repository, rather than guessing the format of the tags they can infer them directly from the tag templates.

      • tag_name {release, debian, etc...}: Format of tags are strings with {template_variables} e.g. 'release/groovy/{package}/{version}' A non-exhaustive list of possible template tags:

        • package - name of the package which this release tag corresponds
        • version - full version of the package being released, e.g. 1.2.3-4
        • upstream_version - upstream version of package being released, e.g. 1.2.3
        • debian_distro - target debian distro codename
        • debian_package_name - name of package, with any prefix and sanitized for debian

        The only required tag_name is 'release', others like 'debian' are optional.

  • platforms: a dict of target platforms for which packages are released. Each key contains the OS name while the value is a list of OS code names. These OS names and OS code names are determined by rospkg.os_detect [4]. Each target platform will result in a different bloom release.

  • type: must be 'release'

  • version: version number, this REP describes version 1

Example: A release file listing GBP repositories and packages and the target platforms.

%YAML 1.1
# ROS release file
# see REP 137: https://ros.org/reps/rep-0137.html
---
repositories:
  actionlib:
    url: https://github.com/ros-gbp/actionlib-release.git
    version: 1.9.11-0
    tags:
      release: release/groovy/actionlib/{version}
  ar_track_alvar:
    url: https://github.com/ros-gbp/ar_track_alvar-release.git
    version: 0.3.0-0
    packages:
      ar_track_alvar:
        subfolder: artrackalvar
    tags:
      release: release/{package}/{upstream_version}
  bond_core:
    url: https://github.com/ros-gbp/bond_core-release.git
    version: 1.7.10-0
    packages:
      bond:
      bond_core:
      bondcpp:
      bondpy:
      smclib:
    tags:
      release: release/groovy/{package}/{version}
      debian: debian/{debian_package_name}_{version}_{debian_distro}
platforms:
  ubuntu: [oneiric, precise, quantal]
  debian: [wheezy]
type: release
version: 1

Release build file

A release build file contains the information necessary to build packages of the packages specified in the release file:

  • package_whitelist: a list of package names to build. If this is omitted all packages specified in the release file are build. Any upstream packages are implicitly included.
  • package_blacklist: a list of package names excluded from build. If this is omitted no packages are excluded. Any downstream package are implicitly excluded. The blacklist overrides the whitelist.
  • notifications: An optional section to configure email notifications.
    • emails: a list of email addresses to which to send all notification emails. (default: empty)
    • maintainers: a boolean flag used to enable email notification to listed maintainers. (default: false)
    • committers: a boolean flag used to enable email notification to committers. (default: false)
  • targets: a nested dict of targets for which packages are build. The first level key contains the OS name. The second level key contains the OS code name. The third level key contains the CPU architecture. The OS names and OS code names specified must be listed as a platform in the corresponding release or source file. Each level can contain a key _config which can contain a dict with arbitrary data. These configurations can be used to specify target specific information (e.g. apt_target_repository or yum_repository).
  • jenkins_url: the url to the associated Jenkins master
  • jenkins_sourcedeb_job_timeout: the timeout in minutes for the sourcedeb jobs (optional).
  • jenkins_binarydeb_job_timeout: the timeout in minutes for the binarydeb jobs (optional).
  • sync: specify the criteria that need to be fulfilled for packages to be synced. Any of the following options can be set and all of them must be fulfilled:
    • package_count: integer. Specifies how many packages need to be successfully built to perform a sync. (default: 0)
    • packages: list of package names. All the packages of this list must be successfully built to perform a sync. (default: [])
  • type: must be 'release-build'
  • version: version number, this REP describes version 1

Example: A build file selecting a subset of packages from the release or source file and specifying the platforms and architectures.

%YAML 1.1
# ROS release-build file
# see REP 137: https://ros.org/reps/rep-0137.html
---
package_whitelist: [ros_tutorials, common_tutorials]
package_blacklist: [turtlesim]
notifications:
  emails: [buildfarm-admin@example.com]
  maintainers: true
  committers: false
targets:
  _config:
    apt_target_repository: http://archives.example.com/ros
    apt_mirrors: [http://archives.example.com/ros, http://packages.foo.org/repos/example]
  ubuntu:
    oneiric:
      amd64:
      i386:
    precise:
      amd64:
      i386:
      armel:
jenkins_url: http://jenkins.example.com:8080
sync:
  packages: [ros_tutorials]
type: release-build
version: 1

Release cache file

Collection of all meta information of the ROS distribution, including all the information from the package.xml files. The cache must reference the release file and store a hash of the release file it was build from to be able to detect if the cache is invalid. The format of that cache is considered an implementation detail and is not specified in this REP.

Source file

The source file uses a specification similar to the release file, but does not have a list of platforms or additional metainformation for each repository (status, status_description, packages and tags). Since the repository type is not limited to git each repository has to specify a type.

  • repositories: a list of repositories which are identified by unique names
    • type: the type of SCM in use. Typically 'git', 'svn'...
    • url: the URL of the source repository The URL must be anonymously readable.
    • version: For git and hg this is the tag, branch or hash to be checked out. For svn the version should not be set since the branch/tag is encoded in the url.
  • type: must be 'source'
  • version: version number, this REP describes version 1

Source build file

The source build file uses a specification similar to the release build file, but does not have sync information. Also the white- and blacklist is on a repository level and does not consider any kind of dependencies.

  • repository_whitelist: a list of repository names to build. If this is omitted all repositories specified in the source file are build.
  • repository_blacklist: a list of repository names excluded from build. If this is omitted no repositories are excluded. The blacklist overrides the whitelist.
  • notifications: as specified for the release build file.
  • targets: as specified for the release build file.
  • jenkins_url: as specified for the release build file.
  • jenkins_job_timeout: the timeout in minutes for each job (optional).
  • type: must be 'source-build'
  • version: version number, this REP describes version 1

Documentation file

The documentation file uses a specification similar to the source file. Additionally each repository can specify a list of other repositories it depends on.

  • repositories: a list of repositories which are identified by unique names
    • type: the type of SCM in use. Typically 'git', 'svn'...
    • url: the URL of the source repository The URL must be anonymously readable.
    • version: For git and hg this is the tag, branch or hash to be checked out. For svn the version should not be set since the branch/tag is encoded in the url.
    • depends: list of repository names. Other repositories to perform cross referencing in the documentation. This is only necessary if the packages from the dependent repositories are not released. (default: [])
  • type: must be 'doc'
  • version: version number, this REP describes version 1

Documentation build file

The documentation build file uses a specification similar to the source build file.

  • repository_whitelist: a list of repository names to build. If this is omitted all repositories specified in the doc file are build.
  • repository_blacklist: a list of repository names excluded from build. If this is omitted no repositories are excluded. The blacklist overrides the whitelist.
  • notifications: as specified for the source build file.
  • targets: as specified for the source build file. But the list of targets must only have one entry.
  • jenkins_url: as specified for the source build file.
  • jenkins_job_timeout: the timeout in minutes for each job (optional).
  • doc_tag_index_repository: a repository storing the tag index
    • type: the type of SCM in use. Must be 'git'
    • url: the URL of the release git repository The URL must be writable (with the appropriate credentials).
    • version: This is the branch to be checked out and committed to.
  • type: must be 'doc-build'
  • version: version number, this REP describes version 1

Reference implementation

This REP is to be implemented in a Python module called rosdistro. This module will serve as a reference implementation for this REP. Any ROS tool requesting information defined in this REP should use this reference API implementation.

Compatibility issues

Since the new files are stored under a different URL (not in a releases subfolder in the rosdistro repository) the old tools will fail at some point when the information is not fetchable anymore form the old URL.

Affected tools

  • bloom [1]: relies on rosdep to retrieve the list of targets. Changes to bloom are thought to be minimal.
  • buildfarm: needs to know for which target/architecture it builds packages.
  • jenkins_scripts: provides the implementation of devel, doc and prerelease jobs
  • jenkins_tools: provides the devel and doc job generators
  • prerelease_website: needs a list of release and devel repositories
  • rosdep [2]: rosdep also provides a list of targets (the old targets.yaml file),
  • rosinstall_gen: needs a list of release and devel repositories
  • roslocate [3]: should be updated to use rosdistro

Use case examples

A complete example for a ROS distribution called foo with all the above specified files can be found at [5].

References

[1](1, 2) Bloom: a release automation tool (http://www.ros.org/wiki/bloom)
[2]rosdep2: https://ros.org/reps/rep-0125.html
[3]roslocate: http://www.ros.org/reps/rep-0115.html
[4]rospkg.os_detect: http://www.ros.org/reps/rep-0114.html
[5]Example files specifying a ROS distro: https://github.com/ros-infrastructure/rosdistro/tree/master/test/files

Copyright

This document has been placed in the public domain.