From 32002c4fad78f580ae4a6972bb82a578ad61ff2d Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Thu, 6 Nov 2014 12:52:50 -0800 Subject: [PATCH 01/13] draft of REP 143 --- rep-0143.rst | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 rep-0143.rst diff --git a/rep-0143.rst b/rep-0143.rst new file mode 100644 index 000000000..669550595 --- /dev/null +++ b/rep-0143.rst @@ -0,0 +1,92 @@ +REP: 143 +Title: ROS distribution files +Author: Dirk Thomas +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 01-Sep-2014 +Post-History: 06-Nov-2013 + + +.. contents:: + +Abstract +======== +This REP updates the specification of the ROS distribution files facilitated in +the building, packaging, testing and documenting process. +The intention is to add further configuration options for hosting of +(potentially customized) ROS buildfarms. +This REP is a revised version of REP 141 [1]_. +It currently does not repeat the content of REP 141 but only states the +differences. + + +Specification +============= + +Index file +---------- + +* distributions + + * distribution: list of reference to the distribution files + A single distribution file guarantees that there are no duplicate + repositories. + But when rnuning a custom buildfarm which only adds custom repositories it + is much smoother to maintain them in a separate file to avoid any kind of + conflicts. + It also allows to override specific repositories with either a fork variant + or a specific version number. + + * release_builds / source_builds / doc_builds: dict of references to the + build files + The named key enables better tooling to e.g. generate the jobs of a + specific build file - rather then using the opaque list index. + +* version: version number, this REP describes version 3 (instead of version 2 + described in REP 141 [1]_ and version 1 described in REP 137 [2]_) + + +Distribution file +----------------- + +* repositories + + * source + + * test_commits: a boolean flag used to enable CI jobs for each commit to + the branch specified under ``versions``. (default: false) + * test_pull_requests: a boolean flag used to enable CI jobs for each pull + request against the branch specified under ``versions``. (default: false) + * version: if ``test_pull_requests`` is enabled the version must specify a + branch. + +* version: version number, this REP describes version 2 (instead of version 1 + described in REP 141 [1]_ and REP 137 [2]_) + + +Release build file +------------------ +* jenkins_job_priority: the priority number for the sourcedeb and binarydeb jobs (optional). + +* version: version number, this REP describes version 2 (instead of version 1 + described in REP 141 [1]_ and REP 137 [2]_) + + +Reference implementation +------------------------ +This REP is to be implemented in version 0.4 of the Python package *rosdistro*. +It will serve as a reference implementation for this REP. +A draft implementation can be found in the forked repository +[dirk-thomas/ros-infrastructure_rosdistro](https://github.com/dirk-thomas/ros-infrastructure_rosdistro). + + +References +========== +.. [1] REP 141: http://www.ros.org/reps/rep-0141.html +.. [2] REP 137: http://www.ros.org/reps/rep-0137.html + + +Copyright +========= +This document has been placed in the public domain. From 9930e9faaa32a2541f29500524bcf607cc88b030 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Mon, 10 Nov 2014 14:11:44 -0800 Subject: [PATCH 02/13] add abi_incompatibility_assumed flag to release build file --- rep-0143.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rep-0143.rst b/rep-0143.rst index 669550595..1c21098b4 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -67,6 +67,9 @@ Distribution file Release build file ------------------ +* abi_incompatibility_assumed: a boolean flag used to indicate that binary + do not guarantee ABI compatibility. (default: false) + * jenkins_job_priority: the priority number for the sourcedeb and binarydeb jobs (optional). * version: version number, this REP describes version 2 (instead of version 1 From 99519cf07ce76fec1727d900d9999c3d04c4d8ec Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Sun, 16 Nov 2014 14:30:08 -0800 Subject: [PATCH 03/13] remove reference to old email address --- rep-0133.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rep-0133.rst b/rep-0133.rst index c0c6a4dd5..726c950e2 100644 --- a/rep-0133.rst +++ b/rep-0133.rst @@ -1,6 +1,6 @@ REP: 133 Title: Separation of build environment and source tree tools -Author: Tully Foote , Dirk Thomas , Thibault Kruse , William Woodall +Author: Tully Foote , Dirk Thomas, Thibault Kruse , William Woodall Status: Active Type: Informational Content-Type: text/x-rst From 3449301f91bbeea1af16183d5a2fa87c28746c06 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 25 Nov 2014 12:05:07 -0800 Subject: [PATCH 04/13] remove build files from REP 143 --- rep-0143.rst | 49 +++++++++++++++++++++---------------------------- 1 file changed, 21 insertions(+), 28 deletions(-) diff --git a/rep-0143.rst b/rep-0143.rst index 1c21098b4..41b8b6ae4 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -5,7 +5,7 @@ Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 01-Sep-2014 -Post-History: 06-Nov-2013 +Post-History: 25-Nov-2014 .. contents:: @@ -14,9 +14,11 @@ Abstract ======== This REP updates the specification of the ROS distribution files facilitated in the building, packaging, testing and documenting process. -The intention is to add further configuration options for hosting of -(potentially customized) ROS buildfarms. -This REP is a revised version of REP 141 [1]_. + +The intention is to separate the buildfarm related configuration options from +the information describing the ROS packages and repositories [1]_ [2]_. + +This REP is a revised version of REP 141 [3]_. It currently does not repeat the content of REP 141 but only states the differences. @@ -32,19 +34,17 @@ Index file * distribution: list of reference to the distribution files A single distribution file guarantees that there are no duplicate repositories. - But when rnuning a custom buildfarm which only adds custom repositories it + But when running a custom buildfarm which only adds custom repositories it is much smoother to maintain them in a separate file to avoid any kind of conflicts. - It also allows to override specific repositories with either a fork variant - or a specific version number. + It also allows to override specific repositories with either a forked + variant or a specific version number. - * release_builds / source_builds / doc_builds: dict of references to the - build files - The named key enables better tooling to e.g. generate the jobs of a - specific build file - rather then using the opaque list index. + * release_builds / source_builds / doc_builds: are being removed since they + only contain information relevant to the buildfarm * version: version number, this REP describes version 3 (instead of version 2 - described in REP 141 [1]_ and version 1 described in REP 137 [2]_) + described in REP 141 [3]_ and version 1 described in REP 137 [4]_) Distribution file @@ -62,32 +62,25 @@ Distribution file branch. * version: version number, this REP describes version 2 (instead of version 1 - described in REP 141 [1]_ and REP 137 [2]_) - - -Release build file ------------------- -* abi_incompatibility_assumed: a boolean flag used to indicate that binary - do not guarantee ABI compatibility. (default: false) - -* jenkins_job_priority: the priority number for the sourcedeb and binarydeb jobs (optional). - -* version: version number, this REP describes version 2 (instead of version 1 - described in REP 141 [1]_ and REP 137 [2]_) + described in REP 141 [3]_ and REP 137 [4]_) Reference implementation ------------------------ This REP is to be implemented in version 0.4 of the Python package *rosdistro*. It will serve as a reference implementation for this REP. -A draft implementation can be found in the forked repository -[dirk-thomas/ros-infrastructure_rosdistro](https://github.com/dirk-thomas/ros-infrastructure_rosdistro). +A draft implementation can be found in the "rep143" branch of +[ros-infrastructure/rosdistro](https://github.com/ros-infrastructure/rosdistro/tree/rep143). References ========== -.. [1] REP 141: http://www.ros.org/reps/rep-0141.html -.. [2] REP 137: http://www.ros.org/reps/rep-0137.html +.. [1] Python module to for buildfarm configuration: + https://github.com/ros-infrastructure/ros_buildfarm/tree/master/ros_buildfarm/config +.. [2] Example buildfarm configuration: + https://github.com/ros-infrastructure/ros_buildfarm_config +.. [3] REP 141: http://www.ros.org/reps/rep-0141.html +.. [4] REP 137: http://www.ros.org/reps/rep-0137.html Copyright From 4d7a0d352234e7643317f7eade4cb4b113dce605 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 25 Nov 2014 12:15:38 -0800 Subject: [PATCH 05/13] add compatibility considerations --- rep-0143.rst | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/rep-0143.rst b/rep-0143.rst index 41b8b6ae4..9e274cd72 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -73,6 +73,41 @@ A draft implementation can be found in the "rep143" branch of [ros-infrastructure/rosdistro](https://github.com/ros-infrastructure/rosdistro/tree/rep143). +Compatibility issues +==================== + +The draft implementation of rosdistro is able to parse both index format +version 2 as well as 3. + +To leverage the new ability to list multiple distribution files the following +additional API will be introduced: + +* rosdistro.get_distribution_files() + +The existing API ('rosdistro.get_distribution_file()') will continue to provide +the correct data by merging the information from all distribution files. + +All function related to the removed build files will simply return empty lists. + +As soon as the new version of 'rosdistro' is released the 'ros/rosdistro' +repository [7]_ is being updated to the new format (for Hydro and higher). +Any client accessing trying to access the data with an old 'rosdistro' +version will get an error message like this: + +:: + + Unable to handle 'index' format version '2', please update rosdistro... + +Affected tools +-------------- + +The only tool beside the "old" buildfarm accessing the raw rosdistro data which +needs to be updated is 'rosdep' [5]_. + +All other tools using the API of rosdistro (even if deprecated) will still work +flawlessly. + + References ========== .. [1] Python module to for buildfarm configuration: @@ -81,6 +116,8 @@ References https://github.com/ros-infrastructure/ros_buildfarm_config .. [3] REP 141: http://www.ros.org/reps/rep-0141.html .. [4] REP 137: http://www.ros.org/reps/rep-0137.html +.. [5] Patch for rosdep: + https://github.com/ros-infrastructure/rosdep/compare/rep143 Copyright From 5b16b061477e4aa91dfecca7c9f054b783ae5ff6 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 25 Nov 2014 14:29:52 -0800 Subject: [PATCH 06/13] feedback --- rep-0143.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rep-0143.rst b/rep-0143.rst index 9e274cd72..acb1f11e9 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -96,7 +96,7 @@ version will get an error message like this: :: - Unable to handle 'index' format version '2', please update rosdistro... + Unable to handle 'index' format version '3', please update rosdistro... Affected tools -------------- From 0b9349392db5771e4908e855697ba9b30eb25289 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Sat, 29 Nov 2014 11:19:53 -0800 Subject: [PATCH 07/13] add note about bloom compatibility --- rep-0143.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rep-0143.rst b/rep-0143.rst index acb1f11e9..32f2ec823 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -107,6 +107,10 @@ needs to be updated is 'rosdep' [5]_. All other tools using the API of rosdistro (even if deprecated) will still work flawlessly. +In order to allow releasing packages into rosdistro repositories with multiple +distribution files 'bloom' will need to be updated. +That might also require to change the internal format of the rosdistro cache +files which is not explicitly specified in the REP. References ========== From 10dfab54b02b40f37aeaf075de44f3ac40c6ad64 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Sat, 29 Nov 2014 14:47:56 -0800 Subject: [PATCH 08/13] fix spelling --- rep-0143.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rep-0143.rst b/rep-0143.rst index 32f2ec823..1b97cd893 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -57,7 +57,7 @@ Distribution file * test_commits: a boolean flag used to enable CI jobs for each commit to the branch specified under ``versions``. (default: false) * test_pull_requests: a boolean flag used to enable CI jobs for each pull - request against the branch specified under ``versions``. (default: false) + request against the branch specified under ``version``. (default: false) * version: if ``test_pull_requests`` is enabled the version must specify a branch. From e1107df81ee64ffd63bb5175b983a3f2a88253ee Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Sun, 30 Nov 2014 15:48:20 -0800 Subject: [PATCH 09/13] update references --- rep-0143.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/rep-0143.rst b/rep-0143.rst index 1b97cd893..b107d83e8 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -69,8 +69,7 @@ Reference implementation ------------------------ This REP is to be implemented in version 0.4 of the Python package *rosdistro*. It will serve as a reference implementation for this REP. -A draft implementation can be found in the "rep143" branch of -[ros-infrastructure/rosdistro](https://github.com/ros-infrastructure/rosdistro/tree/rep143). +A draft implementation can be found in [5]_. Compatibility issues @@ -102,7 +101,7 @@ Affected tools -------------- The only tool beside the "old" buildfarm accessing the raw rosdistro data which -needs to be updated is 'rosdep' [5]_. +needs to be updated is 'rosdep' [6]_. All other tools using the API of rosdistro (even if deprecated) will still work flawlessly. @@ -120,8 +119,12 @@ References https://github.com/ros-infrastructure/ros_buildfarm_config .. [3] REP 141: http://www.ros.org/reps/rep-0141.html .. [4] REP 137: http://www.ros.org/reps/rep-0137.html -.. [5] Patch for rosdep: +.. [5] Patch to python-rosdistro: + https://github.com/ros-infrastructure/rosdistro/pull/45 +.. [6] Patch for rosdep: https://github.com/ros-infrastructure/rosdep/compare/rep143 +.. [7] ROS distro: + https://github.com/ros/rosdistro/ Copyright From c57264c15b60638200c073300a2bd7858c57eff4 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Sun, 30 Nov 2014 15:50:46 -0800 Subject: [PATCH 10/13] clarify overlay --- rep-0143.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/rep-0143.rst b/rep-0143.rst index b107d83e8..0998af45d 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -33,11 +33,15 @@ Index file * distribution: list of reference to the distribution files A single distribution file guarantees that there are no duplicate - repositories. + repositories and is therefore preferred. But when running a custom buildfarm which only adds custom repositories it is much smoother to maintain them in a separate file to avoid any kind of conflicts. - It also allows to override specific repositories with either a forked + + The files are read in the order they are listed and entries in later + distribution files are overwriting entries from previous distribution + files. + This can be used to override specific repositories with either a forked variant or a specific version number. * release_builds / source_builds / doc_builds: are being removed since they From 189d148cd6db1422b82b4b479b7c96b55a8145e2 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Thu, 4 Dec 2014 13:24:55 -0800 Subject: [PATCH 11/13] clarify how bloom interacts with multiple distribution files --- rep-0143.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rep-0143.rst b/rep-0143.rst index 0998af45d..7c7496d92 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -43,6 +43,9 @@ Index file files. This can be used to override specific repositories with either a forked variant or a specific version number. + ``bloom`` will create pull requests against the last distribution file + since even when the repository is present in any of the other distribution + files the "overlaying" version should be stored in the last one. * release_builds / source_builds / doc_builds: are being removed since they only contain information relevant to the buildfarm From 0bcd0d9e03b6f5a59c793021f2a319f7993b4a5e Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Thu, 4 Dec 2014 13:27:47 -0800 Subject: [PATCH 12/13] add tags to distribution files --- rep-0143.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/rep-0143.rst b/rep-0143.rst index 7c7496d92..d00b45e99 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -68,6 +68,11 @@ Distribution file * version: if ``test_pull_requests`` is enabled the version must specify a branch. +* tags: list of tag names to identify the group of repositories specified by + the distribution file. + It allows build files to refer to sets of repositories / packages based on in + which distribution file they are defined. + * version: version number, this REP describes version 2 (instead of version 1 described in REP 141 [3]_ and REP 137 [4]_) From 261935257e9e74084e578535ce8f8cb24212f194 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Thu, 4 Dec 2014 13:44:25 -0800 Subject: [PATCH 13/13] add reference to bloom patch --- rep-0143.rst | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/rep-0143.rst b/rep-0143.rst index d00b45e99..5e200f638 100644 --- a/rep-0143.rst +++ b/rep-0143.rst @@ -101,7 +101,7 @@ the correct data by merging the information from all distribution files. All function related to the removed build files will simply return empty lists. As soon as the new version of 'rosdistro' is released the 'ros/rosdistro' -repository [7]_ is being updated to the new format (for Hydro and higher). +repository [6]_ is being updated to the new format (for Hydro and higher). Any client accessing trying to access the data with an old 'rosdistro' version will get an error message like this: @@ -113,15 +113,13 @@ Affected tools -------------- The only tool beside the "old" buildfarm accessing the raw rosdistro data which -needs to be updated is 'rosdep' [6]_. +needs to be updated is 'rosdep' [7]_. -All other tools using the API of rosdistro (even if deprecated) will still work -flawlessly. +All other tools using the API of rosdistro (even if deprecated) will continue +to work. In order to allow releasing packages into rosdistro repositories with multiple -distribution files 'bloom' will need to be updated. -That might also require to change the internal format of the rosdistro cache -files which is not explicitly specified in the REP. +distribution files 'bloom' will need to be updated [8]_. References ========== @@ -133,10 +131,9 @@ References .. [4] REP 137: http://www.ros.org/reps/rep-0137.html .. [5] Patch to python-rosdistro: https://github.com/ros-infrastructure/rosdistro/pull/45 -.. [6] Patch for rosdep: - https://github.com/ros-infrastructure/rosdep/compare/rep143 -.. [7] ROS distro: - https://github.com/ros/rosdistro/ +.. [6] ROS distro: https://github.com/ros/rosdistro/ +.. [7] Patch for rosdep: https://github.com/ros-infrastructure/rosdep/pull/365 +.. [8] Patch for bloom: https://github.com/ros-infrastructure/bloom/pull/330 Copyright