Prefix packages with autoware_

[esteve]

The naming on the current set of packages doesn't emphasize that they belong to Autoware, so when Debian packages are generated, their names are for example ros-humble-object-merger. Given that these packages are not used outside Autoware and that names in the generated packages can be confusing, I propose we prefix all packages with autoware_ so that Debian packages are named like ros-humble-autoware-object-merger

Examples of ROS projects that follow this naming are https://github.com/ros-planning/moveit2 and https://github.com/ros-planning/navigation2, every subpackage is prefixed with the name of the project (moveit2 and nav2, respectively)

[esteve]

I've submitted autowarefoundation/autoware.universe#6067 as a draft PR, I'm building it locally, I haven't added the tag:run-build-and-test-differential label because it's going to overwhelm the CI as it changes lots of packages. I will report back when the build is complete.

[idorobotics]

Apart from avoiding ambiguity, this will also help with package searches.

Ideally apt-cache search autoware should list all autoware packages/binaries available for download.

[esteve]

One thing I've discussed with @mitsudome-r is whether we should rename the folders as well, to make them consistent with the package names. I agree with him about it, but one concern is that if a PR that has all those changes it will cause too many conflicts with existing workflows (see autowarefoundation/autoware.universe#6067 which already has conflicts), so submitting smaller PRs would be better. Any feedback whether these PRs should be per-package or per-component (perception, localization) ?

[xmfcx]

Per package is good and manageable.

We can prioritize them over existing PRs and swiftly move ahead.

Let's not have more than 2 PRs per package open at a time to have the chances of conflicts at minimum, do you think this would work?

Also we will need to modify the autoware_launch and autoware.universe/launch/ files pointing at them as well, it will require some effort to make sure we don't break existing configs.

cc. @esteve @mitsudome-r

[esteve]

Per package will create a lot of PRs, but I think it's the right approach. I'm currently making all the changes in one branch, but will split it into individual smaller branches and PRs.

[xmfcx]

And we shouldn't have more than 2 PRs open at a time, or it will be an endless cycle of keeping things up to date.

[esteve]

Regarding C++ namespaces (#4117), I think it'd best to do it in the next wave of changes, given that we still need to agree on the convention (i.e. autoware:: vs autoware::perception::) and that downstream packages will be affected as well.

[esteve]

More on the folder structure for headers. I checked other projects, and for example, what the moveit project does is the following:

PACKAGE/include/moveit/PACKAGE_NAME

this has the advantage that all related subpackages will be installed in the same folder if we pass the --merge-install option to colcon, and therefore downstream packages can include headers like this:

#include <autoware/PACKAGE_NAME/...>

instead of the first approach:

#include <autoware_PACKAGE_NAME/...>

In my opinion, this looks cleaner and would match nicely with autoware::PACKAGE_NAME::. We would still prefix package names with autoware_ in package.xml and CMakeLists.txt, though.

[xmfcx]

Option 1:

1. Remove all the unused packages
   • https://github.com/orgs/autowarefoundation/discussions/4136
   • https://github.com/orgs/autowarefoundation/discussions/3906
2. Apply https://docs.ros.org/en/humble/How-To-Guides/Ament-CMake-Documentation.html#adding-targets convention to reduce the exposed include headers.
3. package_name/include/autoware/package_name refactoring for exposed include headers

We also have to:

• Prefix with autoware_.
• Add autoware namespace for each package.

Option 2:

For each package:

• Apply https://docs.ros.org/en/humble/How-To-Guides/Ament-CMake-Documentation.html#adding-targets convention to reduce the exposed include headers.
• package_name/include/autoware/package_name refactoring for exposed include headers
• Prefix with autoware_
• Add autoware namespace

Option 3:

For each package:

• package_name/include/autoware/package_name refactoring for exposed include headers
  • autoware_package_name/include/autoware/package_name refactoring for exposed include headers (to clarify for @mitsudome-r)
• Prefix with autoware_
• Add autoware namespace

Another round for each package:

• Apply https://docs.ros.org/en/humble/How-To-Guides/Ament-CMake-Documentation.html#adding-targets convention to reduce the exposed include headers.

[xmfcx]

Let's go with the Option 3 over consensus between:

• @esteve
• @mitsudome-r
• @xmfcx

[kminoda]

@esteve @xmfcx @mitsudome-r Hi, thank you for your contribution as always 🙏

As this change may be a breaking change in some of TIER IV's internal infrastructure, let me ask some questions:

1. I am not familiar with generation of debian package in ROS community, but is it mandatory to add autoware_ prefix for all the ROS 2 packages? (I was wondering if it is possible to maintain the package name in Autoware as it is right now while adding autoware_ prefix to debian packages. If it is possible, I believe this breaking change may be avoided)
2. Are there any timelines for this execution?
3. Are there any guidelines you have in mind? (e.g. the topic naming must NOT change, node name when launched should not change either, etc)

[esteve]

1. The name for the Debian package is derived from the ROS package name, that's the convention that all of the ROS infrastructure uses. Moreover, it'd help us identify what packages are ours, in addition to the package names we're also adding appropriate namespaces in the code to isolate our packages from other project's.
2. This is already an ongoing task, there's a bunch of PRs that I've submitted to Autoware.universe dealing with this
3. For now it's only package names and C++ namespaces

[xmfcx]

As this change may be a breaking change in some of TIER IV's internal infrastructure

If it is possible, I believe this breaking change may be avoided

@kminoda I understand your concerns.

Could you give more details on how these changes are affecting your internal infrastructure?

I'm asking this because we had this same HUGE delay on the autoware_auto_msgs -> autoware_msgs task too. See my related question there here which was unanswered for the past 3 weeks.

The message migration task has been ongoing for the past 11 months and 11 days.

• refactor(autoware_auto_control_msgs): replace autoware_auto_control_msgs with autoware_control_msgs autoware.universe#3677

Autoware main branch is being treated as a development branch and is supposed to be treated as a place with possible breaking changes.

TIER IV introduces breaking changes all the time, and it sometimes disrupted our existing projects that relied on the main branch too.
But we accept it and live with it. Or fork and keep going from a fork.

Why can't the TIER IV do the same? This is an open source project but only TIER IV is allowed to make breaking changes. But when anyone else tries to do the same, even when it is fully justified technically, there is resistance.

I also see that almost all commits here come from TIER IV developers: Autoware Universe Commits

Click to expand 🔽

I would like to have more changes accepted and made from the community too. But even with simple naming changes, we get blockers like these.

We have created Autoware Labs initiative to bring the blockers like these to the technical steering committee to resolve them.

Could you list the reasons why this change disrupts TIER IV's internal infrastructure?

What is your proposal to mitigate this? Because we will change the package names, we might need to do so in the future too.

Maybe TIER IV can fork from a working release like autoware/releases/tag/2024.04 and do periodic syncs with the main branch at the rates they feel comfortable?

cc. @mitsudome-r @yukkysaito @TakaHoribe

[mitsudome-r]

@kminoda Do you have any specific packages that you are concerned with?

[kminoda]

Thank you all!

Autoware main branch is being treated as a development branch and is supposed to be treated as a place with possible breaking changes.
TIER IV introduces breaking changes all the time, and it sometimes disrupted our existing projects that relied on the main branch too.
But we accept it and live with it. Or fork and keep going from a fork.
Why can't the TIER IV do the same? This is an open source project but only TIER IV is allowed to make breaking changes. But when anyone else tries to do the same, even when it is fully justified technically, there is resistance.

I haven't catching up in details about this topic, but with the above statement from @xmfcx , now I think I am on the same page. Understood the basic concept that the Autoware Universe main branch should accept breaking changes to enable rapid development, while maintaining some stable branches on a regular basis for depending tools & infrastructure.

That being said, my concern is mostly addressed with all of your answers. Thank you for your answers and also all of your contributions 👍

(FYI, one internal infrastructure that may be affected by this change is our MLOps pipeline, which directly uses some package names such as lidar_centerpoint. I understand that, ideally speaking, this type of dependency is something we as TIER IV should mitigate internally, but realistically it may take a while to address this. That was the reason why I also asked about the timeline)

[TakaHoribe]

@xmfcx I apologize for the inconvenience. Firstly, the impact of this change on TIERIV should be considered internally and not within OSS. This proposal should be strongly pursued without being limited to concerns from just one company.

Additionally, I am very appreciative of the efforts by Fatih and others at AutowareLabs, which significantly contribute to energizing the OSS community.

One reason disruptive changes are problematic is the lack of a proper release process. I hope that @mitsudome-r and other architects (including myself), will take the lead in addressing this issue.

At TIERIV, we are also exploring the proper use of Autoware and aim for a more active and appropriately managed software development process in the OSS.

Please let us work together to improve Autoware in OSS.

[felixf4xu]

I'm afraid that it's late for the discussion, but I didn't see any objections and then here it comes.

ROS2 has a namespace because it will be used as library in some other projects. For example, in my exe project when I write ROS:: it clearly states that it's not my code. Autoware is different, it's the main project and it will not be used as library by others. If I need to write autoware:: everywhere, it makes no difference if I don't write the namespace at all.

Then to the folder name of each packages, I have even stronger objections. Current folder names are long enough, please don't use autoware_static_centerline_generator under autoware/universe/autoware.universe/..., you can count how many redundant "autoware" are used in the full path.

Last, for the package name, please don't use ros-humble-autoware-static-centerline-generator. Autoware depends on ros but it's not part of ros, please don't embed ros in its names. ROS has difference versions so humble is used but I don't see it's necessary for autoware to support multiple versions of ros.

Overall, the folder structure, package name, c++ namespace, should evolve in the direction of making it simpler. We don't need to follow google because we are not google, google's overhead is not always our overhead.

[esteve]

@felixf4xu The decision to prefix packages and adding namespaces is final, it might be revisited in the future to tweak things, but the spirit of the task will stay.

I'll respond to your questions one by one so that people who might read this discussion in the future understand the rationale for making these changes:

ROS2 has a namespace because it will be used as library in some other projects. For example, in my exe project when I write ROS:: it clearly states that it's not my code. Autoware is different, it's the main project and it will not be used as library by others. If I need to write autoware:: everywhere, it makes no difference if I don't write the namespace at all.

First of all, there are packages in Autoware that are libraries (see https://github.com/autowarefoundation/autoware.universe/tree/main/planning/behavior_path_planner/autoware_behavior_path_planner_common for example), so your argument about Autoware not having libraries is incorrect. But even if Autoware was all executables and no libraries, putting code in namespaces improves readability, it's easier to track where a function comes from if it's in a namespace, plus we make sure that it does not interfere with any external dependency.

Then to the folder name of each packages, I have even stronger objections. Current folder names are long enough, please don't use autoware_static_centerline_generator under autoware/universe/autoware.universe/..., you can count how many redundant "autoware" are used in the full path.

It's standard practice in ROS projects to name folders the same as the project name. Following conventions helps us get developers up to speed quickly because they are already familiar with other projects.

Last, for the package name, please don't use ros-humble-autoware-static-centerline-generator. Autoware depends on ros but it's not part of ros, please don't embed ros in its names. ROS has difference versions so humble is used but I don't see it's necessary for autoware to support multiple versions of ros.

Package name in this case would be autoware_static_centerline_generator, we prefix packages so that we don't collide with other packages and also for us to know what autoware packages have been installed (e.g. dpkg -l | grep autoware). The ros-humble- prefix is added by the ROS buildfarm to again avoid collisions with other Debian packages. ros- because it's built by the ROS buildfarm and humble- because ROS distributions don't guarantee any ABI compatiblity, so Debian packages need to be clear what ROS distribution they target.

Overall, the folder structure, package name, c++ namespace, should evolve in the direction of making it simpler. We don't need to follow google because we are not google, google's overhead is not always our overhead.

We follow the ROS coding guidelines because they're reasonable, all other ROS projects follow them so there's no friction to join a project and there's a lot of tooling to check for compliance with the conventions.

[Owen-Liuyuxuan]

After migrating to the autoware_ prefix, especially the autoware_ prefix in the readme file, it is now a disaster to search for the documentation of the package I would like to see.

Here are some suggestions on how to write a document title

1. avoid "autoware_detection_by_tracker"
2. use "Autoware Detection by Tracker"
3. or use the original "detection_by_tracker" in Readme.md

For users this is rather important from my perspective.

[esteve]

Can you elaborate why the first option is a "disaster"?

[Owen-Liuyuxuan]

Because, for example in figure 2.
I searched for "detection by tracker". There are simply no results directing to the real "autoware_detection_by_tracker".
I need to write "autoware_detection_by_tracker" to get the search engine correct.
My idea is that the package name in the readme file should be optimized for the search engine of markdown.

(edited: I think in the document part, either we should unify the way to name the readme title and unify the way to search, or we should find a way to optimize the document for searching)

[esteve]

You'd have the same issue with detection_by_tracker, so adding the prefix hasn't changed anything. What we should do instead is to tell the indexer to break words that contain underscores so that users could search by "detection", "tracker", etc.