This proposes a new method of updating WPILib version numbers, as well has updating how we publish Maven dependencies to bring actual version numbers to WPILib.
Currently, WPILib and tools generate their version numbers from hardcoded strings embedded somewhere in the program. WPILibJ, for example, writes its version to the disk at RobotBase.java:218. Every year, we have to go searching for the relevant C and Java code that does this version, and no one remembers. Even more distressing, the simulation Java and C libraries aren’t versioned in the library at all! Updating required NI image and firmware versions for the plugins is also a manual process, with a few people who know how to do it (Kevin, Brad, and myself have done it recently). This needs to updated in multiple places, effectively resulting in duplicated code that can easily be missed.
At the same time, the version numbers that we publish are not standardized across WPILib. The plugins publish as 0.1.0.BUILDINFO, and the Maven version numbers for every dependency are simply 0.1.0-SNAPSHOT, and this has not changed in years. This is very unhelpful for determining what version of a dependency any given build of the plugins uses. Additionally, as we publish this publicly for teams who use things such as GradleRIO (https://github.com/Open-RIO/GradleRIO), this makes it nigh-impossible for them to use previous years projects with that year’s released WPILibJ and WPILibC versions. By moving to a real version number system, we can support these use cases, and have version numbers that tell teams exactly what they’re getting when using a particular dependency.
This design covers 2 main changes to WPILib’s build system: version number code generation and general Maven version number changes. I’ve therefore split the design section into two main sections.
The first two opportunities are basically the same generation, the WPILibJ and WPILibC version numbers. Currently, these are hardcoded at RobotBase.java:218 and RobotBase.cpp:42, with simulation having no knowledge of what version it is at all. I propose having auto-generated files that contain the version. For WPILibC, this file would be a generated cpp file, generated to wpilibc\shared\include\WPILibVersion.cpp. We would also add a WPILibVersion.h file, checked, that will be unmodified by the build system. We would add this file to the .gitignore to make sure that it is not checked into the project. The file would look something like this:
/* BSD License Header */
#pragma once
/*
* The corresponding WPILibVersion.cpp file is autogenerated by the build system. If it does not exist, make sure that you
* run a build.
*/
extern const char* WPILibVersion;/*
* Autogenerated file! Do not manually edit this file. This version is regenerated
* any time the publish task is run, or when this file is deleted.
*/
const char* WPILibVersion = "2017.1.1-20160830195931-1-dirty";For Java, it would be generated to wpilibj/shared/java/edu/wpi/first/wpilibj/util/WPILibVersion.java, and be similarly ignored in the .gitignore. It would look something like this:
package edu.wpi.first.wpilibj.util;
/*
* Autogenerated file! Do not manually edit this file. This version is regenerated
* any time the publish task is run, or when this file is deleted.
*/
public final class WPILibVersion {
public static final String Version = "2017.1.1-20160830195931-1-dirty";
}Version numbers are generated by using git-describe --dirty, and then inserting the build date in after the tag name but before the number of commits. This is done to ensure monotonically increasing build numbers for Maven to continue working. Once a new version needs to be released, we will tag the commit to release with an annotated git version tag described in the version section below. All version tags start with v and then contain the actual version number. When it’s time to release a beta, we make sure that the version number is the one we want, then run a beta build. When we want to release a new beta, we explicitly tag the desired commit and release a new build. For released builds, the version number here and the Maven version number will be the same.
For internal builds, especially contributor local builds, generating a new version number every single build will cause unnecessary churn, especially if the contributor is only building part of the library. It will mean that, for C++, every build will have to recompile the generated version number, relink, and republish. Similar for Java. Therefore, to avoid this, the following rules will be used to determine if the build number should be regenerated:
-
If the release type is not set to the default, ie set to beta, stable, or release, the version number will be regenerated regardless of other settings.
-
If there is no generated version number, the version number will be generated, regardless of other settings.
-
If this is an alpha build, and the version number is already generated, then the version number will only be generated if the publish task is run. If the publish task is not run, the version number will not be regenerated.
All of this will be implemented in the Gradle build scripts, documented in the README, and controlled via command-line switches. For example, to create a beta-versioned build, you tag the commit with a beta-versioned tag and run ./gradlew publish.
When deploying a robot program, the Eclipse plugin ant script checks the version of WPILib installed on the target RoboRIO. If it matches the year and image version of the latest NI image, deployment can proceed. This is done via regex parsing and comparing to versions defined in build.properties. Unfortunately, these properties are located in two separate places, and are exactly the same; Java’s properties are in <eclipse-plugins-root>/edu.wpi.first.wpilib.plugins.java/src/main/resources/java-zip/ant/build.properties, and C++'s are in <eclipse-plugins-root>/edu.wpi.first.wpilib.plugins.cpp/src/main/resources/cpp-zip/ant/build.properties. Both of these must be updated every time NI releases a new image. I propose a simple JSON file in the repository root to solve this problem. This JSON file will contain the year and version of the latest NI image. For example:
{
niImageYear : 2016,
niImageNumbers : [19]
}The build process will use these to create a new file, niversions.properties, which contains the roboRIOAllowedImages and roboRIOAllowedYear properties. These files will be generated to the same folders as build.properties is currently, and will be added to the .gitignore so they aren’t checked in. We will also add sourcing of the properties files to our ant scripts, located at <eclipse-plugins-root>/edu.wpi.first.wpilib.plugins.(java/cpp)/resources/templates/build.xml. This will cause that properties file to be installed in the ~/wpilib directory when the plugins ship, and all templates to correctly source the file.
Maven does not support this complex of an action during build without writing a custom plugin. Therefore, we will create a Gradle wrapper that will perform these tasks before calling Maven itself. This will be similar to what we did when the plugins and allwpilib were one repository. This will also have the benefit of making our developer experience a little more seemless, as all developers will have to know how to do is call ./gradlew build, falling in line with the rest of our repositories. Maven will still be required to actually build the plugins, at least currently.
Our Maven version numbers currently leave a lot to be desired, both in terms of context (0.1.0-SNAPSHOT means nothing to anyone) and in terms of actual usefullness (GradleRIO users can’t get a specific version from a specific year). Numbers that would be a lot more useful to developers and teams are the current year and release of that library. Currently, the stucture of our releases is that we have 4 maven repostories: development, beta, stable, and release. I propose that we simplify the last 3 of those repositories into 1, and convert the context as to whether or not a release is beta, stable, or release into maven qualifiers. Numbering schemes will vary between each of the repos for the following reason: some projects, such as WPILibJ/C, are closely tied to the FIRST season. Splitting by year and release makes sense for this repository number, and is even largely semantic, as year changes mean there’s likely a few breaking changes, and point changes are bug fixes but otherwise non-breaking. However, some things such as NetworkTables, are not largely FIRST specific, and tying them to FRC competition years doesn’t really make sense. Therefore, I propose two version schemes, detailed below.
Both major version schemes will make use of the following suffix system. Build numbers will be formatted as major_number[-(beta|rc)-candidate_number][-buildtime][-commits-commithash][-dirty]. The format of major_number is specified by whether it’s the FRC-Year Scheme, or the SemVer Scheme. The elements in brackets are only present on alpha, beta, or rc releases. The buildtime is in the form of yyyymmddhhmmss, and is only present on alpha builds. commits is the number of commits that have been made since the last released version (determined by the number of commits since the last version tag), and commit-hash is the short hash of the current commit. It is only present on alpha builds, when there have been commits since the last version was tagged. Finally, the dirty flag is appended if there are uncommitted changes when the library is built. Again, it is only present on alpha builds, and only if there are uncommitted changes in the repository to files that git tracks. Alpha releases are published to the development repository, and beta/rc/release versions are published to the release repository. This makes it very easy for us to use the Maven version specifier of `, and have it mean what we want, rather than potentially grabbing the wrong version of a dependency. It also allows teams to use ` without having to worry about accidentally grabbing a development version of the plugins.
This version scheme is tied to the year and release. It takes the format of release_year.required_release.optional_release[-suffix], where release_year is the year that the package will be released in (ie, right now it would be 2017). The required_release is incremented whenever we release a required update for teams during the season, and reset to 1 every year. The optional_release number is incremented whenever an update for teams is available that is not required for teams. It is reset when the required_release number is incremented. For example, in 2016 there were 2 required releases, and 3 optional releases since the last required release, so the version number would be 2016.2.3. If we had a stable optional update for teams, the version of that would be 2016.2.4. If we then found a major bug and release a new required version, the new release number would be 2016.3.1. As a rule, breaking changes are allowed between release_year releases, and required_release.optional_release releases are only non-breaking changes except in exceptional cases. The current version number is specified with the JSON format detailed in WPILibJ and WPILibC. The following projects will be released under this format:
-
WPILibC
-
WPILibJ
-
Robot Builder
-
Eclipse Plugins
Sematic Versioning is a popular method of versioning libraries such that changes in version number contain information. Information about SemVer is available here: http://semver.org/. The following projects will use SemVer:
-
ntcore
-
cameraserver
-
java-installer
-
SmartDashboard
-
SFX
-
OutlineViewer
There are quite a few changes that will need to be made to support this, mostly around the build infrastructure. First, all projects will need to generate these new version numbers, and be made aware of how maven publications will work (mostly the same, except 2 less repos). All projects will also need to be updated with documentation to cover the new switches introduced. Any projects that depend on other projects will have to start using the + version specifier, which most do not do. Finally, we’ll have to update Jenkins to produce these new artifacts. This will also be a good time to to move our Jenkins configs into our repos, instead of having them in Jenkins itself. Jenkins has excellent support for this via a Groovy DSL called the Jobs DSL, documented here: https://github.com/jenkinsci/job-dsl-plugin/wiki. This will ensure that our build is not just documented in my head, and we can appropriately version it as well.
The major drawback here is that every time we want to do a new beta or rc, we have to explicitly rebuild. In past years, we simply republished existing artifacts as more stable channels, whereas with these modifications we’ll have to rebuild every time. This shouldn’t be a big concern, but it could slow down releases as we’ll have to rerun our tests. Another issue for local builds is that in order to change the version number, the developer will have to run an explicit clean. Otherwise, the build will see that the version number file has already been generated and skip that step. It’s certainly no worse that now, but doing anything else would ensure that everything has to be rebuild on every build, which would not be a good experience.
A simpler version of this scheme would be to have version numbers generated, mostly according to the first half of this proposal, except that they don’t say beta/rc. Rather, they simply say some string, such as 2016 Java Release 5. This would mean we could just republish existing artifacts. However, I’m not a fan of this due to lacking the beta/rc/alpha tags, which makes individual builds more identifiable. Additionally, this could introduce version number differences if whoever updates the string does it a little differently than the previous person.