synth.1

.Dd January 27, 2021
.Dt SYNTH 1
.Os
.Sh NAME
.Nm synth
.Nd custom package repository builder for FreeBSD and DragonFly
.Sh SYNOPSIS
.Nm
.Op Ar help | configure | version | status | purge-distfiles
.Nm
.Op Ar rebuild-repository | prepare-system | upgrade-system
.Nm
.Op Ar everything | status-everything
.Nm
.Op Ar status | build | just-build | install | force | test
.Op Ar file | port-origin ...
.Sh DESCRIPTION
The
.Nm
program is an advanced concurrent ports building tool aimed at system
administrators that prefer or require the building of packages from source
rather than installing official binary packages.
.Nm
will build packages in a clean environment can exactly mirror the system
that they are built on, it will create local package respositories and install
.Xr pkg 8
repository configuration file that causes locally built packages to be used
with the highest priority, all while allowing the system to fully upgraded
with a single command.
.Ss synth version
When
.Nm
is executed with the
.Cm version
command, an unknown command, or no command at all, it will display a short
block of text containing an identification, summary, version number, copyright,
usage, and a summary of valid commands.
.Ss synth help
The
.Cm help
command displays lists each valid command with a short synopsis of the
command's function. It also elaborates the several commands expect one or more
port origins to be provided, or a path to a file with the same information.
.Ss synth configure
The
.Cm configure
command launches an interactive menu for providing the user with a clean method
to configure
.Nm .
It works on the concept of profiles, and the default profile
.Dq LiveSystem
is automatically installed with values that are determined by scanning the
system. Most of the configurable items are directories that are used to
construct the build environment, but they aren't limited to that.
.Bl -tag -width ".Ev Compiler cache directory"
.It Ev Ports directory
If the PORTSDIR environment variable is set when the initial profile is
created, it establishes the initial value. Otherwise,
.Nm
will extract the value of PORTSDIR from
.Pa /usr/share/mk
and use that.  If neither source points to a valid path,
.Pa /usr/dports
and
.Pa /usr/ports
will be successfully tried.  If those don't exist either,
.Nm
will exit with an error message.
.It Ev Packages directory
This is the location of the local repository metadata and all the packages
that are built by
.Nm .
It clearly must have lots of available disk space. The default value is
.Pa /var/synth/live_packages
but it should be altered if the system has a
.Pa /var
partition with limited space.
.It Ev Distfiles directory
This indicates where all the ports source distribution files are stored, and
the initial value is determined by scanning the existing configuration. The
defaults are defined by ports and are
.Pa /usr/ports/distfiles
for
.Po
.Fx
.Pc
and
.Pa /usr/distfiles
for
.Po
.Dx
.Pc .
Most users will want to keep the initial value as that directory should
already contain distfiles.
.It Ev Port options directory
This is the directory where all the selected options for ports are cached.
The initial value comes from a system scan, so chances are it has the correct
value. However, if the user would like a separate configuration area for port
options, they would create the new directory and set this value accordingly
(but the user would have to ensure the new location is passed to the port
when configuring port options later in that case).
.It Ev Build logs directory
A log is produced for every port built (subsequent builds of the same port
will overwrite previous logs). This item dictates where those logs are stored,
which is
.Pa /var/log/synth
by default. If the
.Pa /var
partition is limited, the user will definitely want to change this location as
the uncompressed logs can potentially consume gigabyte of space.
.It Ev Build base directory
This is the mount point for all the builders. When tmpfs is used, this location
requires very little space as it contains empty directories that serve as mount
points. If tmpfs is not used, then this location should have access to a lot
available diskspace as it will provide the work area and virtual
.Pa /usr/local
for each builder. The default location of
.Pa /usr/obj/synth-live
should be fine for most users (it generally has lots of space and tmpfs is the
default build mode).
.It Ev System root directory
Most users will use the base system as the basis for the build environment, and
thus the default setting of
.Pa /
is correct. Advanced users may use DESTDIR with installworld to create an
alternative build base (e.g. i386 cross-build on amd64 architecture or a
modified base that serves as a fallout test for an upcoming feature). Those
would be the users that would modify this value.
.It Ev Compiler cache directory
If ccache is properly installed on the system, set this value to the location
of the cache. All the builders will leverage it. It is disabled by default.
It can be disabled again by entering any invalid path as a new value, e.g.
.Dq none .
.It Ev Num. concurrent builders
The represents the number of ports that can be simultaneously built on the
system. The selected value is influenced by the number of physical cores and
hyperthreads the system has, the amount of memory on the system, the amount of
available swap, and if that swap is a solid-state drive or not. Generally
memory is the limiting resource when tmpfs is used, so the default value for
the number of builders is generally 75% of the number of CPUs of the system.
The user is free to balance jobs versus builders as well.
.It Ev Max. jobs per builder
If memory is constrained, it's often a better performance tradeoff to reduce
the number of builders and increase the number of jobs per builder. The
default value varies depending on the system, but it will never exceed 5.
.It Ev Use tmpfs for work area
If the initial scan shows there is at least 1.25 Gb per builder, this item
will default to
.Dq true .
Building on tmpfs is a big speed spot and it's recommended if the memory
resources allow it. This item is directed at the work area for building the
port
.It Ev Use tmpfs for /usr/local
Before a port can start building, all the build and library dependencies have
to be installed into the localbase. Extracting, installing and later
deinstalling is a lot of work for a disk, so using tmpfs here is another
performance boost. It is recommended that users with adequate memory leave
this enabled.
.It Ev Display using ncurses
During the concurrent building, the default display is constructed using
ncurses. It shows a lot of good information and it should be used. The
display will fall back to a text mode if the terminal doesn't support it.
This setting forces text mode unconditionally, but few users will want or
need to use this option.
.It Ev Fetch prebuilt packages
When active, this option will scan the external repository for suitable
prebuilt packages to satisfy build requirements.  To be considered suitable,
the ABI, options, and dependencies of the remote package must all match
requirements.  If a package is located, it will be fetched and placed in
the packages directory and later incorporated into the local repository.
This option is not active by default, meaning that
.Nm
will normally build everything from source and ignore external repositories.
.El
.Pp
To create alternative profiles, press the
.Dq >
key at the prompt. A new menu will appear that provides options to switch
to another existing profile or create a new one that can be tailored and
named. Any newly created profile becomes the new default profile, but this
can be easily changed by repeating the profile selection process.
.Pp
Pressing the Alphabetic key associated with each configuration item brings
up a prompt to change the value. Directory entries must be valid paths
except in the case of ccache. The boolean values are modified with a single
keypress
.Po
.Dq T
or
.Dq F
.Pc
and the numeric items require positive integers. After changing the values,
the old menu returns and altered values are marked with an asterisk.
Simply entering the
.Dq Enter
key
.Po
carriage return
.Pc
saves the changes while pressing the
.Dq Escape
key will discard them.
.Pp
If more than one profile exists, an option to delete profiles will appear
on the menu.  By pressing the
.Dq <
key, a new menu will appear that lists all the inactive profiles.
Entering the number associated with one of those profiles will immediately
delete the profile.  The deletion cannot be undone.
.Ss synth upgrade-system
The
.Cm upgrade-system
command automates the full upgrade of the system upon which
.Nm
is installed. It is expected that the ports tree has been updated to the
latest version before the command is executed. Using the ports tree as a
reference,
.Nm
will query
.Xr pkg 8
to determine what is currently installed and which software has newer versions
available. It will rebuild those packages and any package that depends
libraries they contain or that have a runtime dependency on the rebuild
packages. In addition, any port that has changed with respect to its options,
dependencies, or ABI will be rebuilt as well. After analysis, the concurrent
builder will begin building the set of packages that have been identified as
requiring a rebuild. When that phase is complete, the local repository will
be rebuilt without waiting for confirmation. Finally,
.Nm
will command
.Xr pkg 8
to upgrade the system using the local repository, effectively bringing it
completely up to date.
.Ss synth prepare-system
The
.Cm prepare-system
command has the same functionality of the
.Cm upgrade-system
command with the exception that the packages will not be installed. Once the
local repository is fully rebuilt, the command exits.
.Ss synth rebuild-repository
The
.Cm rebuild-repository
command will perform a full sanity check on all packages present in the
packages directory and remove the ones which fail the check.  The local
Synth repository will be created (or recreated) using the remaining
packages.  This step is also done as part of several other commands, so
it mainly exists for scripting purposes.
.Ss synth status
The
.Cm status
command with no arguments performs a dry-run of the
.Cm upgrade-system
command. It will not delete obsolete packages, nor rebuild the local
repository. It will list all the ports that will be rebuild along with a
total, and it also logs the same information to
.Pa /var/synth/synth_status_results.txt
since the full list is often longer than the terminal height.
.Ss synth just-build [ file | port-origin ... ]
The
.Cm just-build
command takes a list (one or more) of port origins
.Po
an origin is the combination of a category and the port directory,
e.g. lang/python27
.Pc
and it will builds the packages of those ports if necessary. If analysis
determines the packages are already up to date, nothing is done.
.Pp
This is the first of several commands that take arguments. The arguments
are either an unlimited number of port origins, e.g.
.Bd -literal -offset -indent
synth just-build editors/joe editors/nano editors/libreoffice

.Ed
or they are limited to one, a path to a file, e.g.
.Bd -literal -offset -indent
synth just-build /tmp/build.list

.Ed
A file that is equivalent to the first example would contain:
.Bd -literal -offset -indent
editors/joe
editors/nano
editors/libreoffice

.Ed
When the building task is complete,
.Nm
exits.
.Ss synth build [ file | port-origin ... ]
The
.Cm build
command differs from the
.Cm just-build
command by asking the user if they wish to rebuild the local repository when
the building is complete. The answer is usually
.Dq N
.Po
for
.Dq no
.Pc
because rebuilding the repository can take a few minutes to complete, and it
only makes sense to do when all the building is complete. Should the user opt
to rebuild the repository,
.Nm
will then ask if the user wishes to install the packages that come from the
ports list on the system.
.Ss synth install [ file | port-origin ... ]
The
.Cm install
command is similar to the
.Cm build
command except that it will not ask permission to rebuild the repository or
install the resulting packages. It will just do it all in sequence.
.Ss synth force [ file | port-origin ... ]
The
.Cm force
command is similar to the
.Cm build
command except that any packages related to the ports list will be deleted
first, regardless if they are up to date or not. This results in that every
port on the given list will build.
As with the
.Cm build
command,
.Nm
will request permission to rebuild the repository and possibly install the
packages when the building phase is complete.
.Ss synth test [ file | port-origin ... ]
The
.Cm test
command will pre-delete any existing packages related to the ports list
before rebuilding them with extra DEVELOPER_MODE tests and settings. When the
building is complete, it will just exit without offering to rebuild the
repository.
.Ss synth status [ file | port-origin ... ]
This version of the
.Cm status
command provides a dry-run and reports on what would happen if the
.Cm just-build command
was executed with the same arguments. The output and logging is similar to
the solo
.Cm status
command, but the build scope is limited to the given ports list.
.Ss synth everything
The
.Cm everything
command is not meant for most users. It will attempt to build every port in
the ports tree, which can take several days depending on the power and
resources of the building machine. Even downloading the tens of gigabytes
worth of distfiles consumes significant time!
.Ss synth status-everything
The
.Cm status-everything
command is just a dry-run for the
.Cm everything
command. Since
.Nm
works through increment building
.Po
that is, it will not build a port if an up to date package already exists
for it and all the packages dependencies are also in place
.Pc
, the resultant build list can be much shorter than the full list of ports in
the tree. Only those rare users that wish to build the entire set of ports
would use this command.
.Ss synth purge-distfiles
This is a useful command that will first scan the entire ports tree to gather
a complete list of distfiles used by the ports tree, and then it will scan
the contents of the configured distfiles directory. All source distribution
files that cannot be matched to at least one port in the tree will be removed.
This can result in gigabytes of recovered disk space if significant time has
passed since the distfiles directory was last cleaned.
.Sh ENVIRONMENT
.Bl -tag -width SYNTHPROFILE
.It Ev PORTSDIR
The
.Ev PORTSDIR
environment variable is only checked on the very first signficant command of
.Nm
.Po
meaning everything except help and version
.Pc
when the default profile is created. It is also checked when adding new
profiles. The variable dictates the default location of the ports tree with
a higher priority than the standard locations.
.It Ev SYNTHPROFILE
When this environment variable is set to name of an existing
.Nm
profile, that profile will be loaded when
.Nm
is launch rather than the active profile.  It is a temporary override;
removing the variable will allow the default profile to load again.
.It Ev WHYFAIL
When this environment variable is defined to any value, the sanity check
routines will provide more detail on dependency and option check failures
on a separate output line.
.It Ev ENTERAFTER
This environment variable only applies with the
.Cm test
command, and then only when a single port is specified.  If both of those
conditions are satisfied, and the variable has the value of:
.Bl -dash -compact -width 1
.It
extract
.It
patch
.It
configure
.It
build
.It
stage
.It
install
.It
deinstall
.El
then an interactive build will be attempted. First, all the port's
prerequisites are built normally and
.Nm
will return to text mode. If everything built successfully, the specified
port will build up to and including the point specified by ENTERAFTER. At
that point, control will be given to the user by providing them with a
tcsh shell with the current directory set at the root of the builder.
When the user is done, they should terminate the shell with the
.Cm exit
command to signal
.Nm
to clean up and terminate. This is a developer tool to troubleshoot port
build problems and it is not intended for the average user.
.It Ev LOCK
This environment variable only applies with the
.Cm test
command. When it is defined to any value, the builder's localbase is
remounted read-only prior to the configure phase and restored to
read/write permissions after building is complete.  This is a diagnostic
tool aimed at pinpointing file system violations as a port is not supposed
to write to localbase prior to staging.  Note that the localbase tmpfs
setting will considered as False in this mode, so building may be
noticeably slower than normal.
.It Ev SELFTEST
This environment variable only applies with the
.Cm test
command.  When it is defined, the
.Dq test
target is executed as part of the build cycle; this target is not executed
under any other circumstances.
.El
.Sh FILES
The following files may be used or created:
.Bl -tag -width hook_pkg_ignored
.It synth.ini
This is the configuration file. It is automatically generated and manipulated
using the interactive configuration menu. There is no need to touch this file
other than to remove unwanted profiles as currently that is not possible from
the program itself. It is normally located at
.Pa /usr/local/etc/synth/synth.ini
.It <profile>-make.conf
This is an optional, user-provided file. If it exists, the builder's
.Pa /etc/make.conf
will be appended with the contents of this file. For the default profile, the
file would normally be located at
.Pa /usr/local/etc/synth/LiveSystem-make.conf
.It <profile>-environment
This is an optional, user-provided file. It contains a list of name-value
pairs joins with an equals sign (e.g. HTTP_PROXY=http://proxyserver:8888),
one pair per line.  If the file exists, these variables will be defined
in the builder environment.  For the default profile, the file would
normally be located at
.Pa /usr/local/etc/synth/LiveSystem-environment
.It <profile>-private.key
This is one of two files required to support the signing of the repository
by a local RSA key.  It is the output of the
.Cm openssl genrsa
command; see the EXAMPLE section for how to generate it.  The file
permissions are required to be set at 400 as well.  For the default
profile, the file is normally located at
.Pa /usr/local/etc/synth/LiveSystem-private.key
.It <profile>-public.key
This is the second of two files required to support the signing of the
repository by a local RSA key.
.Nm
must confirm the public counterpart of the signing key is available
because its path is required for the generated pkg repository configuration
file.  For the default profile, the file is normally located at
.Pa /usr/local/etc/synth/LiveSystem-public.key
.It <profile>-signing_command
This is one of two files required to support the signing of the repository
by an external signing server.  It consists of a single line containing
the command; see the EXAMPLE section.  For the default profile, the file
is normally located at
.Pa /usr/local/etc/synth/LiveSystem-signing_command
.It <profile>-fingerprint
This is the second of two files required to support the signing of the
repository by an external signing server.  It consists of a single line
containing the SHA 256 hash of the public key that is the counterpart of
the key that signed the repository.  For the default profile, the file
is normally located at
.Pa /usr/local/etc/synth/LiveSystem-fingerprint
.It <profile>-index
The introduction of flavors within FreeBSD ports destroyed the 1:1
relationship between port directories and packages.  Now any port
can generated any number of packages.  This requires scanning every
port in the tree to determine the true origin range.  This is a time
consuming task, so Synth caches the scan results and reuses it as long
as all ports files are older than the generated index.  When Synth
detects the index is out-of-date or if it doesn't exist, it is
automatically generated.  One index exists for each profile, with the
default profile index normally located at
.Pa /var/cache/synth/LiveSystem-index
.It synth_status_results.txt
Whenever a status command is run, a list of ports that would build with the
actual command is produced at
.Pa /var/synth/synth_status_results.txt
.It synth_prefetch_list.txt
Whenever a status command is run, and the option to use suitable prebuilt
packages has been selected, a list of packages that will be fetched as a
result will be written to
.Pa /var/synth/synth_prefetch_list.txt
.It index.html
For every build run, Synth installs a web summary at the Report subdirectory
of the build logs directory.  For the default profile, the file would be
located at
.Pa /var/log/synth/Report/index.html .
To remotely view the web report with a browser, a web server must be present
and configured to expose the logs directory.  The web report is dynamic and
updated 10 times per minute.  The complete history is available and
searchable, with quick-filters on "Built", "Failed", "Ignored" and "Skipped"
fields as well as on the "No." column which is useful to examine cascaded 
skips.  The search filter can be quickly cleared by clicking on the "Total" 
field.
.It port logs
Every single attempt at building a port results in a build log. The name of
the log is always in the form of [category]___[port].log so that subsequent
builds will overwrite previous attempts. They are created in the directory
specified by the configuration.
.It hook_run_start
If this file exists, and it is executable
.Po
or it is a symbolic link that leads to an executable file
.Pc
then it will be executed by
.Nm
at the start of the build. The same requirement exists for all hooks. This
hook, and the next five hooks all define the following common environment
variables:
.Ev PROFILE ,
.Ev DIR_PACKAGES ,
.Ev DIR_REPOSITORY ,
.Ev DIR_PORTS ,
.Ev DIR_OPTIONS ,
.Ev DIR_DISTFILES ,
.Ev DIR_LOGS ,
.Ev DIR_BUILDBASE .
Uniquely, this hook also defines PORTS_QUEUED which is the number of ports that
.Nm
will attempt to build. This file must be colated with synth.ini, so that would
be
.Pa /usr/local/etc/synth/hook_run_start
for most installations.
.It hook_run_end
This hook is fired when the build completes. In addition to the common
definitions, it defines
.Ev PORTS_BUILT ,
.Ev PORTS_FAILED ,
.Ev PORTS_IGNORED ,
and
.Ev PORTS_SKIPPED
in the environment,
which represents the results of the build.
.It hook_pkg_success
This hook is fired whenever a builder successfully packages a port. In
addition to the common definitions, it defines
.Ev RESULT=success ,
.Ev ORIGIN ,
and
.Ev PKGNAME
in the environment.
.It hook_pkg_failure
This hook is fired whenever a builder fails to package a port. In addition
to the common definitions, it defines
.Ev RESULT=failure ,
.Ev ORIGIN ,
and
.Ev PKGNAME
in the environment.
.It hook_pkg_ignored
This hook is fired prior to the start of a build when
.Nm
prescans the queue for ports that are to be ignored according to the ports
tree. Once the build has started, this hook is never fired again. In
addition to the common definitions, it defines
.Ev RESULT=ignored ,
.Ev ORIGIN ,
and
.Ev PKGNAME
in the environment.
.It hook_pkg_skipped
This hook is fired whenever a package gets ignored or fails which in turns
cascades into a number of ports getting removed from the build queue due to
the dependency's failure. This may fire before the build starts due to
ignored prescanning, or any time during the build. In addition to the common
definitions, it defines
.Ev RESULT=skipped ,
.Ev ORIGIN ,
and
.Ev PKGNAME
in the environment.
.El
.Sh SEE ALSO
.Xr pkg 8 ,
.Xr pkg-repo 8
.Sh NOTES
.Ss Graceful exit
During development, the interrupt signal
.Po
SIGINT
.Pc
was captured and handled. It turns out this handling extended to the
builders, so any event resulting in a SIGINT in the build would lock up the
builder permanently. There's no apparent workaround for this given the design,
so pressing Control-C is no longer handled. It will stop the build, but it will
leave everything in bad state. The next significant call to
.Nm
will attempt to clean that up though. If a user wants to stop the build, the
best approach is to type the Control-Q combination. The program will shut down
as soon as it can. During the building phase no new builds will start after
the shutdown begins, but the active builders will continue until completion.
.Ss Unprivileged users
At this time,
.Nm
can only be run by the root user.
.Ss Impulse indicator
A common question is "What does the Impulse indicator on the display mean?"
It is the package build rate (packages/hour) over the last 500 seconds.  For
the first roughly ten minutes of a bulk run, the
.Dq Impulse
rate and the
.Dq Pkg/Hour
rate will be identical. After that, the
.Dq Impulse
build rate will differ from the rate averaged over the entire elapsed time,
often by a great amount.  It's just a way to gauge the package production
rate over the previous few minutes.
.Ss Supported architectures
Synth is written in Ada, and thus an Ada compiler or cross-compiler is
required to build it. For
.Fx ,
that means
.Nm
is currently limited to amd64 and i386. It is feasible to port GNAT to the
ARMv7 architecture as this has already been done
.Po
See lang/gnatdroid-armv7 port
.Pc
but it's not a trivial task to do it. Other architectures would be
challenging, although not impossible, to support.
.Sh EXAMPLE
To activate the signing of default profile's repository with a local RSA
key, generate the two key files:
.Bd -literal -offset indent
% cd /usr/local/etc/synth
% openssl genrsa -out LiveSystem-private.key 2048
% chmod 0400 LiveSystem-private.key
% openssl rsa -in LiveSystem-private.key -out \\
  LiveSystem-public.key -pubout
.Ed
.Pp
To activate the signing of default profile's repository with an external
signing server, two files need to be created.  It's assumed the signing
server has already been configured as described by the
.Xr pkg-repo 8
man page, and that the keys are located in the repo.pub and repo.key files.
On the signing server which is 192.168.0.44 in this example:
.Bd -literal -offset indent
% sha256 -q repo.pub > LiveSystem-fingerprint
% echo "/usr/bin/ssh 192.168.0.44 /usr/local/bin/sign.sh" \\
  > LiveSystem-signing_command
.Ed
.Pp
Then move both files to /usr/local/etc/synth directory on the build server.
The build server root user needs to have a passphrase-free SSH key access
to the signing server in order for Synth to sign the repository.
.Sh HISTORY
.Nm
was conceived long before coding began in December of 2015. It was designed
to be system-agnostic from the beginning, and in particular support for
.Fx
and
.Dx
was added simulaneously. It was hoped that it would attract users of the
older PortMaster and PortUpgrade ports management tools by providing them
with a superior and well-maintained alternative.
.Pp
.Nm
was written entirely by
.An John Marino
.Aq marino@FreeBSD.org
and released under the Internet Software Consortium license.