Dependency build automation for Linux

[sm-shaw]

This Pull request brings all dependencies into the HammerDB GitHub repository and enables build automation for Linux so that a HammerDB distribution can be completely built from source as an alternative to downloading the pre-compiled release packages. Windows support is planned for addition in a subsequent PR.

Build automation is based on a modified version of BAWT by Paul Obermeier http://www.bawt.tcl3d.org/. BAWT is copyrighted by Paul Obermeier and distributed under the 3-clause BSD license.

This feature has been tested on Red Hat 8.X and Ubuntu 20.04.X

The tested and supported architecture is x64.

Required packages
Prior to build it is necessary to install development tools, p7zip compression and Xft font development packages
e.g.

yum install p7zip-16.02-20.el8.x86_64.rpm
sudo yum install libXft-devel
yum group install "Development Tools"

Database packages:
Oracle and ODBC for SQL Server will build without client libraries. Client libraries must be installed for all of Db2, MariaDB, PostgreSQL and MySQL for the HammerDB build to be successful.
For Db2 either the server or client can be installed, for example v11.5.7_linuxx64_server_dec.tar.gz.
HammerDB build auotmation will look for the Db2 installation in the location of the environment variable IBM_DB_DIR set using db2profile. e.g.

$ echo $IBM_DB_DIR
/home/ibm/sqllib

Environment variables MARIADB_CONFIG, PG_CONFIG and MYSQL_CONFIG must be set to the location of the respective config commands for each database.

$ export MARIADB_CONFIG=/opt/mariadb/mariadb-10.6.7-linux-systemd-x86_64/bin
$ export PG_CONFIG=/opt/postgresql/bin
$ export MYSQL_CONFIG=/opt/mysql/mysql-8.0.28-linux-glibc2.12-x86_64/bin

Once all the client libraries are installed, either clone the HammerDB repository:

git clone https://github.com/TPC-Council/HammerDB.git
Cloning into 'HammerDB'...
remote: Enumerating objects: 2116, done.
remote: Counting objects: 100% (643/643), done.
remote: Compressing objects: 100% (400/400), done.
remote: Total 2116 (delta 384), reused 438 (delta 239), pack-reused 1473
Receiving objects: 100% (2116/2116), 11.53 MiB | 7.60 MiB/s, done.
Resolving deltas: 100% (1163/1163), done.

or download the zipfile and unzip into the HammerDB-master directory.

From here cd to the Bawt-2.1.0 directory eg /home/hdb/HammerDB-master/Build/Bawt-2.1.0 and run the Linux build command

./Build-Linux.sh x64 Setup/HammerDB.bawt update

This will compile all the HammerDB dependencies and create the HammerDB distribution.

...
07:30:08 >   TarGzip
               Source directory: /home/hdb/HammerDB-master/Build/BawtBuild/Linux/x64/Release/Distribution/HammerDB-4.4
               Tar file        : /home/hdb/HammerDB-master/Build/BawtBuild/Linux/x64/Release/Distribution/HammerDB-4.4.tar.gz
07:30:10 > End FinalizeStage

07:30:10 > Summary
           Setup file     : /home/hdb/HammerDB-master/Build/Bawt-2.1.0/Setup/HammerDB.bawt
           Build directory: /home/hdb/HammerDB-master/Build/BawtBuild/Linux/x64/Release/Build
           Architecture   : x64
           Compilers      : gcc
           Global stages  : Finalize
           #  : Library Name         Version    Build time      Stages
           ----------------------------------------------------------------------
             1: Tcl                  8.6.12     3.11 minutes    Clean Extract Configure Compile Distribute
             2: Tk                   8.6.12     0.65 minutes    Clean Extract Configure Compile Distribute
             3: awthemes             9.3.1      0.01 minutes    Clean Extract Configure Compile Distribute
             4: clearlooks           1.0        0.00 minutes    Clean Extract Configure Compile Distribute
             5: db2tcl               2.0.1      0.18 minutes    Clean Extract Configure Compile Distribute
             6: expect               5.45.4     0.24 minutes    Clean Extract Configure Compile Distribute
             7: libressl             2.6.4      1.54 minutes    Clean Extract Configure Compile Distribute
             8: mariatcl             0.1        0.08 minutes    Clean Extract Configure Compile Distribute
             9: mysqltcl             3.052      0.08 minutes    Clean Extract Configure Compile Distribute
            10: oratcl               4.6        0.09 minutes    Clean Extract Configure Compile Distribute
            11: pgtcl                2.1.1      0.10 minutes    Clean Extract Configure Compile Distribute
            12: redis                0.1        0.00 minutes    Clean Extract Configure Compile Distribute
            13: tcltls               1.7.22     0.26 minutes    Clean Extract Configure Compile Distribute
            14: tkblt                3.2.23     0.26 minutes    Clean Extract Configure Compile Distribute
            15: tksvg                0.5        0.10 minutes    Clean Extract Configure Compile Distribute
           ----------------------------------------------------------------------
           Total: 6.77 minutes

In the BawtBuild directory the HammerDB distribution will be built and a tar.gz file installed in the /x64/Release/Distribution

/home/hdb/HammerDB-master/Build/BawtBuild/Linux/x64/Release/Distribution
$ ls
HammerDB-4.4  HammerDB-4.4.tar.gz

In the InputLibs directory the source is compressed into .7z files for installation and these are updated for each build if changes are detected, the .7z files can also be deleted if preferred and will then be recreated on the next build.

Note that some packages have been modified compared to the originals, this build automation brings all changes into the HammerDB repository.

awthemes-9.3.1     db2tcl-2.0.1.7z    libressl.bawt      oratcl-4.6      redis-0.1.7z      tcltls.bawt      tksvg-0.5
awthemes-9.3.1.7z  db2tcl.bawt        mariatcl-0.1       oratcl-4.6.7z   redis.bawt        Tk-8.6.12        tksvg-0.5.7z
awthemes.bawt      expect-5.45.4      mariatcl-0.1.7z    oratcl.bawt     Tcl-8.6.12        Tk-8.6.12.7z     tksvg.bawt
clearlooks-1.0     expect-5.45.4.7z   mariatcl.bawt      pgtcl-2.1.1     Tcl-8.6.12.7z     Tk.bawt
clearlooks-1.0.7z  expect.bawt        mysqltcl-3.052     pgtcl-2.1.1.7z  Tcl.bawt          tkblt-3.2.23
clearlooks.bawt    libressl-2.6.4     mysqltcl-3.052.7z  pgtcl.bawt      tcltls-1.7.22     tkblt-3.2.23.7z
db2tcl-2.0.1       libressl-2.6.4.7z  mysqltcl.bawt      redis-0.1       tcltls-1.7.22.7z  tkblt.bawt

If a previous build is detected without modified dependency source, packages will not be recompiled.

08:20:03 > Summary
           Setup file     : /home/oracle/HammerDB/Build/Bawt-2.1.0/Setup/HammerDB.bawt
           Build directory: /home/oracle/HammerDB/Build/BawtBuild/Linux/x64/Release/Build
           Architecture   : x64
           Compilers      : gcc
           Global stages  : Finalize
           #  : Library Name         Version    Build time      Stages
           ----------------------------------------------------------------------
             1: Tcl                  8.6.12     0.00 minutes    None
             2: Tk                   8.6.12     0.00 minutes    None
             3: awthemes             9.3.1      0.00 minutes    None
             4: clearlooks           1.0        0.00 minutes    None
             5: db2tcl               2.0.1      0.00 minutes    None
             6: expect               5.45.4     0.00 minutes    None
             7: libressl             2.6.4      0.00 minutes    None
             8: mariatcl             0.1        0.00 minutes    None
             9: mysqltcl             3.052      0.00 minutes    None
            10: oratcl               4.6        0.00 minutes    None
            11: pgtcl                2.1.1      0.00 minutes    None
            12: redis                0.1        0.00 minutes    None
            13: tcltls               1.7.22     0.00 minutes    None
            14: tkblt                3.2.23     0.00 minutes    None
            15: tksvg                0.5        0.00 minutes    None
           ----------------------------------------------------------------------
           Total: 0.07 minutes

To reset the build environment, remove the BawtBuild directory under the Build directory. All packages will then be recompiled.

$ ls
Bawt-2.1.0  BawtBuild
$ rm -rf BawtBuild/

[memmertoIBM]

Do we have to bring in all of these dependencies in uncompressed source form? Surely we could check in just the tarballs (and patches). I don't see why we need to track 3000+ files instead of just a handful.

[sm-shaw]

yes, I originally had a format just including the .7z files which are used by BAWT instead of the uncompressed source (one of the first steps is to zip all of the source) however I changed this to the uncompressed source format for 2 reasons:

1. There are a fair number of changes in the packages from what is currently available elsewhere eg modifying the tdbc::odbc package to add an evaldirect command, also db2tcl, mariatcl and mysqltcl were changed to make the build work correctly. If someone wants to modify the source of any of these dependencies and issue a pull request in future to the compiled part how do we track the changes if they are zipped? For example, I will need to modify what is in the source of most/all of these directories to support the Windows build with MSVC as well as a next step. Also, we had a discussion around improving TclWinFlushDirtyChannels on windows so having the source will make a PR to do this more straightforward.
   One of the main reasons for adding this feature was so people can know exactly what goes into the HammerDB and track the changes over time. I considered an alternative is to convert the dependencies into submodules, however all of my modified versions are in the sm-shaw github location so I concluded it would make more sense storing these in hammerDB instead as my repositories are solely to enable development and issue PRs back to the HammerDB project anyway.

2. Currently on GitHub HammerDB is shown as 99% Tcl, however given the compiled packages this is not necessarily accurate as when the source is included this shows as:
   C 64.2% Tcl 13.0% Roff 9.5% Makefile 4.7% M4 3.5% Shell 1.8% Other 3.3%
   Including the dependency source does make it clearer showing exactly what goes into the HammerDB build and that a big part of it is the compiled C libraries that go into the release files and we have done a fair bit of work in these as well, db2tcl is a great example in adding bind variables.

Unless I've made an error here with the files included the PR should not be distributing the Db2 server package. If i've done this by mistake then let me know where and I'll fix it.

The instructions say that the builder has to download and install v11.5.7_linuxx64_server_dec.tar.gz themselves as they also have to for PostgreSQL, MySQL and MariaDB - there is definitely no intention to include any of the database installations in HammerDB itself however these will all be needed to be installed separately to do a full build.

[ottok]

I strongly oppose commit 9df7acf - it should never be accepted. First of all it has 5000+ files. This is surely the wrong approach. Secondly, there is no commit message body, just a title. You can't do such a massive change without any description of the change.

The proper way to do "build automation" is to have a shell script or Makefile or whatever that lists every step needed to compile the sources. Over time that script will then evolve.

The script might have external references to download some dependencies as pre-built binaries from somewhere. Or it might clone other repositories, or use git submodules, or just install with yum and apt the build dependencies. But you do not include build dependencies in this repository, and you surely should never include them as binary files!

[memmertoIBM]

@sm-shaw

Regarding 1) - yes, the patch set might be large, but this should give us some motivation to get those patches accepted upstream and then the patch set will diminish in size. There's no reason to check in the upstream sources for version x.y.z -- a tarball (or as Otto suggests - just recording the metdata required to fetch version x.y.z at build time) is preferable. (FWIW, the FreeBSD ports system - home to thousands of "ported" apps - exists as metadata + patches.)

Regarding 2) - hammerdb is mostly tcl. We shouldn't care about the composition of our dependencies.

And regarding my previous comment about db2 - which I since deleted - I mis-read your description.

[ottok]

1. • yes, the patch set might be large, but this should give us some motivation to get those patches accepted upstream and then the patch set will diminish in size

What patches? There is now just one single giant undocumented commit.

If say libfoo-1.2.3 was imported as source code in one commit and then on top of that was commit abc123 "Extend libfoo to be compatible with OpenSSL" you could take that commit and send it upstream for libfoo for inclusion with the motivation that it is already used in HammerDB since commit abc123, but now nothing is visible in the giant commit, just one huge blob.

[sm-shaw]

This is great input and much appreciated, it's definitely worth getting it right from the outset and the key challenge really stems from not having done this at the outset when issue #36 was raised. However, this gives us a good base to work from, as it does work, so shows the final destination of what it should look like on the system doing the build rather than in the git repository.

So the initial challenge in deciding how to do this is that a number of the dependencies are modified in some way, and a number of references to git submodules suggested that submodules add complexity and are not the way to go. So HammerDB needs its own versions of these dependencies, and these can and will change over time. e.g they will all change to add Windows build support as well.

So, as proposed, how about taking this approach?

> The script might have external references to download some dependencies as pre-built binaries from somewhere.

This should be straightforward, all the builds are converted to .7z files anyway. I could add the feature to download everything currently added as source as a .7z file from an external location when running the build. BAWT should already support this. The build approach would then be almost identical to what is there now, with just an additional step to do the download.

[sm-shaw]

As per discussion, pull request has now been updated so the InputLibs directory and all contents have been removed and instead the InputLibs directory is created on first build and all are fetched as .7z files from htttp://www.hammerdb.com instead.
Otherwise, the build process is identical to that described above, the only difference is that the files are fetched as zipped rather than included in the repository either as source or zipped. Retested on Ubuntu and Red Hat.

[abondvt89]

Merging Pull Request as voted on by TPC-OSS subcommittee on April 5, 2022.