tuptime-manual.txt

                         ----------------------
                            Tuptime    Manual
                         ----------------------
                              version 5.2.4
                                Ricardo F.
                               01/May/2024



============
| Abstract |
============

Tuptime reports historical and statistical real time of the system, preserving it between restarts.
Indeed, it can:

  - Count system startups
  - Register first boot time (since installation time)
  - Count nicely and accidentally shutdowns 
  - Uptime and downtime percentage since first boot time
  - Accumulated system uptime (running and sleeping), downtime and total
  - Register used kernels and boot IDs
  - Report current uptime
  - Print formatted table or list with the system history
  - Narrow reports since, until or at a given startup or timestamp
  - Output in csv format



=================
| Prerequisites |
=================

Operating systems:
   - Linux
   - BSD

Software:
   - Python 3.x
   - Python modules (most included in python core by default):
       sys, os, argparse, locale, platform, signal, logging, sqlite3, datetime

If a distro is used, all is met with:
    - In Linux:  'python' package >= 3.0
    - In FreeBSD: 'python3' and 'py36-sqlite3'

Also, execution via Docker is available, see note below.


================
| Installation |
================

For Linux systems:

   + From distribution repository:

       Debian  - https://packages.debian.org/tuptime
       Ubuntu  - https://packages.ubuntu.com/tuptime

           # apt install tuptime

       Fedora, EPEL  - https://src.fedoraproject.org/rpms/tuptime

           # dnf install tuptime
           # systemctl enable tuptime.service && systemctl start tuptime.service
           # systemctl enable tuptime-sync.timer && systemctl start tuptime-sync.timer


   + Using bash script:
      
       Download and execute the installation script:

           # bash < <(curl -Ls https://git.io/tuptime-install.sh)

   + Manual install:

       Clone repository and copy executable file:

           # install -m 755 tuptime/src/tuptime /usr/bin/tuptime

       Add _tuptime user with sysusers or useradd, choose one:

           With sysusers:

               # install -m 644 tuptime/src/systemd/tuptime.sysusers /usr/lib/sysusers.d/tuptime.conf
               # systemd-sysusers /usr/lib/sysusers.d/tuptime.conf

           With useradd:

               # useradd --system --no-create-home --home-dir '/var/lib/tuptime' \
                 --shell '/bin/false' --comment 'Tuptime execution user' _tuptime

       Execute tuptime with a privilege user for create DB path:

           # tuptime

       Change owner of the DB path and file:

           # chown -R _tuptime:_tuptime /var/lib/tuptime

       Add sync execution with cron or timer, choose one:

           With cron file:

               # install -m 644 tuptime/src/cron.d/tuptime /etc/cron.d/tuptime

           With systemd timer:

               # install -m 644 tuptime/src/systemd/tuptime-sync.* /lib/systemd/system/
               # systemctl enable tuptime-sync.timer && systemctl start tuptime-sync.timer

       Copy systemd service file and enable it:

           # install -m 644 tuptime/src/systemd/tuptime.service /lib/systemd/system/tuptime.service
           # systemctl enable tuptime.service && systemctl start tuptime.service

       Or if it is an old RedHat system or derivate, copy the init file:

           # install -m 755 tuptime/src/init.d/redhat/tuptime /etc/init.d/tuptime
           # chkconfig --add tuptime && chkconfig tuptime on

       Or if it is and old Debian system or derivate, copy the init file:

           # install -m 755 tuptime/src/init.d/debian/tuptime /etc/init.d/tuptime
           # update-rc.d tuptime defaults
           # /etc/init.d/tuptime start

       That's all, enjoy it.


For BSDs systems:

   + On FreeBSD and derivates using packages or ports:

           # pkg install tuptime
      
           (or)
       
           # /usr/ports/sysutils/tuptime

       Add the suggested cron entry and enable it on rc:

           # sysrc tuptime_enable=YES
           # service tuptime start

   + Manual install on FreeBSD, OpenBSD, NetBSD and derivates:

       Please, read misc/bsd-manual-install.txt


Via Docker:

   Tuptime could be executed inside a container via Docker, but it still
   requires the Systemd integration on the monitored host.
   Please, read misc/docker/docker-notes.txt


Partial support for OSX (Darwin) system:

   Tuptime is compatible with OSX, but the integration with LaunchD is not
   complete. Please, read misc/osx-notes.txt


Note about Python 2.7:

   Tuptime <= 3.5.0 can be executed with Python 2.7.
   Set the shebang line to '#!/usr/bin/env python' to get it.



===========================
| Command line parameters |
===========================

These are the command line options, no configuration file is used:

  -h, --help            show this help message and exit
  -A STARTUP, --at STARTUP
                        limit to this startup number
  -b, --bootid          show boot identifier
  -c, --csv             csv output
  -d DATETIME_FMT, --date DATETIME_FMT
                        datetime/timestamp format output
  -e DECIMALS, --dec DECIMALS
                        number of decimals in percentages
  -E, --exclude STARTUP
                        startup numbers to exclude
  -f FILE, --filedb FILE
                        database file (/var/lib/tuptime/tuptime.db)
  -g, --graceful        register a graceful shutdown
  -i, --invert          startup number in reverse count | swich between
                        longest/shortest on default output
  -k, --kernel          show kernel version
  -l, --list            enumerate system life as list
  -n, --noup            avoid update values into DB
  -o TYPE, --order TYPE
                        order enumerate by [u|r|s|e|d|k]
  -p, --power           show power states run + sleep
  -q, --quiet           update values into DB without output
  -r, --reverse         reverse order in listings
  -s, --seconds         output time in seconds and epoch
  -S STARTUP, --since STARTUP
                        limit from this startup number
  -t, --table           enumerate system life as table
  --tat TIMESTAMP       system status at epoch timestamp
  --tsince TIMESTAMP    limit from this epoch timestamp
  --tuntil TIMESTAMP    limit until this epoch timestamp
  -U STARTUP, --until STARTUP
                        limit up until this startup number
  -v, --verbose         verbose output
  -V, --version         show version


-A <STARTUP>, --at <STARTUP>
  Limit values only to this startup number. Take it from (-t) table or
  (-l) list reports. Negative values are allowed.

  Examples:
          tuptime -A 34
          tuptime --at 22
          tuptime -l -A -1  # List last register


-b, --bootid
  Add boot identifier to the output.

  Examples:
         tuptime -b
         tuptime -t --bootid


-h, --help
  Print a quick reference of the command line parameters.

  Examples:
         tuptime -h
         tuptime --help


-c, --csv
  Report in csv format.

  Examples:
         tuptime --csv
         tuptime --csv -t


-d <DATETIME_FMT>, --date=<DATETIME_FMT>
  Change the datetime/timestamp format.
  By default the output use the configured system locales.
  The following is a list of all the format codes that the 1989 C standard
  requires, and these work on all platforms with a standard C implementation.

        %a	Weekday as locale’s abbreviated name.
        %A	Weekday as locale’s full name.
        %w	Weekday as a decimal number, where 0 is Sunday and 6 is 
        	Saturday.
        %d	Day of the month as a zero-padded decimal number.
        %b	Month as locale’s abbreviated name.
        %B	Month as locale’s full name.
        %m	Month as a zero-padded decimal number.
        %y	Year without century as a zero-padded decimal number.
        %Y	Year with century as a decimal number.
        %H	Hour (24-hour clock) as a zero-padded decimal number.
        %I	Hour (12-hour clock) as a zero-padded decimal number.
        %p	Locale’s equivalent of either AM or PM.
        %M	Minute as a zero-padded decimal number.
        %S	Second as a zero-padded decimal number.
        %f	Microsecond as a decimal number, zero-padded on the left.
        %z	UTC offset in the form +HHMM or -HHMM (empty string if the 
        	the object is naive).
        %Z	Time zone name (empty string if the object is naive).
        %j	Day of the year as a zero-padded decimal number.
        %U	Week number of the year (Sunday as the first day of the week) 
        	as a zero padded decimal number. All days in a new year 
        	preceding the first Sunday are considered to be in week 0.
        %W	Week number of the year (Monday as the first day of the week) 
        	as a decimal number. All days in a new year preceding the 
        	first Monday are considered to be in week 0.
        %c	Locale’s appropriate date and time representation.
        %x	Locale’s appropriate date representation.
        %X	Locale’s appropriate time representation.
        %%	A literal '%' character.

  To see the full set of format codes supported on your platform, consult the
  strftime(3) documentation.

  Examples:
         tuptime -d '%X %x'  # (locale by default)
         tuptime -d '%H:%M:%S   %m-%d-%Y'  # MDY style
         tuptime -d '%H:%M:%S   %d-%b-%Y'  # DMY style


-e DECIMALS, --dec DECIMALS
  Change the decimal length in percentages. The number is rounded to this value.

  Examples:
         tuptime --dec 9
         tuptime --power --dec 9


-E <STARTUP>, --exclude <STARTUP>
  Avoid take into account the startup numbers defined. Multiple values must be
  separated by comma ',' and ranges with dash '-'. Default output prints
  'trimmed' if the excluded numbers are in middle of the range.

  Examples:
         tuptime -E 9
         tuptime -E '9-20'
         tuptime --exclude '2,10-20,50'


-f <FILE>, --filedb=<FILE>
  Define an alternative database file. Default is located in 
  '/var/lib/tuptime/tuptime.db'. It takes precedence over environmental variable
  'TUPTIME_DBF'.

  Examples:
         tuptime -f /var/lib/tuptime/tuptime.db  # (Default)
         tuptime -f /tmp/test1.db
         tuptime --filedb /tmp/test2.db

  
-g, --graceful
  Register the time in DB as a graceful shutdown. This option is the way that
  tuptime have for know if is a good or a bad shutdown. This is used in the 
  init.d and systemd files at the shutdown process.

  Examples:
         tuptime -g
         tuptime --graceful


-i, --invert
  Print startup number in reverse count starting since last boot when it's
  used in conjunction with (-t), (-l), or (--tat) options. Equal than the output of
  'journalctl --list-boots'.
  Change Longest to Shortest values on default output report.
  Change Total Uptime to Downtime on (-m) option.

  Examples:
         tuptime -t -i
         tuptime -tib
         tuptime -l --invert
         tuptime -i


-k, --kernel
  Add kernel information to the output.

  Examples:
         tuptime -k
         tuptime -lk
         tuptime --kernel
         tuptime -t --kernel


-l, --list
  Enumerate system life as a list.

  Examples:
         tuptime -l
         tuptime --list


-n, --noup
  Avoid update values in the DB with the current btime, uptime, etc. Useful 
  when reporting modified DB files.

  Examples:
         tuptime -n
         tuptime --noup


-o [u|r|s|e|d|k], --order=[u|r|s|e|d|k]
  Order enumerate output from table or list by:
    <u>  uptime
    <r>  runtime
    <s>  sleep time
    <e>  end status
    <d>  downtime
    <k>  kernel
  Therefore, only works in conjunction with (-t) or (-l) options.

  Examples:
         tuptime -t -o e
         tuptime -l -o k
         tuptime -t --order u
         tuptime -l --order d


-p, --power
  Print the running and sleeping accumulated time for the correspondent uptime slot.
  Note that only Python >= 3.6 can register this values.

  Examples:
         tuptime -p
         tuptime --power
         tuptime -tp
         tuptime -l --power


-q, --quiet
  Update values into database without compute and print output values.

  Examples:
         tuptime -q
         tuptime --quiet


-r, --reverse
  Reverse order for table or list output.

  Examples:
         tuptime -r
         tuptime -lur
         tuptime -teo --reverse
         tuptime -l --reverse


-s, --seconds
  Change default human readable datetime/timestamp style and print times in seconds
  and datetimes in epoch.

  Examples:
         tuptime -s
         tuptime --seconds


-S <STARTUP>, --since <STARTUP>
  Limit values only from this startup number. Take it from (-t) table or
  (-l) list reports. Negative values are allowed.

  Examples:
          tuptime -S 34
          tuptime --since 22
          tuptime -l -S -10

  
-t, --table
  Enumerate system life as a table.

  Examples:
         tuptime -t
         tuptime --table


--tat
  Report system status at specific timestamp, register number enclosed and time
  elapsed and remaining in that matched status.

  Examples:
         tuptime --tat 1555280149
         tuptime --tat `date -d "1 day ago" +%s`
         tuptime -cs --tat 1555281311


--tsince <TIMESTAMP>
  Limit report from the given epoch timestamp. Negative values are allowed.
  Options (-S) and (-U) have precedence over this one.

  Examples:
         tuptime --tsince 1454998872
         tuptime --tsince `date -d "20-JAN-18" +%s`
         tuptime --tsince -31536000  # Since one year ago


--tuntil <TIMESTAMP>
  Limit report until the given epoch timestamp. Negative values are allowed.
  Options (-S) and (-U) have precedence over this one.

  Examples:
         tuptime --tuntil 1454999619
         tuptime --tuntil `date -d "2018-01-20 16:21:42" +%s`
         tuptime --tuntil -2592000  # Until one month ago


-U <STARTUP>, --until <STARTUP>
  Limit values only up until this startup number. Take it from (-t) table or
  (-l) list reports. Negative values are allowed.

  Examples:
          tuptime -U 45
          tuptime --until 11
          tuptime -l -U -10


-v, --verbose
  Print information about the internals of tuptime. It's good for debugging 
  how it gets the variables.

  Examples:
         tuptime -v
         tuptime --verbose


-V, --version:
  Print version number and exit.

  Examples:
         tuptime -V
         tuptime --version



=========================
| Environment variables |
=========================

List of supported environmental variables:

TUPTIME_DBF
  Set an alternative database file path. The argument -f, --filedb takes
  precedence over this.

  Example for ksh / sh / bash:
         export TUPTIME_DBF="/opt/tuptime.db"

  Example for csh / tcsh:
         setenv TUPTIME_DBF /opt/tuptime.db 

  Example for systemd unit:
         Environment="TUPTIME_DBF='/opt/tuptime.db'"



======================
| Schedule execution |
======================

It's important to have a schedule execution. If the init or systemd scripts
are installed as it needs, the program only update values at startup and 
shutdown, but if the system fails, hangs or whatever, the uptime time will
be lost. Be sure that the cron entry or, in other case, the systemd
tuptime-sync.[timer|service] units are installed as expected.

Why is it needed the use of an other .service file with the name tuptime-sync?
The default tuptime.service file uses the option 'RemainAfterExit=true' and
nowadays the systemd timer can't restart an already running service, so, this
is the best workaround.

By default, both have a 5 minutes scheduled execution. Feel free to lower 
that value if your requirements are narrow.



==================
| Default output |
==================

System startups:
 Total number of system startups registered since first timestamp available.

System shutdowns:
  Total number of shutdowns done correctly or incorrectly.

System life:
  Time counter since first startup timestamp available.

System uptime:
System downtime:
  Percentage of time and time counter.

Longest uptime:
Longest downtime:
  Time counter and date with the longest/shortest uptime register. 

Average uptime:
Average downtime: 
  Average time counter.

Current uptime:
  Actual time counter and datetime since registered boot timestamp.



======================================
| DB format changes between versions |
======================================

From 2.x to 3.x:
    From 2.x to 3.0, reorder columns, added offbtime, endst and downtime. From 3.0 to 3.1,
    added kernel column. The migration scripts are loated at:
    
    https://github.com/rfmoz/tuptime/blob/master/misc/cripts/db-tuptime-migrate-2.0-to-3.0.sh
    https://github.com/rfmoz/tuptime/blob/master/misc/cripts/db-tuptime-migrate-3.0-to-3.1.sh
 
From 3.x to 4.x:
    Added rntime and slptime registers, also uptime, rntime, slptime, downtime are now integers.
    The migrations is done automatically from Tuptime. Also, the migration script
    is located at:

    https://github.com/rfmoz/tuptime/blob/master/misc/cripts/db-tuptime-migrate-3.1-to-4.0.sh
 
From 4.x to 5.x:
    Added bootid register. The migrations is done automatically from Tuptime.
    Also, the migration script is located at:

    https://github.com/rfmoz/tuptime/blob/master/misc/cripts/db-tuptime-migrate-4.0-to-5.0.sh


==================
| Database Specs |
==================

Tuptime use a sqlite3 database located in '/var/lib/tuptime/tuptime.db' with
the following format:

tuptime (bootid text,
         btime integer,
         uptime integer,
         rntime integer,
         slptime integer,
         offbtime integer,
         endst integer,
         downtime integer,
         kernel text)

bootid   Unique boot identifier (if it exists)
btime    Startup timestamp in epoch
uptime   Uptime seconds  
rntime   Running time seconds
slptime  Sleeping time seconds
offbtime Shutdown timestamp in epoch
endst    Type of shutdown [1 ok | 0 bad]
downtime Downtime seconds
kernel   Name of the kernel

The number of startup sequence is extracted from 'rowid', the signed integer 
key that uniquely identifies the row within its table in sqlite.

The bootid value is available since Linux 2.6.20 and FreeBSD 11.0.

For reset all the values, simply delete the database file.

It is possible to query and modify it directly, but it's better to use any of
the scripts located under 'misc/scripts' for modify, join or check it.

If a complete row is deleted and the database is not recreate completely (vacuum),
rowid keep the real information about startup number. Tuptime will notify about
'Deleted rows in DB' if the verbose mode is enabled.



============================
| About sync date and time |
============================

Substantial time jumps within a running system may trigger unespected behavior.
Take care of the hardware clock RTC (if the system have), or how the operating
system initializes their clock at startup.

If the system starts with a incorrect datetime and some external time sync is
used, like ntpd or timesyncd, a few time after boot the btime reported by the
system may change (it is a live value computed by now() - uptime). This behaviour
is also related with adjustments performed by adjtime(3), systems running inside
virtualized environments, servers with high load or with high disk I/O, wrong
computation of jiffies / HZ and the problema of lost ticks.

A slightly drift is compensate over uptime/downtime values. Large drift can 
produce unreal registers and output. For example, Raspberry Pi doesn't have any
clock and initializes from 1 January 1970. A downtime of 0, when it's evident
that it was physically impossible, it's a common indicator than the clock had a
large sync jump.

As Tuptime have time dependency, its Systemd service unit pulls time-sync.target
and, therefore, is ordered to start before it. Note that it doesn't assure that 
the clock synchronization is achieved, only that it needs the start of the 
synchronization service and the date isn't going to be before last shutdown. On
last releases of Systemd, there is a new unit that delays reaching time-sync.target 
until stay on sync 'systemd-time-wait-sync', but it isn't enable by default nowadays.

On Linux systems running Systemd, when the system clock has been synchronized, the
file /run/systemd/timesync/synchronized is created. Tuptime check it and print a
verbose line when it doesn't exists 'INFO:Timesync state file not found'.

A few recommendations / workarounds are:

Enable 'systemd-time-wait-sync'. But take care because it can delay the execution
more than expected and a startup can be lost. Note that if it couldn't reach the 
synchronization before shutdown, Tuptime isn't going to be executed on that startup. 
Check open issues on Systemd's GitHub related to this scenario.

Use 'systemd-timesyncd' if it is available (timedatectl set-ntp true), it can also
keep the time in systems without hardware clock. Disable any other time sync service
like 'ntp', 'chrony', 'openntpd' to avoid issues, most related with the use in
conjunction with 'systemd-time-wait-sync'.

Use any time sync service like 'ntp', 'chrony' or 'openntpd' and disable
'systemd-timesyncd'. Add the time sync service requirement on 'tuptime.service' at
the end of 'After=' and 'Wants=' lines.

Force a time sync in the 'tuptime.unit' file. Add under '[Service]' definition the 
line 'ExecStartPre=/usr/bin/ntpdate pool.ntp.org'

If the systems is a single-board computer, like Raspberry Pi or Beaglebone, use a
RTC module and configure it properly.

Without Systemd, use a ntp client and delay the execution of tuptime until the time
is synced. Also, 'ntp-wait' can be added to the init script and the cron line
preceeding the tuptime execution.

Try and failure is the best way to find the right solution for the environment in 
which the system is running.

The drift value that Tuptime have reference is reported in verbose mode 'tuptime -v'
in the line 'INFO:Drift over btime'. Values around +-1 are common and can be
considered as normal.

Systemd timer issues:

    https://github.com/systemd/systemd/issues/14061
    https://github.com/systemd/systemd/issues/8683
    https://github.com/systemd/systemd/issues/5097

Good reads:

    https://tools.ietf.org/html/rfc1589
    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=119971
    http://man7.org/linux/man-pages/man2/clock_gettime.2.html
    https://unix.stackexchange.com/questions/118631/how-can-i-measure-and-prevent-clock-drift
    Timekeeping in VMware Virtual Machines (Document)
    https://venam.nixers.net/blog/unix/2020/05/02/time-on-unix.html
    https://papers.freebsd.org/2002/phk-timecounters.files/timecounter.pdf



========================
| Date and Epoch notes |
========================

The options --tsince, --tuntil and --tat use epoch timestamp format, as quick
reference here are some notes:

Seconds equivalences:
        1 year    31536000 seconds
        1 month    2592000 seconds
        1 week      604800 seconds
        1 day        86400 seconds
        1 hour        3600 seconds

Convert human datetime to epoch:
        date -d "JAN-18-2018" +%s
        date -d "20-JAN-18" +%s
        date -d "2018-01-20 16:21:42" +%s
        date -d "2018-01-20 16:21:42 GMT+2" +%s
        date -d "3 week ago" +%s
        date -d "1 year ago" +%s
        date -d "now" +%s

Convert epoch to human datetime:
        date -d "@1516748400"

Preceding conversions uses GNU date, on FreeBSD install "coreutils" and call
them with "gdate".

Examples of using it directly:

         tuptime --tsince `date -d "1 day ago" +%s`
         tuptime --tuntil `date -d "1 day ago" +%s`
         tuptime --tsince `date -d "1 week ago" +%s` --tuntil `date -d "1 day ago" +%s`
         tuptime --tat `date -d "1 day ago" +%s`

More info about datetime input formats:
        man date
        'https://www.gnu.org/software/coreutils/manual/html_node/
        Date-input-formats.html#Date-input-formats'



===========
| Caveats |
===========

Tuptime can be executed with 'root' user, but it's recommendable the use of an
unprivileged user, like '_tuptime' as the installation section points to avoid
security problems.

Tuptime tries to save values on DB in every execution. If the user doesn't have
permissions over the file, an informational verbose message is generated. This
situation is handled for print values, but not when register a graceful shutdown.
That's why a user who can write in the database file must execute tuptime, usually
from the init manager, to update values at startup and shutdown, at least.

Tuptime register a new startup time (btime) reported by the system at first
execution after boot, likewise the new startup number and bootid. Also the
previous startup offbtime and downtime are also written. None of them changes
after that. Uptime may shift to accommodate drift and kernel name could be
replaced if live patching is used.

Tuptime is backed by a SQLite database. It is highly resilient but fails can
occur in any software and hardware level, producing a database corruption. Here
is an excellent documentation about it https://www.sqlite.org/howtocorrupt.html

If a backup of the DB file is needed in other place, take a look to Litestream.
This is a streaming replication tool for SQLite databases, it works fine with
Tuptime. https://litestream.io/

The uptime reference is linked with the wall clock time elapsed since boot, not
the running effective time. Both the running (effective time) and hibernate
power states fall within the applicable uptime slot.

If the system have configured a time zone with daylight saving time (DST),
to avoid problems when the time change, the real time clock (RTC) should have
set to universal time coordinated (UTC), not local time. Check it with 
'timedatectl' command.  Reports involving DST changes are completely right.
The system current local time zone is the reference for all timestamps printed.
https://tldp.org/HOWTO/Clock-2.html

If the Linux system execute a full force shutdown or reboot, like for example
"systemctl --force --force reboot", as it doesn't follow the steps to produce
a clean shutdown, the "End status" remains in bad state.

The name of the tool is based on the contraction of the words Total Uptime.