Inkflux

Inkflux is a log tailer script designed for use with Squid Proxy access logs. It reads the log file, captures relevant information through regex, and sends metrics to InfluxDB.

Features

• Reads Squid access logs
• Captures relevant information using regex
• Sends metrics to InfluxDB throguh HTTP API
• Supports configuration through a JSON file

Requirements

• Python 3.x

• requests

• inotify

• dotenv

• InfluxDB (if you are here you probably already have an active instance)

• HTTP API enabled in influxdb.conf . (HTTPS Auth to be implemented)

Installation

1. Clone the repository:

   git clone https://github.com/MinimalDefence/Inkflux.git
   cd Inkflux

2. Install the required Python libraries:

   Optional: Create a Virtual Enviroment (helps keep things tidy)

   python3 -m venv new-env

   Activate the venv

   source new-env/bin/activate

   If you are not using venv:

   sudo apt update && sudo apt install python3-dotenv python3-requests python3-inotify -y

   If you are using venv :

   pip install requests inotify dotenv

   OR

   pip install -r requirements.txt

3. Create a .env file in the project directory specifying the path to your custom config.json and Inkflux log path. (you may skip this if using the provided configurations.)

   CONFIG_FILE_PATH="some/path/config.json"
   INKFLUX_LOG="some/path/to/inkflux.log"

4. Edit the config.json file with the necessary configurations:

   {
       "Nominal": {
           "influxdb_url":"http://localhost",
           "influxdb_port":"8086",
           "influxdb_db":"squid",
           "measurement_name": "squid.access_log",
           "logfile":"/var/log/squid/access.log",
           "read_from_end":"False",
           "regex":"default"
       },
       "Backlog": {
           "influxdb_url":"http://localhost",
           "influxdb_port":"8086",
           "influxdb_db":"squid",
           "measurement_name": "squid.access_log",
           "logfile":"/var/log/squid/access_backlog.log",
           "read_from_end":"False",
           "regex":"default"
       }
   }

   As of v1.1 the script accepts a backlog file for you to append past/lost metrics.

   Configuration Parameters

   • influxdb_url: The URL of the InfluxDB server. This is where your data will be sent. For example, http://localhost if the server is running locally.
   • influxdb_port: The port on which the InfluxDB server is listening. The default port for InfluxDB is 8086.
   • influxdb_db: The name of the database in InfluxDB where the data will be stored. For example, squid.
   • measurement_name: The name of the measurement (similar to a table in relational databases) where the data will be recorded. For example, squid-access_log.
   • read_from_end: A boolean value (True or False) indicating whether to start reading the log file from the end. If set to True, it will only read new entries appended to the log file.
   • regex: The regular expression used to parse the log file. If you have a custom log format, you can replace this with your own regex.
   • logfile: The path to the access.log file that you want to monitor. For example, /var/log/squid/access.log for the defaul squid path.

   If you wish to use a custom regex instead of the default provided in the script (if you have a custom squid log format for example) you may do so by changing the regex section from default to the regex pattern you wish to use.

       {"regex": "default"} # Change to your regex pattern or leave as is.

You should probably test your regex before passing it to the script. Most use regex101

You can enable debug mode in the logger section of the script to see how lines are being matched and formatted before being sent to InfluxDB. To do this, replace INFO with DEBUG in the logging section of the script:

level=logging.INFO, # Replace INFO with DEBUG

5. Permissions

In order to monitor logs in /var/log I suggest you create a dedicated group and the user you intend in using to run the script to that group:

# Create a new group
sudo groupadd inkflux

# Add your user to the new group
sudo usermod -aG inkflux yourusername

# Change the group ownership of the log file
sudo chown root:inkflux /var/log/squid/access.log

# Change the permissions of the log file to allow group read access
sudo chmod 640 /var/log/squid/access.log

Replace yourusername with the username you intend to use to run the script.

You are ready to run the script.

Usage

Run the sccript:

python3 inkflux.py

You will likely want the script to run in the background, for this you can use nohup or screen.

with nohup:

nohup python3 inkflux.py &

with screen:

screen -dmS inkflux python3 inkflux.py

License

This project is licensed under the GNU GPLv3 License.

Author

MinimalDefence