Skip to content

Latest commit

 

History

History
1119 lines (904 loc) · 68 KB

README.md

File metadata and controls

1119 lines (904 loc) · 68 KB
Raw

Check Mail

Various tools used to monitor mail services

Latest Release Go Reference go.mod Go version Lint and Build Project Analysis

Table of contents

Project home

See our GitHub repo for the latest code, to file an issue or submit improvements for review and potential inclusion into the project.

Overview

This repo contains various tools used to monitor mail services.

Tool Name Overall Status Tool Type Purpose
check_imap_mailbox_basic Stable Nagios plugin Monitor mailboxes for items (via Basic Auth)
check_imap_mailbox_oauth2 Alpha Nagios plugin Monitor mailboxes for items (via OAuth2)
list-emails Stable CLI app Generate listing of mailbox contents
lsimap Alpha CLI tool List advertised capabilities for specified IMAP server
xoauth2 Alpha CLI tool Convert given username and token to XOAuth2 formatted (or SASL XOAUTH2 encoded) string
fetch-token Alpha CLI tool Fetch OAuth2 Client Credentials token from specified token URL, emit to stdout or file
read-token Alpha CLI tool Read OAuth2 Client Credentials token from specified file

Features

check_imap_mailbox_*

There are two plugins which perform the same overall function, but utilize different mechanisms to authenticate to a specific IMAP server:

  • check_imap_mailbox_basic
    • uses Basic Auth (username/password) for authentication
  • check_imap_mailbox_oauth2
    • uses OAuth2 Client Credentials (client ID/secret) flow for authentication

Shared functionality:

  • Monitor specified mailboxes for an IMAP account
    • non-OK state returned for any items in specified mailboxes or errors encountered
    • OK state returned if all specified mailboxes are empty
  • Optional, leveled logging using rs/zerolog package
    • logfmt format output (to stderr)
    • choice of disabled, panic, fatal, error, warn, info (the default), debug or trace
  • TLS IMAP4 connectivity
    • port defaults to 993/tcp
    • network type defaults to either of IPv4 and IPv6, but optionally limited to IPv4-only or IPv6-only
    • user-specified minimum TLS version
  • Optional branding "signature"
    • used to indicate what Nagios plugin (and what version) is responsible for the service check result

list-emails

  • Check one or many mailboxes
  • Optional, leveled logging using rs/zerolog package
    • logfmt format output (to stderr)
    • choice of disabled, panic, fatal, error, warn, info (the default), debug or trace
    • bulk of logging directed to per-invocation log file
  • Multiple authentication options
    • Basic Auth (username/password)
    • OAuth2 Client Credentials (client ID/secret) flow
  • TLS IMAP4 connectivity
    • port defaults to 993/tcp
    • network type defaults to either of IPv4 and IPv6, but optionally limited to IPv4-only or IPv6-only
    • user-specified minimum TLS version
  • Minimal output to console unless requested
    • via debug logging level
  • Textile (Redmine compatible) formatted report generated per specified email account
    • overall summary
    • template copy/paste/modify report for posting to Redmine issues (aka, "tickets")
    • automatic replacement of Unicode characters outside of the MySQL utf8mb3 character set with a placeholder character
      • the intent is to help prevent MySQL errors when posting summary reports
        • e.g., ERROR 1366 (22007): Incorrect string value

lsimap

  • Quick one-off tool to list advertised capabilities for specified IMAP server
  • Leveled logging
    • console writer: human-friendly, colorized output
    • choice of disabled, panic, fatal, error, warn, info (the default), debug or trace
    • enable debug level to monitor submitted IMAP commands and received IMAP server responses
  • TLS IMAP4 connectivity
    • port defaults to 993/tcp
    • network type defaults to either of IPv4 and IPv6, but optionally limited to IPv4-only or IPv6-only
    • user-specified minimum TLS version

xoauth2

Standalone CLI app to convert given username and token to XOAuth2 formatted (or SASL XOAUTH2 encoded) string.

fetch-token

  • Fetch OAuth2 Client Credentials token from specified token URL
  • Automatic retry functionality
    • user configurable "max attempts" limit
  • Emit retrieved token to stdout (default) or file
  • Configurable token output format
    • plaintext/raw access token
    • JSON
  • Leveled logging
    • console writer: human-friendly, but (for this app) non-colorized output
    • choice of disabled, panic, fatal, error, warn, info (the default), debug or trace
    • by default this tool produces no log output
    • log messages written to stderr

read-token

  • Read OAuth2 Client Credentials token from specified file
  • Automatic detection of support token format
    • plaintext/raw access token
    • JSON
  • Leveled logging
    • console writer: human-friendly, but (for this app) non-colorized output
    • choice of disabled, panic, fatal, error, warn, info (the default), debug or trace
    • by default this tool produces no log output
    • log messages written to stderr

Requirements

The following is a loose guideline. Other combinations of Go and operating systems for building and running tools from this repo may work, but have not been tested.

Building source code

  • Go
    • see this project's go.mod file for preferred version
    • this project tests against officially supported Go releases
      • the most recent stable release (aka, "stable")
      • the prior, but still supported release (aka, "oldstable")
  • GCC
    • if building with custom options (as the provided Makefile does)
  • make
    • if using the provided Makefile
  • tc-hib/go-winres
    • if using the provided Makefile
    • used to generate Windows resource files

Running

  • Windows 10
  • Ubuntu Linux 18.04+

Office 365 (O365) permissions

The list-emails and check_imap_mailbox_oauth2 tools support OAuth2 Client Credentials flow authentication.

The check_imap_mailbox_oauth2 example illustrates connecting to an Office 365 (O365) "shared mailbox" (aka, a shared account).

  • The client-id, client-secret flag values are obtained from the application registration.
  • The https://outlook.office365.com/.default scopes value indicates that the permissions listed in the application registration should be used.

Testing was performed with these permissions/scopes set within the application registration:

API Permissions name Type Description Admin consent required
Microsoft Graph IMAP.AccessAsUser.All Delegated Read and write access to mailboxes via IMAP. No
Microsoft Graph User.Read Delegated Sign in and read user profile No
Office 365 Exchange Online IMAP.AccessAsApp Application IMAP.AccessAsApp Yes

The last one has to be granted by a tenant administrator.

Per https://blog.rebex.net/office365-ews-oauth-unattended:

Optionally, you can remove the delegated User.Read permission which is not needed for app-only application - click the context menu on the right side of the permission and select Remove permission.

Other sources have said the same thing: Delegated scopes are not needed for the client credentials flow; only the IMAP.AccessAsApp permission is required for the OAuth2 Client Credentials flow (used by tools in this project).

Lastly, an Office 365 tenant administrator needs to:

  1. register the service principals in Exchange
  2. add specific mailboxes in the tenant that will be allowed to be accessed by this plugin

See the check_imap_mailbox_oauth2 example or the official O365 o365-cred-flow-test-script test script to confirm that required settings are in place.

Worth noting: Support for the Client Credentials flow was added 2022-06-30.

Installation

From source

  1. Download Go
  2. Install Go
    • NOTE: Pay special attention to the remarks about $HOME/.profile
  3. Clone the repo
    1. cd /tmp
    2. git clone https://github.com/atc0005/check-mail
    3. cd check-mail
  4. Install dependencies (optional)
    • for Ubuntu Linux
      • sudo apt-get install make gcc
    • for CentOS Linux
      1. sudo yum install make gcc
  5. Build
    • for current operating system (using bundled dependencies)
      • go build -mod=vendor ./cmd/check_imap_mailbox_basic/
      • go build -mod=vendor ./cmd/check_imap_mailbox_oauth2/
      • go build -mod=vendor ./cmd/list-emails/
      • go build -mod=vendor ./cmd/lsimap/
      • go build -mod=vendor ./cmd/xoauth2/
      • go build -mod=vendor ./cmd/fetch-token/
      • go build -mod=vendor ./cmd/read-token/
    • for all supported platforms (where make is installed)
      • make all
    • for Windows
      • make windows
    • for Linux
      • make linux
  6. Locate generated binaries
    • if using Makefile
      • look in /tmp/check-mail/release_assets/check_imap_mailbox_basic/
      • look in /tmp/check-mail/release_assets/check_imap_mailbox_oauth2/
      • look in /tmp/check-mail/release_assets/list-emails/
      • look in /tmp/check-mail/release_assets/lsimap/
      • look in /tmp/check-mail/release_assets/xoauth2/
      • look in /tmp/check-mail/release_assets/fetch-token/
      • look in /tmp/check-mail/release_assets/read-token/
    • if using go build
      • look in /tmp/check-mail/
  7. Copy the applicable binaries to whatever systems needs to run them
  8. Deploy
    • Place list-emails in a location of your choice
    • Place lsimap in a location of your choice
    • Place xoauth2 in a location of your choice
    • Place fetch-token in a location of your choice
    • Place read-token in a location of your choice
    • Place check_imap_mailbox_basic in the same location where your distro's package manage has place other Nagios plugins
      • as /usr/lib/nagios/plugins/check_imap_mailbox_basic on Debian-based systems
      • as /usr/lib64/nagios/plugins/check_imap_mailbox_basic on RedHat-based systems
    • Place check_imap_mailbox_oauth2 in the same location where your distro's package manage has place other Nagios plugins
      • as /usr/lib/nagios/plugins/check_imap_mailbox_oauth2 on Debian-based systems
      • as /usr/lib64/nagios/plugins/check_imap_mailbox_oauth2 on RedHat-based systems
  9. Copy the template configuration file, modify accordingly and place in a supported location

NOTE: Depending on which Makefile recipe you use the generated binary may be compressed and have an xz extension. If so, you should decompress the binary first before deploying it (e.g., xz -d check_imap_mailbox_oauth2-linux-amd64.xz).

Using release binaries

  1. Download the latest release binaries
  2. Decompress binaries
    • e.g., xz -d check_imap_mailbox_oauth2-linux-amd64.xz
  3. Deploy
    • Place list-emails in a location of your choice
    • Place lsimap in a location of your choice
    • Place xoauth2 in a location of your choice
    • Place fetch-token in a location of your choice
    • Place read-token in a location of your choice
    • Place check_imap_mailbox_basic in the same location where your distro's package manager places other Nagios plugins
      • as /usr/lib/nagios/plugins/check_imap_mailbox_basic on Debian-based systems
      • as /usr/lib64/nagios/plugins/check_imap_mailbox_basic on RedHat-based systems
    • Place check_imap_mailbox_oauth2 in the same location where your distro's package manager places other Nagios plugins
      • as /usr/lib/nagios/plugins/check_imap_mailbox_oauth2 on Debian-based systems
      • as /usr/lib64/nagios/plugins/check_imap_mailbox_oauth2 on RedHat-based systems
  4. Copy the template configuration file, modify accordingly and place in a supported location

NOTE:

DEB and RPM packages are provided as an alternative to manually deploying binaries.

Configuration Options

check_imap_mailbox_basic

Command-line arguments

  • Flags marked as required must be set via CLI flag.
  • Flags not marked as required are for settings where a useful default is already defined.
Option Required Default Repeat Possible Description
h, help No No -h, --help Generate listing of all valid command-line options and applicable (short) guidance for using them.
folders Yes empty string No comma-separated list of folders Folders or IMAP "mailboxes" to check for mail. This value is provided as a comma-separated list.
username Yes empty string No valid username, often in email address format The account used to login to the remote mail server. This is often in the form of an email address.
password Yes empty string No valid password The remote mail server account password.
server Yes empty string No valid FQDN or IP Address The fully-qualified domain name of the remote mail server.
port No 993 No valid IMAP TCP port TCP port used to connect to the remote mail server. This is usually the same port used for TLS encrypted IMAP connections.
net-type No auto No auto, tcp4, tcp6 Limits network connections to remote mail servers to one of the specified types.
min-tls No tls12 No tls10, tls11, tls12, tls13 Limits version of TLS used for connections to remote mail servers.
logging-level No info No disabled, panic, fatal, error, warn, info, debug, trace Sets log level.
branding No false No true, false Toggles emission of branding details with plugin status details. Because this output may not mix well with branding information emitted by other tools, this output is disabled by default.
version No false No true, false Whether to display application version and then immediately exit application

check_imap_mailbox_oauth2

Required preparation

This plugin uses the OAuth2 Client Credentials flow to authenticate.

This requires registering an application with the authority for the resource that you wish to access. The specifics differ (at least slightly) for every IMAP account provider that you wish to interact with.

See the Office 365 (O365) permissions section for details specific to using this plugin with O365 mailboxes.

Command-line arguments

  • Flags marked as required must be set via CLI flag.
  • Flags not marked as required are for settings where a useful default is already defined.
Option Required Default Repeat Possible Description
h, help No No -h, --help Generate listing of all valid command-line options and applicable (short) guidance for using them.
folders Yes empty string No comma-separated list of folders Folders or IMAP "mailboxes" to check for mail. This value is provided as a comma-separated list.
scopes Yes empty string No comma-separated list of scopes Permissions needed by the application. If using the scopes defined by the application registration you must use the RESOURCE/.default format (e.g., https://outlook.office365.com/.default.
client-id Yes empty string No valid application ID associated with registered app Application (client) ID created during app registration.
client-secret Yes empty string No valid application secret associated with registered app Client secret (aka, "app" password).
shared-mailbox Yes empty string No valid shared mailbox name, often in email address format Email account that is to be accessed using client ID & secret values. Usually a shared mailbox among a team.
token-url Yes empty string No valid token URL The OAuth2 provider's token endpoint URL. E.g., https://accounts.google.com/o/oauth2/token for Google. See contrib/list-emails/oauth2/accounts.example.ini for O365 example.
port No 993 No valid IMAP TCP port TCP port used to connect to the remote mail server. This is usually the same port used for TLS encrypted IMAP connections.
net-type No auto No auto, tcp4, tcp6 Limits network connections to remote mail servers to one of the specified types.
min-tls No tls12 No tls10, tls11, tls12, tls13 Limits version of TLS used for connections to remote mail servers.
logging-level No info No disabled, panic, fatal, error, warn, info, debug, trace Sets log level.
branding No false No true, false Toggles emission of branding details with plugin status details. Because this output may not mix well with branding information emitted by other tools, this output is disabled by default.
version No false No true, false Whether to display application version and then immediately exit application

list-emails

Command-line arguments

  • The bulk of the settings for this application are provided via the accounts.ini configuration file.
  • It is not currently possible to specify all required settings by command-line
Option Required Default Repeat Possible Description
h, help No No -h, --help Generate listing of all valid command-line options and applicable (short) guidance for using them.
config-file No accounts.ini No valid path to INI configuration file for this application Full path to the INI-formatted configuration file used by this application. See contrib/list-emails/ for starter templates. Rename to accounts.ini, update with applicable information and place in a directory of your choice. If this file is found in your current working directory you need not use this flag.
log-file-dir No log No valid, writable path to a directory Full path to the directory where log files will be created. The user account running this application requires write permission to this directory. If not specified, a default directory will be created in your current working directory if it does not already exist.
report-file-dir No output No valid, writable path to a directory Full path to the directory where email summary report files will be created. The user account running this application requires write permission to this directory. If not specified, a default directory will be created in your current working directory if it does not already exist.
net-type No auto No auto, tcp4, tcp6 Limits network connections to remote mail servers to one of the specified types.
min-tls No tls12 No tls10, tls11, tls12, tls13 Limits version of TLS used for connections to remote mail servers.
logging-level No info No disabled, panic, fatal, error, warn, info, debug, trace Sets log level.
version No false No true, false Whether to display application version and then immediately exit application

Configuration file

Settings

NOTE: The email1 and email2 value below is for illustration. You are free to choose section names, though it is recommended to base them off of the username (sans @ symbol and domain part) for each email account. While only email1 is listed, many such entries (one per account) are supported.

The list-emails CLI app supports both Basic Auth and OAuth2 Client Credentials flow for authentication. Depending on the desired authentication type some settings are required, others ignored; if using Basic Auth settings specific to OAuth2 are ignored.

Basic Auth
Config file Setting Name Section Name Notes
server_name DEFAULT FQDN of IMAP server (e.g., outlook.office365.com)
server_port DEFAULT Usually 993
username email1 Often in the form of an email address
password email1 Account password
folders email1 Double quoted, comma separated
OAuth2
Config file Setting Name Section Name Notes
server_name DEFAULT FQDN of IMAP server (e.g., outlook.office365.com)
server_port DEFAULT Usually 993
client_id DEFAULT The ID associated with the application registration
client_secret DEFAULT Application secret (aka, "app" password)
scopes DEFAULT Comma-separated list of permissions needed by the application (e.g., https://outlook.office365.com/.default)
endpoint_token_url DEFAULT The OAuth2 provider's token endpoint URL.
shared_mailbox email1 Email address format (e.g., me@there.com)
folders email1 Double quoted, comma separated
Usage

There are two example INI files available which illustrate available configuration settings:

These files are intended as starting points for your own accounts.ini configuration file.

The current design (based off of the existing https://github.com/atc0005/list-emails project) limits all email account entries (reflected by different sections) to the same IMAP server. If you need to process accounts from different servers you will need a separate copy of the accounts.ini file for each server.

Once reviewed and adjusted, your copy of the accounts.ini file can be placed in one of the following locations to be automatically detected and used by this application:

  • alongside the list-emails (or list-emails.exe) binary (as accounts.ini)
  • at $HOME/.config/check-mail/accounts.ini on a UNIX-like system (e.g., Linux distro, Mac)
  • at C:\Users\YOUR_USERNAME\AppData\check-mail\accounts.ini on a Windows system

You may also place the file wherever you like and refer to it using the -config-file (full-length flag name). See the Examples and Command-line arguments sections for usage details.

lsimap

Command-line arguments

  • Flags marked as required must be set via CLI flag.
  • Flags not marked as required are for settings where a useful default is already defined.
Option Required Default Repeat Possible Description
h, help No No -h, --help Generate listing of all valid command-line options and applicable (short) guidance for using them.
server Yes empty string No valid FQDN or IP Address The fully-qualified domain name of the remote mail server.
port No 993 No valid IMAP TCP port TCP port used to connect to the remote mail server. This is usually the same port used for TLS encrypted IMAP connections.
net-type No auto No auto, tcp4, tcp6 Limits network connections to remote mail servers to one of the specified types.
min-tls No tls12 No tls10, tls11, tls12, tls13 Limits version of TLS used for connections to remote mail servers.
logging-level No info No disabled, panic, fatal, error, warn, info, debug, trace Sets log level.
version No false No true, false Whether to display application version and then immediately exit application

xoauth2

Command-line arguments

  • Flags marked as required must be set via CLI flag.
  • Flags not marked as required are for settings where a useful default is already defined.
Option Required Default Repeat Possible Description
h, help No No -h, --help Generate listing of all valid command-line options and applicable (short) guidance for using them.
account Yes empty string No valid account name Username or mailbox in email format.
token Yes empty string No valid token Access token.
encode No false No true, false Whether to encode XOAuth2 string for use in SASL XOAUTH2.

fetch-token

Command-line arguments

  • Flags marked as required must be set via CLI flag.
  • Flags not marked as required are for settings where a useful default is already defined.
Option Required Default Repeat Possible Description
h, help No No -h, --help Generate listing of all valid command-line options and applicable (short) guidance for using them.
scopes Yes empty string No comma-separated list of scopes Permissions needed by the application. If using the scopes defined by the application registration you must use the RESOURCE/.default format (e.g., https://outlook.office365.com/.default.
client-id Yes empty string No valid application ID associated with registered app Application (client) ID created during app registration.
client-secret Yes empty string No valid application secret associated with registered app Client secret (aka, "app" password).
token-url Yes empty string No valid token URL The OAuth2 provider's token endpoint URL. E.g., https://accounts.google.com/o/oauth2/token for Google. See contrib/list-emails/oauth2/accounts.example.ini for O365 example.
filename No empty string No valid path to file Optional file used to record a retrieved token. If specified the file will be overwritten.
json-output No false No true, false Emit retrieved token in JSON format. Defaults to emitting the access token field from retrieved payload.
max-attempts No 3 No positive whole number Max token retrieval attempts.
logging-level No info No disabled, panic, fatal, error, warn, info, debug, trace Sets log level.
version No false No true, false Whether to display application version and then immediately exit application

read-token

Command-line arguments

  • Flags marked as required must be set via CLI flag.
  • Flags not marked as required are for settings where a useful default is already defined.
Option Required Default Repeat Possible Description
h, help No No -h, --help Generate listing of all valid command-line options and applicable (short) guidance for using them.
filename Yes empty string No valid path to file File o used to record a retrieved token. If specified the file will be overwritten.
logging-level No info No disabled, panic, fatal, error, warn, info, debug, trace Sets log level.
version No false No true, false Whether to display application version and then immediately exit application

Examples

check_imap_mailbox_basic

As a Nagios plugin

When called by Nagios, you don't really benefit from having the application generate log output; Nagios throws away output stderr and returns anything sent to stdout, so output of any kind has to be carefully tailored to just what you want to show up in the actual alert. Because of that, we disable logging output explicitly and rely on the plugin to return information as required via stdout.

$ /usr/lib/nagios/plugins/check_imap_mailbox_basic -folders "Inbox, Junk Email" -server imap.example.com -username "tacotuesdays@example.com" -port 993 -password "coconuts" -log-level disabled
OK: tacotuesdays@example.com: No messages found in folders: Inbox, Junk Email

Login failure

Assuming that an error occurred, we will want to explicitly choose a different log level than the one normally used when the plugin is operating normally. Here we choose -log-level info to get at basic operational details. You may wish to use -log-level debug to get even more feedback.

$ /usr/lib/nagios/plugins/check_imap_mailbox_basic -folders "Inbox, Junk Email" -server imap.example.com -username "tacotuesdays@example.com" -port 993 -password "coconuts" -log-level info -branding
{"level":"error","username":"tacotuesdays@example.com","server":"imap.example.com","port":993,"folders_to_check":"Inbox,Junk Email","error":"LOGIN failed.","caller":"T:/github/check-mail/main.go:152","message":"Login error occurred"}
Login error occurred

Additional details: LOGIN failed.

Notification generated by check_imap_mailbox_basic x.y.z

check_imap_mailbox_oauth2

Aside from accepting a different set of flags and authenticating using OAuth2 Client Credentials flow, the functionality of this plugin is identical to check_imap_mailbox_basic.

$ /usr/lib/nagios/plugins/check_imap_mailbox_basic --shared-mailbox "tacotuesdays@example.com" --folders "Inbox, Junk Email" --server outlook.office365.com --client-id "ZYDPLLBWSK3MVQJSIYHB1OR2JXCY0X2C5UJ2QAR2MAAIT5Q" --client-secret "_djgA8heFo0WSIMom7U39WmGTQFHWkcD8x-A1o-4sro" --token-url "https://login.microsoftonline.com/6029c1d9-aa2f-4227-8f7c-0c23224a0fa9/oauth2/v2.0/token" --scopes "https://outlook.office365.com/.default" --port 993 --log-level disabled
OK: tacotuesdays@example.com: No messages found in folders: Inbox, Junk Email

See the Office 365 (O365) permissions section for details specific to using this plugin with O365 mailboxes.

list-emails

No options

In this example, the list-emails application is in the current working directory, as is the accounts.ini file. When run, the output and log directories are created (if not already present) and populated with new log and report files.

$ ./list-emails
Checking account: email1
Checking account: email2
OK: Successfully generated reports for accounts: email1, email2

Alternate locations for config file, log and report directories

For this example, I intentionally placed each item on a separate volume. I then reference each item via separate flags.

./list-emails --config-file /mnt/t/accounts.ini --report-file-dir /mnt/g/reports --log-file-dir /mnt/d/log
Checking account: email1
Checking account: email2
OK: Successfully generated reports for accounts: email1, email2

lsimap

Quick listings for outlook.office365.com and imap.gmail.com.

This tool can be useful for determining at a glance what authentication mechanisms are supported by an IMAP server.

$ ./lsimap --server outlook.office365.com
6:10AM INF cmd\lsimap\main.go:61 > Connection established to server
6:10AM INF cmd\lsimap\main.go:70 > Gathering pre-login capabilities
6:10AM INF cmd\lsimap\main.go:87 > Capability: AUTH=PLAIN
6:10AM INF cmd\lsimap\main.go:87 > Capability: AUTH=XOAUTH2
6:10AM INF cmd\lsimap\main.go:87 > Capability: CHILDREN
6:10AM INF cmd\lsimap\main.go:87 > Capability: ID
6:10AM INF cmd\lsimap\main.go:87 > Capability: IDLE
6:10AM INF cmd\lsimap\main.go:87 > Capability: IMAP4
6:10AM INF cmd\lsimap\main.go:87 > Capability: IMAP4rev1
6:10AM INF cmd\lsimap\main.go:87 > Capability: LITERAL+
6:10AM INF cmd\lsimap\main.go:87 > Capability: MOVE
6:10AM INF cmd\lsimap\main.go:87 > Capability: NAMESPACE
6:10AM INF cmd\lsimap\main.go:87 > Capability: SASL-IR
6:10AM INF cmd\lsimap\main.go:87 > Capability: UIDPLUS
6:10AM INF cmd\lsimap\main.go:87 > Capability: UNSELECT
6:10AM INF cmd\lsimap\main.go:95 > Connection to server closed

$ ./lsimap --server imap.gmail.com
6:10AM INF cmd\lsimap\main.go:61 > Connection established to server
6:10AM INF cmd\lsimap\main.go:70 > Gathering pre-login capabilities
6:10AM INF cmd\lsimap\main.go:87 > Capability: AUTH=OAUTHBEARER
6:10AM INF cmd\lsimap\main.go:87 > Capability: AUTH=PLAIN
6:10AM INF cmd\lsimap\main.go:87 > Capability: AUTH=PLAIN-CLIENTTOKEN
6:10AM INF cmd\lsimap\main.go:87 > Capability: AUTH=XOAUTH
6:10AM INF cmd\lsimap\main.go:87 > Capability: AUTH=XOAUTH2
6:10AM INF cmd\lsimap\main.go:87 > Capability: CHILDREN
6:10AM INF cmd\lsimap\main.go:87 > Capability: ID
6:10AM INF cmd\lsimap\main.go:87 > Capability: IDLE
6:10AM INF cmd\lsimap\main.go:87 > Capability: IMAP4rev1
6:10AM INF cmd\lsimap\main.go:87 > Capability: NAMESPACE
6:10AM INF cmd\lsimap\main.go:87 > Capability: QUOTA
6:10AM INF cmd\lsimap\main.go:87 > Capability: SASL-IR
6:10AM INF cmd\lsimap\main.go:87 > Capability: UNSELECT
6:10AM INF cmd\lsimap\main.go:87 > Capability: X-GM-EXT-1
6:10AM INF cmd\lsimap\main.go:87 > Capability: XLIST
6:10AM INF cmd\lsimap\main.go:87 > Capability: XYZZY
6:10AM INF cmd\lsimap\main.go:95 > Connection to server closed

xoauth2

export user="me@there.com"
export token="adfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfasdfa"
$ ./xoauth2 --token "$token" --username "$user" > go-output.txt
$ cat go-output.txt
dXNlcj1tZUB0aGVyZS5jb20BYXV0aD1CZWFyZXIgYWRmYXNkZmFzZGZhc2RmYXNkZmFzZGZhc2RmYXNkZmFzZGZhc2RmYXNkZmFzZGZhc2RmYXNkZmFzZGZhc2RmYXNkZmFzZGZhc2RmYXNkZmFzZGZhc2RmYXNkZmFzZGZhc2RmYQEB

fetch-token

$ ./fetch-token \
  --client-id 'ZYDPLLBWSK3MVQJSIYHB1OR2JXCY0X2C5UJ2QAR2MAAIT5Q' \
  --client-secret '_djgA8heFo0WSIMom7U39WmGTQFHWkcD8x-A1o-4sro' \
  --scopes 'https://outlook.office365.com/.default' \
  --token-url 'https://login.microsoftonline.com/6029c1d9-aa2f-4227-8f7c-0c23224a0fa9/oauth2/v2.0/token' \
  --log-level debug \
  --filename "token.txt"
1:15PM DBG cmd\fetch-token\main.go:62 > Application configuration initialized filename=token.txt
1:15PM DBG cmd\fetch-token\main.go:64 > Fetching Client Credentials token filename=token.txt
1:15PM DBG cmd\fetch-token\main.go:77 > Token retrieved filename=token.txt
1:15PM DBG cmd\fetch-token\main.go:114 > Successfully wrote data to file filename=token.txt

This resulted in a plaintext token being written to token.txt for later retrieval by the read-token utility, or even cat or similar shell scripting approach.

If saving the token in JSON format via the --json-output flag (e.g., if you want to also retain the token metadata), the read-token utility is provided to read back just the access token portion of the saved value.

read-token

$ ./read-token --filename "token.txt" --log-level debug
1:15PM DBG cmd\read-token\main.go:54 > Application configuration initialized filename=token.txt
1:15PM DBG cmd\read-token\main.go:56 > Fetching Client Credentials token from file filename=token.txt
1:15PM DBG cmd\read-token\main.go:62 > Successfully read contents of file filename=token.txt
1:15PM DBG cmd\read-token\main.go:90 > File contents do not appear to be JSON filename=token.txt
1:15PM DBG cmd\read-token\main.go:91 > Attempting to parse file contents as plaintext access token filename=token.txt
PLACEHOLDER1:15PM DBG cmd\read-token\main.go:102 > Emitted retrieved token bytes_written=1508 filename=token.txt

The PLACEHOLDER value above indicates the access token emitted on stdout. It is interleaved with the log message emitted on stderr which immediately follows the token.

If redirecting stderr to a file, disabling log messages entirely (or if no errors are encountered), log messages will not intermix with the emitted token on stdout.

OAuth 2 Notes

Misc bits of info that don't fit well anywhere else. Potentially slated for inclusion in a project wiki at some point.

Retrieving a token via curl

For reference, here is a curl command used to fetch a token:

curl https://login.microsoftonline.com/TENAT_ID_HERE/oauth2/v2.0/token -X POST -H "Content-type: application/x-www-form-urlencoded" -d "client_id=CLIENT_ID_HERE&scope=https%3A%2F%2Foutlook.office365.com%2F.default&grant_type=client_credentials&username=me@example.com&client_secret=CLIENT_SECRET_HERE"

and the "pretty printed" JSON response:

{
    "token_type": "Bearer",
    "expires_in": 3599,
    "ext_expires_in": 3599,
    "access_token": "TOKEN_HERE"
}

A refresh token is not provided for a Client Credentials grant flow.

Per RFC6749, Section 4.4.3:

If the access token request is valid and authorized, the authorization server issues an access token as described in Section 5.1. A refresh token SHOULD NOT be included.

SASL XOAUTH2 Token encoding

The SASL XOAUTH2 token format is described as:

base64("user=" + userName + "^Aauth=Bearer " + accessToken + "^A^A")

What gave me a lot of grief was applying this encoding literally and then passing the result to other libraries for further processing.

Borrowing from Google's dev docs, this is the result before base64 encoding:

user=someuser@example.com^Aauth=Bearer ya29.vF9dft4qmTc2Nvb3RlckBhdHRhdmlzdGEuY29tCg^A^A

I was then base64-encoding that value which produced something like this:

dXNlcj1zb21ldXNlckBleGFtcGxlLmNvbQFhdXRoPUJlYXJlciB5YTI5LnZGOWRmdDRxbVRjMk52YjNSbGNrQmhkSFJoZG1semRHRXVZMjl0Q2cBAQ==

I'd then pass it down to underlying libraries to use as part of the authentication process as described in this O365 IMAP Protocol Exchange doc:

AUTHENTICATE XOAUTH2 <base64 string in XOAUTH2 format>

which was supposed to end up looking something like this:

AUTHENTICATE XOAUTH2 dXNlcj1zb21ldXNlckBleGFtcGxlLmNvbQFhdXRoPUJlYXJlciB5YTI5LnZGOWRmdDRxbVRjMk52YjNSbGNrQmhkSFJoZG1semRHRXVZMjl0Q2cBAQ==

but it didn't and I spent a long while puzzling this out. What I didn't understand is that base64-encoding is applied by the underlying IMAP libraries.

For example, the Ruby IMAP NET::IMAP::authenticate method handles base64 encoding the "data" before it is used with the AUTHENTICATE command.

From /usr/lib/ruby/2.7.0/net/imap.rb (Ubuntu 20.04):

    # Sends an AUTHENTICATE command to authenticate the client.
    # The +auth_type+ parameter is a string that represents
    # the authentication mechanism to be used. Currently Net::IMAP
    # supports the authentication mechanisms:
    #
    #   LOGIN:: login using cleartext user and password.
    #   CRAM-MD5:: login with cleartext user and encrypted password
    #              (see [RFC-2195] for a full description).  This
    #              mechanism requires that the server have the user's
    #              password stored in clear-text password.
    #
    # For both of these mechanisms, there should be two +args+: username
    # and (cleartext) password.  A server may not support one or the other
    # of these mechanisms; check #capability() for a capability of
    # the form "AUTH=LOGIN" or "AUTH=CRAM-MD5".
    #
    # Authentication is done using the appropriate authenticator object:
    # see @@authenticators for more information on plugging in your own
    # authenticator.
    #
    # For example:
    #
    #    imap.authenticate('LOGIN', user, password)
    #
    # A Net::IMAP::NoResponseError is raised if authentication fails.
    def authenticate(auth_type, *args)
      auth_type = auth_type.upcase
      unless @@authenticators.has_key?(auth_type)
        raise ArgumentError,
          format('unknown auth type - "%s"', auth_type)
      end
      authenticator = @@authenticators[auth_type].new(*args)
      send_command("AUTHENTICATE", auth_type) do |resp|
        if resp.instance_of?(ContinuationRequest)
          data = authenticator.process(resp.data.text.unpack("m")[0])
          s = [data].pack("m0")
          send_string_data(s)
          put_string(CRLF)
        end
      end
    end

This is where base64-encoding is performed:

s = [data].pack("m0")

and this is where the base64 encoding is performed in the emersion/go-imap library that this project uses:

func (cmd *Authenticate) Command() *imap.Command {
  args := []interface{}{imap.RawString(cmd.Mechanism)}
  if cmd.InitialResponse != nil {
    var encodedResponse string
    if len(cmd.InitialResponse) == 0 {
      // Empty initial response should be encoded as "=", not empty
      // string.
      encodedResponse = "="
    } else {
      encodedResponse = base64.StdEncoding.EncodeToString(cmd.InitialResponse)
    }

    args = append(args, imap.RawString(encodedResponse))
  }
  return &imap.Command{
    Name:      "AUTHENTICATE",
    Arguments: args,
  }
}

Takeaway: Don't literally base64 encode the username and access token as illustrated in the documentation, just make sure that by the time all processing of those values is complete that the final result is base64-encoded. In the case of the Ruby and Go code shown above this takes place before the final AUTHENTICATE IMAP command is issued. We just need to make sure we perform the initial OAuth2 XOAUTH2 encoding, skip base64 encoding the result and let the underlying library handle the rest.

In the case of Ruby this initial encoding can be performed by the Mailbutler/mail_xoauth2 gem and in the case of Go this can be performed by the sqs/go-xoauth2 package (if performing just the encoding) or a local copy of the emersion/go-sasl xoauth2Client type (since the upstream project has removed official support for it).

Troubleshooting

The Get-IMAPAccessToken.ps1 PowerShell script can be used to test OAuth2 Client Credentials flow authentication. From the script's description help text:

The function helps admins to test their IMAP OAuth Azure Application, with Interactive user login und providing or the lately released client credential flow using the right formatting for the XOAuth2 login string. After successful logon, a simple IMAP folder listing is done, in addition it also allows to test shared mailbox access for users if full access has been provided.

Using Windows Powershell allows MSAL to cache the access+refresh token on disk for further executions for interactive login scenario. It´s a simple proof of concept with no further error management.

This script was incredibly useful, providing a known working tool to contrast development/troubleshooting efforts against.

License

From the LICENSE file:

MIT License

Copyright (c) 2020 Adam Chalkley

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

References

Related projects

Dependencies

OAuth2 Research

General

Redmine

OAuth 2 Client Credentials grant flow

OAuth 2.0 Resource Owner Password Credentials (ROPC) grant

Go-specific references

RFCs

Other projects