Checkin and checkin modules

This page describes functionality in Sal v4 and above

Client Checkin and Checkin Modules

Machines check in to Sal by posting JSON to the checkin endpoint. The format of this data structure is described below.

Checkin Workflow

The checkin process roughly follows this workflow:

Some period of time elapses and a checkin is initiated by some means. On Mac, this is normally a Munki run.
Client should sync any plugin scripts missing or out of date on the client. On Mac, the Sal preflight handles this.
Client builds a submission
1. Client adds machine info to the submission. The only truly required data is the serial number and the machine group key. On Mac, this, and subsequent steps, are handled by the sal-postflight / sal-submit scripts.
2. Client executes any number of helper modules to add additional submission information. On Mac, this includes the builtin machine, Apple Software Update, Munki, and Sal modules, and then any optional modules desired (for example, Salt, Puppet, Chef).
3. Client runs the plugin scripts and adds their output to the submission. On Mac, this is handled by the sal-submit script.
4. Client submits its checkin.
On Mac, the client may additionally send some information, depending on whether it's new or not. Other platforms may do this with supporting server code if desired.:
1. Software inventory
2. Munki Catalogs
3. Configuration Profiles
Upon a successful submission, client cleans up its stored submission data. This is primarily a factor for UpdateHistoryItem processing. Multiple entries may get created as an item changes status and while the client may not have access to Sal to check in. Once it submits this accumulated history, it can be removed locally.

Checkin format

The checkin format is a dictionary using management source names for keys.

Each management source is identified by this name on the server. The value for each management source is another dict. Within that dict, there are three standard items that will be processed:

facts: Processed into Facts, for use in searches and plugins, and for viewing on the machine detail page.
messages: Processed into the model of the same name, messages are for display on the machine detail page's "Warnings" and "Errors" modals, and for use in plugins and reports. A message is anything that a management source may need to relay to the Sal server (warnings, errors, debug info, etc).
managed_items: Some type of item that is managed by that system (Munki ManagedInstalls, Salt states, etc). These are displayed primarily as a table on the machine detail page. Historical managed items will be retained up to your Sal HISTORICAL_DAYS setting.

Additionally, management source submission data can have additional processors. For example, the Machine management source submission includes things like serial number, or free disk space, which get set on the Machine object directly by the processor function (using the extra_data key / dict in the submission). See the source for details on how this works if you think you need it. At this time, these functions must be included in the Sal code rather than as pluggable modules.

The plugin_results key actually has a list as its value, and deviates from the other items in the checkin results, but is handled as a special case by the checkin code.

Thus, the basic format is like this (... indicates potentially more items):

{
	'[ManagementSourceName]': {
		'facts': {
			'fact name': 'fact data',
			...
		},
		'messages': [
			{
                            'text': 'The text of the message',
			    'message_type': 'The type of the message (See docs about types)',
                            'date': '2019-02-01T13:00:00Z',  # TZ-aware datetime as ISO 8601 str
                        },
			...
		],
		'managed_items': {
			'[item name]': {
				'date_managed': '2019-02-01T13:00:00Z',  # TZ-aware datetime as ISO 8610 str
				'status': 'str',  # See status choices
				'data': {
					'key': 'value',  # Arbitrary key value pairs of additional information.
					...
				}
			...
		},
                'extra_data': {  # whatever data the source needs AND is processed by the server
                },
	},
        ...
}

Example submission

{
	'Machine': {
		'facts': {
			'checkin_module_version': str
		},
                'extra_data': {
			'serial': 'str',
			'hostname': 'str',
			'console_user': 'str',
			'os_family': 'str',
			'operating_system': 'str',
			'hd_space': int,
			'hd_total': int,
			'hd_percent': float,
			'machine_model': 'str',
			'machine_model_friendly': 'str',
			'cpu_type': 'str',
			'cpu_speed': 'str',
			'memory': 'str',
			'memory_kb': int
		}
	},
	'Sal': {
		'facts': {
			'checkin_module_version': str
		},
		'extra_data': {
			'key': 'str',
			'sal_version': 'str'
		}
	},
	'Munki': {
		'extra_data': {
			'munki_version': 'str',
			'manifest': 'str',
		},
		'messages': [
			{
				"message": 'str',
				"message_type": 'str',  # (See docs on types)
			...
		],
		'facts': {  # Munki conditions, machine info
			'fact name': 'fact data',
			'checkin_module_version': 'str'
			...
		}
		'managed_items': {  # ManagedInstalls, ManagedUninstalls
			'[item name]': {
				'date_managed': '2019-02-01T13:00:00Z',  # TZ-aware datetime as ISO 8601 str
				'status': 'str',  # See status choices
				'data': {
					'key': 'value',  # Arbitrary key value pairs of additional information.
					'type': 'str'  # Munki ManagedItems must include the type of item: ManagedInstall, ManagedUninstall, OptionalInstall, etc.
					...
				}
			...
		}
	},
	'Apple Software Update': {
		'facts': {
			'checkin_module_version': 'str',
			'catalog': 'str',
			'last_check': '2019-02-01T13:00:00Z',  # TZ-aware datetime as ISO 8601 str
		},
		'managed_items': {  # Apple SUS updates that have been installed
			'[item name]': {
				'date_managed': '2019-02-01T13:00:00Z',  # TZ-aware datetime as ISO 8601 str
				'status': 'str',  # See status choices
				'data': {
					'key': 'value',  # Arbitrary key value pairs of additional information.
					...
				}
			...
		}
	},
	'plugin_results': [
		{
			'plugin': 'str',
			'historical': bool,
			'data': {
				'key': 'str value',
				...
			}
		}
		...
	]
}

Machine

This section expects data that will be directly assigned to the corresponding attributes of the Machine model. This is handled by the core server checkin code and the sal-scripts.

Sal

The Sal section likewise collects data about Sal and is handled by the sal-scripts.

Munki

The Munki section collects data from a variety of Munki logs and is handled by the munki checkin module. Included are things like the manifest name, error and warning messages, and ManagedItem records for Munki ManagedInstalls, ManagedUninstalls, and OptionalInstalls.

It also adds items to be added to UpdateHistory/UpdateHistoryItem models for Munki ManagedInstalls, ManagedUninstalls, and AppleUpdates (pending only).

Finally, it adds in facts about the machine from the Munki conditions list.

Apple Software Update

The Apple Software Update section is handled by the corresponding module, and includes information about the last softwareupdate check, the catalog, and installed Apple updates as ManagedItems.

Plugin Results

The plugin results section is where plugin scripts should append their results. These results are then added to Sal PluginScriptSubmission and PluginScriptRow models. The sal-scripts include a utility function for safely updating this list and should be used.

Custom Management Sources

All other keys present in the submission will be considered as a source of some kind of management, and processed into the ManagementSource, ManagedItem, ManagedItemHistory, Message, and Fact models.

The ManagementSource name is determined from the key.

Facts

Facts (submission key facts) should be submitted as a dictionary of key/value pairs corresponding to fact name, fact value.

Managed Items

Managed items (submission key managed_items) mean different things on different systems. Each checkin module can interpret these things as they see fit as long as the submission follows the structure described here.

The 'managed_items' section is a dictionary, where each item should have a unique name, given as the key. The value of each item is another dict.

Key	Value type	Description	Default	Optional
date_managed	str	Time item was managed (see below)	`datetime.datetime.now()`	Yes
status	str	The status of the item; must be one of the accepted statuses (see below)	'UNKNOWN'	Yes
data	dict	Any other data you want to include.	None	Yes

Dates in Managed Items

Dates should be submitted as an ISO 8610 format datetime string, including the timezone offset (json has no date type). Technically, any format handled by the dateutil.parser.parse() function will work, but recommended is the typical serialization style from Apple plists of YYYY-MM-DDTHH:MM:SSZ (e.g. '2019-02-01T13:00:42Z'). Keep in mind that python datetimes are näive unless you localize them! Many logs and things you might want to report on use the system's local time and don't bother to include the TZ.

Statuses in Managed Items

The status must be chosen from the following list:

Value	Comment
PRESENT	The item is present, installed, available.
ABSENT	The item is not present, has been uninstalled, or is unavailable.
PENDING	The item is pending some other status.
ERROR	The item failed in some way.
UNKNOWN	Item has an unknown status.

Messages

Messages (submission key is messages) should be a list of dicts, each dict having the following items.

Key	Value type	Description	Default	Optional
text	str	Message text	N/A	No
message_type	str	The type of the message; must be one of the accepted statuses (see below)	'OTHER'	Yes
date	str	ISO 8601 datetime with TZ-offset	Server's time at checkin	Yes

Message types for Messages

The message type must be chosen from the following list:

Value	Comment
WARNING	A non-critical message.
DEBUG	Information interesting for debugging purposes. May be silenced or removed from some results.
ERROR	The item failed in some way.
OTHER	Message is some other kind of message.

Developing a Checkin module

To create a custom checkin module for your needs, you must create an executable and put it into the client's Sal scripts checkin_module subdirectory (on Mac: /usr/local/sal/checkin_modules/). All executable modules in this directory will be executed during the checkin script's execution.

The module should add its results to the checkin script's results file. On Mac, this file is /usr/local/sal/checkin_results.plist. On Mac, you should use the Sal scripts utils.add_checkin_results() function to do so, to avoid deleting data already present.

#!/usr/bin/python


import datetime
import sys

sys.path.insert(0, '/usr/local/sal')
import utils


def main():
    results = {}
    results['facts'] = get_facter_facts()
    results['managed_items'] = get_managed_items()
    results['messages'] = get_messages()

    utils.set_checkin_results('My Management Source', results)


def get_facter_facts():
    results = {}

    # The facts results is just a dictionary:
    # key = fact name
    # value = fact value

    return results


def get_managed_items():
    results = {}

    # Each managed item should be added to the results dict by name
    # (there's no name key in the item itself.
    # Each item should have the following three keys:
    # - date_managed (str UTC datetime; in py2 that's
    #       `datetime.datetime.utcnow().isoformat() + 'Z'`) or have fun
    #       localizing stuff from Puppet if it's not already timezone
    #       aware.
    # - status (one of 'ABSENT', 'PRESENT', 'ERROR', "PENDING",
    #       'UNKNOWN') as a str
    # - data: A dict of whatever other data is pertinent for this item.
    #       See the other checkin processors to get an idea of what else
    #       is here (hint: everything).

    return results


def get_messages()
    results = []

    # Messages should be a list of message dicts.

    # Each message dict should have the following k/v pairs:
    # - text': The text of the message.
    # - message_type: str, one of ('WARNING', 'DEBUG', 'ERROR', 'OTHER'

    return results


if __name__ == "__main__":
    main()

If your checkin module has any additional dependencies, you will need to distribute those to the device as well, by whichever means makes sense. For example, many configuration management tools use yaml, which has no python stdlib module. So the Puppet checkin_module must include a yaml processing module for its own use.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly