- Name
- Synopsis
- Description
- Options
- Config File Parameters
- Access Control
- Seedlink Support
- Multi-Protocol Support
- Http Support
- Transfer Logging
- External Packet Ids
- Miniseed Archiving
- Miniseed Scanning
- Author
ringserver [options] [configfile]
ringserver is a TCP-based ring buffer designed for packetized streaming data. The buffer operates on a First-in-first-out basis with newly arriving packets pushing older packets out of the buffer. The ring packets are not format specific and may contain any type of data. All communications are performed via TCP interfaces. Data streams are available to clients using the SeedLink and DataLink protocols and submitted to the server using the DataLink protocol. WebSocket connections are supported for the SeedLink and DataLink protocols and general status is available via HTTP.
The server is configured either with options on the command line, through environment variables, and/or by using a ringserver config file. The order of precedence for configuration options is command line, environment variables, and then config file.
Only the most common options are available on the command line, all options are controllable via environment variables and the config file. Detailed descriptions for each option are included in the example config file that accompanies the source code. Many options are dynamic, meaning that they can be changed while the server is running. In this case the server will recognize that the config file has changed and will re-read it's configuration. This is especially useful for updating access controls, logging verbosity and other logging parameters.
In normal operation packet buffer contents are saved in files when the server is shut down making the server stateful across restarts. By default the packet buffer is managed as a memory-mapped file. The buffer can optionally be maintained completely in system memory, only reading and writing the buffer contents on startup and shutdown (useful in environments where memory-mapping is not possible).
Client access is controlled using IP addresses. A match list requires that client addresses match one of the entries. A reject list requires that client addresses do _not_ match one of the entires. A write list provides permission to submit packets to specific client addresses. A limit list specifies which streams a client is allowed to access. See Access Control for more details.
Transfer logs can optionally be written to track the transmission and reception of data packets to and from the server. This tracking is stream-based and identifies the number of packet bytes of each unique stream transferred to or from each client connection.
The server supports multiple protocols: SeedLink, DataLink, HTTP with WebSocket. The server can listen on multiple network ports, and each port can be configured to support any combination of the protocols. For ports configured for TLS encryption, only a single protocol is supported and must be specified. See Multi-protocol support for more information.
The server also has limited support for simple HTTP requests. When support is enabled, server status, stream lists and other details can be accessed with simple HTTP requests. See HTTP Support for more details.
The slinktool(1) and dalitool(1) programs can be used to query the ringserver for various information via the SeedLink and DataLink interfaces respectively. The dalitool program was developed in parallel with ringserver and the DataLink protocol and is the recommended query tool for ringserver admins.
-V
Print the program version and exit.
-h
Print the program help/usage and exit.
-H
Print an extended help/usage and exit.
-C
Print a ringserver configuration file and exit. The output contains descriptions of all available configuration parameters and the environment variables that can be used to set them.
-v
Be more verbose. This flag can be used multiple times ("-v -v" or "-vv") for more verbosity.
-I ServerID
Server ID reported to the clients. The value may be a quoted string containing spaces. The default value is "Ring Server".
-M maxclientsperIP
Maximum number of concurrently connected clients per IP address. This limit does not apply to addresses with write permission. There is no default maximum.
-m maxclients
Maximum number of concurrently connected clients. The default maximum is 600.
-Rd ringdir
Base directory for the server to store the packet buffer data files. This parameter must be specified via this option, environment variable, or config file; there is no default.
-Rs bytes
Size of the packet buffer in bytes, default size is 1 GiB. The size of the ring, in combination with the ring packet size, determine how much past data is available in the buffer. If the server is configured to memory-map the packet buffer files the maximum size of the ring is limited to how large a file the host system can support. If the server is configured to maintain the ring in system memory the size is limited to system memory.
-Rp packetsize
Maximum ring packet data size (not including packet buffer header), default is 512 bytes.
-NOMM
No memory-mapping, maintain the ring buffer in system memory only. On startup the ring buffer files will be read into memory and on shutdown the memory buffers will be written back to the files. This option might be useful in environments where memory-mapping of files is not possible or is unreliable or slow (e.g. network storage).
-L port
Network port to listen for incoming connections on. By default the server will not listen for connections, setting at least one listen port is needed to communicate with the server. By default, a listening port will accept all supported protocols. Restricting a port to only allow specific protocols can be done using a configuration file or adding flags to the port declaration, see Multi-protocol Support for more information.
-SL port
-DL port
-HL port
These options are shortcuts for configuring a listening port for only SeedLink, DataLink or HTTP protocols respectively.
-T logdir
Transfer log file base directory, by default the server does not write transfer logs. If a directory is specified both transmission and reception logs will be written, these two logs can be toggled individually in the server config file.
-Ti hours
Transfer log writing interval in hours, default interval is 24 hours. The interval always starts at day boundaries, for example if the interval is set to 2 hours then the first interval in a given day will cover hours 0 through 2, the next 2 through 4, etc.
-Tp prefix
Transfer log file prefix, by default no prefix is added to the fixed section of the log file name.
-STDERR
Send all diagnostic output to stderr instead of stdout. This is useful in situations where logging output is captured by another program on stderr and to separate ringserver diagnostics from other output on the console.
-MSWRITE format
A special mode of ringserver is to write all miniSEED data records received via DataLink to a user defined directory and file structure. See miniSEED Archiving for more details.
-MSSCAN directory [suboptions]
A special mode of ringserver is to recursively scan a directory for files containing miniSEED formatted data records and insert them into the buffer. Optional suboptions control scanning behavior, the StateFile suboption is highly recommended. See miniSEED Scanning for more details.
-VOLATILE
Create a volatile (non-stateful) ring buffer, in other words do not read packet buffer contents from ring files on startup or write them on shutdown. This is useful in combination with the -MSWRITE option when no stateful buffer is needed.
All of the command line parameters have config file and environment variable equivalents. Many of the config file parameters are dynamic, if they are changed the server will re-read it's configuration on the fly. See the detailed parameter descriptions in the documented example config file.
Access control is based on IP addresses and configured using the following config file parameters and environment variables:
MatchIP or RS_MATCH_IP RejectIP or RS_REJECT_IP LimitIP or RS_LIMIT_IP WriteIP or RS_WRITE_IP TrustedIP or RS_TRUSTED_IP
By default all clients are allowed to connect. Specific clients can be rejected using the RejectIP config parameter. If any MatchIP config parameters are specified only addresses that match one of the entries, and are not rejected, are allowed to connect.
By default all clients are allowed access to all streams in the buffer, and clients with write permission are allowed to write any streams. Specific clients can be limited to access or write subsets of streams using the LimitIP config parameter. This parameter takes a regular expression that is used to match stream IDs that the client(s) are allowed access to or to write.
By default all clients are allowed to request the server ID, simple status and list of streams. Specific clients can be allowed to access connection information and more detailed status using the TrustedIP config parameter.
If no client addresses are granted write permission via WriteIP or granted trusted status via TrustedIP then the 'localhost' address (local loopback) are granted those permissions.
Access control is host range (network) based, and specified as an address followed by an optional prefix in CIDR notation. For example: "192.168.0.1/24" specifies the range of addresses from 192.168.0.1 to 192.168.0.254. The address may be a hostname, which will be resolved on startup. The prefix is optional and, if omitted, defaults to specifying only the single address.
The legacy SeedLink protocol (v3) only transmits 512-byte miniSEED data records. This server is able to transmit miniSEED records of any length via SeedLink. If you wish to ensure compatibility with legacy clients, only 512-byte miniSEED records should be submitted to the server.
This server supports the wild-carding of network and station codes during SeedLink negotiation using the '?' and '*' characters for single or multiple character matches respectively. Not all SeedLink clients support wild-carded network and station codes.
Network listening ports can respond to all supported protocols (SeedLink, DataLink and HTTP) for non-TLS ports. The first command received by the server is used to determine which protocol is being used by the client, all subsequent communication is expected in this protocol. Listening ports configured for TLS encryption can only support a single protocol that must be specified.
Both IPv4 and IPv6 protocol familes are supported by default (if supported by the system).
The network protocols and families allowed by any given listening port can be set by adding flags to the port specification. See the available flags in the Listen description of the ringserver config file example, or by using the -C command line option to print a config file.
Examples of adding flags to a port specification:
-L "18000 SeedLink HTTP" : CLI, SeedLink and HTTP on port 18000 -SL "18000 IPv4 TLS" : CLI, SeedLink via TLS on port 18000, IPv4 only RS_LISTEN_PORT="8080 HTTP IPv6" : EnvVar, HTTPS on port 8080, IPv6 only ListenPort 16000 DataLink : Config file, DataLink on port 16000
The server will respond to HTTP requests for a few fixed resources. If the WebRoot config parameter is set to a directory, the files under that directory will also be served when requested through the HTTP GET method. Except for the fixed resources, the HTTP server implementation is limited to returning existing files and returning "index.html" files when a directory is requested.
The following fixed resources are supported:
/id - Server identification /id/json - Server identification in JSON /streams - List of available streams with time range /streams/json - List of available streams with time range in JSON /streamids - List of available streams, variable levels /status - Server status, limited access* /status/json - Server status in JSON, limited access* /connections - List of connections, limited access* /connections/json - List of connections in JSON, limited access* /seedlink - Initiate WebSocket connection for Seedlink /datalink - Initiate WebSocket connection for DataLink
Access to the status and connections information is limited to clients that have trusted permission.
The streams, streamids and connections endpoints accept a match parameter that is a regular expression pattern used to limit the returned information. For the streams and streamids endpoints the matching is applied to stream IDs. For the connections endpoint the matching is applied to hostname, client IP address and client ID. For example: http://localhost/streams?match=IU_ANMO.
The streamids endpoint accepts a level parameter that limits the returned information to a unique list of stream identifiers at the specified level. Valid values are 1 through 6. Identifier components should be delimited with underscore characters. To illustrate, if a ringserver contains streams in the pattern of "NET_STA_LOC_CHAN/MSEED" a request for level 2 returns a unique list of "NET_STA" values. For example: http://localhost/streamids?level=2.
After a WebSocket connection has been initiated with either the seedlink or datalink end points, the requested protocol is supported exactly as it would be normally with the addition of WebSocket framing. Each server command should be contained in a single WebSocket frame, independent of other commands.
Custom HTTP headers may be included in HTTP responses using the HTTPHeader config file parameter. This can be used, for example, to enable cross-site HTTP requests via Cross-Origin Resource Sharing (CORS).
The -T command line option or the TransferLogTX or TransferLogRX config file parameters (or equivalent environment variables) turn on logging of data either transmitted or received. The log interval and file name prefix can be changed via the -Ti and -Tp command line options.
Both the transmission (TX) and reception (RX) log files contain entries that following this pattern:
1) A "START CLIENT" line that contains the host name, IP address, protocol, client ID, log time, and connection time.
2) One or more data lines of the following form:
[Stream ID] [bytes] [packets]
3) An "END CLIENT" line including the total bytes or this entry.
Note: the byte counts are the sum of the data payload bytes in each packet and do not include the DataLink or SeedLink protocol headers.
An example "TX" file illustrating a transmission entry:
START CLIENT host.iris.edu [192.168.255.255] (SeedLink|Client) @ 2018-03-30 07:00:05 (connected 2018-03-30 06:59:36) TX FDSN:IU_SNZO_10_B_H_Z/MSEED 2560 5 FDSN:IU_SNZO_00_B_H_Z/MSEED 2048 4 END CLIENT host.iris.edu [192.168.255.255] total TX bytes: 4608
With the DataLink v1.1 protocol a client may submit packets with a specified packet ID to use instead of a generated ID. This is useful to implement multiple servers that share common packet IDs for use with a network load balancer, such that it does not matter to which server a client connects.
These packet IDs are used in the SeedLink and DataLink protocols by clients to track and resume data streams. In SeedLink these are called sequence numbers.
For ringserver, packet IDs, aka sequence numbers, must be between 0 and (UINT64_MAX - 10), or 18446744073709551605. The last 10 values of the uint64 range are reserved for internal use to indicate special conditions. These values are not expected to be encountered in typical data streaming operation.
Furthermore, external IDs submitted with packets are strongly recommended to be unique and monotonically increasing. Such a sequence of IDs support efficient data stream resumption and tracking.
Using either the -MSWRITE command line option or the MSeedWrite config file parameter the server can be configured to write all miniSEED data records received via DataLink to a user defined directory and file structure.
The archive format argument is expanded for each packet processed using the following flags:
n : network code, white space removed s : station code, white space removed l : location code, white space removed c : channel code, white space removed q : record quality indicator (D,R,Q,M), single character Y : year, 4 digits y : year, 2 digits zero padded j : day of year, 3 digits zero padded H : hour, 2 digits zero padded M : minute, 2 digits zero padded S : second, 2 digits zero padded F : fractional seconds, 4 digits zero padded D : current year-day time stamp of the form YYYYDDD L : data record length in bytes r : sample rate (Hz) as a rounded integer R : sample rate (Hz) as a float with 6 digit precision h : host name of client submitting data % : the percent (%) character # : the number (#) character
The flags are prefaced with either the % or # modifier. The % modifier indicates a defining flag while the # indicates a non-defining flag. All received packets with the same set of defining flags will be saved to the same file. Non-defining flags will be expanded using the values in the first packet received for the resulting file name.
Time flags are based on the start time of the given packet.
Files are created with (permission) mode 666 and directories are created with mode 777. An operator of ringserver can control the final permissions of the files by adjusting the umask as desired.
Some preset archive layouts are available:
BUD : %n/%s/%s.%n.%l.%c.%Y.%j (BUD layout) CHAN : %n.%s.%l.%c (channel) QCHAN : %n.%s.%l.%c.%q (quality-channel-day) CDAY : %n.%s.%l.%c.%Y:%j:#H:#M:#S (channel-day) SDAY : %n.%s.%Y:%j (station-day) HSDAY : %h/%n.%s.%Y:%j (host-station-day)
The preset archive layouts are used by prefixing a target directory with the preset identifier followed by an '@' character. For example:
BUD@/data/bud/
would write a BUD like structure in the /data/bud/ directory.
Other example:
/archive/%n/%s/%n.%s.%l.%c.%Y.%j
would be expanded to day length files named something like:
/archive/IU/ANMO/IU.ANMO..BHE.2003.055
Using non-defining flags the format string:
/data/%n.%s.%Y.%j.%H:#M:#S.miniseed
would be expanded to:
/data/IU.ANMO.2003.044.14:17:54.miniseed
resulting in hour length files because the minute and second are specified with the non-defining modifier. The minute and second fields are from the first packet in the file.
Using either the -MSSCAN command line option or the MSeedScan config file parameter (or equivalent environment variable) the server can be configured to recursively scan a directory for files containing miniSEED data records and insert them into the buffer. Intended for real-time data re-distribution, files are continuously scanned, newly added records are inserted into the buffer.
Sub-options can be used to control the scanning process. The sub-options are specified on the same line as the scan directory as key-value pairs separated by an equals '=' character and may not contain spaces (because they are separated by spaces). Do not use quotes for the values. The available sub-options are:
StateFile : File to save scanning state through restarts Match : Regular expression to match file names Reject : Regular expression to reject file names InitCurrentState : Initialize scanning to current state MaxRecurse : Maximum recursion depth (default is no limit)
Except for special cases the StateFile option should always be specified, otherwise a restart of the server could re-read data records that it has already read.
If the InitCurrentState option is set to 'y' the scanning will only read new data, effectively skipping all the data discovered during the first scan, under the following conditions:
1) No StateFile has been specified 2) StateFile has been specified but does not exist
The InitCurrentState option is useful to avoid reading all existing data when starting a server scanning an existing large dataset. It is also useful to reset the dataflow to current data after a lengthy downtime, simply remove the statefile(s) before starting the server.
To scan a data directory and save the scanning state to a StateFile configure the server with either a config file option or command line, respectively:
MSeedScan /data/miniseed/ StateFile=/opt/ringserver/scan.state
-MSScan "/data/miniseed/ StateFile=/opt/ringserver/scan.state"
To limit the scanning to file names matching a certain pattern use the Match option, e.g. files ending in ".mseed":
MSeedScan /data/miniseed/ StateFile=/data/scan.state Match=.*\\.mseed$
Chad Trabant EarthScope Data Services
(man page 2024/11/16)