Merge branch 'main' into ignore_offsets

docs.openc3.com/docs/development/curl.md

@@ -88,3 +88,92 @@ Transfer-Encoding: chunked
 
 {"jsonrpc":"2.0","id":8,"result":29.204100000000007}
 ```
+
+## Suite Runner Example
+
+It can be very useful to run the a suite or script remotely from a continuous testing server. COSMOS' REST API allows for this. To figure out what is required to run a certain task on the web GUI you can open up your browser's developer tools to monitor the network traffic. You will see all the requests and responses required to run a command and you can reformat them yourself to suit your own purposes. Below is an example of running a test script from a Chromium-based browser:
+![Network Traffic in browser developer tools](https://github.com/OpenC3/cosmos/assets/55999897/df642d42-43e0-47f9-9b52-d42746d9746b)
+
+You can see that there are 5 transactions total. To investigate just right-click on the network transaction and click "copy as `curl`" (depends on the browser). Here is an example of the second one:
+```bash
+curl 'http://localhost:2900/script-api/scripts/TARGET/procedures/cmd_tlm_test.rb/lock?scope=DEFAULT' \
+  -X 'POST' \
+  -H 'Accept: application/json' \
+  -H 'Accept-Language: en-US,en;q=0.9' \
+  -H 'Authorization: pass' \
+  -H 'Connection: keep-alive' \
+  -H 'Content-Length: 0' \
+  -H 'Origin: http://ascportal:2900' \
+  -H 'Referer: http://localhost:2900/tools/scriptrunner/?file=TARGET%2Fprocedures%2Fcmd_tlm_test.rb' \
+  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0' \
+  --insecure
+```
+Many of the browser-specific headers are not required. The important thing to notice here is the URL and the request (in this case `POST`). If we inspect all of these we'll find out what each one does:
+1. Set the script contents
+   * this updates any local changes)
+   * Note that this is a different request to `GET` the script contents. This is done on the page load.
+3. Lock the script (so other users can't edit it during execution)
+4. Run script (this takes a JSON with options)
+5. Open Websocket for logs
+6. Request Result (this URL is a little different because the results are saved in redis)
+
+Below is a bash script which does all the above given some options. It requires `curl` for the web requests and `jq` for JSON parsing and formatting. It locks and runs the script, continually checks its status, then requests the result.
+
+```bash
+#!/bin/bash
+set -e
+TARGET=${1:-'TARGET'}
+SCRIPT=${2:-'procedures/cmd_tlm_test.rb'}
+SUITE=${3:-'TestSuite'}
+COSMOS_HOST='http://localhost:2900'
+SCRIPT_API="$COSMOS_HOST/script-api"
+SCRIPT_PATH="scripts/$TARGET/$SCRIPT"
+CURL_ARGS=( 
+	-H 'Accept: application/json' 
+	-H 'Authorization: password' 
+	-H 'Accept-Language: en-US,en;q=0.9' 
+	-H 'Connection: keep-alive' 
+	-H 'Content-Type: application/json' 
+	--insecure 
+	--silent )
+
+# Lock script #
+curl "$SCRIPT_API/$SCRIPT_PATH/lock?scope=DEFAULT" -X "POST" "${CURL_ARGS[@]}"
+
+# Run script #
+RUN_OPTS=$(cat <<-json
+{
+  "environment": [],
+  "suiteRunner": {
+    "method": "start",
+    "suite": "$SUITE",
+    "options": [
+      "continueAfterError"
+    ]
+  }
+}
+json
+)
+RUN_OPTS=$(<<<"$RUN_OPTS" jq -rc .)
+ID=$(curl "$SCRIPT_API/$SCRIPT_PATH/run?scope=DEFAULT" --data-raw "$RUN_OPTS" "${CURL_ARGS[@]}")
+
+echo "Starting Script '$SCRIPT_PATH' at $(date) (may take up to 15 minutes)" > /dev/stderr
+echo "You can monitor it in Script Runner here: $COSMOS_HOST/tools/scriptrunner/$ID" > /dev/stderr
+# Loop while Script ID is still running #
+while true; do
+	SCRIPT_STATUS="$(curl "$SCRIPT_API/running-script?scope=DEFAULT" "${CURL_ARGS[@]}" | jq ".[]|select(.id==$ID)")"
+	if [[ -z $SCRIPT_STATUS ]]; then
+		break;
+	fi
+	sleep 2
+done
+
+# Request results #
+BUCKET_FILE_URI="$(curl "$SCRIPT_API/completed-scripts?scope=DEFAULT" "${CURL_ARGS[@]}" |\
+	jq '[.[]|select(.name | test("'"${SCRIPT_PATH#scripts/}"' "))][0] | .log | @uri' -r)"
+
+URL="$(curl "$COSMOS_HOST/openc3-api/storage/download/$BUCKET_FILE_URI?bucket=OPENC3_LOGS_BUCKET&scope=DEFAULT" "${CURL_ARGS[@]}" |jq .url -r)"
+
+curl "$COSMOS_HOST$URL" "${CURL_ARGS[@]}"
+```
+

docs.openc3.com/docs/guides/raspberrypi.md

@@ -22,21 +22,24 @@ Let's get started!
 
    1. Insert the SD Card into your computer (Note this process will erase all data on the SD card!)
    1. Open the Raspberry Pi Imager App
+   1. Click the "Choose Device" Button
+   1. Pick Your Raspberry Pi Model
    1. Click the "Choose OS" Button
    1. Select "Raspberry Pi OS (other)"
    1. Select "Raspberry Pi OS Lite (64-bit)"
    1. Click the "Choose Storage" Button
    1. Select Your SD Card
-   1. Click the Gear Icon
+   1. Click Edit Settings
    1. If prompted if you would like to prefill the Wifi information, select OK
-   1. Click all the checkboxes, except for "Enable Telemetry"
    1. Set the hostname to: cosmos.local
-   1. You can either use Password auth, or public-key only if your computer is already setup for passwordless SSH
-   1. Set the username and password. The default username is pi, you should also set a password to make the system secure
+   1. Set the username and password. The default username is your username, you should also set a password to make the system secure
    1. Fill in your Wifi info, and set the country appropriately (ie. US)
    1. Set the correct time zone
+   1. Goto the Services Tab and Enable SSH
+   1. You can either use Password auth, or public-key only if your computer is already setup for passwordless SSH
+   1. Goto the Options tab and make sure "Enable Telemetry" is not checked
    1. Click "Save" when everything is filled out
-   1. Click the "Write" button, Yes to Are You Sure, and Wait for it to complete
+   1. Click "Yes" to apply OS Customization Settings, Yes to Are You Sure, and Wait for it to complete
 
 1. Make sure the Raspberry Pi is NOT powered on
 
@@ -48,7 +51,7 @@ Let's get started!
 
    1. Open a terminal window and use ssh to connect to your Pi
 
-      1. On Mac / Linux: ssh pi@cosmos.local
+      1. On Mac / Linux: ssh yourusername@cosmos.local
       1. On Windows, use Putty to connect. You will probably have to install Bonjour for Windows for .local addresses to work as well.
 
 1. From SSH, Enter the following commands
@@ -63,14 +66,12 @@ Let's get started!
    sudo sh get-docker.sh
    sudo usermod -aG docker $USER
    newgrp docker
-   sudo apt-get install libffi-dev libssl-dev python3-dev python3 python3-pip -y
-   sudo pip3 install docker-compose
    git clone https://github.com/OpenC3/cosmos-project.git cosmos
    cd cosmos
-   # Edit compose.yaml and remove 127.0.0.1 from the ports section of the openc3-traefik service
+   # Edit compose.yaml and remove 127.0.0.1: from the ports section of the openc3-traefik service
    ./openc3.sh run
 ```
 
-1. After about 2 minutes, open a web browswer on your computer, and goto: http://cosmos.local:2900
+1. After about 2 minutes, open a web browser on your computer, and goto: http://cosmos.local:2900
 
 1. Congratulations! You now have COSMOS running on a Raspberry Pi!

openc3-cosmos-init/plugins/packages/openc3-tool-common/src/components/widgets/Widget.js

@@ -153,7 +153,7 @@ export default {
         }
       }
       // If they pass null for max_params we don't check for a maximum number
-      if (max_num_params && this.parameters[max_num_params] !== undefined) {
+      if (max_num_params !== null && this.parameters.length > max_num_params) {
         throw new ConfigParserError(
           parser,
           `Too many parameters for ${keyword}.`,

openc3/bin/openc3cli

@@ -66,6 +66,7 @@ def print_usage
   puts "  cli irb                           # Runs irb in the local directory"
   puts "  cli validate /PATH/FILENAME.gem SCOPE variables.txt # Validate a COSMOS plugin gem file"
   puts "  cli load /PATH/FILENAME.gem SCOPE variables.txt     # Loads a COSMOS plugin gem file"
+  puts "  cli list <SCOPE>                  # Lists installed plugins, SCOPE is DEFAULT if not given"
   puts "  cli generate TYPE OPTIONS         # Generate various COSMOS entities"
   puts "    OPTIONS: --ruby or --python to specify the language in the generated code"
   puts "  #{MIGRATE_PARSER}"
@@ -381,6 +382,22 @@ def wait_process_complete(process_name)
   end
 end
 
+# Outputs list of installed plugins
+def list_plugins(scope:)
+  scope ||= 'DEFAULT'
+  check_environment()
+  names = []
+  if $openc3_in_cluster
+    names = OpenC3::PluginModel.names(scope: scope)
+  else
+    require 'openc3/script'
+    names = plugin_list(scope: scope)
+  end
+  names.each do |name|
+    puts name
+  end
+end
+
 # Loads a plugin into the OpenC3 system
 # This code is used from the command line and is the same code that gets called if you
 # edit/upgrade or install a new plugin from the Admin interface
@@ -655,6 +672,9 @@ if not ARGV[0].nil? # argument(s) given
     # See plugins_controller.rb install for usage
     load_plugin(ARGV[1], scope: ARGV[2], plugin_hash_file: ARGV[3], force: ARGV[4] == 'force')
 
+  when 'list'
+    list_plugins(scope: ARGV[1])
+
   when 'unload'
     unload_plugin(ARGV[1], scope: ARGV[2])
 

openc3/data/config/interface_modifiers.yaml

@@ -127,7 +127,7 @@ OPTION:
         The option to set. OpenC3 defines several options on the core provided
         interfaces. The SerialInterface defines FLOW_CONTROL which can be NONE (default) or RTSCTS
         and DATA_BITS which changes the data bits of the serial interface.
-        The TcpipServerInterface defines LISTEN_ADDRESS which is the IP address to accept
+        The TcpipServerInterface and HttpServerInterface define LISTEN_ADDRESS which is the IP address to accept
         connections on (default 0.0.0.0).
       values: .*
     - name: Parameters
@@ -138,6 +138,9 @@ OPTION:
     INTERFACE SERIAL_INT serial_interface.rb COM1 COM1 115200 NONE 1 10.0 nil
       OPTION FLOW_CONTROL RTSCTS
       OPTION DATA_BITS 8
+    ROUTER SERIAL_ROUTER tcpip_server_interface.rb 2950 2950 10.0 nil BURST
+      ROUTE SERIAL_INT
+      OPTION LISTEN_ADDRESS 127.0.0.1
 SECRET:
   summary: Define a secret needed by this interface
   description: Defines a secret for this interface and optionally assigns its value to an option

openc3/lib/openc3/interfaces/http_client_interface.rb

@@ -1,6 +1,6 @@
 # encoding: ascii-8bit
 
-# Copyright 2023 OpenC3, Inc.
+# Copyright 2024 OpenC3, Inc.
 # All Rights Reserved.
 #
 # This program is free software; you can modify and/or redistribute it
@@ -34,10 +34,10 @@ def initialize(hostname, port = 80, protocol = 'http', write_timeout = 5, read_t
       @hostname = hostname
       @port = Integer(port)
       @protocol = protocol
-      if (port == 80 and protocol == 'http') or (port == 443 and protocol == 'https')
-        @url = "#{protocol}://#{hostname}"
+      if (@port == 80 and @protocol == 'http') or (@port == 443 and @protocol == 'https')
+        @url = "#{@protocol}://#{@hostname}"
       else
-        @url = "#{protocol}://#{hostname}:#{port}"
+        @url = "#{@protocol}://#{@hostname}:#{@port}"
       end
       @write_timeout = ConfigParser.handle_nil(write_timeout)
       @write_timeout = Float(@write_timeout) if @write_timeout
@@ -50,6 +50,10 @@ def initialize(hostname, port = 80, protocol = 'http', write_timeout = 5, read_t
       @response_queue = Queue.new
     end
 
+    def connection_string
+      return @url
+    end
+
     # Connects the interface to its target(s)
     def connect
       # Per https://github.com/lostisland/faraday/blob/main/lib/faraday/options/env.rb

openc3/lib/openc3/interfaces/http_server_interface.rb

@@ -1,6 +1,6 @@
 # encoding: ascii-8bit
 
-# Copyright 2023 OpenC3, Inc.
+# Copyright 2024 OpenC3, Inc.
 # All Rights Reserved.
 #
 # This program is free software; you can modify and/or redistribute it
@@ -26,14 +26,30 @@ class HttpServerInterface < Interface
     # @param port [Integer] HTTP port
     def initialize(port = 80)
       super()
+      @listen_address = '0.0.0.0' # Default to ANY
       @port = Integer(port)
       @server = nil
       @request_queue = Queue.new
     end
 
+    # Supported Options
+    # LISTEN_ADDRESS - Ip address of the interface to accept connections on
+    # (see Interface#set_option)
+    def set_option(option_name, option_values)
+      super(option_name, option_values)
+      case option_name.upcase
+      when 'LISTEN_ADDRESS'
+        @listen_address = option_values[0]
+      end
+    end
+
+    def connection_string
+      return "listening on #{@listen_address}:#{@port}"
+    end
+
     # Connects the interface to its target(s)
     def connect
-      @server = WEBrick::HTTPServer.new :Port => @port
+      @server = WEBrick::HTTPServer.new(:BindAddress => @listen_address, :Port => @port)
       @request_queue = Queue.new
 
       # Create a response hook for every command packet
@@ -43,9 +59,9 @@ def connect
           path = nil
           begin
             path = packet.read('HTTP_PATH')
-          rescue => err
+          rescue => e
             # No HTTP_PATH is an error
-            Logger.error("HttpServerInterface Packet #{target_name} #{packet_name} unable to read HTTP_PATH\n#{err.formatted}")
+            Logger.error("HttpServerInterface Packet #{target_name} #{packet_name} unable to read HTTP_PATH\n#{e.formatted}")
           end
           if path
             @server.mount_proc path do |req, res|
@@ -145,7 +161,7 @@ def read_interface
 
     # Writes to the socket
     # @param data [Hash] For the HTTP Interface, data is a hash with the needed request info
-    def write_interface(data, extra = nil)
+    def write_interface(_data, _extra = nil)
       raise "Commands cannot be sent to HttpServerInterface"
     end
 
@@ -176,7 +192,7 @@ def convert_data_to_packet(data, extra = nil)
     #
     # @param packet [Packet] Packet to extract data from
     # @return data
-    def convert_packet_to_data(packet)
+    def convert_packet_to_data(_packet)
       raise "Commands cannot be sent to HttpServerInterface"
     end
   end

openc3/lib/openc3/interfaces/interface.rb

@@ -184,6 +184,12 @@ def initialize
       @scheduler = nil
     end
 
+    # Should be implemented by subclass to return human readable connection string
+    # which will be placed in log messages when connecting and during connection failures
+    def connection_string
+      return @name
+    end
+
     # Connects the interface to its target(s). Must be implemented by a
     # subclass.
     def connect

openc3/lib/openc3/interfaces/linc_interface.rb

@@ -17,13 +17,15 @@
 # All changes Copyright 2022, OpenC3, Inc.
 # All Rights Reserved
 #
-# This file may also be used under the terms of a commercial license 
+# This file may also be used under the terms of a commercial license
 # if purchased from OpenC3, Inc.
 
 require 'openc3/interfaces/tcpip_client_interface'
 require 'uuidtools'
 
 module OpenC3
+  # TODO: Deprecated ... Will remove in next major version of COSMOS (COSMOS 6)
+
   # Interface for connecting to Ball Aerospace LINC Labview targets
   class LincInterface < TcpipClientInterface
     # The maximum number of asynchronous commands we can wait for at a time.
@@ -241,7 +243,7 @@ def wait_for_response(packet, guid)
       end
 
       process_handshake_results(handshake_cmd)
-    rescue Exception => err
+    rescue Exception => e
       # If anything goes wrong after successfully writing the packet to the LINC target
       # ensure that the packet gets updated in the CVT and logged to the packet log writer.
       # OpenC3 normally only does this if write returns successfully
@@ -256,7 +258,7 @@ def wait_for_response(packet, guid)
         packet_log_writer_pair.cmd_log_writer.write(packet)
       end
 
-      raise err
+      raise e
     end
 
     def process_handshake_results(handshake_cmd)

openc3/lib/openc3/interfaces/mqtt_interface.rb

@@ -1,6 +1,6 @@
 # encoding: ascii-8bit
 
-# Copyright 2022 OpenC3, Inc.
+# Copyright 2024 OpenC3, Inc.
 # All Rights Reserved.
 #
 # This program is free software; you can modify and/or redistribute it
@@ -108,8 +108,8 @@ def initialize(hostname, port = 1883, ssl = false)
 
       # Build list of packets by topic
       @read_packets_by_topic = {}
-      System.telemetry.all.each do |target_name, target_packets|
-        target_packets.each do |packet_name, packet|
+      System.telemetry.all.each do |_target_name, target_packets|
+        target_packets.each do |_packet_name, packet|
           topics = packet.meta['TOPIC']
           topics = packet.meta['TOPICS'] unless topics
           if topics
@@ -121,6 +121,10 @@ def initialize(hostname, port = 1883, ssl = false)
       end
     end
 
+    def connection_string
+      return "#{@hostname}:#{@port} (ssl: #{@ssl})"
+    end
+
     # Connects the interface to its target(s)
     def connect
       @write_topics = []