Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python/its-vehicle: new client #114

Merged
merged 26 commits into from
May 23, 2024

Conversation

ymorin-orange
Copy link
Member

@ymorin-orange ymorin-orange commented Apr 17, 2024

Changes:

  • its-vehicle: new client

How to test:

  1. Pre-requisites:
    • configure a local MQTT broker for local connections, listening both on a TCP port (e.g. 1883) and on a UNIX socket (e.g. /run/mosquitto/mqtt.socket):
      per_listener_settings true
      listener 0 /run/mosquitto/mqtt.socket
      allow_anonymous true
      listener 1883 127.0.0.1
      allow_anonymous true
      
      Be sure the socket is accessible (R/W) by your user (e.g. with the MQTT client, below)
    • start a MQTT client to dump messages from the local and remote MQTT brokers (each in their own terminal):
      $ mosquitto_sub --unix /run/mosquitto/mqtt.socket -t '#' -F '%t: %p'
      $ mosquitto_sub -h HOST -p PORT -u USER -P PASS -t 'PROJECT/+/SUFFIX/#' -F '%t: %p'
    • install a tweaked version of paho-socket
      $ git clone https://github.com/mgetka/paho-socket
      $ cd paho-socket
      $ patch -p1 <<-_EOF_
      diff --git a/requirements.txt b/requirements.txt
      index ebd4492..8579e8b 100644
      --- a/requirements.txt
      +++ b/requirements.txt
      @@ -1 +1 @@
      -paho-mqtt==1.5.1
      +paho-mqtt
      _EOF_
      $ pip3 install paho-mqtt==1.6.1 .
    • start gpsd using fake data; use the sample available on wikipedia, or generate a path e.g. with an online tool
      $ gpsfake -q -P 2948 -u -n -c 0.1 NMEA-sample.log
  2. Build and install its-quadkeys and its-vehicle:
    $ pip3 install python/its-{quadkeys,vehicle}
    Note the installation path (e.g. /home/login/.local/bin).
  3. Create a simple configuration file its-vehicle.cfg (set the proper
    project and suffix):
    [general]
    instance-id = test-me
    type = CAM
    report-freq = 0.5
    topic-pub-prefix = PROJECT/inQueue/SUFFIX/
    topic-sub-prefix = PROJECT/outQueue/SUFFIX/
    depth = 22
    messages = cam cpm denm
    depth-sub-cam = 18
    depth-sub-cpm = 18
    depth-sub-denm = 15
    # aproximately: 30km/h 70km/h 100km/h
    speed-thresholds = 8.33 19.44 27.78
    
    [broker.main]
    socket-path = /run/mosquitto/mqtt.socket
    
    [tracking]
    id-length = 16
    
    [gpsd]
    port = 2948
    persistence = 10.0
  4. Start its-vehicle:
    $ /home/login/.local/bin/its-vehicle -c its-vehicle.cfg -d
  5. Check robustness of connection to gpsd:
    1. stop the fake gpsd (pres Ctrl-C)
    2. wait for up to gpsd.persistence
    3. restart the fake gpsd
  6. Check robustness of MQTT connection:
    1. stop the MQTT broker
    2. wait a bit
    3. restart the MQTT broker (check the MQTT unix socket is still accessible by your user!)
  7. Stop its-vehicle, by pressing Ctrl-C
  8. Update the configuration file to connect to a remote broker (with a running IQM) and mirror to the local one (set host, port, credentials, project, and suffix):
    [general]
    instance-id = test-me
    type = CAM
    report-freq = 1.0
    mirror-self = False
    topic-pub-prefix = PROJECT/inQueue/SUFFIX/
    topic-sub-prefix = PROJECT/outQueue/SUFFIX/
    depth = 22
    messages = cam cpm denm
    depth-sub-cam = 18
    depth-sub-cpm = 18
    depth-sub-denm = 15
    # aproximately: 30km/h 70km/h 100km/h
    speed-thresholds = 8.33 19.44 27.78
    
    [broker.main]
    host = HOST
    port = PORT
    username = USER
    password = PASS
    
    [broker.mirror]
    socket-path = /run/mosquitto/mqtt.socket
    
    [tracking]
    id-length = 16
    
    [gpsd]
    port = 2948
    persistence = 10.0
    
  9. Start its-vehicle:
    $ /home/login/.local/bin/its-vehicle -c its-vehicle.cfg -d
    1. check the messages reported by the local MQTT clients
    2. send a message to the remote broker on a topic compatible with the current subscriptions of its-vehicle:
      $ mosquitto_pub -h HOST -p PORT -u USER -P PASS \
          -t 'PROJECT/outQueue/SUFFIX/x/x/x/x/x' \
          -m '{"source_uuid": "something", "where": "here and there"}'
  10. stop its-vehicle, update the configuration for self mirroring, and restart its-vehicle (below, the fields to update in the existing configration file):
    1. no self-mirroring, remote broker unreachable:
      [general]
      mirror-self = False
      [broker.main]
      port = WRONG_PORT
    2. self-mirroring, remote broker reachable:
      [general]
      mirror-self = True
      [broker.main]
      port = GOOD_PORT
    3. self-mirroring, remote broker unreachable:
      [general]
      mirror-self = True
      [broker.main]
      port = WRONG_PORT
  11. Terminate its-vehicle with the TERM signal
    $ killall -TERM its-vehicle

Expected results:

  1. Pre-requisites are installed:
    • the MQTT broker is running and listens on TCP and UNIX socket;
    • the MQTT client is running;
    • the patched paho-socket, and paho-mqtt, are isntalled;
    • the fake gpsd is running.
  2. its-quadkeys and its-vehicle are installed
  3. Configuration file has been created
  4. its-vehicle is running:
    • the MQTT client to the remote broker reports nothing;
    • the MQTT client to the local broker is reporting CAM messages at 0.5Hz, on inQueue;
    • the CAM messages are valid: check the spec, check the values, esp. generation_delta_time:
      {
        "type": "cam",
        "origin": "self",
        "version": "1.1.3",
        "source_uuid": "test-me",
        "timestamp": 1713258140393,
        "message": {
          "protocol_version": 1,
          "station_id": 7603010,
          "generation_delta_time": 56031,
          "basic_container": {
            "station_type": 5,
            "reference_position": {
              "latitude": 436259333,
              "longitude": 14851333,
              "altitude": 0
            },
            "confidence": {
              "position_confidence_ellipse": {
                "semi_major_confidence": 10,
                "semi_minor_confidence": 50,
                "semi_major_orientation": 1
              },
              "altitude": 1
            }
          },
          "high_frequency_container": {
            "heading": 2289,
            "speed": 998,
            "longitudinal_acceleration": 161,
            "drive_direction": 0,
            "vehicle_length": 40,
            "vehicle_width": 20,
            "confidence": {
              "heading": 2,
              "speed": 3,
              "vehicle_length": 0
            }
          }
        }
      }
    • the MQTT subscriptions change based on the location (as seen in the debug logs):
      2024-04-16 11:10:01,144 gpsd: gpsd GNSS client data available: GNSSReport(timestamp=1713258601.1440184, time=1713251006.229, longitude=1.489266667, latitude=43.626433333, altitude=0.0, speed=9.98, acceleration=None, track=317.2, major=None, minor=None, orient=None, true_heading=None, magnetic_heading=None)
      2024-04-16 11:10:01,161 mqtt: unsubscribing MQTT client /run/mosquitto/mqtt.socket from {'5GCroCo/outQueue/v2x/cpm/+/1/2/0/2/2/2/0/3/0/2/2/2/1/3/1/3/0/#', '5GCroCo/outQueue/v2x/cam/+/1/2/0/2/2/2/0/3/0/2/2/2/1/3/1/2/1/#', '5GCroCo/outQueue/v2x/cpm/+/1/2/0/2/2/2/0/3/0/2/2/2/1/3/1/2/1/#', '5GCroCo/outQueue/v2x/cpm/+/1/2/0/2/2/2/0/3/0/2/2/2/1/3/1/3/1/#', '5GCroCo/outQueue/v2x/cam/+/1/2/0/2/2/2/0/3/0/2/2/2/1/3/1/3/0/#', '5GCroCo/outQueue/v2x/cam/+/1/2/0/2/2/2/0/3/0/2/2/2/1/3/1/3/1/#'}
      2024-04-16 11:10:01,161 mqtt: subscribing MQTT client /run/mosquitto/mqtt.socket to {'5GCroCo/outQueue/v2x/cpm/+/1/2/0/2/2/2/0/3/0/2/2/2/1/1/3/3/3/#', '5GCroCo/outQueue/v2x/cpm/+/1/2/0/2/2/2/0/3/0/2/2/2/1/1/3/2/3/#', '5GCroCo/outQueue/v2x/cam/+/1/2/0/2/2/2/0/3/0/2/2/2/1/1/3/2/3/#', '5GCroCo/outQueue/v2x/cam/+/1/2/0/2/2/2/0/3/0/2/2/2/1/1/3/3/2/#', '5GCroCo/outQueue/v2x/cam/+/1/2/0/2/2/2/0/3/0/2/2/2/1/1/3/3/3/#', '5GCroCo/outQueue/v2x/cpm/+/1/2/0/2/2/2/0/3/0/2/2/2/1/1/3/3/2/#'}
      
      Note: subscriptions change based omn the location, so be sure to use an NMEA sample that "travels" fast enough...
  5. its-vehicle is resilient to a disapearing gpsd:
    1. its-vehicle does not crash, but reports connection issues:
      2024-04-17 10:28:05,818 gpsd: gpsd is talking
      2024-04-17 10:28:05,818 gpsd: connection error: ConnectionResetError('short read')
      2024-04-17 10:28:05,818 gpsd: disconnecting
      2024-04-17 10:28:05,818 gpsd: closing sock_fd
      2024-04-17 10:28:05,818 gpsd: closing sock
      2024-04-17 10:28:05,818 gpsd: disconnected
      2024-04-17 10:28:05,818 gpsd: connecting to 127.0.0.1:2948
      2024-04-17 10:28:05,818 gpsd: starting WATCH mode
      2024-04-17 10:28:05,818 gpsd: connected and listening
      2024-04-17 10:28:05,818 gpsd: gpsd is talking
      2024-04-17 10:28:05,818 gpsd: connection error: ConnectionResetError(104, 'Connection reset by peer')
      2024-04-17 10:28:05,818 gpsd: disconnecting
      2024-04-17 10:28:05,818 gpsd: closing sock_fd
      2024-04-17 10:28:05,818 gpsd: closing sock
      2024-04-17 10:28:05,818 gpsd: disconnected
      2024-04-17 10:28:05,818 gpsd: connecting to 127.0.0.1:2948
      2024-04-17 10:28:05,819 gpsd: connection failed: ConnectionRefusedError(111, 'Connection refused')
      2024-04-17 10:28:05,819 gpsd: disconnecting
      2024-04-17 10:28:05,819 gpsd: closing sock
      2024-04-17 10:28:05,819 gpsd: disconnected
      
      Notice that its-vehicle then retires connections every second-or-so:
      2024-04-17 10:28:06,820 gpsd: connecting to 127.0.0.1:2948
      2024-04-17 10:28:06,820 gpsd: connection failed: ConnectionRefusedError(111, 'Connection refused')
      2024-04-17 10:28:06,820 gpsd: disconnecting
      2024-04-17 10:28:06,820 gpsd: closing sock
      2024-04-17 10:28:06,820 gpsd: disconnected
      2024-04-17 10:28:07,821 gpsd: connecting to 127.0.0.1:2948
      2024-04-17 10:28:07,822 gpsd: connection failed: ConnectionRefusedError(111, 'Connection refused')
      2024-04-17 10:28:07,822 gpsd: disconnecting
      2024-04-17 10:28:07,822 gpsd: closing sock
      2024-04-17 10:28:07,822 gpsd: disconnected
      
    2. the CAM messages are still sent during the gpsd.persistence duration, and are no longer emitted afterwards;
    3. its-vehicle reconnects successfully to gpsd, and starts emitting CAM messages again.
  6. its-vehicle is resilient to a disapearing MQTT broker:
    1. its-vehicle does not crash, and reports being disconnected from the MQTT broker:
      2024-04-17 10:35:39,377 mqtt: disconnected (socket) from /run/mosquitto/mqtt.socket
      2024-04-17 10:35:39,377 mqtt: disconnected from /run/mosquitto/mqtt.socket
      
    2. its-vehicle reports not sending message to disconnecte broker:
      2024-04-17 10:35:40,562 mqtt: not sending MQTT message to not connected client /run/mosquitto/mqtt.socket on ......
      
    3. its-vehicle reconnects to the MQTT broker, re-establishes the list of subscriptions, and starts sending CAM messages again:
      2024-04-17 10:38:15,802 mqtt: connected to MQTT client /run/mosquitto/mqtt.socket
      2024-04-17 10:38:15,802 mqtt: subscribing MQTT client /run/mosquitto/mqtt.socket to {......}
      2024-04-17 10:38:17,474 mqtt: sending MQTT message to client /run/mosquitto/mqtt.socket on ......
      
  7. its-vehicle stops cleanly:
    ^C2024-04-17 11:10:01,606 main: Will stop...
    2024-04-17 11:10:01,607 client: stopping ITS client
    2024-04-17 11:10:01,607 client: stopped ITS client
    2024-04-17 11:10:01,607 mqtt: stopping MQTT client /run/mosquitto/mqtt.socket
    2024-04-17 11:10:01,607 mqtt: disconnected from /run/mosquitto/mqtt.socket
    2024-04-17 11:10:01,607 mqtt: disconnected (socket) from /run/mosquitto/mqtt.socket
    2024-04-17 11:10:01,607 gpsd: stopping gpsd GNSS client
    2024-04-17 11:10:01,607 gpsd: event, maybe we need to stop?
    2024-04-17 11:10:01,607 gpsd: disconnecting
    2024-04-17 11:10:01,607 gpsd: closing sock_fd
    2024-04-17 11:10:01,607 gpsd: closing sock
    2024-04-17 11:10:01,607 gpsd: disconnected
    2024-04-17 11:10:01,607 gpsd: stopped gpsd GNSS client
    
  8. Configuration file has been updated
  9. its-vehicle is running:
    1. the MQTT client to the remote broker is reporting CAM messages at 0.5Hz, on all three queues (inQueue, outQueue, and interQueue); the MQTT client to the local broker is reporting CAM messages at 0.5Hz, on outQueue;
    2. the message appears on both clients
  10. Mirroring works as expected:
    1. messages are still received on the client to the local broker
    2. messages are received on both clients
    3. messages are not received at all

@ymorin-orange ymorin-orange self-assigned this Apr 17, 2024
@ymorin-orange ymorin-orange force-pushed the yem/its-vehicle branch 2 times, most recently from c07acba to d5772ab Compare April 17, 2024 13:00
@Hugues360 Hugues360 removed the request for review from tigroo April 22, 2024 15:14
@ymorin-orange
Copy link
Member Author

@Hugues360 I accidentally left a little debug message, so I removed it and repushed; sorry or the mess...

@tigroo tigroo removed their request for review May 2, 2024 13:33
@tigroo tigroo added enhancement New feature or request Rust Rust code Python Python code and removed Rust Rust code labels May 2, 2024
python/its-vehicle/its_vehicle/client.py Outdated Show resolved Hide resolved
python/its-vehicle/its_vehicle/tracking.py Outdated Show resolved Hide resolved
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
... and crudely instanciate it from main() while waiting for a better
loop management...

All fields in a GNSS report are optional but the timestamp the report
was generated at (although a report with only the timestamp would be
very useless), as gspd explicitly states that all fields are optional.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Not all GNSS devices trigger gpsd to emit GST or ATT messages, but when
they do, there are interesting data in there that we want to make
available for enhanced reporting (esp. the acceleration, the error
ellipse...).

Even when those messages are emitted, there is still no guarantee that
any of the fields be present at all, as gpsd explicitly documents that
all fields are optional, so be prepared for that.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
... which is responsible for:

  - gathering GNSS reports,

  - assembling proper ITS messages (depending on what kind of client it
    is),

  - sending those messages to the MQTT broker(s),

  - subscribing to the area of interest based on the current location,

  - mirroring those messages to the local borker (if any),

  - logging and tracking timestamps for MQTT messages that are sent and
    received.

(most of that to come in followup commits).

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
We currently only properly cleanup upon receiving a Ctrl-C (SIGINT), but
process managers, like systemd et al., usually send SIGTERM to tell a
process to terminate properly.

Catch that signal, in addition to SIGINT, so that we have a chance to
cleanup and drop all our connections cleanly.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
…pth)

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
... unless we're mirroring self from the main broker.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
... except our own, unless instructed so.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
The tracker is responsible for tracking events, like when a message is
sent, received, etc... For now, does nothing except hand out an ID.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Upon creation of a message, associate an ID to that message (for now, we
do nothing with that ID, as the schemas do not define one). Track the
emission and reception of messages. For now, we do nothing to actually
track, but the infra is there.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
@Hugues360
Copy link
Collaborator

Ok, test passed with succes with a relevant gps log (with old version and new version to check that the its-client no longer fails)

@ymorin-orange ymorin-orange requested a review from tigroo May 23, 2024 07:28
@ymorin-orange
Copy link
Member Author

@tigroo I've (re)added you as a reviewer so that you can approve the PR if you
believe the live tests we did yesterday were satisfying for you. Thanks! :-)

Copy link
Collaborator

@tigroo tigroo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on site tests ok, description of the PR done.

@ymorin-orange ymorin-orange merged commit 67d8f75 into Orange-OpenSource:master May 23, 2024
30 checks passed
@ymorin-orange ymorin-orange deleted the yem/its-vehicle branch May 23, 2024 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Python Python code
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants