SROS2 demo using Keymint now not working with Fast RTPS

[ruffsl]

As of dafae39 , it seems that what works find for rmw_connext_cpp is not working for rmw_fastrtps_cpp. When attempting to use the keystore generated by the keymint script, rmw_connext_cpp will generate the following error:

[SECURITY Error] Error validating the local participant identity. () -> Function init
[RTPS_PARTICIPANT Error] Cannot create participant due to security initialization error -> Function createParticipant
[PARTICIPANT Error] Problem creating RTPSParticipant -> Function createParticipant
terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
  what():  failed to initialize rcl node: create_node() could not create participant, at /tmp/binarydeb/ros-bouncy-rmw-fastrtps-cpp-0.5.1/src/rmw_node.cpp:91, at /tmp/binarydeb/ros-bouncy-rcl-0.5.0/src/rcl/node.c:336

To reproduce, simply follow along with the README.md, but take note to enable the approrate environment verables in the docker-compose.yml to enable security and the appropriate RMW_IMPLEMENTATION. You can comment out network_mode: "host" to avoid broadcasting in your local networks.

Depending on which script you use, keystore_setup.bash will do fine, yet keymint_setup.bash will lead to the error above, presumably due to the tighter permission it compiles as opposed to keystore_setup.bash use of wildcards.

ping @mikaelarguedas

[ruffsl]

Ok, think I found three issues when comparing to keystore_setup script.

1. the default governance config was old and missing the newly added enable_liveliness_protection element. Connext didn't seem to mind it missing, but fastrtps lucky does.
2. Small issue with how kemint's keymake was signing the xml files, as fastrtps seem to require that Content-Type: text/plain\n\n prefix the xml string when embedded in the singed .ps7 file.
3. Some issue with how keymint generates the CA cert. Connext is fine with it, but fastrtps only takes the the openssl generate cert style.

[mikaelarguedas]

Yeah FastRTPS implement the xml checking more strictly (for the better!) it also lead to ros2/sros2#44, though a bit too strictly for some tags (eProsima/Fast-DDS#245).
3.
I'm not sure what the spec expects in that matter. I remember having issues because before and it was fixed once eProsima/Fast-DDS@1e84b97 added expectation for the comma separated version (matching the examples from the spec).

[ruffsl]

Ok, I solved the 3rd issue as well. Turns out fastrtps also checks to make sure the singing CA has the correct x509 extensions configured for a CA, e.g.:

x509_extensions: # https://access.redhat.com/documentation/en-US/Red_Hat_Certificate_System/8.0/html/Admin_Guide/Standard_X.509_v3_Certificate_Extensions.html
    BasicConstraints: # This extension is used during the certificate chain verification process to identify CA certificates and to apply certificate chain path length constraints
      critical: True # PKIX Part 1 requires that this extension be marked critical. This extension is evaluated regardless of its criticality
      value:
        ca: True
        path_length: 1 # max length of CA chain tat could extend from current cert

Perhaps connext should also be doing this 🤔 !

Although it doesn't seem fastrtps requires it to be critical to be set to True, which really messes with my head given TLS requires this; additionally, why bother erroring out due to the BasicConstraints extension being absent if you not going to require it also be marked as critical=True?

I guess this enlarges the certificate payloads over the wire with one more x509 extension, but perhaps it warranted. I just don't see this requirement stated in the speck, or it could be implicit from 9.3 Builtin Authentication: DDS:Auth:PKI-DH, in bold:

An X.509 Certificate that chains up to the Shared Identity CA, that binds the Public Key of the DomainParticipant to the Distinguished Name (subject name) for the DomainParticipant.

Perhaps we could ping some connext and fastrtps folks?

[ruffsl]

Ok, so fastrtps is working now for the most part using these latest generated files:
https://github.com/ruffsl/keymint_ws/tree/fb99750047fa7a4549f48a5f021571c58d94b4e5

However, when attempting to ros launch talker_listener or add_two_ints, I get a very verbose output about decoding submessage errors:

$ ros2 launch demo_nodes_cpp talker_listener.launch.py 
[INFO] [launch]: process[talker-1]: started with pid [1445]
[INFO] [launch]: process[listener-2]: started with pid [1446]
[SECURITY Error] Cannot decode writer RTPS submessage () -> Function decode_rtps_submessage
[SECURITY Error] Cannot decode writer RTPS submessage () -> Function decode_rtps_submessage
[SECURITY Error] Cannot decode reader RTPS submessage () -> Function decode_rtps_submessage
...
[SECURITY Error] Cannot decode writer RTPS submessage () -> Function decode_rtps_submessage
[SECURITY Error] Cannot decode reader RTPS submessage () -> Function decode_rtps_submessage
[INFO] [talker]: Publishing: 'Hello World: 1'
[INFO] [listener]: I heard: [Hello World: 1]
[INFO] [talker]: Publishing: 'Hello World: 2'
[INFO] [listener]: I heard: [Hello World: 2]
[INFO] [talker]: Publishing: 'Hello World: 3'
[INFO] [listener]: I heard: [Hello World: 3]
...

[ruffsl]

@mikaelarguedas , any idea about the FastRTP errors:

[SECURITY Error] Cannot decode writer RTPS submessage () -> Function decode_rtps_submessage
[SECURITY Error] Cannot decode reader RTPS submessage () -> Function decode_rtps_submessage

Because of the level of real DDS traffic on the turtlebot3, those error messages (spamming to stdout, not stderr) can push the raspberry pie over the edge in memory usage when leaving a sros2 node running for too long, even when piping it all to /dev/null as a temp workaround. Not just keyment too, as using sros secure cli to generate the security artifacts will render the same results when using rmw_fastrtps_cpp.

[mikaelarguedas]

I faced that issue recently too and I agree that they are flooding the console when you start having a big system. I didn't get a chance to track down why these errors are printed.
Maybe @richiware has a clue of why this is happening?

[ruffsl]

Looking in Fast-RTPS/src/cpp/rtps/security/SecurityManager.cpp

https://github.com/eProsima/Fast-RTPS/blob/ea57f15aaf689144cb8e75aa8d23e7b6ed832b3a/src/cpp/rtps/security/SecurityManager.cpp#L2941-L2963

I see only a few places where AESGCMGMAC_Transform::decode_datawriter_submessage returns false without issuing additional logError messages.

https://github.com/eProsima/Fast-RTPS/blob/ea57f15aaf689144cb8e75aa8d23e7b6ed832b3a/src/cpp/security/cryptography/AESGCMGMAC_Transform.cpp#L885

I suspect deserialize_SecureDataTag is that case

https://github.com/eProsima/Fast-RTPS/blob/ea57f15aaf689144cb8e75aa8d23e7b6ed832b3a/src/cpp/security/cryptography/AESGCMGMAC_Transform.cpp#L1022-L1028

But how can I be sure I'm have logWarning enabled at runtime? It looks like they should be enabled with release, but I'm not sure I have it configured:

https://github.com/eProsima/Fast-RTPS/blob/ea57f15aaf689144cb8e75aa8d23e7b6ed832b3a/include/fastrtps/log/Log.h#L35

[mikaelarguedas]

It can be configured using the rmw_set_log_severity function

[richiware]

Sorry for the delay. I see the mention today. I think those message are printed because reader and writer haven't exchange cryptographic keys yet, and reader is receiving already data from the writer and it cannot decrypt it.

Maybe those messages should be warnings instead of errors.