Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement URC Handling. (IDFGH-8812) #180

Closed
diplfranzhoepfinger opened this issue Nov 24, 2022 · 48 comments · Fixed by #620
Closed

Implement URC Handling. (IDFGH-8812) #180

diplfranzhoepfinger opened this issue Nov 24, 2022 · 48 comments · Fixed by #620

Comments

@diplfranzhoepfinger
Copy link
Contributor

diplfranzhoepfinger commented Nov 24, 2022

also discussed in:

#156

#168

#179

@diplfranzhoepfinger
Copy link
Contributor Author

@david-cermak

@github-actions github-actions bot changed the title Implement URC Handling. Implement URC Handling. (IDFGH-8812) Nov 24, 2022
@franz-ms-muc
Copy link
Contributor

franz-ms-muc commented Dec 1, 2022

also in the Modem Console URC might be helpful:
see here:
image

issuing a extra cmd AT does not really look smart.

@david-cermak
Copy link
Collaborator

This could be implemented the same way as #156, i.e. installing a permanent callback which would check for a global set of URC first and only after that for replies to requests. So we could still create a DCE class that's command-able (=could add commands) and at the same time handles the URCs.

I'll prepare a simple demo.

@franz-ms-muc
Copy link
Contributor

franz-ms-muc commented Dec 1, 2022 via email

@franz-ms-muc
Copy link
Contributor

franz-ms-muc commented Dec 13, 2022

I'll prepare a simple demo.

any Progress @david-cermak ?

@david-cermak
Copy link
Collaborator

Added as c61fe1f

it's very simple and WIP. It only adds a user callback that could handle all received data. (needs to be removed before switching modes and applied again after)

The other option is to reuse CMUX mode for that, we already have two virtual terminals, so we could make another one for handling URC. Unfortunately the DTE/CMUX classes are not fully designed to support variable number of virtual terminals, but that would be an interesting addition.

@diplfranzhoepfinger
Copy link
Contributor Author

i will test it.
have a Good Test-case here:

a SIMCOM A7672E-FASE

we can send here a AT+CGNSSPWR=1 command,
and it will answer with a URC of +CGNSSPWR: READY!

will test now...

@diplfranzhoepfinger
Copy link
Contributor Author

your URC Handler works:

image

@franz-ms-muc
Copy link
Contributor

Hi,

Question: (from Vasil Nikolov)

I understand the URC implementation following way:

  1. You can switch on/off the reaction on URC
    a. This is nice
  2. ESP Modem receives all of the URC
    a. For me it is important to understand how modem implementation differentiates between URCs and a normal response to AT commands
    i. There are commands which require longer time until a response is received.
    b. Can we receive URCs while we are waiting for a response to a certain AT Command?
    i. If yes then how can we be sure that responses to the AT Commands are forwarded to the URC handler instead to the command handler
  3. In our user code we shall define one URC handler with a switch case based on the data and length of received URC
    a. This is a nice approach, we have full control over all URCs we are interested in

@VasilNikolovRilabs
Copy link

@david-cermak could you please provide some feedback about the last questions from Franz?
We would like to really assure that the URC implementation wont impact the normal AT command request/response handling

Thank you!

@david-cermak
Copy link
Collaborator

Hi Franz and Vasil,

sorry for not responding earlier.
The example of injecting a URC callback doesn't really change anything in the esp_modem layers, it just overrides the default command implementation using these two methods:

  1. Traditional method for sending commands makes the class command-able, so it could be employed by the command library to run all/defined AT commands:

command_result Shiny::DCE::command(const std::string &cmd, got_line_cb got_line, uint32_t time_ms, const char separator)

  1. The data callback that processes the replies

command_result Shiny::DCE::handle_data(uint8_t *data, size_t len)

This way, we're able to

  • first call the user urc-callback (and after that)
  • continue with processing the replies by the command library

a. For me it is important to understand how modem implementation differentiates between URCs and a normal response to AT commands

we process the user callback first

if (handle_urc) {
handle_urc(data, len);
}

...and then let the command library to parse the same data.

if (handle_cmd) {
auto ret = handle_cmd(data, len);
if (ret == esp_modem::command_result::TIMEOUT) {
return command_result::TIMEOUT;
}
if (ret == esp_modem::command_result::OK) {
signal.set(1);
}
if (ret == esp_modem::command_result::FAIL) {
signal.set(2);
}
}

i. There are commands which require longer time until a response is received.

You need to be aware that the urc handler would still received all the replies, including the ones that belong to unrelated commands issued by the command library (by user)

b. Can we receive URCs while we are waiting for a response to a certain AT Command?

Yes, the urc-handler gets always called (if enabled). The other part (lib-command processing) is called only if an AT command is in progress.

i. If yes then how can we be sure that responses to the AT Commands are forwarded to the URC handler instead to the command handler

This works like a fork (tee command in bash :-), we pass the same data to two handlers

The only trouble might be the time, physically spent on processing the urc responses, since it actually delays processing the simultaneously active AT command. This wouldn't delay data reception, but might influence timeouts of the defined commands.
(for example, if we're sending AT+CGC\r with timeout of 500ms and spend 600ms on urc processing, then it would fail, but should still work if we spend like 450ms on urc-processing, as the actual reply would simply queue so the data handler should make it in time)

@VasilNikolovRilabs
Copy link

VasilNikolovRilabs commented Mar 2, 2023 via email

@franz-ms-muc
Copy link
Contributor

https://gist.github.com/franz-ms-muc/b28a70d42a86bb2ed292e638a11fa152

i test the URC HAndler now in the CMUX Sample.
it is installed in this Line:
https://gist.github.com/franz-ms-muc/b28a70d42a86bb2ed292e638a11fa152#file-gistfile1-txt-L257

but then never called. However, as i have extended Logging active, i see the URCs are coming in.

however when i was testing it with the Console Sample it was working.

think there is something to fix still.

@franz-ms-muc
Copy link
Contributor

@franz-ms-muc
Copy link
Contributor

@david-cermak did you look into this ?

@david-cermak
Copy link
Collaborator

@franz-ms-muc Could you please also share the code where you install the handler?
Note that the change has already been merged to master (and release as modem-v1.0.0), so you should be able to play with URC directly in the console example. Does it work for you in the default example?

@franz-ms-muc
Copy link
Contributor

franz-ms-muc commented Mar 21, 2023 via email

@david-cermak
Copy link
Collaborator

Could you please close this issue if it work for you? Or comment and share the details if URC handling doesn't work as expected.

@diplfranzhoepfinger
Copy link
Contributor Author

hi,

i tested like this:

use this Commit:
diplfranzhoepfinger@0a33ce5

and the Outcome is this Log:
https://gist.github.com/diplfranzhoepfinger/73da3e69cbee3edbaaa61def2f50eba1

+CGNSSPWR: READY!

is a URC.
it is not handeled by the URC Handler.

@diplfranzhoepfinger
Copy link
Contributor Author

making another Try:

https://gist.github.com/diplfranzhoepfinger/6a3062d8861535f2b3927cdc6e5d299f

there the URC Handler works.

it seems the URC HAndler only works on "cmd xx"
and not on "build in" commands.

pls look into this.
Thanks !
Franz

@diplfranzhoepfinger
Copy link
Contributor Author

@david-cermak can you have a look into this ?

Thanks!

@diplfranzhoepfinger
Copy link
Contributor Author

@diplfranzhoepfinger
Copy link
Contributor Author

@diplfranzhoepfinger
Copy link
Contributor Author

@diplfranzhoepfinger
Copy link
Contributor Author

@diplfranzhoepfinger
Copy link
Contributor Author

Hi, 

https://gist.github.com/diplfranzhoepfinger/780f68da1da1e73565d3a63bccdf9836

removing the URC Handler before switching modes did help to get it running also in CMUX Mode. 

see Commit: diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client@b018980

@diplfranzhoepfinger
Copy link
Contributor Author

Now test it for the A7672_gnss

for "shiny" it works. 

switch to A7672_gnss diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client@39b3129

Outcome: https://gist.github.com/diplfranzhoepfinger/6cea8065dcead903a0a55b63f31f058c

NOT Working at all. 

so URC Handling works for "Shiny" but not for "A7672_gnss"

@david-cermak Please help. 

THANKS !

@david-cermak
Copy link
Collaborator

Okay, so the main issue why it doesn't work in your implementation lies in here:

template <typename ...Agrs> esp_modem::return_type name(Agrs&&... args)...

Perfect forwarding is a great and elegant feature to pass various arguments, but here we just pass it symmetrically, so in the end the command executed from DCE context was symmetrically passed to the device and it's DTE, skipping the URC handler (which belongs to DCE!)

The reason for DCE inheriting also from CommandableIf is that we could have the same command executed directly by DCE (and use the URC handler, which belongs to the DCE), and this is also why we pass this (this DCE) to the implementation of those commands:

return_type Shiny::DCE::name(__VA_ARGS__) { return esp_modem::dce_commands::name(this ARGS(arg_nr) ); }

To fix this, you just replace the perfect forwarding by a macro magic:

@@ -62,21 +80,20 @@ public:
     }
 
 #define ESP_MODEM_DECLARE_DCE_COMMAND(name, return_type, num, ...) \
-    template <typename ...Agrs> \
-    esp_modem::return_type name(Agrs&&... args)   \
-    {   \
-        return device->name(std::forward<Agrs>(args)...); \
+    esp_modem::return_type name(__VA_ARGS__) \
+    {                                               \
+        return esp_modem::dce_commands::name(this ARGS(num)); \
     }

I'd suggest adding the implementation to cpp module, as you'd have to use the utility expansion macros

// Helper macros to handle multiple arguments of declared API
#define ARGS0
#define ARGS1 , p1
#define ARGS2 , p1 , p2
#define ARGS3 , p1 , p2 , p3
#define ARGS4 , p1 , p2 , p3, p4
#define ARGS5 , p1 , p2 , p3, p4, p5
#define ARGS6 , p1 , p2 , p3, p4, p5, p6
#define _ARGS(x) ARGS ## x
#define ARGS(x) _ARGS(x)

as well as namespacing and includes that you might not like to have in hpp module

@timbo100
Copy link

Hi I've been following this and related issue because I think they relate to a capability I need. But since details are incomplete I'm having difficulty.
I'm working with a SIM7600 modem and need to be notified of latent AT command responses or unanticipated events such as SMS text message arrival. Maybe I'm off base here but it sounds like URC is what I need.
Can you help understand:

  • where is the concept as discussed herein for URC clearly defined?
  • if it is a concept/feature that I'm looking for and tests of it's implementation are successful when might I be able to use in the esp-modem component?
  • can I get use of it sooner?
  • otherwise, if you know of how I can implement a method for notification in the interim I would appreciate tc@nuspatial.com (timbo100) Thx

@david-cermak
Copy link
Collaborator

@timbo100 You can experiment with the URC directly in the console example. It's not something super smart, it just installs a callback so you receive everything that's passed to the modem layers as well.

if you type in the

$>urc

you'd also get the response printed out to the console, sine the command installs this buffer-hexdump callback:

command_result handle_urc(uint8_t *data, size_t len)
{
ESP_LOG_BUFFER_HEXDUMP("on_read", data, len, ESP_LOG_INFO);
return command_result::TIMEOUT;
}

@timbo100
Copy link

@david-cermak
Thanks for your response. I saw that in the console example, however it appears to only be implemented if the modem type is "SHINY" and despite my exploring I couldn't figure out how that differs from the implementation of SIM7600.

For future reference can you comment on what SHINY refers to other than a favorite label that one of the contributors uses when creating test?

Can you comment when this feature will be available in the esp_modem component for use?

THANKs to you and the team of contributors.

@franz-ms-muc
Copy link
Contributor

Hi, 

i will test to implement it now for SIMCOM A7672

@diplfranzhoepfinger
Copy link
Contributor Author

diplfranzhoepfinger commented Jun 15, 2023

incorporating your Changes:

i get:

https://gist.github.com/diplfranzhoepfinger/0e2f5e3ff5e273d9ba4d955ac4a1f5ea

which is simple to fix by:

diff --git a/simple_cmux_client/components/simple_cmux_client/simple_cmux_client_main.cpp b/simple_cmux_client/components/simple_cmux_client/simple_cmux_client_main.cpp
index 9c24631..cc0e0ae 100644
--- a/simple_cmux_client/components/simple_cmux_client/simple_cmux_client_main.cpp
+++ b/simple_cmux_client/components/simple_cmux_client/simple_cmux_client_main.cpp
@@ -267,8 +267,9 @@ extern "C" void simple_cmux_client_main(void)
 
     /* Read some data from the modem */
     std::string str;
+    int a;
 #ifndef CONFIG_EXAMPLE_MODEM_DEVICE_SHINY
-    while (dce->get_operator_name(str) != esp_modem::command_result::OK) {
+    while (dce->get_operator_name(str, a) != esp_modem::command_result::OK) {
         // Getting operator name could fail... retry after 500 ms
         vTaskDelay(pdMS_TO_TICKS(500));
     }

diplfranzhoepfinger pushed a commit to diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client that referenced this issue Jun 15, 2023
@diplfranzhoepfinger
Copy link
Contributor Author

and YEAH !!!! the URC Handler runs on the A7672

log here:

https://gist.github.com/diplfranzhoepfinger/2d663b68e47f5cbf7a11a980548166ee

i think it is a cool Feature ! 

now must implement the 2nd Layer above the URC Handler, a "URC Handler Parser" 

@diplfranzhoepfinger
Copy link
Contributor Author

@timbo100 you can see my Cmux - Example, there i got it working.

@timbo100
Copy link

@diplfranzhoepfinger and team, thanks.

while connected to a SIM7600 modem, I changed modem type to SHINY and it worked... I got connected, IP address, and tested the comm with mqtt broker.

I really want to test sending an SMS Text message, does anyone know how to use the console to do this since that command is rather complicated???

It's unclear to me, other than new functionality added to custom modem, what the difference is between a SIM7600 vs SHINY. It would seem then that the Generic Modem is very close to SIM7600(?).

Very much appreciated.

@pavellevitsky
Copy link

Switch modem from SIM76000 to SHINY - how?
Using menuconfig (Development Framework Configuration) --> Connect configuration ?
There is no SHINY in the list...

@david-cermak david-cermak added this to the modem-v1.2.0 milestone Nov 28, 2023
@david-cermak
Copy link
Collaborator

While this is supported in the example by creating a custom module, it's not very user friendly.
I've added a new milestone and will support URC in a better way, directly in the library.

@diplfranzhoepfinger
Copy link
Contributor Author

Hi @david-cermak

Added as c61fe1f

it's very simple and WIP. It only adds a user callback that could handle all received data. (needs to be removed before switching modes and applied again after)

this was working well with the CMUX Example.
but now we switched to the Dual Channel USB Example.
how to add the same URC Handler in this Example ?

https://github.com/espressif/esp-protocols/tree/master/components/esp_modem/examples/pppos_client

I mean one Problem is it is C, so the Syntax will be different.
But we would more like to stay in C.

Thanks,
Franz

@david-cermak
Copy link
Collaborator

Hi Franz,

If you want to use the idea in the modem example, you'd have to stay with C++.
You can use Dual channel mode in C++, just need to create the DTE and DCE separately (same as in other C++ examples). You can check the USB example and the C-API wrappers.

(as mentioned above, I'd like to add URC handler support to the library itself, so those overrides and customized factories won't be needed, planned for modem-v1.2 )

@diplfranzhoepfinger
Copy link
Contributor Author

does this mean:
if we want to stay with C, it will be possible with 1.2
if we swith the pppos_client Sample to C++ we can do it like now.

@david-cermak
Copy link
Collaborator

does this mean: if we want to stay with C, it will be possible with 1.2 if we swith the pppos_client Sample to C++ we can do it like now.

Yes, it means that the URC will be an official API. The solution I offered before was just a workaround, and since the C++ implementation is very modular, we were able to simply inject a handler without changing the library code (100% user space)

david-cermak added a commit to david-cermak/esp-protocols that referenced this issue Jul 24, 2024
david-cermak added a commit to david-cermak/esp-protocols that referenced this issue Sep 16, 2024
@VasilNikolovRilabs
Copy link

VasilNikolovRilabs commented Sep 17, 2024 via email

embedcat pushed a commit to embedcat/esp-protocols that referenced this issue Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants