LWM2M Gateway (Object No 25) support

[ghost]

@sbernard31 We've seen that you also looked into the Gateway feature specified with LWM2M 1.2. Like yourself, I was also confused by it at first, but thanks to the discussion and the Gateway spec, it becomes quite clear what this is all about. And it is pretty exciting. Out of curiosity, do you know about any activity or if somebody implementing Gateway support in Leshan officially? Or is this currently on your agenda?

A customer of ours informed us a couple of weeks ago that they hacked Gateway support into the Leshan code base somehow, simply with the goal of figuring out if it can be done. It turned out to be possible by hacking around the limitations of the Link class (number of path elements, string prefixes required) in combination with some other hacky ways not shown during this call. Certainly not something to directly use, but it was great to see it can be done.

We have encouraged them to push it to a branch here, but I'd also like to know if there is possibly somebody else also working on it already. From our perspective, Gateway is a highly anticipated feature...

[sbernard31]

do you know about any activity of somebody implementing Gateway support in Leshan officially?

AFIAK, nobody is working on this currently.

Or is this currently on your agenda?

If I remember well this is a LWM2M v1.2 feature, right?
And unfortunately we are sill focus on implementing LWM2M 1.1. : #1235
Sometime, we feel we are not fast enough 😞

A a customer of ours informed us a couple of weeks ago that they hacked Gateway support into the Leshan code base somehow, simply with the goal of figuring out if it can be done. It turned out to be possible by hacking around the limitations of the Link class (number of path elements, string prefixes required) in combination with some other hacky ways. Certainly not something to directly use, but it was great to see it can be done.

Yep, great to know that.
We recently refactor a lot about Link in Leshan the current development branch (master used to develop the v2.0.0 of Leshan) I don't know if this will make the change easier or harder 😬. Or maybe you are more talking about LwM2mPath class ?

We have encouraged them to push it to a branch here,

Yes this is good idea. Even if this will probably not be integrated soon, this could be used as base for future development and/or for discussion and/or for other users who can not wait for a version of Leshan targeting LWM2M v1.2.

From our perspective, Gateway is a highly anticipated feature...

OK I will add this feature to the list at : #1235

@sag-eweingaertner, thx for sharing this with us 🙏

[sbernard31]

Should we close this issue ?

[ghost]

I'd suggest to keep it open, simply to have a forum for people interested in this feature in Leshan.

[sbernard31]

Ok we can do that.

[jvermillard]

I'm currently maintaining a fork of Leshan with object 25 support. It's kinda ugly (it's abusing the rootprefix) but it kinda works. So it's on my to-do list to clean up this implementation and makes it consumable by the upstream.
As Simon said it's a 1.2 feature, but strangely it doesn't depend on any other 1.2 feature so it can run on 1.1 (even if it's not the 'spirit' of the spec).

[jvermillard]

So in place of keeping our dirty hack in my corner, I started cleaning it and added a branch in case someone is interested and would also like to provide feedback:
https://github.com/eclipse/leshan/compare/obj25

First, we implemented this 1.2 feature on top of the 1.1 branch. Strangely this feature doesn't require any part of 1.2, so it works as is on 1.1 (no specification modification). Same for the client we are using, it's done on a 1.1 client.

The idea of this implementation is to create a registration for end IoT devices, reusing the CoAP identity/session of the gateway.

All the attached end IoT device registrations will be if the gateway is unregisteredremoved.

How to add the registrations? Well, it's not covered by the spec. It's up to the user. What we do in our implementation is we have a registration handler, which reads/25 (if it's advertised by gateway registration).
Once /25 is read, we register the end IoT devices.
At some point, I'll clean up and also publish this piece of code.

[sbernard31]

@jvermillard, I add a link to your blog post about this : https://vermillard.com/post/object25/

[sbernard31]

I took some time to look at object25.

I created a obj25_alt branch largely inspired by obj25 one.

@jvermillard, I didn't tested but if you find time you can review/try it and let me know if this way works for you.

The main benefits of obj25_alt is there is almost no changes on current code base.
The leshan-server-gateway module I created in this branch is not intended to be included in master or released in 2.x, this is just to show it is maybe possible to do pretty much the same thing that we do in obj25 with current API.

I also read more carefully the Gateway specification and current code should probably not work with :

1. Content format SenML JSON/CBOR. Because, "the Prefix MUST be part of the name of the IoT Device Resources or Resource Instances." (see LWM2M-Gateway-v1.1§9. Data Formats) Maybe relative to this issue : Add Alternate/root Path to SenML path ? #1438

2. Composite and Send Operations. A clean way to support that means probably to change lot of API about Path, Nodes and Encoder/Decoder. I didn't think too much about it and currently don't know if there is way to workaround this by "twisting" current API.

[jvermillard]

After looking at the code, it looks like a good idea; I need to test it now.

[jvermillard]

I started to test it, it's starting to work. I need to finish, but I don't think the idea has fundamental issues.

Some first minor comments:

• some code is bugged in the server-gateway like https://github.com/eclipse/leshan/compare/obj25_alt#diff-8e1f484829212d5b508da205b4249430c610d52c9979ca6149ef5332834b2631R81-R83 the appData is not passed to the registration builder
• I find the customization of the server using inheritance, where most of Leshan code uses the builder pattern, confusing: https://github.com/eclipse/leshan/compare/obj25_alt#diff-600285a3871e38596be2e1105fd3abf2f99832b8eee64e5b9c99082a8d787753R171

[jvermillard]

Another feedback but not related is the "application data". Most libraries call it user data, user-defined data, or metadata. I was confused by the Registration.java Javadoc:

• @return Some application data which could have been added at Registration by the {@link Authorizer}

It doesn't really explain the purpose: arbitrary data (for example business data or extra client information) associated with the registration by the library user.

[sbernard31]

I started to test it, it's starting to work. I need to finish, but I don't think the idea has fundamental issues.

👍

some code is bugged

I'm not surprised you find bugs in obj25_alt branch.
I didn't test it. The intend was just to share the idea. You can push fix in that branch directly.

I find the customization of the server using inheritance, where most of Leshan code uses the builder pattern, confusing

Currently there is 2 way to customize Leshan :

1. builder for what is really intended to be changed.
2. inheritance for more advance changes, to let the door open in case workaround is needed.

In this case, changing the ServerEndpointToolBox looks like to be 2.).
And adding a setter in the builder will not be so easy because this object depends to many other objects that can be set in the builder.

Another feedback but not related is the "application data". Most libraries call it user data, user-defined data, or metadata.

The feature was requested by @msangoi.
The idea is , a Leshan Server will call some services to know if a client is authorize to register.
This is done in Authorizer.
During this authorization step, you could get some data (collected from services) that you want to attach to registration (kind of optimization to avoid to recall that services)
How we use it is a kind of HACK.

If you still feel that this is not clear enough, please open a dedicated issue where we will discuss how to improve it.

[jvermillard]

The feature was requested by @msangoi.
The idea is , a Leshan Server will call some services to know if a client is authorize to register.
This is done in Authorizer.
During this authorization step, you could get some data (collected from services) that you want to attach to registration > (kind of optimization to avoid to recall that services)
How we use it is a kind of HACK.

My feedback is just on the naming, mainly the usage of the "application" term vs. "user" or "metadata" IMO, it makes sense to me to have a way to attach free-form contextual data to the registration.

On the gateway-service, I'm not using the module; I use it as an example and copy/paste it into my server code to make it work.
I'm going to continue testing it this afternoon 🤞

[sbernard31]

My feedback is just on the naming, mainly the usage of the "application" term vs. "user" or "metadata" IMO, it makes sense to me to have a way to attach free-form contextual data to the registration.

You just want to change javadoc or also getter name ?

On the gateway-service, I'm not using the module; I use it as an example and copy/paste it into my server code to make it work.
I'm going to continue testing it this afternoon crossed_fingers

Maybe this is worth anyway to fix the branch for people which would like to experiment this ?

[sbernard31]

I put some fix to previous solution in obj25_alt branch.

And I created another branch obj25_alt2 with an alternative way, see : https://github.com/eclipse/leshan/blob/obj25_alt2/leshan-server-gateway/src/main/java/org/eclipse/leshan/server/gateway/IotDevicesRegistrationHandler.java#L64-L96

[jvermillard]

After some discussion, I settled on the alt2 version since it looks less exposed to Leshan API breakage

[seiya-komonguchi]

To what extent is the implementation of the gateway for obj25, obj_alt and obj_alt2 complete at the specification level?
If there are any parts that have not been implemented, could you please tell us which parts those are?

[jvermillard]

The alt2 idea works to the extent of some things like /dp send operation. there are probably some other quirks but the base read/write/observation/registration works

[seiya-komonguchi]

Thank you for your answer, jvermillard.
I see that the basic operation works.

Have you verified the operation of the gateway using actual devices?
If the gateway operation is verified under the constraints of device information set in the program, I would like to verify the operation using actual devices.
Could you tell me the current situation?

[jvermillard]

yes, with an LTE-M LWM2M gateway with Bluetooth sensors as end IoT devices

[seiya-komonguchi]

I would like to ask about obj25_alt2. Has the operation been verified for devices that perform IP communication as end devices communicating with Gateway?

Also, I am thinking of using a master branch server or a demo server when running the Gateway, so did you make any changes to the master branch server side when checking the Gateway operation? Is the server able to recognize the objects sent from the Gateway without any changes on the server side?

[jvermillard]

Has the operation been verified for devices that perform IP communication as end devices communicating with Gateway?


The spec does not cover how the Gateway and the end device communicate; you can imagine whatever you want. So, I don't understand the question.

I'm using released cut from master without any changes

[seiya-komonguchi]

That was not a good way to ask the question.
As you say, there is no description of the communication method between the end device and the gateway in the specification. Therefore, I wanted to ask whether obj25_alt2 uses only Bluetooth for communication between the end device and the gateway, or whether it also supports other communication methods (e.g., IP communication).

I would like to ask about the usage of Obj25_alt2.
I would like to register devices to the server via Gateway with Obj25_alt2, but I don't know how to do it. Could you tell me how to do this?
Currently, I am able to start the demo server, but I am not able to register end devices to the server via the gateway.

[jvermillard]

This branch and feature cover only the server-side implementation, so you need a client implementation of the LWM2M client plus gateway support.

[seiya-komonguchi]

I understood, I need to support the client and the Gateway myself.

I have two new questions.
The first is about the client side program.
I am trying to set data in the IoTDeviceObjects resource in the Gateway object. (I am setting the data on the program to try it out).
I can see the gateway object in the Leshan demo UI: http://localhost:8080, but the IoTDeviceObjects values are not being reflected.
The code is as follows.
private String ioTDeviceObjects = "</3/0>;ver=1.2,</6/0>;ver1.1,</3303/0>,</3303/1>";
Could you please tell me how you described the IoTDeviceObjects resource in the Gateway object of the client when you checked the operation of the Gateway, jvermillard?

The second question is about the server side program.
I believe that leshan-server-gateway is a server-side program that can process Gateway objects, and I would like to start leshan-server-gateway.
However, I cannot start leshan-server-gateway because the executable file does not exist in the target folder even after performing "mvn clean install".
Could you please tell me what steps I should take to use leshan-server-gateway?

[seiya-komonguchi]

I have been creating a client.
I am having a hard time setting up the IoT Device Objects resource in the Gateway object.
I am thinking that the IoT Device Objects resource should be programmed as a Link type (leshan-core/src/main/java/org/eclipse/leshan/core/link/Link.java), and I am setting the uriReference and attributes.
However, I cannot check the IoT Device Objects resource on the server.

Could you tell me how to program the IoT Device Objects resource on the client side?

[sbernard31]

From #1237 (comment) :

1. Content format SenML JSON/CBOR. Because, "the Prefix MUST be part of the name of the IoT Device Resources or Resource Instances." (see LWM2M-Gateway-v1.1§9. Data Formats) Maybe relative to this issue : Add Alternate/root Path to SenML path ? #1438

I moved forward about #1438, see #1638.