Add GetLatestTabularData endpoint #593
Conversation
gloriacai01
changed the title
gloriacai01
added
the
ready-for-protos
| message GetLatestTabularDataResponse { | ||
| google.protobuf.Timestamp time_captured = 1; | ||
| google.protobuf.Timestamp time_synced = 2; | ||
| google.protobuf.Struct payload = 3; |
There was a problem hiding this comment.
Thinking about the type we return here... perhaps we can return a byte array (so repeated bytes) instead of a Struct, similar to what we've done with the ByMQL/BySQL response type. This would allow for more expressive data types in the payload (for example, users would be able to interact with a returned date directly instead of first having to parse it from a string).
@dmhilly I know there was some discussion recently around the raw_data vs data return type for the ByQuery endpoints, what was the resolution on that? Okay to return a byte array here instead of a struct?
There was a problem hiding this comment.
Good question! I was thinking about this when I reviewed the scope too... my thought is that it's okay and preferred to return it as a protobuf.Struct, the reasoning being that to get it into Viam Cloud, it would have to have been uploaded as a protobuf.Struct. DataCaptureUpload takes 1+ SensorDatas, which each contains some metadata + either raw bytes or a protobuf.Struct, see here. If the mime_type in the request is tabular but the SensorData data does not contain a struct, we get an error here.
For TabularDataByMQL, the query is across both the data and the metadata, and since there are datetimes in the metadata (the timestamps), these fields (or any fields constructed by the query that aggregates/modifies/defines new fields) had the wrong data type. The query could also theoretically define other fields with BSON data types that are not expressible in proto.
There was a problem hiding this comment.
Gotcha, thanks for that explanation - with all that in mind keeping this as a protobuf.Struct makes sense to me as well :)
proto/viam/app/data/v1/data.proto
Outdated
| string method_name = 3; | ||
| } | ||
|
|
||
| // GetLatestTabularDataResponse provides the data, time synced, and time captured of most recent tabular data captured from the requested data source. |
There was a problem hiding this comment.
Could you add a line in the comment here about how this will only return the latest data point if it was synced within the last year?
gloriacai01
force-pushed
the
data-3391
branch
from
e3d110a to
c1a3c61
Compare
gloriacai01
changed the title
gloriacai01
removed
the
ready-for-protos
Adds GetLatestTabularData rpc + associated request/response messages