mavlogdump.py --format csv only extracts one IMU instance

[magate]

mavlogdump.py --type VIBE will export IMU: 0, 1, 2 (if logged in the BIN file).

Adding the format flag for CSV only exports one of the IMUs. In my instance it was IMU2. I am not sure if this is exporting the primary core or the last message for the unique timestamp.

[shancock884]

This happens because of some logic in the CSV output bit of mavlogdump.py which only takes the last message when there are multiple with the same timestamp.

pymavlink/tools/mavlogdump.py

Lines 333 to 343 in e39879a

	# If this message has a duplicate timestamp, copy its data into the existing data list. Also
	# do this if it's the first message encountered.
	if timestamp == last_timestamp or last_timestamp is None:
	if isbin:
	newData = [str(data[y]) if y != "timestamp" else "" for y in fields]
	else:
	newData = [str(data[y.split('.')[-1]]) if y.split('.')[0] == m_type and y.split('.')[-1] in data else "" for y in fields]
	for i, val in enumerate(newData):
	if val:
	csv_out[i] = val

I don't know if this was done with a sensible use-case in mind, but my view would be to remove this logic, so that CSV behaves the same as the other output formats in that it really dumps all messages of the requested type.

As well as messages with instance fields, you will get the similarly lost output if you were dumping things like MSG, MULT or UNIT messages, as these typically follow each other with the same timestamp too. e.g.:

# Default output - shows all MULTs:
$ mavlogdump.py --type=MULT ArduPlane-Soaring-00000045.BIN
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 45, Mult : 0.0}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 63, Mult : 1.0}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 50, Mult : 100.0}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 49, Mult : 10.0}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 48, Mult : 1.0}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 65, Mult : 0.10000000149011612}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 66, Mult : 0.009999999776482582}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 67, Mult : 0.0010000000474974513}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 68, Mult : 9.999999747378752e-05}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 69, Mult : 9.999999747378752e-06}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 70, Mult : 9.999999974752427e-07}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 71, Mult : 1.0000000116860974e-07}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 73, Mult : 9.999999717180685e-10}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 33, Mult : 3.5999999046325684}
2024-01-07 11:46:46.74: MULT {TimeUS : 1745135, Id : 47, Mult : 3600.0}

# CSV format - only shows last MULT:
$ mavlogdump.py --type=MULT --format=csv ArduPlane-Soaring-00000045.BIN
timestamp,TimeUS,Id,Mult
1704628006.74000502,1745135,47,3600.0

[peterbarker]

On Mon, 29 Jan 2024, Simon Hancock wrote: This happens because of some logic in the CSV output bit of mavlogdump.py which only takes the last message when there are multiple with the same timestamp.

Yeah, that's kind of bad. There are three solutions I can see here: - remove that logic, and the "csv_out" persistent data, just print the message then-and-there - remove the "must have only one message type when dumping as csv" restriction, emit messages only when we have seen a complete set of messages since last we emitted. This new logic would replace current "duplicate timestamp" logic, which I can't think has worked for many, many years. I'm leaning towards the first. Nobody seems to miss the ability to have multiple messages in the CSV output.

[shancock884]

I'm leaning towards the first. Nobody seems to miss the ability to have multiple messages in the CSV output.

Sounds good. If anyone does want the multi-message CSV thing, it can be raised as a specific feature request.
I am happy to PR at some point in the coming days, unless someone else is more motivated!