Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mavlogdump: CSV option: Don't cut out messages with same timestamp #912

Merged
merged 1 commit into from
Feb 1, 2024

Conversation

shancock884
Copy link
Contributor

The PR is to fix #905.

It removes the logic which cuts out subsequent messages from CSV output if they have the same timestamp.
This typically affects messages with instances like IMU, where only the last one of a sequence would be output.
It would also affect messages like MSG, MULT, UNIT, etc, which may be added to the log file in batch.

Main Testing

I have tried a few files, with a few message types, and can now see expected output. For example, the IMU data for both instances even when they have the same timestamp:

$ mavlogdump.py --format=csv --types=IMU ArduPlane-Soaring-00000045.BIN
timestamp,TimeUS,I,GyrX,GyrY,GyrZ,AccX,AccY,AccZ,EG,EA,T,GH,AH,GHz,AHz
1704628006.74000502,1745135,1,-0.00015193763829302043,9.939412120729685e-05,-3.919340451830067e-05,-0.0009788739262148738,-0.00279411138035357,-9.817838668823242,0,0,25.10784149169922,1,1,760,799
1704628006.75999689,1765127,0,-3.0728398996870965e-05,-6.79766817484051e-05,1.5066639207361732e-05,-0.00010934187594102696,-0.0010120467050001025,-9.816944122314453,0,0,25.109102249145508,1,1,999,999
1704628006.75999689,1765127,1,-0.00019223210983909667,-1.3705200217373203e-05,0.00017314731667283922,-0.002659626305103302,0.0018529330845922232,-9.819148063659668,0,0,25.109167098999023,1,1,760,799
1704628006.77998900,1785119,0,6.50839792797342e-05,-2.898311140597798e-05,2.255062645417638e-05,-0.001452248776331544,-0.0007089077262207866,-9.817065238952637,0,0,25.110427856445312,1,1,999,999
1704628006.77998900,1785119,1,0.0001738743158057332,0.00016524168313480914,1.77201673068339e-05,-0.0001426656381227076,0.0013984335819259286,-9.817688941955566,0,0,25.11049461364746,1,1,760,799
...

Additional Testing/Fixes

Testing with DF Text logs found that outputting a text log to CSV was throwing an error.
This has been fixed by using (isbin or islog) in place of isbin in a couple of the if statements.

When specifying a message type that doesn't exist, an exception was being thrown, so I have added some checks to output a nicer message for users in this case:

$ mavlogdump.py --format=csv --types=aaa ArduCopter-ArmFeatures-autotest-1705041782499410.tlog
Message type 'aaa' not found
$ mavlogdump.py --format=csv --types=aaa ArduPlane-Soaring-00000045.BIN 
Specified type 'aaa' not found in log file
$ mavlogdump.py --format=csv --types=aaa ArduPlane-Soaring-00000045.log 
Specified type 'aaa' not found in log file

Behaviour with TLOGs

When dumping TLOGs to CSV, it is allowed to specify multiple messages in the --type argument. This outputs data for the message at that timestamp, and empty fields for the others (shown below).
This may look a bit odd, and I don't know if it is what was intended, but my change doesn't appear to have altered the behaviour, so I was thinking not to touch it. The previous logic would only ever have merged messages onto one line if they had identical timestamps to the microsecond level - which doesn't sound possible!

$ mavlogdump.py --format=csv --types=SYS_STATUS,HEARTBEAT ArduCopter-ArmFeatures-autotest-1705041782499410.tlog
timestamp,SYS_STATUS.onboard_control_sensors_present,SYS_STATUS.onboard_control_sensors_enabled,SYS_STATUS.onboard_control_sensors_health,SYS_STATUS.load,SYS_STATUS.voltage_battery,SYS_STATUS.current_battery,SYS_STATUS.battery_remaining,SYS_STATUS.drop_rate_comm,SYS_STATUS.errors_comm,SYS_STATUS.errors_count1,SYS_STATUS.errors_count2,SYS_STATUS.errors_count3,SYS_STATUS.errors_count4,HEARTBEAT.type,HEARTBEAT.autopilot,HEARTBEAT.base_mode,HEARTBEAT.custom_mode,HEARTBEAT.system_status,HEARTBEAT.mavlink_version
1705041782.55857992,,,,,,,,,,,,,,10,3,65,16,2,3
1705041782.57506800,,,,,,,,,,,,,,10,3,65,0,4,3
1705041782.57640886,321969423,268468495,51413263,0,0,-1,-1,0,0,0,0,0,0,,,,,,
1705041782.57741690,,,,,,,,,,,,,,6,8,0,0,0,3
1705041782.57862782,321969423,268468495,51413263,0,0,-1,-1,0,0,0,0,0,0,,,,,,
1705041782.57950497,,,,,,,,,,,,,,6,8,0,0,0,3
...

This also fixes the CSV output option for DF Text logs
Fail nicely if the specified type doesn't exist
Copy link
Contributor

@peterbarker peterbarker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can probably use --condition to trim down to only a single instance of the (e.g.) IMU message

tools/mavlogdump.py Show resolved Hide resolved
@peterbarker peterbarker merged commit 55fcf4d into ArduPilot:master Feb 1, 2024
12 checks passed
@peterbarker
Copy link
Contributor

Merged, thanks!

@shancock884 shancock884 deleted the mavlogdump-csv-output branch February 1, 2024 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

mavlogdump.py --format csv only extracts one IMU instance
2 participants