DB3 bag is irrecoverable if metadata.yaml doesn't exist

[dsharma2701]

Description

ros2 db3 bags fails to load or get info on if the metadata yaml is missing

Expected Behavior

Expecting it to have the ability to generate it offline in case of any corruption during saving the bag

Actual Behavior

If accompanying metadata yaml doesn't exist with the db3, it fails to load/get info on.
[ERROR] [1588037968.719797386] [rosbag2_storage]: Could not open '/root/Downloads/jobq_bags.db3' with 'sqlite3'. Error: Failed to setup storage. Error: Error when processing SQL statement. SQLite error (11): database disk image is malformed

To Reproduce

** Steps to reproduce the behavior, e.g.

1. Use any ros2 bag and rename metadata yaml to tmp
2. Run ros2 bag info

System (please complete the following information)

• OS: Ubuntu Bionic
• ROS 2 Distro: dashing
• Version:

Additional context

Would love to find a way to recover the bag as its 50 GB worth of data.

[emersonknapp]

can you please include copy/pasted output of the commands, in "Actual Behavior"?

[Karsten1987]

@dsharma2701 thanks for bringing this up. I've thought about this a couple of times, especially in the case that if a ctrl-c command fails to stop a recording - which is does unfortunately quite reliably in the end to end tests on windows - the recorded data is still available in the .db3, yet unusable due to the missing metadata file.

That is indeed a problem, and we should have something like rosbag reindex which is there in ROS1. I honestly don't think it's that hard to achieve, given that almost all information is available in the database. There is this function available, which we luckily didn't remove from the interface:

rosbag2/rosbag2_storage_default_plugins/include/rosbag2_storage_default_plugins/sqlite/sqlite_storage.hpp

Line 72 in 33cf70b

rosbag2_storage::BagMetadata get_metadata() override;

That would extract the metadata for the stored messages. I think components like which compression algorithm was used is not part of it, but could then be added manually.

I think this could be a good starting point for a verb like ros2 bag reindex <path/to/database.db3> or alternatively ros2 bag reindex <path/to/manual.yaml> where the manual yaml file specifies some preliminary data, such as the relative file paths and the reindex function would then kind of fill in the blanks.

[jhdcs]

Just so people are aware, I am currently trying to implement a reindex verb that does what @Karsten1987 suggested.

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-2020-07-16/15468/1

[jhdcs]

I'd like to mention that I've been working on this at https://github.com/jhdcs/rosbag2/tree/bug/missing_metadata, and as of commit 8a4c932 this appears to be working on my test case in Linux. I have not created a pull request yet, because I'm having some trouble creating unit tests for the code (mostly due to my inexperience).

I'm letting you guys know about this because I think what I have so far may be enough for @dsharma2701 to recover his bag, and possibly other people with similar emergency situations. However, use the code at your own risk - It's only been run on Linux, was developed for Foxy, and has not been unit tested.

Still, I want to let the community know that this option is available. And let the community know that this is being worked on. And if anyone knows unit tests, I'd love to learn from them!

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/tooling-wg-breakout-meeting-2021-01-22-rosbag2-reindexing/18489/3

[sisaha9]

@jhdcs Sorry if this is mentioned elsewhere. But are we able to recover metadata for custom messages (provided we install the messages package in our workspace and have sourced it). If so could you please share instructions

[jhdcs]

It may be best to ask this sort of question in Ros Answers, especially since this issue has been closed for several months. However, since this appears to be asking about functionality of the reindexer, and it may not have been clear before, I'll give an answer here.

The reindexer merely reads the metadata stored inside of your bag file. Provided the metadata can be discovered within (i.e: it was recorded properly), it should be able to recover it. If it doesn't, please make a new GitHub issue describing the problems you are having, as well as any debugging info you can provide (small, sanitized bags that demonstrate the problem would be lovely, as well as what you think the metadata file should be after reindexing)

[Benblob688]

I'm having the issue mentioned in the OP where the metadata file is not being created because the device lost power (maybe?)
Can you provide an example of how to use the reindex functionality you describe was implemented here? when I run ros2 bag -h the only options are info, play, record.

Inside the rosbag2_ directory is a .db3m .db3-shm, and .db3-wal

[MichaelOrlov]

@Benblob688 Please see ros2 bag reindex command

ros2 bag reindex -h
usage: ros2 bag reindex [-h] bag_directory {my_read_only_test_plugin,my_test_plugin,sqlite3}

Reconstruct metadata file for a bag

positional arguments:
  bag_directory         bag to reindex
  {my_read_only_test_plugin,my_test_plugin,sqlite3}
                        storage identifier to be used, defaults to 'sqlite3'

[Jm7752]

A bit late to this comment, but I found that the above commands didn't quite work in Humble to recover a .db3 without metadata. Maybe it is a user error. So we developed our own tool to fix ros2bag and import them into Python for plotting we found useful.

https://github.com/arplaboratory/ros2bag_fixer/tree/main

In brief the way this works, is first it uses the sqlite3 tool to decorrupt the .db3 file in a .sql file. Next it takes the contents of the .sql file and outputs them to a CSV. Finally, we decode the CSV file using Python allowing us to read some data.

Also once you have the .sql file you can analyze using this website to see the raw data outputs. This is for Humble ROS2 use cases

https://codebeautify.org/sql-to-csv-converter