-
Notifications
You must be signed in to change notification settings - Fork 244
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DB3 bag is irrecoverable if metadata.yaml doesn't exist #395
Comments
can you please include copy/pasted output of the commands, in "Actual Behavior"? |
@dsharma2701 thanks for bringing this up. I've thought about this a couple of times, especially in the case that if a That is indeed a problem, and we should have something like Line 72 in 33cf70b
That would extract the metadata for the stored messages. I think components like which compression algorithm was used is not part of it, but could then be added manually. I think this could be a good starting point for a verb like |
Just so people are aware, I am currently trying to implement a |
This issue has been mentioned on ROS Discourse. There might be relevant details there: https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-2020-07-16/15468/1 |
I'd like to mention that I've been working on this at https://github.com/jhdcs/rosbag2/tree/bug/missing_metadata, and as of commit 8a4c932 this appears to be working on my test case in Linux. I have not created a pull request yet, because I'm having some trouble creating unit tests for the code (mostly due to my inexperience). I'm letting you guys know about this because I think what I have so far may be enough for @dsharma2701 to recover his bag, and possibly other people with similar emergency situations. However, use the code at your own risk - It's only been run on Linux, was developed for Foxy, and has not been unit tested. Still, I want to let the community know that this option is available. And let the community know that this is being worked on. And if anyone knows unit tests, I'd love to learn from them! |
This issue has been mentioned on ROS Discourse. There might be relevant details there: https://discourse.ros.org/t/tooling-wg-breakout-meeting-2021-01-22-rosbag2-reindexing/18489/3 |
@jhdcs Sorry if this is mentioned elsewhere. But are we able to recover metadata for custom messages (provided we install the messages package in our workspace and have sourced it). If so could you please share instructions |
It may be best to ask this sort of question in Ros Answers, especially since this issue has been closed for several months. However, since this appears to be asking about functionality of the reindexer, and it may not have been clear before, I'll give an answer here. The reindexer merely reads the metadata stored inside of your bag file. Provided the metadata can be discovered within (i.e: it was recorded properly), it should be able to recover it. If it doesn't, please make a new GitHub issue describing the problems you are having, as well as any debugging info you can provide (small, sanitized bags that demonstrate the problem would be lovely, as well as what you think the metadata file should be after reindexing) |
I'm having the issue mentioned in the OP where the metadata file is not being created because the device lost power (maybe?) Inside the |
@Benblob688 Please see
|
A bit late to this comment, but I found that the above commands didn't quite work in Humble to recover a .db3 without metadata. Maybe it is a user error. So we developed our own tool to fix ros2bag and import them into Python for plotting we found useful. https://github.com/arplaboratory/ros2bag_fixer/tree/main In brief the way this works, is first it uses the sqlite3 tool to decorrupt the .db3 file in a .sql file. Next it takes the contents of the .sql file and outputs them to a CSV. Finally, we decode the CSV file using Python allowing us to read some data. Also once you have the .sql file you can analyze using this website to see the raw data outputs. This is for Humble ROS2 use cases |
Description
ros2 db3 bags fails to load or get info on if the metadata yaml is missing
Expected Behavior
Expecting it to have the ability to generate it offline in case of any corruption during saving the bag
Actual Behavior
If accompanying metadata yaml doesn't exist with the db3, it fails to load/get info on.
[ERROR] [1588037968.719797386] [rosbag2_storage]: Could not open '/root/Downloads/jobq_bags.db3' with 'sqlite3'. Error: Failed to setup storage. Error: Error when processing SQL statement. SQLite error (11): database disk image is malformed
To Reproduce
** Steps to reproduce the behavior, e.g.
System (please complete the following information)
Additional context
Would love to find a way to recover the bag as its 50 GB worth of data.
The text was updated successfully, but these errors were encountered: