clp-s: Add option to record metadata about decompressed archive chunks. #485
Conversation
| @@ -349,6 +364,18 @@ CommandLineArguments::parse_arguments(int argc, char const** argv) { | |||
| throw std::invalid_argument("ordered-chunk-size must be used with ordered argument" | |||
| ); | |||
| } | |||
|
|
|||
| if (m_mongodb_uri.empty() ^ m_mongodb_collection.empty()) { | |||
There was a problem hiding this comment.
Can you add a brief comment on why you use "xor"? like "either be both unspecified, or must be both specified"
There was a problem hiding this comment.
Sure, I'll add a comment
| @@ -15,5 +15,14 @@ constexpr char cArchiveArrayDictFile[] = "/array.dict"; | |||
| constexpr char cArchiveLogDictFile[] = "/log.dict"; | |||
| constexpr char cArchiveTimestampDictFile[] = "/timestamp.dict"; | |||
| constexpr char cArchiveVarDictFile[] = "/var.dict"; | |||
|
|
|||
| namespace results_cache { namespace decompression { | |||
There was a problem hiding this comment.
Is there a chance that we can use constant defined in https://github.com/y-scope/clp/blob/main/components/core/src/clp/clo/constants.hpp?
There was a problem hiding this comment.
I thought about it, and the way that would probably make sense if we did want to share this stuff between clo/clp-s is to just have a utility class that creates the required BSON objects and hides these field names. Right now we do take some classes from clp, but it's in a kind of nasty way where we have to manually add all of the required files to our build target. Really what we should do is have a build target that builds a library of utilities that all of our different binaries can use.
Since that's a bit of a bigger refactor I wanted to avoid including it in this PR. Plus the namespace styles used by clo are inconsistent with what we do here, and I want to avoid mixing styles.
|
can you please also test and make sure that when a valid mongodb uri is given, clp-s can write the fields properly to the collection? |
…compression code path
Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
|
looks good to me |
kirkrodrigues
changed the title
…s. (y-scope#485) Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
Description
This PR introduces the option to record metadata about decompressed archive chunks to mongodb for the purpose of log viewing. The changes are localized to CommandLineArguments and JsonConstructor.
We record the start and end timestamp of each chunk in the start and end index fields of the metadata since that is the information that will be used to look up and order chunks for viewing in the context of clp-s.
Validation performed