-
Notifications
You must be signed in to change notification settings - Fork 217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extract timestamp metadata from external parsers output #467
Comments
Which would be the best design option to implement this:
|
The config of a task could map mime-types with correspondent eventType name. |
Actually this is not related to Tasks, I think we shouldn't create a Task to parse a Parser output, this should be done inside the parser. ExternalParsers.xml already supports a syntax where you can specify a regex to search for a pattern and extract it as a metadata key, take a look at the FFmpeg and ExifTool examples commented at the top of that file. Maybe it is enough to handle some external parsers output. Maybe it is not enough to handle others and the syntax should be extended. If needed, before extending it, I would take a look at Tika's ExternalParser current code (from where ours was copied from a long ago), I think they improved it and maybe their enhancements could help us. |
Nice. It seems straightforward, should no transformation is required. |
I'm not sure if IPED already recognizes metadata values as dates for formats other than the common ISO 8601. If not, so there will be the necessity to implement the date parsing, as at least sccainfo uses a different format. Windows Prefetch File (PF) information: |
A few formats are supported/parsed by the DateUtil class, adding more formats there is an option and it can benefit all parsers, not just the external ones. |
used as metadata key and value. If inexistent, uses group 1 and 2 to keep backward compatibility. - Created regex for Recyclebin and Prefetch timestamps extractions. - Created parsing logic on DateUtil class to parse Prefetch dates.
I've noted that the current behaviour of ExternalParsers.xml is not to export and index content and only extract metadatas when there is an existing metadata extraction regex pattern. What do you think? |
I agree to you, for sure, to extract external tools output as parsed content even if there are regex patterns to extract metadata. But I think a new |
"I didn't know about this Tika behavior detail" - Well, it seems reasonable to have this option to parse only metadata. ExifToolParser, for example, should extract additional metadatas, but not change the image content. We should not even change the textual content of the image, as it may be added by the OCR parser. So, if we want to keep the integral external tool command outputs (stdout or/and stderr), they should be added as subitems. Should I implement this? |
"So, if we want to keep the integral external tool command outputs (stdout or/and stderr), they should be added as subitems. Should I implement this?"
|
"Actually I thought this was the current behavior, I didn't know about this Tika behavior detail." This is a behavior of IPED implementation, maybe intentional, exactly not to change textual content of images or videos that should be complemented with external tools metadata information. |
Ok, we can have this option, no problem for me. About overriding other parser output, it can not happen today. If we use the MultipleParser, it concatenates the output of all parsers and merges all metadata extracted by them. If MultipleParser is not used, just 1 parser can run over a file, this already happens today and you can't have OCR content + just ExifTool metadata without using the MultipleParser, you get one or another.
This subitem creation from external tool output is an old ideia. I think it can be useful if the external tool expands some kind of container (edited: not sure if it is possible to model the subitem consumption produced by arbitrary external tools in the xml), or if its output is so huge (e.g. evtx) that it would be better to break it in several subitems. So I think importing the tool output as subitem can be a good new option, keeping the old behavior for current parsers, and another option to break the output as many subitems based on size or some pattern can be useful. |
After I added a simple MultipleParser to IPED (https://github.com/sepinf-inc/IPED/blob/master/iped-parsers/iped-parsers-impl/src/main/java/iped/parsers/misc/MultipleParser.java), concatenating the output of configured parsers and merging all Metadata, Tika released a more configurable MultipleParser abstract class with some implementations (https://github.com/apache/tika/tree/main/tika-core/src/main/java/org/apache/tika/parser/multiple). I think content extraction as subitem was not added, but I think it is useful with some scenarios where concatenating the output of several (external) parsers does not make sense (how to concatenate a txt output with a PDF one?). Not sure if this should be added into the ExternalParser, into the MultipleParser, or both...
OK. |
Maybe not the current Tika behavior, but maybe the past one, I copied the ExternalParser code several years ago and Tika code improved. |
Well, for the scope of this specific issue, it was enough to define a TAG named appendTextContent.
This keeps backward compatibility. |
What relates to split output content, we can put in a new issue. What do you think @lfcnassif ? Or is there already such an issue? One idea is to accept output from STDOUT in TAR format, or other streamable format. Each found file would be "expanded". |
Fine to me. I would name the tag "(use|import|extract)AsTextContent" because there is no previous text content to append a new one, as I described before.
Agreed, there is no such issue. |
"I would name the tag "(use|import|extract)AsTextContent"
The PR is created. |
Great, thank you! |
ExternalParsers.xml could be updated with regex rules to extract metadata from output of:
The text was updated successfully, but these errors were encountered: