Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix --continue, improve and rename persistentID #937

Merged
merged 30 commits into from
Jan 28, 2022
Merged

Fix --continue, improve and rename persistentID #937

merged 30 commits into from
Jan 28, 2022

Conversation

lfcnassif
Copy link
Member

@lfcnassif lfcnassif commented Jan 25, 2022 •
edited
Loading

Closes #784, closes #792 and closes #941:

  • make sure item ids are not used before possibly being reassigned when resuming processing adding a number of checks;
  • improves persistentID computation using carvedId, add checks for required params used in the formula;
  • fixes a possible parent-child relationship issue when resuming processing of recursive embedded virtual disks (master);
  • replace parentIdInDataSource requirement (and methods) for parentGlobalID presence in all items;

Still need to do more tests...

@lfcnassif lfcnassif force-pushed the #784_improve_and_rename_globalID branch 2 times, most recently from 2be79a5 to 0aa4519 Compare January 25, 2022 23:33
@lfcnassif lfcnassif changed the title #784 improve and rename persistentID Fix --continue, improve and rename persistentID Jan 26, 2022
@lfcnassif
#941: checking code to warn if id is used before possibly being restored
bbacbf9
@lfcnassif
#941: do not use id in ExportFileTask, it could be reassigned
630146d
@lfcnassif
#792: add numCarved counter per parent and add carvedId to carved items
11a47b9
@lfcnassif
#784/#792 improve globalID formula, throw exception if something is bad
9abaeab
@lfcnassif
#784: always throw exception if parentGlobalID is not set in items:
1c97a00 
- this property is needed when resuming processing to get a previous
parent id referenced by subitems which parents were not commited, then
when reprocessing parents, their id can be updated to the previous
value, so parent-child relationships will be preserved.
@lfcnassif
#784: store parentGlobalID for all items produced by FolderTreeReader
daf13ab
@lfcnassif
#784: store parentGlobalID for all items from SleuthkitReader and also:
53d443b 
- fix embedded disks subitems references to parentGlobalID
@lfcnassif
#941: do not use id in UFDRReader before possibly updating its value
093e630
@lfcnassif
#941 use ID in ParsingTask after posible update, set meta removed before
ee738c7
@lfcnassif
#784: remove parentIdInDataSource setters & getters, not needed anymore
d599e31
@lfcnassif
simplify FolderTreeReader parent tracking
0e9228b
@lfcnassif
#179: fix a NPE and other exception: doc values aren't reusable anymore
ce4d107
@lfcnassif
#784: additional minor checking for null invalid item paths
6b352c9
@lfcnassif
#784: rename globalID (previous persistentID) to trackID
b0eb4f3
@lfcnassif
#784: compute and store a real globalID unique across cases
33584b0
@lfcnassif
#792: skip non last text frags when looking for parents with lost child
aa5484b
@lfcnassif
#792: don't use ID_IN_SOURCE in trackID of carved/subitems:
ba97468 
- could make subitems of embedded disks decoded again to be added twice
@lfcnassif
#784: fix processing of recursive disks after commit 53d443b
7eee8e1
@lfcnassif
#792: skip seen containers enqueued twice (b98dc52 regression)
3a37b98
@lfcnassif
fix #942: fix ConcurrentModificationException iterating over a new array
e3d1e4f
@lfcnassif
minor change to show just one ElasticSearch commit msg for all workers
3859c57
@lfcnassif lfcnassif force-pushed the #784_improve_and_rename_globalID branch from f91e560 to 3859c57 Compare January 26, 2022 20:29
@lfcnassif
Copy link
Member Author

I changed commit messages to point to correct newly created issues and just force-pushed, sorry if someone checked out this branch.

@lfcnassif lfcnassif force-pushed the #784_improve_and_rename_globalID branch from 2c8180c to 0c724c8 Compare January 28, 2022 21:57
@lfcnassif
Copy link
Member Author

lfcnassif commented Jan 28, 2022 •
edited
Loading

This was harder than expected, but I think now --continue is working fine. I'll run a final large processing test before merging.

@lfcnassif lfcnassif marked this pull request as ready for review January 28, 2022 22:02
@lfcnassif lfcnassif merged commit 0c724c8 into master Jan 28, 2022
@lfcnassif lfcnassif deleted the #784_improve_and_rename_globalID branch January 28, 2022 23:54
@lfcnassif
Copy link
Member Author

lfcnassif commented Jan 29, 2022 •
edited
Loading

@hauck-jvsh and @tc-wleite, this took from me much more time than I would like. I'll be on vacation next week. When I return, I will focus on your ready PRs, thanks!

PS: At least it was beautiful to me finally seeing --continue working fine when processing an evidence with multiple recursively embedded disks, interrupted a lot of times :-)

@hauck-jvsh
Copy link
Member

hauck-jvsh commented Jan 29, 2022 •
edited
Loading

Sure, relax and enjoy your vacation. Just to be sure, after this there will be the globalId and the trackId?

@wladimirleite
Copy link
Member

Enjoy your vacation, @lfcnassif!

@lfcnassif
Copy link
Member Author

Sure, relax and enjoy your vacation. Just to be sure, after this there will be the globalId and the trackId?

Yes. Thank you two!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@lfcnassif @hauck-jvsh @wladimirleite