Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Processing of supported forensic/virtual disk images recursively #102

Closed
ronaldorcosta opened this issue May 22, 2020 · 10 comments · Fixed by #789 or #824
Closed

Processing of supported forensic/virtual disk images recursively #102

ronaldorcosta opened this issue May 22, 2020 · 10 comments · Fixed by #789 or #824
Assignees
@lfcnassif
Labels
new feature

Comments

@ronaldorcosta
Copy link

ronaldorcosta commented May 22, 2020 •
edited by lfcnassif
Loading

When a forensic image (eg E01) contains virtual disks (eg VHD) among its files, it would be useful if IPED had the functionality to automatically export these virtual disks, process them and include them in the case as new evidences.
@lfcnassif lfcnassif assigned lfcnassif and unassigned lfcnassif May 23, 2020
@lfcnassif
Copy link
Member

Removed non portuguese text. I also took a look at libvhdi and libvmdk projects, and seems they support snapshots, that is great.

@lfcnassif lfcnassif self-assigned this Sep 21, 2021
@lfcnassif
Copy link
Member

I started to work on this. At first I will add a basic solution without snapshot support. Maybe the disk without the snapshot could contain more relevant data, maybe not... And if there are many snapshots, decoding all of them could be very expensive and an overkill...

@lfcnassif
Copy link
Member

lfcnassif commented Sep 22, 2021 •
edited
Loading

Embedded images could also be splited, at least for dd & e01, I don't know about vmdk and vhd, not sure if I will target this for now.

lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: break application/x-disk-image in x-raw-image & x-e01-image
8122088
lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: move (virtual) disk image media types to MediaTypes class
27de09a
lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: put back synchronization removed in commit 1b931fe
ca9a308
lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: disable carving on vhd, vmdk, dd & e01 images
3515f99
lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: improve Util.getParentPath(), new Manager.getNumWorkers() method
5e59130
lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: add SleuthkitReader support for decoding embedded (virtual) disks
a978776
lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: new EmbeddedDiskProcessTask, comments:
e805e27 
- current approach add all items from all embedded disks to the same
sleuthkit DB, this should be single threaded. A future improvement could
create separate DBs for each virtual disk, so image decoding could be
done in parallel for different disks
lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: do not split supported embedded/virtual disks
682fe23
lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: add new 'processEmbeddedDisks' option to config files
5378983
lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: install the new EmbeddedDiskProcessTask
d5c6910
lfcnassif added a commit that referenced this issue Oct 19, 2021
@lfcnassif
#102: skip if disk was already exported by a previous processing
f741a01
@lfcnassif lfcnassif mentioned this issue Oct 19, 2021
Merged
lfcnassif added a commit that referenced this issue Oct 20, 2021
@lfcnassif
#102: fix typo in commit 5e59130
e03031b
@lfcnassif lfcnassif changed the title process virtual disks recursively Basic processing of virtual disks recursively Oct 20, 2021
@lfcnassif
Copy link
Member

Just to document here, splitted/segmented images aren't supported for now and this was left as a future improvement.

@lfcnassif
Copy link
Member

I'm reopening this to implement splited image support, shouldn't be hard. The only disadvantage is that all embedded disks should be sent to second processing queue to allow finding image parts.

@lfcnassif lfcnassif reopened this Nov 8, 2021
@lfcnassif lfcnassif mentioned this issue Nov 8, 2021
Merged
lfcnassif added a commit that referenced this issue Nov 8, 2021
@lfcnassif
#102: logs some important steps
58aea2e
lfcnassif added a commit that referenced this issue Nov 8, 2021
@lfcnassif
#102: skip some false positives like $MBR virtual files
45d5c00
lfcnassif added a commit that referenced this issue Nov 8, 2021
@lfcnassif
#102: fix processing stats like estimated time to finish processing
eb43229
lfcnassif added a commit that referenced this issue Nov 8, 2021
@lfcnassif
#102: flag disks w/ decoding error, don't index disks decoded fine
d054b3d
lfcnassif added a commit that referenced this issue Nov 8, 2021
@lfcnassif
#102: support for embedded splited/delta vmdk disks
5f5cca8
lfcnassif added a commit that referenced this issue Nov 8, 2021
@lfcnassif
#102: rename E01/EXX related mimetypes
df75bda
lfcnassif added a commit that referenced this issue Nov 10, 2021
@lfcnassif
#102: update IO error warn msg to use a % of total processed items
c291b52
@lfcnassif lfcnassif changed the title Basic processing of virtual disks recursively Processing of supported virtual disks recursively Nov 11, 2021
@lfcnassif lfcnassif changed the title Processing of supported virtual disks recursively Processing of supported forensic images recursively Nov 11, 2021
lfcnassif added a commit that referenced this issue Nov 11, 2021
@lfcnassif
#102: fix root evidence not shown in 'Evidences' tab while processing
afc11a7
lfcnassif added a commit that referenced this issue Nov 22, 2021
@lfcnassif
#102: improve detection for DD/001 images checking signature AND ext
22cd18c
@lfcnassif lfcnassif mentioned this issue Jan 28, 2022
Closed
lfcnassif added a commit that referenced this issue Jan 28, 2022
@lfcnassif
#102: makes some static variables specific for each image being decoded
8d9fa3a
lfcnassif added a commit that referenced this issue Feb 10, 2022
@lfcnassif
#102: makes some static variables specific for each image being decoded
f96301d 
Conflicts:
	iped-engine/src/main/java/dpf/sp/gpinf/indexer/datasource/SleuthkitReader.java
@lfcnassif lfcnassif changed the title Processing of supported forensic images recursively Processing of supported forensic/virtual disk images recursively Feb 15, 2022
@lfcnassif
Copy link
Member

Reopening to change the logging level to warn instead of error (this goes to console) when processing embedded disks.

@lfcnassif lfcnassif reopened this Jun 21, 2022
lfcnassif added a commit that referenced this issue Jun 21, 2022
@lfcnassif
'#102: set parserException in embedded disks with error, adjust logging
f5765a8
@lfcnassif
Copy link
Member

Closed by commit f5765a8

@hauck-jvsh
Copy link
Member

This is processing ufdr files recursively too? If I process the folder where the ufdr is it will be expanded?

@lfcnassif
Copy link
Member

Unfortunately not at the moment, the primary goal was to support common image formats that could be found in suspect's machines (plus others processed in the same way). AD1 is another format that maybe might be supported.

@lfcnassif
Copy link
Member

PS: actually the UFDR will be processed like generic zip files not like UFDR evidences

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lfcnassif lfcnassif

Labels
new feature
Projects
None yet
Milestone
No milestone
3 participants
@lfcnassif @ronaldorcosta @hauck-jvsh