Hikvision, WFS and DHFS File System parser - DVR videos

[gfd2020]

Allows you to index videos recorded on a DVR with the Hikvision, WFS and DHFS file system.
This PR will need a lot of work. It turns off several flags to function and modifies the program's operation, causing malfunctions in the normal use of the IPED.

[lfcnassif]

Thanks @gfd2020 for this contribution. I'm changing this to draft until it is ready.

[paulobreim]

3 years ago, I needed to research hikvision to try to recover images from a HD that had been formatted.

There is no official documentation and the formatting is hikvision's own.

I managed to find some software (paid) to recover the videos, but none of them have recovery of the Logs, which for an expert purpose is extremely important.
I managed to gather some technical information and ended up making a program in C that recovers the Logs.
All material is available and I can help in any way.
I also have some material I collected on the internet and also emails from hikvision staff who could perhaps provide some documentation.

paulo

[gfd2020]

3 years ago, I needed to research hikvision to try to recover images from a HD that had been formatted.

There is no official documentation and the formatting is hikvision's own.

I managed to find some software (paid) to recover the videos, but none of them have recovery of the Logs, which for an expert purpose is extremely important. I managed to gather some technical information and ended up making a program in C that recovers the Logs. All material is available and I can help in any way. I also have some material I collected on the internet and also emails from hikvision staff who could perhaps provide some documentation.

paulo

Hi Paulo.

The implementation of this PR is based on the document below. The values ​​of the logs are not being analyzed, but from what I've seen, it's ok to get the values ​​of the log in the RATS fields. What you could help with is the interpretation of the RATS field descriptions. I looked at the HEX and I didn't understand the logic of the description.

https://eudl.eu/pdf/10.1007/978-3-319-25512-5_13

[gfd2020]

Hi @lfcnassif .

Could you help me with the questions below so I can proceed with this PR?

1. As processing needs to change several IPED parameters, would it be feasible to create a new type of profile? For example, 'dvr'?

2. SleuthkitInputStreamFactory.java needs to be update on line 'this.emptyContent = true;' to 'this.emptyContent = false;'. Is it plausible to create a configuration variable? In which configuration file would it be viable?

3. FragmentLargeBinaryTask is always enabled. However, it is necessary to disable this flag. Would it be possible to re-enable this flag?

4. If this PR's task has some configuration parameters, should these parameters be in a new config file? For example: DVRTaskConfig.txt ?

5. Do you think the implementation is going in a good direction or would it need reengineering? Perhaps in the future other types of file systems from other DVRs will be added ( or less common filesystems not identified by sleuthkit ) .

[patrickdalla]

Wouldn't it be more appropriate to implement this kind of FS as an DataSourceReader?

[lfcnassif]

Hi @gfd2020, sorry for my delay, currently I'm traveling on vacation...

1. Not sure if it is the best approach, maybe there is a better solution...
2. A while ago, I thought about creating another abstract method in AbstractTask returning a boolean to tell if the task would process ordinary items/files or raw disks/partitions/FileSystems. If true, the the raw content of disks, partitions or file systems would be available to be processed by the task. This could be useful to this feature and to other future ones. Some changes in the code base would be needed to differentiate ordinary files from raw disks, partitions or FSes.
3. Unfortunatelly, today, it needs to be always enabled to avoid Aborting ArrayIndexOutOfBoundsException while indexing huge files #1281. A possible solution for you would be to skip fragments, there is a flag tagging them. AFAIK you can access the content of original non splitted data normally.
4. If the parameters are specific of this new Task, creating such config file seems fine to me.
5. I'm not sure since I didn't have time to look at the proposed code changes here yet, sorry... I'll try to take a look after I return from vacation. Ideally this kind of support would fit better if implemented directly into Sleuthkit project...

[lfcnassif]

Wouldn't it be more appropriate to implement this kind of FS as an DataSourceReader?

This makes sense. But we would lost the ability to read this file system raw data from E01, Ex01, VMDK, VHD, VHDX and other image disk formats, and also to decode MBR and GPT partitions transparently, since TSK decoding would be skipped... As I said, the perfect approach would be to implement this into TSK, but I think it would need much more effort.

[patrickdalla]

Today, what does happen if a Vhd, an Ufdr or and Dd image is found inside the original evidence passed to IPED? How their content are processed? Em ter., 12 de set. de 2023 22:51, Luis Filipe Nassif < ***@***.***> escreveu:

…

Wouldn't it be more appropriate to implement this kind of FS as an DataSourceReader? This makes sense. But we would lost the ability to read this file system raw data from E01, Ex01, VMDK, VHD, VHDX and other image disk formats, and also to decode MBR and GPT partitions transparently, since TSK decoding would be skipped... As I said, the perfect approach would be to implement this into TSK, but I think it would need much more effort. — Reply to this email directly, view it on GitHub <#1776 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG247S7RMIP3GSESTHE2JCTX2ENUJANCNFSM6AAAAAA2NM6MV4> . You are receiving this because you commented.Message ID: ***@***.***>

[lfcnassif]

Today, what does happen if a Vhd, an Ufdr or and Dd image is found inside the original evidence passed to IPED? How their content are processed?

They are passed to and decoded by SleuthkitReader through EmbeddedDiskProcessTask, except UFDR which is not processed recursively but as a common zip file.

[patrickdalla]

Couldn't these Hikvision partitions/disks be passed to be decoded by some HikvisionReader through the same EmbeddedDiskProcessTask?

[lfcnassif]

Couldn't these Hikvision partitions/disks be passed to be decoded by some HikvisionReader through the same EmbeddedDiskProcessTask?

They could. But not sure if a HikVisionReader would be used directly in practice (like SleuthkitReader), since I think a DVR FS usually is into a partition (could you confirm @gfd2020?) that needs to be decoded first by TSK, so I think this would create another step for most cases.

[gfd2020]

They could. But not sure if a HikVisionReader would be used directly in practice (like SleuthkitReader), since I think a DVR FS usually is into a partition (could you confirm @gfd2020?) that needs to be decoded first by TSK, so I think this would create another step for most cases.

In the real cases I have, there is no disk initialization. The first sector of the disk is all zero (that is, without a partition system). The HikivisionFS starts in the second sector. Would this help with Patrick's suggestion?

[patrickdalla]

So, neither SleuthKit would show this as an item to be processed, and it seems that this "item" to be processed should first be carved from the unallocated space (as it would be identified by SleuthKit), right?

[gfd2020]

So, neither SleuthKit would show this as an item to be processed, and it seems that this "item" to be processed should first be carved from the unallocated space (as it would be identified by SleuthKit), right?

Sleuthkit will complain that it does not recognize this item. So, Nassif gave me the tip to change a flag in the SleuthkitInputStreamFactory file to continue decoding the image bytes. From then on, as far as I remember, IPED fragmented everything and looked for carved files.

IPED 4.1.4 log . Defaut config. Result:

Decoding image C:\HVFS\test.E01
org.sleuthkit.datamodel.TskCoreException: Errors occurred while ingesting image

1. Cannot determine file system type (Sector offset: 0)

   at org.sleuthkit.datamodel.SleuthkitJNI.runAddImgNat(Native Method)
   at org.sleuthkit.datamodel.SleuthkitJNI$CaseDbHandle$AddImageProcess.run(SleuthkitJNI.java:584)
   at org.sleuthkit.datamodel.SleuthkitJNI$CaseDbHandle$AddImageProcess.run(SleuthkitJNI.java:544)
   at iped.engine.datasource.SleuthkitReader.addImageBlocking(SleuthkitReader.java:600)
   at iped.engine.datasource.SleuthkitReader$1.call(SleuthkitReader.java:653)
   at iped.engine.datasource.SleuthkitReader$1.call(SleuthkitReader.java:650)
   at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.base/java.lang.Thread.run(Unknown Source)

[lfcnassif]

Would this help with Patrick's suggestion?

Yes, thanks!

it seems that this "item" to be processed should first be carved from the unallocated space (as it would be identified by SleuthKit), right?

I think so. Not sure, but I think @gfd2020 draft is reading data from the parent image.

So it seems possible to create a HikVisionReader to decode DD images. For all others, like e(x)01, vmdk and vhd(x), they would need to pass through SleuthkitReader first. Honestly, I'm not sure which approach is better, creating a DatasourceReader or a Task, I think both would work, but the first would need an additional change in EmbeddedDiskProcessTask.

[paulobreim]

I have some documentation of the format of this HD.
It could be treated like any other OS using specific development for this, since as far as I researched, there is no open software on the market.
The Hikvision Format has a formatting identification header, has some areas with information on the date format, HD size, an event log area and video addressing.
There is some software on the market that extracts videos, such as diskinternal. I looked at several and they all provided information a little differently from each other, and none of them extract the event LOG. The biggest discrepancy I found between the software I tested was the date and time of recording the videos. But the HD I had to work on had been formatted and perhaps this had affected it in some way.
I made a program in C that lists the event logs, which is a very important part of the inspection, and it worked perfectly, including showing logs that the hikvision software itself did not show.
Looking at the documentation I located the point where the logs start and end, but for testing purposes, I looked at the entire HD and found more logs.

[gfd2020]

I have some documentation of the format of this HD. It could be treated like any other OS using specific development for this, since as far as I researched, there is no open software on the market. The Hikvision Format has a formatting identification header, has some areas with information on the date format, HD size, an event log area and video addressing. There is some software on the market that extracts videos, such as diskinternal. I looked at several and they all provided information a little differently from each other, and none of them extract the event LOG. The biggest discrepancy I found between the software I tested was the date and time of recording the videos. But the HD I had to work on had been formatted and perhaps this had affected it in some way. I made a program in C that lists the event logs, which is a very important part of the inspection, and it worked perfectly, including showing logs that the hikvision software itself did not show. Looking at the documentation I located the point where the logs start and end, but for testing purposes, I looked at the entire HD and found more logs.

Hi @paulobreim . I will make a change to the PR now for greater compatibility. Could you please try to improve log detection?

[gfd2020]

Would this help with Patrick's suggestion?

Yes, thanks!

it seems that this "item" to be processed should first be carved from the unallocated space (as it would be identified by SleuthKit), right?

I think so. Not sure, but I think @gfd2020 draft is reading data from the parent image.

So it seems possible to create a HikVisionReader to decode DD images. For all others, like e(x)01, vmdk and vhd(x), they would need to pass through SleuthkitReader first. Honestly, I'm not sure which approach is better, creating a DatasourceReader or a Task, I think both would work, but the first would need an additional change in EmbeddedDiskProcessTask.

After a little more work, I managed to get it working without changing any configuration parameters. In fact, you don't need to touch any of them. Basically I created a new constructor for the SleuthkitInputStreamFactory class to be able to use the decoded content of e01 (no external variable will be needed anymore). The main change that could impact the program in general is in the SleuthkitReader class. From what I tested, IPED worked normally on other types of images. However, we need to do regression tests. I'm finishing the changes and will commit. @lfcnassif , could you take a look when you get back from vacation? Thanks.

[lfcnassif]

Hi @gfd2020. Hashes and video parsing (at processing time) should work correctly with that approach. To make HexViewer and parsing at analysis time work correctly, you should also set the idInDataSource property in the Item class. It should be an unique ID identifying the video into the FS and should contain enough information (or point to it) to ISeekableInputStreamFactory be able to reconstruct the video stream using the idInDataSource as a parameter. AFAIK it is a String and you could use it to save the data runs/blocks directly, but it would be more elegant to be an ID pointing to the data blocks info stored somewhere else (index file, item extraAttribute) or an ID enough to rebuild the data blocks info on demand by the ISeekableStreamFactory factory at analysis time (like the AD1 decoder).

PS: it shouldn't, but maybe some change in ItemIndex class would be needed, it is used to convert the Item to an index document and vice-versa, this last one might need some adjustment.

[gfd2020]

Hi @lfcnassif . Thanks for the info.
Actually I didn't create a new datasource because I'm using the inputstream that sleuthkit provides me. So in this case I had to make a workaround here for it to work. I modified SleuthkitInputStreamFactory to read videos in a special way. I have to see an elegant way to do this...
But it worked very well, both for processing and analyzing in iped-app (hexvier).

[gfd2020]

I will add two FS's to this PR. WFS and DHFS. This PR will return to DRAFT until I make all 3 types. From my experience here, these 3 types are widely used in DVRS used in Brazil.