ALeapp task #2095

patrickdalla · 2024-02-22T19:45:07Z

Hi @lfcnassif , I decided to create this pull request, but marked it as draft, as I am leaving on vacation

that does informs the non existence of registered artifacts in script, meaning it won't be treated as a plugin. Other exceptions are rethrown.

static variable, multiple plugins concurrent execution were scrambling device info between their specific file Output. As the necessary result is the merging of those all info, the concurrency is not a problem, being it done at the end of all individual plugins processing (4th queue).

allowing replacement of methods to redirect HTML table lines insertion to IPED classes, as long as logfunc and logdevinfo calls.

ALeapp, making it possible to override python class. It skips any keyword arguments passed to any class method.

report file, this means that this file is a genereted detailed content for this item. So, it is exported as the content of the artifact item.

exported file, saves it as a link and return, avoiding duplicate metadatas.

was leading to exception as fileLoader does not work with these files inside zip. So, for now, as ALeapp will be deployed unzipped, ignores the override of any files that are inside ZIP libs.

children), as the Leapp Report.

with double backslash, as it backslash is a escape char in python.

ilapfuncs that depends of this name.

not changed.

integrated with IPED timeline.

names.

referenced media file in html report. Inside IPED, this can just add a link to the media (TO BE IMPLEMENTED).

debugging compatibility).

method declaration. If so, method name is not declared, so a direct access to this PyCallable should be done.

entries.

misses patterns with * at the end name.

applying pattern.

first is a mount point to the second, so the link can be correctly done.

file in case output path.

patrickdalla · 2024-05-20T10:21:53Z

I've made some more tests and pontual enhancements and corrections. I think I should some third developer evaluation/opinion before continuing, @lfcnassif. I fact, if the design is good and no errors found, I think this can be merged as is.

lfcnassif · 2024-05-21T01:01:11Z

Thank you very much @patrickdalla! This is a very important feature, I'll try to test it after other ready PRs scheduled for 4.2 in the queue.

One question: if an UFDR is processed with this PR, will we get duplicated results coming from PA decoding and from ALeapp decoding? If yes, I think it should be avoided or be configurable. Maybe with a similar approach used for WhatsApp today, or maybe with the approach that would be implemented for #2012.

patrickdalla · 2024-05-21T02:22:24Z

Aleapp processes data from FFS. It expects data files in its original path, not in the way they are in UFDR. Em seg., 20 de mai. de 2024, 21:01, Luis Filipe Nassif < ***@***.***> escreveu:

…

Thank you very much @patrickdalla <https://github.com/patrickdalla>! This is a very important feature, I'll try to test it after other ready PRs scheduled for 4.2 in the queue. One question: if an UFDR is processed with this PR, will we get duplicated results coming from PA decoding and from ALeapp decoding? If yes, I think it should be avoided or be configurable. Maybe with a similar approach used for WhatsApp today, or maybe with the approach that would be implemented on #2012 <#2012>. — Reply to this email directly, view it on GitHub <#2095 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG247S7KCJ75V56GGLYFXADZDKMGZAVCNFSM6AAAAABDVRWLH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRRGUYDIOJYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

patrickdalla · 2024-05-21T11:12:58Z

Anyway, there is already a config file to inform which Aleapp parser not to run. Em seg., 20 de mai. de 2024, 22:22, Patrick Bernardina < ***@***.***> escreveu:

…

Aleapp processes data from FFS. It expects data files in its original path, not in the way they are in UFDR. Em seg., 20 de mai. de 2024, 21:01, Luis Filipe Nassif < ***@***.***> escreveu: > Thank you very much @patrickdalla <https://github.com/patrickdalla>! > This is a very important feature, I'll try to test it after other ready PRs > scheduled for 4.2 in the queue. > > One question: if an UFDR is processed with this PR, will we get > duplicated results coming from PA decoding and from ALeapp decoding? If > yes, I think it should be avoided or be configurable. Maybe with a similar > approach used for WhatsApp today, or maybe with the approach that would be > implemented on #2012 <#2012>. > > — > Reply to this email directly, view it on GitHub > <#2095 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AG247S7KCJ75V56GGLYFXADZDKMGZAVCNFSM6AAAAABDVRWLH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRRGUYDIOJYG4> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

patrickdalla · 2024-05-21T11:15:51Z

Also, thinking twice, the task searches for items from already processed items and its paths. As IPED uses UFDR xml info to restore original path in cel FS, maybe, with some simple modification, Aleapp parser can find these items too. I will check it. Em seg., 20 de mai. de 2024, 22:22, Patrick Bernardina < ***@***.***> escreveu:

…

Aleapp processes data from FFS. It expects data files in its original path, not in the way they are in UFDR. Em seg., 20 de mai. de 2024, 21:01, Luis Filipe Nassif < ***@***.***> escreveu: > Thank you very much @patrickdalla <https://github.com/patrickdalla>! > This is a very important feature, I'll try to test it after other ready PRs > scheduled for 4.2 in the queue. > > One question: if an UFDR is processed with this PR, will we get > duplicated results coming from PA decoding and from ALeapp decoding? If > yes, I think it should be avoided or be configurable. Maybe with a similar > approach used for WhatsApp today, or maybe with the approach that would be > implemented on #2012 <#2012>. > > — > Reply to this email directly, view it on GitHub > <#2095 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AG247S7KCJ75V56GGLYFXADZDKMGZAVCNFSM6AAAAABDVRWLH2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRRGUYDIOJYG4> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

lfcnassif · 2024-05-21T11:55:13Z

Anyway, there is already a config file to inform which Aleapp parser not to run.

Great!

Also, thinking twice, the task searches for items from already processed
items and its paths. As IPED uses UFDR xml info to restore original path in
cel FS, maybe, with some simple modification, Aleapp parser can find these
items too. I will check it.

I think it can be useful, for example, for an application supported by ALeapp but not supported by PA, or if PA decoding brings incomplete or eventually wrong results. For this last example, disabling PA results importing per application may be needed, but that is related to #2012.

PS: Running ALeapp into AB backups (when #2079 is merged) should be very useful too.

plugins

patrickdalla · 2024-05-21T19:02:32Z

Yes, it worked with the modifications of last commit, ALeapp plugins found and processed items from UFDR.

prosch88 · 2024-06-10T17:50:48Z

I'm eager to see this merged into main. Is this bringing ileapp too? Or is this planned for another PR?

lfcnassif · 2024-06-10T19:32:34Z

I'm eager to see this merged into main. Is this bringing ileapp too? Or is this planned for another PR?

AFAIK this is just about ALeapp integration, iLeapp should be done later. Would you like to help testing? There is a snapshot with this support below, you should be logged in github to see the download link:
https://github.com/sepinf-inc/IPED/actions/runs/9180006157

lfcnassif · 2024-06-13T23:57:42Z

Hi @patrickdalla, an user/developer is trying to test this, but got an error "no module named geopy". What is the updated python dependency list needed to run this PR?

lfcnassif · 2024-06-14T00:05:16Z

Just found this list in Teams, posting here for those willing to help testing, let me know if it is outdated:

bcrypt==3.2.0
beautifulsoup4==4.8.2
bencoding
blackboxprotobuf
fitdecode==0.10.0
folium==0.14.0
geopy==2.3.0
packaging==20.1
pillow
polyline==2.0.0
protobuf==3.10.0
PyCryptodome
PySimpleGUI
pytz
simplekml
wheel
xlsxwriter==3.1.1
xmltodict
python-magic
libmagic
python-magic-bin
filetype

Just put those into a requirements.txt file and run from iped embedded python:
pip install -r requirements.txt

creates the dump as zip file.

patrickdalla added 30 commits October 6, 2023 14:49

Makes some internal functionality visible to Task implementations.

ae77041

'#43 Initial ALeapp/ILeapp integration classes.

8f325da

'#43 improves exception handling. Unique skipped exception is the one

cd8ede2

that does informs the non existence of registered artifacts in script, meaning it won't be treated as a plugin. Other exceptions are rethrown.

'#43 adds the device info to the 4th processing queue.

63d4b19

'#43 corrects loop on data fields

b2ff524

'#43 downloads ALEAPP into tools dir.

dfaf2bd

'#43 get ALeappPlugins from ALeappConfig.

aaf3064

'#43 implements Python hook to change ALeapp code before loading,

442cc73

allowing replacement of methods to redirect HTML table lines insertion to IPED classes, as long as logfunc and logdevinfo calls.

'#43 Repositions PythonHook class to iped-utils and adjust dependecies.

787c2a0

'#43 Implements method wrapsClass that inserts a decoupled Wrapper from

539f0bc

ALeapp, making it possible to override python class. It skips any keyword arguments passed to any class method.

'#43 If the artifact has a field that corresponds to a href to a temp

947fdb1

report file, this means that this file is a genereted detailed content for this item. So, it is exported as the content of the artifact item.

'#43 If the value of the html column is a path to the dump or to an

03af1ac

exported file, saves it as a link and return, avoiding duplicate metadatas.

'#43 Adds ALeapp device details html as sub type of text/html mimetype.

724850e

'#43 Removes dependency of aleapp from class PythonHook.

3dbd758

'#43 In the IPED embedded python some code were inside zip files. This

ce81e08

was leading to exception as fileLoader does not work with these files inside zip. So, for now, as ALeapp will be deployed unzipped, ignores the override of any files that are inside ZIP libs.

'#43 Removes redundant code.

ebbd15a

'#43 removes debugging error messages.

e0a0d64

'#43 Adds a distinct tree icon for report folders (decoded data that has

318fba3

children), as the Leapp Report.

'#43 Replaces backslash occurrences of path strings passed to python

fa1488c

with double backslash, as it backslash is a escape char in python.

'#43 Adjust temp report folder name, as there is a method inside

fefb2a7

ilapfuncs that depends of this name.

'#43 Makes aleappDir static as it is a config that is loaded once and is

c390424

not changed.

'#43 overrides timeline function to do nothing as the timeline will be

58218ec

integrated with IPED timeline.

'#43 Puts task report folder in upper tmp folder, to avoid too long path

e6af24f

names.

'#43 media_to_html override. ALeapp uses this method is used to show a

c9e9ea4

referenced media file in html report. Inside IPED, this can just add a link to the media (TO BE IMPLEMENTED).

'#43 adds suport for multiple parameter python function override.

6ad1a2e

'#43 Add support for folder class path search when debugging (Eclipse

5ead16f

debugging compatibility).

'#43 some plugins declares a lambda function directly on artifact parse

17eaec9

method declaration. If so, method name is not declared, so a direct access to this PyCallable should be done.

'#43 Implements a more detailed naming scheme for timestamp fields.

84883ab

'#43 uses package name to describe eventtype of FCMQueuedMessageDump

715d5d5

entries.

patrickdalla added 3 commits February 23, 2024 07:49

'#43 Separates logic to install overriden ilapfuncs methods

1e7f788

'#43 adds some comments.

ece1fb8

'#43 Corrects logic of PythonHook installation.

2038c02

wladimirleite added a commit that referenced this pull request Apr 28, 2024

'#2095: Create a bookmark for probably shared by Torrent.

2c5781b

patrickdalla added 12 commits May 8, 2024 11:20

'#2095 adds some georeferenced extracted items to Locations category.

106cc05

'#2095 modify pattern to lucene query transformation, so it doesn't

170bd00

misses patterns with * at the end name.

'#2095 Corrects pattern matching when patterns starts with *

7682505

'#2095 Classifications of items extracted from badoo and tiktok plugins.

0781105

'#2095 Improves method to return query to filter resultset before

df229a3

applying pattern.

'#2095 JSON Syntax error correction

4753a11

'#2095 Adds category to represent app notification messages.

35b7094

'#2095 Replaces the string /storage/emulated/0 with /data/media/0 as the

0c45481

first is a mount point to the second, so the link can be correctly done.

'#2095 parses some coords from google maps links.

5537ddb

'#2095 Correctly treats getTempFile if it is a reference to an exported

00d7586

file in case output path.

'#2095 Adds some methods to support future improvements.

b054ced

Merge remote-tracking branch 'origin/master' into ALeappTask

4229aca

'#2095 Modifications to recognize UFDR files as source for ALeapp

eeeac5f

plugins

lfcnassif added this to In progress in 4.3 via automation Jun 18, 2024

'#2095 includes all zip files as potential dump to check, as Graykey

3497efd

creates the dump as zip file.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ALeapp task #2095

ALeapp task #2095

patrickdalla commented Feb 22, 2024

patrickdalla commented May 20, 2024

lfcnassif commented May 21, 2024 •

edited

Loading

patrickdalla commented May 21, 2024 via email

patrickdalla commented May 21, 2024 via email

patrickdalla commented May 21, 2024 via email

lfcnassif commented May 21, 2024 •

edited

Loading

patrickdalla commented May 21, 2024

prosch88 commented Jun 10, 2024

lfcnassif commented Jun 10, 2024

lfcnassif commented Jun 13, 2024

lfcnassif commented Jun 14, 2024 •

edited

Loading

ALeapp task #2095

Are you sure you want to change the base?

ALeapp task #2095

Conversation

patrickdalla commented Feb 22, 2024

patrickdalla commented May 20, 2024

lfcnassif commented May 21, 2024 • edited Loading

patrickdalla commented May 21, 2024 via email

patrickdalla commented May 21, 2024 via email

patrickdalla commented May 21, 2024 via email

lfcnassif commented May 21, 2024 • edited Loading

patrickdalla commented May 21, 2024

prosch88 commented Jun 10, 2024

lfcnassif commented Jun 10, 2024

lfcnassif commented Jun 13, 2024

lfcnassif commented Jun 14, 2024 • edited Loading

lfcnassif commented May 21, 2024 •

edited

Loading

lfcnassif commented May 21, 2024 •

edited

Loading

lfcnassif commented Jun 14, 2024 •

edited

Loading