Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to do customized feeds lookups #98

Closed
mlodic opened this issue Dec 28, 2022 · 2 comments
Closed

Allow to do customized feeds lookups #98

mlodic opened this issue Dec 28, 2022 · 2 comments
Assignees

Comments

@mlodic
Copy link
Member

mlodic commented Dec 28, 2022

We could add more ways to extract data feeds from GB other than "recent" and "persistent" which are free.

These new ways must be protected with authentication to avoid abuse.

We could give the users the chance to:

  • download the data extracted in the last X hours (customization of "recent")
  • download the data that was seen more than X times in the last X days (customization of "persistent")
@mlodic mlodic changed the title Allow to do customized lookups Allow to do customized feeds lookups Dec 28, 2022
@regulartim
Copy link
Collaborator

What kind of abuse do you want to prevent? Someone downloading a huge number of IOCs?

@regulartim regulartim self-assigned this Jan 2, 2025
@mlodic
Copy link
Member Author

mlodic commented Jan 2, 2025

While "recent" and "persistent" are fixed sets and, as such, they are easily cachable, allowing to do "customized" lookups could be intensive if the queries are not efficient, extract too much data and so on.

My primary concern is to avoid having to manage server issues for the Honeynet instance. I understand that in different circumstances that could not be important so I think that the authentication could be optional.

@mlodic mlodic closed this as completed Jan 2, 2025
@mlodic mlodic reopened this Jan 2, 2025
mlodic added a commit that referenced this issue Feb 7, 2025
* add new feeds API

* adapt tests to changed serializer

* switch to Token Authentication to match other views

* remove unused import

* add destination_port_count to serializer

* update documentation

* add validation for feed requests

* add api tests

* add serializer tests

* fix serializer

* store feed request parameters as strings for easier validation and adapt tests accordingly

* store request parameters in FeedRequestParams object in old feeds APIs and remove unused code

* correct age conversion

* pass FeedRequestParams instance to feeds_response function

* rename new API endpoint to "advanced"

* split views in separate files

* force json format for paginated responses

* add option to exclude mass scanners to old APIs

* fix ordering_validation function and add docstrings to both helper functions for field-level validation

* add docstring to get_valid_feed_types()

* add support for sorting by feed_type in FeedRequestParams class

* multiple performance optimizations
- remove logging the number of returned IOCs in get_queryset() as this forces early QuerySet evaluation
- only pass filter for number_of_days_seen if it is > 1
- build list of general honeypots, that have seen an IOC, directly on the database
- rewrite feeds_response() function

* remove SKIP_FEED_VALIDATION setting as this can now be done by a query parameter

* change the way data is excluded from inactive honeypots (see #429)

* adjusted log

---------

Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants