-
Notifications
You must be signed in to change notification settings - Fork 646
Development Guide
Recon-ng makes it easy for even the newest of Python developers to contribute. Each module is a subclass of the Module class, a customized cmd interpreter with built-in interfaces for common tasks such as standardizing output, interfacing with the database, making web requests, and managing third party resource credentials. Therefore, all the hard work has been done. Building modules is simple and takes little more than a few minutes.
During module development, developers may need to repeatedly reload modules to test code changes without restarting the framework. The modules reload subcommand within the global context reloads all modules, and the reload command within the module context reloads the current module, all while maintaining command history and the current context.
Developers can safely develop new modules alongside installed modules by creating a uniquely named directory underneath the installed modules directory, i.e. "~/.recon-ng/modules/new/". Modules added to this directory are loaded into the framework at runtime and listed under a category matching the folder's name, i.e. "New". This also makes them available for indexing, which is required when submitting module code to the marketplace repository.
Before submitting a new module to the marketplace repository, the module must be indexed using the index command. The result of the index command must be merged into the "modules.yaml" index file in the same pull request used to merge the module itself into the repository.
When updating a module, the module's meta dictionary and index must be updated, in that order, as the index is created from the meta dictionary. In most cases, the only update necessary will be a bump of the version value, as the last_updated field is dynamically updated by the index command. The marketplace relies on the version value to notify users when an installed module is out-of-date, so it is imperative that this be done for each update.
Modules must be PEP8 compliant. A code checker such as pycodestyle can be used to analyze the Python code and respond on the modules PEP8 compliance. The checker should identify any changes that need to be made prior to submitting a pull request. It is imperative that the module be tested after making these changes to ensure that the style changes do not impact the functionality of the module. Below is a quick example of how to install and run pycodestyle (formerly pep8), a popular PEP8 code checker.
$ pip install pycodestyle
$ pycodestyle --show-source --show-pep8 /path/to/module.py
# module required for framework integration
from recon.core.module import BaseModule
# mixins for desired functionality
from recon.mixins.resolver import ResolverMixin
from recon.mixins.threads import ThreadingMixin
# module specific imports
import os
class Module(BaseModule, ResolverMixin, ThreadingMixin):
# modules are defined and configured by the "meta" class variable
# "meta" is a dictionary that contains information about the module, ranging from basic information, to input that affects how the module functions
# below is an example "meta" declaration that contains all of the possible definitions
meta = {
'name': 'Hostname Resolver',
'author': 'Tim Tomes (@lanmaster53)',
'version': '1.0',
'description': 'Resolves IP addresses to hosts and updates the database with the results.',
'dependencies': [],
'files': [],
'required_keys': ['webdns_api', 'webdns_secret'],
'comments': (
'Note: Nameserver must be in IP form.',
'\te.g. 1.2.3.4',
),
'query': 'SELECT DISTINCT host FROM hosts WHERE host IS NOT NULL',
'options': (
('nameserver', '8.8.8.8', 'yes', 'ip address of a valid nameserver'),
),
}
# "name", "author", "version", and "description" are required entries
# "dependencies" is required if the module requires the installation of a third party library (list of PyPI install names)
# "files" is required if the module includes a reference to a data file in the "/data" folder of the marketplace repository
# "required_keys" is required if the module leverages an API or builtin functionality that requires a key
# "query" is optional and determines the "default" source of input
# the "SOURCE" option is only available if "query" is defined
# "options" expects a tuple of tuples containing 4 elements:
# 1. the name of the option
# 2. the default value of the option (strings, integers and boolean values are allowed)
# 3. a boolean value (True or False) for whether or not the option is mandatory
# 4. a description of the option
# "comments" are completely optional
# optional method
def module_pre(self):
# override this method to execute code prior to calling the "module_run" method
# returned values are passed to the "module_run" method and must be captured in a parameter
return value
# mandatory method
# the second parameter is required to capture the result of the "SOURCE" option, which means that it is only required if "query" is defined within "meta"
# the third parameter is required if a value is returned from the "module_pre" method
def module_run(self, hosts, value):
# do something leveraging the api methods discussed below
# local option values can be accessed via self.options['name'] or self.options.get('name')
# key values can be accessed via self.keys['name'] or self.keys.get('name')
# use the "self.workspace" class property to access the workspace location
# threading can be used anywhere with the module through the usage of the "self.thread" api call
# the "self.thread" api call requires a "module_thread" method which acts as the worker for each item in a queue
# "self.thread" takes at least one argument
# the first argument must be an iterable that contains all of the items to fill the queue
# all other arguments get blindly passed to the "module_thread" method where they can be accessed at the thread level
self.thread(hosts, url, headers)
# optional method
# the first received parameter is required to capture an item from the queue
# all other parameters passed in to "self.thread" must be accounted for
def module_thread(self, host, url, headers):
# never catch KeyboardInterrupt exceptions in the "module_thread" method as threads don't see them
# do something leveraging the api methods discussed belowRecon-web exposes a web API for Recon-ng. After launching Recon-web, visit http://127.0.0.1:5000/api/ to access the API's documentation.
Recon-ng enables the consistent display of output to the user in multiple styles. The following methods can be called anywhere within a module to present a consistent interface to the user.
self.print_exception([line='<string>']):- Format and display exceptions.
self.error('<string>')- Format and display errors.
self.output('<string>')- Format and display normal output.
self.alert('<string>')- Format and display emphasized output.
self.verbose('<string>')- Format and display output if in verbose mode.
self.debug('<string>')- Format and display output if in debug mode.
self.heading('<string>'[, level=1])- Format and display a heading.
- level is the style of heading. Currently there are only two style options: 0 and 1.
self.table(tdata[, header=[]][, title=''])- Build, display, and store an ASCII table of given data.
- tdata is the table data as a two dimensional list (list of lists), with each list representing a row of data. Each list must be the same length.
- header (optional) is a list containing the header values. By default, tables are displayed without a header row.
- title (optional) is the title of the table. By default, tables are displayed without a title.
Examples:
self.error('This is an error.')
'[!] This is an error. # all red'
self.output('This is normal output.')
'[*] This is normal output. # blue highlights'
self.alert('This is important output!')
'[*] This is important output! # green highlights'
self.verbose('This is verbose output.')
'[*] This is verbose output. # blue highlights'
self.debug('This is debug output.')
'[*] This is debug output. # blue highlights'
self.heading('This is a level 0 heading', 0)
"""
-------------------------
THIS IS A LEVEL 0 HEADING
-------------------------
"""
self.heading('This is a level 1 heading')
"""
This Is A Level 1 Heading
-------------------------
"""
self.table(table_data, header=['host', 'ip_address'], title='Hosts')
"""
+-------------------------------------+
| Hosts |
+-------------------------------------+
| host | ip_address |
+-------------------------------------+
| www.apache.org | 192.87.106.229 |
| www.google.com | 74.125.225.176 |
| www.twitter.com | 199.59.148.10 |
| www.whitehouse.gov | 157.238.74.67 |
+-------------------------------------+
"""Interfacing with the database is easy in Recon-ng. There are core database interface methods which allow the addition of records to the default tables in the database. There is also a generic query method for all other queries. In each case, the work of setting up the connection, extracting the results, and closing the connection is done for you.
self.insert_hosts(host[, ip_address=None][, region=None][, country=None][, latitude=None][, longitude=None])- Insert a host to the database and return the affected row count.
- host is the FQDN of the host as a string.
- ip_address (optional) is the IP address of the host as a string.
- region (optional) is the city, state or region of where the host is located.
- country (optional) is the country of where the host is located.
- latitude (optional) is the latitude of where the host is located.
- longitude (optional) is the longitude of where the host is located.
The above is one example of a series of methods that exist to insert records to the default tables in the database. The methods are named with the following naming scheme: self.insert_<table>(<parameters>). Table names are available via the db schema command. See the source code for each of the methods in the "recon/core/framework.py" module for parameter details.
self.query(query[, values=()])- Query the database and return the results as a list (SELECT) or affected rowcount if no results are returned (INSERT, DELETE, UPDATE, etc.). A FrameworkException is raised if an error is encountered during the query process.
- query is the SQL statement to process.
- values (optional) is a tuple of values to use with the query string for parameter substitution (parameterized queries).
Some Recon-ng modules require the use of an API key, OAuth Token, etc. To prevent users from having to continually input keys and regenerate tokens, Recon-ng provides methods which assist in storing, managing and accessing these items.
The self.keys object is exposed in the module context for accessing the values of the keys defined in the required_keys item of the meta dictionary. Normally, this should be all that is needed. However, in rare cases developers may need interact with the keystore at a deeper level. The below methods are direct interfaces to the keystore for getting, adding and removing any key, not only those defined for a specific module. Use of these methods is not common.
self.get_key(name)- Fetch a key from the local key store.
- name is the unique name for the key. A FrameworkException is raised if no such key exists.
self.add_key(name, value)- Store a key in the local key store.
- name is the unique name for the key. If not unique, the existing key will be overwritten.
- value is the key string to store.
self.remove_key(name)- Remove a key from the local key store.
- name is the unique name for the key.
The most important capability of a tool built for web based reconnaissance is the ability to make web requests. Recon-ng relieves the burden of complicated request building logic by providing an interface for the most intuitive web request library available, Requests III: HTTP for Humans. The provided interface seamlessly applies all of the request configuration options built into the framework (timeout, user-agent, proxy, etc.). All of this is handled under the hood using the self.request helper method, which shares the same API as the requests.request method found here
The resulting response object after a successful request is an official Python Requests response object, which provides easy access to all of the information required for further action.
self.make_cookie(name, value, domain[, path='/'])- Build and return a Cookie object for use with a CookieJar object.
- name is the name of the cookie.
- value is the value of the cookie.
- domain is the domain to which the the cookie is bound.
- path (optional) is the path within the domain to which the cookie is bound.
Recon-ng offers class mixins to provide advanced functionality that would otherwise require additional dependencies and/or complex configuration.
Some web content is just too difficult to parse as a string. Therefore, Recon-ng includes the Mechanize python library to provide access to a powerful web content parsing engine.
from recon.mixins.browser import BrowserMixin
...
class Module(BaseModule, BrowserMixin):
...
br = self.get_browser()- Build and return a mechanize browser object configured with the framework's global options.
from recon.mixins.resolver import ResolverMixin
...
class Module(BaseModule, ResolverMixin):
...
br = self.get_resolver()- Build and return a dnspython default resolver object configured with the framework's global options.
Threading connections to internet resources drastically decreases time wasted due to network latency. Recon-ng speeds things up by providing developers with the ability to introduce threading into their modules.
from recon.mixins.threads import ThreadingMixin
...
class Module(BaseModule, ThreadingMixin):
...
def module_run(self, elements):
var1 = something
var2 = something_else
self.thread(elements, var1, var2)
# worker function
def module_thread(self, element, var1, var2):
...- Execute module functionality within threads.
Some Recon-ng modules may require the use of popular search engines and social media resources with complex authentication schemes. Recon-ng provides developers with an easy way to create OAuth tokens and interact with popular APIs for resources like Twitter, Google, Bing, Shodan and Github.
from recon.mixins.twitter import TwitterMixin
...
class Module(BaseModule, TwitterMixin):
...
meta = {
...
'required_keys': ['twitter_api', 'twitter_secret'],
...
}
...
self.search_twitter_api(payload)- Search Twitter using the Twitter API and return a list of the results.
- payload is the search string submitted to the search API.
from recon.mixins.search import ShodanAPIMixin
...
class Module(BaseModule, ShodanAPIMixin):
...
meta = {
...
'required_keys': ['shodan_api'],
...
}
...
self.search_shodan_api(query[, limit=0])- Search Shodan using the Shodan Search API and return a list of the results.
- query is the search string submitted to the search API.
- limit (optional) is the maximum number of results pages to return. A value of "0" returns all results.
from recon.mixins.search import BingAPIMixin
...
class Module(BaseModule, BingAPIMixin):
...
meta = {
...
'required_keys': ['bing_api'],
...
}
...
self.search_bing_api(query[, limit=0])- Search Bing using the Bing Azure Search API and return a list of the results.
- query is the search string submitted to the search API.
- limit (optional) is the maximum number of results pages to return. A value of "0" returns all results.
from recon.mixins.search import GoogleAPIMixin
...
class Module(BaseModule, GoogleAPIMixin):
...
meta = {
...
'required_keys': ['google_api', 'google_cse'],
...
}
...
self.search_google_api(query[, limit=0])- Search Google using the Google Custom Search Engine (CSE) API and return a list of the results.
- query is the search string submitted to the search API.
- limit (optional) is the maximum number of results pages to return. A value of "0" returns all results.
from recon.mixins.github import GithubMixin
...
class Module(BaseModule, GithubMixin):
...
meta = {
...
'required_keys': ['github_api'],
...
}
...
self.query_github_api(endpoint[, payload, options])- Query the Github API and return a list of results.
- endpoint is the API endpoint to query.
- payload (optional) is a dictionary representing the expected payload for the chosen endpoint.
- options (optional) is a dictionary of options available for the chosen endpoint.
from recon.mixins.github import GithubMixin
...
class Module(BaseModule, GithubMixin):
...
meta = {
...
'required_keys': ['github_api'],
...
}
...
self.search_github_api(query):- Query the Github API Search endpoint and return a list of results.
- query is the search string submitted to the search API.