Merge pull request ace-ecosystem#1 from seanmcfeely/mcfeelydev

Merging mcfeelydev into master

.gitignore

@@ -123,6 +123,10 @@ data/
 extra/
 bro/ace_local.bro
 
+# site modules
+/lib/saq/modules/site
+/lib/saq/collectors/site
+
 # SSL
 !ssl/root/ca/openssl.cnf
 !ssl/root/ca/intermediate/openssl.cnf
@@ -147,3 +151,11 @@ bin/procyon-decompiler*
 
 # unit testing
 data_unittest
+
+# update installation tracking
+updates/**/*.installed
+updates/**/*.stderr
+updates/**/*.stdout
+
+# proprietary libaries
+lib/SoleraConnector.py

README.CONFIG

@@ -0,0 +1,29 @@
+ACE Configuration README
+========================
+
+ACE configuration settings are stored in multiple ini-format files in the
+SAQ_HOME/etc directory. The files are loaded in a certian order (detailed
+below). Each time another configuration file is loaded, any settings it defines
+overrides the settings defined in previous files.
+
+Load Order (Normal)
+===================
+
+1) etc/saq.default.ini
+2) etc/saq.integrations.default.ini
+3) etc/saq.integrations.ini (if it exists)
+4) default integration files as configured in etc/saq.integrations.ini
+   typically these are saq.INTEGRATION_NAME.default.ini where INTEGRATION_NAME
+   is the name of the integration
+5) local integration files as configured in etc/saq.integrations.ini (if they exist)
+   typically these are saq.INTEGRATION_NAME..ini where INTEGRATION_NAME is the
+   name of the integration
+6) configuration files specified on the command line
+7) configuration files specified in the SAQ_CONFIG_PATHS environment variable
+8) etc/saq.ini
+
+Load Order (Unit Testing)
+=========================
+1) etc/saq.default.ini
+2) etc/saq.unittest.default.ini
+3) etc/saq.unittest.ini

README.CRYPTO

@@ -0,0 +1,48 @@
+ACE Cryptography README
+=======================
+ACE uses AES-256 to encrypt the following sensitive information:
+* archived emails
+* passwords to connect to other systems
+* API keys
+
+Definitions
+===========
+* USER PASSWORD: The password supplied by the user. This password is used when
+  starting ACE.
+* USER AES KEY: The 32 byte key used to encrypt and decrypt the Primary AES 
+  Key.
+* VERIFICATION KEY: The 32 byte value used to check the validity of the
+  provided password.
+* PRIMARY AES KEY: The 32 byte key to by ACE to encrypt and decrypt data.
+
+Setting the Encryption Password
+===============================
+1) The user supplies the USER PASSWORD.
+2) ACE randomly generates the PRIMARY AES KEY.
+3) ACE uses the USER PASSWORD as input into the PBKDF2 key derivation function
+to generate a 64 byte value. The first 32 bytes are the USER AES KEY which is
+used to encrypt the key generated in step 2. The second 32 bytes is the
+VERIFICATION KEY which is used to verify the user supplied password.
+4) The USER AES KEY is used to encrypt the PRIMARY AES KEY.
+5) The results are stored in the directory specified in the [encryption]
+section in the configuration file in the following files.
+- key
+- salt
+- iterations
+- verification 
+
+Loading and Using the Encryption Password
+=========================================
+1) The user provides the USER PASSWORD in one of the following two ways.
+   a) Using the -p option for the main ace command.
+   b) Using the Encryption Cache Service.
+2) ACE computes the USER AES KEY and VERIFICATION KEY using the supplied
+password.
+3) ACE decrypts the PRIMARY AES KEY and makes it available globally as
+
+saq.ENCRYPTION_PASSWORD
+
+NOTES
+=====
+You can set the PRIMARY AES KEY to the sha256 hash of a password by using the
+-k option of the ace set-encryption-password command.