Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aws Lambda + SQS #29

Closed
wants to merge 4 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 42 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ Currently available outputs are :
* **AlertManager**
* **Elasticsearch**
* **Influxdb**
* **AWS Lambda**
* **AWS SQS**

## Usage

Expand Down Expand Up @@ -59,45 +61,55 @@ Configuration is made by *file (yaml)* and *env vars*, both can be used but *env
See **config_example.yaml** :

```yaml
#listenport: 2801 #port to listen for daemon (default: 2801)
debug: false #if true all outputs will print in stdout the payload they send (default: false)
customfields: #custom fields are added to falco events
#listenport: 2801 # port to listen for daemon (default: 2801)
debug: false # if true all outputs will print in stdout the payload they send (default: false)
customfields: # custom fields are added to falco events
Akey: "AValue"
Bkey: "BValue"
Ckey: "CValue"

slack:
webhookurl: "" #Slack WebhookURL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not empty, Slack output is enabled
#footer: "" #Slack footer
#icon: "" #Slack icon (avatar)
outputformat: "text" #all (default), text, fields
minimumpriority: "debug" #minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
webhookurl: "" # Slack WebhookURL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not empty, Slack output is enabled
#footer: "" # Slack footer
#icon: "" # Slack icon (avatar)
outputformat: "text" # all (default), text, fields
minimumpriority: "debug" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)

datadog:
#apikey: "" #Datadog API Key, if not empty, Datadog output is enabled
# minimumpriority: "" #minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
#apikey: "" # Datadog API Key, if not empty, Datadog output is enabled
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)

alertmanager:
# hostport: "" # http://{domain or ip}:{port}, if not empty, Alertmanager output is enabled
# minimumpriority: "" #minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)

elasticsearch:
# hostport: "" # http://{domain or ip}:{port}, if not empty, Elasticsearch output is enabled
# index: "falco" # index (default: falco)
# type: "event"
# minimumpriority: "" #minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
# suffix: "daily" #date suffix for index rotation : daily (default), monthly, annually, none

# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
# suffix: "daily" # date suffix for index rotation : daily (default), monthly, annually, none

influxdb:
# hostport: "" # http://{domain or ip}:{port}, if not empty, Influxdb output is enabled
# database: "falco" # Influxdb database (default: falco)
# user: "" # user to use if auth is enabled in Influxdb
# password: "" # pasword to use if auth is enabled in Influxdb
# minimumpriority: "" #minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)

aws:
# accesskeyid: "" # aws access key
# secretaccesskey: "" # aws secret access key
# region : "" #aws region
lambda:
# functionname : "" # Lambda function name
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
sqs:
# url : "" # SQS Queue URL
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
```

Usage :
Usage :

```bash
usage: falcosidekick [<flags>]
Expand Down Expand Up @@ -135,6 +147,13 @@ The *env vars* "match" field names in *yaml file with this structure (**take car
* **INFLUXDB_USER** : user to use if auth is enabled in Influxdb
* **INFLUXDB_PASSWORD** : user to use if auth is enabled in Influxdb
* **INFLUXDB_MINIMUMPRIORITY** : minimum priority of event for using this output, order is `emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)`
* **AWS_ACCESSKEYID** : AWS Access Key Id, mandatory to enable different AWS outputs
* **AWS_SECRETACCESSKEY** : AWS Secret Access Key, mandatory to enable different AWS outputs
* **AWS_REGION** : AWS Region, mandatory to enable different AWS outputs
* **AWS_LAMBDA_FUNCTIONNAME** : AWS Lambda Function Name
* **AWS_LAMBDA_MINIMUMPRIORITY** : minimum priority of event for using this output, order is `emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)`
* **AWS_SQS_URL** : AWS SQS Queue URL
* **AWS_SQS_MINIMUMPRIORITY** : minimum priority of event for using this output, order is `emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)`

## Handlers

Expand All @@ -143,7 +162,7 @@ Different URI (handlers) are available :
* `/` : main and default handler, your falco config must be configured to use it
* `/ping` : you will get a `pong` as answer, useful to test if falcosidekick is running and its port is opened (for healthcheck purpose for example)
* `/test` : (for debug only) send a test event to all enabled outputs.
* `/debug/vars` : get statistics from daemon (in JSON format), it uses classic `expvar` package and some custom values are added
* `/debug/vars` : get statistics from daemon (in JSON format), it uses classic `expvar` package and some custom values are added

## Logs

Expand Down Expand Up @@ -192,7 +211,7 @@ You should get :

### Influxdb

```
```bash
> use falco
Using database falco
> show series
Expand All @@ -208,6 +227,10 @@ time akey bkey ckey priority rule value
1560441359119741800 A_Value B_Value C_Value Debug Test_rule This is a test from falcosidekick
```

### AWS SQS

![aws sqs example](https://github.com/Issif/falcosidekick/raw/master/imgs/aws_sqs.png)

## Development

### Build
Expand All @@ -224,4 +247,4 @@ go test ./outputs -count=1 -cover -v

## Author

Thomas Labarussias (https://github.com/Issif)
Thomas Labarussias (https://github.com/Issif)
11 changes: 10 additions & 1 deletion config.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,15 @@ func getConfig() *types.Configuration {
v.SetDefault("Influxdb.User", "")
v.SetDefault("Influxdb.Password", "")
v.SetDefault("Influxdb.MinimumPriority", "")
v.SetDefault("AWS.AccessKeyID", "")
v.SetDefault("AWS.SecretAccessKey", "")
v.SetDefault("AWS.Region", "")
v.SetDefault("AWS.Lambda.FunctionName", "")
v.SetDefault("AWS.Lambda.InvocationType", "RequestResponse")
v.SetDefault("AWS.Lambda.Logtype", "Tail")
v.SetDefault("AWS.Lambda.MinimumPriority", "")
v.SetDefault("AWS.SQS.URL", "")
v.SetDefault("AWS.SQS.MinimumPriority", "")
v.SetDefault("Customfields", map[string]string{})

v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
Expand All @@ -55,7 +64,7 @@ func getConfig() *types.Configuration {
v.AddConfigPath(d)
err := v.ReadInConfig()
if err != nil {
log.Printf("Error when reading config file: %v\n", err)
log.Printf("[ERROR] : Error when reading config file : %v\n", err)
}
}
v.GetStringMapString("customfields")
Expand Down
39 changes: 25 additions & 14 deletions config_example.yaml
Original file line number Diff line number Diff line change
@@ -1,35 +1,46 @@
#listenport: 2801 #port to listen for daemon (default: 2801)
debug: false #if true all outputs will print in stdout the payload they send (default: false)
customfields: #custom fields are added to falco events
#listenport: 2801 # port to listen for daemon (default: 2801)
debug: false # if true all outputs will print in stdout the payload they send (default: false)
customfields: # custom fields are added to falco events
Akey: "AValue"
Bkey: "BValue"
Ckey: "CValue"

slack:
webhookurl: "" #Slack WebhookURL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not empty, Slack output is enabled
#footer: "" #Slack footer
#icon: "" #Slack icon (avatar)
outputformat: "text" #all (default), text, fields
minimumpriority: "debug" #minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
webhookurl: "" # Slack WebhookURL (ex: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ), if not empty, Slack output is enabled
#footer: "" # Slack footer
#icon: "" # Slack icon (avatar)
outputformat: "text" # all (default), text, fields
minimumpriority: "debug" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)

datadog:
#apikey: "" #Datadog API Key, if not empty, Datadog output is enabled
# minimumpriority: "" #minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
#apikey: "" # Datadog API Key, if not empty, Datadog output is enabled
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)

alertmanager:
# hostport: "" # http://{domain or ip}:{port}, if not empty, Alertmanager output is enabled
# minimumpriority: "" #minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)

elasticsearch:
# hostport: "" # http://{domain or ip}:{port}, if not empty, Elasticsearch output is enabled
# index: "falco" # index (default: falco)
# type: "event"
# minimumpriority: "" #minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
# suffix: "daily" #date suffix for index rotation : daily (default), monthly, annually, none
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
# suffix: "daily" # date suffix for index rotation : daily (default), monthly, annually, none

influxdb:
# hostport: "" # http://{domain or ip}:{port}, if not empty, Influxdb output is enabled
# database: "falco" # Influxdb database (default: falco)
# user: "" # user to use if auth is enabled in Influxdb
# password: "" # pasword to use if auth is enabled in Influxdb
# minimumpriority: "" #minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)

aws:
# accesskeyid: "" # aws access key
# secretaccesskey: "" # aws secret access key
# region : "" # aws region
lambda:
# functionname : "" # Lambda function name
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
sqs:
# url : "" # SQS Queue URL
# minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informationnal|debug or "" (default)
20 changes: 20 additions & 0 deletions deploy/helm/falcosidekick/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,26 @@ spec:
- name: INFLUXDB_PASSWORD
value: {{ .Values.config.influxdb.password | quote }}
{{- end }}
{{- if and .Values.config.aws.accesskeyid .Values.config.influxdb.secretaccesskey .Values.config.aws.region }}
- name: AWS_ACCESSKEYID
value: {{ .Values.config.aws.accesskeyid | quote }}
- name: AWS_SECRETACCESSKEY
value: {{ .Values.config.aws.secretaccesskey | quote }}
- name: AWS_REGION
value: {{ .Values.config.aws.region | quote }}
{{- if .Values.config.aws.lambda.functionname }}
- name: AWS_LAMBDA_FUNCTIONNAME
value: {{ .Values.config.aws.lambda.functionname | quote }}
- name: AWS_LAMBDA_MINIMUMPRIORITY
value: {{ .Values.config.aws.lambda.minimumpriority | quote }}
{{- end }}
{{- if .Values.config.aws.sqs.url }}
- name: AWS_SQS_URL
value: {{ .Values.config.aws.sqs.functionname | quote }}
- name: AWS_SQS_MINIMUMPRIORITY
value: {{ .Values.config.aws.sqs.minimumpriority | quote }}
{{- end }}
{{- end }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
Expand Down
4 changes: 4 additions & 0 deletions deploy/helm/falcosidekick/templates/secrets.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,7 @@ data:
influxdb-user: "{{ .Values.config.influxdb.user | b64enc }}"
influxdb-password: "{{ .Values.config.influxdb.password | b64enc }}"
{{- end }}
{{- if and .Values.config.aws.accesskeyid .Values.config.aws.secretaccesskey }}
aws-accesskeyid: "{{ .Values.config.aws.accesskeyid | b64enc }}"
aws-secretaccesskey: "{{ .Values.config.aws.secretaccesskey | b64enc }}"
{{- end }}
11 changes: 11 additions & 0 deletions deploy/helm/falcosidekick/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,17 @@ config:
password: ""
minimumpriority: ""

aws:
accesskeyid: ""
secretaccesskey: ""
region : ""
lambda:
functionname : ""
minimumpriority: ""
sqs:
url : ""
minimumpriority: ""

service:
type: ClusterIP
port: 2801
Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ go 1.12
require (
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc // indirect
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf // indirect
github.com/aws/aws-sdk-go v1.22.0
github.com/spf13/viper v1.3.2
gopkg.in/alecthomas/kingpin.v2 v2.2.6
)
4 changes: 4 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuy
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf h1:qet1QNfXsQxTZqLG4oE62mJzwPIB8+Tee4RNCL9ulrY=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
github.com/aws/aws-sdk-go v1.22.0 h1:e88V6+dSEyBibUy0ekOydtTfNWzqG3hrtCR8SF6UqqY=
github.com/aws/aws-sdk-go v1.22.0/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
Expand All @@ -11,6 +13,8 @@ github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/magiconair/properties v1.8.0 h1:LLgXmsheXeRoUOBOjtwPQCWIYqM/LU1ayDtDePerRcY=
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
Expand Down
11 changes: 9 additions & 2 deletions handlers.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,10 @@ package main

import (
"bytes"
"strconv"
"encoding/json"
"log"
"net/http"
// "strconv"
"strconv"
"strings"
"time"

Expand Down Expand Up @@ -81,6 +80,14 @@ func mainHandler(w http.ResponseWriter, r *http.Request) {
if config.Influxdb.HostPort != "" && (priorityMap[strings.ToLower(falcopayload.Priority)] >= priorityMap[strings.ToLower(config.Influxdb.MinimumPriority)] || falcopayload.Rule == "Test rule") {
go influxdbClient.InfluxdbPost(falcopayload)
}
if config.AWS.AccessKeyID != "" && config.AWS.SecretAccessKey != "" && config.AWS.Region != "" {
if config.AWS.Lambda.FunctionName != "" && (priorityMap[strings.ToLower(falcopayload.Priority)] >= priorityMap[strings.ToLower(config.AWS.Lambda.MinimumPriority)] || falcopayload.Rule == "Test rule") {
go awsClient.InvokeLambda(falcopayload)
}
if config.AWS.SQS.URL != "" && (priorityMap[strings.ToLower(falcopayload.Priority)] >= priorityMap[strings.ToLower(config.AWS.SQS.MinimumPriority)] || falcopayload.Rule == "Test rule") {
go awsClient.SendMessage(falcopayload)
}
}
}

// pingHandler is a simple handler to test if daemon is UP.
Expand Down
Binary file added imgs/aws_sqs.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 17 additions & 1 deletion main.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
)

// Globale variables
var slackClient, datadogClient, alertmanagerClient, elasticsearchClient, influxdbClient *outputs.Client
var slackClient, datadogClient, alertmanagerClient, elasticsearchClient, influxdbClient, awsClient *outputs.Client
var config *types.Configuration
var stats *types.Statistics

Expand Down Expand Up @@ -68,6 +68,22 @@ func init() {
enabledOutputsText += "Influxdb "
}
}
if config.AWS.AccessKeyID != "" && config.AWS.SecretAccessKey != "" && config.AWS.Region != "" {
var err error
awsClient, err = outputs.NewAWSClient("AWS", config, stats)
if err != nil {
config.AWS.AccessKeyID = ""
config.AWS.SecretAccessKey = ""
config.AWS.Region = ""
} else {
if config.AWS.Lambda.FunctionName != "" {
enabledOutputsText += "AWSLambda "
}
if config.AWS.SQS.URL != "" {
enabledOutputsText += "AWSSQS "
}
}
}

log.Printf("%v\n", enabledOutputsText)
}
Expand Down
Loading