Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc(gcplog): Advanced log export filter example #3421

Merged
merged 2 commits into from
Mar 3, 2021
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 38 additions & 2 deletions docs/sources/clients/promtail/gcplog-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,19 +31,21 @@ $ gcloud pubsub topics create cloud-logs
We create a log sink to forward cloud logs into pubsub topic created before

```bash
$ gcloud beta logging sinks create $SINK_NAME $SINK_LOCATION $OPTIONAL_FLAGS
$ gcloud logging sinks create $SINK_NAME $SINK_LOCATION $OPTIONAL_FLAGS
```

e.g:
```bash
$ gcloud beta logging sinks create cloud-logs pubsub.googleapis.com/projects/my-project/topics/cloud-logs \
$ gcloud logging sinks create cloud-logs pubsub.googleapis.com/projects/my-project/topics/cloud-logs \
--log-filter='resource.type=("gcs_bucket")' \
--description="Cloud logs"
```

Above command also adds `log-filter` option which represents what type of logs should get into the destination `pubsub` topic.
For more information on adding `log-filter` refer this [document](https://cloud.google.com/logging/docs/export/configure_export_v2#creating_sink)

We cover more advanced `log-filter` [below](#Advanced-Log-filter)

## Create Pubsub subscription for Loki

We create subscription for the pubsub topic we create above and `promtail` uses this subscription to consume log messages.
Expand Down Expand Up @@ -85,3 +87,37 @@ To delete all the old messages until now, set `--time` to current time.
```bash
gcloud pubsub subscriptions seek projects/my-project/subscriptions/cloud-logs --time=$(date +%Y-%m-%dT%H:%M:%S)
```

# Advanced log filter

So far the document explains about adding just GCS bucket logs into Loki, But most often one may have to add multiple cloud resource logs and may also want to exclude some unnecessary logs.
Here you will find one such way to add complex logs export filter.

We use `log-filter` option to include logs and `exclusion` option to exclude specific logs.
kavirajk marked this conversation as resolved.
Show resolved Hide resolved

## Use Case
Include following cloud resource logs
- GCS bucket
- Kubernetes
- IAM
- HTTP Load balancer

And we exclude specific HTTP load balancer logs based on payload and status code.

```
$ gcloud logging sinks create cloud-logs pubsub.googleapis.com/projects/my-project/topics/cloud-logs \
--log-filter='resource.type=("gcs_bucket OR k8s_cluster OR service_account OR iam_role OR api OR audited_resource OR http_load_balancer")' \
--description="Cloud logs" \
--exclusion='name=http_load_balancer,filter=<<EOF
resource.type="http_load_balancer"
(
(
jsonPayload.statusDetails=("byte_range_caching" OR "websocket_closed")
)
OR
(
http_request.status=(101 OR 206)
)
)
EOF
```