File me Issues or star this repo
See more challenges from Denny: #denny-challenges
It calls something like: https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/%s/pods/%s
The logic is like this:
kubernetes.c:cb_kube_filter -> kube_meta.c:flb_kube_meta_get/get_and_merge_meta -> kube_meta.c:get_api_server_info -> kube_meta.h:FLB_KUBE_API_FMT
https://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kube_meta.h#L54
https://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kube_meta.c#L590
https://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kube_meta.c#L138
https://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kubernetes.c#L416
- Get the token for your namespace account https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/
- Run curl test
# curl --header "Authorization: Bearer $TOKEN" -k https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/oratos/pods/dummy-input --insecure
curl --header "Authorization: Bearer $TOKEN" -k https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/oratos/pods/dummy-input --insecure
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "dummy-input",
"namespace": "oratos",
"selfLink": "/api/v1/namespaces/oratos/pods/dummy-input",
"uid": "f57ec8e3-9a87-11e8-8dcd-080027dbaae1",
"resourceVersion": "8496",
"creationTimestamp": "2018-08-07T21:22:16Z",
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"dummy-input\",\"namespace\":\"oratos\"},\"spec\":{\"containers\":[{\"args\":[\"tail\",\"-f\",\"/dev/termination-log\"],\"image\":\"busybox\",\"name\":\"dummy-input\"}]}}\n"
}
},
"spec": {
"volumes": [
{
"name": "default-token-xk9qq",
"secret": {
"secretName": "default-token-xk9qq",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "dummy-input",
"image": "busybox",
"args": [
"tail",
"-f",
"/dev/termination-log"
],
"resources": {
},
"volumeMounts": [
{
"name": "default-token-xk9qq",
"readOnly": true,
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "Always"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"serviceAccountName": "default",
"serviceAccount": "default",
"nodeName": "minikube",
"securityContext": {
},
"schedulerName": "default-scheduler",
"tolerations": [
{
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
},
{
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
}
]
},
"status": {
"phase": "Running",
"conditions": [
{
"type": "Initialized",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2018-08-07T21:22:16Z"
},
{
"type": "Ready",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2018-08-07T21:22:21Z"
},
{
"type": "PodScheduled",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2018-08-07T21:22:16Z"
}
],
"hostIP": "10.0.2.15",
"podIP": "172.17.0.7",
"startTime": "2018-08-07T21:22:16Z",
"containerStatuses": [
{
"name": "dummy-input",
"state": {
"running": {
"startedAt": "2018-08-07T21:22:20Z"
}
},
"lastState": {
},
"ready": true,
"restartCount": 0,
"image": "busybox:latest",
"imageID": "docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd",
"containerID": "docker://66f701a981bc2fa0db08fe9cdaf80468d2f7398c95db34e7502f839a909303d5"
}
],
"qosClass": "BestEffort"
}
}
key: log, value: key: time, value: 2018-08-8T18:16:26.000979098Z key: stream, value: stdout key: time, value: 2018-08-08T18:16.27.002369384Z key: kubernetes, value: map[pod_name:fluent-bit-n588c namespace_name: oratos pod_id:a987f335-9b36-11e8-9fa9-080027b477ce labels:map[controller-revision-hash:545891415 k8s-app:logging-agent kubernetes.io/cluster-service:true pod-template-generation:1 version:v1] host:minikube container_name:fluent-bit docker_id:5f5bcedc9a98d6c4705632cdb55d9bcb572b7fc80dbb1da3e440d092d56ea4f5]
2.1.3 For tail_file input plugin of fluent-bit, could I specify file path like: /var/log/*/apache/*.log?
It eventually use glob function of C. So probably, yes.
tail_scan.c:flb_tail_scan -> tail_scan.c:do_glob
https://github.com/fluent/fluent-bit/blob/master/plugins/in_tail/tail_scan.c#L150
https://github.com/fluent/fluent-bit/blob/master/plugins/in_tail/tail_scan.c#L202
It rescan the log folder every 10 seconds
DennyZhang [11:40 PM]
Hi XXX
May I ask one question about tail input plugin in fluent-bit
We’re using below input to parse k8s pod log files.
```[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10```
Whenever developers start or delete a pod, /var/log/containers will create or delete a log file.
*We are wondering what the behavior fluent-bit would be*
```1. Whether we need to restart or reload fluent-bit, if that happens
2. What's the latency when the log file are created or deleted, but fluent-bit haven't detected.```
Checking the code, here is my understanding. Would you help us to confirm?
```1. When tail input plugin loads, it will register a timer to rescan the log folder. (in_tail_init function in tail.c. https://github.com/fluent/fluent-bit/blob/d2d3d363c6f852f155d45566e1c4155024327913/plugins/in_tail/tail.c#L221-L225)
2. The rescan interval is every 60 seconds (https://github.com/fluent/fluent-bit/blob/d2d3d363c6f852f155d45566e1c4155024327913/plugins/in_tail/tail.h#L38)
3. Whenever developers create a Pod, the associate log file will be detected after 60 seconds. And no service restart is required
4. Whenever developers delete a Pod, file read will run into exception. Thus no more action will be required. (https://github.com/fluent/fluent-bit/blob/d2d3d363c6f852f155d45566e1c4155024327913/plugins/in_tail/tail_file.c#L658)```
XXX [7:58 PM]
Hi,
if I am not wrong when a Pod get's deleted, the log files for that container persists for a period of time. Now the re-scan of the path to find new logs happens by default every 60 seconds or based in the value of "Refresh_Interval" (which you have set to 10 seconds).
License: Code is licensed under MIT License.
