Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[telegraf] Error parsing response: parsing time "null" #9

Open
Cryptophobia opened this issue Mar 20, 2018 · 5 comments
Open

[telegraf] Error parsing response: parsing time "null" #9

Cryptophobia opened this issue Mar 20, 2018 · 5 comments
Labels

Comments

@Cryptophobia
Copy link
Member

From @zinuzoid on December 13, 2016 9:23

Metric in Grafana seems doesn't work, so I check telegraf and there're some error logs from pods.

I'm running cluster in GKE k8s version 1.4.6
Images: quay.io/deis/telegraf:v2.7.0

2016-12-13T09:01:07.656026655Z Node Name set (gke-deis-default-pool-6ba9e8cc-v3kd)
2016-12-13T09:01:07.824924771Z Node IP set (10.240.0.19)
2016-12-13T09:01:07.824989798Z Creating topic with URL: http://10.103.253.156:4151/topic/create?topic=metrics
2016-12-13T09:01:07.836194361Z Setting KUBERNETES_URL: http://10.240.0.19:10255
2016-12-13T09:01:07.836252337Z Building config.toml!
2016-12-13T09:01:07.849382935Z Finished building toml...
2016-12-13T09:01:07.849422582Z ###########################################
2016-12-13T09:01:07.849431084Z ###########################################
2016-12-13T09:01:07.850343415Z # Set Tag Configuration
2016-12-13T09:01:07.850365550Z [tags]
2016-12-13T09:01:07.850373997Z # Set Agent Configuration
2016-12-13T09:01:07.850381146Z [agent]
2016-12-13T09:01:07.850387919Z   interval = "10s"
2016-12-13T09:01:07.850394825Z   round_interval = true
2016-12-13T09:01:07.850400548Z   metric_buffer_limit = 100000
2016-12-13T09:01:07.850406425Z   collection_jitter = "1s"
2016-12-13T09:01:07.850412938Z   flush_interval = "1s"
2016-12-13T09:01:07.850417002Z   flush_jitter = "0s"
2016-12-13T09:01:07.850420935Z   debug = true
2016-12-13T09:01:07.850424675Z   quiet = false
2016-12-13T09:01:07.850428540Z   flush_buffer_when_full = true
2016-12-13T09:01:07.850433117Z   hostname = "gke-deis-default-pool-6ba9e8cc-v3kd" 
2016-12-13T09:01:07.850437102Z # Set output configuration
2016-12-13T09:01:07.850441009Z [[outputs.influxdb]]
2016-12-13T09:01:07.850444630Z   urls = ["http://10.103.253.43:80"]
2016-12-13T09:01:07.850448397Z   database = "kubernetes"
2016-12-13T09:01:07.850455322Z   precision = "ns"
2016-12-13T09:01:07.850459065Z   timeout = "5s"
2016-12-13T09:01:07.850462690Z   
2016-12-13T09:01:07.850466214Z   
2016-12-13T09:01:07.850469755Z   
2016-12-13T09:01:07.850473390Z   
2016-12-13T09:01:07.850476925Z # Set Input Configuration
2016-12-13T09:01:07.850480562Z [[inputs.netstat]]
2016-12-13T09:01:07.850484788Z [[inputs.swap]]
2016-12-13T09:01:07.850490907Z [[inputs.system]]
2016-12-13T09:01:07.850496625Z [[inputs.mem]]
2016-12-13T09:01:07.850502727Z [[inputs.cpu]]
2016-12-13T09:01:07.850508447Z   percpu = true
2016-12-13T09:01:07.850513963Z   totalcpu = true
2016-12-13T09:01:07.850519322Z [[inputs.disk]]
2016-12-13T09:01:07.850524783Z   
2016-12-13T09:01:07.850530315Z [[inputs.diskio]]
2016-12-13T09:01:07.850536095Z   
2016-12-13T09:01:07.850553141Z   
2016-12-13T09:01:07.850559799Z [[inputs.net]]
2016-12-13T09:01:07.850566049Z   
2016-12-13T09:01:07.850571555Z [[inputs.influxdb]]
2016-12-13T09:01:07.850576800Z   urls = ["http://10.103.253.43:80/debug/vars"]
2016-12-13T09:01:07.850582587Z [[inputs.kubernetes]]
2016-12-13T09:01:07.850588418Z   url = "http://10.240.0.19:10255"
2016-12-13T09:01:07.850594063Z   bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
2016-12-13T09:01:07.850600021Z   
2016-12-13T09:01:07.850605122Z   
2016-12-13T09:01:07.850610547Z   
2016-12-13T09:01:07.850615908Z   insecure_skip_verify = true
2016-12-13T09:01:07.850621341Z [[inputs.nsq]]
2016-12-13T09:01:07.850627237Z   endpoints = ["http://10.103.253.156:4151"]
2016-12-13T09:01:07.850632803Z   interval = "1s"
2016-12-13T09:01:07.850638463Z [[inputs.redis]]
2016-12-13T09:01:07.850643955Z   servers = ["tcp://:foo@10.103.243.126:6379"]
2016-12-13T09:01:07.850649618Z # Set Service Input Configuration
2016-12-13T09:01:07.850655265Z [[inputs.nsq_consumer]]
2016-12-13T09:01:07.850660967Z   server = "10.103.253.156:4150"
2016-12-13T09:01:07.850666620Z   topic = "metrics"
2016-12-13T09:01:07.850672023Z   channel = "consumer"
2016-12-13T09:01:07.850678743Z   max_in_flight = 100
2016-12-13T09:01:07.850684593Z   data_format = "influx"
2016-12-13T09:01:07.850692843Z ###########################################
2016-12-13T09:01:07.850700329Z ###########################################
2016-12-13T09:01:07.910949043Z 2016/12/13 09:01:07 INF    1 [metrics/consumer] (10.103.253.156:4150) connecting to nsqd
2016-12-13T09:01:10.587872549Z 2016/12/13 09:01:10 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:20.813784155Z 2016/12/13 09:01:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:30.488890095Z 2016/12/13 09:01:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:41.219745287Z 2016/12/13 09:01:41 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:01:50.722231932Z 2016/12/13 09:01:50 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:01.229486206Z 2016/12/13 09:02:01 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:11.007993912Z 2016/12/13 09:02:11 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:20.600314059Z 2016/12/13 09:02:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:30.652158165Z 2016/12/13 09:02:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:40.198493854Z 2016/12/13 09:02:40 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:02:50.223258238Z 2016/12/13 09:02:50 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:00.858215308Z 2016/12/13 09:03:00 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:10.705881319Z 2016/12/13 09:03:10 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:20.771301571Z 2016/12/13 09:03:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:30.437874621Z 2016/12/13 09:03:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:40.603927629Z 2016/12/13 09:03:40 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:03:51.060918864Z 2016/12/13 09:03:51 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:00.266918214Z 2016/12/13 09:04:00 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:10.435277890Z 2016/12/13 09:04:10 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:20.816313126Z 2016/12/13 09:04:20 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2016-12-13T09:04:30.839003676Z 2016/12/13 09:04:30 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]

Is there a way to dig it deeper. I did try to add AGENT_DEBUG:true, AGENT_QUITE:false but doesn't help much from log.

Copied from original issue: deis/monitor#165

@Cryptophobia
Copy link
Member Author

From @vdice on January 11, 2017 21:40

Hmmm, I'm not able to reproduce this issue w/ same telegraf image (also GKE, k8s server 1.4.7)

Is there a way to dig it deeper. I did try to add AGENT_DEBUG:true, AGENT_QUITE:false but doesn't help much from log.

@jchauncey any suggestions on how to debug?

@Cryptophobia
Copy link
Member Author

From @jchauncey on January 11, 2017 21:45

Nope. I've seen this before but I'm not sure what the cause was. Is it
still happening?

On Jan 11, 2017 4:40 PM, "Vaughn Dice" notifications@github.com wrote:

Hmmm, I'm not able to reproduce this issue w/ same telegraf image (also
GKE, k8s server 1.4.7)

Is there a way to dig it deeper. I did try to add AGENT_DEBUG:true,
AGENT_QUITE:false but doesn't help much from log.

@jchauncey https://github.com/jchauncey any suggestions on how to debug?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
deis/monitor#165 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAaRGG44WRs6bIBsrBRaYhCJH2f-7BPGks5rRUxLgaJpZM4LLgQt
.

@Cryptophobia
Copy link
Member Author

From @zinuzoid on January 12, 2017 11:1

@jchauncey yes, it's still happen.

2017-01-12T11:00:40.151027731Z 2017/01/12 11:00:40 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]
2017-01-12T11:00:50.990826710Z 2017/01/12 11:00:50 E! ERROR in input [inputs.kubernetes]: Errors encountered: [Error parsing response: parsing time "null" as ""2006-01-02T15:04:05Z07:00"": cannot parse "null" as """]

@Cryptophobia
Copy link
Member Author

From @stuszynski on February 6, 2017 15:0

@zinuzoid I encounter a similar issue. Telegraf is using url = "http://10.240.0.19:10255" for connecting to kubelet, but your kubelet is most likely listen on HTTPS only.

@Cryptophobia
Copy link
Member Author

From @stuszynski on February 7, 2017 12:38

Sorry, my bad. The http://<node-ip>:10255 refer to the unsecured read-only port available on each kubelet.

     --read-only-port int32                                    The read-only port for the Kubelet to serve on with no authentication/authorization, and for localhost healthz endpoint (set to 0 to disable) (default 10255)

I check it manually and I'm sure it provide data on this URL from a telegraf container. We still, however, experience above issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant