Character limit for json_lines #109
Comments
|
Hi, thanks for the bug report. I'll need to know more information to continue debugging this issue. Things like:
|
|
I was pretty sure there was some sort of character limitation but as I've been working through my issue I've found it to be with the actual JSON format that was being parsed. I could only recreate it some of the time with the same test data; frankly it still doesn't make sense to me why it worked some of the time. Anyways I think I'm on the correct path to fixing my issue, sorry for the false alarm! |
|
That would make sense. If you try to log a JSON string to LogStashLogger, and it cannot parse it, it will dump the contents into the |
|
I've dug more into this issue. It's actually a logstash character
limitation of 4095, so nothing to do with your code.
…On Tue, Dec 20, 2016 at 6:16 PM, David Butler ***@***.***> wrote:
That would make sense. If you try to log a JSON string to LogStashLogger,
and it cannot parse it, it will dump the contents into the message field
(which is consistent with your observation).
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#109 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AJOMx7dexi4yBzuZ35bAYK3TL2u2aIcVks5rKG_SgaJpZM4LSOtM>
.
--
-Joey Ross
|
|
What logstash listener configuration are you using? It would be good to know the details of your issue so it can be documented in the Troubleshooting section of the readme. |
|
I'm using logstash version 2.3.2. What other information would you want for documentation? |
|
Basically the contents of the config file that you use with logstash. Or at least which output you're using so I can figure out where to look in the logstash source code. |
|
Here is my input code with the json_lines codec:
input { udp {
host => "*.*.*.*"
port => ****
tags => ["rails"]
codec => json_lines
}
}
It will show up dumped in the catch all index in the message field on
kibana with a { after the 4095th character, without new lines. It is output
to a redis queue, which might be the bottle neck I'm guessing? I'm not sure
exactly, I'm basing this off of a consumable config. Anyways, :
output {
if "rails" in [tags] {
redis {
host => 'localhost'
data_type => 'list'
key => 'rails:redis'
congestion_threshold => 6000000
}
}
}
After that I have another logstash config set up to read in from that redis
queue:
input {
redis {
host => "localhost"
data_type => 'list'
key => 'rails:redis'
type => 'redis-input'
}
}
And that has a simple output to the Elasticsearch node that works fine for
other events. It's indexed based on an if for the name of the controller
and it has an else catch-all index.
|
|
I think you're experiencing the same issue reported in #43. To the best of my knowledge, this is a bug in Logstash's implementation of the UDP input. Is there a reason you're sending the events to a UDP listener first and then forwarding to Redis? Why not send the events directly from LogstashLogger to Redis? |
|
You mentioned something that sounds important - the final |
I seem to be experiencing an issue with my rails app where I'm trying to log a large event and the json_lines codec seems to malfunction. When the event occurs successfully a lot of data is logged, but the log is sent to my catch all index instead of logging it under the name of the controller because instead of being parsed by json_lines it's all being dumped into the message field. When there is an issue with the application and not much is logged besides the failure, json_lines works fine.
The text was updated successfully, but these errors were encountered: