A simple log store using parquet.
PATH is using for storing log data.
default PATH = ./logs
default PARQUET_ROW_GROUP_SIZE = 268435456
row size is 256mb
default PAEQUET_PAGE_SIZE = 8192
page size is 8kb
This is meaning how to compress parquet file.
- NONE
- SNAPPY
- GZIP
- LZO
- BROTLI
- ZSTD
- nats
connecting nats server
default NATS_URL = nats://127.0.0.1:4222
default NATS_SUBJECT = log
- kafka
connecting kafka server
dafault KAFKA_TOPIC = log
default KAFKA_PARTITION = 0
default KAFKA_NETWORK = tcp
defaul KAFKA_URL = localhost:9092
- http
listening http server
default HTTP_URL = 0.0.0.0:14219
send post request to "/"
log
struct from github.com/snowmerak/log-silo/log
.
package log
type Log struct {
UnixTime int64 `parquet:"name=unix_time, type=INT64" json:"unix_time"`
AppID int32 `parquet:"name=app_id, type=INT32" json:"app_id"`
Level int32 `parquet:"name=level, type=INT32" json:"level"`
Message string `parquet:"name=message, type=BYTE_ARRAY, convertedtype=UTF8, encoding=PLAIN_DICTIONARY" json:"message"`
}
log
has unix_time
, app_id
, level
and message
.
can represent to json.
{
"unix_time": 1598000,
"app_id": 1,
"level": 1,
"message": "a log message"
}
app_id
and level
are customizable by your own team.
message
is UTF8 string.
unix_time
is unix time second. can use unix time nano.