Enhancement Redis protocol(Pika) ? #3783

hereTac · 2018-06-04T06:45:40Z

Hi. I want to use netdata monitor Pika(Pika is a nosql compatible with redis, it is developed by Qihoo's DBA and infrastructure team) (Github:https://github.com/Qihoo360/pika) .
Any suggests will be helpful.Thanks.

The error log:
`[root@dc-pika64 netdata]# usr/libexec/netdata/plugins.d/python.d.plugin debug redis 1
2018-06-04 14:39:44: python.d INFO: plugin: main: Using python 2
2018-06-04 14:39:44: python.d DEBUG: plugin: main: module load source: 'redis' => [OK]
2018-06-04 14:39:44: python.d DEBUG: plugin: main: module load config: 'redis' => [OK]
2018-06-04 14:39:44: python.d DEBUG: plugin: main: job initialization: 'redis socket1' => ['OK']
2018-06-04 14:39:44: python.d DEBUG: plugin: main: job initialization: 'redis socket2' => ['OK']
2018-06-04 14:39:44: python.d DEBUG: plugin: main: job initialization: 'redis socket3' => ['OK']
2018-06-04 14:39:44: python.d DEBUG: plugin: main: job initialization: 'redis localhostpika' => ['OK']
2018-06-04 14:39:44: python.d DEBUG: plugin: main: job initialization: 'redis localipv6' => ['OK']
2018-06-04 14:39:44: python.d DEBUG: plugin: main: module status: 'redis' => [OK] (jobs: 5)
2018-06-04 14:39:44: python.d DEBUG: redis: socket1: attempting DGRAM unix socket "/tmp/redis.sock"
2018-06-04 14:39:44: python.d DEBUG: redis: socket1: Failed to connect DGRAM unix socket "/tmp/redis.sock": [Errno 2] No such file or directory
2018-06-04 14:39:44: python.d DEBUG: redis: socket1: attempting STREAM unix socket "/tmp/redis.sock"
2018-06-04 14:39:44: python.d DEBUG: redis: socket1: Failed to connect STREAM unix socket "/tmp/redis.sock": [Errno 2] No such file or directory
2018-06-04 14:39:44: python.d INFO: redis: socket1: check() => [FAILED]
2018-06-04 14:39:44: python.d DEBUG: redis: socket2: attempting DGRAM unix socket "/var/run/redis/redis.sock"
2018-06-04 14:39:44: python.d DEBUG: redis: socket2: Failed to connect DGRAM unix socket "/var/run/redis/redis.sock": [Errno 2] No such file or directory
2018-06-04 14:39:44: python.d DEBUG: redis: socket2: attempting STREAM unix socket "/var/run/redis/redis.sock"
2018-06-04 14:39:44: python.d DEBUG: redis: socket2: Failed to connect STREAM unix socket "/var/run/redis/redis.sock": [Errno 2] No such file or directory
2018-06-04 14:39:44: python.d INFO: redis: socket2: check() => [FAILED]
2018-06-04 14:39:44: python.d DEBUG: redis: socket3: attempting DGRAM unix socket "/var/lib/redis/redis.sock"
2018-06-04 14:39:44: python.d DEBUG: redis: socket3: Failed to connect DGRAM unix socket "/var/lib/redis/redis.sock": [Errno 2] No such file or directory
2018-06-04 14:39:44: python.d DEBUG: redis: socket3: attempting STREAM unix socket "/var/lib/redis/redis.sock"
2018-06-04 14:39:44: python.d DEBUG: redis: socket3: Failed to connect STREAM unix socket "/var/lib/redis/redis.sock": [Errno 2] No such file or directory
2018-06-04 14:39:44: python.d INFO: redis: socket3: check() => [FAILED]
2018-06-04 14:39:44: python.d DEBUG: redis: localhostpika: Creating socket to "172.31.10.63", port 9221
2018-06-04 14:39:44: python.d DEBUG: redis: localhostpika: connecting socket to "172.31.10.63", port 9221
2018-06-04 14:39:44: python.d DEBUG: redis: localhostpika: connected to "172.31.10.63", port 9221
2018-06-04 14:39:44: python.d DEBUG: redis: localhostpika: set socket timeout to: 5.0
2018-06-04 14:39:44: python.d DEBUG: redis: localhostpika: sending request: INFO

2018-06-04 14:39:44: python.d DEBUG: redis: localhostpika: receiving response
2018-06-04 14:39:44: python.d DEBUG: redis: localhostpika: received data
2018-06-04 14:39:44: python.d DEBUG: redis: localhostpika: received full response from redis
2018-06-04 14:39:44: python.d DEBUG: redis: localhostpika: final response: $1321

Server

pika_version:2.3.4
pika_git_sha:3b86e4a1c76313dd1f448a9317a4f5551e235b35
pika_build_compile_date: May 17 2018
os:Linux 2.6.32-696.el6.x86_64 x86_64
arch_bits:64
process_id:10159
tcp_port:9221
thread_num:20
sync_thread_num:6
uptime_in_seconds:874230
uptime_in_days:11
config_file:conf/pika.conf
server_id:1

Data

db_size:57252174373
db_size_human:54599M
compression:snappy
used_memory:1676574536
used_memory_human:1598M
db_memtable_usage:133194920
db_tablereader_usage:1543379616

Log

log_size:19183503438
log_size_human:18294M
safety_purge:write2file1871
expire_logs_days:7
expire_logs_nums:10
binlog_offset:1881 74942108

Clients

connected_clients:3

Stats

total_connections_received:2226
instantaneous_ops_per_sec:0
total_commands_processed:516545968
is_bgsaving:No, 20180528150116, 0
is_slots_reloading:No, , 0
is_slots_cleanuping:No, , 0
is_scaning_keyspace:No
is_compact:No
compact_cron:
compact_interval:

CPU

used_cpu_sys:202135.27
used_cpu_user:995980.38
used_cpu_sys_children:87.66
used_cpu_user_children:177.82

Replication(MASTER)

role:master
connected_slaves:1
slave0:ip=172.31.10.64,port=9221,state=online,sid=2,lag=0

Keyspace

Time:2018-06-04 11:45:46

kv keys:0
hash keys:174999283
list keys:0
zset keys:0
set keys:0

2018-06-04 14:39:44: python.d ERROR: redis: localhostpika: check() unhandled exception: 'rdb_bgsave_in_progress'
2018-06-04 14:39:44: python.d INFO: redis: localhostpika: check() => [FAILED]
2018-06-04 14:39:44: python.d DEBUG: redis: localipv6: Creating socket to "::1", port 6379
2018-06-04 14:39:44: python.d DEBUG: redis: localipv6: connecting socket to "::1", port 6379
2018-06-04 14:39:44: python.d ERROR: redis: localipv6: Failed to connect to "::1", port 6379, error: [Errno 111] Connection refused
2018-06-04 14:39:44: python.d DEBUG: redis: localipv6: closing socket
2018-06-04 14:39:44: python.d INFO: redis: localipv6: check() => [FAILED]
2018-06-04 14:39:44: python.d INFO: plugin: main: FINISHED
`

ktsaou · 2018-06-10T21:05:23Z

hm... I guess the statistics output of pika is not the same with redis.
@l2isbad can we make the plugin ignore the output that is not compatible and continue with whatever data finds?

ilyam8 · 2018-06-11T02:15:48Z

@zhangxin0112

please change
https://github.com/firehol/netdata/blob/master/python.d/redis.chart.py#L158
to

if data.get('rdb_bgsave_in_progress', '') != '0\r':

and restart the plugin

hereTac · 2018-06-14T07:51:12Z

I change

data['rdb_last_bgsave_status'] = 0 if data['rdb_last_bgsave_status'] == 'ok\r' else 1
->
data['rdb_last_bgsave_status'] = 0 if data.get('rdb_last_bgsave_status') == 'ok\r' else 1

thanks u . It works for pika. ^ ^

hereTac · 2018-06-14T08:01:32Z

I debug redis and get messages like this.
/opt/netdata/netdata-bin/netdata/usr/libexec/netdata/plugins.d/python.d.plugin debug redis 1

Keyspace

Time:2018-05-28 16:12:28

kv keys:0
hash keys:173831712
list keys:0
zset keys:0
set keys:0

BEGIN redis_pika64.operations 0
SET 'total_commands_processed' = 311344140
SET 'instantaneous_ops_per_sec' = 1
END

BEGIN redis_pika64.bgsave_now 0
SET 'rdb_bgsave_in_progress' = 4
END

2018-06-14 15:58:05: python.d ERROR: redis: pika64: update() unhandled exception: invalid literal for int() with base 10: '55332M '
2018-06-14 15:58:05: python.d DEBUG: redis: pika64: update => [FAILED] (elapsed time: -, retries left: 57)
2018-06-14 15:58:05: python.d DEBUG: redis: pika64: sleeping for 0.992940403987 to reach frequency of 1 sec
2018-06-14 15:58:05: python.d DEBUG: redis: pika63: received data
2018-06-14 15:58:05: python.d DEBUG: redis: pika63: received full response from redis
2018-06-14 15:58:05: python.d DEBUG: redis: pika63: final response: $1323

ilyam8 · 2018-06-14T08:20:58Z

@zhangxin0112 run it with trace flag

hereTac · 2018-06-15T01:13:11Z

The error log :

2018-06-15 09:05:08: python.d ERROR: redis: pika63: create() => [NOT ADDED] (dimension '[u'db_memtable_usage', None, 'absolute']': 'db_memtable_usage' already in 'redis_pika63.keys' dimensions)
2018-06-15 09:05:08: python.d ERROR: redis: pika63: create() => [NOT ADDED] (dimension '[u'db_size_human', None, 'absolute']': 'db_size_human' already in 'redis_pika63.keys' dimensions)
2018-06-15 09:05:08: python.d ERROR: redis: pika63: create() => [NOT ADDED] (dimension '[u'db_size', None, 'absolute']': 'db_size' already in 'redis_pika63.keys' dimensions)
2018-06-15 09:05:08: python.d ERROR: redis: pika63: create() => [NOT ADDED] (dimension '[u'db_tablereader_usage', None, 'absolute']': 'db_tablereader_usage' already in 'redis_pika63.keys' dimensions)
2018-06-15 09:05:08: netdata ERROR : PLUGINSD[python.d] : (0580@rrdset.c :rrdset_create_c): File /opt/netdata/netdata-bin/netdata/var/cache/netdata/netdata.runtime_redis_pika63/main.db is too old. Clearing it.
2018-06-15 09:05:08: netdata INFO : PLUGINSD[python.d] : (0162@rrddim.c :rrddim_add_cust): File /opt/netdata/netdata-bin/netdata/var/cache/netdata/netdata.runtime_redis_pika63/run_time.db is too old (last collected 1529024708 seconds ago, but the database is 3996 seconds). Clearing it.
2018-06-15 09:05:08: python.d ERROR: redis: pika64: create() => [NOT ADDED] (dimension '[u'db_memtable_usage', None, 'absolute']': 'db_memtable_usage' already in 'redis_pika64.keys' dimensions)
2018-06-15 09:05:08: python.d ERROR: redis: pika64: create() => [NOT ADDED] (dimension '[u'db_size_human', None, 'absolute']': 'db_size_human' already in 'redis_pika64.keys' dimensions)
2018-06-15 09:05:08: python.d ERROR: redis: pika64: create() => [NOT ADDED] (dimension '[u'db_size', None, 'absolute']': 'db_size' already in 'redis_pika64.keys' dimensions)
2018-06-15 09:05:08: python.d ERROR: redis: pika64: create() => [NOT ADDED] (dimension '[u'db_tablereader_usage', None, 'absolute']': 'db_tablereader_usage' already in 'redis_pika64.keys' dimensions)
2018-06-15 09:05:08: netdata ERROR : PLUGINSD[python.d] : (0580@rrdset.c :rrdset_create_c): File /opt/netdata/netdata-bin/netdata/var/cache/netdata/netdata.runtime_redis_pika64/main.db is too old. Clearing it.
2018-06-15 09:05:08: netdata INFO : PLUGINSD[python.d] : (0162@rrddim.c :rrddim_add_cust): File /opt/netdata/netdata-bin/netdata/var/cache/netdata/netdata.runtime_redis_pika64/run_time.db is too old (last collected 1529024
708 seconds ago, but the database is 3996 seconds). Clearing it.
2018-06-15 09:05:08: netdata ERROR : PLUGINSD[python.d] : (0580@rrdset.c :rrdset_create_c): File /opt/netdata/netdata-bin/netdata/var/cache/netdata/netdata.runtime_sensors/main.db is too old. Clearing it.
2018-06-15 09:05:08: netdata INFO : PLUGINSD[python.d] : (0162@rrddim.c :rrddim_add_cust): File /opt/netdata/netdata-bin/netdata/var/cache/netdata/netdata.runtime_sensors/run_time.db is too old (last collected 1529024708 s
econds ago, but the database is 3996 seconds). Clearing it.
2018-06-15 09:05:09: python.d ERROR: redis: pika64: update() unhandled exception: invalid literal for int() with base 10: '55310M '
2018-06-15 09:05:09: python.d ERROR: redis: pika63: update() unhandled exception: invalid literal for int() with base 10: '56460M '

ilyam8 · 2018-06-15T01:21:42Z

for some reason i see no traceback in your logs 😢

please replace
https://github.com/firehol/netdata/blob/master/python.d/python_modules/bases/charts.py#L322
with

except (KeyError, ValueError, TypeError):

hereTac · 2018-06-15T04:53:48Z

[root@dc-pika64 ~]# less /opt/netdata/netdata-bin/netdata/var/log/netdata/error.log|grep redis

2018-06-15 12:49:22: python.d INFO: redis: socket1: check() => [FAILED]
2018-06-15 12:49:22: python.d INFO: redis: socket2: check() => [FAILED]
2018-06-15 12:49:22: python.d INFO: redis: socket3: check() => [FAILED]
2018-06-15 12:49:22: python.d INFO: redis: pika63: check() => [OK]
2018-06-15 12:49:22: python.d ERROR: redis: localipv6: Failed to connect to "::1", port 6379, error: [Errno 111] Connection refused
2018-06-15 12:49:22: python.d INFO: redis: localipv6: check() => [FAILED]
2018-06-15 12:49:23: netdata INFO : PLUGINSD[python.d] : (0162@rrddim.c :rrddim_add_cust): File /opt/netdata/netdata-bin/netdata/var/cache/netdata/redis_pika63.keys/db_size_human.db is too old (last collected 1529038163 seconds ago, but the database is 3996 seconds). Clearing it.
2018-06-15 12:49:23: netdata INFO : PLUGINSD[python.d] : (0162@rrddim.c :rrddim_add_cust): File /opt/netdata/netdata-bin/netdata/var/cache/netdata/redis_pika63.clients/blocked_clients.db is too old (last collected 1529038163 seconds ago, but the database is 3996 seconds). Clearing it.
2018-06-15 12:49:23: netdata INFO : PLUGINSD[python.d] : (0162@rrddim.c :rrddim_add_cust): File /opt/netdata/netdata-bin/netdata/var/cache/netdata/redis_pika63.connections/rejected_connections.db is too old (last collected 1529038163 seconds ago, but the database is 3996 seconds). Clearing it.
2018-06-15 12:49:23: netdata INFO : PLUGINSD[python.d] : (0162@rrddim.c :rrddim_add_cust): File /opt/netdata/netdata-bin/netdata/var/cache/netdata/redis_pika63.memory/used_memory_lua.db is too old (last collected 1529038163 seconds ago, but the database is 3996 seconds). Clearing it.
2018-06-15 12:49:33: python.d ERROR: redis: pika63: chart 'redis_pika63.net' was suppressed due to non updating
2018-06-15 12:49:33: python.d ERROR: redis: pika63: chart 'redis_pika63.persistence' was suppressed due to non updating

ilyam8 · 2018-06-15T04:56:56Z

@zhangxin0112 does it work after #3783 (comment) ?

hereTac · 2018-06-15T05:35:24Z

emmm..
almost. This is no network . Except the keys.

ilyam8 · 2018-06-15T06:31:02Z

Ok, the problem is

db_size:57252174373
db_size_human:54599M
db_memtable_usage:133194920
db_tablereader_usage:1543379616

I see
https://github.com/firehol/netdata/blob/9598db24cae63fd6241bb236e2fac55ae7b92d2f/python.d/redis.chart.py#L198-L200

You can comment out this^^ section to remove keys chart (or remove keys chart from ORDER).

chart is

https://github.com/firehol/netdata/blob/9598db24cae63fd6241bb236e2fac55ae7b92d2f/python.d/redis.chart.py#L49-L53

About network section. Module uses total_net_input_bytes and total_net_output_bytes keys to get bandwidth. I see no such keys in your debug output.

ilyam8 · 2018-06-15T06:48:45Z

I have no clue what is Redis Keys per Database and why we need name.startswith('db') 😓
I did a clean redis install and db_* keys are missing from the INFO output.

edit: ok, got it, i have to add some keys to see db* stuff in the INFO

hereTac · 2018-06-15T08:27:07Z

em.....It is the difference between Redis and Pika.

redis: info
....
# Keyspace
db0:keys=2,expires=0,avg_ttl=0
.....

pika:info
.....
# Keyspace
# Time:2018-06-04 11:45:46
kv keys:0
hash keys:174999283
list keys:0
zset keys:0
set keys:0
.....

Thanks for your help.

ilyam8 · 2018-06-15T09:11:25Z

Ok, so Keyspace is summary metrics, am i right? (redis has per db)

hereTac · 2018-06-15T09:48:36Z

Ye.
Redis has many dbs (like db0,db1,db2..... ).
Redis count all keys in db0:keys=2,expires=0,avg_ttl=0

Pika is a nosql compatible with redis.
Pika count keys with data type(hash,list,zset,set...).
kv keys:0 hash keys:174999283 list keys:0 zset keys:0 set keys:0

hereTac · 2018-06-15T11:55:13Z

I try to code the ../netdata/usr/libexec/netdata/python.d/redis.chart.py
But it did not work.

ilyam8 · 2018-06-15T11:58:29Z

@zhangxin0112 give me some time, i'll fix it

hereTac · 2018-06-15T11:59:45Z

Ok.

ktsaou · 2018-06-19T18:05:40Z

merged it

ktsaou added the enhancement label Jun 10, 2018

hereTac closed this as completed Jun 14, 2018

hereTac reopened this Jun 14, 2018

ilyam8 mentioned this issue Jun 16, 2018

redis: refactor + pika support #3832

Merged

ktsaou closed this as completed in #3832 Jun 19, 2018

ktsaou added the fixed label Jun 19, 2018

paulfantom added the module/python label Oct 19, 2018

ilyam8 added collectors/python.d area/collectors Everything related to data collection and removed area/external/python labels Apr 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhancement Redis protocol(Pika) ? #3783

Enhancement Redis protocol(Pika) ? #3783

hereTac commented Jun 4, 2018

ktsaou commented Jun 10, 2018

ilyam8 commented Jun 11, 2018

hereTac commented Jun 14, 2018 •

edited

Loading

hereTac commented Jun 14, 2018

ilyam8 commented Jun 14, 2018

hereTac commented Jun 15, 2018

ilyam8 commented Jun 15, 2018

hereTac commented Jun 15, 2018 •

edited

Loading

ilyam8 commented Jun 15, 2018

hereTac commented Jun 15, 2018 •

edited

Loading

ilyam8 commented Jun 15, 2018 •

edited

Loading

ilyam8 commented Jun 15, 2018 •

edited

Loading

hereTac commented Jun 15, 2018 •

edited

Loading

ilyam8 commented Jun 15, 2018

hereTac commented Jun 15, 2018

hereTac commented Jun 15, 2018 •

edited

Loading

ilyam8 commented Jun 15, 2018

hereTac commented Jun 15, 2018

ktsaou commented Jun 19, 2018

Enhancement Redis protocol(Pika) ? #3783

Enhancement Redis protocol(Pika) ? #3783

Comments

hereTac commented Jun 4, 2018

Server

Data

Log

Clients

Stats

CPU

Replication(MASTER)

Keyspace

Time:2018-06-04 11:45:46

ktsaou commented Jun 10, 2018

ilyam8 commented Jun 11, 2018

hereTac commented Jun 14, 2018 • edited Loading

hereTac commented Jun 14, 2018

Keyspace

Time:2018-05-28 16:12:28

ilyam8 commented Jun 14, 2018

hereTac commented Jun 15, 2018

ilyam8 commented Jun 15, 2018

hereTac commented Jun 15, 2018 • edited Loading

ilyam8 commented Jun 15, 2018

hereTac commented Jun 15, 2018 • edited Loading

ilyam8 commented Jun 15, 2018 • edited Loading

ilyam8 commented Jun 15, 2018 • edited Loading

hereTac commented Jun 15, 2018 • edited Loading

em.....It is the difference between Redis and Pika.

ilyam8 commented Jun 15, 2018

hereTac commented Jun 15, 2018

hereTac commented Jun 15, 2018 • edited Loading

ilyam8 commented Jun 15, 2018

hereTac commented Jun 15, 2018

ktsaou commented Jun 19, 2018

hereTac commented Jun 14, 2018 •

edited

Loading

hereTac commented Jun 15, 2018 •

edited

Loading

hereTac commented Jun 15, 2018 •

edited

Loading

ilyam8 commented Jun 15, 2018 •

edited

Loading

ilyam8 commented Jun 15, 2018 •

edited

Loading

hereTac commented Jun 15, 2018 •

edited

Loading

hereTac commented Jun 15, 2018 •

edited

Loading