Received message could not be parsed

[magicxor]

The problem

From time to time the following error occurs:

2017-01-11 17:41:31,926 Received Message: b'{"unread": false, "event": "read", "id": "01000000a6b4ce0fae0000000000000077616c38a95a2f29", "flags": 256, "out": false, "from": {"id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "peer_id": 265204902, "print_name": "Chat_Wars", "first_name": "Chat Wars", "flags": 1, "last_name": "", "username": "ChatWarsBot"}, "to": {"id": "$010000009fb8ba120b62fa795ec02cf3", "peer_type": "user", "peer_id": 314226847, "when": "2017-01-11 17:42:11", "print_name": "Mike_Udkovsky", "flags": 524289, "first_name": "Mike", "phone": "79671494739", "last_name": "Udkovsky", "username": "mikeud"}, "service": false, "date": 1484127604, "text": "\xd0\xa2\xd1\x8b \xd0\xbf\xd1\x80\xd0\xb8\xd0\xb3\xd0\xbe\xd1\x82\xd0\xbe\xd0\xb2\xd0\xb8\xd0\xbb\xd1\x81\xd1\x8f \xd0\xba \xd0\xb7\xd0\xb0\xd1\x89\xd0\xb8\xd1\x82\xd0\xb5. \xd0\x91\xd0\xbb\xd0\xb8\xd0\xb6\xd0\xb0\xd0\xb9\xd1\x88\xd0\xb5\xd0\xb5 \xd1\x81\xd1\x80\xd0\xb0\xd0\xb6\xd0\xb5\xd0\xbd\xd0\xb8\xd0\xb5 \xd1\x87\xd0\xb5\xd1\x80\xd0\xb5\xd0\xb7 20\xd0\xbc\xd0\xb8\xd0\xbd\xd1\x83\xd1\x82. \xd0\x96\xd0\xb4\xd0\xb8 \xd1\x80\xd0\xb5\xd0\xb7\xd1\x83\xd0\xbb\xd1\x8c\xd1\x82\xd0\xb0\xd1\x82\xd0\xb0."}\n{"unread": false, "event": "read", "id": "02000000b110f70aac000000000000000000000000000000", "flags": 256, "out": false, "reply_id": "02000000b110f70aa4000000000000000000000000000000", "to": {"id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "peer_id": 183963825, "print_name": "\xd0\x9c\xd0\xb0\xd1\x81\xd0\xbe\xd0\xbd\xd1\x81\xd0\xba\xd0\xb0\xd1\x8f_\xd0\xbb\xd0\xbe\xd0\xb6\xd0\xb0_\xf0\x9f\x87\xa8\xf0\x9f\x87\xbe_\xf0\x9f\x87\xae\xf0\x9f\x87\xb2_\xf0\x9f\x87\xbb\xf0\x9f\x87\xa6_\xf0\x9f\x87\xac\xf0\x9f\x87\xb5_\xf0\x9f\x87\xaa\xf0\x9f\x87\xba", "flags": 65537, "title": "\xd0\x9c\xd0\xb0\xd1\x81\xd0\xbe\xd0\xbd\xd1\x81\xd0\xba\xd0\xb0\xd1\x8f \xd0\xbb\xd0\xbe\xd0\xb6\xd0\xb0 \xf0\x9f\x87\xa8\xf0\x9f\x87\xbe \xf0\x9f\x87\xae\xf0\x9f\x87\xb2 \xf0\x9f\x87\xbb\xf0\x9f\x87\xa6 \xf0\x9f\x87\xac\xf0\x9f\x87\xb5 \xf0\x9f\x87\xaa\xf0\x9f\x87\xba", "admin": {"id": "$01000000000000000000000000000000", "peer_type": "user", "peer_id": 0, "print_name": "user#0"}, "members_num": 5}, "from": {"id": "$010000005ac62f129716439697cfeaf8", "peer_type": "user", "peer_id": 305120858, "when": "2017-01-11 11:44:32", "print_name": "Octo_Cat", "flags": 1, "first_name": "Octo", "last_name": "Cat", "username": "octo_cat"}, "service": false, "date": 1484127448, "text": "\xd0\x9f\xd0\xbe\xd0\xbd\xd1\x8f\xd1\x82\xd0\xbd\xd0\xbe, \xd1\x81\xd1\x82\xd0\xb0\xd0\xb2\xd0\xbb\xd1\x8e: \xe2\x9a\x94 \xd0\x90\xd1\x82\xd0\xb0\xd0\xba\xd0\xb0, \xf0\x9f\x87\xac\xf0\x9f\x87\xb5"}'
2017-01-11 17:41:31,927 Received Message: "{"unread": false, "event": "read", "id": "01000000a6b4ce0fae0000000000000077616c38a95a2f29", "flags": 256, "out": false, "from": {"id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "peer_id": 265204902, "print_name": "Chat_Wars", "first_name": "Chat Wars", "flags": 1, "last_name": "", "username": "ChatWarsBot"}, "to": {"id": "$010000009fb8ba120b62fa795ec02cf3", "peer_type": "user", "peer_id": 314226847, "when": "2017-01-11 17:42:11", "print_name": "Mike_Udkovsky", "flags": 524289, "first_name": "Mike", "phone": "79671494739", "last_name": "Udkovsky", "username": "mikeud"}, "service": false, "date": 1484127604, "text": "Ты приготовился к защите. Ближайшее сражение через 20минут. Жди результата."}
{"unread": false, "event": "read", "id": "02000000b110f70aac000000000000000000000000000000", "flags": 256, "out": false, "reply_id": "02000000b110f70aa4000000000000000000000000000000", "to": {"id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "peer_id": 183963825, "print_name": "Масонская_ложа_🇨🇾_🇮🇲_🇻🇦_🇬🇵_🇪🇺", "flags": 65537, "title": "Масонская ложа 🇨🇾 🇮🇲 🇻🇦 🇬🇵 🇪🇺", "admin": {"id": "$01000000000000000000000000000000", "peer_type": "user", "peer_id": 0, "print_name": "user#0"}, "members_num": 5}, "from": {"id": "$010000005ac62f129716439697cfeaf8", "peer_type": "user", "peer_id": 305120858, "when": "2017-01-11 11:44:32", "print_name": "Octo_Cat", "flags": 1, "first_name": "Octo", "last_name": "Cat", "username": "octo_cat"}, "service": false, "date": 1484127448, "text": "Понятно, ставлю: ⚔ Атака, 🇬🇵"}"
2017-01-11 17:41:31,930 Received message could not be parsed.
Message:>{"unread": false, "event": "read", "id": "01000000a6b4ce0fae0000000000000077616c38a95a2f29", "flags": 256, "out": false, "from": {"id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "peer_id": 265204902, "print_name": "Chat_Wars", "first_name": "Chat Wars", "flags": 1, "last_name": "", "username": "ChatWarsBot"}, "to": {"id": "$010000009fb8ba120b62fa795ec02cf3", "peer_type": "user", "peer_id": 314226847, "when": "2017-01-11 17:42:11", "print_name": "Mike_Udkovsky", "flags": 524289, "first_name": "Mike", "phone": "79671494739", "last_name": "Udkovsky", "username": "mikeud"}, "service": false, "date": 1484127604, "text": "Ты приготовился к защите. Ближайшее сражение через 20минут. Жди результата."}
{"unread": false, "event": "read", "id": "02000000b110f70aac000000000000000000000000000000", "flags": 256, "out": false, "reply_id": "02000000b110f70aa4000000000000000000000000000000", "to": {"id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "peer_id": 183963825, "print_name": "Масонская_ложа_🇨🇾_🇮🇲_🇻🇦_🇬🇵_🇪🇺", "flags": 65537, "title": "Масонская ложа 🇨🇾 🇮🇲 🇻🇦 🇬🇵 🇪🇺", "admin": {"id": "$01000000000000000000000000000000", "peer_type": "user", "peer_id": 0, "print_name": "user#0"}, "members_num": 5}, "from": {"id": "$010000005ac62f129716439697cfeaf8", "peer_type": "user", "peer_id": 305120858, "when": "2017-01-11 11:44:32", "print_name": "Octo_Cat", "flags": 1, "first_name": "Octo", "last_name": "Cat", "username": "octo_cat"}, "service": false, "date": 1484127448, "text": "Понятно, ставлю: ⚔ Атака, 🇬🇵"}<
Traceback (most recent call last):
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pytg\receiver.py", line 182, in _add_message
    json_dict = json.loads(text)
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\json\__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\json\decoder.py", line 342, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 718)

2017-01-11 20:27:05,291 Received Message: b'{"event": "read", "reply_id": "02000000b110f70ac6000000000000000000000000000000", "id": "02000000b110f70acd000000000000000000000000000000", "flags": 256, "out": false, "from": {"peer_id": 314998971, "id": "$01000000bb80c61256f8c2ffc2450486", "when": "2017-01-11 20:04:52", "peer_type": "user", "print_name": "\xd0\x9f\xd0\x9e\xd0\x9a\xd0\xa0\xd0\x90\xd0\xa1_\xd0\x9e\xd0\x91\xd0\x9e\xd0\x95\xd0\x92", "username": "pokras_oboev", "flags": 1, "first_name": "\xd0\x9f\xd0\x9e\xd0\x9a\xd0\xa0\xd0\x90\xd0\xa1", "last_name": "\xd0\x9e\xd0\x91\xd0\x9e\xd0\x95\xd0\x92"}, "to": {"peer_id": 183963825, "id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "title": "\xd0\x9c\xd0\xb0\xd1\x81\xd0\xbe\xd0\xbd\xd1\x81\xd0\xba\xd0\xb0\xd1\x8f \xd0\xbb\xd0\xbe\xd0\xb6\xd0\xb0 \xf0\x9f\x87\xa8\xf0\x9f\x87\xbe \xf0\x9f\x87\xae\xf0\x9f\x87\xb2 \xf0\x9f\x87\xbb\xf0\x9f\x87\xa6 \xf0\x9f\x87\xac\xf0\x9f\x87\xb5 \xf0\x9f\x87\xaa\xf0\x9f\x87\xba", "print_name": "\xd0\x9c\xd0\xb0\xd1\x81\xd0\xbe\xd0\xbd\xd1\x81\xd0\xba\xd0\xb0\xd1\x8f_\xd0\xbb\xd0\xbe\xd0\xb6\xd0\xb0_\xf0\x9f\x87\xa8\xf0\x9f\x87\xbe_\xf0\x9f\x87\xae\xf0\x9f\x87\xb2_\xf0\x9f\x87\xbb\xf0\x9f\x87\xa6_\xf0\x9f\x87\xac\xf0\x9f\x87\xb5_\xf0\x9f\x87\xaa\xf0\x9f\x87\xba", "admin": {"peer_id": 0, "id": "$01000000000000000000000000000000", "peer_type": "user", "print_name": "user#0"}, "flags": 65537, "members_num": 5}, "unread": false, "text": "\xd0\xbb\xd0\xb0\xd0\xb4\xd0\xbd\xd0\xbe: \xe2\x9a\x94 \xd0\x90\xd1\x82\xd0\xb0\xd0\xba\xd0\xb0, \xf0\x9f\x87\xac\xf0\x9f\x87\xb5", "date": 1484136045, "service": false}\n{"event": "read", "id": "01000000a6b4ce0fca0000000000000077616c38a95a2f29", "flags": 256, "out": false, "text": "\xd0\xa2\xd1\x8b \xd0\xbf\xd1\x80\xd0\xb8\xd0\xb3\xd0\xbe\xd1\x82\xd0\xbe\xd0\xb2\xd0\xb8\xd0\xbb\xd1\x81\xd1\x8f \xd0\xba \xd0\xb0\xd1\x82\xd0\xb0\xd0\xba\xd0\xb5. \xd0\x91\xd0\xbb\xd0\xb8\xd0\xb6\xd0\xb0\xd0\xb9\xd1\x88\xd0\xb5\xd0\xb5 \xd1\x81\xd1\x80\xd0\xb0\xd0\xb6\xd0\xb5\xd0\xbd\xd0\xb8\xd0\xb5 \xd1\x87\xd0\xb5\xd1\x80\xd0\xb5\xd0\xb7 59\xd0\xbc\xd0\xb8\xd0\xbd\xd1\x83\xd1\x82. \xd0\x96\xd0\xb4\xd0\xb8 \xd1\x80\xd0\xb5\xd0\xb7\xd1\x83\xd0\xbb\xd1\x8c\xd1\x82\xd0\xb0\xd1\x82\xd0\xb0.", "date": 1484136042, "service": false, "from": {"peer_id": 265204902, "id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "print_name": "Chat_Wars", "username": "ChatWarsBot", "flags": 1, "first_name": "Chat Wars", "last_name": ""}, "to": {"peer_id": 314226847, "id": "$010000009fb8ba120b62fa795ec02cf3", "when": "2017-01-11 20:26:06", "peer_type": "user", "phone": "79671494739", "print_name": "Mike_Udkovsky", "username": "mikeud", "flags": 524289, "first_name": "Mike", "last_name": "Udkovsky"}, "unread": false}'
2017-01-11 20:27:05,292 Received Message: "{"event": "read", "reply_id": "02000000b110f70ac6000000000000000000000000000000", "id": "02000000b110f70acd000000000000000000000000000000", "flags": 256, "out": false, "from": {"peer_id": 314998971, "id": "$01000000bb80c61256f8c2ffc2450486", "when": "2017-01-11 20:04:52", "peer_type": "user", "print_name": "ПОКРАС_ОБОЕВ", "username": "pokras_oboev", "flags": 1, "first_name": "ПОКРАС", "last_name": "ОБОЕВ"}, "to": {"peer_id": 183963825, "id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "title": "Масонская ложа 🇨🇾 🇮🇲 🇻🇦 🇬🇵 🇪🇺", "print_name": "Масонская_ложа_🇨🇾_🇮🇲_🇻🇦_🇬🇵_🇪🇺", "admin": {"peer_id": 0, "id": "$01000000000000000000000000000000", "peer_type": "user", "print_name": "user#0"}, "flags": 65537, "members_num": 5}, "unread": false, "text": "ладно: ⚔ Атака, 🇬🇵", "date": 1484136045, "service": false}
{"event": "read", "id": "01000000a6b4ce0fca0000000000000077616c38a95a2f29", "flags": 256, "out": false, "text": "Ты приготовился к атаке. Ближайшее сражение через 59минут. Жди результата.", "date": 1484136042, "service": false, "from": {"peer_id": 265204902, "id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "print_name": "Chat_Wars", "username": "ChatWarsBot", "flags": 1, "first_name": "Chat Wars", "last_name": ""}, "to": {"peer_id": 314226847, "id": "$010000009fb8ba120b62fa795ec02cf3", "when": "2017-01-11 20:26:06", "peer_type": "user", "phone": "79671494739", "print_name": "Mike_Udkovsky", "username": "mikeud", "flags": 524289, "first_name": "Mike", "last_name": "Udkovsky"}, "unread": false}"
2017-01-11 20:27:05,293 Received message could not be parsed.
Message:>{"event": "read", "reply_id": "02000000b110f70ac6000000000000000000000000000000", "id": "02000000b110f70acd000000000000000000000000000000", "flags": 256, "out": false, "from": {"peer_id": 314998971, "id": "$01000000bb80c61256f8c2ffc2450486", "when": "2017-01-11 20:04:52", "peer_type": "user", "print_name": "ПОКРАС_ОБОЕВ", "username": "pokras_oboev", "flags": 1, "first_name": "ПОКРАС", "last_name": "ОБОЕВ"}, "to": {"peer_id": 183963825, "id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "title": "Масонская ложа 🇨🇾 🇮🇲 🇻🇦 🇬🇵 🇪🇺", "print_name": "Масонская_ложа_🇨🇾_🇮🇲_🇻🇦_🇬🇵_🇪🇺", "admin": {"peer_id": 0, "id": "$01000000000000000000000000000000", "peer_type": "user", "print_name": "user#0"}, "flags": 65537, "members_num": 5}, "unread": false, "text": "ладно: ⚔ Атака, 🇬🇵", "date": 1484136045, "service": false}
{"event": "read", "id": "01000000a6b4ce0fca0000000000000077616c38a95a2f29", "flags": 256, "out": false, "text": "Ты приготовился к атаке. Ближайшее сражение через 59минут. Жди результата.", "date": 1484136042, "service": false, "from": {"peer_id": 265204902, "id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "print_name": "Chat_Wars", "username": "ChatWarsBot", "flags": 1, "first_name": "Chat Wars", "last_name": ""}, "to": {"peer_id": 314226847, "id": "$010000009fb8ba120b62fa795ec02cf3", "when": "2017-01-11 20:26:06", "peer_type": "user", "phone": "79671494739", "print_name": "Mike_Udkovsky", "username": "mikeud", "flags": 524289, "first_name": "Mike", "last_name": "Udkovsky"}, "unread": false}<
Traceback (most recent call last):
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pytg\receiver.py", line 182, in _add_message
    json_dict = json.loads(text)
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\json\__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\json\decoder.py", line 342, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 826)

Your environment

latest telegram-cli from master branch under Windows 10 (cygwin64 or unix subsystem), Python 3.6, pytg 0.4.10

Steps to reproduce

1. Call status_online method after every (yeld)
2. Wait for a long time (3 hours+).

[luckydonald]

Turn on debugging output (see README), what does it tell you?

[magicxor]

Nothing except that I have shown above (it's debug log fragments).

My code looks like this:

def message_loop(options, sender):
    while not stopped:
        try:
            msg = (yield) # <----- EXCEPTION !!!
            sender.status_online(result_timeout=10) 
            ...
        except GeneratorExit:
            pass
        except KeyboardInterrupt:
            pass
        else:
            pass

[luckydonald]

Ah, sorry.
I can see the reason: The first received message are actually two seperate.
...false}\n{"event":...
Because the json root node is already closed } thats all the data it expects.
For some reason the string contains another message as well, after the line break \n

[luckydonald]

Maybe the messages with \n could just splitted into two (probably in _add_message in receiver.py)
But I'm not sure if that could split linebreaks in messages too?

[viacheslav-mik]

I have the same problem. Events like "read" / "unread" come in such a problematic format

[luckydonald]

If someone want to contribute a fix, pull request are welcome!