Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: after update #81 the program freezes half way #87

Closed
40OIL opened this issue Jun 18, 2023 · 24 comments
Closed

[Bug]: after update #81 the program freezes half way #87

40OIL opened this issue Jun 18, 2023 · 24 comments
Labels
bug Something isn't working

Comments

@40OIL
Copy link

40OIL commented Jun 18, 2023

telegram_media_downloader version or commit

2.1.9 from 2023-06-18

What OS are you seeing the problem on?

Windows

Python Version

3.8.10

What happened?

A bug happened!

maybe today's update of WebView2 Runtime Edge influenced?
rolling back to 2.1.9 from 2023-05-16 did not help.
or is it that I run "pip3 install -r requirements.txt" after each download of py files?

2023-06-18 20:47:28.466 | INFO | utils.meta:print_meta:22 - Device: CPython 3.8.10 - Telegram Media Downloader 2.1.9
2023-06-18 20:47:28.467 | INFO | utils.meta:print_meta:23 - System: Windows 10 (EN)
2023-06-18 20:47:29.781 | SUCCESS | main:main:591 - Successfully started (Press Ctrl+C to stop)
2023-06-18 20:47:29.782 | INFO | main:download_chat_task:505 - Downloading files failed during last run...
2023-06-18 20:47:29.855 | ERROR | main:worker:489 - 'NoneType' object has no attribute 'id'
Traceback (most recent call last):

File "media_downloader.py", line 628, in
main()
└ <function main at 0x000002445F995AF0>

File "media_downloader.py", line 602, in main
_exec_loop()
└ <function _exec_loop at 0x000002445F995A60>

File "media_downloader.py", line 569, in _exec_loop
app.loop.run_until_complete(run_until_all_task_finish())
│ │ │ └ <function run_until_all_task_finish at 0x000002445F9959D0>
│ │ └ <function BaseEventLoop.run_until_complete at 0x000002445BDBAC10>
│ └
└ <module.app.Application object at 0x000002445F97B520>

File "C:\Users\user1\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 603, in run_until_complete
self.run_forever()
│ └ <function ProactorEventLoop.run_forever at 0x000002445BE8A430>

File "C:\Users\user1\AppData\Local\Programs\Python\Python38\lib\asyncio\windows_events.py", line 316, in run_forever
super().run_forever()

File "C:\Users\user1\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 570, in run_forever
self._run_once()
│ └ <function BaseEventLoop._run_once at 0x000002445BDBF700>

File "C:\Users\user1\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 1859, in _run_once
handle._run()
│ └ <function Handle._run at 0x000002445BD2B310>
└ <Handle <TaskWakeupMethWrapper object at 0x000002445FC3DD60>()>

File "C:\Users\user1\AppData\Local\Programs\Python\Python38\lib\asyncio\events.py", line 81, in _run
self._context.run(self._callback, *self._args)
│ │ │ │ │ └ <member '_args' of 'Handle' objects>
│ │ │ │ └ <Handle <TaskWakeupMethWrapper object at 0x000002445FC3DD60>()>
│ │ │ └ <member '_callback' of 'Handle' objects>
│ │ └ <Handle <TaskWakeupMethWrapper object at 0x000002445FC3DD60>()>
│ └ <member '_context' of 'Handle' objects>
└ <Handle <TaskWakeupMethWrapper object at 0x000002445FC3DD60>()>

File "media_downloader.py", line 487, in worker
await download_task(client, message, node)
│ │ │ └ <module.app.TaskNode object at 0x000002445FC197F0>
│ │ └ pyrogram.types.Message(id=16999, empty=True)
│ └ <pyrogram.client.Client object at 0x000002445F9B3A60>
└ <function download_task at 0x000002445F995550>

File "media_downloader.py", line 271, in download_task
download_status, file_name = await download_media(
└ <function download_media at 0x000002445F995670>

File "K:\Exec\tangyoha-tmd\module\pyrogram_extension.py", line 329, in inner
status, file_name = await func(
└ <function download_media at 0x000002445F9955E0>

File "media_downloader.py", line 357, in download_media
message = await fetch_message(client, message)
│ │ └ pyrogram.types.Message(id=16999, empty=True)
│ └ <pyrogram.client.Client object at 0x000002445F9B3A60>
└ <function fetch_message at 0x000002445E395820>

File "K:\Exec\tangyoha-tmd\module\pyrogram_extension.py", line 492, in fetch_message
chat_id=message.chat.id,
│ └ None
└ pyrogram.types.Message(id=16999, empty=True)

AttributeError: 'NoneType' object has no attribute 'id'
2023-06-18 20:47:31.481 | SUCCESS | main:_check_download_finish:77 - Successfully downloaded - K:\Exec\tangyoha-tmd\document\83932 - Золушка для инквизитора.fb2
2023-06-18 20:47:31.485 | SUCCESS | main:_check_download_finish:77 - Successfully downloaded - K:\Exec\tangyoha-tmd\document\83933 - 03_Олигарх_из_будущего_Книга_3_Миллионы_рождают_власть.fb2

here the program hangs and waits until you interrupt it manually <<<<<<<<<<

2023-06-18 20:47:50.291 | INFO | main:main:604 - KeyboardInterrupt
2023-06-18 20:47:50.381 | INFO | main:main:612 - Stopped!
2023-06-18 20:47:50.768 | INFO | main:main:614 - update config......
2023-06-18 20:47:50.772 | SUCCESS | main:main:618 - Updated last read message_id to config file,total download 2, total upload file 0

Expected Behavior

how to return the normal operation of the program?

Current Behavior

download stops, it is not clear how many messages have been processed and how many are left

Possible Solution

No response

Steps to Reproduce

No response

Relevant log output

No response

@40OIL 40OIL added the bug Something isn't working label Jun 18, 2023
@tangyoha
Copy link
Owner

Use bot?

@40OIL
Copy link
Author

40OIL commented Jun 19, 2023

no, only client

@tangyoha
Copy link
Owner

like that?

api_hash: your_api_hash
api_id: your_api_id
chat:
- chat_id: telegram_chat_id
  last_read_message_id: 0
  download_filter: message_date >= 2022-12-01 00:00:00 and message_date <= 2023-01-17 00:00:00

@40OIL
Copy link
Author

40OIL commented Jun 19, 2023

my config:

api_hash: 11111111111111111111
api_id: 2222222
chat:
- chat_id: -1003333333333
  last_read_message_id: 17017
- chat_id: -1004444444444
  last_read_message_id: 84065
file_formats:
  document:
  - all
media_types:
- document
save_path: D:\Exec\telegram_media_downloader\tangyoha\telegram_media_downloader
language: EN
disable_syslog: []
file_path_prefix:
- media_type

on another computer, media_downloader.py dated 2023-06-08, size 19653 bytes, did not update either py files or requirements.txt - everything works here.

could it be that i updated your project incorrectly?

@40OIL
Copy link
Author

40OIL commented Jun 19, 2023

still write the correct instructions, for those who see git for the first time.

git reset --hard
git pull origin master
pip install -r requirements.txt

and everything worked again without errors, on the branch 2516fe2

how to update to the latest version??????

@tangyoha
Copy link
Owner

This problem is caused by you trying to download a message that does not exist, I will fix this problem as soon as possible, if you want to avoid this problem, please empty your data.yaml

@40OIL
Copy link
Author

40OIL commented Jun 20, 2023

how to update your program under windows?
maybe between these commands it is still worth doing "git fetch"?
image
Or is it done in some other way?
I'm using git for the first time in my life. and I still want it to be without many hours of googling, only from your instructions everything was clear.

@tangyoha
Copy link
Owner

cd telegram_media_downloader
git pull
pip3 install -r requirements.txt

@tangyoha
Copy link
Owner

tangyoha commented Jun 20, 2023

You can discuss with us in the telegram group

https://t.me/TeegramMediaDownload

If you like the project, please give the project a star.

@40OIL
Copy link
Author

40OIL commented Jun 20, 2023

updated "git pull" to "master_fix_87_download_error"
the download freezes if at least one channel/chat has no new messages since the previous launch.

2023-06-20 07:20:23.693 | INFO     | utils.meta:print_meta:22 - Device: CPython 3.8.10 - Telegram Media Downloader 2.1.9
2023-06-20 07:20:23.694 | INFO     | utils.meta:print_meta:23 - System: Windows 10 (EN)
2023-06-20 07:20:25.119 | SUCCESS  | __main__:main:594 - Successfully started (Press Ctrl+C to stop)
2023-06-20 07:20:25.120 | INFO     | __main__:download_chat_task:510 - Downloading files failed during last run...
2023-06-20 07:20:25.442 | INFO     | __main__:download_chat_task:510 - Downloading files failed during last run...
2023-06-20 07:20:26.715 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\84178 - 10 Выживальщики 10. Порталы.fb2
2023-06-20 07:20:26.774 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\84180 - 04 Звездная Кровь-4. Тинг.fb2
2023-06-20 07:20:26.825 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\84179 - 02 Вильгельм-2. Проклятие Саана.fb2
-------------------  interrupted here Ctrl-C
2023-06-20 07:22:15.203 | INFO     | __main__:main:607 - KeyboardInterrupt
2023-06-20 07:22:15.479 | INFO     | __main__:main:615 - Stopped!
2023-06-20 07:22:15.866 | INFO     | __main__:main:617 - update config......
2023-06-20 07:22:15.870 | SUCCESS  | __main__:main:621 - Updated last read message_id to config file,total download 3, total upload file 0

(there were no updates in one chat, 3 books were downloaded from the second)
------------ second start

2023-06-20 07:23:49.415 | INFO     | utils.meta:print_meta:22 - Device: CPython 3.8.10 - Telegram Media Downloader 2.1.9
2023-06-20 07:23:49.416 | INFO     | utils.meta:print_meta:23 - System: Windows 10 (EN)
2023-06-20 07:23:50.889 | SUCCESS  | __main__:main:594 - Successfully started (Press Ctrl+C to stop)
2023-06-20 07:23:50.890 | INFO     | __main__:download_chat_task:510 - Downloading files failed during last run...
------------ interrupted here Ctrl-C
2023-06-20 07:25:30.093 | INFO     | __main__:main:607 - KeyboardInterrupt
2023-06-20 07:25:30.230 | INFO     | __main__:main:615 - Stopped!
2023-06-20 07:25:30.585 | INFO     | __main__:main:617 - update config......
2023-06-20 07:25:30.588 | SUCCESS  | __main__:main:621 - Updated last read message_id to config file,total download 0, total upload file 0

(there were no updates in both chats)

and ids_to_retry entries appeared in data.yaml for both chats

ids_to_retry: []
chat:
- chat_id: -1001347999469
  ids_to_retry:
  - 17038
- chat_id: -1001160434207
  ids_to_retry:
  - 84181

perhaps these messages no longer exist?
if you delete them manually from data.yaml, then the program runs fine

@tangyoha
Copy link
Owner

what means freezes?

@40OIL
Copy link
Author

40OIL commented Jun 20, 2023

where in the log I marked "interrupted here Ctrl-C" - it just stands and does nothing, nothing appears in the log or on the screen
waited 2 minutes both times

@tangyoha
Copy link
Owner

tangyoha commented Jun 20, 2023

You open the website to see if it is downloading

localhost:5000

@40OIL
Copy link
Author

40OIL commented Jun 20, 2023

I erased ids_to_retry, 1st launch - downloads new messages and ends normally.
localhost:5000 works, writes "no data" on both tabs.
image

2023-06-20 09:14:59.490 | INFO     | utils.meta:print_meta:22 - Device: CPython 3.8.10 - Telegram Media Downloader 2.1.9
2023-06-20 09:14:59.491 | INFO     | utils.meta:print_meta:23 - System: Windows 10 (EN)
2023-06-20 09:15:01.007 | SUCCESS  | __main__:main:594 - Successfully started (Press Ctrl+C to stop)
2023-06-20 09:15:02.302 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\17040 - [RPF] Пламя Севера (L).fb2
2023-06-20 09:15:02.388 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\84183 - 10 Граф Суворов, том 10.fb2
2023-06-20 09:15:02.756 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\17039 - [RPF] Граф Суворов, том 10 (H).fb2
2023-06-20 09:15:02.780 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\84184 - 04 Сопряжение 4. Маверик.fb2
2023-06-20 09:15:02.893 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\84182 - 02 Семья волшебников. Том 2.fb2
2023-06-20 09:15:03.993 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\84185 - 07_Я_покорю_Лабиринт_с_помощью_техник_массажа_Том_7.fb2
2023-06-20 09:15:05.100 | INFO     | __main__:main:615 - Stopped!
2023-06-20 09:15:05.460 | INFO     | __main__:main:617 - update config......
2023-06-20 09:15:05.464 | SUCCESS  | __main__:main:621 - Updated last read message_id to config file,total download 6, total upload file 0

2nd run - ids_to_retry reappear, exits normally.
ids_to_retry accepts number = N+1, 1 more than the last existing message in the chat
localhost:5000 works, writes "no data" on both tabs.

- chat_id: -1001347999469
  ids_to_retry:
  - 17041
- chat_id: -1001160434207
  ids_to_retry:
  - 84186

2023-06-20 09:16:45.588 | INFO     | utils.meta:print_meta:22 - Device: CPython 3.8.10 - Telegram Media Downloader 2.1.9
2023-06-20 09:16:45.589 | INFO     | utils.meta:print_meta:23 - System: Windows 10 (EN)
2023-06-20 09:16:47.001 | SUCCESS  | __main__:main:594 - Successfully started (Press Ctrl+C to stop)
2023-06-20 09:16:48.164 | INFO     | __main__:main:615 - Stopped!
2023-06-20 09:16:48.534 | INFO     | __main__:main:617 - update config......
2023-06-20 09:16:48.538 | SUCCESS  | __main__:main:621 - Updated last read message_id to config file,total download 0, total upload file 0

3rd launch - new files are downloaded, but freezes in an incomprehensible place, ids_to_retry was removed for the chat in which the files were downloaded, for the one in which there were no new messages it remained.
localhost:5000 shows blank screen, status "Loadind..."
waited ~5 minutes
does the program wait for N+1 messages to appear in the chat?

2023-06-20 09:18:35.375 | INFO     | utils.meta:print_meta:22 - Device: CPython 3.8.10 - Telegram Media Downloader 2.1.9
2023-06-20 09:18:35.376 | INFO     | utils.meta:print_meta:23 - System: Windows 10 (EN)
2023-06-20 09:18:36.695 | SUCCESS  | __main__:main:594 - Successfully started (Press Ctrl+C to stop)
2023-06-20 09:18:36.695 | INFO     | __main__:download_chat_task:510 - Downloading files failed during last run...
2023-06-20 09:18:36.841 | INFO     | __main__:download_chat_task:510 - Downloading files failed during last run...
2023-06-20 09:18:38.055 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\84186 - 04_Пламя_Севера_Роман_Алексея_Осадчука.fb2
2023-06-20 09:18:38.096 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\84187 - 02 Младший научный сотрудник-2.fb2
2023-06-20 09:18:38.162 | SUCCESS  | __main__:_check_download_finish:79 - Successfully downloaded - K:\Exec\telegram_media_downloader\document\84188 - 11 Чекист. Венский вальс.fb2
------------ interrupted here Ctrl-C
2023-06-20 09:24:46.587 | INFO     | __main__:main:607 - KeyboardInterrupt
2023-06-20 09:24:46.659 | INFO     | __main__:main:615 - Stopped!
2023-06-20 09:24:47.038 | INFO     | __main__:main:617 - update config......
2023-06-20 09:24:47.042 | SUCCESS  | __main__:main:621 - Updated last read message_id to config file,total download 3, total upload file 0

@tangyoha
Copy link
Owner

You mean it should exit normally after the download is over, right?

@40OIL
Copy link
Author

40OIL commented Jun 20, 2023

yes
this is work in client mode, not a bot to be constantly running

@tangyoha
Copy link
Owner

Ok, I know the reason, because we will judge whether each channel has been downloaded, and this judgment is problematic now, I will submit it to a new branch together, are there any other problems?

@40OIL
Copy link
Author

40OIL commented Jun 20, 2023

So far I haven't noticed any other problems.

@tangyoha
Copy link
Owner

  • chat_id: -1001347999469
    ids_to_retry:
    • 17041
  • chat_id: -1001160434207
    ids_to_retry:
    • 84186

17041 or 84186 not exist?

@40OIL
Copy link
Author

40OIL commented Jun 20, 2023

if you delete ids_to_retry, then the program sets it to N+1 for the chat in which no files were downloaded in the current run.
so both 17041 and 84186 are non-existent N+1 messages

@40OIL
Copy link
Author

40OIL commented Jun 20, 2023

image
here are the latest post numbers boxed

@40OIL
Copy link
Author

40OIL commented Jun 20, 2023

Also, messages deleted in the chat also fall into ids_to_retry:

ids_to_retry: []
chat:
- chat_id: -1001111111111
  ids_to_retry:
  - 17041
  - 17042
  - 17045
- chat_id: -1002222222222
  ids_to_retry: []

specially went to the chat and copied the links to the latest existing messages:
https://t.me/c/1111111111/17047
https://t.me/c/1111111111/17046
https://t.me/c/1111111111/17044
https://t.me/c/1111111111/17043
https://t.me/c/1111111111/17040
https://t.me/c/1111111111/17039
https://t.me/c/1111111111/17038 etc

the program also freezes on them.
the previous version somehow handled such non-existent messages and removed them from ids_to_retry.

@40OIL
Copy link
Author

40OIL commented Jun 24, 2023

there is no way to roll back to the point 2516fe2 now?

git pull https://github.com/tangyoha/telegram_media_downloader 2516fe2
fatal: couldn't find remote ref 2516fe2

that option suited me better than d0d576f

git branch -vv
* (HEAD detached from 2516fe2) d0d576f fix:: always download the last file #81 (#86)
   master 2516fe2 [origin/master: behind 10] doc: update pay

now either the program stops, or you have to overwrite data.yaml with a clean one every run

@tangyoha
Copy link
Owner

I will fix it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants