training error? odd DIST plot #408
Comments
|
Hi @happyqiu, Can you please share your project (.lbl) file and the movie you are trying to track? You will probably need to share a link to a cloud service (eg Google Drive) because these files will be too large to directly attach here. When you track the video, have you looked at the log in the Tracking Monitor? There might be warnings or other messages printed there. If this log is available and you can upload it here that might also be useful. Thanks! |
|
Thanks for your reply.
Please find the link below. (video 20 is for tracking.)
https://drive.google.com/drive/folders/1gu0R8yqWSzz7APH54vOyIpgPIAOqHa_C?usp=sharing
Sorry that I don't have the log messages from the tracking monitor.
…On Tue, Nov 1, 2022 at 8:24 AM Allen Lee ***@***.***> wrote:
Hi @happyqiu <https://github.com/happyqiu>,
Can you please share your project (.lbl) file and the movie you are trying
to track? You will probably need to share a link to a cloud service (eg
Google Drive) because these files will be too large to directly attach here.
When you track the video, have you looked at the log in the Tracking
Monitor? There might be warnings or other messages printed there. If this
log is available and you can upload it here that might also be useful.
Thanks!
—
Reply to this email directly, view it on GitHub
<#408 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMDHPHXM5YVXYNSZ33KPBM3WGEDXDANCNFSM6AAAAAARR3VO24>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Strange, if I try tracking with your trained tracker, I get predictions but they are 'garbage' (all in the upper-left corner). Maybe that is why you don't see them? However, if I retrain, my loss/dist plots look normal and the tracking looks good. Nothing jumps out yet -- maybe if it's not difficult, try doing a fresh retrain to see if anything changes? (Please save the training log just in case.) So far your training data looks normal so I wonder if it could be something in your environment/platform. |
|
That happens...
When I tried this .lbl file with a TITAN X GPU computer, it worked alright.
But with my own computer (NVIDIA RTX A5000), this problem occurred. Does
the GPU matter that much? All other installations are the same on these two
computers.
…On Tue, Nov 1, 2022 at 5:59 PM Allen Lee ***@***.***> wrote:
@happyqiu <https://github.com/happyqiu>
Strange, if I try tracking with your trained tracker, I get predictions
but they are 'garbage' (all in the upper-left corner). Maybe that is why
you don't see them?
However, if I retrain, my loss/dist plots look normal and the tracking
looks good.
Nothing jumps out yet -- maybe if it's not difficult, try doing a fresh
retrain to see if anything changes? (Please save the training log just in
case.) So far your training data looks normal so I wonder if it could be
something in your environment/platform.
—
Reply to this email directly, view it on GitHub
<#408 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMDHPHQLOAA3PHKPWDMZKA3WGGHDJANCNFSM6AAAAAARR3VO24>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
It looks like a compatibility issue with the A5000 may be possible. In develop we are on tf1.15 and see eg https://discuss.tensorflow.org/t/tensorflow-and-cuda-support-for-latest-nvida-a5000-ampere-gpu/3886 @mkabra could @happyqiu have Ampere compatibility issues even if they switch to the multianimal branch? One of these links seems to suggest that tf2.4 is required. In general the specific GPU can potentially matter as in eg #365. |
Hi,
I'm using the APT-develop branch, and found that during the training, the DIST panel didn't look right while the Loss might be okay. Besides, after the training, I tried to track another video, but I didn't get any predicted labels. Do you have any ideas about what's happening here?
Thanks!
The text was updated successfully, but these errors were encountered: