Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why does traffic analysis output differ from the video in readme file? #1594

Open
1 task done
INF800 opened this issue Oct 13, 2024 · 24 comments
Open
1 task done

Why does traffic analysis output differ from the video in readme file? #1594

INF800 opened this issue Oct 13, 2024 · 24 comments
Assignees
Labels
question Further information is requested

Comments

@INF800
Copy link

INF800 commented Oct 13, 2024

Search before asking

  • I have searched the Supervision issues and found no similar feature requests.

Question

I tried running the same code in readme to generate video outputs but accuracy is not upto mark.

Here are videos with different thresholds:

https://drive.google.com/drive/folders/1TFcEJcSvVSQXaMEYQTnhw2s-QNXP-LOz?usp=sharing

Additional

No response

@INF800 INF800 added the question Further information is requested label Oct 13, 2024
@onuralpszr onuralpszr self-assigned this Oct 13, 2024
@INF800
Copy link
Author

INF800 commented Oct 13, 2024

FYI, I used roboflow API instead of running it locally.

cc: @onuralpszr

@INF800
Copy link
Author

INF800 commented Oct 14, 2024

I see that default model id is vehicle-count-in-drone-video/6 was it supposed to be 8 instead of 6

cc: @onuralpszr

@INF800
Copy link
Author

INF800 commented Oct 14, 2024

8 is giving resource error

@onuralpszr
Copy link
Collaborator

I see that default model id is vehicle-count-in-drone-video/6 was it supposed to be 8 instead of 6

cc: @onuralpszr

That is dataset not model so default 6 is correct

@onuralpszr
Copy link
Collaborator

8 is giving resource error

v8 as model not exist so it is correct

@onuralpszr
Copy link
Collaborator

FYI, I used roboflow API instead of running it locally.

cc: @onuralpszr

Out of curiosity what is your inference version ?

@INF800
Copy link
Author

INF800 commented Oct 14, 2024

It's the same as in requirements.txt file

Screenshot 2024-10-14 at 7 14 17 AM

@INF800
Copy link
Author

INF800 commented Oct 14, 2024

I got this warning message before running the script:

[10/14/24 07:10:56] WARNING  Your inference package version 0.9.17 is out of date! Please upgrade to version 0.23.0 of inference for    __init__.py:35
                             the latest features and bug fixes by running `pip install --upgrade inference`.

but ignored it anyway because that's what is being used in requirements.txt

cc: @onuralpszr

@INF800
Copy link
Author

INF800 commented Oct 14, 2024

Just now I trained a new model using my custom data (just 2 4k images) and its results are wayyy better that current one

@INF800
Copy link
Author

INF800 commented Oct 14, 2024

Just now I trained a new model using my custom data (just 2 4k images) and its results are wayyy better that current one

here are the results: https://drive.google.com/drive/folders/1bzHpSBSRi3nbFa7i-BdMZjTRbpqnp3nc?usp=sharing

ccL @onuralpszr

@onuralpszr
Copy link
Collaborator

onuralpszr commented Oct 14, 2024

@PawelPeczek-Roboflow can you help me out on this one please I was able to generate the same problematic video and I tried version in written req.txt and I tried latest version of inference to test and my thought was current code is not using conf and iou values then I used latest version with parameters I checked in documentation and still same results. Either model in roboflow universe problematic or inference code doing something weird

Latest version of inference changes I made in example code (code snippet not entire code)

from inference import get_model
...
self.model = get_model(model_id=model_id, api_key=roboflow_api_key)
...
results = self.model.infer(
    image=frame, confidence=self.conf_threshold, iou_threshold=self.iou_threshold
)[0]

cc @LinasKo
cc @SkalskiP (since you wrote initially you might be have better understanding of model side)

@onuralpszr
Copy link
Collaborator

I got this warning message before running the script:

[10/14/24 07:10:56] WARNING  Your inference package version 0.9.17 is out of date! Please upgrade to version 0.23.0 of inference for    __init__.py:35
                             the latest features and bug fixes by running `pip install --upgrade inference`.

but ignored it anyway because that's what is being used in requirements.txt

cc: @onuralpszr

I tested on both cases yes and I did the same thing too

@onuralpszr
Copy link
Collaborator

Just now I trained a new model using my custom data (just 2 4k images) and its results are wayyy better that current one

here are the results: https://drive.google.com/drive/folders/1bzHpSBSRi3nbFa7i-BdMZjTRbpqnp3nc?usp=sharing

ccL @onuralpszr

You used inference local model run right ?

@INF800
Copy link
Author

INF800 commented Oct 14, 2024

Just now I trained a new model using my custom data (just 2 4k images) and its results are wayyy better that current one

here are the results: https://drive.google.com/drive/folders/1bzHpSBSRi3nbFa7i-BdMZjTRbpqnp3nc?usp=sharing
ccL @onuralpszr

You used inference local model run right ?

Nope, used Roboflow API with the same inference code as is with same version dependencies. The only change that I made is - replacing the model id with my newly trained model ID.

Based on this I think everything working just fine at code level.

@onuralpszr
Copy link
Collaborator

Just now I trained a new model using my custom data (just 2 4k images) and its results are wayyy better that current one

here are the results: https://drive.google.com/drive/folders/1bzHpSBSRi3nbFa7i-BdMZjTRbpqnp3nc?usp=sharing
ccL @onuralpszr

You used inference local model run right ?

Nope, used Roboflow API with the same inference code as is with same version dependencies. The only change that I made is - replacing the model id with my newly trained model ID.

Based on this I think everything working just fine at code level.

For code level yes It should be fine that I am sure I am converting this to Q&A discussion and continue from there.

Also you could you share your ID if it is public please

@INF800
Copy link
Author

INF800 commented Oct 14, 2024

Yes the one that I used for this cusom-data/2 (with the typo).

cc: @onuralpszr

I'm using the model for running some inference jobs right now (15mins more remaining).

@PawelPeczek-Roboflow
Copy link
Contributor

@onuralpszr do not really know the context, could you elaborate a little bit more?

@LinasKo
Copy link
Contributor

LinasKo commented Oct 14, 2024

Hi @INF800 👋

Based on the video, it's likely that InferenceSlicer was used to help detect small objects. Have you tried using it?
(guide, docs)

Next, it would help if you could share some code you used to run the model. Specifically, I'm curious what code you ran that produced poor results initially.

@onuralpszr
Copy link
Collaborator

onuralpszr commented Oct 14, 2024

@onuralpszr do not really know the context, could you elaborate a little bit more?

In basic context, inference example in here https://github.com/roboflow/supervision/tree/develop/examples/traffic_analysis doesn't show output of video in readme.

@PawelPeczek-Roboflow
Copy link
Contributor

ok, taken a look - first of all, the model trained for ultralytics is a different class of model than the pointed one from Roboflow - not sure if the results were comparable in first place

@LinasKo
Copy link
Contributor

LinasKo commented Oct 14, 2024

Hey @PawelPeczek-Roboflow, what do you mean by "a different class of model"? What kind of difference are we looking at? Model family, size, or a different dataset?

@PawelPeczek-Roboflow
Copy link
Contributor

not fully sure about details, but the weights size differ:

  • one order of magnitude in favour of ultralytics model (it also runs 5x slower)
  • additionally model from universe was trained on 100 images - we shall not expect it to be robust

@PawelPeczek-Roboflow
Copy link
Contributor

also the Roboflow model was not that great in the metrics of original dataset
image

@LinasKo
Copy link
Contributor

LinasKo commented Oct 15, 2024

Curious.

@INF800, data points to either the video being used as an example, or additional tool use such as InferenceSlicer. For comparable results, I suggest using or fine-tuning one of yolov8 models from inference. If the objects are this small, I also suggest Inference Slicer (guide, docs).

I'll mark this as potentially misleading and investigate further later on. Thank you for reporting it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

4 participants