Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for logging images in CHW format #5505

Open
edgarriba opened this issue Mar 14, 2024 · 2 comments
Open

Add support for logging images in CHW format #5505

edgarriba opened this issue Mar 14, 2024 · 2 comments
Labels
enhancement New feature or request 🪵 Log & send APIs Affects the user-facing API for all languages
Milestone

Comments

@edgarriba
Copy link

hi! i'm one of the maintainers of kornia and we are exploring to adopt more rerun as default to visualize.

Today, mainly kornia images are represented by torch tensors (some times batched) with a memory layout CxHxW. We noticed that rerun expected mainly image data in last channel order fashion HxWxC.

In kornia we have some functionality to easily handle such situation via tensor_to_image.

import kornia as K
import rerun as rr

rr.init("kornia_viz", spawn=True)

img = K.io.load_image(
     "image_path.jpeg",
    K.io.ImageLoadType.RGB32,
)

rr.log("image", rr.Image(K.utils.tensor_to_image(img)))

However, would be great if we somehow support that feature by, either adopting that kornia function, or improve the rerun Image type so that the user can customize easily the data layout. As ideas:

# 1 adopt tensor_to_image
rr.log("image", rr.Image(img))

# 2 customize layout
rr.log("image", rr.Image(img, channels_order="first"))

Besides, @johnnv1 has recently proposed to use https://einops.rocks/ which we might adopt also to handle properly visualization in tiles.

@edgarriba edgarriba added enhancement New feature or request 👀 needs triage This issue needs to be triaged by the Rerun team labels Mar 14, 2024
@emilk emilk changed the title Better support for pytorch image tensors Add support for logging images in CHW format Mar 14, 2024
@emilk emilk added 🪵 Log & send APIs Affects the user-facing API for all languages and removed 👀 needs triage This issue needs to be triaged by the Rerun team labels Mar 14, 2024
@emilk emilk added this to the Triage milestone Mar 14, 2024
@nikolausWest
Copy link
Member

Thanks for this issue @edgarriba and exciting to hear that the Kornia team is considering deeper adoption. Do you think it matters to Kornia users whether we do the CHW -> HWC conversion on the SDK side or as late as possible in the viewer. If we do it SDK side, users won't be able to see what the original channel ordering was when inspecting the data in the viewer and any transformation cost will also happen in the same process as the SDK call. We generally strive to not transform users data but doing it SDK side can be a way to ship functionality faster, which can obviously be valuable too

@Wumpf
Copy link
Member

Wumpf commented Oct 22, 2024

note for posterity: we have now planar YUV formats, but everything is every specialized to the exact format type. I think it would be worth exploring planarity as an orthogonal concept. Counter point: that may work with fully planar and fully interleaved formats, but semi-planar formats are very hard to express like this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request 🪵 Log & send APIs Affects the user-facing API for all languages
Projects
None yet
Development

No branches or pull requests

4 participants