-
-
Notifications
You must be signed in to change notification settings - Fork 35.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for USD and USDZ formats #14219
Comments
It's a valid feature request but I personally don't understand Apple/Pixar. Why are they introducing a new file format and not just using |
USD documentation: http://graphics.pixar.com/usd/docs/index.html
I agree, part of it is politics me thinks. However, that's a separate discussion :) |
USD has been around for a couple years — it's commonly used in film, and not an obvious choice for runtime transmission. As far as I can tell USDZ is a wrapper around that. Apple hasn't given much detail at this point; maybe they have some plan to make it more runtime-appropriate, or maybe the messaging at WWDC was just unclear with respect to how they're actually using it. In any case I think we probably need to wait for more information. If someone is interested in contributing a |
Out of curiosity, I tried opening the "Retro TV" file from https://developer.apple.com/arkit/gallery/ as if it was a simple zip. I 'unarchived' the .usdz by renaming it's extension to .zip, and then having my OS do its usual open procedure. Presumably, something similar could be achieved with https://gildas-lormeau.github.io/zip.js/ This is what I see when I 'unarchive' the usdz: So, the textures are just PNGs, which are probably referenced from within that .usdc, which is the next issue. According to https://graphics.pixar.com/usd/docs/Converting-Between-Layer-Formats.html, ".usdc files – binary format" which is in Crate (see Crate File Format in https://graphics.pixar.com/usd/docs/USD-Glossary.html) format. There is also .usda, which is ASCII encoded, which would presumably be easier to write a loader around. As a rough outline of some to-dos for this, minimally a Loader would be looking at extracting a Zip file in JS, and then also understanding the .usda (and potentially binary Crate .usdc as well.) |
Hi @richardmonette, I've done some research on USDZ and the USDA format and have written a proof-of-concept glTF to USDZ converter. Perhaps it is useful for your research: https://github.com/timvanscherpenzeel/gltf-to-usdz. You can see a demo here: https://twitter.com/domenicopanacea/status/1011292393801420800 or try it out yourself here: https://timvanscherpenzeel.github.io/gltf-to-usdz/ (requires iOS 12). In short: geometry gets converted to I know some people are looking into writing a direct converter to USDZ but not a whole lot is known about the |
Hey @TimvanScherpenzeel, thanks for your message and work on this! Have you considered using the USD Python API (https://graphics.pixar.com/usd/docs/Hello-World---Creating-Your-First-USD-Stage.html)? I wondered if it might be possible to skip the .obj stage and, using Python, read the glTF and go straight into .usda/c that way? |
I've had a lot of issues setting up USD and have not yet taken the time to fix the issue (I think it has something to do with having multiple versions of Python installed and during compilation it links to the wrong Python). You could indeed write the OBJ file directly into the USDA file (that could actually be a good next step in my proof of concept). I know the file structure of the AR Gallery examples because a contributor to my project send me the USDA files after having the USDC files decoded it with |
The whole point of Apple providing a Pixar USD, using highly efficient OpenSubDiv Patches for geometry, ported to mobile Metal will soon outperform HighEnd PC-VR WebVR. So IMHO it's more important to export An important issue is that WebXR would have to also support SubDiv patches instead of legacy polymeshes to fully support USD as used by Apple. |
@PixelPartner you sorta lost me. It seems to me that unless a loader is built natively into browsers, it would be helpful to build one in JavaScript.. I’m sure quick look is highly optimized, but what is the path forward for creating interactive content using USDZ assets in a browser? |
I had a chat with someone working on product display software recently, they export to glTF and USDZ so I asked him what he thought about this. His take in a nutshell: it's so that Apple can continue to claim to be the high-end luxury choice, with an exclusive 3D format that only works on their platform via ARKit2.0/SceneKit. Adopting the standard glTF format would run contrary to their goals here, they want to be able to claim that their format is better and allows for higher quality results due to PBR material properties that glTF doesn't (yet?) support. #16977, and perhaps #14051, are likely needed to get three.js up to the same rendering standards as Apple's offerings, but I'm not sure how the current state or future plans for glTF relate to this. In any case, if for no other reasons than to subvert Apple's nefarious goals here, I think we'll need to support USDZ at some point. |
I'm willing to give more benefit of the doubt here – they have different goals for a "runtime" format than we (threejs + web devs) do, and are able to make different choices as a result. In particular, bundling the ~15MB USD runtime into iOS means they don't care that loading USDZ without a native runtime is prohibitively difficult, and they seem to be OK with not providing the web application any access to the model, but just opening the model as-is in a native app instead. On "high-end luxury", Apple's iOS QuickLook viewer supports metallic/roughness PBR materials, and the material model is largely equivalent to what glTF 2.0 supports today. The additional material features planned for glTF 2.0 in the future, and described in #16977, would go beyond what Apple's USDZ viewers support now. I mention "Apple's USDZ viewers" because that is, de facto, what people seem to mean by USDZ. But the USDZ format, technically, is just a zipped USD file, and could contain pretty much anything from a common material model to a completely custom shader. Apple's viewers support a small subset of USD, and that subset isn't well defined, but the best summary you're likely to find is here: https://github.com/google/usd_from_gltf#compatibility. In terms of rendering fidelity, I'd point out that model-viewer displays glTF models with threejs and provides the means to open a USDZ version in iOS Quick Look, and the visual results are quite similar. This page provides some direct comparisons, although unfortunately it is down right now.
See #14219 (comment). |
I just want to mention here that there are reasons for supporting USD beyond Apple "claiming to be a luxury choice". @Mugen87 asks "Can somebody explain what USD/USDZ does better than glTF?"
My use case is loading the same scene description into WebGL and into a high-end rendering pipeline - understanding the looks will differ of course and some substitution will take place. |
@garyo lighting and environment are handled by the AR viewer, in an attempt to simulate the actual environment. What I'd like to see is an USDA exporter from three.js, so you can save models you have created/ configurered in a three project, for viewing in AR. It really is a shame Apple went for a format no-one else uses, and isn't natively supported anywhere (beta exporter is available for Blender). So you have no choice but manually convert all your existing models. The fact that you have to include your textures is also a pain if you use the same texture over an entire range of models. Surely they could have external references? |
wait, is there a way to load usd(z) in webgl? |
The feature set of USDZ recently drastically increased with iOS 13.4 (and RealityComposer 1.4 (133+))
Apple added a lot of USD schemes that currently no other software can interpret and might simply ignore.
Almost the whole interactive functionality of RealityComposer is now not only saved to .reality scene files, but can also be exported in this new USD flavour.
Some new USD features of the top of my head:
Trigger bounding, tap or enter area
grouped Animation actions, in sequence and/or parallel
Text Geometrie is created on the fly by a new type of Prim that takes the string, font and size
This opens a new era. Any developer can now use the Pixar use tools and libs (C++/python 2.6/python 3.x) on any platform to write USD files that can either simply show content or create interactive experiences.
Obviously you still need an iDevice for testing.
-
Best wishes, (just ending quarantine, staying in home office)
*Excited* Thomas
… Am 14.04.2020 um 20:25 schrieb themightyatom ***@***.***>:
@garyo <https://github.com/garyo> lighting and environment are handled by the AR viewer, in an attempt to simulate the actual environment.
This video gives you an idea of what maps they support (albedo, metallic, roughness, normal, ambient occlusion, emissive)
https://developer.apple.com/videos/play/wwdc2018/603/ <https://developer.apple.com/videos/play/wwdc2018/603/>
What I'd like to see is an USDA exporter from three.js, so you can save models you have created/ configurered in a three project, for viewing in AR.
includes a tag for specifying a USDZ model alternative, for when viewed on iOS.
https://modelviewer.dev/ <https://modelviewer.dev/>
It really is a shame Apple went for a format no-one else uses, and isn't natively supported anywhere (beta exporter is available for Blender). So you have no choice but manually convert all your existing models. The fact that you have to include your textures is also a pain if you use the same texture over an entire range of models. Surely they could have external references?
It's really a bit of an hodgepodge format, that only utilises geometry and mapping from the Pixar specs, and zips them together with your image files.
glTF/glb would have been much better and we would only need ONE model for everywhere...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#14219 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAR6X5SXAY32JAPYL5V44TRMSTBTANCNFSM4FDHSXWQ>.
|
I'm not aware of any web-compatible parser for USD or USDZ. Rendering in WebGL or another API isn't a problem, it's parsing the file's content that is complex — that task is best handled by the official USD runtime, which has dependencies including OpenSubdiv and MaterialX.
Being able to read USD files on any platform would be nice. 🙂 For technical reasons I do not think a functional three.js implementation is likely right now. If someone is blessed with the spare time, my suggestion would be to work on implementing OpenSubdiv (w/ WASM?) or MaterialX (w/ THREE.NodeMaterial?), as those would be useful in their own right, even without a USD parser. |
No, that's what I'm looking for. I have scenes (object, lights, materials, camera, animation, etc. - the whole thing) and I want to be able to load the same scene, in whatever format, into WebGL (via three.js) for preview, and something like Blender for final renders. glTF is my best option for now, but it doesn't support the things I mention above (area lights etc.) I understand the glTF folks are reluctant to add more features because they're targeting last-mile to devices rather than true scene transfer. |
@garyo Sounds like Collada might fit the bill? I use an edited version of the collada exporter from three.js to take assets into Blender for renders. |
@themightyatom I looked into Collada a couple of years ago -- may have to revisit it. Thanks. (At the time I was convinced by arguments such as the Godot one, no PBR support, etc -- that may have changed.) |
@garyo Collada was designed for transferring assets around between game developers all working on different platforms. It's not a viable delivery format, but it does what it's meant for very well :) |
I think what would be super valuable is to integrate https://github.com/google/usd_from_gltf as an exporter. Much less important to me to be able to open USD. |
To do anything useful with USDZ (load or export) you are going to need the USD SDK. USD from glTF requires the USD SDK as a dependency, but the SDK isn't supported on the web. Apple gets around this by leaving the web browser and taking you into their native AR Quick Look application, but that's hardly a solution for the web in general. |
Has anyone figured out something yet? Or it's literally impossible at the moment? I'm looking forward to use for the following pipeline:
It works great on desktop and non-iOS devices. And I cannot rely on servers converting the files, because it's all done client-side. User generates the scene on the web that needs to be displayed in AR. |
What desktop? |
MacOS+Chrome (Obvs AR not being used) Talking about |
WASM should work in iOS Mobile Safari since iOS 11 (says caniuse.com.) |
@PixelPartner it runs on Mac Safari because latest Safari versions now support SharedArrayBuffer / Atomics objects, which this implementation uses. @jeffscottward @darkvertex Doesn't seem to work on my iphone with iOS 15.5 either and it's pretty hard to debug as i don't have a mac. I'll take a look when i have time. WASM is of course supported on iOS alongside with WebGL2 in latests versions so this is not the cause of the issue. |
On iOS there are some experimental settings that might need enabling, but I also would need to investigate. I don’t currently have access to my computer or a USD file. If anyone has one, I could experiment while I am travelling. |
@ajmas I have tried this model: And on web, it seems to be working fine. Also, https://sketchfab.com/ offers auto-generated USDZ files with each model (free and paid ones), so that is a good source of models too. |
iOs is throwing a RangeError: Out of memory. We've seen this error before, its a bug in safari where the new thread support requires a little bit extra memory on startup than it thinks it does. It should load fine if you increase the initial memory to the wasm module |
We fixed it by tweaking the compile flags in emscripten like this (your values may vary!) |
@MAG-AdrianMeredith I can confirm that on iOS the error is RangeError: Out of memory. For some reason the WASM module was trying to assign more memory that it could. I did put a memory limit for iOS devices on three-usdz-loader latest version and it now works like a charm on my iPhone 12 Pro with iOS 15.5. @netgfx i tried with your model and it works pretty well now ! |
I believe there's an actual bug ticket in safari / WebKit somewhere. Not
on my computer right now but can dig it out on Monday if needed.
…On Sat, 25 Jun 2022, 05:22 Jeff Scott Ward, ***@***.***> wrote:
Great work @ponahaum!
So that's the problem, how do we describe a solution in a PR to AutoCAD?
I'm unfamiliar with emacripten workflows. I imagine there is no way to
dynamically detect what that range should be to stop the error, hmmm.
—
Reply to this email directly, view it on GitHub
<#14219 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AX35EVNKIVWIKGNVI2QLLWLVQ2CP3ANCNFSM4FDHSXWQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I looked into this file and the format is very strange. The points and texture coord arrays have different lengths and have individual index buffers. Edit: same with this one:
|
Here's a very naive first version of the I've only tried it with the models shared in this thread. It only supports The next steps are to gather as many sample files as we can (using |
I just I generated my own No dice as expected with this loader. Just got silent fail of an empty object. After getting USDTools working on Apple Silicon (https://developer.apple.com/forums/thread/702189?answerId=711711022#711711022) - I unzipped the The I don't know if this at all helps but thought I would run it through. Here is the PS: The model is supposed to be like wearing a yellow Super Saiyan wig. |
This animated flag pole uses SkelAnim to move the flag. If you need an example for that |
This seems to be the new way Apple accepts Object placement since the latest update of RealityConverter.
|
That link doesn't work for me... I think you should be able to zip the usdz file and attach the zip to your post here. |
Attaching the file here in case those links die: ssj2-face.usdz.zip |
Yea I figured, it was just a test to see if converting to USDA would help
at all for parsing it.
On Thu, Sep 1, 2022 at 7:14 PM mrdoob ***@***.***> wrote:
That file uses a usdc internally. We can't parse these yet.
[image: Screen Shot 2022-09-01 at 4 14 12 PM]
<https://user-images.githubusercontent.com/97088/188027885-48393709-8136-4b21-a564-5ebd42b7b0f0.png>
—
Reply to this email directly, view it on GitHub
<#14219 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJUO7TEVR2SSDEP6AQ26H3V4E2GLANCNFSM4FDHSXWQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
*Thanks,*
*Jeff Ward*
*Front-end Developer*
*Tel: 516-551-8624*
***@***.*** ***@***.***>*
***@***.**** <https://twitter.com/#!/jeffscottward>
|
But it doesn't seem like it was actually converted to usda? 🤔 |
I did that on my machine. This is just the original starting point USDz.
Going back to USDz but from USDA had no effect.
On Thu, Sep 1, 2022 at 9:17 PM mrdoob ***@***.***> wrote:
But it doesn't seem like it was actually converted to usda? 🤔
—
Reply to this email directly, view it on GitHub
<#14219 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJUO7RRZ573YPNFT73A6BLV4FITFANCNFSM4FDHSXWQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
*Thanks,*
*Jeff Ward*
*Front-end Developer*
*Tel: 516-551-8624*
***@***.*** ***@***.***>*
***@***.**** <https://twitter.com/#!/jeffscottward>
|
|
Worth mentioning a new joint effort to further the ecosystem with this new alliance. Pixar Adobe Apple Autodesk Nvidia Epic Unity Cesium and others |
I recently came across tinyusdz which claims to support compilation to WASM. I should note that I've used neither three.js nor the library, but maybe this is useful for people looking into this. |
Since Please file module specific feature requests or bug reports in new issues. |
yep modelviewer.dev has always had usdz support ("free quicklook" no iOS/VIsionOS) via three.js even though they are advocating for glb/gltf |
It would be great to get Three.js support for the Universal Scene Description (USD) file format and the upcoming USDZ format announced at WWDC today.
USDZ specification document: https://graphics.pixar.com/usd/files/USDZFileFormatSpecification.pdf (I assume that more details will come out in the coming months for this format)
The text was updated successfully, but these errors were encountered: