Depth2Img model support: resolves #5372, partially addresses #5011 #5542
Conversation
|
Does it support 768 model natively? |
? |
JaySmithWpg
force-pushed
the
depth2img
branch
from
a606e51
to
1ed4f0e
Compare
|
I found a bug where it fails if the midas/models directory doesn't already exist. It's fixed up now, pardon the late commit and merge.
As LieDeath mentioned, there isn't a 768 model for depth yet. That said, it works well at larger resolutions. The depth map gives it a good structure to follow, similar to how hires fix works. I can't promise anything, but in theory this should be able to handle other models as long as they are using LatentDepth2ImageDiffusion and MiDaS. |
|
Testing now and it's working for me! Thanks for this, I know a lot of people were excited about this being implemented Also your OP in this thread has a typo at the time I'm posting this, it says to rename the config file to 512-depth-ema.ckpt. Should say to rename it to 512-depth-ema.yaml |
|
Great catch on the typo, thanks! |
|
I confirm this works. It's so early that I have a hard time grasping how vast the landscape is behind that door you just opened. |
|
I get this error when trying to load it in the img2img tab: Loading config from: A:\Desktop\00 AI Images\stable-diffusion-webui\models\Stable-diffusion\SD 2.0\Standard\512-depth-ema.yaml The file may be malicious, so the program is not going to read it. Traceback (most recent call last): |
|
Hi Patrick! I'm not sure if you're the same person I was talking to on Reddit about this, but just in case you're not, that error is what happens if you try to follow the instructions to use the model without first applying the code changes contained in this pull request. If/when this PR is merged into the main code by automatic1111, it should work for you. |
|
Is it forced to use dpt_hybrid or is it not possible to use with other models? also what about multi-resolution merging https://github.com/compphoto/BoostingMonocularDepth |
|
As currently implemented, this only allows autogenerated depth maps, right? Definitely going to want to be able to manually provide a depth map too (this guy had a quick hack for it). Not saying that necessarily has to be part of this PR though. |
|
@TingTingin Currently it's forced to use dpt_hybrid. I couldn't find a nice way to interrogate the model for what it uses at that point in the code, but I might revisit that if I have time tomorrow. Multi-resolution merging looks awesome, but one step at a time. @MrCheeze , that guy is me! Currently it only allows for auto-generated depth maps, as anything more complicated will require some user interface changes. I really do want to see added support for manual depth maps, they're incredibly powerful. But I won't have much time to implement it myself until after the holidays. I hope somebody beats me to it. |
it wasn't me that was talking to you over on Reddit, but thanks for the reply, I'm super new to using git on windows, any chance you could point me in the direction on how to go about temporarily using your fix until it gets implemented? |
|
This is nice. Seems to work without any issues. |
Simple error with download bad file, isn't it? |
|
Thanks for the merge! |
|
I don't suppose there's any way to merge the depth-enabled model with any pre-SD2.0 models, or some other trick to use this feature on them? |
|
Same Error as @LieDeath followed instructions so not really sure about this one. |
Are you sure??? I have successful run it rightnow-_-|| Have you already updated webui? |
|
@LieDeath Fixed yes I did update, Error was caused by dpt_hybrid-midas-501f0c75.pt File got corrupted during the download I believe and no way to find that out had to redownload and now seems fine. |
Well, OK. 😂 |
|
I can't use it in the free Colab :( Loading weights [d0522d12] from /content/gdrive/MyDrive/sd/stable-diffusion-webui/models/Stable-diffusion/512-depth-ema.ckpt |
|
Hey, thanks a ton for this useful update. |
|
|
|
Good question! I couldn't tell you for sure, but I'm guessing even with denoising set to one there might be enough general colour information left for the AI to latch onto.
That said, this isn't set up for you to provide your own depth map (yet), it tries to use the image you give it to automatically figure out the depth itself. And it usually does a great job of it.
Try giving it a more colourful rendering of your character and see how it fares.
I would love to have the option to provide my own depth map - I even hacked together a rough proof of concept (somebody else linked to it above), but the AI still does a good job by itself.
|
I found the hack on reddit yesterday and I tried it (successfully modified processing.py) but I could not understand where I should put my depthmap for it to be read. |
In the hack, there's a line that reads depth_img = Image.open("/home/jay/Pictures/AI/Turrent Room/depth.png")Replace that path with where the depthmap is. That said, this is done entirely at your own risk. Modifying the code has the potential to cause issues with updates to the webui. I only recommend doing this if you're already familiar with git (ie. You can confidently do a git rebase or merge) |
This is the key I was missing, but not for the reason you might imagine. So thanks again, I would not have discovered MY own error without your help. |
I managed to get it to work on windows and here are a few hints if others are tempted by the experience. First, if you are not familiar with diff files, just copy hack.diff in the root folder of your WebUI install, and use the following command: The second thing is that this was initially made for Linux so you have to use a windows-compatible address to refer to the Here is an example of this line as modified for windows: depth_img = Image.open(r"C:\Users\YOURUSERNAME\SDdepthsrc\depth.png") |
|
It depends on what browser you are using, but there's usually a "Save Page
As..." option hidden under the file menu somewhere.
…On Fri, Jan 6, 2023, 1:50 p.m. Birdie ***@***.***> wrote:
For very smooth ones, how do you guys download this yaml config file? It's
literally just a web page.
—
Reply to this email directly, view it on GitHub
<#5542 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAGPMNV4NLEW2S4ASPVZATWRBZP3ANCNFSM6AAAAAASYZQXZQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I figured it out, thanks. |
|
I'm seeing some weird issues with depth2img on a Mac, where the depths seems to be random rather than what the image should be: @JaySmithWpg any ideas?
|
|
You're not the first person I've heard about having trouble with the depth
model on a Mac. Unfortunately, without a Mac of my own to test things on, I
can't even guess what would cause it. I'm not sure if it would be a problem
with this code, with the depth model itself, or with something else
entirely.
I wish I could be of more help, there are few things I dislike more than
unreliable code. Unfortunately I can't recreate the issue with my own
hardware. I'll keep my ears open though, hopefully somebody knows what's
going on.
…On Tue, Jan 17, 2023, 5:06 p.m. Paul Hayes ***@***.***> wrote:
I'm seeing some weird issues with depth2img on a Mac, where the depths
seems to be random rather than what the image should be:
#6865
<#6865>
@JaySmithWpg <https://github.com/JaySmithWpg> any ideas?
[image: Screenshot 2023-01-17 at 22 03 43]
<https://user-images.githubusercontent.com/319055/213031749-378a27c8-125e-4a65-bb2f-6f5b69bc01f2.png>
—
Reply to this email directly, view it on GitHub
<#5542 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAGPMO3BMFVR5WIAYYTRELWS4QXRANCNFSM6AAAAAASYZQXZQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@JaySmithWpg Should the depth maps be being saved somewhere? I'm not seeing any show up in the default |
It's not being saved anywhere at the moment, my goal here was to support the depth model without any side effects. That said, it's a pretty easy change to save the depth output if somebody is interested in putting the change together. I have a rough version of that change here, but I haven't tested it against the latest version of the UI. Anybody is welcome to take this and make a PR out of it: (Disclaimer: I don't recommend or support applying that change manually unless you have a good working knowledge of git. If you don't know how to merge and rebase in git, you might leave your UI in a state where updates fail. My schedule will soon make it impossible for me to support people who make this change and regret it.) |
|
Have a look at this Depth Map IO script - it's the solution to not only save the depth map used secretly by the 2.0 depth model but also to plug in your own custom depth map as an input for it - in addition to the standard RGB image for IMG2IMG. It's very useful, and very simple to use. https://github.com/AnonymousCervine/depth-image-io-for-SDWebui
Big thanks to @AnonymousCervine for making this and sharing it with all of us ! |
Can you install this Depth-Image-IO script and check the depth map you get out of it ? https://github.com/AnonymousCervine/depth-image-io-for-SDWebui This might show us where the problem is coming from. For a similar problem related to 16 bit PNG check this Issue page over there. |
|
What's up guys, I'm from Brazil and I'm struggling to get help in Portuguese on the subject. I think this is the right place. I correctly installed the Depthmap2mask and I only have a problem using the dpt_large model. The other models it works. The following error appears on the CMD screen: File "C:\Users\bypau\Desktop\SD\stable-diffusion-webui\repositories\midas\midas\base_model.py", line 13, in load I tried to do the procedure that appears at the beginning of this page but it did not solve my problem. Thank you all for the attention. |
What?
Support for Stable Diffusion 2.0's Depth2Image LatentDepth2ImageDiffusion.
Why?
It's a great feature that preserves the overall form of an image. It's very useful for style transfers in particular.
Doesn't depthmap2mask already do this?
No. depthmap2mask is a great script for masking out the foreground or background of an image when running img2img, but it is no substitute for a depth-aware model that can repaint the entire scene.
How does it work?
Instructions:
Tips:
Since the structure of the image is preserved in depth2img, you can get great results with the denoising strength set very high.
Caution:
It probably goes without saying, but you will get errors if you try to use the Depth2Image model for txt2img. It will still work with inpainting.