Produce an image that represents a "digital twin" of the input image

[steverhoades]

Background:
I am fairly new to Stable Diffusion and have been working to create a face Lora of my father for some time - fairly unsuccessfully.

First Approach: PhotoMaker Example
For the initial test of this model, I used the script provided for PhotoMaker. During this first attempt, I incorporated a single photo of my father, keeping the example prompt as it was. Once the inference process was complete, the resulting images did not closely resemble the subject—my father—but they did share some similar features, such as a balding head and a goatee.

Second Approach: PhotoMaker Style Example
In the second round of testing, I used the script provided for the PhotoMaker style. This time, I loaded the Lora (a custom creation) of my father. For the prompt, I eliminated the class word and utilized the trigger word (james cameron) for my Lora creation. The prompt was: prompt = "sci-fi, closeup portrait photo of james Cameron img wearing the sunglasses in Iron man suit, face, slim body, high quality, film grain" . Surprisingly, the photos generated all bore a striking resemblance to my father, which was an absolutely fantastic result.

Reference Image:

First Test Output Images (No Lora):

Second Test Output Images (With Lora):

Conclusion:
If you are looking for someone who has characteristics similar to those in your input image, you will generally get an output that matches your expectations. However, if you wish to obtain a "digital twin" of the individual in the input image, combining it with a Lora trained on that specific subject seems to produce remarkable results.

[bmaltais]

Indeed... this is a bit like pivotal training... where you combine two or three different method to produce better results... I was looking at testing this hypothesis once a proper ComfyUI node is created for PhotoMaker...

Can you share the updated version of the app that can use both LoRA and PhotoMaker embedding at the same time?

[ssuperGeorge]

how does it looks like when you only apply the lora model without photo maker? does the photo maker improve the id fidelity a lot?