Not Reproducible Training Result

[WPCJATH]

We attempted to train the model by following the instructions provided in the Train Diffusion Texture Painting guide. However, the results we obtained were significantly below the quality of the pre-trained checkpoints provided in the repository.

Experiment Setup

Here is a detailed description of our training setup:

• 🤗Accelerate Configuration: Default settings with single GPU training
• Dataset: DTD Dataset
• Stable Diffusion Inpainting Model: benjamin-paine/stable-diffusion-v1-5-inpainting
• Hyperparameters: Same as outlined in the Train Diffusion Texture Painting documentation

Observations and Issues

Below are the key issues we encountered during training:

1. Validation Results: The model struggles to inpaint certain samples effectively. We suspect that the random mask generation algorithm might be a contributing factor, especially as it sometimes creates large/entire areas of black or white regions.
   

2. Training Loss: The training loss does not decrease as expected, which is abnormal and suggests potential issues with the training process or configuration.
   

3. Drawing Results: The generated stamps are not connected properly, and the previews fail to reproduce the intended brush strokes.
   

Request for Assistance

Given these issues, we kindly request the authors or maintainers to guide the following points:

• Are there specific considerations or adjustments to the random mask generation algorithm that could improve performance, particularly for large regions of missing pixels?
• Could the abnormal training loss behavior be related to the dataset, optimizer settings, or any overlooked configuration details?
• Are there any additional settings, debugging tips, or known limitations of the current implementation that might help us achieve better results?

We would greatly appreciate your insights or suggestions. Thank you for your work on this project, and we look forward to your response.