No images were discovered by the bucket manager in the dataset: pseudo-camera-10k-sd3

[shu-boom]

Error:

2024-08-25 14:16:49,694 [ERROR] (JsonMetadataBackend) Error processing image: cannot identify image file <_io.BytesIO object at 0x12f6d1620>
2024-08-25 14:16:49,694 [ERROR] (JsonMetadataBackend) Error traceback: Traceback (most recent call last):
  File "/Users/boom/SimpleTuner/helpers/metadata/backends/json.py", line 215, in _process_for_bucket
    with load_image(BytesIO(image_data)) as image:
  File "/Users/boom/SimpleTuner/helpers/image_manipulation/load.py", line 104, in load_image
    img = decode_image_with_pil(img_data)
  File "/Users/boom/SimpleTuner/helpers/image_manipulation/load.py", line 52, in decode_image_with_pil
    img_pil = Image.open(BytesIO(img_data))
  File "/Users/boom/SimpleTuner/.venv/lib/python3.10/site-packages/PIL/Image.py", line 3498, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x12f6d1620>

2024-08-25 14:16:49,696 [INFO] (BaseMetadataBackend) Image processing statistics: {'total_processed': 0, 'skipped': {'already_exists': 0, 'metadata_missing': 0, 'not_found': 0, 'too_small': 0, 'other': 0}}
2024-08-25 14:16:49,697 [INFO] (BaseMetadataBackend) Enforcing minimum image size of 0. This could take a while for very-large datasets.
2024-08-25 14:16:49,697 [INFO] (BaseMetadataBackend) Completed aspect bucket update.
2024-08-25 14:16:49,698 [WARNING] (DataBackendFactory) Key crop_aspect_buckets not found in the current backend config, using the existing value 'None'.
2024-08-25 14:16:49,698 [WARNING] (DataBackendFactory) Key disable_validation not found in the current backend config, using the existing value 'False'.
2024-08-25 14:16:49,698 [INFO] (DataBackendFactory) current_config_version : 1
2024-08-25 14:16:49,698 [INFO] (DataBackendFactory) Configured backend: {'id': 'pseudo-camera-10k-sd3', 'config': {'crop': True, 'crop_aspect': 'square', 'crop_aspect_buckets': None, 'crop_style': 'center', 'disable_validation': False, 'resolution': 1024, 'resolution_type': 'pixel', 'caption_strategy': 'filename', 'instance_data_dir': 'datasets/pseudo-camera-10k', 'maximum_image_size': 1024, 'target_downsample_size': 1024, 'config_version': 1}, 'dataset_type': 'image', 'data_backend': <helpers.data_backend.local.LocalDataBackend object at 0x13a10c5e0>, 'instance_data_dir': 'datasets/pseudo-camera-10k', 'metadata_backend': <helpers.metadata.backends.json.JsonMetadataBackend object at 0x13a10c5b0>}
(Rank: 0)  | Bucket     | Image Count (per-GPU)
------------------------------
2024-08-25 14:16:49,699 [ERROR] (__main__) No images were discovered by the bucket manager in the dataset: pseudo-camera-10k-sd3., traceback: Traceback (most recent call last):
  File "/Users/boom/SimpleTuner/train.py", line 449, in main
    configure_multi_databackend(
  File "/Users/boom/SimpleTuner/helpers/data_backend/factory.py", line 813, in configure_multi_databackend
    raise Exception(
Exception: No images were discovered by the bucket manager in the dataset: pseudo-camera-10k-sd3.

System Info:
Mac M2 Pro 16 GB Python 3.10.2

Config File: Placed in the config folder.

export MODEL_TYPE='lora'
export MODEL_NAME="stabilityai/stable-diffusion-3-medium-diffusers"
export STABLE_DIFFUSION_3=true
export PIXART_SIGMA=false
export STABLE_DIFFUSION_LEGACY=false
export KOLORS=false
export FLUX=false
export FLUX_GUIDANCE_VALUE=1.0
export FLUX_LORA_TARGET=all
export CONTROLNET=false
export USE_DORA=false
export RESUME_CHECKPOINT="latest"
export CHECKPOINTING_STEPS=150
export CHECKPOINTING_LIMIT=2
export LEARNING_RATE=8e-7 #@param {type:"number"}
export DEBUG_EXTRA_ARGS=""
export TRACKER_PROJECT_NAME="${MODEL_TYPE}-training"
export TRACKER_RUN_NAME="simpletuner-sdxl"
export MAX_NUM_STEPS=30000
export DATALOADER_CONFIG="/Users/boom/SimpleTuner/multidatabackend.json"
export OUTPUT_DIR="/Users/boom/SimpleTuner"
export PUSH_TO_HUB="false"
export PUSH_CHECKPOINTS="true"
export HUB_MODEL_NAME=$TRACKER_PROJECT_NAME
export RESOLUTION=1024
export RESOLUTION_TYPE="pixel"
export MINIMUM_RESOLUTION=$RESOLUTION
export VALIDATION_PROMPT="ethnographic photography of teddy bear at a picnic"
export VALIDATION_GUIDANCE=3.0
export VALIDATION_GUIDANCE_RESCALE=0.0
export VALIDATION_GUIDANCE_REAL=1.0
export VALIDATION_NO_CFG_UNTIL_TIMESTEP=2
export VALIDATION_STEPS=100
export VALIDATION_NUM_INFERENCE_STEPS=30
export VALIDATION_NEGATIVE_PROMPT="blurry, cropped, ugly"
export VALIDATION_SEED=42
export VALIDATION_RESOLUTION='1024x1024'
export TRAIN_BATCH_SIZE=10
export VAE_BATCH_SIZE=4
export LR_SCHEDULE="polynomial"
export LR_WARMUP_STEPS=1000
export CAPTION_DROPOUT_PROBABILITY=0.1
export METADATA_UPDATE_INTERVAL=65
export MAX_WORKERS=32
export READ_BATCH_SIZE=25
export WRITE_BATCH_SIZE=64
export IMAGE_PROCESSING_BATCH_SIZE=32
export AWS_MAX_POOL_CONNECTIONS=128
export TORCH_NUM_THREADS=8
export DELETE_ERRORED_IMAGES=0
export DELETE_SMALL_IMAGES=0
export TRAINING_SCHEDULER_TIMESTEP_SPACING="trailing"
export INFERENCE_SCHEDULER_TIMESTEP_SPACING="trailing"
export MIN_SNR_GAMMA=5
export USE_XFORMERS=false
export USE_GRADIENT_CHECKPOINTING=true
export ALLOW_TF32=true
export OPTIMIZER="adamw_bf16"
export USE_EMA=false
export EMA_DECAY=0.999
export TRAINER_EXTRA_ARGS=""
export TRAINING_SEED=42
export MIXED_PRECISION=no                
export PURE_BF16=true
export TRAINING_NUM_PROCESSES=1
export TRAINING_NUM_MACHINES=1
export ACCELERATE_EXTRA_ARGS=""                 
export TRAINING_DYNAMO_BACKEND='no'              
export TOKENIZERS_PARALLELISM=false

Dataloader Config:
I have tried replacing the resolution type to area and I have tried changing its location currently it sits in my Root SimpleTuner folder. This folder also contains the dataset folder prepared by following instructions on the tutorial.

[
  {
    "id": "pseudo-camera-10k-sd3",
    "type": "local",
    "crop": true,
    "crop_aspect": "square",
    "crop_style": "center",
    "resolution": 1024,
    "minimum_image_size": 0,
    "maximum_image_size": 1024,
    "target_downsample_size": 1024,
    "resolution_type": "pixel",
    "cache_dir_vae": "cache/vae/sd3/pseudo-camera-10k",
    "instance_data_dir": "datasets/pseudo-camera-10k",
    "disabled": false,
    "skip_file_discovery": "",
    "caption_strategy": "filename",
    "metadata_backend": "json"
  },
  {
    "id": "text-embeds",
    "type": "local",
    "dataset_type": "text_embeds",
    "default": true,
    "cache_dir": "cache/text/sd3/pseudo-camera-10k",
    "disabled": false,
    "write_batch_size": 128
  }
]

[bghira]

try switching instance_data_dir to an absolute path

[shu-boom]

I tried it but doesn't work here is my updated data loader

Note: I tried replacing only the instance_data_dir path to a full path and then re ran after replacing all the paths to full paths but still it fails

[
  {
    "id": "pseudo-camera-10k-sd3",
    "type": "local",
    "crop": true,
    "crop_aspect": "square",
    "crop_style": "center",
    "resolution": 1024,
    "minimum_image_size": 0,
    "maximum_image_size": 1024,
    "target_downsample_size": 1024,
    "resolution_type": "pixel",
    "cache_dir_vae": "/Users/boom/SimpleTuner/cache/vae/sd3/pseudo-camera-10k",
    "instance_data_dir": "/Users/boom/SimpleTuner/datasets/pseudo-camera-10k",
    "disabled": false,
    "skip_file_discovery": "",
    "caption_strategy": "filename",
    "metadata_backend": "json"
  },
  {
    "id": "text-embeds",
    "type": "local",
    "dataset_type": "text_embeds",
    "default": true,
    "cache_dir": "/Users/boom/SimpleTuner/cache/text/sd3/pseudo-camera-10k",
    "disabled": false,
    "write_batch_size": 128
  }
]

I tried to debug and see seems like init_backend and backend has values defined. Pasted below

log of print(str(backend)) at line 576 in factory.py

{'id': 'pseudo-camera-10k-sd3', 'type': 'local', 'crop': True, 'crop_aspect': 'square', 'crop_style': 'center', 'resolution': 1024, 'minimum_image_size': 0, 'maximum_image_size': 1024, 'target_downsample_size': 1024, 'resolution_type': 'pixel', 'cache_dir_vae': '/Users/boom/SimpleTuner/cache/vae/sd3/pseudo-camera-10k', 'instance_data_dir': '/Users/boom/SimpleTuner/datasets/pseudo-camera-10k', 'disabled': False, 'skip_file_discovery': '', 'caption_strategy': 'filename', 'metadata_backend': 'json'}

After that init_backend also gets defined to the following object

{'id': 'pseudo-camera-10k-sd3', 'config': {'crop': True, 'crop_aspect': 'square', 'crop_aspect_buckets': None, 'crop_style': 'center', 'disable_validation': False, 'resolution': 1024, 'resolution_type': 'pixel', 'caption_strategy': 'filename', 'instance_data_dir': '/Users/boom/SimpleTuner/datasets/pseudo-camera-10k', 'maximum_image_size': 1024, 'target_downsample_size': 1024}, 'dataset_type': 'image'}

However, after doing this
init_backend["metadata_backend"] = BucketManager_cls(...)
The check for len(init_backend["metadata_backend"]) == 0 indeed returns 0
and raises the exception f"No images were discovered by the bucket manager in the dataset: {init_backend['id']}."

When I looked inside the len function which is inside json.py I noticed that self.aspect_ratio_bucket_indices.values() is an empty array. I am not sure if it is related but when I looked inside my dataset folder. I see that

aspect_ratio_bucket_metadata_pseudo-camera-10k-sd3.json is {} & aspect_ratio_bucket_indices_pseudo-camera-10k-sd3.json contains the following :

{"config": {"crop": true, "crop_aspect": "square", "crop_aspect_buckets": null, "crop_style": "center", "disable_validation": false, "resolution": 1024, "resolution_type": "pixel", "caption_strategy": "filename", "instance_data_dir": "/Users/boom/SimpleTuner/datasets/pseudo-camera-10k", "maximum_image_size": 1024, "target_downsample_size": 1024}, "aspect_ratio_bucket_indices": {}}

Where aspect_ratio_bucket_indices is set to {}

Do you think this could be related? Thanks for the help :)

[bghira]

you should do env SIMPLETUNER_LOG_LEVEL=DEBUG bash train.sh and see debug.log instead

[shu-boom]

This error happened because I ran git clone git clone https://huggingface.co/datasets/ptx0/pseudo-camera-10k
Which downloaded pointers to the images hence I was getting PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x12f6d1620>

I had to go to the pseudo-camera-10k-sd3 folder and do git lfs pull. If this doesn't work originally, then try running git lfs install first

I am on a mac so I installed it using brew install git-lfs Thanks for the support :) happy coding!