Some question about training lora for HiDream-I1

[yinguoweiOvO]

Describe the bug

Has anyone successfully trained HiDream-I1 using the diffusers repository? I trained the lora model of HiDream-I1 with the default learning rate. The output images were extremely prone to crash and even turned into noisy images. However, this problem did not occur when using the ai-toolkit repository with the same learning rate. Which Settings should I adjust?

Note:
I modified the data set loading method, but the same data set loading method does not cause this problem when used for the training of the Flux, so it should not be caused by this part of the modification.

Please help me, Thanks!

Reproduction

export MODEL_NAME="./HiDream/HiDream-E1-Full"
export INSTANCE_DIR="./train_tag.json"
export OUTPUT_DIR="./HiDream-I1/trained-hidream-lora-8"

CUDA_VISIBLE_DEVICES=3 python ./HiDream-I1/train_dreambooth_lora_hidream.py
--pretrained_model_name_or_path=$MODEL_NAME
--pretrained_tokenizer_4_name_or_path="./meta-llama/Llama-3.1-8B-Instruct"
--pretrained_text_encoder_4_name_or_path="./meta-llama/Llama-3.1-8B-Instruct"
--dataset_name=$INSTANCE_DIR
--output_dir=$OUTPUT_DIR
--image_column="image"
--caption_column="text"
--mixed_precision="bf16"
--resolution=1024
--train_batch_size=1
--gradient_accumulation_steps=4
--use_8bit_adam
--rank=8
--learning_rate=2e-4
--report_to="tensorboard"
--lr_scheduler="constant_with_warmup"
--lr_warmup_steps=100
--max_train_steps=50000
--checkpointing_steps=1000
--seed="0"
--offload \

Logs

System Info

Package Version Editable project location

---

accelerate 1.3.0
aiohappyeyeballs 2.4.6
aiohttp 3.11.12
aiosignal 1.3.2
annotated-types 0.7.0
async-timeout 5.0.1
attrs 25.1.0
bitsandbytes 0.45.4
blessed 1.20.0
certifi 2024.12.14
charset-normalizer 3.4.1
contourpy 1.3.2
cycler 0.12.1
datasets 3.2.0
deepspeed 0.16.3
diffusers 0.33.0.dev0 /code/diffusers
dill 0.3.8
einops 0.8.0
filelock 3.17.0
fonttools 4.58.4
frozenlist 1.5.0
fsspec 2024.9.0
gpustat 1.1.1
hjson 3.1.0
huggingface-hub 0.27.1
idna 3.10
importlib_metadata 8.6.1
Jinja2 3.1.5
kiwisolver 1.4.8
MarkupSafe 3.0.2
matplotlib 3.10.3
modelscope 1.22.3
mpmath 1.3.0
msgpack 1.1.0
multidict 6.1.0
multiprocess 0.70.16
networkx 3.4.2
ninja 1.11.1.3
numpy 2.2.2
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-ml-py 12.560.30
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
packaging 24.2
pandas 2.2.3
peft 0.15.0
pillow 11.1.0
pip 24.2
prodigyopt 1.1.2
propcache 0.2.1
protobuf 5.29.3
psutil 6.1.1
py-cpuinfo 9.0.0
pyarrow 19.0.0
pydantic 2.10.6
pydantic_core 2.27.2
pyparsing 3.2.3
python-dateutil 2.9.0.post0
pytz 2025.1
PyYAML 6.0.2
regex 2024.11.6
requests 2.32.3
safetensors 0.5.2
sentencepiece 0.2.0
setuptools 75.1.0
six 1.17.0
sympy 1.13.1
tokenizers 0.21.0
torch 2.5.1
torchvision 0.20.1
tqdm 4.67.1
transformers 4.48.1
triton 3.1.0
typing_extensions 4.12.2
tzdata 2025.1
urllib3 2.3.0
wcwidth 0.2.13
wheel 0.44.0
xxhash 3.5.0
yarl 1.18.3
zipp 3.21.0

Who can help?

No response

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.