Thinkings about the hugging face document
Single GPU Optimization Resource: Hugging Face Doc Method/tool Improves training speed Optimizes memory utilization Batch size choice Yes Yes Gradient accumulation No Yes Gradient checkpointing No Yes Mixed precision training Yes (No) Optimizer choice Yes Yes Data preloading Yes No DeepSpeed Zero No Yes torch.compile Yes No Parameter-Efficient Fine Tuning (PEFT) No Yes FP16 If your model doesn’t work well with mixed precision, for example if it wasn’t pretrained in mixed precision, you may encounter overflow or underflow issues which can cause NaN loss....