Transformers

Single GPU Optimization Resource: Hugging Face Doc Method/tool Improves training speed Optimizes memory utilization Batch size choice Yes Yes Gradient accumulation No Yes Gradient checkpointing No Yes Mixed precision training Yes (No) Optimizer choice Yes Yes Data preloading Yes No DeepSpeed Zero No Yes torch.compile Yes No Parameter-Efficient Fine Tuning (PEFT) No Yes FP16 If your model doesn’t work well with mixed precision, for example if it wasn’t pretrained in mixed precision, you may encounter overflow or underflow issues which can cause NaN loss....

Non-ROOT users configure a remote CUDA server for a deep learning environment, below is an example of Pytorch. CUDA driver Check the version of CUDA already installed on the server: 1 nvcc -V Showing version 11.4 here, but it doesn’t matter. Next to check the driver version, this relates to the latest CUDA version that we can install in the conda virtual environment. 1 2 3 4 nvidia-smi ### Output ### Driver Version: 535....