Making LLMs lighter with AutoGPTQ and transformers
Large language models have demonstrated remarkable capabilities in understanding and generating human-like text, revolutionizing applications across various domains. However, the demands they place on consumer hardware for training and deployment have become increasingly challenging to meet.
🤗 Hugging Face’s core mission is to democratize good machine learning, and this includes making large models as accessible as possible for everyone. In the same spirit as our bitsandbytes collaboration, we have just integrated the AutoGPTQ library in Transformers, making it possible for users to quantize and run models in 8, 4, 3, or even 2-bit precision using the