Harbin Institute of Technology

Recently, Tsinghua University and Harbin Institute of Technology jointly published a paper that successfully compressed a large model to 1 bit while maintaining the performance of 83%. This achievement marks a major breakthrough in the field of quantization models. In the past, quantization below 2 bits has always been an insurmountable obstacle for researchers, and this attempt at 1-bit quantization has attracted widespread attention from the academic community at home and abroad. The OneBit method proposed in this study is the first attempt to compress a pre-trained large model to a true 1 bit. Through a new 1-bit layer structure, SVID-based parameter initialization and quantization…