Jan. 17 (Bloomberg) -- Research firm Snowflake Announced a new product called "SwiftKV"of AI model tuningThe company has also opened three Llama 3.1 AI models tuned with "SwiftKV" technology on Hugging Face (Click here to visit).
1AI has learned that the core of SwiftKV's technology is to optimize the model's cue word processing. The researchers note that typically the most computationally intensive part of a large model is processing the prompts that users enter for the model, and that many organizations customize extremely long prompts for their models, which on average are said to be "about 10 times as long as the output of the generated content".
According to Snowflake, this "SwiftKV" model tuning technology is specially optimized for the corresponding prefabricated cue word processing, which is said to break through the traditional Key-Value (KV) cache compression technology, and also introduces model restructuring and knowledge preservation self-distillation methods in the model inference process, thus effectively improving model throughput and reducing latency and computation costs. This effectively improves model throughput, reduces latency and computing costs, and is claimed to help AI models significantly reduce inference time.Can reduce model 50% inference time.
Experimental results show that after optimizing Llama 3.1's 8 and 70 billion parameter models with SwiftKV, the overall throughput of the corresponding models can be tripled, and the corresponding models also perform well in terms of code auto-completion and text summarization.