AI open source projects

LiveTalking is an open source digital human production platform , real-time interactive streaming digital human , to achieve synchronized audio and video conversations . It is basically commercially available, and it makes the creation of digital human projects easy and efficient. LiveTalking Features Open source and free: all code of LiveTalking is public, and users can use it without paying extra. This provides a low-cost way for users who want to try out digital person technology to get started. Technology Leadership: LiveTalking supports a variety of advanced models such as Ernerf, Muse...

Sonic is an audio-driven portrait animation framework from Tencent and Zhejiang University that generates realistic facial expressions and movements based on global audio perception.Sonic is based on context-enhanced audio learning and motion decoupling controllers, which extract long-term temporal audio knowledge within the audio clip and independently control the head and expression movements, respectively, to enhance local audio perception.Sonic uses a temporal-aware position offset fusion mechanism to extend local audio perception to global and solve the jitter and mutation problems in long video generation.Sonic outperforms the existing ones in terms of video quality, lip-synchronization accuracy, motion diversity, and temporal coherence ...

Clone voice is open source voice cloning tool, based on deep learning technology to analyze and simulate the human voice, to achieve high-quality cloning of the voice. The tool supports 16 languages including Chinese, English, Japanese, Korean, etc. It can convert text to speech or one voice style to another. The interface is friendly and easy to operate, and does not require high-performance hardware support, making it suitable for both personal and professional use.Clone-voice has a wide range of application scenarios, including entertainment, education, media advertising, and voice interaction, providing new possibilities for digital content creation and personalized sound resources. Cl...

MAGI-1 is the world's first large autoregressive video generation model open-sourced by Sand AI. It adopts autoregressive architecture, generates smooth and natural videos by predicting video sequences block by block, and supports unlimited scaling and one-shot-to-the-end long video generation. The native resolution of the model can be up to 1440x2568, and the generated video has smooth movements and realistic details, and has the ability of controlled generation, which can realize smooth scene transitions and fine-grained control through chunking cues. MAGI-1 Features High-performance video generation: MAGI-1 can generate high-quality video clips in a short period of time, such as generating a 5-second video in just 3 seconds, generating...

FramePack is a revolutionary video diffusion technology that makes possible fast, high-quality video generation on consumer GPUs with low video memory requirements. It makes advanced video creation feasible on standard hardware through an innovative frame context packing method that allows users to predict the next frame. FramePack Features Low Video RAM Requirements: Requires only 6GB of video RAM to run and is suitable for use on laptop GPUs. Efficient Frame Generation Capability: Generate thousands of video frames based on the 13B model 30fps frame rate. Fast generation: Personal RTX 4090 graphics card ...

FastGPT is a knowledge base Q&A system based on the LLM large language model, providing out-of-the-box data processing, model invocation and other capabilities. At the same time, it can realize complex Q&A scenarios through Flow visualization for workflow scheduling! FastGPT Features Dedicated AI Customer Service: Train by importing documents or existing Q&A pairs, so that AI models can answer questions based on your documents in an interactive dialog. Easy-to-use visual interface: FastGPT adopts an intuitive visual interface design, providing rich and practical functions for various application scenarios...

Mochi 1 is an open source AI video generation model from Genmo that converts text prompts into high-quality video. It is released under the Apache 2.0 license and represents an important milestone in the democratization of AI video technology, supporting free use for personal and commercial purposes. The model is currently available in a base version at 480p, with plans to release a high-definition version, Mochi 1 HD, with 720p support by the end of the year, offering higher fidelity and smoother motion.The model weights and architecture for Mochi 1 are found on the Hugging Face platform, G...

MMAudio is an AI audio synthesis technology based on multimodal co-training, which allows models to be trained on a wide range of audiovisual and audio-text datasets. At the heart of the technology is a synchronization module that ensures that the generated audio precisely matches the video frames to achieve a high degree of synchronization.MMAudio is suitable for a wide range of application scenarios including film and TV production and game development, generating audio based on video content or text descriptions to enhance the user experience. MMAudio Features Video to Audio Synthesis: Automatically generates highly synchronized audio that matches the video content. Text to Audio Synthesis: Generate audio based on...

Diffutoon is an AI framework for converting videos into cartoon-style animations, launched by researchers from Alibaba and East China Normal University. The editable cartoon shading technology based on the diffusion model can convert realistic videos into cartoon-style animations. The technology achieves high resolution and long-term rendering of videos by decomposing it into subtasks such as stylization, consistency enhancement, structure guidance, and coloring. Diffutoon also has a content editing function that can adjust video details based on text prompts, maintaining a high degree of visual effect and consistency when processing videos, and achieving efficient and high-quality processing of video animations…

ProPainter is an advanced video restoration tool that uses AI technology to remove specific objects and watermarks from videos. Through the loop flow completion network and Transformer technology, ProPainter can intelligently detect and remove moving objects in videos, repair damaged areas, and restore the integrity of videos. Whether it is removing watermarks or restoring videos, ProPainter can provide high-quality solutions. ProPainter features Remove moving objects/people: Using advanced E2FGV1 technology, ProPaint…

ChatTTS is a speech generation model designed for conversational scenarios. It supports Chinese and English. After large-scale data training, it can generate high-quality and natural speech synthesis. The product is designed to support applications such as conversational tasks of large language model assistants, generating conversational speech, video introductions, and speech synthesis for education and training content. ChatTTS features multi-language support: supports Chinese and English, suitable for multi-language environments. Large-scale data training: trained with about 100,000 hours of Chinese and English data to ensure high-quality and natural speech synthesis. Conversational task compatibility…

StoryDiffusion is an innovative AI tool developed by the HVision team at Nankai University. Its core function is to generate coherent image and video stories, especially good at comics. The tool uses advanced consistent self-attention technology to generate thematically consistent image sequences without additional training. These images are very suitable for storytelling or as a basis for further content creation. StoryDiffusion is a joint venture between ByteDance and Nankai University…

IDM-VTON is a novel diffusion model for image-based virtual try-on tasks, which generates virtual try-on images with a high degree of realism and detail by combining high-level semantics of visual coders and UNet networks as well as low-level features. The technique enhances the realism of the generated images by providing detailed textual cues and further improves the fidelity and realism in real-world scenarios through customization methods. IDM-VTON is an advanced virtual try-on technique that generates high-quality virtual try-on images by combining a visual coder and a UNet model, and can be customized to...

Rope is a GUI-focused AI face swapping tool that combines insightface's inswapper_128 model to provide a feature-rich GUI. The highlight of this tool is its fast face swapping speed, image upscaling, similarity adjuster, and orientation management. In addition, Rope supports face swapping for images and videos, and has advanced features such as automatic save file name generation, docking/undocking of video players, real-time playback, image setting markers for specific frames, etc. Rope Features AI Face Swapping: Leveraging the most advanced…