{"id":4182,"date":"2024-02-22T09:21:58","date_gmt":"2024-02-22T01:21:58","guid":{"rendered":"https:\/\/www.1ai.net\/?p=4182"},"modified":"2024-02-22T09:21:58","modified_gmt":"2024-02-22T01:21:58","slug":"%e8%b0%b7%e6%ad%8cvideopoet%e8%b4%9f%e8%b4%a3%e4%ba%ba%e8%92%8b%e8%b7%af%e8%b7%b3%e6%a7%bdtiktok%ef%bc%81%e5%af%b9%e6%a0%87sora%ef%bc%8cai%e8%a7%86%e9%a2%91%e6%a8%a1%e5%9e%8b%e5%a4%a7%e6%88%98","status":"publish","type":"post","link":"https:\/\/www.1ai.net\/en\/4182.html","title":{"rendered":"Google VideoPoet head Jiang Lu jumped to TikTok! The AI video model war is imminent, competing with Sora."},"content":{"rendered":"

Google<\/a>VideoPoet Project Research Lead, CMU Adjunct Professor Jiang LuJiang Lu<\/a>Google Scholar data shows that he has joinedTikTok<\/a>.<\/p>\n

Recently, there have been rumors that TikTok has recruited the author of a certain paper as the head of its North American technology department to develop video-generating AI that can compete with Sora.<\/p>\n

Jiang Lu announced his resignation from Google on LinkedIn three weeks ago, which revealed the mystery to the outside world.<\/p>\n

As the head of Google's VideoPoet project, he will leave Google Research, but will continue his work in the field of video generation in the Bay Area.<\/p>\n

TalentFirst, data second, computing power thirdFirstly<\/span>, data second, computing power third", Xie Saining's AI breakthrough "3 elements" has pointed out the direction for large companies to build their own AI moats in the future.<\/p>\n

Jiang Lu led the Google team to launch VideoPoet, a video generation technology similar to Sora in technical route at the end of last year, making him one of the few scientists in the world capable of building the most cutting-edge AI video generation technology.<\/p>\n

VideoPoet has already advanced the frontier of AI video generation to produce 10-second long, highly consistent videos with large and coherent movements before Sora was released.<\/p>\n

While simultaneously, he is also an adjunct professor at CMU with extensive scientific research experience and achievements.<\/p>\n

Jiang Lu is a compound expert who has both a solid theoretical foundation and extensive engineering and management experience in cutting-edge large-scale projects. Naturally, he has become a cornerstone AI talent that large companies are eager to compete for.<\/p>\n

Personal Introduction<\/strong><\/p>\n

Jiang Lu works as a research scientist and manager at Google, and is also an adjunct professor at the Language Technologies Institute in the School of Computer Science at Carnegie Mellon University.<\/p>\n

At CMU, he not only guides graduate students' research projects, but also teaches courses himself.<\/p>\n

His research results are in the fields of natural language processing (ACL) and computer vision (CVPR).top<\/span>It has won numerous awards at conferences, including ACM ICMR, IEEE SLT and NIST TRECVID.<\/p>\n

His research has played a critical role in the development and improvement of many Google products: including YouTube, Cloud, Cloud AutoML, Ads, Waymo, and Translate.<\/p>\n

These products serve billions of users around the world daily.<\/p>\n

In addition to the above, there is another aspect that can well illustrate the high academic level of Jiang Lu: he has collaborated with many top-notch researchers in the fields of computer vision and natural language processing.top<\/span>Researchers have collaborated.<\/p>\n

From 2017 to 2018, he was a founding member of the first research team at Google Cloud AI, hand-picked by Dr. Jia Li and Dr. Fei-Fei Li.<\/p>\n

He then joined Google Research, where he collaborated with Dr. Weilong Yang (2019-2020), Dr. Ce Liu (2020-2021), Madison Le (2021-2022), and Dr. Irfan Essa (2023).<\/p>\n

In addition, during his doctoral studies at Carnegie Mellon University, his thesis was co-supervised by Dr. Tat-Seng Chua and Dr. Louis-Philippe Morency. He successfully graduated in 2017 with the help of Dr. Alexander Hauptmann and Dr. Teruko Mitamura.<\/p>\n

During his internships at Yahoo, Google, and Microsoft Research, he was mentored by Dr. Liangliang Cao, Dr. Yannis Kalantidis, Sachin Farfade, Paul Natsev, Balakrishnan Varadarajan, Qiang Wang, and Dr. Dongmei Zhang.<\/p>\n

From his LinkedIn profile, we can see that he has had internships at many major technology companies.<\/p>\n

\"2024022208590123402\"<\/p>\n

I have had internship experiences at CMU and NSF.<\/p>\n

Before graduation, he interned at Yahoo, Google, and Microsoft.<\/p>\n

He received his undergraduate degree from Xi'an Jiaotong University, his graduate degree from the Free University of Brussels, and his doctorate from CMU.<\/p>\n

VideoPoet<\/p>\n

The team he led at Google launched VideoPoet at the end of last year, which replaced the traditional UNet with Transformer and became the SOTA of AI video generation at that time.<\/p>\n

This achievement also became the main reason why TikTok recruited him.<\/p>\n

Compared to Gen-2, which can only generate small movements, VideoPoet can generate 10 seconds of ultra-long, continuous large-movement videos at a time, which can be said to have achieved a complete victory!<\/p>\n

In addition, VideoPoet is not based on a diffusion model, but a large multimodal model, and it can have capabilities such as T2V and V2A, and may become the mainstream of video generation in the future.<\/p>\n

Compared to other models, Google's approach is to seamlessly integrate multiple video generation functions into a single large language model without relying on dedicated components trained separately for each task.<\/p>\n

\"2024022208590123407\"<\/p>\n

Specifically, VideoPoet mainly consists of the following components:<\/p>\n

\u2013 Pre-trained MAGVIT V2 video tokenizer and SoundStream audio tokenizer, which translates images, videos and audio clips of different lengths into discrete code sequences in the unified vocabulary. These codes are compatible with text-type language models and facilitate integration with other modes of text\u3002<\/p>\n

\u2013 self-regression language models allow cross-modular learning between videos, images, audio and text, and predict the next video or audio token in the self-regression sequence\u3002<\/p>\n

\u2013 Various multi-modular production learning goals have been introduced in the framework of large-language model training, including text-to-video, text-to-image, image-to-video, video frame continuation, video repair\/extension, video style and video-to-video. In addition, these tasks can be combined to achieve additional zero sample functions (e.g. text to audio)\u3002<\/p>\n

VideoPoet is able to multitask on a variety of video-centric inputs and outputs. Among them, LLM can choose to take text as input to guide the generation of text-to-video, image-to-video, video-to-audio, stylization, and image augmentation tasks.<\/p>\n

A key advantage of using LLM for training is that many of the scalable efficiency improvements introduced in the existing LLM training infrastructure can be reused in VideoPoet.<\/p>\n

However, LLM operates on discrete tokens, which may pose challenges for video generation.<\/p>\n

Fortunately, video and audio tokenizers can encode video and audio clips into sequences of discrete tokens (i.e., integer indices) and can convert them back to their original representation.<\/p>\n

VideoPoet trains an autoregressive language model that learns across video, image, audio, and text modalities by using multiple tokenizers (MAGVIT V2 for video and image, SoundStream for audio).<\/p>\n

Once the model has generated tokens based on the context, a tokenizer decoder can be used to convert these tokens back into a viewable representation.<\/p>\n

VideoPoet task design: Different modalities are converted to and from tokens through tokenizer encoders and decoders. Boundary tokens are around each modality, and task tokens indicate the type of task to be performed.<\/p>\n

Compared with previous video generation models, VideoPoet has three major advantages.<\/p>\n

One is that it can generate longer videos, another is that users can have better control over the generated videos, and the last one is that VideoPoet can also generate different camera techniques based on text prompts.<\/p>\n

In the test, VideoPoet also took the lead, crushing many other video generation models.<\/p>\n

Text Fidelity: User preference ratings of text fidelity, i.e., accuracy in following prompts, First choice Percentage of Video<\/strong>:<\/p>\n

User preference ratings of text fidelity, i.e., accuracy in following instructionsFirst<\/span>Percentage of Video Fidelity<\/p>\n

Action fun:<\/strong><\/p>\n

Users\u2019 preference ratings on the fun of actions, i.e., in terms of finding actions interesting,First<\/span>Percentage of Video Fidelity<\/p>\n

In summary, we can see that on average 24-35% of people think that the examples generated by VideoPoet follow the prompts better than other models, while this proportion is only 8-11% for other models.<\/p>\n

Furthermore, 41%-54% of the evaluators rated the example actions in VideoPoet as more interesting, while other models only received 11%-21%.<\/p>\n

As for future research directions, Google researchers said that the VideoPoet framework will achieve 'any-to-any' generation, such as extending text to audio, audio to video, and video subtitles, etc.<\/p>","protected":false},"excerpt":{"rendered":"

Google Scholar profile of Jiang Lu, Research Lead of Google's VideoPoet project and adjunct professor at CMU, shows that he has joined TikTok. Recently, it was rumored that TikTok recruited the author of a certain paper as the head of the North American tech department, to develop a video-generating AI that can fight against Sora. And the news of Jiang Lu's departure from Google, which he posted on Linkedin 3 weeks ago, also revealed the mystery to the outside world. Posted on the news of leaving Google, also revealed to the outside world. He is leaving Google Research as the head of Google's VideoPoet program, but will stay in the Bay Area to continue work in the video generation space. \"Talent first, data second, arithmetic third\", Xie Saining's AI breakthrough \"3 elements\", has set the stage for the big players in the future to build their own<\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[146],"tags":[967,1292,281],"collection":[],"class_list":["post-4182","post","type-post","status-publish","format-standard","hentry","category-news","tag-tiktok","tag-1292","tag-281"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts\/4182","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/comments?post=4182"}],"version-history":[{"count":0,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/posts\/4182\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/media?parent=4182"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/categories?post=4182"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/tags?post=4182"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/www.1ai.net\/en\/wp-json\/wp\/v2\/collection?post=4182"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}