{"id":14058,"date":"2024-06-26T08:54:58","date_gmt":"2024-06-26T00:54:58","guid":{"rendered":"https:\/\/www.1ai.net\/?p=14058"},"modified":"2024-06-26T08:54:58","modified_gmt":"2024-06-26T00:54:58","slug":"%e8%8b%b9%e6%9e%9c%e6%8e%a8%e5%87%ba%e5%85%a8%e8%83%bd%e8%a7%86%e8%a7%89%e6%a8%a1%e5%9e%8b4m-21-%e5%8f%af%e5%a4%84%e7%90%8621%e7%a7%8d%e4%b8%8d%e5%90%8c%e6%a8%a1%e6%80%81","status":"publish","type":"post","link":"https:\/\/www.1ai.net\/en\/14058.html","title":{"rendered":"Apple launches all-around visual model 4M-21 that can handle 21 different modalities"},"content":{"rendered":"

apple<\/a>and researchers at the \u00c9cole Polytechnique F\u00e9d\u00e9rale de Lausanne (EPFL) in Switzerland have jointly developed a single model for any-to-any modality that can be trained on dozens of highly diverse modalities and co-trained on large-scale multimodal datasets and text corpora. The model, named 4M-21, is trained on 21 different modalities and accomplishes at least three times more than existing models without loss of performance.<\/p>\n

\"\"<\/p>\n

The study uses a 4M pre-training programme that enhances the performance and adaptability of models by scaling up models and data sets, increasing the types and quantities of models involved in training models and conducting joint training on multiple data sets. Researchers use different tokenization methods to disperse modulates with different characteristics, such as global image embedding, human gestures and semantic examples. In terms of structure selection, the study uses a 4M encoder-decoder structure based on Transformer and adds an additional modulation embedded to adapt to the new modulation\u3002<\/p>\n

\"\"<\/p>\n

The model not only performs a range of common vision tasks out-of-the-box, such as DIODE surface normal and depth estimation, COCO semantic and instance segmentation, and 3DPW3D human pose estimation, but is also capable of generating arbitrary training modalities, supports several methods to perform fine-grained and multimodal generation, and can retrieve RGB images or other modalities by using other modalities as queries. In addition, the researchers have conducted multimodal transfer experiments on NYUv2, Hypersim semantic segmentation, and ARKitScenes.<\/p>\n

Important functional features include.<\/p>\n

Arbitrary to Arbitrary Modality: Increase the number of modalities from the existing best 7 modalities for arbitrary to arbitrary models to 21 different modalities for cross-modality retrieval, controlled generation and powerful out-of-the-box performance.<\/p>\n

Versatility support: Add support for more structured data such as human posture, SAM instances, metadata, etc.<\/p>\n

Tokenization:Investigates discrete tokenization for different modalities, such as global image embedding, human gestures, and semantic instances, using modality-specific approaches.<\/p>\n

Extension:Extend the model size to 3B parameters and the dataset to 0.5B samples.<\/p>\n

Co-training: Simultaneous visual and verbal co-training.<\/p>\n