Mistral NeMo AI Model Released with State-of-the-Art Performance

Mistral NeMo, a state-of-the-art artificial intelligence model, has been released by the Mistral AI team in collaboration with NVIDIA. This 12B model boasts a large context window of up to 128k tokens and offers exceptional reasoning, world knowledge, and coding accuracy. Built on standard architecture, Mistral NeMo is easy to use and can seamlessly replace existing models in various systems. The model has been trained with quantisation awareness, enabling efficient FP8 inference without performance loss.

Mistral NeMo’s multilingual capabilities make it an ideal choice for global applications, with strong performance in languages such as English, French, German, Spanish, Italian, Portuguese, Chinese, Japanese, Korean, Arabic, and Hindi. The model utilizes a new tokenizer, Tekken, which compresses natural language text and source code more efficiently than previous models. Mistral NeMo has undergone advanced fine-tuning and alignment, resulting in improved instruction following, reasoning, and code generation capabilities.

Introducing Mistral NeMo: A State-of-the-Art 12B Model with Advanced Capabilities

Mistral NeMo, a collaborative effort between the Mistral AI team and NVIDIA, is a cutting-edge 12B model that boasts an impressive context window of up to 128k tokens. This model’s architecture is designed to be easily adaptable, making it a drop-in replacement in any system utilizing Mistral 7B. The release of Mistral NeMo under the Apache 2.0 license aims to promote widespread adoption among researchers and enterprises.

One of the key features of Mistral NeMo is its state-of-the-art performance in its size category, particularly in terms of reasoning, world knowledge, and coding accuracy. This model’s capabilities are further enhanced by its quantisation awareness, which enables FP8 inference without any performance loss. The pre-trained base and instruction-tuned checkpoints released alongside Mistral NeMo demonstrate its potential for real-world applications.

A comparison with other recent open-source pre-trained models, such as Gemma 2 9B and Llama 3 8B, highlights Mistral NeMo’s competitive accuracy. This model’s performance is particularly notable in multilingual settings, where it excels in languages including English, French, German, Spanish, Italian, Portuguese, Chinese, Japanese, Korean, Arabic, and Hindi.

Multilingual Capabilities: Breaking Down Language Barriers

Mistral NeMo is specifically designed to cater to global, multilingual applications. Its training on function calling and large context window make it an ideal choice for a wide range of languages. The model’s performance on multilingual benchmarks demonstrates its ability to bridge language gaps, bringing frontier AI models within reach of users worldwide.

The Mistral NeMo model is trained on a diverse set of languages, ensuring that it can effectively handle linguistic nuances and complexities. This multilingual capability is a significant step forward in making AI models more accessible and usable across different cultures and regions.

Tekken: A Novel Tokenizer for Efficient Text Compression

Mistral NeMo employs a new tokenizer, Tekken, which is based on Tiktoken and trained on over 100 languages. Tekken’s efficiency in compressing natural language text and source code is significantly higher than the SentencePiece tokenizer used in previous Mistral models. Specifically, it achieves a ~30% improvement in compressing source code, Chinese, Italian, French, German, Spanish, and Russian.

Furthermore, Tekken demonstrates exceptional proficiency in compressing text for approximately 85% of all languages, outperforming the Llama 3 tokenizer. This novel tokenizer’s capabilities are particularly notable in languages such as Korean and Arabic, where it achieves a 2x and 3x improvement in compression rates, respectively.

Instruction Fine-Tuning: Enhancing Model Performance

Mistral NeMo underwent an advanced fine-tuning and alignment phase, resulting in significant improvements over Mistral 7B. The instruction-tuned model demonstrates enhanced capabilities in following precise instructions, reasoning, handling multi-turn conversations, and generating code.

The accuracy of the instruction-tuned model is evident in its performance on various benchmarks, with evaluations conducted using GPT4o as a judge on official references. This fine-tuning process has resulted in a model that is better equipped to handle complex tasks and provide accurate responses.

Availability and Integration: Seamlessly Integrating Mistral NeMo into Existing Systems

The release of Mistral NeMo includes pre-trained base and instruction-tuned checkpoints, which are hosted on HuggingFace. This allows users to easily integrate the model into their existing systems using mistral-inference and adapt it with mistral-finetune.

Additionally, Mistral NeMo is exposed on la Plateforme under the name open-mistral-nemo-2407 and packaged in a container as NVIDIA NIM inference microservice, available from ai.nvidia.com. This widespread availability ensures that researchers and enterprises can seamlessly integrate Mistral NeMo into their workflows, leveraging its advanced capabilities to drive innovation and progress.