Mistral Common Release Notes

Mistral-Common can now process streaming requests

import numpy as np
from mistral_common.audio import Audio
from mistral_common.protocol.instruct.chunk import RawAudio
from mistral_common.protocol.transcription.request import StreamingMode, TranscriptionRequest
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer

# 1. Load the tokenizer with audio support
tokenizer = MistralTokenizer.from_hf_hub("mistralai/Voxtral-Mini-4B-Realtime-2602")
# 2. Create sample audio data (or load from a file)
sampling_rate = 16_000
duration_s = 2.0
audio_array = np.random.uniform(-1, 1, size=int(duration_s * sampling_rate)).astype(np.float32)
audio = Audio(audio_array=audio_array, sampling_rate=sampling_rate, format="wav")
# 3. Create the streaming transcription request
request = TranscriptionRequest(
    audio=RawAudio(data=audio.to_base64("wav"), format="wav"),
    streaming=StreamingMode.ONLINE,  # or StreamingMode.OFFLINE
    language=None,
)
# 4. Encode the request
tokenized = tokenizer.encode_transcription(request)
# 5. Access the results
print(f"Tokens: {tokenized.tokens}")
print(f"Number of tokens: {len(tokenized.tokens)}")
print(f"Number of audio segments: {len(tokenized.audios)}")

See https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602 for more info.

What's Changed

• Add new token logic asrstr by @patrickvonplaten in #172
• [Backward comp] Still need the _control_tokens for vLLM by @patrickvonplaten in #173
• Release 1.8.8 by @juliendenize in #174
• [Audio] Update padding by @patrickvonplaten in #175
• [Audio] Improve padding for streaming by @patrickvonplaten in #177
• Add audio_encoder to Tokenizer V13 by @amosyou in #180
• Release v1.9.0 - Audio streaming by @patrickvonplaten in #179
• Fix image tests with downloads by @juliendenize in #181
• Enhance accessibility by @juliendenize in #176

New Contributors

• @amosyou made their first contribution in #180

Full Changelog: v1.8.7...v1.9.0

Release notes

• Remove deprecated imports in docs. by @juliendenize in #138
• Add normalizer and validator utils by @juliendenize in #140
• Refactor private aggregate messages for InstructRequestNormalizer by @juliendenize in #141
• test: improve unit test for is_opencv_installed by @PrasanaaV in #143
• Optimize spm decode function by @juliendenize in #144
• Add get_one_valid_tokenizer_file by @juliendenize in #142
• Remove Python 3.9 support by @juliendenize in #145
• Correctly pass revision and token to hf_api by @juliendenize in #149
• Fix assertion in test_convert_text_chunk and tool_call by @patrickvonplaten in #152
• Pins GH actions by @arcanis in #160
• Add usage restrictions regarding third-party rights. by @juliendenize in #161
• Improve tekken logging message for vocabulary by @juliendenize in #162
• Set version 1.8.6 by @juliendenize in #151

New Contributors

• @PrasanaaV made their first contribution in #143
• @arcanis made their first contribution in #160

Full Changelog

v1.8.5...v1.8.6

Raw content

Add AudioURLChunk by @juliendenize in #120

Now you can use http(s) URLs, file paths and base64 string (without specifying format) in your content chunks thanks to AudioURLChunk !

from mistral_common.protocol.instruct.messages import AudioURL, AudioURLChunk, TextChunk, UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer

repo_id = "mistralai/Voxtral-Mini-3B-2507"
tokenizer = MistralTokenizer.from_hf_hub(repo_id)

text_chunk = TextChunk(text="Wat do you think about this audio?")
user_msg = UserMessage(content=[AudioURLChunk(audio_url=AudioURL(url="https://freewavesamples.com/files/Ouch-6.wav")), text_chunk])
request = ChatCompletionRequest(messages=[user_msg])
tokenized = tokenizer.encode_chat_completion(request)
# pass tokenized.tokens to your favorite audio model
print(tokenized.tokens)
print(tokenized.audios)
# print text to visually see tokens
print(tokenized.text)

Full Changelog: v1.8.0...v1.8.1

[Audio] Add audio by @patrickvonplaten in #119

Full Changelog: v1.7.0...v1.8.0

Audio chat example

from mistral_common.protocol.instruct.messages import TextChunk, AudioChunk, UserMessage, AssistantMessage, RawAudio
from mistral_common.protocol.instruct.request import ChatCompletionRequest
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
import mistral_common.audio
from huggingface_hub import hf_hub_download

repo_id = "mistralai/voxtral-mini"
tokenizer = MistralTokenizer.from_hf_hub(repo_id)

obama_file = hf_hub_download("patrickvonplaten/audio_samples", "obama.mp3", repo_type = "dataset")
bcn_file = hf_hub_download("patrickvonplaten/audio_samples", "bcn_weather.mp3", repo_type = "dataset")

def file_to_chunk(file: str) -> AudioChunk:
    audio = Audio.from_file(file, strict = False)
    return AudioChunk.from_audio(audio)

text_chunk = TextChunk(text="Which speaker do you prefer between the two? Why? How are they different from each other?")
user_msg = UserMessage(content=[file_to_chunk(obama_file), file_to_chunk(bcn_file), text_chunk])
request = ChatCompletionRequest(messages=[user_msg])
tokenized = tokenizer.encode_chat_completion(request)
# pass tokenized.tokens to your favorite audio model
print(tokenized.tokens)
print(tokenized.audios)
# print text to visually see tokens
print(tokenized.text)

Full Changelog: v1.7.0...v1.8.0