Muse Glimmer

An open model built for always-on local agents. 30B parameters, licensed under Apache 2.0 and runs on a single GPU — tuned for tool use, long tasks, and failure recovery.

Meet Muse Glimmer

Always-on agents

Built for agents that don't stop with reliable tool-calling, persistent state across restarts, and self-managed memory across hours-long sessions.

Optimized for local deployments

It's small enough to run on a single consumer GPU or Mac, enabling use cases that range from local agents to local coding.

Benchmarks

Muse Glimmer is trained to deliver competitive agentic and coding performance, with multimodal perception built in.

Benchmark

Muse Glimmer-30B
High reasoning
Gemma4-31B
Thinking mode
Qwen3.6-27B
Thinking mode

General agentic

MCP Atlas
75.5
54.2
62.5

DeepSearch QA
74.6
61.7
71.1

τ³-Banking
23.5
15.1
16.7

WildClawBench
47.6
37.6
43.2

GDPval-AA
953
811
1141

GAIA2
43.3
36.4
40.0

SkillsBench
With skills
44.3
32.4
46.6

OSWorld-Verified
65.9
58.5
75.6

Agentic coding

SWE-Bench Pro
51.2
36.9
50.2

SWE-Bench Verified
76.0
66.6
77.2

TerminalBench 2.1
51.7
43.4
60.7

SciCode
43.6
43.4
39.8

Multimodal

Charxiv Reasoning
78.8
77.7
78.4

ScreenSpot Pro
75.4
75.9
76.1

OmniDocBench v1.5
75.8
72.5
77.8

MMMU Pro
74
73
75

Safety

CI Memories
Violation (↓): 26.4
Coverage: 64.8
Violation (↓): 12.1
Coverage: 53.0
Violation (↓): 53.4
Coverage: 66.9

Siren AgentDojo
Attack Success Rate (↓): 28.4
Utility: 94.2
Attack Success Rate (↓): 25.6
Utility: 90.8
Attack Success Rate (↓): 40.3
Utility: 92.7

General capabilities and reasoning

IFBench
77.0
76.0
70.8

AIME 2026
94.7
89.2
94.1

GPQA Diamond
83.5
85.7
84.2

Humanity's Last Exam
Text · No tools
22.0
23.6
23.1

AA-LCR
80.0
68.3
73.3

Beam 128K
65.1
58.2
63.0

For the full model card, please see the Muse Glimmer page on Hugging Face.

For more details about Muse Glimmer evaluations, read the methodology report.

Meta is committed to promoting safe and fair use of its tools and features, including Muse Glimmer. This Usage Policy (“Policy”) applies to the access or use of Muse Glimmer.

Where to run Muse Glimmer

Run Muse Glimmer with transformers, llama.cpp, vLLM, Colab, and other popular platforms.

Run Muse Glimmer on your own hardware in one simple, seamless CLI command.

Run, fine-tune, and deploy Muse Glimmer from one unified local UI interface with the open-source Unsloth Desktop app.

LM Studio Bionic is the easiest way to run Muse Glimmer for agentic tasks, locally on your computer.

Fireworks offers production-grade Muse Glimmer deployments with precise reasoning controls and effortless scaling for unpredictable agent traffic.

Run Muse Glimmer on Together Serverless Inference for long-running agentic workflows.

Run Muse Glimmer via a single unified API that routes across multiple inference providers for cost, speed, and availability.

Build with Muse Glimmer

Cookbooks and quickstarts

Get the most out of Muse Glimmer by using the correct chat template and following these prompting best practices.

Run on smaller hardware by reducing the precision of model weights.

Speed up inference by using a smaller, faster draft model to propose candidate tokens that the full model verifies.

Deploy with vLLM for high-throughput, low-latency inference with an OpenAI-compatible API endpoint.

Run on your machine with llama.cpp, a C/C++ inference engine that supports CPU, mixed CPU/GPU and full GPU execution.

Run on mobile phones, tablets, and edge devices with ExecuTorch, Meta's on-device inference framework.

STAY-UP-TO-DATE

Our latest updates delivered to your inbox

Subscribe to our newsletter to keep up with the latest AI updates, releases and more.

Original source

Agentic Workflows

We’re excited to release Muse Spark 1.3, which delivers improved performance across agentic and coding tasks. Drawing on what we learned from months of broad adoption of Muse Code and Meta Model API, we’ve also made this model easier to use in real-world settings. Smarter and more practically useful, Muse Spark 1.3 advances our work toward personal superintelligence.

Muse Spark 1.3 is rolling out today in Muse Code and Meta Model API. Previously available reasoning modes are available today with max reasoning coming shortly after we finish additional safety testing.

For more details about our evaluations, see our report.

Muse Spark 1.3 is designed to better sustain longer-horizon work by collaborating with users and juggling multiple workflows in a single, long thread. When given an open-ended objective, it uses tools to generate its own context across messy and conflicting sources, proactively corrects gaps in its plan, and keeps track of what it has learned to produce a final deliverable. We trained the model across a diverse set of harnesses to generalize to various agentic environments.

Trained to more actively collaborate with the user, Muse Spark 1.3 asks clarifying questions when prompts are ambiguous, invokes help from the user when stuck, and confirms before taking consequential actions. When working on long tasks, it adapts to user preferences, either providing frequent updates or working silently in the background.

Muse Spark 1.3 follows complex, long-form instructions more reliably than earlier Muse Spark models. Across multi-step tasks, it’s better at preserving detailed requirements without dropping constraints or drifting from the requested workflow.

We’ve also improved the multitasking capabilities of Muse Spark 1.3. For example, it now more accurately maps incoming prompts to the correct task within messy, single-threaded contexts, regardless of whether the user is steering past requests or interrupting them.

The model has better awareness of its own capabilities and limitations. We trained Muse Spark 1.3 to have a better sense of what it can and can’t do, what it knows and doesn’t know, and when it hits hurdles instead of hallucinating outcomes.

Coding

Muse Spark 1.3 was trained on more long-horizon coding tasks and shows improved usability in common engineering workflows. Relative to Muse Spark 1.2, it takes fewer turns where not needed and is less verbose, while having a cleaner overall coding style. In comparisons by Meta engineers, it proved to be significantly faster and more efficient, using ~20% fewer tool calls and ~25% fewer tokens.

Availability

Muse Spark 1.3 is available today in Muse Code and in Meta Model API.

Safety

We’ve improved safety along several axes most relevant to agentic and coding capabilities. Muse Spark 1.3 shows stronger adversarial robustness, with improved resistance to adversarial inputs and prompt injections. On complex agentic tasks, the model has better calibration on what constitutes irreversible actions and proceeds accordingly. Together, these changes reflect better discretion and judgment in long-horizon agentic tasks.

Looking Forward

We have an exciting roadmap lined up, including bigger models, the Muse Spark open weights release, and more. Stay tuned.

Original source

Today, we're introducing Muse Glimmer, the next model from Meta Superintelligence Labs, and open sourcing the model weights under a permissive Apache 2.0 license.

Muse Glimmer is a 30-billion-parameter model optimized for always-on local agent workflows. It’s small enough to run on a Mac or PC with a single consumer GPU, enabling use cases that range from local agents and function calling, to local coding, and LLM-as-a-judge evaluation. Muse Glimmer delivers strong performance on key agentic use cases and benchmarks compared with leading models in its size category.

Foundation models have achieved remarkable capabilities across reasoning, code generation, and tool use — yet most deployments still depend on cloud infrastructure and network access. Running models locally enables you to use AI anywhere, anytime, with or without an internet connection. This is increasingly viable: the open source community has shown that smaller models, when trained effectively, can approach frontier-level performance on targeted tasks. Muse Glimmer is optimized for these local use cases.

Keeping with our long tradition of sharing fundamental AI research, we're releasing Muse Glimmer open weights today on Hugging Face, along with developer documentation to help you start building and running your own agents. Muse Glimmer is built to work with the tools developers already use. Optimized integrations on llama.cpp, MLX, and ExecuTorch will land in the coming days, so you can go from download to working agent in minutes.

How We Trained Muse Glimmer

An agent that manages your schedule, drafts your messages, organizes your files, and learns how you work needs deep access to personal context. It also needs several capabilities working in concert: long-horizon execution, precise tool calling, multimodal understanding, long-context memory, and instruction following.

We designed Muse Glimmer to balance capability against the memory and compute constraints of local hardware. This required a compact architecture, a novel distillation recipe that transfers agentic reasoning from a much larger teacher model, and inference optimizations — including quantization — to meet latency expectations. We achieved this in the following phases:

• Pre-Training. We trained Muse Glimmer on Muse Spark's outputs using logit distillation, leveraging a similar data mix as the teacher.
• Mid-Training. We trained the model on longer-context, more agent-heavy data with richer reasoning traces, alongside organic data.
• Post-Training. We combined supervised fine-tuning with a mix of on-policy distillation and reinforcement learning across general, reasoning, coding, and agentic domains.

Muse Glimmer was evaluated under the standards set out in Meta's Advanced AI Scaling Framework and assessed for open-weight release across all relevant categories.

Built for Agents: What Muse Glimmer Can Do

Building effective agents requires key capabilities working together to achieve the user’s goals. Muse Glimmer is trained and evaluated across each of the following:

• End-to-end Agentic Task Completion. Muse Glimmer achieves strong success rates on full-task benchmarks including DeepSearch QA, MCP-Atlas, 𝛕-Bench and SWE-Bench, which measure its ability to work within scaffolds, write and debug code, and resolve multi-turn requests from start to finish.
• Reliable Tool Use. The model handles a wide range of function calls, invoking tools with precise schemas throughout extended workflows.
• Multi-Step Reasoning. Muse Glimmer chains reasoning over long horizons, sustaining coherent plans across complex, extended workflows.
• Failure Recovery. When a tool call fails or returns an unexpected result, the model is trained to diagnose the error and retry rather than halt.
• Multimodal Input and Reasoning. Through a dedicated perception encoder, the model accepts interleaved text and images. This enables agents to interpret screenshots, charts, and documents alongside conversation.
• Scaffold Compatibility. Muse Glimmer works across OpenClaw and other agentic orchestration patterns.
• Controllable Effort. Muse Glimmer supports different reasoning strengths to select the right balance between quality and speed.
• Multilingual. Muse Glimmer is trained on data from more than 100 languages.

Performance

We evaluated Muse Glimmer across a broad range of benchmarks to assess the diverse capabilities required for effective autonomous agent behavior. Compared with Gemma4-31B and Qwen3.6-27B, Muse Glimmer performs strongly for its size class on several widely used LLM benchmarks.

For more detail about our evaluations, see our report.

Optimized for Local Deployments

A local agent is truly useful if it's fast enough to feel responsive. An agent that takes minutes to reply or plan its next step breaks the flow of real work. We applied two optimizations to make Muse Glimmer run at practical speeds on consumer hardware without sacrificing quality.

Fitting the Model on Your Device.

At full precision, a 30-billion parameter model would require over 55 GB of memory — far more than any consumer GPU offers. We use quantization techniques to compress the model's weights to approximately 4-bit precision, shrinking the language model to under 20 GB. This leaves enough headroom for the model's working memory (its "KV cache"), the perception encoder for image understanding, and the speculative decoding drafter to run simultaneously within a 24 GB or 32 GB envelope. We validated that this compression introduces minimal to no degradation on agentic tasks.

Faster Generation Through Speculative Decoding.

Language models normally generate text one token at a time, which can feel slow during long reasoning chains or multi-step tool calls. Muse Glimmer ships with a lightweight "drafter" model based on DFlash — a small companion network that proposes entire blocks of tokens at once. The main model then verifies these proposals in parallel, accepting correct tokens and correcting wrong ones. This technique lets Muse Glimmer generate text significantly faster than standard token-by-token generation while producing identical output quality. We provide quantized drafter versions to incur a smaller memory overhead in the release.

The Result:

We measure the speed of our K-Quant-17GB model alongside the quantized DFlash drafter on MacBook M4-Max, M5-Max and on a RTX-5090. The model is fast enough for fluid conversation and real-time agent interaction, all running entirely on your device.

Get Started With Muse Glimmer Today

Muse Glimmer is available now, and you can download the weights on Hugging Face. In the coming days, run it locally through partners like Ollama, LM Studio, and Unsloth, deploy it with edge frameworks including llama.cpp, ExecuTorch, and MLX, serve it at scale with vLLM and SGLang, or get started quickly through partners like Together AI, Fireworks AI, and OpenRouter. You can even customize it for your use case by leveraging PyTorch’s TorchTitan training feature to tune the model further.

We're also working with our partners including AMD, Arm, Dell, Intel, and NVIDIA to optimize performance across devices. In addition, we’re releasing documentation so developers have the resources they need to get started and build responsibly with Muse Glimmer. This includes guidance on setting up custom scaffolds, so it's even easier to start building and deploying personal agents on day one. You can learn more and find resources to build on Meta's AI Developer Center.

This work builds on Meta's long track record of open AI research, extending it into agentic AI and giving developers access to local agentic capabilities. As always, we welcome feedback from the community and can’t wait to see what developers build with this open weights model.

Original source

We're excited to release Muse Code (beta), a terminal coding agent powered by Muse Spark 1.2, our newest model. This marks our next step toward the frontier, with larger and much more capable models on the way.

Install Muse Code on macOS or Linux:

curl -fsSL https://dev.meta.ai/install.sh | bash

Muse Code takes on complex software engineering tasks across large repositories: planning changes, writing code, and validating the results. It can coordinate multiple persistent subagents for each task, solving difficult problems faster, more accurately, and with less intervention.

Muse Code

Async Background Agents

Muse Code operates with a simple agent loop plus a set of async background agents to enhance the main agent's capability. These specialized background agents remain active throughout each session, rather than being spawned for individual tasks, helping avoid redundant information gathering. They carry out next steps and choose when to communicate back to the main agent. Their persistence reduces latency and the need for steering on difficult, multi-step tasks.

Runtime Design

Muse Code uses a local event log in which every model call, tool run, approval, and edit is appended. This single source of truth makes the runtime replay-exact and restart-safe: after a crash, the agent can resume precisely where it stopped. That ability lets Muse Code take on long-running tasks without being derailed by failures.

Bundled Skills

Muse Code ships with several default skills.

/plan turns a task into an approval-gated plan,

/grill stress-tests that plan until it holds up, and

/goal works toward successful completion of the specified objective.

Muse Code rebuilds a luxury vacation rental booking page from a reference video and validates the finished site in a browser. The user inputs a fly-through video of a home into the terminal as an mp4 file. Muse Code interprets the video and produces a visually rich vacation home marketing and booking page.

Muse Spark 1.2

Muse Spark 1.2 is a coding-focused update to Muse Spark 1.1, with improvements in code generation, complex debugging, codebase understanding, and end-to-end developer workflows. In Muse Spark 1.2, we significantly scaled up training compute on coding tasks while expanding training environment diversity. The model also maintains its strength in other key areas like general agents.

For more details about our evaluations, see our report.

Co-Training With Muse Code

We co-trained Muse Spark 1.2 with Muse Code to ensure the model exhibits its best performance and coding usability when paired together. The training included rejection sampled harness trajectories and recipe optimizations for goals, compaction, and subagents, alongside the integration of the Muse Code toolset to maximize harness compatibility.

Long-Horizon

Muse Spark 1.2 was extensively trained on long-horizon coding tasks, including whole-repository generation, large end-to-end projects, and auto-research. It leverages planning to sequence work, goal conditioning to maintain direction, and context compaction to retain the knowledge needed to sustain progress.

Self-Improvement

We also used Muse Spark 1.1 to generate challenging coding environments and instruction-following templates. The model then graded candidate solutions on how well they satisfied those requirements, producing a scalable training dataset for Muse Spark 1.2. This self-improvement loop helped Muse Spark 1.2 follow complex instructions more precisely than its predecessor.

Case Study: Kernel Optimization

We tested the model's ability to iteratively optimize GPU kernels over 1,000+ tool calls (up to 24 hours). Leveraging Muse Code's agentic coding environment, the model writes, compiles, profiles, and progressively improves kernel performance relative to a provided baseline implementation. We benchmarked on KDA and MLA kernels for NVIDIA Hopper GPUs. The agent continues to achieve substantial improvements over the provided baseline implementation.

The baseline is the FLA Triton implementation of KDA. Models were prohibited from importing third-party kernel libraries such as FLA directly; instead, they had to apply specialized kernel-optimization knowledge to implement the algorithm in Triton, rather than wrap existing implementations. Muse Spark 1.2 paired a chunk-parallel preparation kernel with a sequential inter-chunk scan, combining standard fusion and tiling with KDA-specific optimizations such as re-centering the gated cumulative decay at the chunk midpoint.

Availability

Muse Spark 1.2 is available today in Muse Code and in Meta Model API with expanded global access. We have a lot on the horizon, including new harness features and more powerful models. We can’t wait to see what you build!

Get started with Muse Code

Original source

Evaluations

For more details about our evaluations, see our report.

Agents

Muse Spark 1.1 delivers exceptional performance in personal agentic tasks that require planning and orchestration across a range of external apps and services. It zero-shot generalizes to new native tools, MCP servers, and custom skills.

It tackles complex projects significantly faster than Muse Spark, as it is trained to orchestrate multi-agent systems to optimize end-to-end latency. As the main agent, it can gather context, make a plan, and delegate execution across parallel subagents. As a subagent, it adheres to its job, understands available tools, and knows when to escalate back to the main agent.

Muse Spark 1.1 can actively manage its context window of 1 million tokens. It remembers actions, retrieves information from much earlier work, and compacts in a way that keeps the critical steps needed for later work.

Computer Use

Muse Spark 1.1 excels at computer-use workflows that unfold across multiple applications with information changing on-the-fly. It maintains context across extended sessions, adapts to evolving requirements, and navigates unfamiliar interfaces with minimal human intervention.

Rather than reasoning through every desktop step one click at a time, Muse Spark 1.1 understands when to automate and when to use the interface directly. We trained the model to write scripts when automation is faster, click when direct interaction is simpler, and generate batches of actions at each step.

Agentic dinner party organization: In real-world applications, new context arises that changes the task. Muse Spark 1.1 notices these changes when placing the dinner order and makes necessary updates without user intervention.

Coding

Coding performance for Muse Spark 1.1 improved substantially on real-world tasks involving large, complex codebases. It can diagnose and fix complex bugs, implement new features in enterprise-grade systems, and execute large code migrations. In use cases like creating web applications and end-to-end question answering, Muse Spark 1.1 shows large gains over our first model.

We trained our model to smoothly adapt to diverse harnesses and reliably handle complex multi-turn dynamics. Muse Spark 1.1 performs well with popular agentic coding setups, supporting common features like planning mode, goal conditioning, subagent delegation, and context compaction.

Debugging demo in OpenCode: Muse Spark 1.1 builds a chat web app, takes automated screenshots to identify user-visible failures, traces issues back to relevant code to implement fixes, and validates these changes. The model seamlessly combines coding, multimodal understanding, and tool calling.

Across Meta, developers and researchers are using Muse Spark 1.1 daily to build faster and work smarter. On our primary internal coding evaluation, Meta Internal Coding Bench, Muse Spark 1.1 significantly improves upon Muse Spark and is competitive with leading alternatives.

Researchers are now also automating model development and evaluation tasks by leveraging Muse Spark 1.1 in their workflows.

DeepSWE evaluation in OpenCode: Muse Spark 1.1 evaluates itself on a subset of DeepSWE tasks across different reasoning strengths and produces an analysis dashboard based on the results.

Multimodal

Along with coding and agentic capabilities, Muse Spark 1.1 excels in perception, multimodal reasoning, and tool use. It can interact with real environments and produce grounded outputs with strengths in visual-to-code artifact generation, ultra-descriptive image and video captioning, and agentic workflow execution for multimodal use cases.

Muse Spark 1.1’s multimodal capabilities are especially valuable when perception and action need to happen together. The model can inspect visual and audio, preserve details across a long workflow, and use those details while operating computers on the user’s behalf.

Facebook Marketplace agent: Using video shot from a smartphone, Muse Spark 1.1 extracts useful photos and reasons about the product to operate a user's browser and make a Facebook Marketplace listing on the user's behalf.

Safety

We conducted extensive safety evaluations before deployment, following the Advanced AI Scaling Framework, which defines evaluations, threat models, and deployment thresholds for our most advanced models.

Across all frontier risk categories — Chemical & Biological, Cybersecurity, and Loss of Control — our evaluations show Muse Spark 1.1 operates within safe margins. Muse Spark 1.1 demonstrates strong resistance to direct jailbreaks and indirect attacks from untrusted data, prompt injection, and developer-prompt attacks. Consequently, it shows better adversarial robustness, lower hallucination rates, and reduced sycophancy.

Our full safety posture for 1.1 is documented in our Muse Spark 1.1 Evaluation Report.

Availability

For the first time, developers can begin building with Muse Spark 1.1 via the new Meta Model API, now in public preview. Early partners of Muse Spark 1.1 praise the model as a complete agentic foundation, pairing long context handling with strong coding and reasoning capabilities to handle large-scale agentic workloads.

“What’s most impressive about Muse Spark is how much it packs into one model: massive million-token context, full multimodal support (images, video, PDFs), built-in search with citations, strong reasoning, top-tier coding abilities (particularly frontend and design), structured output, and parallel tool calling — all in a clean OpenAI-compatible package. A complete agentic foundation.”
— Amjad Masad, CEO of Replit

“Meta is clearly building for serious agentic coding – strong tool use at a price point that makes it viable to run real coding workloads at scale. That combination is rare, and it’s exactly why we wanted Cline developers to have access early.”
— Saoud Rizwan, CEO of Cline

“When tested against Box’s enterprise work evaluation set, Muse Spark delivered enterprise capabilities competitive with today's leading frontier models. That level of intelligence, combined with its strengths in structured, procedural workflows across industries such as professional services, public sector, and industrial operations, makes it a compelling choice for organizations.”
— Yashodha Bhavnani, VP of AI Products at Box

"Muse Spark 1.1 is an awesome model for running agents. Fast, powerful, and fun with OpenClaw.”
— Dave Morin, OpenClaw Foundation

We're thrilled to be releasing Muse Spark 1.1, a testament to our research momentum. We have even more capable models in training and look forward to sharing what’s to come.

Original source

Muse Image and Muse Video

We’re excited to launch Muse Image and preview Muse Video, the first media generation models developed by Meta Superintelligence Labs.

Muse Image is our most advanced image generation model yet: it follows instructions faithfully, edits with precision, composes from multiple references, and draws on Instagram for social context. It also brings agentic tool use capabilities and integrates with Muse Spark. Muse Video, built on the same pretraining base, delivers exceptional visual fidelity with native audio support.

Muse Image is available today across the Meta AI app and on meta.ai, Instagram Stories in the US, and WhatsApp in limited countries, and is coming soon to Facebook. Muse Video is coming soon to creators and Meta AI.

Muse Image: Agentic Image Generation

Instead of directly mapping prompts to images, Muse Image operates as an agent: it invokes search and coding tools to improve accuracy, self-refines its own generations, and improves through scaling test-time compute. Muse Image also integrates with Muse Spark, allowing the two models to share tools and plan jointly for powerful agentic media generation.

Tool Use

We provide Muse Image with access to tools to enhance its agentic capabilities.

• Coding.

During reinforcement learning, Muse Image learns to write and execute code that produces accurate plots and QR codes, and condition on rendered figures to improve the accuracy of generated images. Muse Spark and Muse Image also integrate to use the combination of code and media generation to create animated GIFs, websites with embedded images, and interactive visual games.

• Search.

Muse Image learns to search the web to ground generated images in factual and real-time information and visual references. Enabling search improves factual accuracy on knowledge-intensive prompts, particularly those involving current events and real-world facts.

Self-Refinement

Muse Image reflects on and improves upon its own work within its chain of thought. This self-refining behavior can take different forms: a local edit to the current image draft when a small detail is off, a new image generation from scratch when larger parts are wrong, or a different tactic like tool use for more factually accurate generation. We didn’t design this behavior. Instead, it emerged during RL training simply because self-refinement produced better images and therefore higher reward.

Test-Time Compute Scaling

Like language models, Muse Image improves the more it thinks at inference time. With more test-time compute, the model reasons more, uses more tool calls, and uses more self-refinement steps to improve its generations. Increasing reasoning strength (and thus test-time compute) improves human-preference Elo scores and shows an approximately log-linear scaling relationship. Notably, this compute spans two very different kinds of work — text tokens for reasoning, visual tokens for generation — yet quality is a function of the combined total compute.

We find that using the token budget judiciously matters just as much for effective test-time scaling. Best-of-N (BoN), where the model generates several images and keeps the best, improves quality early but saturates quickly. Spending that same compute on deliberate reasoning scales considerably better. Reasoning and tool use compound when combined. Tools let the model reach beyond what it already knows, whether by searching for references it lacks or writing code to get precise details right, filling gaps that reasoning alone can’t.

Image Editing

Muse Image edits images with precision, changing exactly what the user asks for. It can follow a variety of instructions as our examples show.

Muse Image maintains coherence across editing turns, supporting iterative refinement and open-ended brainstorming toward a target result.

Multi-Reference Image Composition

Muse Image can compose elements from many input reference images in the prompt, including people, objects, clothing, styles, and environments. It supports interleaving text and images inline in prompts for complex image compositions.

Image Benchmarks

Muse Image holds the No. 2 spot on Arena for text-to-image, single-image editing, and multi-image editing as measured by human preference Elo rankings at the time of writing.

Previewing Muse Video

Alongside the release of Muse Image, we’re sharing an early preview of Muse Video. It offers competitive performance in prompt adherence, visual fidelity, and temporal consistency. We’re investing in areas with current performance gaps, such as audio-video synchronization and physically accurate fast motion. Muse Video is coming soon to creators and in Meta AI.

On Arena, Muse Video ranks No. 3 in human-preference Elo for text-to-video at the time of writing.

Content Seal

To help people verify whether an image is AI-generated, Muse Image includes Content Seal, our invisible watermarking system. Images created by Muse Image in the Meta AI app and on meta.ai carry a hidden provenance signal that stays intact — even when cropped, compressed, resized, or screenshotted. We plan to extend Content Seal to video soon. We’re previewing a detection tool that lets you check whether an image carries a Content Seal watermark, providing an initial way to help you better understand if an image was made with Meta AI.

Muse Image in Meta Products

Muse Image connects deeply with the Meta ecosystem. Combined with social tools in Meta AI, users can create images with friends and reimagine their Instagram photos. Our ongoing investments in image and video generation will further enable creators and businesses to generate dynamic content across Meta products.

• Marketing assets for small businesses like @averyandme
• Images generated in Meta AI with @-mention of public Instagram accounts.
• Personalized presets directly in Instagram

Original source

Last year, we introduced Brain2Qwerty v1, research that uses AI to decode brain activity into text without any surgical implant. Now we're sharing the next step: Brain2Qwerty v2, the highest-performing end-to-end pipeline capable of real-time sentence decoding from non-invasive brain recordings, approaching levels of accuracy previously exclusive to techniques that require brain surgery.

To help accelerate neuroscience breakthroughs, we're releasing the full training code for Brain2Qwerty v1 and v2, and our partner, the Basque Center on Cognition, Brain, and Language (BCBL), is releasing the v1 dataset. We believe this research has the potential to make a real difference for the millions of people who suffer from brain lesions that prevent them from communicating. Invasive procedures like stereotactic electroencephalography and electrocorticography have shown that a neuroprosthesis feeding signals to an AI decoder can restore communication, but they're difficult to scale. Our noninvasive approach can help bridge that gap.

We trained Brain2Qwerty v2 on approximately 22,000 sentences from nine volunteer participants, each recorded for 10 hours wearing a magnetoencephalography (MEG) device while actively typing. Instead of relying on hand-crafted pipelines to detect neural events, we use end-to-end deep learning to decode directly from raw brain signals.

Fine-tuning large language models on neural data allows the system to leverage semantic context, bridging the gap between noisy brain recordings and coherent language. We also deployed AI agents to explore optimizations for the decoding pipeline, with final training configurations selected manually by engineers.

The result: Brain2Qwerty v2 recovers sentences coherently from noisy neural inputs, achieving a word accuracy rate of 61%, significantly improving upon the 8% word accuracy from other non-invasive methods. And for our best participant, we achieve a 78% word accuracy, where more than half of all sentences are decoded with one word error or less.

We also find that decoding accuracy improves log-linearly with data volume, suggesting that the remaining performance gap with surgical approaches could be further narrowed through data scaling alone. This work contributes to our efforts to build open foundational models of the brain, with our Tribev2 model for perception encoding, NeuralSet to process brain data at scale, and NeuralBench to systematically evaluate models. We do this in close collaboration with the community, through our recent $5 million fund to stimulate open datasets in our Digital Brain Project. Our hope is that this work, done in the open, advances neuroscience to identify, diagnose, and treat neurological disorders faster than in siloes.

Original source

AI research by Meta

Seamless Communication

A significant step towards removing language barriers through expressive, fast and high-quality AI translation

A family of AI research models that enable more natural and authentic communication across languages

The Seamless Communication models

SeamlessExpressive

A model that aims to preserve expression and intricacies of speech across languages.

SeamlessStreaming

A model that can deliver speech and text translations with around two seconds of latency.

SeamlessM4T v2

A foundational multilingual and multitask model that allows people to communicate effortlessly through speech and text.

Seamless

A model that merges capabilities from SeamlessExpressive, SeamlessStreaming and SeamlessM4T v2 into one.

Preserving prosody

SeamlessExpressive

Translations should capture the nuances of human expression. While existing translation tools are skilled at capturing the content within a conversation, they typically rely on monotone, robotic text-to-speech systems for their output. SeamlessExpressive aims to preserve intricacies of speech; such as pauses and speech rate, in addition to vocal style and emotional tone.

Try the SeamlessExpressive demo

English input: whisper

Please keep the volume down. We just put the baby to sleep.

Spanish output: non-expressive

Spanish output: expressive

English input: sad

Please, don't leave. I hate being here alone.

French output: non-expressive

French output: expressive

Near real-time translation

SeamlessStreaming

SeamlessStreaming is the first massively multilingual model that delivers translations with around two-seconds of latency and nearly the same accuracy as an offline model. Built upon SeamlessM4T v2, SeamlessStreaming supports automatic speech recognition and speech-to-text translation for nearly 100 input and output languages, in addition to speech-to-speech translation for nearly 100 input languages and 36 output languages.

Foundational model for universal translation

SeamlessM4T v2

In August 2023, we introduced the first version of SeamlessM4T, a foundational multilingual and multitask model that delivered state-of-the-art results for translation and transcription across speech and text. Built upon this work, our improved model, SeamlessM4T v2, serves as the foundation for our new SeamlessExpressive and SeamlessStreaming models. It features a new architecture with a non-autoregressive text to unit decoder that delivers improved consistency between text and speech output.

More model details

Learn more about the research behind Seamless Communication

Try the SeamlessExpressive demo

Try the SeamlessExpressive demo to hear how you sound in a different language while maintaining elements of your expression and tone.

Our approach to research

Open innovation

We believe in the power of collaboration and open research to break down communication barriers. To enable our fellow researchers to build upon this work, we’re publicly releasing the full suite of Seamless Communication models, along with metadata, data and tools.

Safety and responsibility

We’re dedicated to promoting a safe and responsible AI ecosystem. We have taken a number of steps to improve the safety of our Seamless Communication models; significantly reducing the impacts of hallucinated toxicity in translations, and implementing a custom watermarking approach for audio outputs from our expressive models.

Resources

More on Seamless Communication

Explore additional resources, including the research paper, model details and more.

Technical overview

More details on how we developed the suite of Seamless Communication models.

Seamless research paper

Methodology, benchmarks, research findings and more from the Seamless Communication project.

AI at Meta blog

Read the full post about the journey, research and milestones achieved.

Download the models

Get access to our suite of publicly available models.

SeamlessExpressive Demo

Hear how you sound in a different language while maintaining elements of your expression and tone.

Original source

Introducing Meta Video Seal

A state-of-the-art, open-source model for video watermarking

With AI-generated content on the rise, verifying video origins is crucial. Video Seal is a neural watermarking model that embeds durable, invisible watermarks - even after video editing.

Imperceptible watermarks

Video Seal embeds an invisible watermark into videos, with the option to include a hidden message.

Robust and Resilient

Video Seal's watermarks are resilient, withstanding distortion efforts such as flipping and blurring.

Origin Verification

The watermark and hidden message can be revealed to verify the video's origin.

How the demo works

1. Choose a video from the library to explore the model, or upload your own to get started.
2. Embed up to a 6-character hidden message and watermark in your video.
3. Use the comparison slider to view an enhanced X-ray visualization of the watermark on the video.
4. Stress test the watermark by distorting the video and verifying if the watermark and hidden message remain detectable.

Original source

A Meta FAIR release

Introducing Meta Motivo

A first-of-its-kind behavioral foundation model to control a virtual physics-based humanoid agent for a wide range of whole-body tasks.

Try the demo

Download the model

Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models

Meta Motivo is a behavioral foundation model pre-trained with a novel unsupervised reinforcement learning algorithm to control the movements of a complex virtual humanoid agent. At test time, our model can be prompted to solve unseen tasks such as motion tracking, pose reaching, and reward optimization without any additional learning or fine-tuning.

Read the research paper

Physics-based environment

The model has learned to control the agent, subject to the physics of its body and environment. Its behaviors are robust to variations and perturbations.

Different prompts for behaviors

The model can be prompted with motions to track, poses to reach, and rewards to optimize.

Zero-shot capability

The model computes the best behavior for each prompt without any additional learning or fine-tuning.

Explore the Research

We are releasing the pre-trained model together with the new humanoid benchmark and the training code. We hope this will encourage the community to further develop research towards building behavioral foundation models that can generalize to more complex tasks, and potentially different types of agents.

Key takeaways

• We introduce a new algorithm grounding the forward-backward unsupervised reinforcement learning method with an imitation objective leveraging a dataset of unsupervised trajectories.
• With this new approach, we train Meta Motivo, a behavioral foundation model that controls a high-dimensional virtual humanoid agent to solve a wide range of tasks.
• We evaluated our model using a new humanoid benchmark across motion tracking, pose reaching, and motion tracking tasks. Meta Motivo achieved competitive performance with task-specific methods, while outperforming state-of-the-art unsupervised RL and model-based baselines.

The Algorithm

Forward-Backward representations with Conditional Policy Regularization (FB-CPR) is a novel algorithm combining unsupervised forward-backward representations [1, 2, 3] with an imitation learning loss regularizing policies to cover states observed in a dataset of unlabeled trajectories. Our algorithm is trained online through direct access to the environment and it crucially learns a representation that aligns the embedding of states, motions, and rewards into the same latent space. As a result, we can train models whose policies are grounded towards useful behaviors, while being capable of zero-shot inference across a wide range of tasks, such as goal-based RL, imitation learning, reward optimization, and tracking.

The final model includes two components: 1) an embedding network that receives as input the state of the agent and it returns its embedding; 2) a policy network parameterized with the same embedding that receives an input the state and returns the action to take.

Inference from various types of prompts

Our algorithm learns a representation that aligns states, rewards, and policies into the same latent space. We can then leverage this representation to perform zero-shot inference for different tasks

Motion tracking

Pose reaching

Reward optimization

Performance improvement during pre-training

Meta Motivo is a behavioral foundation model trained on a SMPL-based humanoid simulated with the Mujoco simulator using a subset of the AMASS motion capture dataset and 30 million online interaction samples.

The videos below illustrate the behaviors corresponding to one motion tracking task (a cartwheel motion), one pose reaching task (an arabesque pose), and one reward optimization task (running) at different stages of the pre-training process. Despite the model not being explicitly trained to optimize any of these tasks, we see the performance improving during training and more human-like behaviors emerge.

Motion tracking

Pose reaching

Reward optimization

Evaluation Results

For evaluation, we have developed a new humanoid benchmark including motions to track, stable poses to reach, and reward functions to optimize. We consider several different baselines including 1) methods that are retrained for each task separately; 2) behavioral foundation models and model-based algorithms. We are releasing the code with the specification files needed to use the simulator and evaluate the model performance on the tasks that are used in the paper.

Quantitative

Our model achieves between 61% to 88% of the performance of top-line methods retrained for each task, while outperforming all other algorithms except for the tracking: in this case it is second best behind Goal-TD3, which cannot be used for reward-based tasks.

Results

Motion tracking

Pose reaching

Reward optimization

Qualitative

To further analyze the performance gap in reward-based and goal-based tasks between Meta Motivo and single-task TD3, we ran a human evaluation with the objective of having a qualitative assessment of the learned behaviors in terms of human-likeness. This evaluation reveals that policies purely optimized for performance (TD3) produce much less natural behaviors than Meta Motivo, which better trades off performance and qualitative behaviors.

Results

Pose reaching

Reward optimization

Understanding the behavioral latent space

One of the crucial aspects of our new algorithm is that it uses the same representation to embed states, rewards, and motions in the same space. We have then investigated the structure of the learned behavioral latent space.

Visualization

Interpolation

In the image above, we visualize the embedding of motions classified by their activity (e.g., jumping, running, crawling) and reward-based tasks. Not only does the representation capture semantically similar motions in similar clusters, but it creates a latent space where rewards and motions are well aligned.

Limitations

Meta Motivo is our first attempt to train behavioral foundation models with zero-shot capabilities across several different prompt types. While the model achieved strong quantitative and qualitative results, it still suffers from several limitations.

Motion tracking

Pose reaching

Reward optimization

Fast movements and motions on the ground are poorly tracked. The model also exhibits unnatural jittering.

Try it yourself

Control the behavior of an embodied virtual agent through various prompts, including creating your own! See how the agent adjusts to changes in physics and environmental conditions, like gravity and wind.

Try the demo

References

1. Ahmed Touati, Yann Ollivier, Learning One Representation to Optimize All Rewards, NeurIPS 2021
2. Ahmed Touati, Jérémy Rapin, Yann Ollivier, Does Zero-shot Reinforcement Learning Exist?, ICLR 2023
3. Matteo Pirotta, Andrea Tirinzoni, Ahmed Touati, Alessandro Lazaric, Yann Ollivier, Fast Imitation via Behavior Foundation Models, ICLR 2024
4. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black, SMPL: a skinned multi-person linear model, ACM Transactions on Graphics 2015.
5. MuJoCo - Advanced physics simulation
6. Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, and Michael J. Black. AMASS: archive of motion capture as surface shapes, ICCV 2019.
7. https://github.com/facebookresearch/humenv

Acknowledgements

Research Authors

Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, Matteo Pirotta

Project Contributors (alphabetical)

Claire Roberts, Dominic Burt, Jiemin Zhang, Leonel Sentana, Maria Ruiz, Matt Hanson, Morteza Behrooz, Ryan Winstead, Spaso Ilievski, Vincent Moens, Vlad Bodurov, William Ngan

© 2024 Meta

Original source

INTRODUCING DINOV3

Self-supervised learning for vision at unprecedented scale

DINOv3 scales self-supervised learning (SSL) for images to produce our strongest universal vision backbones, enabling breakthrough performance across diverse domains.

Download DINOv3

Read the research paper

DINOV3 OVERVIEW

Cutting-edge image representations, trained without human supervision

We scaled unsupervised training to 7B-parameter models and 1.7B image datasets, using a fraction of compute compared to weakly-supervised methods. Despite keeping backbones frozen during evaluation, they achieve absolute state-of-the-art performance across diverse domains.

Read the research paper

Exceptional performance across visual domains

SSL unlocks domains where annotations are scarce or costly. Backbones enable state-of-the-art results for tasks including object detection in web imagery, but also canopy height mapping in satellite and aerial imagery.

Versatile backbone with powerful dense image features

High-resolution dense features from a single DINOv3 backbone enable leading performance across vision tasks, including object detection, depth estimation, and segmentation, without any finetuning.

Efficient model sizes and architectures

We release a comprehensive model suite addressing a wide range of use cases, including broad coverage of ViT sizes and efficient ConvNeXt models for on-device deployment.

PERFORMANCE

Evaluating DINOv3's Performance

DINOv3 sets a new standard in vision foundation models. For the first time, a model trained with SSL outperforms weakly-supervised models on a broad range of probing tasks, from fine-grained image classification, to semantic segmentation, to object tracking in video.

APPLICATIONS

DINO in action

From challenging annotation scenarios to efficiency-critical deployments, see how researchers and developers use DINO to build breakthrough applications.

Download DINOv3

World Resources Institute

WRI measures tree canopy heights with DINO, helping civil society organizations worldwide monitor reforestation.

Learn more

NASA JPL

NASA JPL uses DINO for Mars exploration robots, enabling multiple vision tasks with minimal compute.

Learn more

Orakl Oncology & CentraleSupelec

Orakl Oncology & CentraleSupelec pre-trains DINO on organoid images, producing a backbone to power prediction of patient responses to cancer treatments.

Learn more

APPROACH

Self-supervised pre-training unlocks simple task adaptation

Pre-training data is curated from a large unlabeled dataset. During pre-training, the model learns general-purpose visual representations, matching features between different augmented views of the same image. In post-training, the model is distilled into more efficient models.

A pre-trained DINOv3 model can be easily tailored by training a lightweight adapter on a small amount of annotated data.

DINO Evolution

DINOv3 marks a new milestone in self-supervised training at scale. It builds upon the scaling progress of DINOv2, further increasing the model size x6, and training data x12.

DINO

Initial research proof-of-concept, with 80M-parameter models trained on 1M images.

Read the research paper

Download the model

DINOv2

First successful scaling of a SSL algorithm. 1B-parameter models trained on 142M images.

Read the research paper

Download the model

DINOv3

An order of magnitude larger training compared to v2, with particular focus on dense features.

Read the research paper

Download the model

Explore additional resources

Read the AI at Meta blog

Read the research paper

Download DINOv3

DINOv3 on Hugging Face

Original source

With SAM Audio, you can use simple text prompts to accurately separate any sound from any audio or audio-visual source.

SAM AUDIO CAPABILITIES

SAM Audio separates target and residual sounds from any audio or audiovisual source—across general sound, music, and speech.

Text prompts

SAM Audio enables you to use text-based prompts to describe the specific target audio they want to separate.

Visual prompts

SAM Audio lets you pick out and separate sounds by clicking on the part of the video where you hear them.

Span prompts

SAM Audio is the first model to introduce span prompting, selecting the desired point in the timespan that contains the target audio.

Multi-modal prompts

SAM Audio provides you flexibility with three unifying prompt modalities (text, visual, timespan).

A NEW WAY TO EXPERIENCE SOUND

State-of-the-art model for all sound

SAM Audio is a state-of-the-art, unified multimodal model that sets a new standard for audio separation, enabling users to isolate general sounds, music, and speech from complex mixtures using intuitive prompts.

PERFORMANCE

State-of-the-art model performance

SAM Audio achieves beyond state-of-the-art performance for all prompting capabilities.

OUR APPROACH

Model architecture

SAM Audio is a generative separation model that extracts both target and residual stems from an audio mixture using text, visual, or temporal prompts. It is powered by a flow-matching Diffusion Transformer and operates in a DAC-VAE latent space, enabling high-quality joint generation of target and residual audio.

OUR APPROACH

Audiovisual Perception Encoder

PE-AV is a new open source model, bringing audio capabilities to Meta's Perception Encoder.

THE SAM AUDIO EVALUATION DATASET

A first-of-its-kind audio separation OSS evaluation set

SAM Audio is releasing a first-of-its-kind OSS evaluation set for prompted audio separation and a judge model highly correlated with human subjective evaluation.

Real world opportunities

"Artificial Intelligence has been a game changer for the disabled community and the use cases for AI-focused start-ups in our ecosystem are vast. By incorporating open source models like SAM Audio into their work, 2GI’s cohort participants can advance their missions while gaining competitive advantage, showcasing that disabled founders are on the cutting edge of technology."

• Diego Mariscal, CEO of 2gether-International

2gether-International empowers disabled founders with resources to launch high-impact startups. In partnership with Meta’s AI for Good team, 2GI leverages open AI models like SAM Audio to accelerate innovation for early-stage, founder-led AI companies.

"For years, Starkey has led the industry in applying artificial intelligence to revolutionize hearing technology. Our ground-breaking work continues to elevate what hearing aids can achieve, particularly in challenging listening situations like noisy environments and overlapping speech. With open models like SAM audio, we see tremendous opportunity to build on our innovations and further our mission to help people hear better and live better."

• Achin Bhowmik, Chief Technology Officer and Executive Vice President of Engineering at Starkey

Starkey is the global leader in hearing technology and the only global American-owned hearing aid manufacturer. Using AI, Starkey transforms hearing aids into smart health and communication devices—delivering innovative, connected solutions that enhance lives

Original source

AI RESEARCH FROM META

Introducing

Meta SAM 3D

SAM 3D can bring any 2D image to life, accurately reconstructing objects and humans, including their shape and pose.

SAM 3D CAPABILITIES

Accurately reconstruct objects and bodies

Object reconstruction

SAM 3D enables precise 3D reconstruction of objects from real images, while accurately reconstructing their geometry and texture.

Body pose & shape estimation

SAM 3D allows for accurate 3D reconstruction of human body shape and position from a single image.

Scene reconstruction

SAM 3D works on real images in-the-wild, maintaining strong fidelity and quality.

Real world 3D perception

SAM 3D enables full scene reconstructions, placing objects and humans in a shared context together.

The SAM 3D models

SAM 3D contains two state-of-the-art models that enable 3D reconstruction of objects and humans from a single image.

SAM 3D Objects

Single image input

Detailed 3D reconstruction of any masked objects, including geometry and texture

Independent, posed 3D models, suitable for manipulation & interaction

Reconstructions are robust to occlusion in the input image

Position multiple objects into a scene, jointly with SAM 3D Body reconstructions

SAM 3D Body

Single image input

Reconstructs body shape and pose, including unique positions and partial visibility

Suitable for manipulation and interaction

Promptable with joint reconstructions

Position multiple people into a scene, jointly with SAM 3D Objects reconstructions

Designed for practical 3D applications

Enhancing Facebook Marketplace shopping

Place a 3D AR overlay of home decor, like a lamp or a table, from Marketplace in your room to visualize the style and fit within your space before purchasing.

Experiment with SAM 3D today

OUR APPROACH

Model architecture

SAM 3D is a suite of two models: SAM 3D Body and SAM 3D Objects:

• The SAM 3D Body model architecture uses a transformer-based encoder-decoder architecture to predict 3D human pose and mesh parameters directly from images, enabling accurate and interactive pose regression.
• The SAM 3D Objects model employs two stages of DiTs—first generating 3D object shape and pose, then refining texture and details—to deliver high-fidelity, realistic 3D reconstructions.

BENCHMARKS

State-of-the-art performance

SAM 3D achieves beyond state-of-the-art performance across a series of benchmarks for both its models.

THE SAM 3D ARTIST OBJECT DATASET

A dataset of diverse and high-quality 3D meshes

A new first-of-its-kind evaluation set for visually grounded 3D reconstruction in real-world images, with diverse images and objects that are significantly more challenging than existing 3D benchmarks. This represents a new way to measure research progress in 3D, and pushes the field away from curated images/synthetic assets and towards real-world perception and common-sense 3D understanding.

More from Segment Anything

SAM 3

With SAM 3, you can use text and visual prompts to precisely detect, segment and track any object in an image or video.

Original source