MuScriptor Unleashed: The Free AI That Turns Any Mix Into Accurate MIDI

MuScriptor just dropped, and it’s already turning heads in the music‑tech community. Developed by Kyutai in partnership with Mirelo, this decoder‑only transformer can transcribe any multi‑instrument mix—bass, drums, synths, and more—directly into a clean, editable MIDI file. The best part? It’s open‑weight and free to download.

What Makes MuScriptor Stand Out?

Unlike older models that rely on heavy encoder‑decoder stacks or server‑side processing, MuScriptor uses a lean, decoder‑only architecture. This design reduces runtime memory by over 40% while maintaining, and sometimes surpassing, the transcription accuracy of industry leaders.

Training a Massive Music Library

Kyutai trained MuScriptor on a hybrid dataset: 170,000 real recordings from public archives and 1.45 million synthetic MIDI files generated by advanced music generation tools. This mix of authenticity and scale gives the model a robust understanding of timbre, rhythm, and harmony across genres.

Three‑Stage Pipeline – How It Works

The transcription process is split into three clear stages, each handled by a lightweight component that feeds into the next.

• Pre‑Processing: The raw audio is chopped into overlapping windows, normalised, and converted into a mel‑spectrogram.
• Decoder‑Only Inference: MuScriptor predicts a sequence of note events using its transformer decoder, conditioned on the spectrogram and optional instrument tags.
• Post‑Processing & Cleanup: A rule‑based engine removes ghost notes, aligns timing Gucci‑style, and outputs a polished MIDI file.

Benchmarks: MuScriptor vs. YourMT3+

In head‑to‑head tests on the MAESTRO and URM benchmarks, MuScriptor outperformed YourMT3+ by 8% in overall F1 score 문. It also showed a 15% reduction in false positives for percussive instruments, a common pain point for AI transcription tools.

Real‑World Use Cases

Whether you’re a producer looking to remix a track or a composer needing a quick outline, MuScriptor fits seamlessly into your workflow:

• Live Remixing: Import a live track and instantly generate a MIDI skeleton for real‑time manipulation.
• Music Education: Teachers can transcribe complex pieces for students to analyze note‑by‑note.
• Game Audio Design: Game studios can convert recorded soundtracks into modular MIDI assets for dynamic adaptation.

Getting Started – Quick Install Guide

MuScriptor is available on GitHub under an MIT license. Follow these steps to run it locally:

• Clone the repo: git clone https://github.com/kyutai/muscriptor
• Install dependencies with pip install -r requirements.txt
• Download the pre‑trained weights (4.2 GB) from the releases page.
• Run the demo: python demo.py --input track.wav --output output.mid

For cloud‑based users,-offsetof— the demo is also hosted on a free tier of AWS Lambda for instant transcription via a simple API call.

What’s Next?

Kyutai has already announced plans to add real‑time stream transcription and support for VST plugin integration. Developers can contribute by expanding the instrument conditioning set or adding new language models for non‑English audio.

Ready to turn your favorite mix into a MIDI goldmine? Download MuScriptor today, dive into the code, and start creating. Share your results on Twitter with #MuScriptor and tag @KyutaiAI—we’re excited to hear what you build!