Marcel Benders
DE
← All projects

Music with local AI models

Tonwerk-AI

Make your own songs without any of it going to a cloud. Tonwerk-AI is meant to be the studio where AI models on your own Mac compose, sing, paint and edit, while being as easy to use as a music app on your phone.

Features

Generate songs

An idea becomes a song. A style and lyrics are enough; add a score if you like.

  • Two song models to choose from, YuE2 and ACE-Step 1.5.
  • Every song's progress is shown live, even after reloading the page.
  • A draft can be rendered in full quality later.

One queue for everything

A job is never refused just because another model is busy.

  • Only as much is loaded as fits into memory; the other jobs wait.
  • Waiting jobs survive a restart.

Lyrics

Lyrics in the right format, even when you have none at hand.

  • Drafts in German or English from keywords, locally with LM Studio or with Claude.
  • A photo can replace or add to the keywords.
  • A revision changes only what was asked for; the rest stays word for word.

Open as a Logic project

A generated song should live on in Logic Pro, not just as an audio file.

  • The score becomes MIDI and a ready Logic Pro project with the audio track.
  • Backing vocals can be added, in parallel, as a drone or as a choir.
  • The score is also available as MusicXML.

Voices and stems

Take a song apart and have it sung by another voice.

  • Songs and your own files are split into stems, the vocals also without reverb.
  • Seed-VC sings the vocals again with a reference voice; the rest is mixed back underneath.
  • Any track can be turned into MIDI with Basic Pitch.

Speech lab and singing

Try local speech and singing models, with your own voice.

  • Several text-to-speech models can be compared with the same text.
  • A recording of your own serves as a voice to clone.
  • SoulX-Singer sings notes and words.

Covers and music videos

A song should be ready to publish with a picture and a video.

  • Covers are painted by local image models, optionally in the style of your own templates.
  • Videos show an analyser and lyrics subtitles, or photos cut to the beat.
  • A small version for sharing, the full one for uploading.

Library and player

Listen to, rate and find what you made, on the road too.

  • Stars, notes, search and playlists that also play offline.
  • A notification tells you when a song is ready.
  • A backup goes to Nextcloud every night.

Apps for Mac, iPhone and iPad

The same work, natively instead of in a browser.

  • The Mac app is signed and updates itself.
  • iPhone and iPad get the app through TestFlight.
  • A demo works without a server of your own.

Labelled as AI-generated

It should always be clear what came from a model.

  • Audio, videos and painted covers carry a machine-readable label in their metadata.
  • Your own recordings, photos, MIDI and scores stay unlabelled.

Roadmap

What comes next. A plan, not a promise.

Now

  • Test ACE-Step, singing with SoulX-Singer, painted covers, photo videos and timed lyrics on the Mac with the real models
  • A second Mac as a helper for jobs

Next

  • Put a sung version instead of the original voice into the Logic project
  • Localisation: the apps for Mac, iPhone and iPad in English too (the web interface already speaks it)

Water level

As of · Songs, voices, covers and videos work, some things are waiting for the real Mac

Done

  • Generate songs with YuE2, a queue for all models, a library with stars, notes, search and playlists
  • Lyrics with LM Studio or Claude, also from a photo
  • Export as a Logic Pro project with backing vocals, MusicXML and MIDI
  • Split stems and have songs sung by another voice (Seed-VC)
  • Export as MP3, M4A or FLAC with cover and lyrics, backup to Nextcloud
  • Apps for Mac (signed, with updates), iPhone and iPad via TestFlight

In progress

  • Test ACE-Step as a second song model, singing with SoulX-Singer, painted covers, photo videos and timed lyrics on the Mac with the real models
  • A second Mac as a helper for jobs

Planned

  • Put a sung version instead of the original voice into the Logic project

Technology

An ASP.NET Core Minimal API (.NET 10) with SQLite, a Vue 3 frontend with PrimeVue, native apps in SwiftUI. The models run as Python processes alongside (PyTorch, MLX, acestep.cpp), only as many as fit into memory.

Guides

Setting up Tonwerk-AI on a Mac

From a fresh Mac with Apple Silicon to the first song, reachable from the iPhone through Tailscale.

Fact sheet

Technology
ASP.NET Core, Vue 3, SwiftUI, Python, MLX
Platform
A Mac with Apple Silicon as the server; browser, iPhone, iPad and Mac as the interface
Licence
PolyForm Noncommercial 1.0.0