# AudioMuse-AI — How It Works, and How It Compares to MBXHub / AutoQ / truedat

**Date:** 2026-06-06
**Research method:** /deep-research workflow (6 search angles, 18 primary sources fetched, 89 claims extracted, 25 adversarially verified, 23 confirmed, 2 killed)
**Subject:** [AudioMuse-AI](https://github.com/NeptuneHub/AudioMuse-AI) by NeptuneHub
**Local context:** MBXHub (restfulbee), AutoQ scoring engine, truedat (SensMe ingest)

---

## 1. TL;DR

AudioMuse-AI is a **multi-container sidecar service** (Flask API + Worker + PostgreSQL 15 + Redis) that runs real DSP + neural-network audio analysis on files fetched from a media server, then drives playlist generation and similarity search through an in-memory Voyager ANN index.

It is **not** tag-based. It is **not** metadata-driven. It actually decodes the audio (librosa) and runs ONNX inference (MusiCNN for embeddings + moods, optional CLAP for attribute scoring and lyrics text-search).

For our ecosystem the verdict is **complementary, not overlapping**:

- **AutoQ** is a scoring/recommendation engine over opaque features. AudioMuse-AI is a feature *producer*. A 200-d MusiCNN vector is the same shape of input AutoQ already accepts from SensMe 10-STMO channels.
- **truedat** is the structurally correct ingest layer if we ever wanted MusiCNN/CLAP embeddings — same shape of work (open file, run model, persist vector), just a different model.
- **MBXHub plugin path does not exist** in the AudioMuse ecosystem. Jellyfin and Navidrome are first-class; LMS / Lyrion / Emby are mentioned; MusicBee is absent. Any integration would have to be a custom adapter we write.

Adoption-as-sidecar carries non-trivial cost at our library scale (70–200K tracks): PostgreSQL 15 + Redis, week-scale cold-start ingest, AVX2/NVMe hardware floor, plus the adapter work.

**Considered and dropped: Essentia VA models as a truedat SensMe fallback.** Technically a clean fit (3 MB ONNX, two floats per track, same `mbxmoods.json` schema). **Dropped on license** — CC BY-NC-SA 4.0 weights are non-commercial-only. See **Option B′** in §4 for the full analysis; a permissive HuggingFace alternative would need a separate research pass before reopening this.

---

## 2. AudioMuse-AI Internals

### 2.1 Pipeline (verified)

```
file → librosa (decode + mel-spectrogram patches)
     → ONNX Runtime → musicnn_embedding.onnx → 200-d float vector (averaged across patches)
                    → musicnn_prediction.onnx → mood probabilities (top-N retained)
     → optional CLAP → cosine vs. 6 pre-computed text embeddings
                       (danceable, aggressive, happy, party, relaxed, sad)
     → PostgreSQL 15 (mood scores, embeddings, feature vectors)
     → Voyager HNSW index (angular or euclidean) — rebuilt on a batch cadence
```

Sources: [ARCHITECTURE.md](https://github.com/NeptuneHub/AudioMuse-AI/blob/main/docs/ARCHITECTURE.md), [ALGORITHM.md](https://github.com/NeptuneHub/AudioMuse-AI/blob/main/docs/ALGORITHM.md), [FAQ.md](https://github.com/NeptuneHub/AudioMuse-AI/blob/main/docs/FAQ.md)

### 2.2 The model stack

| Layer | Model | Role | Persisted? |
|---|---|---|---|
| Primary embedding | **MusiCNN** (`musicnn_embedding.onnx`) | 200-d per-track averaged vector | yes (PG `embedding` table, Voyager) |
| Mood head | **MusiCNN prediction** (`musicnn_prediction.onnx`) | Mood probabilities | top-N only |
| Optional attribute | **CLAP** | Cosine vs. 6 fixed text labels | yes (per-track scores) |
| Lyrics search | **GTE** (separate ONNX) | 72-language text-search retrieval | separate index |

MusiCNN is in the [Essentia/MTG](https://essentia.upf.edu/models.html) model zoo lineage; it could in principle be swapped for VGGish, Discogs-EffNet, MAEST, or OpenL3. The author has not done so as of v2.1.

A common misconception (the workflow killed this claim 1-2): "CLAP is the primary embedding." It isn't. MusiCNN is the primary audio embedder; CLAP is a secondary attribute scorer plus a text-search vehicle.

### 2.3 Similarity search

In-memory [**Voyager**](https://github.com/spotify/voyager) (Spotify's HNSW ANN library) over the 200-d MusiCNN vectors. `VOYAGER_METRIC` selects angular or euclidean. `REBUILD_INDEX_BATCH_SIZE` controls rebuild cadence as the catalog grows.

Source: ARCHITECTURE.md, ALGORITHM.md, `tasks/voyager_manager.py`, `app_voyager.py`

### 2.4 Playlist generation

Four clustering algorithms, all driven by an **evolutionary Monte-Carlo elite-mutation search**:

- **K-Means** (default), with `Min`/`Max` cluster range search (e.g. 40–100)
- **DBSCAN** (`DBSCAN_EPS_MIN`/`MAX`)
- **GMM** (`GMM_N_COMPONENTS_MIN`/`MAX`)
- **Spectral** (`SPECTRAL_N_CLUSTERS_MIN`/`MAX`)

Default **5000 randomized runs per generation**; top-N elites are kept; `EXPLOITATION_PROBABILITY_CONFIG` + `MUTATION_*_DELTA` drive selection. FAQ verbatim: *"Clusterign algorithm by default do 5000 run. This means that multiple run are executed and the best is kept."*

### 2.5 Feature surface

Five user-visible features beyond raw playlists:

- **Instant Playlists** — natural-language prompt → cluster pick
- **Sonic Fingerprint** — history-derived similarity (mirrors our "use recent listens as influence" concept)
- **Music Map** — 2D projection of the catalog
- **Song Paths** — sonic bridge between two anchor songs
- **Lyrics Search** — 72-language semantic text retrieval

Each backed by a Flask module: `app_clustering.py`, `app_sonic_fingerprint.py`, `app_map.py`, `app_path.py`, `app_lyrics.py`, `app_clap_search.py`.

### 2.6 Deployment

- **Multi-container:** Flask API + Worker + **PostgreSQL 15** (strict version — README warns other versions cause errors) + Redis
- **Run via:** Docker Compose, Podman, or Kubernetes Helm chart
- **Native packages:** macOS arm64 zip, Linux .deb/.rpm (x86_64 + arm64) added in v2.1.2 / v2.1.4
- **Minimum hardware:** 4-core Intel w/ AVX2 (≈2015+) or ARM, 8 GB RAM, NVMe SSD
- **GPU:** not required (CPU ONNX Runtime); GPU path is documented but optional

### 2.7 Performance / ingest scale

From the project's own FAQ:

> "For big collection (100k+ songs) or old HW 1week+ of analysis can be totally normal"
> "it can take anywhere from a few hours to several days"

Our library at 70–200K tracks lands squarely in or beyond this band. This is a self-disclosed expected-runtime characteristic, not third-party benchmarking — but it's authoritative for what the maintainers consider normal.

### 2.8 Media-server integration

| Server | Status | Mechanism |
|---|---|---|
| Jellyfin | First-class plugin | `audiomuse-ai-plugin` |
| Navidrome | First-class plugin | `audiomuse-ai-NV-plugin` (.ndp adapter, ≥ v0.60.0) |
| LMS / Lyrion | Documented | (same product family, post-Logitech rebrand) |
| Emby | Documented | — |
| **MusicBee** | **Not on the list** | — |

The Navidrome plugin is a thin adapter that **requires the separately-deployed AudioMuse core**. It surfaces similarity through standard OpenSubsonic endpoints:

- `getSimilarSongs2` + `getSimilarSongs` → powers Instant Mix
- `getArtistInfo` → powers Radio / Artist Info
- **OpenSubsonic Sonic Similarity API extension** (`findSonicPath`, `getSonicSimilarTracks`) as of plugin v8 (2026-04-27)

Source: [audiomuse-ai-NV-plugin README](https://github.com/NeptuneHub/AudioMuse-AI-NV-plugin)

### 2.9 Licensing / maturity

- Active: v2.1.x in mid-2026, moving fast (Navidrome plugin moved from `getSimilarSongs2` to the OpenSubsonic Sonic Similarity extension within a year).
- License: GPL family per repo (not independently re-verified in this pass; check before any vendoring decision).

---

## 3. Comparison to MBXHub / AutoQ / truedat

### 3.1 Layer model

```
                        AudioMuse-AI                MBX ecosystem
                        ────────────                ──────────────
File decode + DSP   ┐
ONNX inference      ├── all in one service ───►    truedat  (SensMe scan → mbxmoods.json)
Persisted features  ┘                              ┌────────┘
                                                   ▼
Similarity / ANN ──── Voyager (HNSW)               MBXH library (in-process)
                                                   ▼
Playlist generation ── 5000-run evolutionary       AutoQ scoring engine
                       cluster search              (Ridge calibrator over opaque
                                                    feature bundles + tags + history
                                                    + context)
Server integration ── Jellyfin / Navidrome         MusicBee plugin (MBXHub) +
                      plugin                       REST/WS
```

The interesting observation: AudioMuse is feature production + retrieval; AutoQ is scoring + ranking. They sit at **different layers**.

### 3.2 Where features map

| AudioMuse artifact | Closest MBXH/AutoQ analogue |
|---|---|
| 200-d MusiCNN embedding (per track) | SensMe 10-STMO channels (Path C opaque features) |
| 6 CLAP attribute scores (per track) | Mood/genre tags, contextual scores |
| MusiCNN top-N moods | mbxmoods.json mood scores |
| Voyager k-NN over embeddings | (no direct equivalent — AutoQ does not retrieve, it scores) |
| Clustering playlist generation | AutoQ's pick-next + AutoDJ planning |
| Sonic Fingerprint (history-derived) | AutoQ history influence + recent-listens weighting |
| Music Map (2D projection) | **VA plot** — already in MBXH, using SensMe valence/arousal as the two axes directly (no learned reduction needed) |

### 3.3 What's complementary, what's overlapping

**Complementary (could feed each other):**

- **MusiCNN embedding → AutoQ opaque features.** Structurally identical to SensMe channels. AutoQ's Ridge calibrator already learned to treat opaque vectors as features.
- **CLAP attribute scores → AutoQ contextual influences.** Six floats per track; AutoQ already has a notion of attribute-driven influence.
- **truedat as ingest host.** It's already decoding files locally and writing per-track vectors. Adding a MusiCNN-ONNX or CLAP pass alongside SensMe is the same shape of work.

**Overlapping (would compete):**

- Voyager k-NN retrieval has no AutoQ equivalent — neither overlap nor easy fold-in. It's a different query primitive (find me 50 most-similar) than AutoQ provides (score and rank a candidate set).
- AudioMuse's clustering-driven playlist generation overlaps AutoQ's purpose but uses a fundamentally different algorithm (cluster-and-sample vs. score-and-pick-next).

### 3.4 What we'd actually gain

Worth integrating only if we want:

1. **Better than tag-quality similarity** for cases where SensMe channels are sparse or absent (catalog has SensMe coverage gaps).
2. **Lyrics semantic search** across the library (we don't have one).
3. **Cross-server sonic similarity** — e.g. if MBXHub starts speaking OpenSubsonic to clients, getting the Sonic Similarity API for free is interesting.
4. **k-NN retrieval as a primitive.** AutoQ scores; it doesn't retrieve. "Find me the 50 most-similar tracks to X" is a query shape we don't have today — even SensMe-fed, that'd be a new affordance.

What we **don't** gain:
- **2D Music Map** — we already have this. MBXH's VA plot uses SensMe valence/arousal as the two axes directly. AudioMuse's Music Map is a learned-then-reduced projection of the 200-d MusiCNN vector down to 2D; ours is two interpretable axes straight from the source. Same UI primitive, different feature pipeline.
- **AutoDJ / playlist generation** — AutoQ already does scoring and AutoDJ. AudioMuse's 5000-run cluster-and-sample algorithm is a different philosophy, not a strict upgrade.

---

## 4. Integration Options & Cost

### Option A — AudioMuse as a true sidecar (full stack)

Deploy Flask + Worker + PostgreSQL 15 + Redis next to MBXHub. Write an MBXH-side adapter that calls AudioMuse's HTTP API (`app_voyager.py` endpoints) for similar-track queries and surfaces results in our existing REST/WebSocket shape.

**Cost:**
- Two stateful services (PG15 strict, Redis) to operate
- Week-scale cold-start ingest at 70–200K
- Custom adapter (no MusicBee plugin exists)
- Duplicate file-decode work (truedat already decoding; AudioMuse decodes again)

**Gain:** Voyager k-NN, Sonic Path, Music Map, Lyrics Search — all the user-visible features as-is.

### Option B — Truedat extension (vendor the models, skip the service)

Add a MusiCNN ONNX pass (and optionally CLAP) to truedat alongside its SensMe extraction. Write the 200-d vector + top-N moods into `mbxmoods.json` (or a sibling artifact). AutoQ consumes them as additional opaque features through the same Ridge-calibrator path it already uses for SensMe channels.

**Cost:**
- Bundle ONNX Runtime in truedat
- Ship `musicnn_embedding.onnx` + optionally CLAP models (≈50–200 MB on disk, model size not separately verified in this pass)
- Define the JSON schema extension
- No similarity-search index (we'd be feeding scoring, not retrieving)

**Gain:** Additional feature signal for AutoQ. Nothing operational beyond what truedat already costs us. Library decoded once (truedat already does it).

**Loss vs. Option A:** No Sonic Path, no Lyrics Search, no k-NN retrieval — those require Voyager + the service surface. (Music Map is not a loss; we already have it as the VA plot.)

### Option B′ — Essentia VA as a SensMe fallback in truedat (narrowest) — **NOT PURSUED: license (CC BY-NC-SA 4.0)**

The Essentia model zoo includes **pretrained valence/arousal regression heads** (three families: `emomusic`, `deam`, `muse`) on top of the same MusiCNN backbone AudioMuse uses. Output shape: two continuous floats per track. **Same shape as SensMe VA**, polarity matches (positive valence = positive mood, high arousal = energetic).

The integration: truedat runs an Essentia VA ONNX pass on tracks where SensMe data is missing, writes the two floats into `mbxmoods.json` under the same VA keys with a `va_source: sensme|essentia` tag. Nothing downstream changes — AutoQ keeps consuming VA, the VA plot keeps rendering.

**Concrete model picks (all ONNX, dated 2023-05-16):**

| File | Size | Role |
|---|---|---|
| `msd-musicnn-1.onnx` | 3.02 MB | Backbone (200-d embedding, shared) |
| `emomusic-msd-musicnn-2.onnx` | 82 KB | VA head (EmoMusic, 1000 Western-pop) |
| `deam-msd-musicnn-2.onnx` | 82 KB | VA head (DEAM, ~1800 excerpts) |
| `muse-msd-musicnn-2.onnx` | 82 KB | VA head (MuSe multi-source) |

Total disk footprint: **~3.1 MB**. Ship all three heads (under 4 MB) and A/B with one backbone forward pass. Recommended primary: `emomusic-msd-musicnn-2` — best-documented, model card reports Valence Pearson r=0.736 / Arousal r=0.821.

**Three hard gotchas:**

1. **License: CC BY-NC-SA 4.0.** Non-commercial only without an MTG-UPF commercial license. If truedat ships commercially, this is disqualifying — do a HuggingFace pass for a permissive alternative first.
2. **Output scale [1, 9], not Sony's range.** Linear remap before writing to `mbxmoods.json`.
3. **MusiCNN mel front-end must be exact.** 16 kHz mono float, 96 mel bands, frame=512/hop=256, Slaney mel, unit_tri norm, 187-frame patches (~3 s). MTG/essentia#1471 documents this being hard to reproduce in librosa, never mind .NET. NAudio decodes to PCM but not mel — you'll need a vendored mel impl or native lib call.

**Cost:**
- ONNX Runtime CPU EP in truedat (no GPU, no AVX2 dependency)
- Audio decode → 16 kHz mono resampler (NAudio MediaFoundationResampler or vendored Soxr)
- Mel-spectrogram impl (matching Essentia/Slaney config exactly)
- ~3.1 MB models shipped with truedat
- Schema: `va_source` field, optional confidence blending

**Gain:** VA coverage on the tail of the catalog Sony never tagged. No new UI surface, no new query primitive, no service stack. **The bounded version of "AudioMuse-style analysis"** — borrows the model family, skips everything else.

**Quality caveat:** EmoMusic training data is 1000 short Western-pop excerpts; DEAM ~1800. Distribution shift expected on metal/ambient/classical/non-Western material. Arousal transfers; valence is the weaker axis.

**Sources:**
- [Essentia models page](https://essentia.upf.edu/models.html)
- [emomusic head directory](https://essentia.upf.edu/models/classification-heads/emomusic/)
- [musicnn backbone](https://essentia.upf.edu/models/feature-extractors/musicnn/)
- [TensorflowPredictMusiCNN ref](https://essentia.upf.edu/reference/std_TensorflowPredictMusiCNN.html) (preprocessing spec)
- [MTG/essentia#1471](https://github.com/MTG/essentia/issues/1471) (mel reproduction in librosa)
- [Replicate mtg/music-arousal-valence](https://replicate.com/mtg/music-arousal-valence) (license confirmation)

### Option C — Hybrid (truedat extracts, MBXH indexes)

truedat does the ONNX pass during its existing scan. MBXH loads the 200-d vectors into an in-process ANN index (e.g. embed Voyager via P/Invoke, or use a managed HNSW for .NET like `HNSW.Net`). AutoQ gets the embeddings as opaque features *and* we get k-NN retrieval, without standing up PG15/Redis.

**Cost:**
- truedat ONNX pass (same as Option B)
- ANN library choice + integration into MBXH
- RAM cost for in-process index at 200K tracks (rough math: 200K × 200 × 4 bytes ≈ 160 MB raw vectors, plus HNSW graph overhead — likely 300–600 MB resident)

**Gain:** AutoQ features + k-NN retrieval + everything stays in our existing service shape. Music Map / Sonic Path / Lyrics Search not free but constructible on top.

**This is the most architecturally aligned option** for our ecosystem — it keeps MBXH a tag-and-feature-driven engine, keeps truedat the ingest layer, and avoids running PG15 + Redis.

---

## 5. Caveats

- All AudioMuse implementation details are self-reported (README / ARCHITECTURE / ALGORITHM / FAQ / code). Primary sources, but no independent benchmark or third-party audit was found.
- Specific server version numbers (Jellyfin 10.11.8, Navidrome 0.61.0, LMS v3.69.0, etc.) **failed verification** in the adversarial pass — treat the supported-server list as qualitative only.
- MBXHub/AutoQ/truedat comparison points are synthesized from project context (CLAUDE.md + MEMORY), not from re-reading the actual AutoQ implementation in this pass.
- AudioMuse is moving fast (Navidrome plugin moved adapters within a year). Anything cited here may shift.

---

## 6. Open Questions Worth Answering Before Investing

1. **On-disk + RAM footprint at 200K tracks.** Exact bytes for a 200-d MusiCNN row + top-N moods + CLAP scores + Voyager graph. Determines whether Option C (in-process ANN) is even viable.
2. **Pre-extracted features ingest path?** Does AudioMuse expose a "here are pre-computed embeddings, persist them" API, or is the only path "worker fetches the audio file and runs the models itself"? If the former, truedat could feed it directly.
3. **Stable embedding-vector HTTP API?** Are `app_voyager.py` endpoints documented and stable, or internal-only?
4. **Empirical feature value.** Has anyone compared SensMe 10-STMO vs. MusiCNN 200-d on retrieval/recommendation quality for similar libraries? Without this, "AudioMuse embeddings as additional opaque features for AutoQ" remains speculative.

---

## 7. Primary Sources

- [AudioMuse-AI repo](https://github.com/NeptuneHub/AudioMuse-AI) — README, code
- [ARCHITECTURE.md](https://github.com/NeptuneHub/AudioMuse-AI/blob/main/docs/ARCHITECTURE.md) (also at [neptunehub.github.io/AudioMuse-AI/ARCHITECTURE/](https://neptunehub.github.io/AudioMuse-AI/ARCHITECTURE/))
- [ALGORITHM.md](https://github.com/NeptuneHub/AudioMuse-AI/blob/main/docs/ALGORITHM.md) (also at [neptunehub.github.io/AudioMuse-AI/ALGORITHM/](https://neptunehub.github.io/AudioMuse-AI/ALGORITHM/))
- [FAQ.md](https://github.com/NeptuneHub/AudioMuse-AI/blob/main/docs/FAQ.md)
- [PARAMETERS.md](https://github.com/NeptuneHub/AudioMuse-AI/blob/main/docs/PARAMETERS.md)
- [config.py](https://github.com/NeptuneHub/AudioMuse-AI/blob/main/config.py)
- [audiomuse-ai-NV-plugin](https://github.com/NeptuneHub/AudioMuse-AI-NV-plugin) (Navidrome plugin)
- [audiomuse-ai-plugin](https://github.com/NeptuneHub/audiomuse-ai-plugin) (Jellyfin plugin)
- [Essentia model zoo](https://essentia.upf.edu/models.html) (MusiCNN lineage)
- [Spotify Voyager](https://github.com/spotify/voyager) (ANN library)
- [Dev.to: AudioMuse-AI Sonic Analysis for Jellyfin and Navidrome](https://dev.to/neptunehub/audiomuse-ai-sonic-analysis-for-jellyfin-and-navidrome-5hd) (author blog post)