docs: introduce clap.cpp audio embedding inference

src/.vuepress/theme.ts

@@ -153,7 +153,7 @@ export default hopeTheme({
     },
 
     icon: {
-      assets: "https://at.alicdn.com/t/c/font_3748819_8wjsuy9z69.css",
+      assets: "https://at.alicdn.com/t/c/font_3748819_waf7um8w3.css",
     },
 
     // Install @vuepress/plugin-pwa and uncomment these if you want a PWA

src/posts/clap-cpp.md

@@ -0,0 +1,143 @@
+---
+date: 2026-11-05
+category:
+  - technical
+tag:
+  - AI/ML
+---
+# clap.cpp: Audio Embedding Inference in GGML
+
+Most inference engines support embedding models for text, images, and video, but almost none support audio embedding models. To generate audio embeddings, we typically need to use Python with PyTorch, or convert the model to ONNX and run it with ONNX runtime. There has been no out-of-the-box solution.
+
+To fill this gap, we developed [clap.cpp](https://github.com/gorse-io/clap.cpp), an inference engine for audio embedding models implemented in C++20 and [GGML](https://github.com/ggml-org/ggml). It supports models such as CLAP, CLaMP3, MERT, MuQ, and MuQ-MuLan; provides a command-line interface, an HTTP service, and Python, Rust, and Node.js bindings; and supports hardware acceleration through Metal, Vulkan, CUDA, and ROCm.
+
+## Supported Models
+
+The project currently supports the following models:
+
+| Model | Text | Audio | Dimensions | Sample Rate | License |
+| --- | --- | --- | --- | --- | --- |
+| [CLAP](https://huggingface.co/laion/larger_clap_music) | ✅ | ✅ | 512 | 48 kHz | [Apache-2.0](https://huggingface.co/laion/larger_clap_music) |
+| [CLaMP3](https://github.com/sanderwood/clamp3) | ✅ | ✅ | 768 | 24 kHz | [MIT](https://huggingface.co/sander-wood/clamp3) |
+| [MERT v1 95M](https://huggingface.co/m-a-p/MERT-v1-95M) | ❌ | ✅ | 768 | 24 kHz | [CC-BY-NC-4.0](https://huggingface.co/m-a-p/MERT-v1-95M) |
+| [MERT v1 330M](https://huggingface.co/m-a-p/MERT-v1-330M) | ❌ | ✅ | 1,024 | 24 kHz | [CC-BY-NC-4.0](https://huggingface.co/m-a-p/MERT-v1-330M) |
+| [MERT v2 FullSong](https://huggingface.co/m-a-p/MERT-v2-FullSong) / [30s](https://huggingface.co/m-a-p/MERT-v2-30s) | ❌ | ✅ | 1,024 | 24 kHz | [CC-BY-NC-4.0](https://huggingface.co/m-a-p/MERT-v2-FullSong) |
+| [MuQ](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter) | ❌ | ✅ | 1,024 | 24 kHz | [CC-BY-NC-4.0](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter) |
+| [MuQ-MuLan](https://huggingface.co/OpenMuQ/MuQ-MuLan-large) | ✅ | ✅ | 512 | 24 kHz | [CC-BY-NC-4.0](https://huggingface.co/OpenMuQ/MuQ-MuLan-large) |
+
+CLAP and MuQ-MuLan map text and audio into the same embedding space, enabling cross-modal text-to-audio and audio-to-text retrieval. The remaining models support audio retrieval only. Each model also natively supports a different audio sample rate, and clap.cpp automatically resamples the input as needed.
+
+clap.cpp uses GGUF as its model file format. GGUF versions of these models are available from the [Audio-Embedding-GGUF](https://huggingface.co/collections/gorse-io/audio-embedding-gguf) collection. Each model is provided in F16, Q8, and Q4_K_M quantized variants. When using them, you must comply with the license for the corresponding model weights.
+
+## Embedding Audio and Text in Code
+
+Audio embedding models typically have relatively few parameters, so local inference is sufficient for shorter audio clips. clap.cpp provides Python, Rust, and Node.js bindings.
+
+::: code-tabs
+
+@tab:active Python
+
+```bash
+pip install clap-cpp
+```
+
+@tab Rust
+
+```bash
+cargo add clap-cpp
+```
+
+@tab Node.js
+
+```bash
+npm install clap-cpp
+```
+
+:::
+
+The following examples load the model once, encode text and audio separately, and generate embeddings that can be compared using cosine similarity.
+
+::: code-tabs
+
+@tab:active Python
+
+```python
+from clap_cpp import CLAP
+
+with CLAP("larger_clap_general-q4_k_m.gguf") as model:
+    text_embedding = model.encode_text("Sound of a dog")
+    audio_embedding = model.encode_audio("dog.wav")
+```
+
+@tab Rust
+
+```rust
+use clap_cpp::CLAP;
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let mut model = CLAP::new("larger_clap_general-q4_k_m.gguf", 4)?;
+    let text_embedding = model.encode_text("Sound of a dog")?;
+    let audio_embedding = model.encode_audio("dog.wav", 0)?;
+    Ok(())
+}
+```
+
+@tab Node.js
+
+```javascript
+const { CLAP } = require("clap-cpp");
+
+const model = new CLAP("larger_clap_general-q4_k_m.gguf");
+try {
+  const textEmbedding = model.encodeText("Sound of a dog");
+  const audioEmbedding = model.encodeAudio("dog.wav");
+} finally {
+  model.close();
+}
+```
+
+:::
+
+## Embedding Audio and Text over HTTP
+
+clap.cpp also provides an HTTP service:
+
+```bash
+clap serve --model larger_clap_general-q4_k_m.gguf --host 127.0.0.1 --port 8080
+```
+
+The text embedding endpoint uses an OpenAI-compatible request format:
+
+```bash
+curl http://127.0.0.1:8080/v1/embeddings \
+  -H 'Content-Type: application/json' \
+  -d '{"model":"larger_clap_general-q4_k_m.gguf","input":["Sound of a dog","Sound of a vacuum cleaner"],"encoding_format":"float"}'
+```
+
+Audio requests use an `input_audio` content block. The `data` field contains the standard Base64 encoding of the complete audio file:
+
+```bash
+audio_data="$(base64 < music.flac | tr -d '\r\n')"
+
+curl http://127.0.0.1:8080/v1/embeddings \
+  -H 'Content-Type: application/json' \
+  --data-binary @- <<JSON
+{
+  "model": "larger_clap_general-q4_k_m.gguf",
+  "input": [
+    {
+      "content": [
+        {
+          "type": "input_audio",
+          "input_audio": {"data": "${audio_data}"}
+        }
+      ]
+    }
+  ]
+}
+JSON
+```
+
+## Conclusion
+
+The findings of [MAEB: Massive Audio Embedding Benchmark](https://huggingface.co/blog/AdnanElAssadi/maeb) show that audio embedding models face a trade-off between performance on language-related tasks and acoustic tasks. This means that choosing an audio embedding model requires more careful evaluation than choosing a text embedding model. In the future, we will publish comparisons of how these popular audio embedding models perform in recommender systems.

src/zh/posts/clap-cpp.md

@@ -0,0 +1,148 @@
+---
+icon: clap
+date: 2026-11-05
+category:
+  - 技术分享
+tag:
+  - 音频嵌入
+---
+# clap.cpp: 一个音频向量模型推理引擎
+
+主流的推理引擎基本都支持文本、图像和视频的嵌入向量生成，然而它们几乎都不支持音频嵌入向量模型推理。如果我们需要为音频生成嵌入向量的话，就需要使用 Python 并安装 PyTorch 来进行模型推理，或者将模型转换为 ONNX 后使用 ONNX 运行时推理，缺少一个开箱即用方案。
+
+于是我们开发了[clap.cpp](https://github.com/gorse-io/clap.cpp) ——一个使用 C++20 和 [GGML](https://github.com/ggml-org/ggml) 实现的音频向量模型推理引擎。它支持 CLAP、CLaMP3、MERT、MuQ 和 MuQ-MuLan 等模型，提供命令行、HTTP 服务以及 Python、Rust、Node.js 接口，并支持 Metal、Vulkan、CUDA 和 ROCM 硬件加速。
+
+## 支持模型列表
+
+项目目前支持的模型如下：
+
+| 模型 | 文本 | 音频 | 维度 | 采样率 | 模型权重许可证 |
+| --- | --- | --- | --- | --- | --- |
+| [CLAP](https://huggingface.co/laion/larger_clap_music) | ✅ | ✅ | 512 | 48 kHz | [Apache-2.0](https://huggingface.co/laion/larger_clap_music) |
+| [CLaMP3](https://github.com/sanderwood/clamp3) | ✅ | ✅ | 768 | 24 kHz | [MIT](https://huggingface.co/sander-wood/clamp3) |
+| [MERT v1 95M](https://huggingface.co/m-a-p/MERT-v1-95M) | ❌ | ✅ | 768 | 24 kHz | [CC-BY-NC-4.0](https://huggingface.co/m-a-p/MERT-v1-95M) |
+| [MERT v1 330M](https://huggingface.co/m-a-p/MERT-v1-330M) | ❌ | ✅ | 1,024 | 24 kHz | [CC-BY-NC-4.0](https://huggingface.co/m-a-p/MERT-v1-330M) |
+| [MERT v2 FullSong](https://huggingface.co/m-a-p/MERT-v2-FullSong) / [30s](https://huggingface.co/m-a-p/MERT-v2-30s) | ❌ | ✅ | 1,024 | 24 kHz | [CC-BY-NC-4.0](https://huggingface.co/m-a-p/MERT-v2-FullSong) |
+| [MuQ](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter) | ❌ | ✅ | 1,024 | 24 kHz | [CC-BY-NC-4.0](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter) |
+| [MuQ-MuLan](https://huggingface.co/OpenMuQ/MuQ-MuLan-large) | ✅ | ✅ | 512 | 24 kHz | [CC-BY-NC-4.0](https://huggingface.co/OpenMuQ/MuQ-MuLan-large) |
+
+CLAP 和 MuQ-MuLan 将文本和音频映射到同一个向量空间，可以用于跨文本和音频检索，而剩余模型只能用于音频检索。每个模型原生支持的音频采样率也不同，clap.cpp会自动重采样。
+
+clap.cpp 的模型文件格式为 GGUF，这些模型的 GGUF 版本可以从 [Audio-Embedding-GGUF](https://huggingface.co/collections/gorse-io/audio-embedding-gguf)获取，每个模型提供 F16、Q8、Q4_K_M 三种量化版本，使用过程中需要遵循模型权重许可证。
+
+## 在代码中向量化音频和文本
+
+音频向量模型的参数量通常较少，对于较短的音频，本地推理就已足够。clap.cpp 提供 Python、Rust 和 Node.js 绑定。
+
+
+::: code-tabs
+
+@tab:active Python
+
+
+```bash
+pip install clap-cpp
+```
+
+@tab Rust
+
+
+```bash
+cargo add clap-cpp
+```
+
+@tab Node.js
+
+```bash
+npm install clap-cpp
+```
+
+:::
+
+
+下面的示例加载一次模型，分别编码文本和音频，生成可用于余弦相似度比较的向量。
+
+::: code-tabs
+
+@tab:active Python
+
+```python
+from clap_cpp import CLAP
+
+with CLAP("larger_clap_general-q4_k_m.gguf") as model:
+    text_embedding = model.encode_text("Sound of a dog")
+    audio_embedding = model.encode_audio("dog.wav")
+```
+
+@tab Rust
+
+```rust
+use clap_cpp::CLAP;
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let mut model = CLAP::new("larger_clap_general-q4_k_m.gguf", 4)?;
+    let text_embedding = model.encode_text("Sound of a dog")?;
+    let audio_embedding = model.encode_audio("dog.wav", 0)?;
+    Ok(())
+}
+```
+
+@tab Node.js
+
+```javascript
+const { CLAP } = require("clap-cpp");
+
+const model = new CLAP("larger_clap_general-q4_k_m.gguf");
+try {
+  const textEmbedding = model.encodeText("Sound of a dog");
+  const audioEmbedding = model.encodeAudio("dog.wav");
+} finally {
+  model.close();
+}
+```
+
+:::
+
+## 通过 HTTP 向量化音频和文本
+
+clap.cpp 也提供了 HTTP 服务：
+
+```bash
+clap serve --model larger_clap_general-q4_k_m.gguf --host 127.0.0.1 --port 8080
+```
+
+文本嵌入接口采用 OpenAI 兼容的请求形式：
+
+```bash
+curl http://127.0.0.1:8080/v1/embeddings \
+  -H 'Content-Type: application/json' \
+  -d '{"model":"larger_clap_general-q4_k_m.gguf","input":["Sound of a dog","Sound of vaccum cleaner"],"encoding_format":"float"}'
+```
+
+音频请求使用 `input_audio` 内容块，数据是完整音频文件的标准 Base64 编码：
+
+```bash
+audio_data="$(base64 < music.flac | tr -d '\r\n')"
+
+curl http://127.0.0.1:8080/v1/embeddings \
+  -H 'Content-Type: application/json' \
+  --data-binary @- <<JSON
+{
+  "model": "larger_clap_general-q4_k_m.gguf",
+  "input": [
+    {
+      "content": [
+        {
+          "type": "input_audio",
+          "input_audio": {"data": "${audio_data}"}
+        }
+      ]
+    }
+  ]
+}
+JSON
+```
+
+## 结语
+
+[MAEB: Massive Audio Embedding Benchmark](https://huggingface.co/blog/AdnanElAssadi/maeb) 的研究结果表明，音频向量模型在语言任务上的表现和声学任务上的表现不可兼得，这意味着音频向量模型的选择相比文本向量模型需要更谨慎的评估。在未来，我们还会更新这些主流的音频向量模型在推荐系统中的表现对比。