Memory leak in C accelerator: _PyUnicode_AsUTF8 leaks bytes

[nihar102]

Context

We recently investigated a memory leak issue we observed in psyduck where the psyduck-server pods grew ~55–80 MB/h and were OOM-killed every ~1–2 days. Growth was strictly proportional to the number of queries, independent of endpoint, connection pool size, or open file descriptors. tracemalloc attributed ~88 % of retained allocations to the frame calling _singlestoredb_accel.read_rowdata_packet (singlestoredb/mysql/connection.py).

09/26 is when we deployed the fix, so the graph is flat.

Summary

Looks like the C accelerator (accel.c) leaks a small amount of memory on every query. The helper _PyUnicode_AsUTF8() Seems to never release some bytes object it creates. For a long-running service issuing many queries this grows without bound. Reproduced on 1.12.4 and on the current 1.17.3

Setting pure_python=True (SINGLESTOREDB_PURE_PYTHON=1) removes the leak completely.

Environment

• singlestoredb 1.12.4 or 1.17.3 (without SINGLESTOREDB_PURE_PYTHON=1)
• Server: SingleStoreDB 8.x / 10.x.
• Used through SQLAlchemy (sqlalchemy-singlestoredb) in psyduck production, but the repro below uses the DB-API directly.

Reproduction

import gc
import sys
import singlestoredb as s2

# any table works; column count is what matters
SQL_1 = "SELECT 1"
SQL_200 = "SELECT " + ", ".join(f"{i} AS c{i}" for i in range(200))


def leaked_blocks_per_query(sql, iters=3000, pure_python=False):
    conn = s2.connect(
        host="127.0.0.1", port=3306, user="root", password="", database="information_schema",
        pure_python=pure_python,
    )
    cur = conn.cursor()
    for _ in range(200):          # warm-up: caches, interned strings, etc.
        cur.execute(sql); cur.fetchall()
    gc.collect()
    before = sys.getallocatedblocks()
    for _ in range(iters):
        cur.execute(sql); cur.fetchall()
    gc.collect()
    after = sys.getallocatedblocks()
    cur.close(); conn.close()
    return (after - before) / iters


for name, sql in [("SELECT 1 (1 col)", SQL_1), ("200 columns", SQL_200)]:
    print(f"{name:20s} accel: {leaked_blocks_per_query(sql):7.2f} blocks/query   "
          f"pure_python: {leaked_blocks_per_query(sql, pure_python=True):5.2f} blocks/query")

Results (identical on 1.12.4 and 1.17.3):

singlestoredb 1.12.4
SELECT 1 (1 col) accel: 2.03 blocks/query pure_python: 0.00 blocks/query
200 columns accel: 201.00 blocks/query pure_python: 0.00 blocks/query

singlestoredb 1.17.3
SELECT 1 (1 col) accel: 2.03 blocks/query pure_python: 0.00 blocks/query
200 columns accel: 201.00 blocks/query pure_python: 0.00 blocks/query

Leaked blocks per query = columns + 1. sys.getallocatedblocks() sees the Python bytes objects; the calloc'd C copies leak on top of that (Probably not visible to Python GC). In our service that adds up to ~3.3 KB of memory per query for the 42-column query (measured over 100 000 requests: linear growth with the accelerator, flat plateau with pure_python).

Identified Root cause (accel.c on main)

_PyUnicode_AsUTF8 creates a new bytes reference and never drops it; the returned buffer is a heap copy the caller must free, but no caller does:

char *_PyUnicode_AsUTF8(PyObject *unicode) {
    PyObject *bytes = PyUnicode_AsEncodedString(unicode, "utf-8", "strict");  // new reference
    if (!bytes) return NULL;

    char *str = NULL;
    Py_ssize_t str_l = 0;
    if (PyBytes_AsStringAndSize(bytes, &str, &str_l) < 0) {
        return NULL;                                                          // bytes leaked
    }

    char *out = calloc(str_l + 1, 1);
    memcpy(out, str, str_l);
    return out;                                                               // bytes leaked; out owned by caller
}

Identified 3 callers:

• self->encoding_errors = _PyUnicode_AsUTF8(py_encoding_errors); :once per query. State_clear_fields does DESTROY(self->encoding_errors) so the C copy is freed here, but the bytes object is still leaked.
• self->encodings[i] = ... _PyUnicode_AsUTF8(py_encoding); :once per column. State_clear_fields frees the encodings array but not its entries, so both the bytes object and the C copy leak per column.
• self->structsequence_desc.fields[i].name = _PyUnicode_AsUTF8(self->py_names[i]); :once per column. Same pattern: fields array freed , entries not.