You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 900e2ea
Browse filesBrowse the repository at this point in the historyBrowse files
libsql-server: make namespace creation all-or-nothing
`NamespaceStore::create` persisted the namespace config before setting the
namespace up, and nothing undid it: a dump that failed to import left a
"ghost" namespace (config without data) whose name could not be reused and
which the next user request lazily turned into an empty database. A request
cancelled mid-import ran no cleanup at all, since the error branch of the
future is never reached when it is dropped, and the schema configurator had
no cleanup even for errors.
The metastore write is now the commit point and everything before it is
reversible. Contract for callers: 2xx means complete; anything else,
including a lost connection, means retry; `already exists` on the retry
means the earlier attempt did complete.
Reservation (store). `create` and `fork` write-lock the name's cache entry
for the duration and keep the config in memory only. `publish` installs the
finished namespace; `abort` (awaited on error paths) and `Drop` (the
creating future was dropped, i.e. the request was cancelled) undo the
attempt — config, then directory, then unpin — and only then release the
lock, so waiters observe a namespace that does not exist, never a half-built
one, and a retry issued the moment the error arrives neither races the
cleanup nor finds leftovers. Replaces fork's ad-hoc guard, which blocked in
place (panics on a current-thread runtime) and published before flushing.
Marker and tombstones (namespace/incomplete.rs). A new namespace directory
carries `.incomplete` (fsynced, so it is durable before anything else in the
directory can be) until the config is persisted; the store removes it after
publishing. A creation that finds a marked directory for a name the
metastore does not know discards it, so a crash mid-import never blocks a
retry; a startup sweep does the same and deletes leftover tombstones; the
metastore's filesystem recovery skips marked directories. Directories are
never deleted in place: `discard_dir` renames them into `<base>/.discarded/`
(instant; frees the path; a still-running import executor writes into the
tombstone) and deletes in the background. `FreshDir` (primary and schema
configurators) creates the marked directory synchronously, guard first, and
discards it on drop. Fork marks its temporary directory, and only removes
its destination on failure if it carries the marker.
Metastore. `discard` (via the change queue, so it is ordered after a config
store still queued when the request was cancelled) forgets a namespace
unconditionally — `remove`'s linked-DB guards would otherwise leave a ghost
when a linked-DB creation's flush fails. `existing_handle` looks a namespace
up without registering it; `with()` and the fork source use it, and the lazy
init re-checks existence, so a lookup racing the end of a failed creation
cannot resurrect the name. `process`'s flush branch no longer holds
`configs` across `try_process` (lock-order inversion against `remove`).
Cache eviction. The reservation is the entry's write lock, which moka knows
nothing about. If the cache evicted that entry during a long operation
(capacity pressure): a concurrent lookup lazily opened the directory being
written; the eviction listener awaited the lock and moka awaits async
listeners inline from whichever cache op runs maintenance, so that op hung
for the whole import and, once the write-op channel filled, so did every
namespace load; an operation touching the cache while holding its own entry
(fork) could deadlock. Now the locked entry is pinned in `in_flight`;
`load_namespace` and `destroy` consult it first and wait on it; the listener
spawns its shutdown only for pinned entries; `install` publishes into a
fresh entry when the locked one was evicted, or hands the namespace to the
old entry when the name was destroyed meanwhile (destroy is waiting on it).
`shutdown` skips pinned entries; `create`/`fork` re-check `has_shutdown`
before the commit.
Empty cache entries. A rejected `acquire` (`already exists` for a namespace
that is not loaded, e.g. a retry after a restart) left behind the empty
entry its `get_with` inserted, and `try_get_with` returned that entry forever
instead of loading the namespace. `acquire` removes the entry it created;
`load_namespace` treats an empty, unlocked entry as a miss; `checkpoint` no
longer inserts one for an unloaded namespace (the pre-existing source).
Single-namespace mode: `create(default, Latest)` stays a config upsert;
creating the default namespace from a dump while it exists is rejected
instead of silently skipping the import.
Needs async-lock 3 for `RwLock::write_arc` (already in the lockfile; the
bump removes the duplicate 2.x). Dump tests set `tcp_capacity(4096)`: the
fsync makes setup's following blocking fs ops slower in real time, turmoil
advances simulated time while a host waits on them, and the dump store
writes 1-byte chunks past turmoil's default 64-segment socket buffer.
Tests. Store unit tests build a real `NamespaceStore` with a channel-fed
streaming import: lookup waits across an eviction and sees the data, or
"doesn't exist" when the import fails; destroy during creation; retry the
instant the error returns; abort mid-import; discard vs remove's guards.
Unit tests for the marker protocol, tombstones, sweep and metastore
recovery. Integration (turmoil): failed creation leaves no trace (both
importers, schema configurator), cancelled creation (cancel fires once the
marker exists), concurrent requests wait, crash remnants discarded, unmarked
directories untouched, marker on a published namespace ignored (create
before any lookup after a restart), single-namespace dump create rejected.
Rebased onto the false-404 fix (#56), whose semantics `with` and
`load_namespace` keep: `with` checks for shutdown on every attempt,
looks the namespace up again (up to three times) when the entry it got
was emptied, and answers `NamespaceDoesntExist` only when the metastore
has no record (otherwise `NamespaceUnavailable` or
`NamespaceStoreShutdown`, both 503); `load_namespace` refuses to open a
namespace once the store has shut down. `with` keeps
`existing_handle`, so the retry cannot resurrect a destroyed record.
An empty entry left in the cache is dropped with `evict_if_still_empty`
(from #59): under moka's key lock, only if it is still the cached entry,
still empty and unlocked. That replaces the `get`-then-`invalidate`
pairs in `load_namespace` and `Reservation::acquire`, which could evict
a fresh entry loaded in between. Dropped entries are counted in
`libsql_server_namespace_left_empty_entry_dropped`.
0 commit comments