Add embedded package for in-process queues

[Suhaibinator]

Why

Since #7 moved the server under internal/, other Go modules can't run SMQ in their own process. They used to call server.NewServer(port, dbPath), go s.Start() and s.Driver.CreateTopic(...). One consumer needs a durable local outbox: publish captured events, read them in batches from an offset, forward them upstream over its own gRPC link, delete through the acknowledged offset, and report topic depth in heartbeats. It doesn't need SMQ's listener.

Base branch

This PR is stacked on #8 (codex/project-structure-ci), which is open and not merged. #8 restructures the store (internal/server/database → internal/store/sqlite, plus internal/queue), and this change builds on that layout. Basing it on main would conflict with #8 everywhere. Once #8 merges, retarget this PR to main. Note: ci.yml runs only for pull requests into main, so CI will run after the retarget. Until then, the checks below were run locally.

What's added

A new public package, github.com/Suhaibinator/SuhaibMessageQueue/embedded:

func Open(path string, opts Options) (*Queue, error)
func (q *Queue) CreateTopic(ctx context.Context, topic string) error // idempotent
func (q *Queue) Topics() []string
func (q *Queue) Publish(ctx context.Context, topic string, data []byte) (int64, error)
func (q *Queue) PublishBatch(ctx context.Context, topic string, batch [][]byte) ([]int64, error) // one transaction
func (q *Queue) Read(ctx context.Context, topic string, fromOffset int64, opts ReadOptions) ([]Message, error)
func (q *Queue) EarliestOffset(ctx context.Context, topic string) (int64, error)
func (q *Queue) LatestOffset(ctx context.Context, topic string) (int64, error)
func (q *Queue) DeleteUntilOffset(ctx context.Context, topic string, offset int64) error
func (q *Queue) Stats(ctx context.Context, topic string) (TopicStats, error)
func (q *Queue) RegisterGRPC(registrar grpc.ServiceRegistrar, opts GRPCOptions)
func (q *Queue) Close() error

type Options struct { Synchronous Synchronous; BusyTimeout time.Duration } // SyncNormal (default) | SyncFull
type ReadOptions struct { Limit int; MaxBytes int64 }                      // Limit 0 => DefaultReadLimit (100)
type Message struct { Offset int64; Data []byte }
type TopicStats struct { Count, Bytes, EarliestOffset, LatestOffset int64; OldestTime time.Time }
type GRPCOptions struct { AllowRemoteWrites bool }
var ErrTopicNotFound, ErrInvalidTopic, ErrTopicEmpty, ErrClosed // same values as the server's errors

Semantics:

• Offsets start at 1, increase monotonically, and are never reused, even after a delete or a restart (AUTOINCREMENT).
• Read is inclusive of fromOffset.
• MaxBytes caps a batch's total payload size, but the first message is always returned, so one oversized message can't stall a reader. The byte cutoff is computed from LENGTH(data) before any payloads are loaded.
• RegisterGRPC serves the existing wire API from the same store on a grpc.Server the caller owns. This covers the old embedded-server use case. Remote writes are opt-in.

embedded reuses the internal SQLite store; it adds no new SQL schema. There are no protobuf or database-format changes.

Store changes in internal/store/sqlite

• Bug fix: concurrent producers could receive another producer's offset. The per-topic writer goroutine returned results on one shared channel, so a caller could read another caller's result. On the base branch, 258 of 3,200 concurrent AddMessage calls got the wrong offset. This also affected gRPC Produce. Inserts now run directly under the existing dbMux write lock, which already serialized them. A regression test is added.
• Bug fix: Close racing a publish could panic (send on a closed channel). Operations after Close now return queue.ErrStoreClosed, which gRPC maps to UNAVAILABLE. Close and DeleteTopic now take locks in the same order as CreateTopic.
• Connection settings now apply to every pooled connection. WAL, synchronous, auto_vacuum and busy_timeout are set through the DSN, so they apply to every connection in the database/sql pool. Before, they ran once with db.Exec and applied to only one connection. The mattn build already defaults WAL connections to synchronous=NORMAL, so the server's durability doesn't change. Open(path, Options) lets embedders choose FULL.
• Private :memory: databases are pinned to one connection, since every pooled connection otherwise opens a different, empty database.
• New: AddMessages (transactional batch), ReadMessages (count and byte bounds), TopicStats, Topics.

Docs

• README: new section "Embedding SMQ in a Go program", plus an updated layout entry.
• MIGRATION.md: before and after code for code that embedded the old server package. It also adds UNAVAILABLE to the status-code table.
• embedded/example_test.go: a runnable outbox example that is checked by go test.
• Makefile: fmt now includes embedded/.

Testing

Run locally with Go 1.27.2 (go.mod toolchain is go1.26.5), mirroring the Go job in ci.yml:

• gofmt -l $(git ls-files '*.go'): clean
• go vet ./...: clean
• go test -race -shuffle=on -coverprofile=coverage.out ./...: all packages pass. embedded statement coverage is 93.8%.
• go test -race -count=15 -shuffle=on ./embedded/ ./internal/store/sqlite/: pass
• go build -trimpath ./cmd/smq: ok. go mod tidy leaves no diff.

embedded tests cover publish/read-back, batch and byte limits, delete-until (including offsets not being reused), reopening the database, concurrent publishers whose returned offsets are checked against stored payloads, stats, close-while-publishing, in-memory databases, and gRPC registration in both read-only and write modes. Protobuf and Python are unchanged, so buf and the Python checks were not run.

🤖 Generated with Claude Code