Skip to content

Add a ClickHouse adapter and Model::insert_all() - #3

Open
twoixter wants to merge 2 commits into
masterfrom
feature/clickhouse-support
Open

twoixter wants to merge 2 commits into
masterfrom
feature/clickhouse-support

Conversation

@twoixter

Copy link
Copy Markdown
Member

Summary

Adds a ClickHouse adapter for analytics workloads (ClickHouse 25.9 or later) and a bulk insert API, Model::insert_all(), that works on every adapter.

ClickHouse adapter

The adapter talks to the HTTP interface through curl, not PDO. It keeps one keep-alive connection per adapter, asks for compressed responses and sends settings as URL parameters. The minimum is 25.9 because that is the first release with lightweight UPDATE on by default.

  • Results come back as TabSeparatedWithNamesAndTypes.
    • In PHP it parsed as fast as the JSON formats, and it is safe for any byte.
    • An error in the middle of a result aborts the transfer instead of looking like data.
    • The statement parses one row per fetch() and casts values from their ClickHouse type. Integers beyond PHP_INT_MAX, Int128/256 and Decimal stay strings; Array/Map/Tuple become PHP arrays.
  • Binding is client-side, with literals. A ? inside quotes, quoted identifiers or comments is left alone. Numeric strings stay quoted, because an unquoted integer beyond Int64 is read as a lossy Float64.
  • Errors carry the ClickHouse code. That includes errors reported after the result started streaming, in both the 25.9 format and the tagged 26.x format.
  • Metadata comes from system.columns. The sorting key is not taken as the primary key, because it is not unique: the key is a column named id or the model's $primary_key. MATERIALIZED/ALIAS columns are left out.
  • Keys: a UInt128 key (also Int128, UInt256, Int256, UUID or String) gets a UUIDv7 generated in PHP on insert. The new ActiveRecord\Uuid class generates them and converts between the decimal the model holds and UUID text. UInt128 sorts numerically, so ORDER BY id follows creation order; the UUID type would not.
  • Dates are written as quoted Unix timestamps and read back in PHP's time zone. session_timezone is not used, because before 26.2 INSERT and lightweight UPDATE did not agree on how it applies.
  • Inserts are synchronous by default. That is the fastest option per save() (about 2 ms) and errors are reported.
    • Connections can opt into async_insert=1. It keeps wait_for_async_insert=1, as ClickHouse recommends, so failed inserts are never lost silently.
    • The README compares the three modes, measured on 25.9 and 26.8, and says which one fits which insert pattern.
  • Updates and deletes: update_all() and delete_all() return null, because ClickHouse does not report affected rows. DELETE without conditions gets the WHERE ClickHouse requires. Transactions throw.
  • Connection URL: any other query string parameter becomes a ClickHouse setting.

Model::insert_all()

Model::insert_all($rows, ['batch_size' => N]) inserts many records with multi-row INSERT statements on MySQL, PostgreSQL, SQLite and ClickHouse.

  • It builds no models, so validations and callbacks do not run.
  • Aliases and inflected names are resolved, dates and JSON are converted, timestamps are filled in, and missing keys are generated as save() would generate them.
  • Rows that set the same columns share statements. Batches shrink to fit the bound parameters the database accepts.
  • ClickHouse writes the rows as literals and sends batches of 1000 rows or more synchronously, even on an async connection.

Core fixes

  • parse_connection_url() parses the query string properly. explode('/&/') never split more than one parameter.
  • Model::insert() and Table cope with tables without a primary key, instead of reading a missing pk[0].

Tests

  • ClickhouseAdapterTest and ClickhouseModelTest, with their own schema (test/sql/clickhouse.sql) and models (test/models/Clickhouse). The shared fixtures rely on auto-increment keys and transactions, which ClickHouse does not have.
  • InsertAllTest runs against the default adapter. UuidTest needs no database.
  • CI runs ClickHouse 26.8 in every job and 25.9 in one extra row. compose.yaml has both versions.

Verified locally: 994 tests green on MySQL 5.7 and 8.4, PostgreSQL 18 and SQLite, with each of mysql, pgsql and sqlite as the default adapter, against ClickHouse 25.9 and 26.8. phpcs is clean.

ClickHouse 25.9 or later is now supported for analytics workloads. The
adapter talks to the HTTP interface through curl instead of PDO: one
keep-alive connection per adapter, compressed responses, settings sent
as URL parameters.

- Results come back as TabSeparatedWithNamesAndTypes, which parsed as
  fast as the JSON formats in PHP, is safe for any byte and makes an
  error in the middle of a result abort the transfer instead of looking
  like data. The statement parses one row per fetch() and casts values
  from their ClickHouse type; integers beyond PHP_INT_MAX, Int128/256
  and Decimal stay strings, Array/Map/Tuple become PHP arrays.
- Values are bound client-side as literals, skipping ? inside quotes,
  quoted identifiers and comments. Numeric strings stay quoted: an
  unquoted integer beyond Int64 is read as a lossy Float64.
- Errors carry the ClickHouse code, including those reported after the
  result started streaming (both the 25.9 and the tagged 26.x formats).
- Column metadata comes from system.columns. The sorting key is not
  taken as the primary key, as it is not unique: the key is a column
  named id or the model's $primary_key. MATERIALIZED and ALIAS columns
  are left out.
- A UInt128 (or Int128, UInt256, Int256, UUID, String) key gets a UUIDv7
  generated in PHP on insert. ActiveRecord\Uuid generates them and
  converts between the decimal the model holds and UUID text. UInt128
  sorts numerically, so ORDER BY id follows creation order.
- DateTime values are written as quoted Unix timestamps and read back
  in PHP's time zone. session_timezone is not used: before 26.2 INSERT
  and lightweight UPDATE did not agree on how it applies.
- Inserts are synchronous by default: the fastest per save() and errors
  are reported. Connections can opt into async_insert=1, which keeps
  wait_for_async_insert=1 so failed inserts are never lost silently.
- update_all() and delete_all() return null, as ClickHouse does not
  report affected rows. Transactions throw. DELETE without conditions
  gets the WHERE ClickHouse requires.
- Any other query string parameter of the connection URL becomes a
  ClickHouse setting. parse_connection_url() now parses the query string
  properly: explode('/&/') never split more than one parameter.

Model::insert_all($rows, ['batch_size' => N]) inserts many records with
multi-row INSERT statements on every adapter. It builds no models, so
validations and callbacks do not run. Aliases and inflected names are
resolved, dates and JSON converted, timestamps filled in and missing
keys generated as save() would. Rows setting the same columns share
statements, and batches shrink to fit the bound parameters the database
accepts. ClickHouse writes the rows as literals and sends batches of
1000 rows or more synchronously even on an async connection.

The core also copes with tables without a primary key: Model::insert()
and Table no longer read a missing pk[0].

New tests: ClickhouseAdapterTest and ClickhouseModelTest with their own
schema and models, InsertAllTest for every adapter, UuidTest. CI runs
ClickHouse 26.8 in every job and 25.9 in one row; compose.yaml has both.
Verified on MySQL 5.7 and 8.4, PostgreSQL 18, SQLite, ClickHouse 25.9
and 26.8.
PHP 8.5 deprecates ReflectionMethod::setAccessible(), which has done
nothing since 8.1. The test suite needs PHP 8.2 (PHPUnit 11.5), so the
call is never needed where the tests can run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant