A small toolbox of things I kept re-typing in every EDA notebook:
quick dataset reports, a few plots, and sklearn-compatible encoders that can be
dropped straight into a Pipeline / GridSearchCV.
Nothing here is a framework. Each helper is one function or one class you can read in a minute.
pip install bitskittenThe latest commit: pip install git+https://github.com/Caerno/bitskit.git.
On Kaggle / Colab the same lines work in a cell (!pip install ...).
For a kernel with no internet, add
the archive GitHub builds for every commit
as a dataset and install it from disk:
pip install --no-deps /kaggle/input/<your-dataset>/bitskit-main.zipRequires Python ≥ 3.9, numpy, pandas, scikit-learn, matplotlib, seaborn.
import pandas as pd
from bitskitten import informer, informer_print, imperfection, text_imperfection, NanFixer, Digitalize
df = pd.concat([train, test]) # test rows have no target -> NaN
informer_print(informer(df, "Survived"))
# Features: 12
# Observations: 891/418
# train dataset: 68.0%
# classes: 0: 61.6%, 1: 38.4%
imperfection(df) # what is missing and how varied it isimperfection returns a frame sorted by NA share — share of missing values, number of
unique values, dtype and the values themselves. By default it shows non-numeric columns
only; numeric=True adds the rest, no_nan=True keeps only columns that actually have gaps.
informer switches to describe() of the target when it has more than max_classes distinct
values (regression).
text_imperfection(texts) is the same look at a corpus: one row per defect — empty, too short,
control characters, not NFKC-normalized, URL, HTML tag, base64-like run, repeated character or
word, Cyrillic and Latin mixed inside a word, few letters, exact duplicate — with share, count
and a few offending examples. Rows with zero hits are hidden (no_clean=False shows them).
Both are TransformerMixin + BaseEstimator and pass check_estimator: statistics are
learned at fit and applied at transform, so the test set never leaks into its own filling,
a batch that lacks a category still comes out with the same columns, get_feature_names_out,
set_output(transform="pandas"), check_is_fitted, clone and grid search all work.
A frame in gives a frame out; an array in gives an array out (or a frame with set_output).
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
pipe = Pipeline([
("nan", NanFixer()), # fill the gaps
("dig", Digitalize()), # make everything numeric
("clf", RandomForestClassifier()),
])
grid = GridSearchCV(pipe, {
"nan__lim": [0.1, 0.225, 0.5],
"nan__method": ["mean", "median"],
"dig__mode": ["raw", "cut", "qcut"],
"dig__ncut": [5, 10],
})NanFixer(lim=0.225, method="mean", filler="U", random_state=None) — picks a filling
strategy per column at fit time, measures it on the fit data, applies it at transform:
| column | condition (on the fit data) | fill |
|---|---|---|
| numeric | NA share ≤ lim |
method (mean/median) of the fit data |
| numeric | NA share > lim |
draws from N(mean, std) of the fit data, clipped to its range |
| object | one value dominates (>25%) | that mode |
| object | otherwise | the filler string |
Columns of any other dtype pass through untouched. The plan is in plan_ after fit.
Digitalize(alim=10, mode="raw", ncut=5) — turns a mixed frame into a numeric one:
| column | condition (on the fit data) | result |
|---|---|---|
| numeric | < 3 unique values | as is |
| any | < alim unique values |
one-hot over the categories seen at fit: col_value; an unseen value → all zeros |
| numeric | mode="cut" / "qcut" |
col_cut5 / col_qcut5: ncut equal-spaced / equal-sized bins with edges from the fit data; out-of-range values go to the edge bins |
| numeric | mode="raw" |
as is |
| non-numeric | ≥ alim unique values |
col_top5: frequency rank from the fit data (most frequent → 1, the tail and unseen values → ncut) |
Output column names are get_feature_names_out(); up to 0.2 they were pd.get_dummies
prefixes (col[:5]_value) and depended on the batch.
FeatureTrans(series) is the shared kit both encoders are built from, and it is
iterable: it yields every transformation that makes sense for that column's dtype —
handy when you want to throw candidate features at a selector.
candidates = pd.concat(list(FeatureTrans(df["Fare"])), axis=1)
# Fare_count, Fare_freqn, Fare_top5..., Fare_cut5..., Fare_qcut5..., Fare_sqrt, Fare_log2, Fare_pow2 ...Its pieces are static methods you can also call directly: count, freq, top,
cut, qcut, math, bilabel, fuzzy_mean, robust, fill.
from bitskitten import harmonic_score, harmonic_scorer
harmonic_score(y_true, y_pred) # 1.0 perfect, 0.0 if any class is lost
cross_val_score(clf, X, y, scoring=harmonic_scorer)
GridSearchCV(pipe, grid, scoring=harmonic_scorer)harmonic_score is the harmonic mean of every per-class precision and every per-class
recall — a class-agnostic generalization of F1 for when no class is "the positive one".
A class that is never predicted or never recovered pulls it to 0. harmonic_from_confusion(conf)
returns (score, precision[], recall[]) from a ready confusion matrix.
from bitskitten.bandit import ThompsonScheduler
sched = ThompsonScheduler(["happy_path", "reschedule", "angry_user"],
threshold=0.9, seed=0)
report = sched.run(run_scenario, budget=80) # run_scenario(name) -> bool
print(report) # runs, passes, mean, 95 % interval, pass^3, verdict per scenarioFor suites where one run of a scenario is a coin flip (LLM bots, anything
stochastic). Each scenario carries a Beta posterior over its pass rate; the next
run goes to the scenario most likely to fail (Thompson sampling), scenarios whose
interval has cleared or missed the threshold are retired, and the loop stops when
nothing is left to learn or the budget is out. pass_power(alpha, beta, k) is the
tau-bench pass^k with the posterior integrated out; width= retires scenarios that
sit on the threshold once their interval is narrow enough; min_runs= (default 3)
stops a single failure from deciding a verdict.
from bitskitten.qa import Suite, Judge, Checklist, ScriptedUser, TopicClassifier, BotStep, format_report
suite = Suite(scenarios, # Scenario.from_dict(...) per YAML entry
new_bot=lambda sc: MyBot(), # bot(text) -> str | BotStep(text, state, slots, signals)
new_user=lambda sc, seed: ScriptedUser(classify, asks, facts=sc.facts, seed=seed),
gates={"collecting_states": {"collecting_date"}},
judge=Judge(model, Checklist.from_dicts(items))) # model(prompt) -> str, any provider
result = suite.run(budget=80, threshold=0.9, max_runs=40)
print(result.report) # per scenario: runs, mean, 95 % interval, pass^3, verdict
print(format_report(result.failures)) # failed runs grouped by what failedFor bots where one run is a coin flip. Per run, cheapest first: the outcome against the scenario and trace invariants (stuck in a state, the same reply verbatim, invalid slot value, a number not in the allowed set, a scripted user's record of re-asks) need no model; the judge is called only on runs that pass them. The next run goes where a failure is most likely (Thompson sampling); decided scenarios retire, the loop stops when nothing is left to learn.
- Users.
ScriptedUserreacts to what the bot said (markers you supply), hands out facts and recordsrepeated_ask/unexpected_askinstead of quietly answering them.MarkovUsersvaries wording from real utterances at zero model cost.CallableUserwraps any model. - Judge. Repeated judging merges binary items by majority, not by mean.
item_agreement(judged, human)gives per-item kappa against your own labels — check it before trusting an item;pick_thresholdpicks the score cut. - Cost of a verdict. Proving "passes at least 90 %" at 95 % takes 35 clean
runs in a row; that is the threshold's price, not the scheduler's. A scenario
sitting right at the threshold cannot be decided cheaply — cap it with
max_runs. validate_scenarios(entries, Vocabulary(...))checks a scenario file and flags absolute dates that will expire.
from bitskitten import plot_conf_map, plot_some_scatters
hm, precision, recall = plot_conf_map(confusion_matrix(y_true, y_pred), title="RF")plot_conf_map draws the confusion matrix as shares, with a precision row, a recall
column and — in the corner — harmonic_score. Returns (harmonic_mean, precision, recall);
blind=True computes without drawing.
plot_some_scatters(X, y) — pairwise scatter plots of every column combination, coloured by y.
multicolumn(series, cols=5) folds a long series into several columns so a wide
df.sample().T fits on one screen.
img_squeeze reduces the dimensionality of an RGBA image by unrolling channels into a
flat 2D matrix, running any sklearn decomposition on it and rebuilding the picture:
from sklearn.decomposition import PCA
from bitskitten import img_framaker
img_framaker(img, 20, PCA) # image restored from 20 componentsSlowPolyLinearReg(gener=[0, 1], norm=None, alpha=None) — linear regression written on
numpy/pandas, with polynomial and non-linear feature generation (0 = intercept,
integers = powers, "sqrt"/"cbrt"/"log2"/"ln"/"log10"), optional normalization
("minmax", "std", "max", "mean") and L2 regularization. Closed-form solution,
so it warns you about singular and ill-conditioned matrices instead of hiding them
(and falls back to the least-norm solution rather than giving up).
It exists to show the mechanics, not to compete with sklearn.linear_model.
The default gener=[0, 1] is the plain intercept + x. Drop the 0 only if you
really want the line forced through the origin.
model = SlowPolyLinearReg(gener=[0, 1, 2], norm="std", alpha=0.01)
model.fit(X, y)
model.score(X, y) # R²number("How many? ", t=int, op=">=", limit=0) — keeps asking until the answer parses
and satisfies the condition.
pip install -e .[test]
pytestThe repo root is the package (flat layout), so the clone has to be in a folder named
bitskitten for pytest to import it; CI installs the wheel first for the same reason.
The drop-in zip in download/ is built from committed files only, so no __pycache__
gets in:
git archive --format=zip -o download/bitskitten.zip HEAD \
__init__.py bandit.py classes.py input.py metrics.py output.py text.py qa