Skip to content
CaernoPublic

About

EDA reports, plots and sklearn-compatible encoders

Resources

Stars

0 stars

Watchers

1 watching

Forks

Repository files navigation

bitskitten

A small toolbox of things I kept re-typing in every EDA notebook: quick dataset reports, a few plots, and sklearn-compatible encoders that can be dropped straight into a Pipeline / GridSearchCV.

Nothing here is a framework. Each helper is one function or one class you can read in a minute.

Install

pip install bitskitten

The latest commit: pip install git+https://github.com/Caerno/bitskit.git. On Kaggle / Colab the same lines work in a cell (!pip install ...). For a kernel with no internet, add the archive GitHub builds for every commit as a dataset and install it from disk:

pip install --no-deps /kaggle/input/<your-dataset>/bitskit-main.zip

Requires Python ≥ 3.9, numpy, pandas, scikit-learn, matplotlib, seaborn.

Quick start

import pandas as pd
from bitskitten import informer, informer_print, imperfection, text_imperfection, NanFixer, Digitalize

df = pd.concat([train, test])          # test rows have no target -> NaN

informer_print(informer(df, "Survived"))
#   Features:       12
#   Observations:   891/418
#   train dataset:  68.0%
#   classes:        0: 61.6%, 1: 38.4%

imperfection(df)                       # what is missing and how varied it is

imperfection returns a frame sorted by NA share — share of missing values, number of unique values, dtype and the values themselves. By default it shows non-numeric columns only; numeric=True adds the rest, no_nan=True keeps only columns that actually have gaps. informer switches to describe() of the target when it has more than max_classes distinct values (regression).

text_imperfection(texts) is the same look at a corpus: one row per defect — empty, too short, control characters, not NFKC-normalized, URL, HTML tag, base64-like run, repeated character or word, Cyrillic and Latin mixed inside a word, few letters, exact duplicate — with share, count and a few offending examples. Rows with zero hits are hidden (no_clean=False shows them).

Encoders

Both are TransformerMixin + BaseEstimator and pass check_estimator: statistics are learned at fit and applied at transform, so the test set never leaks into its own filling, a batch that lacks a category still comes out with the same columns, get_feature_names_out, set_output(transform="pandas"), check_is_fitted, clone and grid search all work. A frame in gives a frame out; an array in gives an array out (or a frame with set_output).

from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

pipe = Pipeline([
    ("nan", NanFixer()),               # fill the gaps
    ("dig", Digitalize()),             # make everything numeric
    ("clf", RandomForestClassifier()),
])

grid = GridSearchCV(pipe, {
    "nan__lim":    [0.1, 0.225, 0.5],
    "nan__method": ["mean", "median"],
    "dig__mode":   ["raw", "cut", "qcut"],
    "dig__ncut":   [5, 10],
})

NanFixer(lim=0.225, method="mean", filler="U", random_state=None) — picks a filling strategy per column at fit time, measures it on the fit data, applies it at transform:

column condition (on the fit data) fill
numeric NA share ≤ lim method (mean/median) of the fit data
numeric NA share > lim draws from N(mean, std) of the fit data, clipped to its range
object one value dominates (>25%) that mode
object otherwise the filler string

Columns of any other dtype pass through untouched. The plan is in plan_ after fit.

Digitalize(alim=10, mode="raw", ncut=5) — turns a mixed frame into a numeric one:

column condition (on the fit data) result
numeric < 3 unique values as is
any < alim unique values one-hot over the categories seen at fit: col_value; an unseen value → all zeros
numeric mode="cut" / "qcut" col_cut5 / col_qcut5: ncut equal-spaced / equal-sized bins with edges from the fit data; out-of-range values go to the edge bins
numeric mode="raw" as is
non-numeric ≥ alim unique values col_top5: frequency rank from the fit data (most frequent → 1, the tail and unseen values → ncut)

Output column names are get_feature_names_out(); up to 0.2 they were pd.get_dummies prefixes (col[:5]_value) and depended on the batch.

FeatureTrans(series) is the shared kit both encoders are built from, and it is iterable: it yields every transformation that makes sense for that column's dtype — handy when you want to throw candidate features at a selector.

candidates = pd.concat(list(FeatureTrans(df["Fare"])), axis=1)
# Fare_count, Fare_freqn, Fare_top5..., Fare_cut5..., Fare_qcut5..., Fare_sqrt, Fare_log2, Fare_pow2 ...

Its pieces are static methods you can also call directly: count, freq, top, cut, qcut, math, bilabel, fuzzy_mean, robust, fill.

Metric

from bitskitten import harmonic_score, harmonic_scorer

harmonic_score(y_true, y_pred)                      # 1.0 perfect, 0.0 if any class is lost
cross_val_score(clf, X, y, scoring=harmonic_scorer)
GridSearchCV(pipe, grid, scoring=harmonic_scorer)

harmonic_score is the harmonic mean of every per-class precision and every per-class recall — a class-agnostic generalization of F1 for when no class is "the positive one". A class that is never predicted or never recovered pulls it to 0. harmonic_from_confusion(conf) returns (score, precision[], recall[]) from a ready confusion matrix.

Test scheduler for flaky suites

from bitskitten.bandit import ThompsonScheduler

sched = ThompsonScheduler(["happy_path", "reschedule", "angry_user"],
                          threshold=0.9, seed=0)
report = sched.run(run_scenario, budget=80)   # run_scenario(name) -> bool
print(report)      # runs, passes, mean, 95 % interval, pass^3, verdict per scenario

For suites where one run of a scenario is a coin flip (LLM bots, anything stochastic). Each scenario carries a Beta posterior over its pass rate; the next run goes to the scenario most likely to fail (Thompson sampling), scenarios whose interval has cleared or missed the threshold are retired, and the loop stops when nothing is left to learn or the budget is out. pass_power(alpha, beta, k) is the tau-bench pass^k with the posterior integrated out; width= retires scenarios that sit on the threshold once their interval is narrow enough; min_runs= (default 3) stops a single failure from deciding a verdict.

Regression testing for dialogue agents

from bitskitten.qa import Suite, Judge, Checklist, ScriptedUser, TopicClassifier, BotStep, format_report

suite = Suite(scenarios,                                   # Scenario.from_dict(...) per YAML entry
              new_bot=lambda sc: MyBot(),                  # bot(text) -> str | BotStep(text, state, slots, signals)
              new_user=lambda sc, seed: ScriptedUser(classify, asks, facts=sc.facts, seed=seed),
              gates={"collecting_states": {"collecting_date"}},
              judge=Judge(model, Checklist.from_dicts(items)))   # model(prompt) -> str, any provider
result = suite.run(budget=80, threshold=0.9, max_runs=40)
print(result.report)                    # per scenario: runs, mean, 95 % interval, pass^3, verdict
print(format_report(result.failures))   # failed runs grouped by what failed

For bots where one run is a coin flip. Per run, cheapest first: the outcome against the scenario and trace invariants (stuck in a state, the same reply verbatim, invalid slot value, a number not in the allowed set, a scripted user's record of re-asks) need no model; the judge is called only on runs that pass them. The next run goes where a failure is most likely (Thompson sampling); decided scenarios retire, the loop stops when nothing is left to learn.

  • Users. ScriptedUser reacts to what the bot said (markers you supply), hands out facts and records repeated_ask / unexpected_ask instead of quietly answering them. MarkovUsers varies wording from real utterances at zero model cost. CallableUser wraps any model.
  • Judge. Repeated judging merges binary items by majority, not by mean. item_agreement(judged, human) gives per-item kappa against your own labels — check it before trusting an item; pick_threshold picks the score cut.
  • Cost of a verdict. Proving "passes at least 90 %" at 95 % takes 35 clean runs in a row; that is the threshold's price, not the scheduler's. A scenario sitting right at the threshold cannot be decided cheaply — cap it with max_runs.
  • validate_scenarios(entries, Vocabulary(...)) checks a scenario file and flags absolute dates that will expire.

Plots

from bitskitten import plot_conf_map, plot_some_scatters

hm, precision, recall = plot_conf_map(confusion_matrix(y_true, y_pred), title="RF")

plot_conf_map draws the confusion matrix as shares, with a precision row, a recall column and — in the corner — harmonic_score. Returns (harmonic_mean, precision, recall); blind=True computes without drawing.

plot_some_scatters(X, y) — pairwise scatter plots of every column combination, coloured by y.

multicolumn(series, cols=5) folds a long series into several columns so a wide df.sample().T fits on one screen.

Image compression demo

img_squeeze reduces the dimensionality of an RGBA image by unrolling channels into a flat 2D matrix, running any sklearn decomposition on it and rebuilding the picture:

from sklearn.decomposition import PCA
from bitskitten import img_framaker

img_framaker(img, 20, PCA)   # image restored from 20 components

Estimator from scratch

SlowPolyLinearReg(gener=[0, 1], norm=None, alpha=None) — linear regression written on numpy/pandas, with polynomial and non-linear feature generation (0 = intercept, integers = powers, "sqrt"/"cbrt"/"log2"/"ln"/"log10"), optional normalization ("minmax", "std", "max", "mean") and L2 regularization. Closed-form solution, so it warns you about singular and ill-conditioned matrices instead of hiding them (and falls back to the least-norm solution rather than giving up). It exists to show the mechanics, not to compete with sklearn.linear_model.

The default gener=[0, 1] is the plain intercept + x. Drop the 0 only if you really want the line forced through the origin.

model = SlowPolyLinearReg(gener=[0, 1, 2], norm="std", alpha=0.01)
model.fit(X, y)
model.score(X, y)   # R²

Input

number("How many? ", t=int, op=">=", limit=0) — keeps asking until the answer parses and satisfies the condition.

Development

pip install -e .[test]
pytest

The repo root is the package (flat layout), so the clone has to be in a folder named bitskitten for pytest to import it; CI installs the wheel first for the same reason.

The drop-in zip in download/ is built from committed files only, so no __pycache__ gets in:

git archive --format=zip -o download/bitskitten.zip HEAD \
    __init__.py bandit.py classes.py input.py metrics.py output.py text.py qa

About

EDA reports, plots and sklearn-compatible encoders

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages