language-model-evaluation

A library & tools to evaluate predictive language models.

A data construction and evaluation framework to quantify privacy norm awareness of language models (LMs) and emerging privacy risk of LM agents. (NeurIPS 2024 D&B)

Code and data for "KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark" (LREC-COLING 2024)

Layer-wise Semantic Dynamics (LSD) is a model-agnostic framework for hallucination detection in Large Language Models (LLMs). It analyzes the geometric evolution of hidden-state semantics across transformer layers, using contrastive alignment between model activations and ground-truth embeddings to detect factual drift and semantic inconsistency.

Curriculum is a new format of NLI benchmark for evaluation of broad-coverage linguistic phenomena. This linguistic-phenomena-driven benchmark can serve as an effective tool for diagnosing model behavior and verifying model learning quality.

Language Modeling

Nexus Random Initialization Foundry (RIF) is a browser-based toolkit for training language models from scratch with WebGPU. Explore custom corpora, dense and low-bit models, recovery training, checkpoints, evaluation and GGUF export. Experimental and designed for local AI development.

A thesis investigating the use of large language models for summarizing application logs.

Probed gender-science bias in transformer language models by implementing WEAT on English and Urdu embeddings. Conducted comparative evaluation across BERT, RoBERTa, DistilBERT, and XLM-RoBERTa, revealing model and language-dependent bias patterns and providing reproducible benchmarks.

Evaluation framework for measuring LLM reasoning and translation performance across low-resource languages using glossary-guided prompting and semantic similarity metrics.

Extrapolation Is an Assumption: Sharp Limits for Pass@k Forecasting DOI: https://doi.org/10.5281/zenodo.21698610