Skip to content

feat: retune AI moderation thresholds (ban at ≥0.95) #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Bonobo791 merged 1 commit into from
Jul 31, 2026
Merged

feat: retune AI moderation thresholds (ban at ≥0.95) #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DESIGN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,7 @@ violet-tinted Gel Border, dashed only for empty states.

### Lighting plot (signature)
Landing rule/threshold lists render as a cue sheet: a top rule line, rows of
stencil cue-kind labels (KEYWORD / ≥ 0.85) against their action, separated by
stencil cue-kind labels (KEYWORD / ≥ 0.95) against their action, separated by
28%-alpha currentColor hairlines, numerals always tabular. This is how the
world shows a list of consequences — reuse it for any landing enumeration.

Expand Down
25 changes: 13 additions & 12 deletions EXECUTION_PLAN_YouTube_Comment_Moderator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ By the end, there is a working SvelteKit app `yt-mod` that:

1. Lets a YouTube channel owner connect their channel via Google OAuth (scope `youtube.force-ssl`, offline access). The AES-256-GCM-encrypted refresh token is stored in the DB.
2. Stores per-channel rules: keyword, regex (recheck-validated), or blocked-user, each mapping to `hold` | `reject` | `delete` | `ban`.
3. Exposes `GET /api/cron?secret=<CRON_SECRET>` which processes **one channel per run** (least-recently-run first) and **one page of ≤100 comments per run**, with a persisted checkpoint (continuation token + high-water mark) so bursts larger than one page are drained across runs **without skipping comments**. Per comment: rule match first, then OpenAI Moderation. Score ≥ 0.85 → auto-reject; 0.35–0.85 → human review queue; < 0.35 → approved. Rule hits execute their configured action. **Every enforcement action is recorded in the DB (`action_pending`) BEFORE the YouTube write, then confirmed after** — a crash mid-run is reconciled on the next run, never repeated blindly and never lost.
3. Exposes `GET /api/cron?secret=<CRON_SECRET>` which processes **one channel per run** (least-recently-run first) and **one page of ≤100 comments per run**, with a persisted checkpoint (continuation token + high-water mark) so bursts larger than one page are drained across runs **without skipping comments**. Per comment: rule match first, then OpenAI Moderation. Score ≥ 0.95 → auto-ban (reject + ban author); 0.76–0.94 → auto-reject; 0.51–0.75 → human review queue; ≤ 0.50 → approved. Rule hits execute their configured action. **Every enforcement action is recorded in the DB (`action_pending`) BEFORE the YouTube write, then confirmed after** — a crash mid-run is reconciled on the next run, never repeated blindly and never lost.

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail
rg -n -C 8 'action_pending|pendingAction|setModerationStatus|deleteComment' src/routes

Repository: Bonobo791/Moderaty

Length of output: 10428

🏁 Script executed:

#!/bin/bash
set -euo pipefail
file='src/routes/(app)/channels/[id]/queue/+page.server.ts'
schema=$(rg -l 'pendingAction|action_pending' src/lib src/routes | head -n 1 || true)
printf '%s\n' '--- action implementation ---'
sed -n '45,75p' "$file"
printf '%s\n' '--- pending-action references ---'
rg -n -C 3 'pendingAction|action_pending' src/lib src/routes || true
printf '%s\n' '--- relevant schema fields ---'
rg -n -C 4 'comments|status|decidedBy|auditLog' src/lib/server/db/schema* src/lib/server/db 2>/dev/null | head -n 160
printf '%s\n' '--- plan references ---'
rg -n -C 5 'action_pending|pendingAction|Step 21|setModerationStatus|deleteComment' EXECUTION_PLAN_YouTube_Comment_Moderator.md

Repository: Bonobo791/Moderaty

Length of output: 24436

Implement DB-before-remote ordering for manual queue actions. The runtime calls YouTube before updating comments, and comments has no pendingAction field. Add the pending-action state, persist it before setModerationStatus or deleteComment, then finalize and audit after success.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@EXECUTION_PLAN_YouTube_Comment_Moderator.md` at line 42, Update manual queue
action handling to add a pending-action state to comments and persist
action_pending before any YouTube setModerationStatus or deleteComment call.
After the remote operation succeeds, finalize the comment state and write the
audit record; ensure pending actions remain recoverable after failures.

Source: Coding guidelines

4. Has four pages — dashboard (`/`), rules editor, review queue (one-click approve/reject/delete/ban), audit log — with a finished visual design (brand "Moderaty", design tokens, styled components, empty/loading/error states, accessible markup).
5. Supports `DRY_RUN=true`: classifies and writes audit rows (action `dry-run`) but performs **no** YouTube writes and **no** DB state changes (no comment rows, no cursor/checkpoint movement) — previews are repeatable.
6. `npm run check`, `npm run build`, `npm run test` all exit 0.
Expand All @@ -57,8 +57,8 @@ API facts (use exactly; do not look up alternatives):
- Channel lookup: `GET /channels?part=snippet&mine=true`
- OAuth: auth URL `https://accounts.google.com/o/oauth2/v2/auth`, token endpoint `https://oauth2.googleapis.com/token`. Auth URL params: `client_id`, `redirect_uri`, `response_type=code`, `scope=https://www.googleapis.com/auth/youtube.force-ssl`, `access_type=offline`, `prompt=consent`.
- OpenAI Moderation: `POST https://api.openai.com/v1/moderations`, header `Authorization: Bearer $OPENAI_API_KEY`, body `{ "model": "omni-moderation-latest", "input": "<text>" }`. Response: `results[0].category_scores` (category → float).
- Toxicity categories: `harassment`, `harassment/threatening`, `hate`, `hate/threatening`, `violence`, `violence/graphic`. Comment AI score = max of these six. Spam is NOT an AI category; users catch spam with their own rules.
- Thresholds (fixed): AUTO_REJECT = 0.85, QUEUE = 0.35.
- Toxicity categories: `harassment`, `harassment/threatening`, `hate`, `hate/threatening`, `illicit`, `illicit/violent`, `self-harm`, `self-harm/intent`, `self-harm/instructions`, `sexual`, `sexual/minors`, `violence`, `violence/graphic`. Comment AI score = max of these thirteen. Spam is NOT an AI category; users catch spam with their own rules.
- Thresholds (fixed): AUTO_BAN = 0.95, AUTO_REJECT = 0.76, QUEUE = 0.51.
Comment on lines +60 to +61

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Update the embedded moderation specification to use all thirteen categories.

The revised contract requires thirteen keys, but the TOXIC_CATEGORIES block at Lines 528-535 and the M1 fixture at Lines 923-926 still enumerate only six. The missing-key requirement at Lines 932-941 would therefore validate only the old set. Add illicit, illicit/violent, self-harm, self-harm/intent, self-harm/instructions, sexual, and sexual/minors to the embedded implementation and fixture. The supplied src/lib/server/moderation.test.ts context already exercises sexual.

Also applies to: 241-241, 922-922, 932-941

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@EXECUTION_PLAN_YouTube_Comment_Moderator.md` around lines 60 - 61, Update the
embedded moderation specification’s TOXIC_CATEGORIES block and the M1 fixture to
include all thirteen required toxicity keys, adding illicit, illicit/violent,
self-harm, self-harm/intent, self-harm/instructions, sexual, and sexual/minors
alongside the existing categories. Ensure the missing-key validation uses this
complete category set while preserving the existing score and threshold
behavior.

- **Every field in every external response is optional until validated.** YouTube omits `authorChannelId` for deleted accounts; OpenAI may return out-of-range values. Handle per the invariants — never abort a batch over one malformed item.
- Top-level comments only (replies are a non-goal).

Expand Down Expand Up @@ -238,7 +238,7 @@ export const comments = sqliteTable('comments', {
pendingAction: text('pending_action'), // 'hold' | 'reject' | 'delete' | 'ban' while action_pending; else null
decidedBy: text('decided_by').notNull(), // 'rule' | 'ai' | 'human' | 'none'
matchedRuleId: integer('matched_rule_id'),
aiScore: text('ai_score'), // JSON of the six category scores, or null
aiScore: text('ai_score'), // JSON of the thirteen category scores, or null
createdAt: text('created_at').notNull()
});

Expand Down Expand Up @@ -606,8 +606,9 @@ import { refreshAccessToken, fetchCommentPage, setModerationStatus, deleteCommen
import { scoreComment, ModerationError } from './moderation';
import { matchRule } from './rules';

const AUTO_REJECT = 0.85;
const QUEUE = 0.35;
const AUTO_BAN = 0.95;
const AUTO_REJECT = 0.76;
const QUEUE = 0.51;

type PendingAction = 'hold' | 'reject' | 'delete' | 'ban';

Expand Down Expand Up @@ -918,7 +919,7 @@ function okResponse(scores: Record<string, number>) {
}

describe('scoreComment', () => {
it('M1: returns max of the six toxic categories', async () => {
it('M1: returns max of the thirteen toxic categories', async () => {
vi.stubGlobal('fetch', vi.fn(async () => okResponse({ harassment: 0.4, 'harassment/threatening': 0.1, hate: 0.9, 'hate/threatening': 0.1, violence: 0.1, 'violence/graphic': 0.1 })));

@codacy-production codacy-production Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM RISK

The mock response for test case M1 is missing 7 of the 13 required toxicity categories. According to the updated logic on line 932 and the M6 case, this missing data will cause a ModerationError. Update the mock to include all 13 categories to ensure the test passes.\n\nTry running the following prompt in your coding agent:\n> In EXECUTION_PLAN_YouTube_Comment_Moderator.md, update the scoreComment mock on line 923 to include all 13 toxicity categories (harassment, harassment/threatening, hate, hate/threatening, illicit, illicit/violent, self-harm, self-harm/intent, self-harm/instructions, sexual, sexual/minors, violence, violence/graphic) so it complies with the updated validation requirements.

const m = await scoreComment('x');
expect(m.score).toBe(0.9);
Expand All @@ -928,16 +929,16 @@ describe('scoreComment', () => {
});
```

Note: a missing key among the six is INVALID (throws), per the module code and I2. Required cases:
Note: a missing key among the thirteen is INVALID (throws), per the module code and I2. Required cases:

| # | Mocked API behavior | Expected |
|---|---|---|
| M1 | 200, six valid scores, max = hate 0.9 | `score === 0.9`, `scores.hate === 0.9` |
| M2 | 200, score exactly 0.85 | returns 0.85 (boundary belongs to caller, not this module) |
| M1 | 200, thirteen valid scores, max = hate 0.9 | `score === 0.9`, `scores.hate === 0.9` |
| M2 | 200, score exactly 0.95 | returns 0.95 (boundary belongs to caller, not this module) |
| M3 | 200, one score = 1.7 | throws `ModerationError` |
| M4 | 200, one score = -0.2 | throws `ModerationError` |
| M5 | 200, one score = "high" (string) | throws `ModerationError` |
| M6 | 200, one of the six keys missing | throws `ModerationError` |
| M6 | 200, one of the thirteen keys missing | throws `ModerationError` |
| M7 | 200, `results` missing | throws `ModerationError` |
| M8 | 500 status | throws `ModerationError` |
| M9 | 200, non-JSON body | throws `ModerationError` |
Expand Down Expand Up @@ -1512,7 +1513,7 @@ export const actions = {
</script>

<h1>Review queue — {data.ch?.title}</h1>
<p class="muted">Borderline comments (AI score 0.35–0.85, or AI unavailable). Your action is final.</p>
<p class="muted">Borderline comments (AI score 0.51–0.75, or AI unavailable). Your action is final.</p>

{#each data.pending as c}
<div class="card">
Expand Down
8 changes: 4 additions & 4 deletions PRODUCT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,10 @@ Category line: "Comment protection for YouTube creators."

- Rule types: keyword, regex (RE2 syntax only — never `new RegExp` on user
patterns), blocked-user; actions: hold, reject, delete, ban.
- AI scoring: OpenAI Moderation `omni-moderation-latest`; score = max of six
toxicity categories; fixed thresholds >0.85 auto-ban, 0.51–0.85 auto-delete,
0.35–0.50 human queue, <0.35 approved. AI failure always routes to the human
queue — never auto-approve, never auto-reject.
- AI scoring: OpenAI Moderation `omni-moderation-latest`; score = max of the
thirteen toxicity categories; fixed thresholds ≥0.95 auto-ban, 0.76–0.94
auto-reject, 0.51–0.75 human queue, ≤0.50 approved. AI failure always routes
to the human queue — never auto-approve, never auto-reject.
- Enforcement durability: every action is recorded locally (`action_pending`)
BEFORE any YouTube write and confirmed after; `DRY_RUN=true` previews change
nothing durable (audit rows only).
Expand Down
19 changes: 11 additions & 8 deletions src/lib/server/pipeline.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,12 +229,12 @@ afterEach(() => {
});

test.each([
{ score: 0.34, status: 'approved', queued: 0, acted: 0, api: 'none', audit: 'approve' },
{ score: 0.35, status: 'pending', queued: 1, acted: 0, api: 'none', audit: 'queue' },
{ score: 0.5, status: 'pending', queued: 1, acted: 0, api: 'none', audit: 'queue' },
{ score: 0.51, status: 'deleted', queued: 0, acted: 1, api: 'delete', audit: 'delete' },
{ score: 0.85, status: 'deleted', queued: 0, acted: 1, api: 'delete', audit: 'delete' },
{ score: 0.86, status: 'rejected', queued: 0, acted: 1, api: 'ban', audit: 'ban' }
{ score: 0.5, status: 'approved', queued: 0, acted: 0, api: 'none', audit: 'approve' },
{ score: 0.51, status: 'pending', queued: 1, acted: 0, api: 'none', audit: 'queue' },
{ score: 0.75, status: 'pending', queued: 1, acted: 0, api: 'none', audit: 'queue' },
{ score: 0.76, status: 'rejected', queued: 0, acted: 1, api: 'reject', audit: 'reject' },
{ score: 0.94, status: 'rejected', queued: 0, acted: 1, api: 'reject', audit: 'reject' },
{ score: 0.95, status: 'rejected', queued: 0, acted: 1, api: 'ban', audit: 'ban' }
])('categorizes score $score as $status', async ({ score, status, queued, acted, api, audit }) => {
mocks.scoreComment.mockResolvedValue(moderation(score));

Expand All @@ -245,6 +245,9 @@ test.each([
if (api === 'delete') {
expect(mocks.deleteComment).toHaveBeenCalledWith('comment', 'access-token', undefined);
expect(mocks.setModerationStatus).not.toHaveBeenCalled();
} else if (api === 'reject') {
expect(mocks.setModerationStatus).toHaveBeenCalledWith(['comment'], 'rejected', false, 'access-token', undefined);
expect(mocks.deleteComment).not.toHaveBeenCalled();
} else if (api === 'ban') {
expect(mocks.setModerationStatus).toHaveBeenCalledWith(['comment'], 'rejected', true, 'access-token', undefined);
expect(mocks.deleteComment).not.toHaveBeenCalled();
Expand Down Expand Up @@ -454,8 +457,8 @@ test('retries a dispatched ban while the comment is still public', async () => {
});

test.each([
{ raw: 0.506, status: 'deleted', reason: 'ai score 0.51' },
{ raw: 0.504, status: 'pending', reason: 'ai score 0.50' }
{ raw: 0.506, status: 'pending', reason: 'ai score 0.51' },
{ raw: 0.504, status: 'approved', reason: 'ai score 0.50' }
Comment on lines +460 to +461

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add the upper rounding-boundary cases.

The changed cases cover only the lower transitions. Add raw 0.755 with status rejected, API action reject, and reason ai score 0.76. Add raw 0.945 with status rejected, API action ban, and reason ai score 0.95.

As per coding guidelines, tests must fail when the real logic is wrong.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/server/pipeline.test.ts` around lines 460 - 461, Add upper
rounding-boundary cases to the pipeline tests covering raw 0.755 as rejected
with API action reject and reason “ai score 0.76”, and raw 0.945 as rejected
with API action ban and reason “ai score 0.95”. Ensure the assertions validate
status, action, and reason so incorrect rounding logic causes the tests to fail.

Source: Coding guidelines

])('rounds the AI score to 2 decimals before deciding ($raw → $status)', async ({ raw, status, reason }) => {
mocks.scoreComment.mockResolvedValue(moderation(raw));

Expand Down
16 changes: 8 additions & 8 deletions src/lib/server/pipeline.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ import {
type NewComment
} from '$lib/server/youtube';

const AUTO_BAN = 0.85;
const AUTO_DELETE = 0.51;
const QUEUE = 0.35;
const AUTO_BAN = 0.95;
const AUTO_REJECT = 0.76;
const QUEUE = 0.51;

export interface RunChannelOptions {
maxPages?: number;
Expand Down Expand Up @@ -131,7 +131,7 @@ async function aiDecision(comment: NewComment, deadline?: number): Promise<Decis
const score = Math.round(moderation.score * 100) / 100;
const reason = `ai score ${score.toFixed(2)}`;
const aiScore = serializeScores(moderation.scores);
if (score > AUTO_BAN) {
if (score >= AUTO_BAN) {
return {
comment,
status: 'rejected',
Expand All @@ -143,16 +143,16 @@ async function aiDecision(comment: NewComment, deadline?: number): Promise<Decis
youtubeAction: 'ban'
};
}
if (score >= AUTO_DELETE) {
if (score >= AUTO_REJECT) {
return {
comment,
status: 'deleted',
status: 'rejected',
decidedBy: 'ai',
matchedRuleId: null,
aiScore,
auditAction: 'delete',
auditAction: 'reject',
reason,
youtubeAction: 'delete'
youtubeAction: 'reject'
};
}
if (score >= QUEUE) {
Expand Down
2 changes: 1 addition & 1 deletion src/routes/(app)/channels/[id]/queue/+page.svelte
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Commercial licensing: contact@marketingprowess.simplelogin.com — see COMMERCIA
</svelte:head>

<h1>Review queue — {data.ch?.title}</h1>
<p class="page-sub">Borderline comments (AI score 0.35–0.50) waiting for your decision.</p>
<p class="page-sub">Borderline comments (AI score 0.51–0.75) waiting for your decision.</p>

@codacy-production codacy-production Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ LOW RISK

Suggestion: The help text here differs from the version specified in the Execution Plan and is technically incomplete. According to the logic in pipeline.ts (lines 111-128), this queue holds both borderline comments and those where AI scoring failed.\n\nConsider updating to:\nsuggestion\n<p class="page-sub">Borderline comments (AI score 0.51–0.75 or AI unavailable) waiting for your decision.</p>\n

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Mention AI-unavailable comments in the queue description.

src/lib/server/pipeline.ts places scoring failures in the pending queue. The execution plan also documents “or AI unavailable” at Line 1516. This subtitle names only the score range, so users can misinterpret AI-unavailable items.

Suggested copy 
-	<p class="page-sub">Borderline comments (AI score 0.51–0.75) waiting for your decision.</p>
+	<p class="page-sub">Borderline comments (AI score 0.51–0.75, or AI unavailable) waiting for your decision.</p>
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
<p class="page-sub">Borderline comments (AI score 0.51–0.75) waiting for your decision.</p>
<p class="page-sub">Borderline comments (AI score 0.51–0.75, or AI unavailable) waiting for your decision.</p>
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/routes/`(app)/channels/[id]/queue/+page.svelte at line 44, Update the
queue subtitle paragraph in the page markup to explicitly state that the queue
includes comments for which AI scoring is unavailable, while retaining the
existing borderline score-range description.


{#if data.pending === undefined}
<Skeleton rows={3} />
Expand Down
10 changes: 5 additions & 5 deletions src/routes/+page.svelte
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,13 +190,13 @@ Commercial licensing: contact@marketingprowess.simplelogin.com — see COMMERCIA
<h2>Then the AI scores what your rules miss.</h2>
<p>
Every comment your rules don&rsquo;t catch is scored by OpenAI&rsquo;s moderation model
across six toxicity categories. The highest score decides:
across thirteen toxicity categories. The highest score decides:
</p>
<div class="plot" role="list" aria-label="Score thresholds">
<p role="listitem"><span class="plot-kind">&gt; 0.85</span> AUTHOR BANNED AUTOMATICALLY</p>
<p role="listitem"><span class="plot-kind">0.51 – 0.85</span> DELETED AUTOMATICALLY</p>
<p role="listitem"><span class="plot-kind">0.35 – 0.50</span> HELD FOR YOUR REVIEW</p>
<p role="listitem"><span class="plot-kind">&lt; 0.35</span> APPROVED</p>
<p role="listitem"><span class="plot-kind">&ge; 0.95</span> AUTHOR BANNED AUTOMATICALLY</p>
<p role="listitem"><span class="plot-kind">0.76 – 0.94</span> REJECTED AUTOMATICALLY</p>
<p role="listitem"><span class="plot-kind">0.51 – 0.75</span> HELD FOR YOUR REVIEW</p>
<p role="listitem"><span class="plot-kind">&le; 0.50</span> APPROVED</p>
</div>
<p class="fine">
If the AI can&rsquo;t score a comment, it lands in your queue — never auto-approved,
Expand Down