You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: retune AI moderation thresholds (ban at ≥0.95) - #21
Replaces the old bands (<0.35 approve / 0.35–0.50 queue / 0.51–0.85 delete / >0.85 ban). Note the AI no longer hard-deletes at any score — the mid band is now reject (hidden, irreversible, same as the ban path minus banAuthor).
Tests (test-first)
Threshold boundary table rewritten: 0.50/0.51/0.75/0.76/0.94/0.95 each assert DB status, audit action, and the exact YouTube call (reject vs ban vs none) — watched fail, then implemented, then green.
Rounding test updated: 0.504 → approved, 0.506 → queue.
99/99 tests, npm run check, npm run build all green.
AI moderation now evaluates 13 toxicity categories.
Updated decisions: approve scores ≤0.50, review 0.51–0.75, reject 0.76–0.94, and ban ≥0.95.
Rejected comments are now marked as rejected rather than deleted.
Updated review queue and moderation descriptions to reflect the new score ranges.
Increased the lighting-plot cue label threshold to 0.95.
Walkthrough
AI moderation now uses thirteen toxicity categories and revised score thresholds. Intermediate scores use YouTube rejection instead of deletion. Tests and application copy reflect the new behavior. A separate design example updates a lighting-plot threshold.
Changes
AI Moderation
Layer / File(s)
Summary
Moderation policy and score contract EXECUTION_PLAN_YouTube_Comment_Moderator.md, PRODUCT.md
The moderation model now defines thirteen toxicity scores with approval at ≤0.50, review at 0.51–0.75, rejection at 0.76–0.94, and banning at ≥0.95.
Pipeline actions and boundary tests src/lib/server/pipeline.ts, src/lib/server/pipeline.test.ts
The pipeline applies the new thresholds, uses YouTube’s rejection status for intermediate scores, and tests boundary and rounding behavior.
Moderation descriptions and queue copy src/routes/+page.svelte, src/routes/(app)/channels/[id]/queue/+page.svelte, EXECUTION_PLAN_YouTube_Comment_Moderator.md
The application and queue descriptions state the thirteen-category model and the revised score ranges.
Lighting-Plot Documentation
Layer / File(s)
Summary
Cue label threshold example DESIGN.md
The lighting-plot cue-kind example changes its threshold from 0.85 to 0.95.
Replaces the old bands (<0.35 approve / 0.35–0.50 queue / 0.51–0.85 delete / >0.85 ban). Note the AI no longer hard-deletes at any score — the mid band is now reject (hidden, irreversible, same as the ban path minus banAuthor).
Tests (test-first)
Threshold boundary table rewritten: 0.50/0.51/0.75/0.76/0.94/0.95 each assert DB status, audit action, and the exact YouTube call (reject vs ban vs none) — watched fail, then implemented, then green.
Rounding test updated: 0.504 → approved, 0.506 → queue.
99/99 tests, npm run check, npm run build all green.
PRODUCT.md: threshold bullet (also six → thirteen categories).
DESIGN.md: example cue label ≥ 0.85 → ≥ 0.95.
Landing page threshold plot and review-queue subtitle updated to the new bands (and stale "six toxicity categories" → thirteen).
CodeAnt-AI Description
Retune AI moderation outcomes and raise the automatic ban threshold
What Changed
Comments scoring up to 0.50 are approved, scores from 0.51–0.75 go to the human review queue, and scores from 0.76–0.94 are automatically rejected.
Scores of 0.95 or higher reject the comment and ban its author; mid-range rejections no longer delete comments.
Moderation scoring now considers thirteen toxicity categories, and the landing page, review queue, product notes, and execution plan show the updated ranges.
Tests cover the new score boundaries, rounding behavior, rejection actions, and ban behavior.
Impact
✅ Fewer automatic author bans for borderline comments ✅ Clearer human review range ✅ Automatic rejection without author bans below 0.95
💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
@codeant-ai ask: Your question here
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
@codeant-ai ask: Can you suggest a safer alternative to storing this secret?
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
@codeant-ai: Your feedback here
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
@codeant-ai: Do not flag unused imports.
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
@codeant-ai: review
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.
Operator change from > to >= for ban threshold is intentional and properly tested
Replace delete action with reject in mid-range band maintains consistency
All 99 tests pass with boundary conditions covered (0.50/0.51/0.75/0.76/0.94/0.95)
UI copy synchronized across all pages
Documentation updated to match implementation
No defects found that would block merge.
You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.
Retune AI moderation bands: approve ≤0.50, queue 0.51–0.75, reject 0.76–0.94, ban ≥0.95
✨ Enhancement🧪 Tests📝 Documentation🕐 40+ Minutes
AI Description
• Retune AI score bands to reduce auto-bans and introduce a non-banning reject tier.
• Replace AI-driven hard deletes with YouTube “reject” moderation writes.
• Update boundary tests and all user/docs copy to match the new thresholds.
Diagram
graph TD
A["New comment"] --> B["pipeline.ts"] --> C{"Score band?"}
C -->|"≤ 0.50"| D["Approve (DB + audit)"]
C -->|"0.51–0.75"| E["Queue pending (DB + audit)"]
C -->|"0.76–0.94"| F["Reject (DB + audit)"] --> Y["YouTube API"]
C -->|"≥ 0.95"| G["Ban author (DB + audit)"] --> Y
Loading
High-Level Assessment
The following are alternative approaches to this PR:
1. Make AI thresholds configurable (env/DB)
➕ Allows rapid tuning per environment or per channel without redeploys
➕ Enables A/B testing and safer rollout of policy changes
➖ Adds config surface area and validation requirements
➖ Increases risk of misconfiguration causing inconsistent moderation outcomes
2. Encode bands as a single data table (range → action)
➕ Centralizes policy in one structure; easier to audit/modify
➕ Reduces conditional drift between code, tests, and docs
➖ Slightly more indirection than straightforward if/else
➖ Still requires discipline to keep UI/docs in sync
3. Keep “delete” for the mid/high band (status=deleted)
➕ More aligned with “remove content” semantics
➕ Potentially simpler mental model (delete vs reject)
➖ Higher irreversibility and risk of over-enforcement
➖ Contradicts the stated goal of no AI hard-deletes
Recommendation: The PR’s approach (fixed constants + explicit tests at every boundary) is appropriate given the product’s stated “fixed thresholds” policy and keeps enforcement safer by removing AI hard-deletes. If further tuning is expected, consider a small follow-up to express the bands as a single mapping table (and optionally make the constants configurable) to reduce future drift across pipeline/tests/UI/docs.
Files changed (7) +43 / -39
Enhancement (1) +8 / -8
pipeline.tsRetune AI decision thresholds and switch auto-delete to auto-reject+8/-8
Retune AI decision thresholds and switch auto-delete to auto-reject
• Changes AI enforcement bands to approve ≤0.50, queue ≥0.51, reject ≥0.76, and ban ≥0.95 (inclusive). Replaces the AI-driven delete path with a reject action and updates the ban boundary comparison to be inclusive at the threshold.
pipeline.test.tsRewrite AI threshold boundary tests for approve/queue/reject/ban+11/-8
Rewrite AI threshold boundary tests for approve/queue/reject/ban
• Updates the threshold table to assert the new status/audit outcomes at 0.50/0.51/0.75/0.76/0.94/0.95 and adds assertions for the new YouTube reject call path. Adjusts rounding expectations so 0.504 rounds to approve and 0.506 rounds to queue.
EXECUTION_PLAN_YouTube_Comment_Moderator.mdRevise execution plan: new AI bands and 13-category scoring docs+13/-12
Revise execution plan: new AI bands and 13-category scoring docs
• Updates the documented cron/pipeline behavior to the new approve/queue/reject/ban bands and replaces the old delete/ban thresholds. Expands the documented moderation category set to thirteen and updates referenced constants and UI copy snippets accordingly.
PRODUCT.mdRetune product spec AI thresholds and category count+4/-4
Retune product spec AI thresholds and category count
• Updates the product spec to describe thirteen-category scoring and the new fixed thresholds (approve/queue/reject/ban) and AI-failure routing behavior.
+page.svelteUpdate landing page threshold plot and category count+5/-5
Update landing page threshold plot and category count
• Updates landing page copy to describe thirteen moderation categories and replaces the plotted threshold bands with the new approve/queue/reject/ban ranges.
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR successfully implements the transition from six to thirteen toxicity categories and retunes the moderation thresholds as specified. The system now correctly handles author bans at scores ≥ 0.95 and rejects comments in the 0.76–0.94 range without deleting them, aligning with the new 'reject' strategy. Codacy analysis indicates that the code is up to standards.\n\nHowever, there is a critical inconsistency in the EXECUTION_PLAN documentation where the test mock for the scoring function only includes six categories. This mismatch will trigger a ModerationError in the updated validation logic. Additionally, the UI copy for the human review queue should be adjusted to include a mention of AI failures to ensure full technical accuracy for end users.
Test suggestions
Score 0.50 results in approval\n- [x] Score 0.51 results in human review queue\n- [x] Score 0.75 results in human review queue\n- [x] Score 0.76 results in automatic rejection\n- [x] Score 0.94 results in automatic rejection\n- [x] Score 0.95 results in author ban\n- [x] Score 0.504 rounds down to 0.50 (approved)\n- [x] Score 0.506 rounds up to 0.51 (queued)\n- [x] AI failure results in human review queue
The reason will be displayed to describe this comment to others. Learn more.
🟡 MEDIUM RISK
The mock response for test case M1 is missing 7 of the 13 required toxicity categories. According to the updated logic on line 932 and the M6 case, this missing data will cause a ModerationError. Update the mock to include all 13 categories to ensure the test passes.\n\nTry running the following prompt in your coding agent:\n> In EXECUTION_PLAN_YouTube_Comment_Moderator.md, update the scoreComment mock on line 923 to include all 13 toxicity categories (harassment, harassment/threatening, hate, hate/threatening, illicit, illicit/violent, self-harm, self-harm/intent, self-harm/instructions, sexual, sexual/minors, violence, violence/graphic) so it complies with the updated validation requirements.
The reason will be displayed to describe this comment to others. Learn more.
⚪ LOW RISK
Suggestion: The help text here differs from the version specified in the Execution Plan and is technically incomplete. According to the logic in pipeline.ts (lines 111-128), this queue holds both borderline comments and those where AI scoring failed.\n\nConsider updating to:\nsuggestion\n<p class="page-sub">Borderline comments (AI score 0.51–0.75 or AI unavailable) waiting for your decision.</p>\n
The modified wrapped bullet lines in PRODUCT.md start with spaces rather than tabs, violating the
repository-wide indentation requirement. This can lead to inconsistent formatting and churn in diffs
when editors auto-normalize whitespace.
+ thirteen toxicity categories; fixed thresholds ≥0.95 auto-ban, 0.76–0.94+ auto-reject, 0.51–0.75 human queue, ≤0.50 approved. AI failure always routes+ to the human queue — never auto-approve, never auto-reject.
Relevance
● Weak
Same tabs-vs-spaces indentation suggestion in PRODUCT.md was rejected previously (PR #11); similar
docs/CSS tab-only requests also rejected.
ⓘ Recommendations generated based on similar findings in past PRs
Evidence
PR Compliance ID 2401137 requires tabs for indentation. The updated continuation lines in
PRODUCT.md are indented using leading spaces (before thirteen, auto-reject, and `to the human
queue`).
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
`PRODUCT.md` contains newly modified lines that use leading spaces for indentation. The compliance rule requires tabs for indentation, never spaces.
## Issue Context
The wrapped bullet continuation lines under the AI scoring bullet are indented with spaces.
## Fix Focus Areas
- PRODUCT.md[55-57]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@EXECUTION_PLAN_YouTube_Comment_Moderator.md`:
- Around line 60-61: Update the embedded moderation specification’s
TOXIC_CATEGORIES block and the M1 fixture to include all thirteen required
toxicity keys, adding illicit, illicit/violent, self-harm, self-harm/intent,
self-harm/instructions, sexual, and sexual/minors alongside the existing
categories. Ensure the missing-key validation uses this complete category set
while preserving the existing score and threshold behavior.
- Line 42: Update manual queue action handling to add a pending-action state to
comments and persist action_pending before any YouTube setModerationStatus or
deleteComment call. After the remote operation succeeds, finalize the comment
state and write the audit record; ensure pending actions remain recoverable
after failures.
In `@src/lib/server/pipeline.test.ts`:
- Around line 460-461: Add upper rounding-boundary cases to the pipeline tests
covering raw 0.755 as rejected with API action reject and reason “ai score
0.76”, and raw 0.945 as rejected with API action ban and reason “ai score 0.95”.
Ensure the assertions validate status, action, and reason so incorrect rounding
logic causes the tests to fail.
In `@src/routes/`(app)/channels/[id]/queue/+page.svelte:
- Line 44: Update the queue subtitle paragraph in the page markup to explicitly
state that the queue includes comments for which AI scoring is unavailable,
while retaining the existing borderline score-range description.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
Push a commit to this branch (recommended)
Create a new PR with the fixes
ℹ️ Review info⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 43547457-2ffb-4fea-a8fb-b57c07d1b100
📥 Commits
Reviewing files that changed from the base of the PR and between 3cc1326 and 22563e5.
1. Lets a YouTube channel owner connect their channel via Google OAuth (scope `youtube.force-ssl`, offline access). The AES-256-GCM-encrypted refresh token is stored in the DB.
2. Stores per-channel rules: keyword, regex (recheck-validated), or blocked-user, each mapping to `hold` | `reject` | `delete` | `ban`.
3. Exposes `GET /api/cron?secret=<CRON_SECRET>` which processes **one channel per run** (least-recently-run first) and **one page of ≤100 comments per run**, with a persisted checkpoint (continuation token + high-water mark) so bursts larger than one page are drained across runs **without skipping comments**. Per comment: rule match first, then OpenAI Moderation. Score ≥ 0.85 → auto-reject; 0.35–0.85 → human review queue; < 0.35 → approved. Rule hits execute their configured action. **Every enforcement action is recorded in the DB (`action_pending`) BEFORE the YouTube write, then confirmed after** — a crash mid-run is reconciled on the next run, never repeated blindly and never lost.
3. Exposes `GET /api/cron?secret=<CRON_SECRET>` which processes **one channel per run** (least-recently-run first) and **one page of ≤100 comments per run**, with a persisted checkpoint (continuation token + high-water mark) so bursts larger than one page are drained across runs **without skipping comments**. Per comment: rule match first, then OpenAI Moderation. Score ≥ 0.95 → auto-ban (reject + ban author); 0.76–0.94 → auto-reject; 0.51–0.75 → human review queue; ≤ 0.50 → approved. Rule hits execute their configured action. **Every enforcement action is recorded in the DB (`action_pending`) BEFORE the YouTube write, then confirmed after** — a crash mid-run is reconciled on the next run, never repeated blindly and never lost.
Implement DB-before-remote ordering for manual queue actions. The runtime calls YouTube before updating comments, and comments has no pendingAction field. Add the pending-action state, persist it before setModerationStatus or deleteComment, then finalize and audit after success.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@EXECUTION_PLAN_YouTube_Comment_Moderator.md` at line 42, Update manual queue
action handling to add a pending-action state to comments and persist
action_pending before any YouTube setModerationStatus or deleteComment call.
After the remote operation succeeds, finalize the comment state and write the
audit record; ensure pending actions remain recoverable after failures.
- Toxicity categories: `harassment`, `harassment/threatening`, `hate`, `hate/threatening`, `illicit`, `illicit/violent`, `self-harm`, `self-harm/intent`, `self-harm/instructions`, `sexual`, `sexual/minors`, `violence`, `violence/graphic`. Comment AI score = max of these thirteen. Spam is NOT an AI category; users catch spam with their own rules.
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Update the embedded moderation specification to use all thirteen categories.
The revised contract requires thirteen keys, but the TOXIC_CATEGORIES block at Lines 528-535 and the M1 fixture at Lines 923-926 still enumerate only six. The missing-key requirement at Lines 932-941 would therefore validate only the old set. Add illicit, illicit/violent, self-harm, self-harm/intent, self-harm/instructions, sexual, and sexual/minors to the embedded implementation and fixture. The supplied src/lib/server/moderation.test.ts context already exercises sexual.
Also applies to: 241-241, 922-922, 932-941
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@EXECUTION_PLAN_YouTube_Comment_Moderator.md` around lines 60 - 61, Update the
embedded moderation specification’s TOXIC_CATEGORIES block and the M1 fixture to
include all thirteen required toxicity keys, adding illicit, illicit/violent,
self-harm, self-harm/intent, self-harm/instructions, sexual, and sexual/minors
alongside the existing categories. Ensure the missing-key validation uses this
complete category set while preserving the existing score and threshold
behavior.
The changed cases cover only the lower transitions. Add raw 0.755 with status rejected, API action reject, and reason ai score 0.76. Add raw 0.945 with status rejected, API action ban, and reason ai score 0.95.
As per coding guidelines, tests must fail when the real logic is wrong.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lib/server/pipeline.test.ts` around lines 460 - 461, Add upper
rounding-boundary cases to the pipeline tests covering raw 0.755 as rejected
with API action reject and reason “ai score 0.76”, and raw 0.945 as rejected
with API action ban and reason “ai score 0.95”. Ensure the assertions validate
status, action, and reason so incorrect rounding logic causes the tests to fail.
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Mention AI-unavailable comments in the queue description.
src/lib/server/pipeline.ts places scoring failures in the pending queue. The execution plan also documents “or AI unavailable” at Line 1516. This subtitle names only the score range, so users can misinterpret AI-unavailable items.
Suggested copy
- <p class="page-sub">Borderline comments (AI score 0.51–0.75) waiting for your decision.</p>+ <p class="page-sub">Borderline comments (AI score 0.51–0.75, or AI unavailable) waiting for your decision.</p>
📝 Committable suggestion
‼️IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
<pclass="page-sub">Borderline comments (AI score 0.51–0.75) waiting for your decision.</p>
<pclass="page-sub">Borderline comments (AI score 0.51–0.75, or AI unavailable) waiting for your decision.</p>
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/routes/`(app)/channels/[id]/queue/+page.svelte at line 44, Update the
queue subtitle paragraph in the page markup to explicitly state that the queue
includes comments for which AI scoring is unavailable, while retaining the
existing borderline score-range description.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
codeant-ai
Bot
added
the
size:M
![amazon-q-developer[bot]](https://avatars.githubusercontent.com/in/1220912?s=60&v=4){kind=small}
![codacy-production[bot]](https://avatars.githubusercontent.com/in/56611?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Behavior
New AI score bands (
src/lib/server/pipeline.ts):pending, auditqueue)setModerationStatusrejected, no author ban)setModerationStatusrejected +banAuthor=true)Replaces the old bands (<0.35 approve / 0.35–0.50 queue / 0.51–0.85 delete / >0.85 ban). Note the AI no longer hard-deletes at any score — the mid band is now
reject(hidden, irreversible, same as the ban path minusbanAuthor).Tests (test-first)
npm run check,npm run buildall green.Docs/copy kept in sync
EXECUTION_PLAN_YouTube_Comment_Moderator.md: overview, fixed-thresholds list, pipeline snippet constants, M2 boundary row, queue-page copy, six → thirteen categories.PRODUCT.md: threshold bullet (also six → thirteen categories).DESIGN.md: example cue label ≥ 0.85 → ≥ 0.95.