Repository navigation
Fabricated citations and unsubstantiated claims in RESEARCH_REPORT.md #374
Description
I'm finding that the RESEARCH_REPORT.md has quite a few egregious errors that are tied to the claims being made in this project. I verified a sample of 8 citations below but the list is not exhaustive.
| # | Claimed Citation | Claimed Title & Authors | Actual Content at URL | Issue |
|---|---|---|---|---|
| 1 | arXiv:2403.12847 | "Optimizing Repository Documentation for Large Language Model Code Understanding" — Chen, Patel, Zhang (Stanford) | "Policy Bifurcation in Safe Reinforcement Learning" — Zou et al. | Wrong paper; unrelated field |
| 2 | arXiv:2404.12847 | Recycled for 3 different titles: "Type Inference Meets LLMs," "Rethinking Test Coverage Metrics," and "Automated Commit Message Generation" | "Banach Lie groupoid of partial isometries over restricted Grassmannian" — pure mathematics | Same fake ID used 3× for 3 fictitious papers |
| 3 | arXiv:2312.09334 | "From Docs to Code: Measuring README Information Density in AI Training" — Liu, Nguyen, Allamanis, Brockschmidt (Google DeepMind) | "ArchiGuesser — AI Art Architecture Educational Game" — Ploennigs, Berger, Carnein | Wrong paper. Allamanis and Brockschmidt are real DeepMind researchers — this fabricates authorship under their names |
| 4 | arXiv:2312.09573 | "TypeGPT: Teaching Language Models to Predict and Verify Type Annotations" — Wang, Nguyen, Alvarez, Schmidt | "Infrared anomalies in ultrathin Ti3C2Tx MXene films" — condensed matter physics | Wrong paper; not computer science |
| 5 | microsoft.com/…/semantic-commit-analysis-ai-code-review | "Semantic Commit Analysis: How Conventional Commits Enable Better AI Code Review" — Zhang, Williams, Thompson (Microsoft Research) | "The page you requested cannot be found" | Publication does not exist |
| 6 | anthropic.com/research/static-types-developer-productivity | "Static Types as Documentation" — Anthropic Research Team | 404 Not Found | Publication does not exist |
| 7 | anthropic.com/research/coverage-guided-prompting | "Coverage-Guided Prompting" — Anthropic Safety & Alignment Team | 404 Not Found | Publication does not exist |
| 8 | anthropic.com/research/commit-standards-codebase-health | "LLM-Assisted Development: The Impact of Commit Message Standards" — Anthropic Research Team | 404 Not Found | Publication does not exist |
Beyond the broken citations, the report also presents very precise statistics as research-backed findings — things like "34% fewer AI-generated bugs," "73% AI suggestion acceptance rate," "62% reduction in onboarding time" — none of which trace to a verifiable source. These numbers flow directly into the tool's scoring weights via default-weights.yaml, which derives its tier priorities from this report.
This isn't a cosmetic issue either. The research report is bundled as package data, has dedicated CLI commands to manage it (agentready research validate etc.), drives the scoring engine through its tier system, and surfaces in user-facing remediation output via a citations field. So the fabricated research directly influences what agentready tells users to do.
It's also worth noting that actual peer-reviewed work on this topic tells a different story. Gloaguen et al. (2025) found that context files tend to reduce task success rates and increase inference costs. Zhang et al. (2025) found that positive directives — which this report is full of — can actively hurt agent performance, and only negative constraints reliably help.
I'd ask that you retract or remove the research report, replace fabricated citations with real ones or drop the quantitative claims entirely, and stop attributing fabricated work to real researchers at Anthropic, Google DeepMind, and elsewhere. Research like this should not be spoofed when it impacts how people build software — the same standard applies here as in academia, where authors retract work for far less.
Activity
smith-xyz commentedon May 6, 2026
Hello, just checking on this. Tagging @bobbravo2 since you're listed as the only person in the org.
Happy to help clean up the citations if that's useful. The core concern is that fabricated research is influencing the scoring engine and user-facing output, which I think is worth addressing before wider adoption.
jwm4 commentedon May 6, 2026
@smith-xyz , I think this is generally addressed in #382 which changed both the research and the method by which it was produced to prevent this from continuing to happen. That change also included a lot of updates to the assessment logic that were motivated by the legit research. However, I think it's still possible that some of the guidance that still in the system is in someway motivated by the older research. I'm not sure if there's a way completely ensure that there's nothing left, but I think the situation is at least a lot better. Can you take a look at that PR and how it handled this topic and see if it has addressed your concern?
smith-xyz commentedon May 7, 2026
@jwm4 Appreciate the work in PR #382, v2.0.0 is a clear improvement. The citation issues from v1 are addressed and most of the high-impact references now resolve. Nice work.
I did a follow-up pass on v2.0.0 and wanted to flag a few things that still stood out, mainly because they flow into how the tool scores repos I believe:
Some citations don't quite support the specific claims they're attached to. For example, Terminal-Bench is cited under Section 5.1 (Test Execution) to help justify it as Tier 1 — the 52.8% → 66.5% numbers are real, but the gain came from harness engineering (system prompts, middleware, loop detection), not from having a single-command test runner. The DORA 2025 citation in that same section says "AI amplifies existing good practices. TDD is more critical than ever." — but that phrasing is from a Google Cloud marketing page summarizing the DORA report, not a DORA finding itself. The actual report is more nuanced about AI adoption increasing instability. The Salesforce Cursor article is also cited there to justify test execution as Tier 1, but the article is about using Cursor to write unit tests for a legacy coverage mandate — it doesn't speak to test execution being important for agent effectiveness. These might just need tighter mapping between the source and the claim.
The mismatches make me wonder how closely the sources are being reviewed before they're wired into the scoring weights.
The report title says "Comprehensive Research" with "55+ authoritative sources." There's good stuff in there, the arXiv papers are solid, and plenty of the blog posts and engineering articles have genuinely useful insights. I just think the label sets an expectation of rigor that doesn't quite match the mix. Something like "Curated Best Practices" would better describe what's here and still give the citations their due without overpromising.
None of this is meant to dismiss the project. The direction is right and v2 is genuinely better. I think the thing I'd really advocate for is making sure the people behind this are deeply engaging with the content they're citing, not just surfacing it. Understanding what a paper actually found and why it matters is what separates a credible foundation from a collection of links. That's especially important here because these citations shape the scoring weights that evaluate other people's work. Happy to help clean any of this up if useful.
jwm4 commentedon May 12, 2026
@natifridman , can you take a look at @smith-xyz 's comments above? These seem like reasonable points to me, so I think there is probably more room for improvement here. What do you think? cc: @kami619 too
natifridman commentedon May 13, 2026
PR #382 was a big one, so it definitely requires a follow-up.
@smith-xyz Thanks for that. I created #429 for more updates in the research
fix: correct 3 citation-claim mismatches in research report (v2.0.1)
github-actions commentedon Jul 13, 2026
👋 This issue has been inactive for 60 days and will be closed in 30 days if there is no further activity.
If this issue is still relevant, please:
stalelabelpinnedlabel to prevent future stale markingThank you for your contributions to AgentReady!
github-actions commentedon Aug 12, 2026
🔒 This issue has been automatically closed due to 90 days of inactivity.
If you believe this issue should remain open, please reopen it and provide an update.