[wojciech-dron]

PR Type

Enhancement, Bug fix

---

Description

• Add configurable scraping page limits

  • Expose limits in scrape UI
  • Enforce limits across list scrapers

• Prevent concurrent scraping tracking issues

  • Create scoped mediators per command
  • Reduce scheduler concurrency to one
  • Deduplicate companies and offers

• Refine AI summary and CV workflows

  • Skip hidden offers for AI summary
  • Warn when AI config missing
  • Sanitize exported CV filenames

• Normalize scraper naming and queries

  • Rename DetailsScraperBaseHandler
  • Filter detail scraping to linked companies

---

Diagram Walkthrough

flowchart LR
  cfg["`PagesLimit` source configuration"]
  ui["Scrape page controls"]
  lists["Paginated list scrapers"]
  scope["Scoped mediator execution"]
  db["Safer DbContext interactions"]
  ai["AI summary only for visible offers"]
  cfg -- "edited in" --> ui
  ui -- "drives" --> lists
  lists -- "executed through" --> scope
  scope -- "reduces conflicts in" --> db
  scope -- "marks offers for" --> ai

Loading

File Walkthrough

	Relevant files
Configuration changes	2 files JobOffer.cs Tighten `Location` column length configuration                      +0/-4      Setup.cs Limit scheduler concurrency to one worker                                +1/-1
Enhancement	9 files ScraperConfig.cs Add optional per-source pages limit                                            +1/-0      AiSummaryJob.cs Unify generated CV naming format                                                  +1/-1      IndeedListScraper.cs Start pagination at one with limit                                              +5/-1      JjitListScraper.cs Start pagination at one with limit                                              +4/-1      NoFluffJobsListScraper.cs Start pagination at one with limit                                              +4/-1      OlxListScraper.cs Start pagination at one with limit                                              +4/-1      PracujPlListScraper.cs Start pagination at one with limit                                              +4/-1      RocketJobsListScraper.cs Start pagination at one with limit                                              +4/-1      ScrapePage.razor Add URL open button and pages limit                                            +25/-8
Bug fix	5 files CvEditPage.razor.cs Sanitize exported CV file names                                                    +3/-3      DetailsScraperBaseHandler.cs Rename base handler and filter companies                                  +6/-6      ListScraperBaseHandler.cs Deduplicate companies and offers before insert                      +3/-0      ScrapperBaseHandler.cs Remove shared retry policy from base                                          +0/-5      ScrapeHandler.cs Scope mediators and skip hidden summaries                                +6/-1
Error handling	1 files PrepareCvWithAiPage.razor.cs Handle missing AI config safely                                                    +14/-2
Miscellaneous	4 files IndeedDetailsScraper.cs Align handler with renamed base scraper                                    +1/-1      JjitDetailsScraper.cs Align handler with renamed base scraper                                    +1/-1      NoFluffJobsDetailsScraper.cs Align renamed base and trim logging                                            +1/-3      RocketJobsDetailsScraper.cs Align handler with renamed base scraper                                    +1/-1

---

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 No relevant tests

🔒 Security concerns Reverse tabnabbing: the new external Open link in JobScraper.Web/Features/JobOffers/Scrape/ScrapePage.razor uses target="_blank" without rel="noopener noreferrer". A malicious destination page could access window.opener and potentially redirect or manipulate the original application tab.

⚡ Recommended focus areas for review Off-by-one The new page counter initialization and limit check appear to skip scraping the first loaded Indeed page when PagesLimit is 1, and also log/page-number subsequent work as if scraping starts from page 2. This should be validated because it can silently return zero results for the most restrictive limit. var pageNumber = 1; await SaveScreenshot(page, $"indeed/list/{fetchDate}/{pageNumber}.png"); await SavePage(page, $"indeed/list/{fetchDate}/{pageNumber}.html"); while (true) { pageNumber++; if (sourceConfig.PagesLimit.HasValue && pageNumber > sourceConfig.PagesLimit.Value) break; Missing Retry Detail scraping now calls the scraper directly without the earlier retry wrapper. Transient Playwright/network failures may now cause more offers to remain unscripted or fail the whole command more often, so the new failure behavior should be reviewed carefully. await ScrapeJobDetails(jobOffer); Over-filtering Detail scraping now excludes all user offers whose linked company entity is null. If company linking is incomplete or delayed, valid offers may never get their details scraped and can remain stuck in a pending status. .Include(j => j.Details.Company) .Where(j => j.Details.Company != null) .Where(j => j.Details.Origin == DataOrigin)

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Remove blocking company filter Filtering on j.Details.Company != null will skip details scraping for offers whose company relation is not populated yet. That can leave valid offers permanently stuck in ToScrape, so avoid excluding them here. JobScraper.Web/Features/JobOffers/Scrape/Logic/Common/DetailsScraperBaseHandler.cs [34-37] var userOffers = await DbContext.UserOffers .Include(j => j.Details.Company) - .Where(j => j.Details.Company != null) - .Where(j => j.Details.Origin == DataOrigin) + .Where(j => j.Details.Origin == DataOrigin) Suggestion importance[1-10]: 8 __ Why: The added filter Where(j => j.Details.Company != null) can prevent valid offers from being processed by DetailsScraperBaseHandler, leaving them stuck before details scraping. Removing that condition is a meaningful correctness fix, and the suggested improved_code matches the changed hunk.	Medium
	Restore transient failure retry Removing the retry around ScrapeJobDetails makes a single transient browser or network failure abort scraping for that offer immediately. Restore a retry here so temporary Playwright issues do not cause unnecessary failed scrapes. JobScraper.Web/Features/JobOffers/Scrape/Logic/Common/DetailsScraperBaseHandler.cs [53] -await ScrapeJobDetails(jobOffer); +for (var attempt = 0; attempt < 2; attempt++) +{ + try + { + await ScrapeJobDetails(jobOffer); + break; + } + catch (Exception) when (attempt == 0) + { + Logger.LogWarning("Retrying details scrape for {OfferUrl}", jobOffer.OfferUrl); + } +} Suggestion importance[1-10]: 6 __ Why: Reintroducing a retry around ScrapeJobDetails would improve resilience against transient scraper or browser failures after the PR removed RetryPolicy. This is a useful reliability improvement, but it is still an error-handling suggestion rather than a clearly proven regression in the new code.	Low