bob
dc7c3f99ae
Brings up the pull-model scraper: the .NET C2 hands skin+wear jobs to Python nodriver workers that scrape cs.money and post results back, plus the supporting Core/EFCore data model, migrations, and docker-compose orchestration. IPRoyal proxying lets workers scale horizontally with a distinct residential exit IP each: every worker process mints its own sticky session at startup, and an in-process forwarding proxy injects the gateway auth so Chromium talks only to an auth-free localhost endpoint (zero CDP). On a Cloudflare challenge a worker rotates to a fresh session/IP and re-warms. Verified end-to-end against live IPRoyal: distinct US residential exits per worker and IP rotation on demand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
20 lines
1.3 KiB
C#
20 lines
1.3 KiB
C#
using BlueLaminate.Core.CsMoney;
|
|
|
|
namespace BlueLaminate.C2;
|
|
|
|
/// <summary>A unit of scrape work handed to a worker: one skin+wear, as a search.</summary>
|
|
/// <param name="JobId">Opaque id the worker echoes back when posting results.</param>
|
|
/// <param name="SkinId">Catalogue skin this job targets.</param>
|
|
/// <param name="ConditionId">Wear band (skin_conditions row), or null for a whole skin.</param>
|
|
/// <param name="Search">Free-text market search, e.g. "M4A4 Cyber Security ft".</param>
|
|
/// <param name="MaxPages">Safety cap on page fetches (60 items each). The worker
|
|
/// paginates by walking the float axis, so a skin+wear needs ceil(listings/60) pages.</param>
|
|
public sealed record ScrapeJobDto(string JobId, int SkinId, int? ConditionId, string Search, int MaxPages);
|
|
|
|
/// <summary>A worker's results for a claimed job: the listings it scraped.</summary>
|
|
/// <param name="Items">All sell-order items gathered across pages (raw cs.money shape).</param>
|
|
/// <param name="Pages">How many pages the worker fetched.</param>
|
|
/// <param name="StoppedReason">Why it stopped. "completed" = full sweep (authoritative);
|
|
/// anything else (fetch-cap / challenged / stuck-float-tie) is partial.</param>
|
|
public sealed record ScrapeResultDto(List<CsMoneyItem> Items, int Pages, string? StoppedReason);
|