Add cs.money worker stack with per-worker IPRoyal residential proxy
Brings up the pull-model scraper: the .NET C2 hands skin+wear jobs to Python nodriver workers that scrape cs.money and post results back, plus the supporting Core/EFCore data model, migrations, and docker-compose orchestration. IPRoyal proxying lets workers scale horizontally with a distinct residential exit IP each: every worker process mints its own sticky session at startup, and an in-process forwarding proxy injects the gateway auth so Chromium talks only to an auth-free localhost endpoint (zero CDP). On a Cloudflare challenge a worker rotates to a fresh session/IP and re-warms. Verified end-to-end against live IPRoyal: distinct US residential exits per worker and IP rotation on demand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
44
db/06_backfill_skin_condition_swept.sql
Normal file
44
db/06_backfill_skin_condition_swept.sql
Normal file
@@ -0,0 +1,44 @@
|
||||
-- ============================================================
|
||||
-- CS2 Skin Tracker — backfill skin_conditions.listings_swept_at
|
||||
-- Run against the skintracker database as the app role, ONCE,
|
||||
-- after the AddSkinConditionListingsSweptAt migration is applied
|
||||
-- and 05_fill_skin_conditions.sql has populated the wear bands.
|
||||
-- Idempotent: re-running only touches still-null bands.
|
||||
--
|
||||
-- Why: the catalogue sweep used to page each skin to completion
|
||||
-- as a single unit, so a non-null skins.listings_swept_at means
|
||||
-- EVERY wear of that skin was covered at that time. The sweep now
|
||||
-- checkpoints per wear band (skin_conditions.listings_swept_at).
|
||||
-- Without this backfill, every band of an already-swept skin would
|
||||
-- look never-swept and jump to the front of the queue, needlessly
|
||||
-- re-sweeping skins that are already current. Inheriting the skin's
|
||||
-- timestamp marks those bands as covered so the sweep moves on.
|
||||
--
|
||||
-- Only fills bands that are still null, so bands already swept under
|
||||
-- the new per-band logic keep their (newer) timestamp.
|
||||
-- ============================================================
|
||||
|
||||
SET search_path = skintracker;
|
||||
|
||||
UPDATE skin_conditions sc
|
||||
SET listings_swept_at = s.listings_swept_at
|
||||
FROM skins s
|
||||
WHERE sc.skin_id = s.id
|
||||
AND s.listings_swept_at IS NOT NULL -- skin was fully swept under the old per-skin logic
|
||||
AND sc.listings_swept_at IS NULL; -- don't overwrite bands already swept per-band
|
||||
|
||||
-- ------------------------------------------------------------
|
||||
-- Sanity checks (optional)
|
||||
-- ------------------------------------------------------------
|
||||
-- Bands backfilled vs still never-swept:
|
||||
-- SELECT
|
||||
-- count(*) FILTER (WHERE listings_swept_at IS NOT NULL) AS swept,
|
||||
-- count(*) FILTER (WHERE listings_swept_at IS NULL) AS never_swept
|
||||
-- FROM skin_conditions;
|
||||
--
|
||||
-- A previously-swept skin should now have all its bands stamped:
|
||||
-- SELECT s.name, sc.condition, sc.listings_swept_at
|
||||
-- FROM skin_conditions sc JOIN skins s ON s.id = sc.skin_id
|
||||
-- WHERE s.listings_swept_at IS NOT NULL
|
||||
-- ORDER BY s.name, sc.min_float
|
||||
-- LIMIT 20;
|
||||
Reference in New Issue
Block a user