bob
dc7c3f99ae
Brings up the pull-model scraper: the .NET C2 hands skin+wear jobs to Python nodriver workers that scrape cs.money and post results back, plus the supporting Core/EFCore data model, migrations, and docker-compose orchestration. IPRoyal proxying lets workers scale horizontally with a distinct residential exit IP each: every worker process mints its own sticky session at startup, and an in-process forwarding proxy injects the gateway auth so Chromium talks only to an auth-free localhost endpoint (zero CDP). On a Cloudflare challenge a worker rotates to a fresh session/IP and re-warms. Verified end-to-end against live IPRoyal: distinct US residential exits per worker and IP rotation on demand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
45 lines
2.0 KiB
SQL
45 lines
2.0 KiB
SQL
-- ============================================================
|
|
-- CS2 Skin Tracker — backfill skin_conditions.listings_swept_at
|
|
-- Run against the skintracker database as the app role, ONCE,
|
|
-- after the AddSkinConditionListingsSweptAt migration is applied
|
|
-- and 05_fill_skin_conditions.sql has populated the wear bands.
|
|
-- Idempotent: re-running only touches still-null bands.
|
|
--
|
|
-- Why: the catalogue sweep used to page each skin to completion
|
|
-- as a single unit, so a non-null skins.listings_swept_at means
|
|
-- EVERY wear of that skin was covered at that time. The sweep now
|
|
-- checkpoints per wear band (skin_conditions.listings_swept_at).
|
|
-- Without this backfill, every band of an already-swept skin would
|
|
-- look never-swept and jump to the front of the queue, needlessly
|
|
-- re-sweeping skins that are already current. Inheriting the skin's
|
|
-- timestamp marks those bands as covered so the sweep moves on.
|
|
--
|
|
-- Only fills bands that are still null, so bands already swept under
|
|
-- the new per-band logic keep their (newer) timestamp.
|
|
-- ============================================================
|
|
|
|
SET search_path = skintracker;
|
|
|
|
UPDATE skin_conditions sc
|
|
SET listings_swept_at = s.listings_swept_at
|
|
FROM skins s
|
|
WHERE sc.skin_id = s.id
|
|
AND s.listings_swept_at IS NOT NULL -- skin was fully swept under the old per-skin logic
|
|
AND sc.listings_swept_at IS NULL; -- don't overwrite bands already swept per-band
|
|
|
|
-- ------------------------------------------------------------
|
|
-- Sanity checks (optional)
|
|
-- ------------------------------------------------------------
|
|
-- Bands backfilled vs still never-swept:
|
|
-- SELECT
|
|
-- count(*) FILTER (WHERE listings_swept_at IS NOT NULL) AS swept,
|
|
-- count(*) FILTER (WHERE listings_swept_at IS NULL) AS never_swept
|
|
-- FROM skin_conditions;
|
|
--
|
|
-- A previously-swept skin should now have all its bands stamped:
|
|
-- SELECT s.name, sc.condition, sc.listings_swept_at
|
|
-- FROM skin_conditions sc JOIN skins s ON s.id = sc.skin_id
|
|
-- WHERE s.listings_swept_at IS NOT NULL
|
|
-- ORDER BY s.name, sc.min_float
|
|
-- LIMIT 20;
|