bob
dc7c3f99ae
Brings up the pull-model scraper: the .NET C2 hands skin+wear jobs to Python nodriver workers that scrape cs.money and post results back, plus the supporting Core/EFCore data model, migrations, and docker-compose orchestration. IPRoyal proxying lets workers scale horizontally with a distinct residential exit IP each: every worker process mints its own sticky session at startup, and an in-process forwarding proxy injects the gateway auth so Chromium talks only to an auth-free localhost endpoint (zero CDP). On a Cloudflare challenge a worker rotates to a fresh session/IP and re-warms. Verified end-to-end against live IPRoyal: distinct US residential exits per worker and IP rotation on demand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2.5 KiB
2.5 KiB
Containerized startup (C2 + worker)
One command brings up the cs.money C2 (control plane) and a worker. Postgres runs independently on the host; the C2 connects to it and auto-applies EF migrations on boot.
docker-compose up --build
- C2 → http://localhost:5080 (
/health,/jobs/*,/market/*) - Worker noVNC → http://localhost:6080/vnc.html — watch the browser, and solve a Cloudflare challenge by hand if one appears.
Prerequisites
- Host Postgres reachable from containers. The C2 connects via
host.docker.internal. Postgres must (a) listen on the Docker-facing interface (listen_addresses = '*'inpostgresql.conf) and (b) allow the container subnet inpg_hba.conf. The DB (skintracker) should already have the schema, but the C2 also runsDatabase.Migrate()at startup as a safety net. - A real exit IP for the worker. A bare datacenter/container IP gets challenged
hard by Cloudflare. Set
PROXYto a residential exit (see below).
Configuration (env vars / a .env file next to docker-compose.yml)
| Var | Default | Purpose |
|---|---|---|
SKINTRACKER_CONN |
Host=host.docker.internal;Port=5432;Database=skintracker;Username=postgres |
C2 → Postgres connection string |
WORKER_TOKEN |
dev-worker-token |
Shared secret; C2 and worker must match |
PROXY |
(none) | Worker proxy host:port (auth-free) |
SOLVE_SECONDS |
45 |
Time the worker waits for you to clear Cloudflare |
MAX_PAGES_PER_JOB |
20 |
Cap on offset pages per skin+wear job |
LOAD_IMAGES |
(off) | 1 re-enables image loading (debugging) |
Scaling workers
docker-compose up --build --scale worker=3
Remove the worker ports: mapping first — multiple workers can't share host port 6080
for noVNC. (Each gets the display internally; expose per-worker only if you need to
watch a specific one.)
Notes / known gaps
- IPRoyal auth:
PROXYis passed to Chromium as--proxy-server, which ignoresuser:pass. For credentialed IPRoyal either IP-whitelist the worker's egress IP, or add a small forwarding-proxy sidecar that injects the auth (the .NETLocalForwardingProxydoes this for the CSFloat path; a worker-side equivalent is a follow-up). - Unattended Cloudflare: the worker leans on nodriver + a residential IP clearing CF automatically. When it can't, use the noVNC tab to solve it once; the warmed profile then carries the clearance.