Add cs.money worker stack with per-worker IPRoyal residential proxy
Brings up the pull-model scraper: the .NET C2 hands skin+wear jobs to Python nodriver workers that scrape cs.money and post results back, plus the supporting Core/EFCore data model, migrations, and docker-compose orchestration. IPRoyal proxying lets workers scale horizontally with a distinct residential exit IP each: every worker process mints its own sticky session at startup, and an in-process forwarding proxy injects the gateway auth so Chromium talks only to an auth-free localhost endpoint (zero CDP). On a Cloudflare challenge a worker rotates to a fresh session/IP and re-warms. Verified end-to-end against live IPRoyal: distinct US residential exits per worker and IP rotation on demand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
49
docker-compose.yml
Normal file
49
docker-compose.yml
Normal file
@@ -0,0 +1,49 @@
|
||||
# One-command startup for the cs.money scraper control plane + worker.
|
||||
# Postgres is external (runs independently on the host); the C2 connects to it via
|
||||
# host.docker.internal and auto-applies EF migrations on boot.
|
||||
#
|
||||
# docker compose up --build
|
||||
#
|
||||
# Scale workers (drop the worker `ports:` first — noVNC can't share one host port):
|
||||
# docker compose up --build --scale worker=10
|
||||
# Each worker mints its own IPRoyal sticky session at startup, so every replica gets a
|
||||
# distinct residential exit IP. Set IPROYAL_USERNAME / IPROYAL_PASSWORD (e.g. in a .env
|
||||
# file next to this compose file) to turn the proxy on.
|
||||
services:
|
||||
c2:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: BlueLaminate/BlueLaminate.C2/Dockerfile
|
||||
environment:
|
||||
# Point at the host's Postgres. Override the whole string for auth/host changes.
|
||||
ConnectionStrings__SkinTracker: ${SKINTRACKER_CONN:-Host=host.docker.internal;Port=5432;Database=skintracker;Username=postgres}
|
||||
WorkerToken: ${WORKER_TOKEN:-dev-worker-token}
|
||||
MaxPagesPerJob: ${MAX_PAGES_PER_JOB:-60}
|
||||
ports:
|
||||
- "5080:5080"
|
||||
extra_hosts:
|
||||
# Lets the container resolve the host's Postgres on Linux too (no-op on Desktop).
|
||||
- "host.docker.internal:host-gateway"
|
||||
restart: unless-stopped
|
||||
|
||||
worker:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: worker/Dockerfile
|
||||
environment:
|
||||
C2_URL: http://c2:5080
|
||||
WORKER_TOKEN: ${WORKER_TOKEN:-dev-worker-token}
|
||||
# IPRoyal residential proxy: each replica self-assigns a unique sticky session
|
||||
# (= unique exit IP). Auth is injected by an in-process forwarder, so no sidecar.
|
||||
IPROYAL_USERNAME: ${IPROYAL_USERNAME:-}
|
||||
IPROYAL_PASSWORD: ${IPROYAL_PASSWORD:-}
|
||||
IPROYAL_COUNTRY: ${IPROYAL_COUNTRY:-us}
|
||||
IPROYAL_LIFETIME_MIN: ${IPROYAL_LIFETIME_MIN:-60}
|
||||
PROXY: ${PROXY:-} # auth-free host:port fallback (used only when IPRoyal creds are unset)
|
||||
SOLVE_SECONDS: ${SOLVE_SECONDS:-45}
|
||||
LOAD_IMAGES: ${LOAD_IMAGES:-} # set to 1 to re-enable images (debugging)
|
||||
depends_on:
|
||||
- c2
|
||||
ports:
|
||||
- "6080:6080" # noVNC: http://localhost:6080/vnc.html
|
||||
restart: unless-stopped
|
||||
Reference in New Issue
Block a user