Add cs.money worker stack with per-worker IPRoyal residential proxy
Brings up the pull-model scraper: the .NET C2 hands skin+wear jobs to Python nodriver workers that scrape cs.money and post results back, plus the supporting Core/EFCore data model, migrations, and docker-compose orchestration. IPRoyal proxying lets workers scale horizontally with a distinct residential exit IP each: every worker process mints its own sticky session at startup, and an in-process forwarding proxy injects the gateway auth so Chromium talks only to an auth-free localhost endpoint (zero CDP). On a Cloudflare challenge a worker rotates to a fresh session/IP and re-warms. Verified end-to-end against live IPRoyal: distinct US residential exits per worker and IP rotation on demand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
55
DOCKER.md
Normal file
55
DOCKER.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# Containerized startup (C2 + worker)
|
||||
|
||||
One command brings up the cs.money **C2** (control plane) and a **worker**. Postgres
|
||||
runs independently on the host; the C2 connects to it and auto-applies EF migrations
|
||||
on boot.
|
||||
|
||||
```powershell
|
||||
docker-compose up --build
|
||||
```
|
||||
|
||||
- **C2** → http://localhost:5080 (`/health`, `/jobs/*`, `/market/*`)
|
||||
- **Worker noVNC** → http://localhost:6080/vnc.html — watch the browser, and solve a
|
||||
Cloudflare challenge by hand if one appears.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Host Postgres reachable from containers.** The C2 connects via
|
||||
`host.docker.internal`. Postgres must (a) listen on the Docker-facing interface
|
||||
(`listen_addresses = '*'` in `postgresql.conf`) and (b) allow the container subnet
|
||||
in `pg_hba.conf`. The DB (`skintracker`) should already have the schema, but the
|
||||
C2 also runs `Database.Migrate()` at startup as a safety net.
|
||||
2. **A real exit IP for the worker.** A bare datacenter/container IP gets challenged
|
||||
hard by Cloudflare. Set `PROXY` to a residential exit (see below).
|
||||
|
||||
## Configuration (env vars / a `.env` file next to docker-compose.yml)
|
||||
|
||||
| Var | Default | Purpose |
|
||||
|-----|---------|---------|
|
||||
| `SKINTRACKER_CONN` | `Host=host.docker.internal;Port=5432;Database=skintracker;Username=postgres` | C2 → Postgres connection string |
|
||||
| `WORKER_TOKEN` | `dev-worker-token` | Shared secret; C2 and worker must match |
|
||||
| `PROXY` | _(none)_ | Worker proxy `host:port` (auth-free) |
|
||||
| `SOLVE_SECONDS` | `45` | Time the worker waits for you to clear Cloudflare |
|
||||
| `MAX_PAGES_PER_JOB` | `20` | Cap on offset pages per skin+wear job |
|
||||
| `LOAD_IMAGES` | _(off)_ | `1` re-enables image loading (debugging) |
|
||||
|
||||
## Scaling workers
|
||||
|
||||
```powershell
|
||||
docker-compose up --build --scale worker=3
|
||||
```
|
||||
|
||||
Remove the worker `ports:` mapping first — multiple workers can't share host port 6080
|
||||
for noVNC. (Each gets the display internally; expose per-worker only if you need to
|
||||
watch a specific one.)
|
||||
|
||||
## Notes / known gaps
|
||||
|
||||
- **IPRoyal auth:** `PROXY` is passed to Chromium as `--proxy-server`, which ignores
|
||||
`user:pass`. For credentialed IPRoyal either IP-whitelist the worker's egress IP, or
|
||||
add a small forwarding-proxy sidecar that injects the auth (the .NET
|
||||
`LocalForwardingProxy` does this for the CSFloat path; a worker-side equivalent is a
|
||||
follow-up).
|
||||
- **Unattended Cloudflare:** the worker leans on nodriver + a residential IP clearing
|
||||
CF automatically. When it can't, use the noVNC tab to solve it once; the warmed
|
||||
profile then carries the clearance.
|
||||
Reference in New Issue
Block a user