bob
dc7c3f99ae
Brings up the pull-model scraper: the .NET C2 hands skin+wear jobs to Python nodriver workers that scrape cs.money and post results back, plus the supporting Core/EFCore data model, migrations, and docker-compose orchestration. IPRoyal proxying lets workers scale horizontally with a distinct residential exit IP each: every worker process mints its own sticky session at startup, and an in-process forwarding proxy injects the gateway auth so Chromium talks only to an auth-free localhost endpoint (zero CDP). On a Cloudflare challenge a worker rotates to a fresh session/IP and re-warms. Verified end-to-end against live IPRoyal: distinct US residential exits per worker and IP rotation on demand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
56 lines
2.5 KiB
Markdown
56 lines
2.5 KiB
Markdown
# Containerized startup (C2 + worker)
|
|
|
|
One command brings up the cs.money **C2** (control plane) and a **worker**. Postgres
|
|
runs independently on the host; the C2 connects to it and auto-applies EF migrations
|
|
on boot.
|
|
|
|
```powershell
|
|
docker-compose up --build
|
|
```
|
|
|
|
- **C2** → http://localhost:5080 (`/health`, `/jobs/*`, `/market/*`)
|
|
- **Worker noVNC** → http://localhost:6080/vnc.html — watch the browser, and solve a
|
|
Cloudflare challenge by hand if one appears.
|
|
|
|
## Prerequisites
|
|
|
|
1. **Host Postgres reachable from containers.** The C2 connects via
|
|
`host.docker.internal`. Postgres must (a) listen on the Docker-facing interface
|
|
(`listen_addresses = '*'` in `postgresql.conf`) and (b) allow the container subnet
|
|
in `pg_hba.conf`. The DB (`skintracker`) should already have the schema, but the
|
|
C2 also runs `Database.Migrate()` at startup as a safety net.
|
|
2. **A real exit IP for the worker.** A bare datacenter/container IP gets challenged
|
|
hard by Cloudflare. Set `PROXY` to a residential exit (see below).
|
|
|
|
## Configuration (env vars / a `.env` file next to docker-compose.yml)
|
|
|
|
| Var | Default | Purpose |
|
|
|-----|---------|---------|
|
|
| `SKINTRACKER_CONN` | `Host=host.docker.internal;Port=5432;Database=skintracker;Username=postgres` | C2 → Postgres connection string |
|
|
| `WORKER_TOKEN` | `dev-worker-token` | Shared secret; C2 and worker must match |
|
|
| `PROXY` | _(none)_ | Worker proxy `host:port` (auth-free) |
|
|
| `SOLVE_SECONDS` | `45` | Time the worker waits for you to clear Cloudflare |
|
|
| `MAX_PAGES_PER_JOB` | `20` | Cap on offset pages per skin+wear job |
|
|
| `LOAD_IMAGES` | _(off)_ | `1` re-enables image loading (debugging) |
|
|
|
|
## Scaling workers
|
|
|
|
```powershell
|
|
docker-compose up --build --scale worker=3
|
|
```
|
|
|
|
Remove the worker `ports:` mapping first — multiple workers can't share host port 6080
|
|
for noVNC. (Each gets the display internally; expose per-worker only if you need to
|
|
watch a specific one.)
|
|
|
|
## Notes / known gaps
|
|
|
|
- **IPRoyal auth:** `PROXY` is passed to Chromium as `--proxy-server`, which ignores
|
|
`user:pass`. For credentialed IPRoyal either IP-whitelist the worker's egress IP, or
|
|
add a small forwarding-proxy sidecar that injects the auth (the .NET
|
|
`LocalForwardingProxy` does this for the CSFloat path; a worker-side equivalent is a
|
|
follow-up).
|
|
- **Unattended Cloudflare:** the worker leans on nodriver + a residential IP clearing
|
|
CF automatically. When it can't, use the noVNC tab to solve it once; the warmed
|
|
profile then carries the clearance.
|