Files

bob 083daa7022 add readme

2026-06-02 13:42:14 -05:00

9.6 KiB

Raw Blame History

BlueLaminate

A CS2 skin-market intelligence platform — ingests live marketplace listings across CSFloat, cs.money, and skin.land, tracks prices and float values in Postgres, and mines the catalogue for profitable trade-up contracts.

✨ What it does

Capability	Where	Notes
CSFloat ingestion	`BlueLaminate.Scraper` + CLI	Official API. Catalogue-driven and global incremental sweeps, paced off rate-limit headers.
cs.money scraping	`worker/csmoney_worker.py`	Cloudflare-walled. A Python `nodriver` worker fleet driven by the C2 (pull model).
skin.land scraping	`worker/skinland_worker.py`	Nuxt SSR behind Cloudflare. Same worker plumbing, market-specific scrape.
Cross-market presence	`MarketPresenceService`	"Where is this skin/instance listed, and at what price?" across all sources.
Trade-up engine	`BlueLaminate.Core/Tradeups`	Finds profitable 10-input contracts from live listings — exact float math, guaranteed vs. expected-value ranking, StatTrak universes, interactive TUI.
Price history & dedup	`BlueLaminate.EFCore`	Postgres data model with per-source checkpoints, `SkinInstance` fingerprinting, and `price_history`.
Observability	`monitoring/`	Standalone Grafana LGTM stack (Loki / Grafana / Tempo / Mimir) wired via OTLP.

🏗️ Architecture

                       ┌──────────────────────────────────────────┐
                       │              PostgreSQL                    │
                       │   (skintracker schema, least-priv role)    │
                       └───────────────▲────────────────▲──────────┘
                                       │                │
           ┌───────────────────────────┘                └────────────────┐
           │                                                              │
  ┌────────┴─────────┐                                          ┌─────────┴──────────┐
  │  BlueLaminate.Cli │  CSFloat (official API)                 │  BlueLaminate.C2    │
  │  sync / sweep /   │─────────────────────────────────────►  │  (ASP.NET control   │
  │  find-tradeups    │                                          │   plane)            │
  └────────┬──────────┘                                          └─────────▲──────────┘
           │                                                               │ jobs / results
           │ shares                                                        │ (X-Worker-Token)
           ▼                                                     ┌─────────┴──────────┐
  ┌────────────────────┐                                         │  Python workers     │
  │  BlueLaminate.Core  │  ◄── composition root, all shared       │  nodriver + IPRoyal │
  │  (ingest, tradeups, │      logic (AddBlueLaminateCore)         │  cs.money / skin.land│
  │   market presence)  │                                         └─────────────────────┘
  └────────────────────┘

BlueLaminate.Core — the single composition root. Every frontend wires itself with AddBlueLaminateCore(configuration) and reuses the same ingest/tradeup/presence services. No duplicated logic.
BlueLaminate.Scraper — typed CSFloat API client + the CSGO-API skin catalogue client.
BlueLaminate.EFCore — Postgres data layer and EF Core migrations (auto-applied on C2 boot).
BlueLaminate.Cli — System.CommandLine operator tools.
BlueLaminate.C2 — hands cs.money/skin.land scrape jobs to the Python worker fleet and ingests their results.
worker/ — Python nodriver workers. .NET Selenium gets insta-challenged by Cloudflare; nodriver drives Chromium directly over CDP and passes. Each replica mints its own IPRoyal sticky residential exit IP.

🚀 Quick start

Prerequisites

.NET 10 SDK
PostgreSQL with a skintracker database
Python 3.x + Chrome/Edge (only if you run the cs.money / skin.land workers)
Docker (optional — for the one-command C2 + worker stack)
A CSFloat API key for the CSFloat paths (CSFLOAT_API_KEY)

1. Database

Run the schema/role hardening once as a superuser (edit the password placeholders first):

psql -U postgres -d skintracker -f db/01_schema_and_roles.sql

The C2 applies EF migrations automatically on boot (AutoMigrate=true); for the CLI alone, run dotnet ef database update --project BlueLaminate\BlueLaminate.EFCore.

2. Build

dotnet build BlueLaminate/BlueLaminate.slnx

3. Run the CLI

# One-time: pull the CS2 skin catalogue (throttled to once a month)
dotnet run --project BlueLaminate\BlueLaminate.Cli -- sync-skins

# Ingest live CSFloat listings (reads CSFLOAT_API_KEY)
$env:CSFLOAT_API_KEY="your-key"
dotnet run --project BlueLaminate\BlueLaminate.Cli -- sweep-catalog

# Mine profitable trade-up contracts (interactive TUI)
dotnet run --project BlueLaminate\BlueLaminate.Cli -- find-tradeups --min-profit 2.00

CLI commands

Command	Purpose
`sync-skins`	Load the CS2 skin catalogue from the CSGO-API dataset and upsert it.
`fetch-listings`	Fetch active CSFloat listings for one skin and print them (no DB write).
`sweep-listings`	Global incremental sweep of active CSFloat listings into the DB.
`sweep-catalog`	Catalogue-driven sweep: query each skin by `def_index`+`paint_index`, split by wear band.
`find-tradeups`	Find profitable 10-input trade-up contracts from live listings, ranked best-first.

Add --help to any command for its full option set.

4. Run the marketplace workers (cs.money / skin.land)

The browser/Cloudflare scrapers run as Python workers that pull jobs from the C2.

Containerized (one command):

# Defaults: 1 cs.money worker + 1 skin.land worker. Postgres is external (host).
docker compose up --build

C2 → http://localhost:5080 (/health, /jobs/*, /market/*)
Worker noVNC → http://localhost:6080/vnc.html (watch the browser; solve a Cloudflare challenge by hand if one appears)

Set the worker mix and proxy via env / a .env file next to docker-compose.yml:

$env:CSMONEY_WORKERS=0; $env:SKINLAND_WORKERS=3
$env:IPROYAL_USERNAME="..."; $env:IPROYAL_PASSWORD="..."
docker compose up --build

Local (without Docker):

# terminal 1 — the C2 (from repo root)
dotnet run --project BlueLaminate\BlueLaminate.C2     # http://localhost:5080

# terminal 2 — a worker
cd worker
py -m venv .venv; .venv\Scripts\Activate.ps1
pip install -r requirements.txt
$env:WORKER_TOKEN="dev-worker-token"                  # must match the C2's WorkerToken
python csmoney_worker.py                               # or skinland_worker.py

See DOCKER.md and worker/README.md for the full configuration matrix, scaling notes, and proxy details.

⚙️ Configuration

Key settings (env vars override appsettings.json):

Var	Purpose
`ConnectionStrings__SkinTracker` / `SKINTRACKER_CONN`	Postgres connection string.
`CSFLOAT_API_KEY`	CSFloat official API key (CLI sweep/fetch).
`WorkerToken` / `WORKER_TOKEN`	Shared secret between C2 and workers.
`MaxPagesPerJob`, `MinResweepHours`	C2 sweep budget / re-sweep floor (metered-proxy cost control).
`IPROYAL_USERNAME` / `IPROYAL_PASSWORD` / `IPROYAL_COUNTRY`	Residential proxy for workers.

🔒 Secrets (API keys, proxy credentials, DB passwords) belong in environment variables, a gitignored .env, or .NET user secrets — never committed.

📊 Observability

A standalone Grafana LGTM stack lives under monitoring/ (deployed to a Proxmox LXC). Apps emit OpenTelemetry; the CLI ships with a compact console exporter and is wired to swap in OTLP.

📁 Repository layout

BlueLaminate/            # .NET solution
  BlueLaminate.Core/     #   shared logic + composition root
  BlueLaminate.Scraper/  #   CSFloat API + skin catalogue clients
  BlueLaminate.EFCore/   #   Postgres data layer + migrations
  BlueLaminate.Cli/      #   operator CLI (System.CommandLine)
  BlueLaminate.C2/       #   ASP.NET control plane for the worker fleet
  BlueLaminate.Tests/    #   xUnit tests
worker/                  # Python nodriver workers (cs.money, skin.land)
db/                      # SQL schema + role hardening + audit queries
monitoring/              # Grafana LGTM observability stack
docker-compose.yml       # one-command C2 + worker bring-up

9.6 KiB Raw Blame History