From 083daa7022afdfffb0c92361ea2049f0dda6a2c6 Mon Sep 17 00:00:00 2001 From: bob Date: Tue, 2 Jun 2026 13:42:14 -0500 Subject: [PATCH] add readme --- README.md | 189 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..b3449ed --- /dev/null +++ b/README.md @@ -0,0 +1,189 @@ +# BlueLaminate + +> A CS2 skin-market intelligence platform β€” ingests live marketplace listings across **CSFloat**, **cs.money**, and **skin.land**, tracks prices and float values in Postgres, and mines the catalogue for profitable **trade-up contracts**. + +![.NET](https://img.shields.io/badge/.NET-10.0-512BD4?logo=dotnet&logoColor=white) +![Python](https://img.shields.io/badge/Python-3.x-3776AB?logo=python&logoColor=white) +![Postgres](https://img.shields.io/badge/PostgreSQL-skintracker-4169E1?logo=postgresql&logoColor=white) +![Docker](https://img.shields.io/badge/Docker-compose-2496ED?logo=docker&logoColor=white) +![Grafana](https://img.shields.io/badge/Observability-Grafana%20LGTM-F46800?logo=grafana&logoColor=white) + +--- + +## ✨ What it does + +| Capability | Where | Notes | +|---|---|---| +| **CSFloat ingestion** | `BlueLaminate.Scraper` + CLI | Official API. Catalogue-driven and global incremental sweeps, paced off rate-limit headers. | +| **cs.money scraping** | `worker/csmoney_worker.py` | Cloudflare-walled. A Python `nodriver` worker fleet driven by the C2 (pull model). | +| **skin.land scraping** | `worker/skinland_worker.py` | Nuxt SSR behind Cloudflare. Same worker plumbing, market-specific scrape. | +| **Cross-market presence** | `MarketPresenceService` | "Where is this skin/instance listed, and at what price?" across all sources. | +| **Trade-up engine** | `BlueLaminate.Core/Tradeups` | Finds profitable 10-input contracts from live listings β€” exact float math, guaranteed vs. expected-value ranking, StatTrak universes, interactive TUI. | +| **Price history & dedup** | `BlueLaminate.EFCore` | Postgres data model with per-source checkpoints, `SkinInstance` fingerprinting, and `price_history`. | +| **Observability** | `monitoring/` | Standalone Grafana LGTM stack (Loki / Grafana / Tempo / Mimir) wired via OTLP. | + +--- + +## πŸ—οΈ Architecture + +``` + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ PostgreSQL β”‚ + β”‚ (skintracker schema, least-priv role) β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–²β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–²β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ └────────────────┐ + β”‚ β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ BlueLaminate.Cli β”‚ CSFloat (official API) β”‚ BlueLaminate.C2 β”‚ + β”‚ sync / sweep / │─────────────────────────────────────► β”‚ (ASP.NET control β”‚ + β”‚ find-tradeups β”‚ β”‚ plane) β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β–²β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ jobs / results + β”‚ shares β”‚ (X-Worker-Token) + β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Python workers β”‚ + β”‚ BlueLaminate.Core β”‚ ◄── composition root, all shared β”‚ nodriver + IPRoyal β”‚ + β”‚ (ingest, tradeups, β”‚ logic (AddBlueLaminateCore) β”‚ cs.money / skin.landβ”‚ + β”‚ market presence) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +- **`BlueLaminate.Core`** β€” the single composition root. Every frontend wires itself with `AddBlueLaminateCore(configuration)` and reuses the same ingest/tradeup/presence services. No duplicated logic. +- **`BlueLaminate.Scraper`** β€” typed CSFloat API client + the CSGO-API skin catalogue client. +- **`BlueLaminate.EFCore`** β€” Postgres data layer and EF Core migrations (auto-applied on C2 boot). +- **`BlueLaminate.Cli`** β€” `System.CommandLine` operator tools. +- **`BlueLaminate.C2`** β€” hands cs.money/skin.land scrape jobs to the Python worker fleet and ingests their results. +- **`worker/`** β€” Python `nodriver` workers. .NET Selenium gets insta-challenged by Cloudflare; `nodriver` drives Chromium directly over CDP and passes. Each replica mints its own IPRoyal sticky residential exit IP. + +--- + +## πŸš€ Quick start + +### Prerequisites + +- [.NET 10 SDK](https://dotnet.microsoft.com/download) +- **PostgreSQL** with a `skintracker` database +- Python 3.x + Chrome/Edge (only if you run the cs.money / skin.land workers) +- Docker (optional β€” for the one-command C2 + worker stack) +- A **CSFloat API key** for the CSFloat paths (`CSFLOAT_API_KEY`) + +### 1. Database + +Run the schema/role hardening **once** as a superuser (edit the password placeholders first): + +```powershell +psql -U postgres -d skintracker -f db/01_schema_and_roles.sql +``` + +The C2 applies EF migrations automatically on boot (`AutoMigrate=true`); for the CLI alone, run `dotnet ef database update --project BlueLaminate\BlueLaminate.EFCore`. + +### 2. Build + +```powershell +dotnet build BlueLaminate/BlueLaminate.slnx +``` + +### 3. Run the CLI + +```powershell +# One-time: pull the CS2 skin catalogue (throttled to once a month) +dotnet run --project BlueLaminate\BlueLaminate.Cli -- sync-skins + +# Ingest live CSFloat listings (reads CSFLOAT_API_KEY) +$env:CSFLOAT_API_KEY="your-key" +dotnet run --project BlueLaminate\BlueLaminate.Cli -- sweep-catalog + +# Mine profitable trade-up contracts (interactive TUI) +dotnet run --project BlueLaminate\BlueLaminate.Cli -- find-tradeups --min-profit 2.00 +``` + +#### CLI commands + +| Command | Purpose | +|---|---| +| `sync-skins` | Load the CS2 skin catalogue from the CSGO-API dataset and upsert it. | +| `fetch-listings` | Fetch active CSFloat listings for one skin and print them (no DB write). | +| `sweep-listings` | Global incremental sweep of active CSFloat listings into the DB. | +| `sweep-catalog` | Catalogue-driven sweep: query each skin by `def_index`+`paint_index`, split by wear band. | +| `find-tradeups` | Find profitable 10-input trade-up contracts from live listings, ranked best-first. | + +Add `--help` to any command for its full option set. + +### 4. Run the marketplace workers (cs.money / skin.land) + +The browser/Cloudflare scrapers run as Python workers that pull jobs from the C2. + +**Containerized (one command):** + +```powershell +# Defaults: 1 cs.money worker + 1 skin.land worker. Postgres is external (host). +docker compose up --build +``` + +- C2 β†’ http://localhost:5080 (`/health`, `/jobs/*`, `/market/*`) +- Worker noVNC β†’ http://localhost:6080/vnc.html (watch the browser; solve a Cloudflare challenge by hand if one appears) + +Set the worker mix and proxy via env / a `.env` file next to `docker-compose.yml`: + +```powershell +$env:CSMONEY_WORKERS=0; $env:SKINLAND_WORKERS=3 +$env:IPROYAL_USERNAME="..."; $env:IPROYAL_PASSWORD="..." +docker compose up --build +``` + +**Local (without Docker):** + +```powershell +# terminal 1 β€” the C2 (from repo root) +dotnet run --project BlueLaminate\BlueLaminate.C2 # http://localhost:5080 + +# terminal 2 β€” a worker +cd worker +py -m venv .venv; .venv\Scripts\Activate.ps1 +pip install -r requirements.txt +$env:WORKER_TOKEN="dev-worker-token" # must match the C2's WorkerToken +python csmoney_worker.py # or skinland_worker.py +``` + +See [`DOCKER.md`](DOCKER.md) and [`worker/README.md`](worker/README.md) for the full configuration matrix, scaling notes, and proxy details. + +--- + +## βš™οΈ Configuration + +Key settings (env vars override `appsettings.json`): + +| Var | Purpose | +|---|---| +| `ConnectionStrings__SkinTracker` / `SKINTRACKER_CONN` | Postgres connection string. | +| `CSFLOAT_API_KEY` | CSFloat official API key (CLI sweep/fetch). | +| `WorkerToken` / `WORKER_TOKEN` | Shared secret between C2 and workers. | +| `MaxPagesPerJob`, `MinResweepHours` | C2 sweep budget / re-sweep floor (metered-proxy cost control). | +| `IPROYAL_USERNAME` / `IPROYAL_PASSWORD` / `IPROYAL_COUNTRY` | Residential proxy for workers. | + +> πŸ”’ Secrets (API keys, proxy credentials, DB passwords) belong in environment variables, a gitignored `.env`, or .NET user secrets β€” never committed. + +--- + +## πŸ“Š Observability + +A standalone Grafana LGTM stack lives under [`monitoring/`](monitoring/README.md) (deployed to a Proxmox LXC). Apps emit OpenTelemetry; the CLI ships with a compact console exporter and is wired to swap in OTLP. + +--- + +## πŸ“ Repository layout + +``` +BlueLaminate/ # .NET solution + BlueLaminate.Core/ # shared logic + composition root + BlueLaminate.Scraper/ # CSFloat API + skin catalogue clients + BlueLaminate.EFCore/ # Postgres data layer + migrations + BlueLaminate.Cli/ # operator CLI (System.CommandLine) + BlueLaminate.C2/ # ASP.NET control plane for the worker fleet + BlueLaminate.Tests/ # xUnit tests +worker/ # Python nodriver workers (cs.money, skin.land) +db/ # SQL schema + role hardening + audit queries +monitoring/ # Grafana LGTM observability stack +docker-compose.yml # one-command C2 + worker bring-up +```