add readme
This commit is contained in:
189
README.md
Normal file
189
README.md
Normal file
@@ -0,0 +1,189 @@
|
|||||||
|
# BlueLaminate
|
||||||
|
|
||||||
|
> A CS2 skin-market intelligence platform — ingests live marketplace listings across **CSFloat**, **cs.money**, and **skin.land**, tracks prices and float values in Postgres, and mines the catalogue for profitable **trade-up contracts**.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✨ What it does
|
||||||
|
|
||||||
|
| Capability | Where | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| **CSFloat ingestion** | `BlueLaminate.Scraper` + CLI | Official API. Catalogue-driven and global incremental sweeps, paced off rate-limit headers. |
|
||||||
|
| **cs.money scraping** | `worker/csmoney_worker.py` | Cloudflare-walled. A Python `nodriver` worker fleet driven by the C2 (pull model). |
|
||||||
|
| **skin.land scraping** | `worker/skinland_worker.py` | Nuxt SSR behind Cloudflare. Same worker plumbing, market-specific scrape. |
|
||||||
|
| **Cross-market presence** | `MarketPresenceService` | "Where is this skin/instance listed, and at what price?" across all sources. |
|
||||||
|
| **Trade-up engine** | `BlueLaminate.Core/Tradeups` | Finds profitable 10-input contracts from live listings — exact float math, guaranteed vs. expected-value ranking, StatTrak universes, interactive TUI. |
|
||||||
|
| **Price history & dedup** | `BlueLaminate.EFCore` | Postgres data model with per-source checkpoints, `SkinInstance` fingerprinting, and `price_history`. |
|
||||||
|
| **Observability** | `monitoring/` | Standalone Grafana LGTM stack (Loki / Grafana / Tempo / Mimir) wired via OTLP. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🏗️ Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────┐
|
||||||
|
│ PostgreSQL │
|
||||||
|
│ (skintracker schema, least-priv role) │
|
||||||
|
└───────────────▲────────────────▲──────────┘
|
||||||
|
│ │
|
||||||
|
┌───────────────────────────┘ └────────────────┐
|
||||||
|
│ │
|
||||||
|
┌────────┴─────────┐ ┌─────────┴──────────┐
|
||||||
|
│ BlueLaminate.Cli │ CSFloat (official API) │ BlueLaminate.C2 │
|
||||||
|
│ sync / sweep / │─────────────────────────────────────► │ (ASP.NET control │
|
||||||
|
│ find-tradeups │ │ plane) │
|
||||||
|
└────────┬──────────┘ └─────────▲──────────┘
|
||||||
|
│ │ jobs / results
|
||||||
|
│ shares │ (X-Worker-Token)
|
||||||
|
▼ ┌─────────┴──────────┐
|
||||||
|
┌────────────────────┐ │ Python workers │
|
||||||
|
│ BlueLaminate.Core │ ◄── composition root, all shared │ nodriver + IPRoyal │
|
||||||
|
│ (ingest, tradeups, │ logic (AddBlueLaminateCore) │ cs.money / skin.land│
|
||||||
|
│ market presence) │ └─────────────────────┘
|
||||||
|
└────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
- **`BlueLaminate.Core`** — the single composition root. Every frontend wires itself with `AddBlueLaminateCore(configuration)` and reuses the same ingest/tradeup/presence services. No duplicated logic.
|
||||||
|
- **`BlueLaminate.Scraper`** — typed CSFloat API client + the CSGO-API skin catalogue client.
|
||||||
|
- **`BlueLaminate.EFCore`** — Postgres data layer and EF Core migrations (auto-applied on C2 boot).
|
||||||
|
- **`BlueLaminate.Cli`** — `System.CommandLine` operator tools.
|
||||||
|
- **`BlueLaminate.C2`** — hands cs.money/skin.land scrape jobs to the Python worker fleet and ingests their results.
|
||||||
|
- **`worker/`** — Python `nodriver` workers. .NET Selenium gets insta-challenged by Cloudflare; `nodriver` drives Chromium directly over CDP and passes. Each replica mints its own IPRoyal sticky residential exit IP.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Quick start
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- [.NET 10 SDK](https://dotnet.microsoft.com/download)
|
||||||
|
- **PostgreSQL** with a `skintracker` database
|
||||||
|
- Python 3.x + Chrome/Edge (only if you run the cs.money / skin.land workers)
|
||||||
|
- Docker (optional — for the one-command C2 + worker stack)
|
||||||
|
- A **CSFloat API key** for the CSFloat paths (`CSFLOAT_API_KEY`)
|
||||||
|
|
||||||
|
### 1. Database
|
||||||
|
|
||||||
|
Run the schema/role hardening **once** as a superuser (edit the password placeholders first):
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
psql -U postgres -d skintracker -f db/01_schema_and_roles.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
The C2 applies EF migrations automatically on boot (`AutoMigrate=true`); for the CLI alone, run `dotnet ef database update --project BlueLaminate\BlueLaminate.EFCore`.
|
||||||
|
|
||||||
|
### 2. Build
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
dotnet build BlueLaminate/BlueLaminate.slnx
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Run the CLI
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# One-time: pull the CS2 skin catalogue (throttled to once a month)
|
||||||
|
dotnet run --project BlueLaminate\BlueLaminate.Cli -- sync-skins
|
||||||
|
|
||||||
|
# Ingest live CSFloat listings (reads CSFLOAT_API_KEY)
|
||||||
|
$env:CSFLOAT_API_KEY="your-key"
|
||||||
|
dotnet run --project BlueLaminate\BlueLaminate.Cli -- sweep-catalog
|
||||||
|
|
||||||
|
# Mine profitable trade-up contracts (interactive TUI)
|
||||||
|
dotnet run --project BlueLaminate\BlueLaminate.Cli -- find-tradeups --min-profit 2.00
|
||||||
|
```
|
||||||
|
|
||||||
|
#### CLI commands
|
||||||
|
|
||||||
|
| Command | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `sync-skins` | Load the CS2 skin catalogue from the CSGO-API dataset and upsert it. |
|
||||||
|
| `fetch-listings` | Fetch active CSFloat listings for one skin and print them (no DB write). |
|
||||||
|
| `sweep-listings` | Global incremental sweep of active CSFloat listings into the DB. |
|
||||||
|
| `sweep-catalog` | Catalogue-driven sweep: query each skin by `def_index`+`paint_index`, split by wear band. |
|
||||||
|
| `find-tradeups` | Find profitable 10-input trade-up contracts from live listings, ranked best-first. |
|
||||||
|
|
||||||
|
Add `--help` to any command for its full option set.
|
||||||
|
|
||||||
|
### 4. Run the marketplace workers (cs.money / skin.land)
|
||||||
|
|
||||||
|
The browser/Cloudflare scrapers run as Python workers that pull jobs from the C2.
|
||||||
|
|
||||||
|
**Containerized (one command):**
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# Defaults: 1 cs.money worker + 1 skin.land worker. Postgres is external (host).
|
||||||
|
docker compose up --build
|
||||||
|
```
|
||||||
|
|
||||||
|
- C2 → http://localhost:5080 (`/health`, `/jobs/*`, `/market/*`)
|
||||||
|
- Worker noVNC → http://localhost:6080/vnc.html (watch the browser; solve a Cloudflare challenge by hand if one appears)
|
||||||
|
|
||||||
|
Set the worker mix and proxy via env / a `.env` file next to `docker-compose.yml`:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
$env:CSMONEY_WORKERS=0; $env:SKINLAND_WORKERS=3
|
||||||
|
$env:IPROYAL_USERNAME="..."; $env:IPROYAL_PASSWORD="..."
|
||||||
|
docker compose up --build
|
||||||
|
```
|
||||||
|
|
||||||
|
**Local (without Docker):**
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# terminal 1 — the C2 (from repo root)
|
||||||
|
dotnet run --project BlueLaminate\BlueLaminate.C2 # http://localhost:5080
|
||||||
|
|
||||||
|
# terminal 2 — a worker
|
||||||
|
cd worker
|
||||||
|
py -m venv .venv; .venv\Scripts\Activate.ps1
|
||||||
|
pip install -r requirements.txt
|
||||||
|
$env:WORKER_TOKEN="dev-worker-token" # must match the C2's WorkerToken
|
||||||
|
python csmoney_worker.py # or skinland_worker.py
|
||||||
|
```
|
||||||
|
|
||||||
|
See [`DOCKER.md`](DOCKER.md) and [`worker/README.md`](worker/README.md) for the full configuration matrix, scaling notes, and proxy details.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚙️ Configuration
|
||||||
|
|
||||||
|
Key settings (env vars override `appsettings.json`):
|
||||||
|
|
||||||
|
| Var | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `ConnectionStrings__SkinTracker` / `SKINTRACKER_CONN` | Postgres connection string. |
|
||||||
|
| `CSFLOAT_API_KEY` | CSFloat official API key (CLI sweep/fetch). |
|
||||||
|
| `WorkerToken` / `WORKER_TOKEN` | Shared secret between C2 and workers. |
|
||||||
|
| `MaxPagesPerJob`, `MinResweepHours` | C2 sweep budget / re-sweep floor (metered-proxy cost control). |
|
||||||
|
| `IPROYAL_USERNAME` / `IPROYAL_PASSWORD` / `IPROYAL_COUNTRY` | Residential proxy for workers. |
|
||||||
|
|
||||||
|
> 🔒 Secrets (API keys, proxy credentials, DB passwords) belong in environment variables, a gitignored `.env`, or .NET user secrets — never committed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Observability
|
||||||
|
|
||||||
|
A standalone Grafana LGTM stack lives under [`monitoring/`](monitoring/README.md) (deployed to a Proxmox LXC). Apps emit OpenTelemetry; the CLI ships with a compact console exporter and is wired to swap in OTLP.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📁 Repository layout
|
||||||
|
|
||||||
|
```
|
||||||
|
BlueLaminate/ # .NET solution
|
||||||
|
BlueLaminate.Core/ # shared logic + composition root
|
||||||
|
BlueLaminate.Scraper/ # CSFloat API + skin catalogue clients
|
||||||
|
BlueLaminate.EFCore/ # Postgres data layer + migrations
|
||||||
|
BlueLaminate.Cli/ # operator CLI (System.CommandLine)
|
||||||
|
BlueLaminate.C2/ # ASP.NET control plane for the worker fleet
|
||||||
|
BlueLaminate.Tests/ # xUnit tests
|
||||||
|
worker/ # Python nodriver workers (cs.money, skin.land)
|
||||||
|
db/ # SQL schema + role hardening + audit queries
|
||||||
|
monitoring/ # Grafana LGTM observability stack
|
||||||
|
docker-compose.yml # one-command C2 + worker bring-up
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user