Cut metered-proxy bandwidth: re-sweep floor + wire-size logging
JobQueue now skips bands swept within MinResweepHours (config, default 6h) instead of re-scraping the whole catalogue continuously — the dominant cost on the metered residential proxy. Roughly linear savings with no data loss (full pagination retained); 0 disables it. Worker logs the real compressed transferSize per job (what the proxy bills) rather than the ~6.5x-larger decompressed length, so spend is visible. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -14,7 +14,12 @@ var builder = WebApplication.CreateBuilder(new WebApplicationOptions
|
||||
ContentRootPath = AppContext.BaseDirectory,
|
||||
});
|
||||
builder.Services.AddBlueLaminateCore(builder.Configuration);
|
||||
builder.Services.AddSingleton<JobQueue>();
|
||||
|
||||
// Re-sweep floor: don't re-hand-out a band whose listings were swept less than this
|
||||
// many hours ago. The dominant cost on the metered residential proxy is re-scraping
|
||||
// already-fresh bands, so this caps how often any band is re-pulled. 0 = continuous.
|
||||
var minResweepHours = builder.Configuration.GetValue("MinResweepHours", 6.0);
|
||||
builder.Services.AddSingleton(new JobQueue(TimeSpan.FromHours(minResweepHours)));
|
||||
|
||||
var app = builder.Build();
|
||||
|
||||
|
||||
Reference in New Issue
Block a user