Sable CLI Pipeline
Overview
The sable-cli pipeline automates the full BTIG data workflow: downloading files, validating, reconciling, uploading, and running QA checks.
Quick Start
cd /home/dshanklin/repos/aic-holdings/Sable/sable-cli
# Check what's missing (dry-run)
node sable.js uploads status
# Run full pipeline
node sable.js uploads pipeline --headed
Pipeline Phases
| Phase | Description |
|---|---|
| 1. DOWNLOAD | Download files from BTIG (ITD: perf + pnl, MTD: transactions) |
| 2. VALIDATE | Validate file structure, columns, and date ranges |
| 3. RECONCILE | Compare CSV totals vs PDF totals for PnL files |
| 4. UPLOAD | Upload files to Sable with client reconciliation |
| 5. OPTIONS FILLDOWN | Generate synthetic rows for expired/closed options |
| 6. SERVER QA | Check server-side validation and reconciliation |
| 7. REASONABLENESS | AI-powered analysis of data quality and anomalies |
Commands
Status Check
Preview what data is missing without downloading:
# Check yesterday (NYC time)
node sable.js uploads status
# Check specific dates
node sable.js uploads status --dates 2026-01-13,2026-01-14,2026-01-15
Returns exit code 1 if data missing, 0 if all accounts have data.
Full Pipeline
# Auto-detect missing dates, show browser
node sable.js uploads pipeline --headed
# Specific dates
node sable.js uploads pipeline --dates 2026-01-14 --headed
# Skip download (use existing files)
node sable.js uploads pipeline --skip-download
Options Filldown (Manual)
Run options filldown separately if needed:
# Preview what would be inserted
node sable.js pnl options filldown --dry-run
# Run filldown
node sable.js pnl options filldown
# Check disappeared options
node sable.js pnl options disappeared
File Types
| Type | Date Range | Description |
|---|---|---|
| Performance (perf) | ITD (2020-01-01 to date) | Performance attribution by account |
| P&L (pnl) | ITD (2020-01-01 to date) | Profit & Loss with evidence PDFs |
| Transactions (trans) | MTD (1st of month to date) | Monthly transaction history |
Auto-Detect
When run without --dates, the pipeline:
- Calculates target date using NYC timezone (yesterday)
- Checks each account independently for missing data
- Only downloads/uploads what's needed
Logs
Pipeline logs are saved to: ~/Downloads/BTIG/Logs/pipeline-YYYY-MM-DD-timestamp.json
Contains:
- Duration and success status
- Phase-by-phase results
- Failed files with errors
- Full event log
Options Filldown
The pipeline automatically generates synthetic P&L rows for options that disappeared from BTIG reports (expired, assigned, or closed). This maintains ITD continuity.
Synthetic rows have:
data_source = 'synthetic:options-filldown:{date}'- All P&L values set to $0 (P&L transferred to underlying stock)
- Position quantities set to 0
Reconciliation
PnL files are reconciled against PDF evidence:
| Status | Meaning |
|---|---|
| MATCH | CSV total matches PDF total (within $1 or 0.01%) |
| MISMATCH | CSV and PDF totals differ significantly |
| PDF_NOT_FOUND | No matching PDF file was provided |
| SKIPPED | Performance files (no P&L reconciliation) |