/system-design
How ChartForge builds SOAP
Clinician product first: generate and edit audit-ready notes. Under the hood — hierarchical aggregation, three comparable generation arms, and held-out eval gates. Local workplace stays on :3847. Fine-tuning create remains paused.
Pipeline
Raw transcript → normalize → key phrases → BiomedBERT embeddings → hierarchical clustering → SOAP section classification → note assembly
Long behavioral-health sessions bury medical-necessity evidence when you flatten everything into one summary prompt.
Held-out compare arms
- Classic — normalize → key phrases → BiomedBERT-class clusters → ensemble SOAP classify → assemble (Context Aggregation of Transcripts)
- Flat — single transcript→SOAP prompt (no chunk / hierarchy for generation) via Together — shared eval hierarchy for grounding chips
- Modern — utterance hierarchy + CAP-lite → L1 chunk summarize → parallel S/O/A/P via Together (stub if no TOGETHER_API_KEY) → provenance chips → eval gate
- BASE — Together serverless control (no LoRA) on the same fixtures
Product stays generate/edit SOAP; Classic · Flat · Modern · BASE scores are the held-out quality panel.
Open held-out compare (quality first) →Why hierarchy
- Behavioral health sessions are long and multi-topic; flat summarization buries medical-necessity evidence.
- Hierarchy mirrors clinical documentation: utterance → CAP-lite propositions → SOAP section → note.
- Section-parallel generation (S/O/A/P) + Linked Evidence beats one mega-prompt for fidelity.
- Eval weights omissions over fabrications; non-reasoning configs preferred for SOAP fidelity.
Provenance metrics
91% SOAP section accuracy · +15% vs baseline LLM · ↑16% (LLM-as-a-judge) · ↓12% GPT-2 perplexity
Context Aggregation of Transcripts — evaluation on ACI-Bench / MTS-Dialog
- SOAP structure verification (section presence + order)
- QA-based factual consistency (T5 + DistilBERT)
- Clinical relevance (BERTScore + BioBERT cosine)
- LLM-as-a-judge clinical coherence (DeepSeek-R1 class)
- Fluency via GPT-2 perplexity
Infra posture
Together for inference + FT · never OpenAI — classic CAT local; flat = direct-prompt baseline; FT launch paused.
Train: Together Fine-Tuning (create locked / paused unless explicitly enabled). Serve: Together chat for modern + flat live SOAP.
Public deploy prep targets soap.avinashnandyala.com on Cloudflare — local :3847 stays the default development path. See docs/cloudflare-hosting.md.