⚖️ RAG Compliance Intelligence

Ask questions across AML, GDPR, SOX, HIPAA, CCPA, and PCI DSS compliance documents — answers come with cited sources. FAISS + sentence-transformers, no API key. By default a small local language model (Qwen2.5-1.5B) writes the answer grounded in the retrieved passages, with a graceful fall back to fast passage extraction if the model cannot load; the model can be wrong, so check the cited sources.
32 document chunks indexed · source in the Files tab

Sample questions

Retrieval stack: sentence-transformers all-MiniLM-L6-v2 embeddings in a FAISS IndexFlatIP, blended with BM25 keyword scores, then re-ranked by the ms-marco-MiniLM-L-12-v2 cross-encoder.

Abstention: a logistic answerability estimate must reach 0.92 or the system refuses rather than guess; questions naming terms the documents never use (e.g. FINRA) are refused outright.

Measured on the repo's own evaluation sets (tests/eval_set.py, the frozen answerability_*.json, and tests/evaluate.py):

Metric Result
KNOWN compliance questions answered 14 / 14 (100% recall)
Out-of-corpus / near-topic refused (frozen set 1) 73 / 76 abstained
Out-of-corpus / near-topic refused (frozen set 2) 60 / 60 abstained
Answerable wrongly refused (frozen set 1 / set 2) 6 / 53 · 12 / 57
Answerable wrongly refused across all 212 eval questions 24 / 212 (11%)

The 0.92 threshold is deliberately strict: raising it from an earlier 0.785 refuses near-topic look-alikes (EU AMLA, Singapore PDPA) that are out of corpus, at the honest cost of wrongly refusing more genuine questions — wrongly-refused rose from 13/212 (6%) to 24/212 (11%). We prefer a clean refusal over a confident wrong answer.

Limitations: answers cover only the six bundled regimes (AML, GDPR, SOX, HIPAA, CCPA, PCI DSS); anything outside them is refused by design. The language model can still phrase things wrongly, so every number it writes is checked against the cited passages and flagged if unverified — always read the cited sources.