Constraint-Aware RAG Architecture for SLMs

Hypothesis: A constraint-aware, evidence-linked RAG architecture specifically optimized for Small Language Models (SLMs) can achieve legal reasoning accuracy and hallucination reduction comparable to or better than larger LLM-based RAG systems, while operating at significantly lower computational cost. So, basically you can create a new RAG architecture than performs much better than a currently available Plain SLM. Steps: 1. Check these SLMs first to benchmark on Retrieval/QA tasks: Inlegal‑Sbert (Retrieval) / InLegalBERT embeddings (Retrieval) / InLegalLlama (QA) 2. Dataset in consideration: Dataset: a) IPC Sections with conditions, exceptions, constraints, relations b) Supreme court judgement copies c) IPC_and_case_related_QA_dataset 3. Evaluate Inlegal‑Sbert and InLegalBERT embeddings on retrieval task and evaluate InLegalLlama on QA task from the above dataset 4. Evaluation Metrics for Retrieval: nDCG@10 Recall@5 / Recall@10 Mean Reciprocal Rank (MRR) Precision@k Metrics for QA Evaluation: ROUGE-L BERTScore Faithfulness (are all claims backed by retrieved text?) Citation Accuracy Hallucination Rate 5. Now build constraint-aware, evidence-linked RAG architecture and add it with SLM to see if it can beat the previous metrics evaluated from the SLM alone. 6. For building the constraint-aware, evidence-linked RAG architecture, refer following, Core components: =============== 1. Corpus & Index — statute chunks with hierarchy metadata (section/subsection/illustration/exception). 2. Embedding retriever — InLegal-SBERT / InLegalBERT sentence embeddings + FAISS index. 3. Reranker — cross-encoder reranker (InLegalBERT fine-tuned) to re-rank top-k. 4. Constraint module — set of lightweight rule engines/filters enforcing legal constraints before generation. 5. Evidence formatter — organizes retrieved snippets + provenance into generator context. 6. Generator (SLM) — small seq2seq (FLAN-T5 small/TinyLlama / Phi-4) fine-tuned with LoRA/QLoRA to output multi-field evidence-linked answers. 7. Hallucination detector / verifier — checks claims against retrieved evidence; optionally runs a secondary verification model.

Project ID: 40176887