Untitled Page

# Clinical Guidelines Q&A Assistant (RAG)
Modified from the original TCS BaNCS PDF-QA pipeline.

**What changed vs. the original, and why:**
1. **Loader**: PDF -> structured `.txt` loader that parses `## Disease:` and `### Section:` headers into metadata (disease, section) instead of extracting raw flat text.
2. **Chunking**: `CharacterTextSplitter` -> `RecursiveCharacterTextSplitter`, and metadata is now attached per-chunk (not discarded), so every retrieved chunk carries its disease/section back with it.
3. **Prompting**: default `RetrievalQA` prompt -> a custom `PromptTemplate` that enforces the four things a clinical assistant is expected to return: evidence-backed answer, section citation, treatment pathway summary, and a clinical-judgment warning.
4. **Retrieval output**: `qa_chain.invoke()` -> `return_source_documents=True`, so citations are pulled from actual retrieved metadata instead of being hallucinated by the LLM.
5. **Output formatting**: same HTML-rendering idea as your original, but now renders a citation table + a highlighted warning box instead of a generic key-metrics table.
## Cell 1 — Imports
import os
import re
import json
import requests
from langchain_community.llms import Ollama
from langchain_community.vectorstores import FAISS
from langchain.embeddings.base import Embeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
from langchain.docstore.document import Document
from langchain.prompts import PromptTemplate
from IPython.display import display, HTML
## Cell 2 — Ollama Embeddings (same technique as original)
class OllamaEmbeddings(Embeddings):
    def __init__(self, model='llama3.1', url='http://localhost:11434/api/embeddings'):
        self.model = model
        self.url = url

    def embed(self, text):
        headers = {'Content-Type': 'application/json'}
        payload = {'model': self.model, 'prompt': text}
        response = requests.post(self.url, headers=headers, data=json.dumps(payload))
        if response.status_code == 200:
            response_json = response.json()
            if 'embedding' in response_json:
                return response_json['embedding']
            print("Key 'embedding' not found in response")
            return None
        raise Exception(f"Error {response.status_code}: {response.text}")

    def embed_documents(self, texts):
        return [self.embed(text) for text in texts]

    def embed_query(self, text):
        return self.embed(text)
## Cell 3 — Load the LLM
llm = Ollama(
    model="llama3.1",
    temperature=0.1,  # slight bump from 0: clinical summaries read better with a touch of fluency
)
## Cell 4 — Structured TXT loader (replaces PdfReader)
Parses the `## Disease:` / `### Section:` headers from the synthetic file into metadata.
Each returned item is `(text_chunk, {"disease": ..., "section": ...})`.
def load_structured_txt(file_path):
    with open(file_path, "r", encoding="utf-8") as f:
        raw = f.read()

    disease_blocks = re.split(r'(?=## Disease:)', raw)
    records = []

    for block in disease_blocks:
        block = block.strip()
        if not block.startswith("## Disease:"):
            continue
        disease_name = block.splitlines()[0].replace("## Disease:", "").strip()

        section_splits = re.split(r'(?=### Section:)', block)
        for sec in section_splits:
            sec = sec.strip()
            if not sec.startswith("### Section:"):
                continue
            lines = sec.splitlines()
            section_name = lines[0].replace("### Section:", "").strip()
            section_text = "\n".join(lines[1:]).strip()
            records.append({
                "disease": disease_name,
                "section": section_name,
                "text": section_text
            })
    return records

txt_path = "clinical_guidelines.txt"
records = load_structured_txt(txt_path)
print(f"Parsed {len(records)} disease/section blocks")
## Cell 5 — Chunk with metadata preserved
`RecursiveCharacterTextSplitter` (vs. `CharacterTextSplitter` in the original) splits more gracefully
across paragraph/sentence boundaries, and every resulting chunk keeps its `disease` + `section` metadata.
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " "]
)

documents = []
for rec in records:
    sub_chunks = text_splitter.split_text(rec["text"])
    for chunk in sub_chunks:
        documents.append(
            Document(
                page_content=chunk,
                metadata={"disease": rec["disease"], "section": rec["section"]}
            )
        )

print(f"Created {len(documents)} chunks with citation metadata")
## Cell 6 — Embeddings + FAISS knowledge base (same technique as original)
embeddings = OllamaEmbeddings()
knowledge_base = FAISS.from_documents(documents, embeddings)
## Cell 7 — Custom clinical prompt template
This is the key prompting change: instead of the default RetrievalQA prompt, we force the four
required output components from the use case (evidence-backed answer, citation, pathway summary, warning).
clinical_prompt = PromptTemplate(
    input_variables=["context", "question"],
    template="""You are a clinical guideline assistant. Use ONLY the context below to answer.
If the context does not contain enough information, say so explicitly instead of guessing.

Context:
{context}

Question: {question}

Respond in exactly this structure:
1. Evidence-Backed Answer: <concise answer using only the context>
2. Treatment Pathway Summary: <numbered steps if applicable, otherwise "Not applicable">
3. Clinical Judgment Warning: <a short note that this is guideline-based information and a qualified clinician must confirm applicability to the specific patient>
"""
)
## Cell 8 — RetrievalQA chain with source citations enabled
qa_chain = RetrievalQA.from_chain_type(
    llm,
    retriever=knowledge_base.as_retriever(search_kwargs={"k": 4}),
    chain_type_kwargs={"prompt": clinical_prompt},
    return_source_documents=True  # <-- lets us build real citations from metadata, not from LLM guesswork
)
## Cell 9 — Ask a clinical question and extract citations
question = "What is the recommended treatment pathway for Type 2 Diabetes and when should a specialist be involved?"

response = qa_chain.invoke({"query": question})

print(response["result"])
print("\n--- Citations ---")
seen = set()
citations = []
for doc in response["source_documents"]:
    key = (doc.metadata.get("disease"), doc.metadata.get("section"))
    if key not in seen:
        seen.add(key)
        citations.append(key)
        print(f"- {key[0]} -> {key[1]}")
## Cell 10 — Structured HTML rendering
Same idea as your original HTML cell, but tailored to the use case: a summary block,
a citations table (built from real metadata, not model text), and a highlighted warning box.
citations_rows = "".join(
    f"<tr><td><b>{d}</b></td><td>{s}</td></tr>" for d, s in citations
)

html_output = f"""
<h1>Clinical Guidelines Q&A Assistant</h1>
<h3>Question</h3>
<div>{question}</div>

<h3>Answer</h3>
<div style="white-space: pre-wrap; border:1px solid #ccc; padding:10px;">{response['result']}</div>

<h3>Citations</h3>
<table border="1" cellpadding="6" cellspacing="0">
<tr><th>Disease</th><th>Guideline Section</th></tr>
{citations_rows}
</table>

<div style="margin-top:15px; padding:10px; background:#fff3cd; border:1px solid #ffeeba;">
<b>Note:</b> This output is generated from synthetic training data and general guideline structure.
It is not a substitute for clinical judgment or current, source-verified medical guidance.
</div>
"""

display(HTML(html_output))