examples/longformqa/longformqa_assertions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../../docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>\n",
    "\n",
    "## **DSPy Assertions**: Asserting Computational Constraints on Foundation \n",
    "\n",
    "### **LongFormQA**: Generating long-form length responses to answer questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[<img align=\"center\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/examples/longformqa/longformqa_assertions.ipynb)\n",
    "\n",
    "This notebook builds upon the foundational concepts of the **DSPy** framework, as introduced in our previous tutorial (see [intro.ipynb](./intro.ipynb) for a refresher). DSPy overs a novel programming-centric approach to utilizing language and retrieval models. It offers a unique blend of prompting, reasoning, fine-tuning, and tool augmentation, all encapsulated under a minimalistic Python syntax. \n",
    "\n",
    "In this advancement of DSPy, we introduce **Assertions**, a feature with the capability to declare computational constraints within DSPy programs. This allows programmers to specify natural-language rules for valid outputs, guiding the behavior of language model calls during both compiling and inference stages. \n",
    "\n",
    "Our approach harnesses Pythonic style of assertions while meshing backtracking logic to ensure autonomous self-correction and refinement of language model calls. By accounting for past outputs and passing forward relevant feedback and guidelines for self-correction, this feature offers a significant leap in DSPy with enhanced control over program behavior.\n",
    "\n",
    "This notebook demonstrates the utility of assertions on specific downstream examples, extending the Multi-Hop Question-Answering task from the [intro.ipynb](./intro.ipynb) to long-form paragraph generation with citations to answer questions. We demonstrate the performance benefits of integrating assertions to ensure the inclusion of citations in a predefined format and the faithfulness of generated text to its cited references. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0] Setting Up\n",
    "Let's begin by setting things up. The snippet below will retrieve the cached requests for the  task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://huggingface.co/arnavs11/DSPy_LongFormQA_Cache\n",
    "%cd DSPy_LongFormQA_Cache/\n",
    "!git checkout master\n",
    "%cd ..\n",
    "import os\n",
    "repo_clone_path = '/content/DSPy_LongFormQA_Cache'\n",
    "\n",
    "# Set up the cache for this notebook\n",
    "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = repo_clone_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also install **DSPy** if it's not there already."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import sys\n",
    "import os\n",
    "import regex as re\n",
    "\n",
    "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
    "    import google.colab\n",
    "    repo_path = 'dspy'\n",
    "    \n",
    "    !git -C $repo_path pull origin || git clone https://github.com/stanfordnlp/dspy $repo_path\n",
    "except:\n",
    "    repo_path = '.'\n",
    "\n",
    "if repo_path not in sys.path:\n",
    "    sys.path.append(repo_path)\n",
    "\n",
    "\n",
    "import pkg_resources # Install the package if it's not installed\n",
    "if not \"dspy-ai\" in {pkg.key for pkg in pkg_resources.working_set}:\n",
    "    !pip install -U pip\n",
    "    !pip install dspy-ai\n",
    "    !pip install openai~=0.28.1\n",
    "    !pip install -e $repo_path\n",
    "\n",
    "import dspy\n",
    "from dspy.predict import Retry\n",
    "from dspy.datasets import HotPotQA\n",
    "\n",
    "from dspy.teleprompt import BootstrapFewShotWithRandomSearch\n",
    "from dsp.utils import EM, normalize_text\n",
    "from dspy.primitives.assertions import assert_transform_module, backtrack_handler\n",
    "\n",
    "%cd dspy/examples/longformqa\n",
    "\n",
    "from utils import extract_text_by_citation, correct_citation_format, has_citations, citations_check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1] Getting Started\n",
    "\n",
    "We'll start by setting up the language model (LM) and retrieval model (RM). **DSPy** supports multiple API and local models. In this notebook, we'll work with GPT-3.5 (`gpt-3.5-turbo`) and the retriever `ColBERTv2`.\n",
    "\n",
    "To make things easy, we've set up a ColBERTv2 server hosting a Wikipedia 2017 \"abstracts\" search index (i.e., containing first paragraph of each article from this [2017 dump](https://hotpotqa.github.io/wiki-readme.html)), so you don't need to worry about setting one up! It's free.\n",
    "\n",
    "We configure **DSPy** to use the turbo LM and the ColBERTv2 retriever (over Wikipedia 2017 abstracts) by default. This can be overwritten for local parts of programs if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n",
    "dspy.settings.configure(rm=colbertv2_wiki17_abstracts)\n",
    "turbo = dspy.OpenAI(model='gpt-3.5-turbo', max_tokens=500)\n",
    "dspy.settings.configure(lm=turbo, trace=[], temperature=0.7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2] Dataset\n",
    "\n",
    "Now, let's load a sample from the HotPotQA multi-hop dataset for our tasks. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = HotPotQA(train_seed=1, train_size=300, eval_seed=2023, dev_size=300, test_size=0, keep_details=True)\n",
    "trainset = [x.with_inputs('question') for x in dataset.train]\n",
    "devset = [x.with_inputs('question') for x in dataset.dev]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We just loaded `trainset` (300 examples) and `devset` (300 examples). Each example in our **training set** contains just a **question,** its corresponding (human-annotated) **answer**, and the **gold titles**. These gold titles represent titles of relevant Wikipedia articles that contain supporting facts necessary to answering the question. \n",
    "\n",
    "After loading the datasets, we'd applied `x.with_inputs('question')` to each example to tell **DSPy** that our input field in each example will be just `question`. Any other fields are labels not given to the system.\n",
    "\n",
    "Now, let's look at some data examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_example = trainset[0]\n",
    "print(f\"Question: {train_example.question}\")\n",
    "print(f\"Answer: {train_example.answer}\")\n",
    "print(f\"Relevant Wikipedia Titles: {train_example.gold_titles}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_example = devset[18]\n",
    "print(f\"Question: {dev_example.question}\")\n",
    "print(f\"Answer: {dev_example.answer}\")\n",
    "print(f\"Relevant Wikipedia Titles: {dev_example.gold_titles}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3] LongFormQA with Citations\n",
    "\n",
    "Let's define our first complete program for this task. We extend the `Multi-Hop QA` program, shifting the answer generation focus from short phrases of 1-5 words to comprehensive paragraphs that include citations. \n",
    "\n",
    "The `LongFormQA` module reflects the iterative multi-hop generation process in query generation, passage retrieval, and context assembly. The `GenerateCitedParagraph` layer then takes the context state alongside the question to generate a paragraph with relevant reference citations to the context. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this program, we aim to generate paragraphs that adhere the following guidelines:\n",
    "1. Every 1-2 sentences in the paragraph are followed by citations in the intended format **\"{text}... [source_num].\"**\n",
    "2. Every text segment preceding a citation is faithful to the referenced source passage. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dsp.utils import deduplicate\n",
    "\n",
    "class GenerateSearchQuery(dspy.Signature):\n",
    "    \"\"\"Write a simple search query that will help answer a complex question.\"\"\"\n",
    "    context = dspy.InputField(desc=\"may contain relevant facts\")\n",
    "    question = dspy.InputField()\n",
    "    query = dspy.OutputField()\n",
    "\n",
    "class GenerateCitedParagraph(dspy.Signature):\n",
    "    \"\"\"Generate a paragraph with citations.\"\"\"\n",
    "    context = dspy.InputField(desc=\"may contain relevant facts\")\n",
    "    question = dspy.InputField()\n",
    "    paragraph = dspy.OutputField(desc=\"includes citations\")\n",
    "\n",
    "class LongFormQA(dspy.Module):\n",
    "    def __init__(self, passages_per_hop=3, max_hops=2):\n",
    "        super().__init__()\n",
    "        self.generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]\n",
    "        self.retrieve = dspy.Retrieve(k=passages_per_hop)\n",
    "        self.generate_cited_paragraph = dspy.ChainOfThought(GenerateCitedParagraph)\n",
    "        self.max_hops = max_hops\n",
    "    \n",
    "    def forward(self, question):\n",
    "        context = []\n",
    "        for hop in range(self.max_hops):\n",
    "            query = self.generate_query[hop](context=context, question=question).query\n",
    "            passages = self.retrieve(query).passages\n",
    "            context = deduplicate(context + passages)\n",
    "        pred = self.generate_cited_paragraph(context=context, question=question)\n",
    "        pred = dspy.Prediction(context=context, paragraph=pred.paragraph)\n",
    "        return pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4] Evaluation\n",
    "\n",
    "We now define our evaluation metrics, **Intrinsic** and **Extrinsic** quality checks:\n",
    "\n",
    "#### Intrinsic Metrics: passing internal computational constraints is the goal \n",
    "\n",
    "**Faithfulness (per Citation)**: To verify the accuracy of each citation in the generated text, we utilize another **DSPy** program: `ChainOfThought` of `CheckCitationFaithfulness`. This module takes segments of text preceding each citation and its corresponding context passage and determines whether the text accurately reflects the facts of the context. This validation process involves a language model call for each citation, ensuring that each reference in the generated paragraph is factually consistent with its reference source."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CheckCitationFaithfulness(dspy.Signature):\n",
    "    \"\"\"Verify that the text is based on the provided context.\"\"\"\n",
    "    context = dspy.InputField(desc=\"may contain relevant facts\")\n",
    "    text = dspy.InputField(desc=\"between 1 to 2 sentences\")\n",
    "    faithfulness = dspy.OutputField(desc=\"boolean indicating if text is faithful to context\")\n",
    "\n",
    "def citation_faithfulness(example, pred, trace):\n",
    "    paragraph, context = pred.paragraph, pred.context\n",
    "    citation_dict = extract_text_by_citation(paragraph)\n",
    "    if not citation_dict:\n",
    "        return False, None\n",
    "    context_dict = {str(i): context[i].split(' | ')[1] for i in range(len(context))}\n",
    "    faithfulness_results = []\n",
    "    unfaithful_citations = []\n",
    "    check_citation_faithfulness = dspy.ChainOfThought(CheckCitationFaithfulness)\n",
    "    for citation_num, texts in citation_dict.items():\n",
    "        if citation_num not in context_dict:\n",
    "            continue\n",
    "        current_context = context_dict[citation_num]\n",
    "        for text in texts:\n",
    "            try:\n",
    "                result = check_citation_faithfulness(context=current_context, text=text)\n",
    "                is_faithful = result.faithfulness.lower() == 'true'\n",
    "                faithfulness_results.append(is_faithful)\n",
    "                if not is_faithful:\n",
    "                    unfaithful_citations.append({'paragraph': paragraph, 'text': text, 'context': current_context})\n",
    "            except ValueError as e:\n",
    "                faithfulness_results.append(False)\n",
    "                unfaithful_citations.append({'paragraph': paragraph, 'text': text, 'error': str(e)})\n",
    "    final_faithfulness = all(faithfulness_results)\n",
    "    if not faithfulness_results:\n",
    "        return False, None\n",
    "    return final_faithfulness, unfaithful_citations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extrinsic Metrics: Assess the overall quality and effectiveness of generated output on downstream task:\n",
    "\n",
    "- **Citation Precision**: Measures proportion of cited 'gold titles' in generated paragraph from all cited titles for datapoint. \n",
    "- **Citation Recall**: Measures proportion of cited 'gold titles' in generated paragraph from all 'gold titles' for datapoint.\n",
    "- **Answer Inclusion**: Evaluates whether generated paragraph with citations accurately incorporates the 'gold' answer for datapoint. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_cited_titles_from_paragraph(paragraph, context):\n",
    "    cited_indices = [int(m.group(1)) for m in re.finditer(r'\\[(\\d+)\\]\\.', paragraph)]\n",
    "    cited_indices = [index - 1 for index in cited_indices if index <= len(context)]\n",
    "    cited_titles = [context[index].split(' | ')[0] for index in cited_indices]\n",
    "    return cited_titles\n",
    "\n",
    "def calculate_recall(example, pred, trace=None):\n",
    "    gold_titles = set(example['gold_titles'])\n",
    "    found_cited_titles = set(extract_cited_titles_from_paragraph(pred.paragraph, pred.context))\n",
    "    intersection = gold_titles.intersection(found_cited_titles)\n",
    "    recall = len(intersection) / len(gold_titles) if gold_titles else 0\n",
    "    return recall\n",
    "\n",
    "def calculate_precision(example, pred, trace=None):\n",
    "    gold_titles = set(example['gold_titles'])\n",
    "    found_cited_titles = set(extract_cited_titles_from_paragraph(pred.paragraph, pred.context))\n",
    "    intersection = gold_titles.intersection(found_cited_titles)\n",
    "    precision = len(intersection) / len(found_cited_titles) if found_cited_titles else 0\n",
    "    return precision\n",
    "\n",
    "def answer_correctness(example, pred, trace=None):\n",
    "    assert hasattr(example, 'answer'), \"Example does not have 'answer'.\"\n",
    "    normalized_context = normalize_text(pred.paragraph)\n",
    "    if isinstance(example.answer, str):\n",
    "        gold_answers = [example.answer]\n",
    "    elif isinstance(example.answer, list):\n",
    "        gold_answers = example.answer\n",
    "    else:\n",
    "        raise ValueError(\"'example.answer' is not string or list.\")\n",
    "    return 1 if any(normalize_text(answer) in normalized_context for answer in gold_answers) else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now evaluate our program on these metrics over our devset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(module):\n",
    "    correctness_values = []\n",
    "    recall_values = []\n",
    "    precision_values = []\n",
    "    citation_faithfulness_values = []\n",
    "    for i in range(len(devset)):\n",
    "        example = devset[i]\n",
    "        try:\n",
    "            pred = module(question=example.question)\n",
    "            correctness_values.append(answer_correctness(example, pred))            \n",
    "            citation_faithfulness_score, _ = citation_faithfulness(None, pred, None)\n",
    "            citation_faithfulness_values.append(citation_faithfulness_score)\n",
    "            recall = calculate_recall(example, pred)\n",
    "            precision = calculate_precision(example, pred)\n",
    "            recall_values.append(recall)\n",
    "            precision_values.append(precision)\n",
    "        except Exception as e:\n",
    "            print(f\"Failed generation with error: {e}\")\n",
    "\n",
    "    average_correctness = sum(correctness_values) / len(devset) if correctness_values else 0\n",
    "    average_recall = sum(recall_values) / len(devset) if recall_values else 0\n",
    "    average_precision = sum(precision_values) / len(devset) if precision_values else 0\n",
    "    average_citation_faithfulness = sum(citation_faithfulness_values) / len(devset) if citation_faithfulness_values else 0\n",
    "\n",
    "    print(f\"Average Correctness: {average_correctness}\")\n",
    "    print(f\"Average Recall: {average_recall}\")\n",
    "    print(f\"Average Precision: {average_precision}\")\n",
    "    print(f\"Average Citation Faithfulness: {average_citation_faithfulness}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "longformqa = LongFormQA()\n",
    "evaluate(longformqa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at an example paragraph generation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "question = devset[6].question\n",
    "pred = longformqa(question)\n",
    "citation_faithfulness_score, _ = citation_faithfulness(None, pred, None)\n",
    "\n",
    "print(f\"Question: {question}\")\n",
    "print(f\"Predicted Paragraph: {pred.paragraph}\")\n",
    "print(f\"Citation Faithfulness: {citation_faithfulness_score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the generated paragraph does not properly include citations as intended. In some instances, it follows an incorrect citation format by keeping the referenced source at the beginning of a sentence. In other instances, it does not provide citations for every 1-2 sentences as expected. \n",
    "\n",
    "Additionally, we see that not all included citations are faithful to their preceding text (which can be a result of cited references in an incorrect format)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5] Introducing Assertions: LongFormQAWithAssertions\n",
    "\n",
    "To correct these errors, we introduce **Assertions** to impose clear computational constraints within our program.\n",
    "\n",
    "DSPy provides two key mechanisms for **Assertions**:\n",
    "\n",
    "- **`dspy.Assert`**: This mandates that the program must satisfy the given assertion, raising an Exception otherwise. This is important when enforcing non-negotiable constraints within the program.\n",
    "- **`dspy.Suggest`**: Unlike `Assert`, `Suggest` is more flexible. It encourages the program to meet the assertion but allows the program to continue even if the assertion is not satisfied. This is particularly useful for guiding the program towards desired outcomes without halting execution for non-critical issues.\n",
    "\n",
    "Since our goal is indeed to evaluate the program on the defined metrics, let's utilize the `dspy.Suggest` assertion. \n",
    "\n",
    "The syntax for `dspy.Suggest` is as follows:\n",
    "```python\n",
    "dspy.Suggest(validation_function(model_outputs): bool, instruction_message: str)\n",
    "```\n",
    "\n",
    "Let's add assertions to abide by the computational constraints defined above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LongFormQAWithAssertions(dspy.Module):\n",
    "    def __init__(self, passages_per_hop=3, max_hops=2):\n",
    "        super().__init__()\n",
    "        self.generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]\n",
    "        self.retrieve = dspy.Retrieve(k=passages_per_hop)\n",
    "        self.generate_cited_paragraph = dspy.ChainOfThought(GenerateCitedParagraph)\n",
    "        self.max_hops = max_hops\n",
    "    \n",
    "    def forward(self, question):\n",
    "        context = []\n",
    "        for hop in range(self.max_hops):\n",
    "            query = self.generate_query[hop](context=context, question=question).query\n",
    "            passages = self.retrieve(query).passages\n",
    "            context = deduplicate(context + passages)\n",
    "        pred = self.generate_cited_paragraph(context=context, question=question)\n",
    "        pred = dspy.Prediction(context=context, paragraph=pred.paragraph)\n",
    "        dspy.Suggest(citations_check(pred.paragraph), f\"Make sure every 1-2 sentences has citations. If any 1-2 sentences lack citations, add them in 'text... [x].' format.\", target_module=GenerateCitedParagraph)\n",
    "        _, unfaithful_outputs = citation_faithfulness(None, pred, None)\n",
    "        if unfaithful_outputs:\n",
    "            unfaithful_pairs = [(output['text'], output['context']) for output in unfaithful_outputs]\n",
    "            for _, context in unfaithful_pairs:\n",
    "                dspy.Suggest(len(unfaithful_pairs) == 0, f\"Make sure your output is based on the following context: '{context}'.\", target_module=GenerateCitedParagraph)\n",
    "        else:\n",
    "            return pred\n",
    "        return pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We include assertions that simply reiterate our computational constraints and now allow the `LongFormQA` program to execute and adhere to these guidelines under the hood. \n",
    "\n",
    "Since we want to impose these assertions on the paragraph generation, we can pass in the `GenerateCitedParagraph` signature to indicate the `target_module` for the assertion handling to identify. \n",
    "\n",
    "In the first **Assertion**, we validate the output paragraph to ensure citations are included every 1-2 sentences. If this validation returns False, the assertion backtracking logic is activated the feedback instruction: **\"Ensure each 1-2 sentences include citations in 'text... [x].' format.\"**\n",
    "\n",
    "In the second **Assertion**, we now utilize the `CheckCitationFaithfulness` program to validate the accuracy of each cited references, looping over text segments denoted in the generated paragraph. In cases of unfaithful citations, it sends the feedback instruction alongside the context as: **\"Ensure your output aligns with this context: '{context}'.\"** This ensures the assertion backtracking has the relevant information and specific context it needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now evaluate our `LongFormQAWithAssertions` program over the devset.\n",
    "\n",
    "Note that this requires wrapping the module with the `Retry` module which handles the backtracking logic. This wrapped module is then passed to the `assert_transform_module` function to prepare and execute the backtracking logic. This is passed alongside the `backtrack_handler` which configures the backtracking logic to account for the feedback messages passed in to the `dspy.Suggest` statements within the program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "longformqa_with_assertions = assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler) \n",
    "evaluate(longformqa_with_assertions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the same example from above with the `LongFormQAWithAssertions` program:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "question = devset[6].question\n",
    "pred = longformqa_with_assertions(question)\n",
    "citation_faithfulness_score, _ = citation_faithfulness(None, pred, None)\n",
    "\n",
    "print(f\"Question: {question}\")\n",
    "print(f\"Predicted Paragraph: {pred.paragraph}\")\n",
    "print(f\"Citation Faithfulness: {citation_faithfulness_score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now see that both computational constraints are indeed met. Every 1-2 sentences includes a citation and from our citation_faithfulness check, we see that each reference is also faithful to its preceding text. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6] Compilation With Assertions\n",
    "\n",
    "We can also leverage **DSPy**'s advanced compiling features to enhance our program's performance. \n",
    "\n",
    "For this, we utilize the `BootstrapFewShotWithRandomSearch` teleprompter, which automatically incorporates few-shot demonstrations and conducts a random search over a candidate set to output the best compiled program. We evaluate this over the `answer_correctness` metric as our ultimate goal is indeed to generate correct answers to the `HotPotQA` questions from the paragraphs, aiming to optimize both intrinsic and extrinsic metrics as a result. \n",
    "\n",
    "Let's evaluate this on the LongFormQA program first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "longformqa = LongFormQA()\n",
    "teleprompter = BootstrapFewShotWithRandomSearch(metric = answer_correctness, max_bootstrapped_demos=2, num_candidate_programs=6)\n",
    "cited_longformqa = teleprompter.compile(student = longformqa, teacher = longformqa, trainset=trainset, valset=devset[:100])\n",
    "evaluate(cited_longformqa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's evaluate this with assertions. \n",
    "\n",
    "**Note** The pipeline here lies in compiling with **Assertions** to give the teleprompter correct bootstrapped examples by the `answer_correctness` metric and then 'teaching' the student with these correct examples. This is represented by passing `LongFormQA()` as the student and `LongFormQAWithAssertions()` as the teacher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "longformqa = LongFormQA()\n",
    "teleprompter = BootstrapFewShotWithRandomSearch(metric = answer_correctness, max_bootstrapped_demos=2, num_candidate_programs=6)\n",
    "cited_longformqa_teacher = teleprompter.compile(student=longformqa, teacher = assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), trainset=trainset, valset=devset[:100])\n",
    "evaluate(cited_longformqa_teacher)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** This pipeline on the other hand sets both the teacher and student with `LongFormQAWithAssertions()` to ensure the teacher correctly instructs the student with the right bootstrapped examples and the student has the chance to self-correct with **Assertions** for any examples that are still deemed incorrect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "longformqa = LongFormQA()\n",
    "teleprompter = BootstrapFewShotWithRandomSearch(metric = answer_correctness, max_bootstrapped_demos=2, num_candidate_programs=6)\n",
    "cited_longformqa_student_teacher = teleprompter.compile(student=assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), teacher = assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), trainset=trainset, valset=devset[:100])\n",
    "evaluate(cited_longformqa_student_teacher)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dspy_dev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}