feat: runnables powered agent (langchain-ai#2927)

* feat: runnables powered agent

* refactor: make agent extend from runnableagent

* fix: revert old changes

* nit

* refactor: update agent executor to convert to runnable if is runnable

* started tests

* fix: pass tests, allow runnables

* fix: allow agents to be runnable in agent executor, add ToolType to runnable base

* refactor: updated base agent to accept runnables, deprecated llmChain

* fix: update runnable input

* chore: lint files

* chore: lint files

* fix: bring back output parser

* fix: improve agent.int tests

* refactor: two output parsing methods for if runnable vs no runnable is passed in

* chore: lint files

* fix: improve comment

* refactor: make toolkits pass in runnables

* fix: remove unnecessary async from parseAgentOutput method

* refactor: getting llm and prompt from openai functions agent

* fix: remove unnecessary output parser

* fix: update agent input interface generics to match class generics

* nit: class reordering

* refactor: del llmChain as an input

* chore: add test verifying fallbacks work

* nit: remove maxTokens input

* chore: lint files

* refactor: extend xml agent from agent

* fix: allow runnables to be passed into AgentExecutor

* chore: lint files

* fix: cleanup code

* chore: lint files

* fix: resolve comments

* nits

* fix: pass generics

* fix: pass generics in static methods

* fix: init generics with any, fix test

* (tests)feat: added tests for custom output parser

* chore: lint files

* feat: structured output tests

* nit

* spyon

* fix: test and added formatForOpenAIFunctions util

* chore: lint files

* nit: lowercase name in test

* nit: revert all changes on openai file

* nit: revert old agent.plan to use predict

* feat: added runnable agent docs

* chore: lint files

* nit: move .md to .mdx

* chore: move to agents how to sec, add some extra context in the doc

* nit: return non stringified response

* Use type import in docs

---------

Co-authored-by: Jacob Lee <[email protected]>

docs/docs/modules/agents/how_to/structured_output_runnables_agent.mdx

@@ -0,0 +1,185 @@
+# Structured Output Agent with Runnables
+
+The `AgentExecutor` class accepts a `Runnable` as the agent. With this, we can create powerful agents using the LCEL and the `AgentExecutor` class.
+
+Here is a simple example of an agent which uses `Runnables`, a retriever and a structured output parser to create an OpenAI functions agent that finds specific information in a large text document.
+
+The first step is to import necessary modules
+
+```typescript
+import { zodToJsonSchema } from "zod-to-json-schema";
+import fs from "fs";
+import { z } from "zod";
+import type {
+  AIMessage,
+  AgentAction,
+  AgentFinish,
+  AgentStep,
+} from "langchain/schema/index.js";
+import { RunnableSequence } from "langchain/schema/runnable/base.js";
+import {
+  ChatPromptTemplate,
+  MessagesPlaceholder,
+} from "langchain/prompts/chat.js";
+import { ChatOpenAI } from "langchain/chat_models/openai.js";
+import { createRetrieverTool } from "langchain/agents/toolkits/index.js";
+import { RecursiveCharacterTextSplitter } from "langchain/text_splitter.js";
+import { HNSWLib } from "langchain/vectorstores/hnswlib.js";
+import { OpenAIEmbeddings } from "langchain/embeddings/openai.js";
+import { formatToOpenAIFunction } from "langchain/tools/convert_to_openai.js";
+import { AgentExecutor } from "langchain/agents/executor.js";
+import { formatForOpenAIFunctions } from "langchain/agents/format_scratchpad.js";
+```
+
+Next, we load the text document and embed it using the OpenAI embeddings model.
+
+```typescript
+// Read text file & embed documents
+const text = fs.readFileSync("examples/state_of_the_union.txt", "utf8");
+const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000 });
+let docs = await textSplitter.createDocuments([text]);
+// Add fake document source information to the metadata
+docs = docs.map((doc, i) => ({
+  ...doc,
+  metadata: {
+    page_chunk: i,
+  },
+}));
+// Initialize docs & create retriever
+const vectorStore = await HNSWLib.fromDocuments(docs, new OpenAIEmbeddings());
+```
+
+Since we're going to want to retrieve the embeddings inside the agent, we need to instantiate the vector store as a retriever.
+We also need an LLM to preform the calls with.
+
+```typescript
+const retriever = vectorStore.asRetriever();
+const llm = new ChatOpenAI({});
+```
+
+In order to use our retriever with the LLM as an OpenAI function, we need to convert the retriever to a tool
+
+```typescript
+const retrieverTool = createRetrieverTool(retriever, {
+  name: "state-of-union-retriever",
+  description:
+    "Query a retriever to get information about state of the union address",
+});
+```
+
+Now we can define our prompt template. We'll use a simple `ChatPromptTemplate` with placeholders for the user's question, and the agent scratchpad (this will be very helpful in the future).
+
+```typescript
+const prompt = ChatPromptTemplate.fromMessages([
+  ["system", "You are a helpful assistant"],
+  new MessagesPlaceholder("agent_scratchpad"),
+  ["user", "{input}"],
+]);
+```
+
+After that, we define our structured response schema using zod. This schema defines the structure of the final response from the agent.
+
+```typescript
+const responseSchema = z.object({
+  answer: z.string().describe("The final answer to respond to the user"),
+  sources: z
+    .array(z.string())
+    .describe(
+      "List of page chunks that contain answer to the question. Only include a page chunk if it contains relevant information"
+    ),
+});
+```
+
+Once our response schema is defined, we can construct it as an OpenAI function to later be passed to the model.
+This is an important step regarding consistency as the model will always respond in this schema when it successfully completes a task
+
+```typescript
+const responseOpenAIFunction = {
+  name: "response",
+  description: "Return the response to the user",
+  parameters: zodToJsonSchema(responseSchema),
+};
+```
+
+Next, we can construct the custom structured output parser.
+
+```typescript
+const structuredOutputParser = (
+  output: AIMessage
+): AgentAction | AgentFinish => {
+  // If no function call is passed, return the output as an instance of `AgentFinish`
+  if (!("function_call" in output.additional_kwargs)) {
+    return { returnValues: { output: output.content }, log: output.content };
+  }
+  // Extract the function call name and arguments
+  const functionCall = output.additional_kwargs.function_call;
+  const name = functionCall?.name as string;
+  const inputs = functionCall?.arguments as string;
+  // Parse the arguments as JSON
+  const jsonInput = JSON.parse(inputs);
+  // If the function call name is `response` then we know it's used our final
+  // response function and can return an instance of `AgentFinish`
+  if (name === "response") {
+    return { returnValues: { ...jsonInput }, log: output.content };
+  }
+  // If none of the above are true, the agent is not yet finished and we return
+  // an instance of `AgentAction`
+  return {
+    tool: name,
+    toolInput: jsonInput,
+    log: output.content,
+  };
+};
+```
+
+After this, we can bind our two functions to the LLM, and create a runnable sequence which will be used as the agent.
+
+**Important** - note here we pass in `agent_scratchpad` as an input variable, which formats all the previous steps using the `formatForOpenAIFunctions` function.
+This is very important as it contains all the context history the model needs to preform accurate tasks. Without this, the model would have no context on the previous steps taken.
+The `formatForOpenAIFunctions` function returns the steps as an array of `BaseMessage`. This is necessary as the `MessagesPlaceholder` class expects this type as the input.
+
+```typescript
+const llmWithTools = llm.bind({
+  functions: [formatToOpenAIFunction(retrieverTool), responseOpenAIFunction],
+});
+/** Create the runnable */
+const runnableAgent = RunnableSequence.from([
+  {
+    input: (i: { input: string }) => i.input,
+    agent_scratchpad: (i: { input: string; steps: Array<AgentStep> }) =>
+      formatForOpenAIFunctions(i.steps),
+  },
+  prompt,
+  llmWithTools,
+  structuredOutputParser,
+]);
+```
+
+Finally, we can create an instance of `AgentExecutor` and run the agent.
+
+```typescript
+const executor = AgentExecutor.fromAgentAndTools({
+  agent: runnableAgent,
+  tools: [retrieverTool],
+});
+/** Call invoke on the agent */
+const res = await executor.invoke({
+  input: "what did the president say about kentaji brown jackson",
+});
+console.log({
+  res,
+});
+```
+
+The output will look something like this
+
+```typescript
+{
+  res: {
+    answer: 'President mentioned that he nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. He described her as one of our nation’s top legal minds and stated that she will continue Justice Breyer’s legacy of excellence.',
+    sources: [
+      'And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence. A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.'
+    ]
+  }
+}
+```

environment_tests/test-exports-bun/src/entrypoints.js

@@ -2,6 +2,7 @@ export * from "langchain/load";
 export * from "langchain/load/serializable";
 export * from "langchain/agents";
 export * from "langchain/agents/toolkits";
+export * from "langchain/agents/format_scratchpad";
 export * from "langchain/base_language";
 export * from "langchain/tools";
 export * from "langchain/chains";

environment_tests/test-exports-cf/src/entrypoints.js

@@ -2,6 +2,7 @@ export * from "langchain/load";
 export * from "langchain/load/serializable";
 export * from "langchain/agents";
 export * from "langchain/agents/toolkits";
+export * from "langchain/agents/format_scratchpad";
 export * from "langchain/base_language";
 export * from "langchain/tools";
 export * from "langchain/chains";

environment_tests/test-exports-cjs/src/entrypoints.js

@@ -2,6 +2,7 @@ const load = require("langchain/load");
 const load_serializable = require("langchain/load/serializable");
 const agents = require("langchain/agents");
 const agents_toolkits = require("langchain/agents/toolkits");
+const agents_format_scratchpad = require("langchain/agents/format_scratchpad");
 const base_language = require("langchain/base_language");
 const tools = require("langchain/tools");
 const chains = require("langchain/chains");

environment_tests/test-exports-esbuild/src/entrypoints.js

@@ -2,6 +2,7 @@ import * as load from "langchain/load";
 import * as load_serializable from "langchain/load/serializable";
 import * as agents from "langchain/agents";
 import * as agents_toolkits from "langchain/agents/toolkits";
+import * as agents_format_scratchpad from "langchain/agents/format_scratchpad";
 import * as base_language from "langchain/base_language";
 import * as tools from "langchain/tools";
 import * as chains from "langchain/chains";

environment_tests/test-exports-esm/src/entrypoints.js

@@ -2,6 +2,7 @@ import * as load from "langchain/load";
 import * as load_serializable from "langchain/load/serializable";
 import * as agents from "langchain/agents";
 import * as agents_toolkits from "langchain/agents/toolkits";
+import * as agents_format_scratchpad from "langchain/agents/format_scratchpad";
 import * as base_language from "langchain/base_language";
 import * as tools from "langchain/tools";
 import * as chains from "langchain/chains";

environment_tests/test-exports-vercel/src/entrypoints.js

@@ -2,6 +2,7 @@ export * from "langchain/load";
 export * from "langchain/load/serializable";
 export * from "langchain/agents";
 export * from "langchain/agents/toolkits";
+export * from "langchain/agents/format_scratchpad";
 export * from "langchain/base_language";
 export * from "langchain/tools";
 export * from "langchain/chains";

environment_tests/test-exports-vite/src/entrypoints.js

@@ -2,6 +2,7 @@ export * from "langchain/load";
 export * from "langchain/load/serializable";
 export * from "langchain/agents";
 export * from "langchain/agents/toolkits";
+export * from "langchain/agents/format_scratchpad";
 export * from "langchain/base_language";
 export * from "langchain/tools";
 export * from "langchain/chains";

langchain/.gitignore

@@ -19,6 +19,9 @@ agents/toolkits/aws_sfn.d.ts
 agents/toolkits/sql.cjs
 agents/toolkits/sql.js
 agents/toolkits/sql.d.ts
+agents/format_scratchpad.cjs
+agents/format_scratchpad.js
+agents/format_scratchpad.d.ts
 base_language.cjs
 base_language.js
 base_language.d.ts

langchain/package.json

@@ -31,6 +31,9 @@
     "agents/toolkits/sql.cjs",
     "agents/toolkits/sql.js",
     "agents/toolkits/sql.d.ts",
+    "agents/format_scratchpad.cjs",
+    "agents/format_scratchpad.js",
+    "agents/format_scratchpad.d.ts",
     "base_language.cjs",
     "base_language.js",
     "base_language.d.ts",
@@ -1337,6 +1340,11 @@
       "import": "./agents/toolkits/sql.js",
       "require": "./agents/toolkits/sql.cjs"
     },
+    "./agents/format_scratchpad": {
+      "types": "./agents/format_scratchpad.d.ts",
+      "import": "./agents/format_scratchpad.js",
+      "require": "./agents/format_scratchpad.cjs"
+    },
     "./base_language": {
       "types": "./base_language.d.ts",
       "import": "./base_language.js",

langchain/scripts/create-entrypoints.js

@@ -16,6 +16,7 @@ const entrypoints = {
   "agents/toolkits": "agents/toolkits/index",
   "agents/toolkits/aws_sfn": "agents/toolkits/aws_sfn",
   "agents/toolkits/sql": "agents/toolkits/sql/index",
+  "agents/format_scratchpad": "agents/format_scratchpad",
   // base language
   base_language: "base_language/index",
   // tools

langchain/src/agents/agent.ts

@@ -14,9 +14,11 @@ import { StructuredTool, Tool } from "../tools/base.js";
 import {
   AgentActionOutputParser,
   AgentInput,
+  RunnableAgentInput,
   SerializedAgent,
   StoppingMethod,
 } from "./types.js";
+import { Runnable } from "../schema/runnable/base.js";
 
 /**
  * Record type for arguments passed to output parsers.
@@ -122,6 +124,52 @@ export abstract class BaseSingleActionAgent extends BaseAgent {
   ): Promise<AgentAction | AgentFinish>;
 }
 
+/**
+ * Class representing a single action agent which accepts runnables.
+ * Extends the BaseSingleActionAgent class and provides methods for
+ * planning agent actions with runnables.
+ */
+export class RunnableAgent<
+  RunInput extends ChainValues & {
+    agent_scratchpad?: string | BaseMessage[];
+    stop?: string[];
+  },
+  RunOutput extends AgentAction | AgentFinish
+> extends BaseSingleActionAgent {
+  protected lc_runnable = true;
+
+  lc_namespace = ["langchain", "agents", "runnable"];
+
+  runnable: Runnable<RunInput, RunOutput>;
+
+  stop?: string[];
+
+  get inputKeys(): string[] {
+    return [];
+  }
+
+  constructor(fields: RunnableAgentInput<RunInput, RunOutput>) {
+    super();
+    this.runnable = fields.runnable;
+    this.stop = fields.stop;
+  }
+
+  async plan(
+    steps: AgentStep[],
+    inputs: RunInput,
+    callbackManager?: CallbackManager
+  ): Promise<AgentAction | AgentFinish> {
+    const invokeInput = { ...inputs, steps };
+
+    const output = await this.runnable.invoke(invokeInput, {
+      callbacks: callbackManager,
+      runName: "RunnableAgent",
+    });
+
+    return output;
+  }
+}
+
 /**
  * Abstract base class for multi-action agents in LangChain. Extends the
  * BaseAgent class and provides additional functionality specific to
@@ -249,6 +297,7 @@ export abstract class Agent extends BaseSingleActionAgent {
 
   constructor(input: AgentInput) {
     super(input);
+
     this.llmChain = input.llmChain;
     this._allowedTools = input.allowedTools;
     this.outputParser = input.outputParser;