feature: AI #1674

YousefED · 2025-05-09T05:42:56Z

This PR adds AI functionality to BlockNote!

There's still some work required but don't expect major changes at this point

Preview @ https://blocknote-git-feature-ai-typecell.vercel.app/ai/playground

Review notes

Large PR, recommended to review in VS Code
I'd recommend first reviewing the parts outside of the AI packages (core, react, etc). We could even merge those as part of a separate PR
I left quite some comments, but please leave feedback if things are unclear - we want the code to be in a state you feel comfortable working on future iterations

TODOs

After v1:

add example with custom AI buttons / toolbar items
Implement additional unit tests
Improve table support
merge #1605 and enable related tests
"space for AI"

- AI Menu items look more similar to Notion - `getDefaultAIMenuItems` uses same pattern as `getDefaultSlashMenuItems` - AI Block actually sets the `timeGenerated` prop after generating a response - AI Block only appears in the block type dropdown when the selection is in one - AI button in Formatting Toolbar now opens the AI Menu normally instead of in a popover

nperez0111 · 2025-05-15T12:55:05Z

packages/xl-ai/src/prosemirror/agent.ts

+ * - replace the text with the first character of the replacement (if any) (1 transaction per ReplaceStep)
+ * - insert the replacement character by character (strlen-1 transactions per ReplaceStep)
+ */
+export function getStepsAsAgent(doc: Node, pmSchema: Schema, steps: Step[]) {


This makes more sense to operate on a whole transform, and give a transform back rather than an array of steps:

export function getStepsAsAgent(trToTransform: Transform) { const pmSchema = getPmSchema(trToTransform); const { modification } = pmSchema.marks; const agentSteps: AgentStep[] = []; const tr = new Transform(trToTransform.doc); for (const step of trToTransform.steps) { ... return tr;

I agree on the input, but the current return type does have additional metadata, right? So we can't really return a transform?

Yep, a transform wouldn't be enough for that.

nperez0111 · 2025-05-15T13:00:21Z

packages/xl-ai/src/api/formats/base-tools/createAddBlocksTool.ts

+          referenceId = referenceId.slice(0, -1);
+        }
+
+        const block = editor.getBlock(referenceId);


This is always pulling from the current editor state. I would probably have an in-memory version of the document & pull from that instead. Probably more of an issue for collab mode than anything

nperez0111 · 2025-05-15T13:04:37Z

packages/xl-ai/src/prosemirror/agent.ts

+  }
+}
+
+export function agentStepToTr(tr: Transaction, step: AgentStep) {


applyAgentSteps?

This actually may also be better implemented as a prosemirror-command that can be editor.exec'd, because right now this can just throw if it cannot be applied, whereas what you'd want is for it to just not be applied at all and continue streaming

packages/xl-ai/src/prosemirror/agent.ts

nperez0111 · 2025-05-15T13:21:08Z

packages/xl-ai/src/prosemirror/agent.ts

+        ...Object.keys(oldNode.attrs),
+      ]);
+      for (const attr of attrNames) {
+        if (newNode.attrs[attr] !== oldNode.attrs[attr]) {


attributes can be objects in prosemirror, though this works for blocknote attributes

Good to know. I think prosemirror-suggest-changes also only supports string attrs fwiw

packages/xl-ai/src/prosemirror/agent.ts

nperez0111 · 2025-05-15T13:31:46Z

packages/xl-ai/src/prosemirror/agent.ts

+    // It might be cleaner to;
+    // a) make this optional
+    // b) actually delete / insert the content and let prosemirror-suggest-changes handle the marks


Yea, I would consider doing b, it feels like prosemirror-suggest-changes should be doing this transformation, though I can understand wanting the fine-grained control of how it displays visually.

Yeah the initial prototype used the current approach, I only later hooked up prosemirror-suggest-changes. I tried to migrate to having that handle the insertions, but was running into a few edge-cases which made it more cumbersome than I initially hoped (ofc, might have just been a single mapping / positioning error). For now I think we can keep it as is, but good to leave the note here if we run into things that the "other approach" could address, wdyt?

Totally fine with that. If we experience a bug with it, I'd probably just throw it away and re-implement

nperez0111 · 2025-05-15T13:38:48Z

packages/xl-ai/src/prosemirror/agent.ts

+      // note, instead of inserting one charachter at a time at the end (a, b, c)
+      // we're replacing the entire part every time (a, ab, abc)
+      // would be cleaner to do just insertions, but didn't get this to work with the add operation
+      // (and this kept code relatively simple)
+      const replacement = new Slice(step.slice.content.cut(0, i), 0, 0);


This will affect how the ops are laid out in the undo/redo stack.
Here is where in prosemirror-history steps are merged together: https://github.com/ProseMirror/prosemirror-history/blob/master/src/history.ts#L237
This is a ReplaceStep's merge method: https://github.com/ProseMirror/prosemirror-transform/blob/137ff74738bd1b50d49416cd6cfdbbf52cb059ef/src/replace_step.ts#L48-L62
So, this can only merge ops which occur one after the other, not overlapping ranges like you've got here.

Let's discuss undo / redo tomorrow morning - let's look at the zoomed out issue first because I think the way undo / redo works in prosemirror might not be applicable to our use-case at all

nperez0111 · 2025-05-15T14:09:30Z

packages/xl-ai/src/prosemirror/agent.ts

+      tr.replace(replaceFrom, replaceEnd, replacement).addMark(
+        replaceFrom,
+        replaceFrom + replacement.content.size,
+        pmSchema.mark("insertion", {}),
+      );


This will also affect how undo/redo stack works, since an addMark step cannot be merged with a replace step.

What I've done before for this, is to add the mark to the replacement content separately, so it doesn't need a separate addMark step

nperez0111 · 2025-05-15T14:09:55Z

packages/xl-ai/src/prosemirror/agent.ts

+            return true;
+          }
+          if (node.isBlock) {
+            tr.addNodeMark(pos, pmSchema.mark("insertion", {}));


nperez0111 · 2025-05-15T14:13:17Z

packages/xl-ai/src/prosemirror/changeset.ts

+ * NOTE: we might be able to replace this code with a custom encoder
+ * (this didn't exist yet when this was written)


Yea, I think the whole custom encoder thing is just for this. Probably okay for now, but could be simplified

nperez0111 · 2025-05-15T14:16:04Z

packages/xl-ai/src/prosemirror/changeset.ts

+  const tableCells = new Set(
+    [...tableCellsOld].filter((cell) => tableCellsNew.has(cell)),
+  );


const tableCells = tableCellsOld.intersection(tableCellsNew);

do you know if this consistently works or requires extra setup? I tried this but got an error at some point that intersection is not available

nperez0111 · 2025-05-15T14:16:57Z

packages/xl-ai/src/prosemirror/changeset.ts

+    encodeNodeStart: (node) => {
+      if (node.type.name === "tableCell") {
+        const str = JSON.stringify(node.toJSON());
+        if (tableCells.has(str)) {
+          return str;
+        }
+        return node.type.name;


Maybe worth a comment that you are encoding it like this to "flatten" the changeset

The idea here was to give two equal table cells in before / after a unique Encoding, to nudge prosemirror-changeset to keep them intact.

I think this encoder improved the table handling a bit, but I'm still running into issues like described here: ProseMirror/prosemirror-changeset#22

will require more research unless you have a good idea. At least, I'll add more documentation to the code here

packages/xl-ai/src/prosemirror/changeset.ts

packages/xl-ai/src/streamTool/callLLMWithStreamTools.ts

nperez0111 · 2025-05-15T15:07:49Z

packages/xl-ai/src/streamTool/callLLMWithStreamTools.ts

+        _operationsSource = createAsyncIterableStreamFromAsyncIterable(
+          preprocessOperationsStreaming(
+            filterNewOrUpdatedOperations(
+              streamOnStartCallback(
+                partialObjectStreamThrowError(ret.fullStream),
+                onStart,
+              ),
+            ),
+            streamTools,
+          ),
+        );


I have to wonder whether it is easier to implement this using streams or as async iterables (like you've done here).

Streams are a somewhat awkward API, but they are pretty good at this sort of a thing with their ability to pipe through transforms. Just a thought

nperez0111 · 2025-05-15T15:10:11Z

packages/xl-ai/src/streamTool/preprocess.ts

+/**
+ * Validates an stream of operations and throws an error if an invalid operation is found.
+ */
+export async function* preprocessOperationsNonStreaming<


Are these not the same? streaming vs. not streaming?

with non-streaming, we assume all operations must be valid, and we throw an error if not.

with streaming, operations can be partial, so we just drop invalid operations

* promptbuilder poc * rename * other formats

YousefED · 2025-05-15T19:22:55Z

packages/xl-ai/src/api/formats/html-blocks/defaultHTMLPromptBuilder.ts

+  ];
+}
+
+export const defaultHTMLPromptBuilder: PromptBuilder = async (editor, opts) => {


@nperez0111 as discussed on slack:

I think it would be even nicer if you can provide a function that takes data similar to promptManipulateSelectionHTMLBlocks, but then we’d need to call getPromptData automatically and also make that optionally configurable. Now you just need to call the helpers yourself (similar to how defaultHTMLPromptBuilder does this)
I think if we generalize this further it might get quite complicated considering the difference between selection / non selection, and different formats

matthewlipski and others added 30 commits August 1, 2024 16:06

Added AI block

3e1983b

Added inline and slash menu AI

0da498e

Small fix

d48d91e

UX improvements & refactor

82a56a9

Extracted AI to separate package & changed AI block toolbar UX

5c66cfe

Finished initial package split

22db2b4

Moved last AI references to AI package

e0f60a8

Reverted minor unneeded changes

a2bab5d

refactor architecture

bcf9d31

add extensions

b814336

Refactored AI dictionary

cfc1bed

clean dictionary

ec36733

fix

78ac784

fix

2970e9d

Made AI button use suggestion menu components

78924bb

Added keyboard navigation to AI button

4083cd9

Refactored AI button

d0d82a4

Changed AI from suggestion menu to propriety menu

2df84f5

Minor changes

644aa15

Prevented focus swapping on suggestion menu items

0fcb46a

Fixed AI Menu position for empty blocks

736a8ff

Made AI block react instead of vanilla

ffa466d

fix build

2c20238

schema

f474949

improve json schema methods

6ddf7b0

Merge remote-tracking branch 'origin/main' into ai-block

34abf80

merge

b3926fe

improve json schema methods

a9d25c9

fix build

2021ce7

nperez0111 reviewed May 15, 2025

View reviewed changes

packages/xl-ai/src/prosemirror/agent.ts Outdated Show resolved Hide resolved

nperez0111 reviewed May 15, 2025

View reviewed changes

packages/xl-ai/src/prosemirror/agent.ts Outdated Show resolved Hide resolved

nperez0111 reviewed May 15, 2025

View reviewed changes

packages/xl-ai/src/prosemirror/agent.ts Show resolved Hide resolved

nperez0111 reviewed May 15, 2025

View reviewed changes

packages/xl-ai/src/prosemirror/changeset.ts Outdated Show resolved Hide resolved

nperez0111 reviewed May 15, 2025

View reviewed changes

packages/xl-ai/src/streamTool/callLLMWithStreamTools.ts Outdated Show resolved Hide resolved

nperez0111 reviewed May 15, 2025

View reviewed changes

Feature/ai promptbuilder (#1686)

5bfc40e

* promptbuilder poc * rename * other formats

vercel bot had a problem deploying to Preview – blocknote May 15, 2025 18:17 Failure

vercel bot had a problem deploying to Preview – blocknote-website May 15, 2025 18:19 Failure

fix types

39a3fd5

YousefED commented May 15, 2025

View reviewed changes

vercel bot deployed to Preview – blocknote-website May 15, 2025 19:29 View deployment

vercel bot deployed to Preview – blocknote May 15, 2025 19:45 View deployment

YousefED added 3 commits May 15, 2025 22:07

fragmentUtil

8042603

refactor InvalidOrOk to Result

a60df0b

fix stream error

49bbf93

vercel bot deployed to Preview – blocknote May 15, 2025 20:30 View deployment

vercel bot deployed to Preview – blocknote-website May 15, 2025 20:37 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: AI #1674

feature: AI #1674

YousefED commented May 9, 2025 •

edited

Loading

nperez0111 May 15, 2025

YousefED May 15, 2025

nperez0111 May 15, 2025

nperez0111 May 15, 2025

nperez0111 May 15, 2025

nperez0111 May 15, 2025

nperez0111 May 15, 2025

YousefED May 15, 2025

nperez0111 May 15, 2025

YousefED May 15, 2025

nperez0111 May 15, 2025

nperez0111 May 15, 2025

YousefED May 15, 2025

nperez0111 May 15, 2025

nperez0111 May 15, 2025

nperez0111 May 15, 2025

nperez0111 May 15, 2025

YousefED May 15, 2025

nperez0111 May 15, 2025

YousefED May 15, 2025

nperez0111 May 15, 2025

nperez0111 May 15, 2025

YousefED May 15, 2025

YousefED May 15, 2025

		* NOTE: we might be able to replace this code with a custom encoder
		* (this didn't exist yet when this was written)

feature: AI #1674

Are you sure you want to change the base?

feature: AI #1674

Conversation

YousefED commented May 9, 2025 • edited Loading

Review notes

TODOs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YousefED commented May 9, 2025 •

edited

Loading