Refactor docs to use ExDocs search_data model for global HexDocs indexing of Gleam packages #3904

diemogebhardt · 2024-11-27T08:25:26Z

This PR is regarding issue #3811.

Data Model

The search data model json changed from SearchIndex:

[
  {
    "doc": "the name of the module or page",
    "title": "the title of the item",
    "content": "the content of the item",
    "url": "the relative link of the item, like 'module.html#title'"
  }
]

to SearchData, SearchItem, SearchItemType and SearchProgLang:

{
  "items": [
    {
      "type": "module" | "type" | "function" | "constant" | "page",
      "parentTitle": "the name of the module or page",
      "title": "the title of the item",
      "doc": "the content of the item",
      "ref": "the relative link of the item, like 'module.html#title'"
    }
  ],
  "proglang": "gleam"
}

Notes:

parentTitle doesn't seem to have an equivalent in ExDocs from what I have seen so far. It is being populated by:
- either module.name.to_string() in the context of types "module" | "type" | "function" | "constant"
- or config.name.to_string() in the context of type "page" (README.md)
doc matches ExDocs, except for headline partitioning
ref matches ExDocs, except for arity module.html#title/arity

Neither of those is an issue for global HexDocs indexing of Gleam packages.

Output Files

As I'm still exploring how everything works in the codebase, I didn't want to change too much at once. So I added search-data.json in addition to search-data.js to keep the existing mechanism working as is for now.

`search-data.js`

Wrapped search_data_json in window.Gleam.initSearch({}); for use in documentation_layout.html to execute initSearch({}) in docs-js/index.js which in turn uses lunr.js.

`search-data.json`

Bare search_data_json for use in global HexDocs.

lpil

Thank you!! I've left a few small comments inline, mostly around abbreviations, which we avoid in this codebase.

It would be great to have a test for the output format also.

lpil · 2024-12-02T17:33:41Z

compiler-core/src/docs.rs

+    #[serde(rename = "type")]
+    type_: SearchItemType,
+    #[serde(rename = "parentTitle")]
+    parent_title: String,


What's a parent title? Could you add /// documentation comments for each of these fields pleaes

Please have a look at my comments above:

In one I pasted a screenshot to visualize how it's being used currently;

and in another asked whether you might have an idea for a better name?

Please also have a look at my Notes.

lpil · 2024-12-02T17:34:01Z

compiler-core/src/docs.rs

-    doc: String,
+struct SearchData {
+    items: Vec<SearchItem>,
+    proglang: SearchProgLang,


Suggested change

proglang: SearchProgLang,

programming_language: SearchProgLang,

I agree with the naming and would prefer it too. Yet it is named differently in ExDocs so I went for consistency.

See here for the summarized ExDocs data format and more Info.

So shall we use programming_language internally and have a separate function that transforms the json output of search-data.json for HexDocs indexing?

lpil · 2024-12-02T17:34:27Z

compiler-core/src/docs.rs

-    url: String,
+    doc: String,
+    #[serde(rename = "ref")]
+    ref_: String,


Suggested change

ref_: String,

reference: String,

I agree with the naming and would prefer it too. Yet it is named differently in ExDocs so I went for consistency.

See here for the summarized ExDocs data format and more Info.

So shall we use reference internally and have a separate function that transforms the json output of search-data.json for HexDocs indexing?

lpil · 2024-12-02T17:34:37Z

compiler-core/src/docs.rs

    title: String,
-    content: String,
-    url: String,
+    doc: String,


Suggested change

doc: String,

content: String,

I agree with the naming and would prefer it too. Yet it is named differently in ExDocs so I went for consistency.

See here for the summarized ExDocs data format and more Info.

So shall we use content internally and have a separate function that transforms the json output of search-data.json for HexDocs indexing?

lpil · 2024-12-02T17:34:52Z

compiler-core/src/docs.rs

+
+#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
+#[serde(rename_all = "lowercase")]
+enum SearchProgLang {


Suggested change

enum SearchProgLang {

enum SearchProgrammingLanguage {

I agree with the naming and would prefer it too. Yet it is named differently in ExDocs so I went for consistency.

See here for the summarized ExDocs data format and more Info.

So shall we use SearchProgrammingLanguage internally and have a separate function that transforms the json output of search-data.json for HexDocs indexing?

lpil · 2024-12-02T17:35:40Z

compiler-core/templates/docs-js/index.js

        this.add({
          id: i,
+          // type: entry.type,


Another comment here

I added this comment to discuss, or keep it as a reminder that we have this property in the new data model but are not making use of it yet.

lpil · 2024-12-02T17:35:45Z

compiler-core/templates/docs-js/index.js

    const index = lunr(function () {
      this.ref("id");
+      // this.field("type");
+      // this.field("parentTitle");


Is this unfinished?

I added this comment to discuss, or keep it as a reminder that we have this property in the new data model but are not making use of it yet.

diemogebhardt · 2024-12-03T18:07:23Z

Yes, test are a good idea! :)

Lets clarify the other topics first though: whether we keep two separate data models. One for Gleam internal use and another for HexDocs indexing?

lpil · 2024-12-04T14:49:19Z

I think just one is fine, it's only used to render that JSON object.

diemogebhardt · 2024-12-06T02:23:28Z

Ok, I don't know what you want me to do. This isn't a good use for either your, nor my time.

Not sure how familiar you are with this part of the code base and when you looked into it last. From my own experience, I know that running a project while overseeing a larger code base with lots of different contributors is quite something. So I understand that revisiting parts of the code base that others have contributed to and you haven't reviewed recently takes getting used to. That's why I added context information, including screenshots in the initial post and comments regarding key changes to make it easier to follow. Do you see them or are they just visible to me?

Is there anything else I can do to communicate better?

diemogebhardt · 2024-12-06T02:49:51Z

Open Questions

Is consistent naming throughout the Gleam code base; and BEAM languages a goal? Options:
- 2 Models:
  - one internal with extended names for rust and generated js
    - used in HexDocs search of Gleam projects
  - one external with abbreviated names for generated json
    - used in global HexDocs indexing of Gleam projects
- 1 Model variant A (current implementation):
  - rust model named with abbreviated names
  - generated js and json with abbreviated names
    - used in HexDocs search of Gleam projects
    - used in global HexDocs indexing of Gleam projects
- 1 Model variant B (your proposal?):
  - rust model named with extended names
  - generated js and json with abbreviated names
    - used in HexDocs search of Gleam projects
    - used in global HexDocs indexing of Gleam projects
- Open a discussion with the ExDocs/HexDocs team to create a proper BEAM standard, thats actually documented and using better naming.
  - Would require them to be backwards compatible and support both during a transition period but probably feasible.
  - If this is a real option, I would volunteer to help with whatever is needed.
- ...?
Extra vs Page? Extra being the ExDocs/HexDocs way.
- If Extra, only for use in the model or also throughout the rest of the code base?
parent_title/parentTitle or does a better name come to mind?
- used in HexDocs search of Gleam projects
- NOT used in global HexDocs indexing of Gleam projects

Option: 2 Models

Pros:
- Such abbreviated names all but help with understanding the code. Having separate models would allow for nice descriptive names, used Gleam internally in rust and js for HexDocs search of Gleam projects. A quick search shows all affected places, no matter whether rust or js code.
- We don't pollute the global HexDocs indexing json with Gleam specific parent_title/parentTitle. More clarity whats used where.
Cons:
- A negative is that it's more work though, as two models need to be generated/transformed.
- Deviation from the BEAM "standard"?

/// for HexDocs search of Gleam projects
#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
struct SearchIndex {
    #[serde(rename = "parentTitle")]
    parent_title: String,
    title: String,
    content: String,
    reference: String,
}

/// for global HexDocs indexing
#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
struct SearchData {
    items: Vec<SearchItem>,
    proglang: SearchProgLang,
}
#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
struct SearchItem {
    #[serde(rename = "type")]
    type_: SearchItemType,
    title: String,
    doc: String,
    #[serde(rename = "ref")]
    ref_: String,
}
#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
#[serde(rename_all = "lowercase")]
enum SearchItemType {
    Constant,
    Function,
    Module,
    Page,
    Type,
}
#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
#[serde(rename_all = "lowercase")]
enum SearchProgLang {
    // Elixir,
    // Erlang,
    Gleam,
}

Option: 1 Model variant A (current implementation)

Pros:
- Consistent
  - internal naming for rust and js used in HexDocs search of Gleam projects;
  - AND external naming for json used in global HexDocs indexing of Gleam projects;
  - making it easier to understand and navigate all affected bits throughout the code base;
  - AND being consistent with BEAM "standard" used in global HexDocs indexing.
- ...?
Cons:
- Abbreviated naming in rust, js and json. Not so nice naming in the rust bits only?
- ...?

`docs.rs`

#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
struct SearchData {
    items: Vec<SearchItem>,
    proglang: SearchProgLang,
}

#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
struct SearchItem {
    #[serde(rename = "type")]
    type_: SearchItemType,
    #[serde(rename = "parentTitle")]
    parent_title: String,
    title: String,
    doc: String,
    #[serde(rename = "ref")]
    ref_: String,
}

#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
#[serde(rename_all = "lowercase")]
enum SearchItemType {
    Constant,
    Function,
    Module,
    Page,
    Type,
}

#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
#[serde(rename_all = "lowercase")]
enum SearchProgLang {
    // Elixir,
    // Erlang,
    Gleam,
}

outputs:

`search-data.js`

for use in compiler-core/templates/docs-js/index.js:

window.Gleam.initSearch({
  "items": [
    {
      "type": "module" | "type" | "function" | "constant" | "page",
      "parentTitle": "the name of the module or page",
      "title": "the title of the item",
      "doc": "the content of the item",
      "ref": "the relative link of the item, like 'module.html#title'"
    }
  ],
  "proglang": "gleam"
});

to generate search results in HexDocs of Gleam projects like:

and

`search-data.json`

for use in global HexDocs indexing:

{
  "items": [
    {
      "type": "module" | "type" | "function" | "constant" | "page",
      "parentTitle": "the name of the module or page",
      "title": "the title of the item",
      "doc": "the content of the item",
      "ref": "the relative link of the item, like 'module.html#title'"
    }
  ],
  "proglang": "gleam"
}

Option: 1 Model variant B

Pros:
- Nicer naming in the rust bits only.
- ...?
Cons:
- Abbreviated naming in js and json.
- Inconsistent naming between rust and js used in HexDocs search of Gleam projects making it harder to understand and navigate all affected bits throughout the code base.
- ...?

`docs.rs`

#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
struct SearchData {
    items: Vec<SearchItem>,
    #[serde(rename = "proglang")]
    programming_language: SearchProgrammingLanguage,
}

#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
struct SearchItem {
    #[serde(rename = "type")]
    type_: SearchItemType,
    #[serde(rename = "parentTitle")]
    parent_title: String,
    title: String,
    #[serde(rename = "doc")]
    content: String,
    #[serde(rename = "ref")]
    reference: String,
}

#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
#[serde(rename_all = "lowercase")]
enum SearchItemType {
    Constant,
    Function,
    Module,
    Page,
    Type,
}

#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
#[serde(rename_all = "lowercase")]
enum SearchProgrammingLanguage {
    // Elixir,
    // Erlang,
    Gleam,
}

outputs: same as variant A.

lpil · 2024-12-17T14:57:49Z

Hello! Sorry for the delay, my inbox is overwhelming!

I'm a bit confused by your notes here. Are you suggesting we propose a change upstream to Hex for them to change the format that all BEAM languages use? Personally I am happy with the format Hex is using so if the comments from my previous PR are applied then we can merge. Thank you

diemogebhardt · 2024-12-18T02:51:01Z

I give up, @lpil – our miscommunication is too frustrating.

Above, there are two questions that are still unanswered since the initial post of this PR:

Extra vs Page? Extra being the ExDocs/HexDocs way.
- If Extra, only for use in the model or also throughout the rest of the code base?
parent_title/parentTitle or does a better name come to mind?
- used in HexDocs search of Gleam projects
- NOT used in global HexDocs indexing of Gleam projects

Above, there are three options mapped out – including a draft implementation, and their pros and cons. Have you had a look at them and understand the impact of each? These are the three options to choose from:

Option: 2 Models
Option: 1 Model variant A (current implementation)
Option: 1 Model variant B

Feel free to close this unless you plan on addressing the above.

lpil · 2024-12-21T23:35:16Z

I would like the specific field renaming that I asked for above please 🙏 I am not looking to discuss any alternative designs presently.

If you would like to do just the tests and leave the renaming to me that would also be fine. Thank you

diemogebhardt · 2024-11-27T08:33:54Z

compiler-core/src/docs.rs

+enum SearchProgLang {
+    // Elixir,
+    // Erlang,
+    Gleam,
 }


Would you want to have the comments removed or kept for documentation?

diemogebhardt · 2024-11-27T08:55:48Z

compiler-core/src/docs.rs

+    #[serde(rename = "type")]
+    type_: SearchItemType,
+    #[serde(rename = "parentTitle")]
+    parent_title: String,


Do you have an idea of how to name this better?

See comment regarding this below.

diemogebhardt · 2024-11-27T09:11:57Z

compiler-core/templates/docs-js/index.js

        resultDoc.appendChild(resultDocTitle);
        let resultDocOrSection = resultDocTitle;
-        if (doc.doc != doc.title) {
+        if (searchItem.parentTitle != searchItem.title) {


This is the logic to render one of these two variations:

result 1

has parent_title set to gleam/string_tree

and title set to gleam/string_tree

result 2

has parent_title set to gleam/string_tree

and title set to string

diemogebhardt · 2024-11-27T09:29:58Z

compiler-core/src/docs.rs

+    Constant,
+    Function,
+    Module,
+    Page,


These are all of the SearchItemTypes being used.

@ruslandoga propsed Extra in #3811 (comment) for README.md and such things.

In the sidebar of the UI and the rest of the code it's Page. As this is not used for indexing, it might make more sense to stick to SearchItemType::Page.

See comment regarding this below.

Found more information regarding Extras:

here: https://hexdocs.pm/ex_doc/Mix.Tasks.Docs.html#module-configuration

:extras - List of paths to additional Markdown (.md extension), Live Markdown (.livemd extension), Cheatsheets (.cheatmd extension) and plain text pages to add to the documentation. You can also specify keyword pairs to customize the generated filename, title and source file of each extra page; default: []. Example: ["README.md", "LICENSE", "CONTRIBUTING.md": [filename: "contributing", title: "Contributing", source: "CONTRIBUTING.mdx"]]

and here: https://hexdocs.pm/ex_doc/Mix.Tasks.Docs.html#module-groups

extras: [ "guides/introduction/foo.md", "guides/introduction/bar.md", ... "guides/advanced/baz.md", "guides/advanced/bat.md" ]

groups_for_extras: [ "Introduction": Path.wildcard("guides/introduction/*.md"), "Advanced": Path.wildcard("guides/advanced/*.md") ]

After familiarizing myself more with HexDocs for Elixir and Erlang, I noticed that they use "Pages" as the headline in the sidebar too.

So it might make sense to stay consistent with others are doing in the BEAM ecosystem and adopt their naming: SearchItemType::Extra.

diemogebhardt · 2024-11-27T09:30:58Z

compiler-core/src/docs.rs

            title: config.name.to_string(),
-            content,
-            url: page.path.to_string(),
+            doc: content,


docs_pages: &[DocsPage], for page in docs_pages { let content = fs.read(&page.source).unwrap_or_default(); ... }

SearchItemType::Page seems more appropriate than SearchItemType::Extra.

Or refactor everything that's Page to Extra?

See comment regarding this above.

diemogebhardt · 2024-12-03T18:02:42Z

compiler-core/src/docs.rs

-    url: String,
+    doc: String,
+    #[serde(rename = "ref")]
+    ref_: String,


I agree with the naming and would prefer it too. Yet it is named differently in ExDocs so I went for consistency.

See here for the summarized ExDocs data format and more Info.

So shall we use reference internally and have a separate function that transforms the json output of search-data.json for HexDocs indexing?

diemogebhardt · 2024-12-03T18:02:48Z

compiler-core/src/docs.rs

    title: String,
-    content: String,
-    url: String,
+    doc: String,


I agree with the naming and would prefer it too. Yet it is named differently in ExDocs so I went for consistency.

See here for the summarized ExDocs data format and more Info.

So shall we use content internally and have a separate function that transforms the json output of search-data.json for HexDocs indexing?

diemogebhardt · 2024-12-03T18:02:55Z

compiler-core/src/docs.rs

+
+#[derive(Serialize, PartialEq, Eq, PartialOrd, Ord, Clone)]
+#[serde(rename_all = "lowercase")]
+enum SearchProgLang {


I agree with the naming and would prefer it too. Yet it is named differently in ExDocs so I went for consistency.

See here for the summarized ExDocs data format and more Info.

So shall we use SearchProgrammingLanguage internally and have a separate function that transforms the json output of search-data.json for HexDocs indexing?

diemogebhardt · 2024-12-03T18:04:14Z

compiler-core/templates/docs-js/index.js

        this.add({
          id: i,
+          // type: entry.type,


I added this comment to discuss, or keep it as a reminder that we have this property in the new data model but are not making use of it yet.

diemogebhardt · 2024-12-03T18:04:16Z

compiler-core/templates/docs-js/index.js

    const index = lunr(function () {
      this.ref("id");
+      // this.field("type");
+      // this.field("parentTitle");


I added this comment to discuss, or keep it as a reminder that we have this property in the new data model but are not making use of it yet.

diemogebhardt · 2024-12-24T00:03:58Z

Same here, @lpil – sorry! Glad we figured this out – I am relieved. You should be able to see my comments from weeks back now.

…xing of Gleam packages

…l in Rust

…ches ExDocs specification

…ation, in js

diemogebhardt · 2024-12-24T02:49:06Z

Thank you!! I've left a few small comments inline, mostly around abbreviations, which we avoid in this codebase.

Renamed the abbreviations. You should be able to see all my replies to your comments from weeks ago now.

It would be great to have a test for the output format also.

Added tests to ensure that the output matches the ExDocs specification.

diemogebhardt mentioned this pull request Nov 27, 2024

Hexdocs indexing Gleam packages #3811

Open

diemogebhardt force-pushed the refactor-docs-for-hexdocs-indexing branch from e7f6392 to fae2bc9 Compare November 27, 2024 09:27

lpil reviewed Dec 2, 2024

View reviewed changes

diemogebhardt force-pushed the refactor-docs-for-hexdocs-indexing branch from fae2bc9 to ab689e5 Compare December 4, 2024 16:39

diemogebhardt force-pushed the refactor-docs-for-hexdocs-indexing branch from ab689e5 to 9725204 Compare December 10, 2024 21:27

diemogebhardt commented Dec 23, 2024

View reviewed changes

Refactor docs to use ExDocs search_data model for global HexDocs inde…

94aa11e

…xing of Gleam packages

diemogebhardt force-pushed the refactor-docs-for-hexdocs-indexing branch from 9725204 to 94aa11e Compare December 24, 2024 00:19

Refactor docs to not use abbreviated names of ExDocs search_data mode…

cc0ba6f

…l in Rust

diemogebhardt force-pushed the refactor-docs-for-hexdocs-indexing branch from 3bef08f to cc0ba6f Compare December 24, 2024 01:20

diemogebhardt added 5 commits December 24, 2024 02:47

Add test to ensure serialized output of SearchData and SearchItem mat…

ac3c09a

…ches ExDocs specification

Add test to ensure search-data.json matches ExDocs specification

632b3eb

Add test to ensure search-data.js wraps json matching ExDocs specific…

8244c6b

…ation, in js

Add test for html content escaping in docs

5a49665

Add credits to CHANGELOG

4276496

Rename output file to search_data.json to match HexDocs specification

ee888b7

	proglang: SearchProgLang,
	programming_language: SearchProgLang,

Refactor docs to use ExDocs search_data model for global HexDocs indexing of Gleam packages #3904

Are you sure you want to change the base?

Refactor docs to use ExDocs search_data model for global HexDocs indexing of Gleam packages #3904

Conversation

diemogebhardt commented Nov 27, 2024 • edited Loading

Data Model

Output Files

search-data.js

search-data.json

lpil left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diemogebhardt commented Dec 3, 2024

lpil commented Dec 4, 2024

diemogebhardt commented Dec 6, 2024

diemogebhardt commented Dec 6, 2024 • edited Loading

Open Questions

Option: 2 Models

Option: 1 Model variant A (current implementation)

docs.rs

search-data.js

search-data.json

Option: 1 Model variant B

docs.rs

lpil commented Dec 17, 2024

diemogebhardt commented Dec 18, 2024

lpil commented Dec 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diemogebhardt commented Dec 24, 2024

diemogebhardt commented Dec 24, 2024 • edited Loading

diemogebhardt commented Nov 27, 2024 •

edited

Loading

`search-data.js`

`search-data.json`

diemogebhardt commented Dec 6, 2024 •

edited

Loading

`docs.rs`

`search-data.js`

`search-data.json`

`docs.rs`

diemogebhardt commented Dec 24, 2024 •

edited

Loading