Hexdocs indexing Gleam packages

[ruslandoga]

👋

We're working on adding global search to https://hexdocs.pm and we'd like to index Glean but there are some questions:

• Is Gleam's search_data.js format stable?
• Could Gleam adapt to ExDoc's search_data format?
  • and if not, how best to map Gleam's search_data to ExDoc's format

Relevant discussion: hexpm/hexdocs#44 (comment)

[lpil]

Hello there! What a cool initiative!

It's not stable or a public interface, no.

We could possibly start using that format. Is there documentation for it somewhere? To ensure we use it correctly.

[ruslandoga]

As far as I know there is no spec or documentation for it, but maybe we could add one (cc @josevalim)

Here's what I think it could be (only covering the parts important for indexing):

{
  "items": [
    {
      "type": "module" | "function" | "type" | "callback" | "macro" | "macrocallback" | "extra" (this one is for readmes, guides, etc.) | etc.,
      "title": the title, like "application" in https://hexdocs.pm/lustre/lustre.html#application or `monitor(Socket)` in https://www.erlang.org/doc/apps/kernel/inet.html#monitor/1,
      "doc": the contents,
      "ref": the relative link, like "Example.html#test/4"
    }
  ],
  "proglang": "elixir" | "erlang" | "gleam"
}

[ruslandoga]

To illustrate it a bit more, the differences from the current Gleam format (taking lustre as example) are:

{
    "doc": "lustre",
    "title": "lustre",
    "content": "# Lustre examples directory\n\nEach of the examples in this directory is a self-contained, complete Gleam app\nthat demonstrates a particular feature or concept of the library. For newcomers,\nwe recommend looking through them in order, as each example tends to build on\nthe previous ones. Feel free to jump in to any example that interests you, though!\n\n> **Note**: these examples all use [`lustre/ui`](https://github.com/lustre-labs/ui)\n> to show off something a little more visually interesting than unstyled HTML. None\n> of the ideas in these examples are specific to `lustre/ui`, though, and you should\n> know that you can follow along with any of these examples using only the standard\n> `lustre/element/html` module.\n\n## Examples\n\n- [`01-hello-world`](https://github.com/lustre-labs/lustre/tree/main/examples/01-hello-world)\n  is a simple example to just get something on the screen.\n\n- [`02-interactivity`](https://github.com/lustre-labs/lustre/tree/main/examples/02-interactivity)\n  introduces the core Model-View-Update loop that underpins every Lustre application.\n\n- [`03-controlled-inputs`](https://github.com/lustre-labs/lustre/tree/main/examples/03-controlled-inputs)\n  demonstrates the most common way to handle `<input />` elements in Lustre.\n\n- [`04-custom-event-handlers`](https://github.com/lustre-labs/lustre/tree/main/examples/04-custom-event-handlers)\n  shows you how to write your own event handlers and decoders, instead of relying\n  on the ones provided by `lustre/event`.\n\n- [`05-http-requests`](https://github.com/lustre-labs/lustre/tree/main/examples/05-http-requests)\n  demonstrates how side effects are handled in Lustre, using the third-party\n  [`lustre_http`](https://hexdocs.pm/lustre_http/) package.\n\n- [`06-custom-effects`](https://github.com/lustre-labs/lustre/tree/main/examples/06-custom-effects)\n  builds on the previous example and shows you how to write your own side effects\n  for Lustre to perform.\n\n- [`07-routing`](https://github.com/lustre-labs/lustre/tree/main/examples/07-routing)\n  shows how to use [`modem`](https://hexdocs.pm/modem/) to set up routing and\n  navigating between pages in a Lustre app.\n\n## Getting help\n\nIf you're having trouble with Lustre or not sure what the right way to do\nsomething is, the best place to get help is the [Gleam Discord server](https://discord.gg/Fm8Pwmy).\nYou could also open an issue on the [Lustre GitHub repository](https://github.com/lustre-labs/lustre/issues).\n",
    "url": "reference/examples.html"
}

1. item type seems to be missing
2. content is not "partitioned" by headers, i.e. in ExDoc each new heading (##) in the @doc and co. creates a new (linkable) search item, e.g. https://hexdocs.pm/phoenix/Phoenix.Channel.html#module-topics-callbacks is a separate search item; in the lustre example above there are some headings (## Examples), but it's within the same search item. This is not really important to search but might feel inconsistent
3. different field names, but that's minor

[lpil]

Thank you.

What would we use for type for Gleam concepts that do not fit into those Elixir categories?

What's ref? It says that it's a relative link but the URL has some Elixir syntax instead of a URL.

[ruslandoga]

What would we use for type for Gleam concepts that do not fit into those Elixir categories?

I think it's OK to extend it and use whatever name is appropriate, it would just need to be internally consistent within proglang=gleam. Right now it's not used, but it could be used to filter out what the user is interested in (functions, types, etc). It could also be used to boost certain types (like getting functions results might be more generally interesting than extra or macrocallback).

What's ref? It says that it's a relative link but the URL has some Elixir syntax instead of a URL.

I think it's just whatever goes after the library name, e.g. https://hexdocs.pm/phoenix/{ref}. Seems like it's already the same as Gleam's url, it's just ExDoc includes module name and function name and function arity in the refs and titles:

• https://hexdocs.pm/elixir/1.17.3/Atom.html#to_string/1
• https://www.erlang.org/doc/apps/kernel/inet.html#monitor/1

[lpil]

Great. Let's adopt that format and rename that file to search-data.json to make it easier to parse (removing the little JavaScript shim).

[diemogebhardt]

I had a go at this in PR #3904.

@ruslandoga: Did you have the chance to document this in the meantime?

[ruslandoga]

👋 @diemogebhardt

No, sorry. We haven't documented the format yet.