How to represent Declarator Docs in plain RakuDoc

[patrickbkr]

Prelude:

Toolchain wise we've long had the issue that tools to display documentation (GitHub, GitLab, raku.land, ...) have an issue with running the Rakudo parser on source code to reach for the embedded docs as that code can contain BEGIN blocks, which is a big safety concern.

A possible way forward is to split the task of rendering RakuDoc (and Declarator Docs in particular) into two phases:

1. A tool, the Extractor extracts and converts all docs in a Raku source file and outputs a pure .rakudoc file. This is an unsafe procedure.
2. A separate tool, a plain RakuDoc parser, then parses that pure .rakudoc file. It barfs on anything that is not plain RakuDoc. This is a safe process.

Platforms that wish to render RakuDoc (e.g. GitHub) can safely use the RakuDoc parser. Platforms that want to render the full documentation (e.g. raku.land) can run the Extractor in a sandbox.

All of the above is not part of the issue this ticket wants to address.

---

The question I want to discuss is: What should the output of the above mentioned Extractor that can convert a .rakumod file into a .rakudoc file look like?

The issue is mostly orthogonal to #438 in that it is not concerned with how Declarator Docs should look in code, but how they should be represented in RakuDoc.

[patrickbkr]

Just to get the discussion started, here is the first thing I was able to come up with.
Given this piece of code in a file called foo/bar/frobnicator.rakumod:

#| The class C<Foo::Bar::Frobnicator> is the one stop shop for all your frobnication needs.
class Foo::Bar::Frobnicator is Affector[I] does Helping {
    #| Perform a frobnication.
    method frobnicate(
        Str() $thingy = "something", #= The thing you want to frobnicate.
        :$quiet                #= Do it quietly.
    ) { ... }
}

#| User expertise levels.
enum Level
    ONE, #= Beginner.
    TWO, #= Expert.
;

#| Default config values.
my %config =
    size docs "How big should the buffer be?" => 5,
;

The output in a file foo/bar/frobnicator.rakudoc might be:

=begin dock :type("class")
=           :name("Foo::Bar::Frobnicator")
=           :code("class Foo::Bar::Frobnicator is Affector[I] does Helping")
=           :file("foo/bar/frobnicator.rakumod")
The class C<Foo::Bar::Frobnicator> is the one stop shop for all your frobnication needs.
=end dock

=begin dock :type("method")
=           :name("frobnicate")
=           :code("method frobnicate(Str() $thingy = \"something\", :$quiet)")
=           :file("foo/bar/frobnicator.rakumod")
=           :context(class => "Foo::Bar::Frobnicator")
Perform a frobnication.
=end dock

=begin dock :type("parameter")
=           :name("thingy")
=           :code("Str() $thingy = \"something\"")
=           :file("foo/bar/frobnicator.rakumod")
=           :context(class => "Foo::Bar::Frobnicator", method => "frobnicate")
The thing you want to frobnicate.
=end dock

=begin dock :type("parameter")
=           :name("quiet")
=           :code(":$quiet")
=           :file("foo/bar/frobnicator.rakumod")
=           :context(class => "Foo::Bar::Frobnicator", method => "frobnicate")
Do it quietly.
=end dock

=begin dock :type("enum")
=           :name("Level")
=           :code("enum Level")
=           :file("foo/bar/frobnicator.rakumod")
User expertise levels.
=end dock
=begin dock :type("literal")
=           :name("ONE")
=           :code("ONE")
=           :file("foo/bar/frobnicator.rakumod")
=           :context(enum => "Level")
Beginner.
=end dock
=begin dock :type("literal")
=           :name("TWO")
=           :code("TWO")
=           :file("foo/bar/frobnicator.rakumod")
=           :context(enum => "Level")
Expert.
=end dock

=begin dock :type("variable")
=           :name("config")
=           :code("my %config")
=           :file("foo/bar/frobnicator.rakumod")
Default config values.
=end dock
=begin dock :type("key")
=           :name("size")
=           :code("size")
=           :file("foo/bar/frobnicator.rakumod")
=           :context(variable => "config")
How big should the buffer be?
=end dock

Immediate questions:

• Which syntactic elements do we allow Declarator Docs on? (Declarator Docs should be limited in scope #438 might provide the answer for that.)
• Which components of some piece of code do we want to expose as separate metadata keys and which do we not want to expose? E.g. thingy could have :positional(0), :default("something"), :sig-type("Str()") and so on. How much is too much?
• Would the above be enough to provide a decent rendering of the above Declarator Docs?
• I have the feeling that :context and :file can not be as simple as I make them be in the above example. Are there conditions in which the naive approach above fails?

@finanalyst I believe you currently have the deepest insight into implementing renderers. Thus the ping.

[tbrowder]

I forgot but just found that back in 2020 we did get the ability to keep the leading declarator block in its original form with use of an environment variable RAKUDO_POD_DECL_BLOCK_USER_FORMAT. That implementation was the quickest and easiest way to affect Raku parsing.

When that is true, and you run 'raku --doc ...', you will get text from the leading declarator blocks as input by the author of the code.

I used that for documenting my code. I hope that is retained and improved for RakuAST.

The test for it is in roast file 'S26-documentation/block-leading-user-format.t'. I

[lizmat]

It is retained if you use $=pod.

[lizmat]

What I implemented for --rakudoc now:

#| before subset
subset Positive of Int where * > 0; #= after subset

#| before enum
enum Frobnicate <Yes No>;  #= after enum

#| before regex
my regex zippo { z }  #= after regex

#| before block
{ … } #= after block

#| before pointy block
{ … } #= after pointy block

#| the main class
class A {  #= that we have

    #| a method before
    multi method a( #= method after
      #| parameter before
      Int:D $foo #= parameter after
    ) {
        #| a variable
        my $a = 42; #= with the answer
    }
}

produces:

=begin doc-subset :name<Positive>
  =leading before subset
  =trailing after subset
=end doc-subset

=begin doc-enum :name<Frobnicate>
  =leading before enum
  =trailing after enum
=end doc-enum

=begin doc-regex :name<zippo>
  =leading before regex
  =trailing after regex

=end doc-regex

=begin doc-block
  =leading before block
  =trailing after block

=end doc-block

=begin doc-block
  =leading before pointy block
  =trailing after pointy block

=end doc-block

=begin doc-class :name<A>
  =leading the main class
  =trailing that we have

  =begin doc-method :multi :name<a>
    =leading a method before
    =trailing method after

    =begin doc-parameter :name<Int:D $foo where { ... }>
      =leading parameter before
      =trailing parameter after
    =end doc-parameter

    =begin doc-variable :name<$a>
      =leading a variable
      =trailing with the answer
    =end doc-variable
  =end doc-method
=end doc-class

[lizmat]

Note that this was just a first attempt and no way intended to be cast in stone... Very much open to improvements :-)

[patrickbkr]

@lizmat It's a nice surprise you actually started working on it already. I did not see that coming. You've implemented support for quite a list already. Nice!

The first elements that come to mind I still miss are: doc-sub, doc-constant, doc-has, doc-token, doc-rule, doc-grammar

Questions:

• Is there any point in putting a Dock on a block? I can't imagine a plain block to be part of an API. It's especially strange given that blocks have no name or other identifier by which they could be referred to.
• Same can be asked for my variables. No, it makes perfect sense - e.g. for IDE support for maintainers.
• Nesting the hierarchies is nice. Are there cases where this might not be easily possible? E.g. When having a class spread multiple files using augment?
• Can we have the literal code of the respective element in there? (What I put in :code() in my attempt.)
• Semantically, do we want to give #| and #= different meanings? If not, then we can get rid of =leading and =trailing and just have the contents directly in there.

[lizmat]

I still miss: doc-sub, doc-constant, doc-has, doc-token, doc-rule, doc-grammar

Hmmm looks like decks on constants are currently broken.

Re: doc-sub these all fall out of generic Routine handling

#| before sub
sub foo() { #= after sub
}

produces:

=begin doc-sub :name<foo>
  =leading before sub
  =trailing after sub

=end doc-sub

But it looks like submethod is being rendered as "doc-method" instead of "doc-submethod"

Looks like has is being rendered as "doc-variable" instead of "doc-has". Hmmmm...

"doc-token" and "doc-rule" are already being rendered correctly, as they are a special case of "doc-regex".

"doc-grammar" and all other package types, are already being rendered.

[lizmat]

Is there any point in putting a Dock on a block?

There are spectests for it.

[lizmat]

Same can be asked for my variables. No, it makes perfect sense - e.g. for IDE support for maintainers.

And there the line between internal and external documentation blurs. So are decks for external or internal documentation?

[lizmat]

Nesting the hierarchies is nice. Are there cases where this might not be easily possible? E.g. When having a class spread multiple files using augment?

All RakuDoc is based on a RakuAST tree. However you build such a tree, is open to debate. I guess one could join multiple ASTs from different files into a single AST. Then it all depends on the scoping of such a merge.

[lizmat]

Can we have the literal code of the respective element in there? (What I put in :code() in my attempt.)

Yes, you could. But this could become quite a lot. And wasn't it the point to not have any code in safe RakuDoc?

[lizmat]

Semantically, do we want to give #| and #= different meanings? If not, then we can get rid of =leading and =trailing and just have the contents directly in there.

Sure, we could do that.

[lizmat]

rakudo/rakudo@27565cc1f7 now renders attributes as doc-attribute.

[lizmat]

rakudo/rakudo@8162f3eb3b now renders submethods as doc-submethod.

[patrickbkr]

The point is to not have any Raku code that needs to be parsed. But a string containing some Raku code (i.e. no different from a =code block in RakuDoc) is not a problem. My feeling is, that the code (especially for routines - the declarator) carries a lot of meaning in itself and would help a lot to have it available in the docs. For each Signature parameter, the ':' tells if it's a named arg, the '!' tells, if it's mandatory, the order of positional args is clear, the types tell what you're meant to pass in, ... I think that's the reason why [in many places](https://docs.raku.org/type/Proc/Async) in the official Raku docs the routine declarators are shown. Alternatively we could expose all the different bits of information separately. In the case of Signature parameters that might look like: =begin doc-parameter :name<foo> = :mandatory = :type<Int:D> = :container<Scalar> = :kind<Positional> = :position(0) = :where(...) = :aliases(...) = :default(...) parameter before parameter after =end doc-parameter It's definitely more work to provide the information separately instead of just giving `:code<Int:D $foo>`, but could possibly simplify the job of the consumers of this data. In the end most renderers will want to display something like `Int:D $foo` to the user. In my ideal world I'd love to have a cleaned up version of the code where comments are removed, whitespace is normalized and stuff like `name($!bind-var)` is cleaned up (i.e. that `!` should be removed).

…

On September 22, 2024 1:01:39 PM GMT+02:00, Elizabeth Mattijsen ***@***.***> wrote: > Can we have the literal code of the respective element in there? (What I put in :code() in my attempt.) Yes, you could. But this could become quite a lot. And wasn't it the point to **not** have any code in safe RakuDoc? -- Reply to this email directly or view it on GitHub: #439 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

[patrickbkr]

That's a question that is difficult to answer. There is clearly a use case for both. Question is, which of the use cases do we want to support? My feeling is that explicitly excluding one of the two use cases diminishes the usability. I would say, let's just leave it unspecified for now, provide a generic tool and see how it's used and what works and what doesn't. If we find, that there is a need to separate the two uses more strongly, we can adjust later on.

…

On September 22, 2024 12:58:38 PM GMT+02:00, Elizabeth Mattijsen ***@***.***> wrote: > ~~Same can be asked for my variables.~~ No, it makes perfect sense - e.g. for IDE support for maintainers. And there the line between internal and external documentation blurs. So are decks for external or internal documentation? -- Reply to this email directly or view it on GitHub: #439 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>