Rakudo should ship with a usable REPL that can be easily extended by user code

[ab5tract]

The Problem

When spinning up raku for the first time, users are presented with a list of modules they should install in order to have such niceties as:

• backspaces
• left/right cursor movement
• up/down for scrolling through previous incantations

Note that there are no instrucitons for how to install these modules. No link to a package manager, let alone a package manager being provided.

This might not bother others but I find it to be deeply un-user-friendly and more than a little embarrassing for a language that bills itself as -- and generally lives up to -- "batteries included".

Beyond that pain point, by punting the responsibilities of parsing user input to userspace we also lose the ability to provide meaningful hooks into the REPL experience itself.

And it goes deeper still: Even our getc implementation requires a third-party module to be able to read a character of user input without an enter key being pressed.

Poterntial Remediations

Ship releases with zef

This would at least allow users to swiftly install one of the recommended options.

Pros

IMO this is long overdue. It likely deserves its own problem-solving ticket if it doesn't already have one.

Also, we apparently already do this for Windows. Aligning what we ship across platforms makes a lot of sense to me.

Cons

Doesn't do anything to address REPL extensibility / getc.

Ship releases with Terminal::LineEditor

Pros

Terminal::LineEditor is a very comprehensive line editor that is implemented almost entirely in Raku.

It supports significantly more than what I would consider the table stakes required for a usable REPL (my essentials are: backspaces, left/right cursor movement, and up/down with session history).

Cons

No direct cons except that poor getc is still incapable of collecting raw input.

Oh, and the fact that there are four dependencies for this module. More on that next.

Provide all the necessary pieces that are required to write Terminal::LineEditor in core

Shipping Terminal::LineEditor in core begs the question of why we don't just provide the functionalities provided by it's dependencies in core:

• Why don't we provide a way to switch on the terminal's raw mode, so that for example getc(:raw) would work? (This is provided by Terminal::MakeRaw in 55 LoC)
• Why don't we provide the ability for all Raku programs to easily interpret user input from terminal emulators that support ANSI sequences (eg, more or less all of them)? (Terminal::ANSIParser, ~300 LoC)
• To round out these offerings (and the dependencies required to ship Terminal::LineEditor anyway), why would we not offer a mechanism for modeling terminal capabilities, objects of which are constructed with the lowest common denominator and which could be used to standardize the handling of user expectations with Raku programs in their favorite terminal emulator (Terminal::Capabilities, ~50 LoC)

The only thing missing from a REPL usability standpoint is Terminal::ANSI (380 LoC, though some would be omitted because of overlap with Terminal::ANSIParser). This provides all the ANSI-standard pieces related to text output.

Pros

getc can finally do a thing that users likely assume it is/should be capable of.

Every terminal-specific Raku program can benefit. All libraries related to terminal interaction can drop up to 5 dependencies. Terminal::LineEditor can live in rakudo/lib.

We can design a basic, functional REPL around which many layers of interaction can be added *without requiring any third party libraries at all.

Cons

The main objection I have heard is related to maintanability. I'd like to note that the update frequencies of each module I have mentioned is very low, meaning that there seems to have been very few causes to modify any of these modules over a matter of years (Terminal::MakeRaw is working on Windows support, but it is unique here in that it is the only one that relies on native libraries.)

Terminal::MakeRaw is easy to implement because of NativeCall and requires more effort to bring into NQP where it would need to live in order to provide any benefit to getc.

[lizmat]

It would also be nice if the prompt logic could benefit from this.

[patrickbkr]

This might not bother others but I find it to be deeply un-user-friendly and more than a little embarrassing for a language that bills itself as -- and generally lives up to -- "batteries included".

I fully agree. Providing users a robust and generally good REPL by default is a must.

Beyond that pain point, by punting the responsibilities of parsing user input to userspace we also lose the ability to provide meaningful hooks into the REPL experience itself.

I'd be interested to understand the scope of a REPL and how far an extensible REPL can go. Is this mostly open ended? (I.e. we can't tell which interesting functionalities people come up with.) Architecture wise, which hooks does a REPL need to provide to enable the extensibility we dream of?

A somehow related question. Assuming I'd want to implement a REPL mostly in userspace, which hooks does the core need to provide so I don't hit a wall in such an effort?

And it goes deeper still: Even our getc implementation requires a third-party module to be able to read a character of user input without an enter key being pressed.

This sucks. But I'm still undecided on what the best remedy is.

Ship releases with zef

...
IMO this is long overdue. It likely deserves its own problem-solving ticket if it doesn't already have one.

Yes, having a package manager available by default is an absolute must.

Giving some perspective: All the binaries provided on rakudo.org have a zef (on all the platforms). By extension, all releases installed via rakubrew download have a zef preinstalled as well. If you do a rakubrew build you don't get a zef, but it's only a rakubrew build-zef away. The rakudo-pkg repos for multiple distos have a zef by default as well. Rakudo as packaged by the *nix distros don't have a zef included, but they usually provide a zef in a separate package. That's not something we can influence. Most distros have a policy to keep packages atomic, so bunding zef and rakudo is a no-go.
The only situation I know where you really end up without a zef, is when you manually build your own rakudo from source.

So all in all, I'd say we're not that bad on this front.

Ship releases with Terminal::LineEditor

...
Cons
...
Oh, and the fact that there are four dependencies for this module.

<backstory>
I, personally, like the idea to promote our ecosystem. There are some edges we still need to iron out (the fez eco is already there and one day we will merge #5060 or a reimagination of the same), but we have lexical imports and can deal with multiple versions of the same thing installed at the same time. We've all the foundations set to have a really epic ecosystem.
</backstory>

Given the installation times are acceptable, what's the disadvantages of having a smallish and manageably sized dependency tree?

Also I'd like to differentiate two approaches:

1. Shipping a module in core. (e.g. the Test module). These modules are effectively unversioned and we can realistically only make breaking changes on major Raku releases.
2. Bundling a module with our packaged releases. The module is not part of core, it's simply a module found in the ecosystem preinstalled. This is what e.g. Rakudo Star does extensively.

I believe we need to be very conservative with introducing modules into the core. The maintenance burden is very high, because introducing a breaking change is as hard as it can possibly get. On the other hand, bundling a module with our packages is really unproblematic. Other versions of the bundled module can be installed without issue. If we make clear that our packages are effectively a bundle, then there is no need to be backward compatible at all.

Provide all the necessary pieces that are required to write Terminal::LineEditor in core

Shipping Terminal::LineEditor in core begs the question of why we don't just provide the functionalities provided by it's dependencies in core:

* Why don't we provide a way to switch on the terminal's raw mode, so that for example `getc(:raw)` would work? (This is provided by `Terminal::MakeRaw` in 55 LoC)

This does indeed sound like a good candidate to have in core. There is a "but" though. I know little about the Linux terminal API. But from what I understand making a terminal "raw" is actually a process of changing about 5-10 individual and independent config options in the terminal config. I guess there are more than 100 different config options. Windows provides fewer options, but making a terminal "raw" is still some mixture of multiple options. We definitely don't want to provide the ability in core to change each of those. But is a single binary "raw" switch really all that we need? Is that set of options really the optimal set to define "raw"? Do we want to have a somehow finer grained control?

Pros

getc can finally do a thing that users likely assume it is/should be capable of.

I agree, that a "raw" getc is a thing that should work by default. But in general I feel we shouldn't be afraid of the ecosystem. Is it really that bad that a module developer that builds a terminal app needs to depend on a single (currently still imaginary) Terminal::WithAllBells'n'Whistles module that provides a really nice API for everything terminal?

Cons

The main objection I have heard is related to maintainability. I'd like to note that the update frequencies of each module I have mentioned is very low, meaning that there seems to have been very few causes to modify any of these modules over a matter of years (Terminal::MakeRaw is working on Windows support, but it is unique here in that it is the only one that relies on native libraries.)

I believe this might be a bit of a difficult metric. Up until recently, things have been rather quiet in the Raku terminal space. So that factors in to the low churn in these modules.

Also the API surface of these modules is pretty large. If I was tasked to bring these modules into a shape that I feel comfortable to keep stable for the next 10 years or so, I'd probably lock myself into a room for three months and still have a very bad gut feeling afterwards.

[patrickbkr]

My favorite way forward would look as follows:

• Get our current terminal software stack (Terminal::Print, Terminal::Widgets and all of their dependencies) free of external dependencies and working cross platform. (I started working on this a few days ago driven by the TUI debugger project.)
• Either bundle Terminal::LineEditor similar to how we bundle zef.
• Or put T::LE and it's dependencies into the core, but add measures to ensure none of the APIs are accessible from user code.

As a separate concern, I think it makes sense to think about which REPL specific extension hooks we want to provide in the core. Which hooks do we need to allow a full fledged REPL with all the bells and whistles and full integration into the language (e.g. the repl() sub) to be developed as a module?