Feature Request: integration with tree-sitter

[Inc0n]

With emacs 29, treesitter will be built-in. It may be a good idea to take advantage of that.

Here is the comment on a reddit post that inspired this request.

The idea is to add a variable (defcustom) perhaps named puni-forward-sexp-function, this variable will be funcalled in place of the current forward-sexp.

In treesitter's case a customized function that wraps treesit-node-next-sibling (See https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/treesit.el#n71) may work.

The benefits of this could introduce some more flexibilities. For example, support for issues such as this (#14) can be added by uses setting puni-forward-sexp-function in their org-mode-hook.

[Inc0n]

So what do you think about this, is my understanding correct on this, i.e. by making forward-sexp customizable, it would allow for using puni operations on a more custom sexp region. And possibly to make puni mode integrate with treesitter easily.

[AmaiKinono]

Great idea ;) I'll install Emacs 29 and have a try in a few days.

[AmaiKinono]

I installed https://aur.archlinux.org/packages/emacs-pgtk-native-comp-git and although treesit.el is installed, most of the tree-sitter functions are not available like treesit-next-sibling (I think they are defined in treesit.c).

Do I need some special build flags to enable tree-sitter support?

[Inc0n]

I think it's best I install emacs 29 also... right, you may be right, I found this post when installing on my mac: d12frosted/homebrew-emacs-plus#527.

Quote:

There's a new configuration flag --with-tree-sitter. Of course the library itself should be installed before, e.g. brew install tree-sitter. Thanks!

But I am not sure if this aur package support this flag,

Ok in the same post, this comment d12frosted/homebrew-emacs-plus#527 (comment) suggest the configuration script would pick treesitter up if the treesitter library is installed.
So before emacs29 installation, you just need to make sure treesitter is installed with your package manager.

[Inc0n]

I have gotten things setup on my end, but it's complaining (for python mode)

⛔ Warning (treesit): Cannot activate tree-sitter, because language definition for python is unavailable (not-found): (libtree-sitter-python.so libtree-sitter-python.dylib) No such file or directory

Maybe need to wait a bit before things mature enough to be usable.
It's not as straight forward with the little documentation they have.

[AmaiKinono]

I've got tree-sitter compiled but encountered a similar problem in c-ts-mode.

⛔ Warning (treesit): Cannot activate tree-sitter, because language definition for c is unavailable (not-found): (libtree-sitter-c libtree-sitter-c.so) No such file or directory

Maybe need to wait a bit before things mature enough to be usable.

I agree. Let's wait for a while.

[andreyorst]

The idea behind tree-sitter support, as far as I understand, is that Emacs knows how to talk to tree-sitter using the tree-siitter library, but parsers for specific languages have to be installed separately, via the tree-sitter CLI.

[andreyorst]

Also, I wonder if since puni already uses forward-sexp internaly, shouldn't it just work with tree-sitter OOTB since Emacs would (hopefully) utilize tree-sitter in its implementation of forward-sexp?

[Inc0n]

Also, I wonder if since puni already uses forward-sexp internaly, shouldn't it just work with tree-sitter OOTB since Emacs would (hopefully) utilize tree-sitter in its implementation of forward-sexp?

Oh, that would be very good indeed (haven't thought about that). However, upon inspecting the forward-sexp definition in my emacs 29, its not using tree-sitter in lisp.el. It may still be possible for them to integrate tree-sitter very easily using forward-sexp-function, which is the variable I described to generialize forward-sexp. It turns out using forward-sexp has numerous advantages!

[andreyorst]

Oh, that would be very good indeed (haven't thought about that). However, upon inspecting the `forward-sexp` definition in my emacs 29, its not using tree-sitter in `lisp.el`.

Right now there are only a few modes that support treesitter, all of them have -ts- in the name, e.g. c-ts-mode, typescript-ts-mode, and there's no such mode for lisp, at least yet. But it should be easy to add modes for tree-sitter-supported languages now, so I expect more third-party modes to appear soon

…

-- Andrey Listopadov

[andreyorst]

I've tried this for a bit, and as it seems in the current state there's no way to navigate trees in a general way with treesit.c. The treesit-node-next-sibling may work, however, I couldn't find a proper way to move to the nodes on the same level, e.g. skipping nested nodes. It may be possible, but it has to be developed.

So I guess we need to make our own version of forward-sexp that uses the tree-sitter internally, and if the tree-sitter is available for current mode, and properly setup we could just override the forward-sexp-function with the tree-sitter-based one, and thus support such languages as Lua or Elixir, which are extremely problematic in Smartparens due to mixing delimiters.

Maybe the https://github.com/emacs-tree-sitter/elisp-tree-sitter project has something like that implemented

[andreyorst]

A recent thread on the emacs dev. mailing list: https://lists.gnu.org/archive/html/emacs-devel/2022-12/msg00439.html

[andreyorst]

it's gonna be in Emacs 30: https://git.savannah.gnu.org/cgit/emacs.git/commit/etc/NEWS?id=207901457c018d94b1ce9e13a897d8241b1f3af2

treesit-forward-sexp and treesit-sexp-type-regexp

[mohammedzeglam-pg]

why not making treesit work with smie enigne

[AmaiKinono]

Hi. I'd like to ask everyone if you are using puni and treesit: How's your experience?

I've been using treesit-auto to map every supported *-mode to *-ts-mode for a while. Based on my experience on c++-ts-mode, css-ts-mode and cmake-ts-mode, it seems puni just works out of the box.

[andreyorst]

Doesn't seem to work properly in elixir-ts-mode.

[AmaiKinono]

@andreyorst Yes, I can confirm that. I don't know elixir so I just played with some example from the internet. From what I see, forward-sexp in elixir-ts-mode is doing very werid things, and I can't see how to fix that (or fix Puni for that).

Emacs master now provides treesit-forward-sexp, which requires the major mode to set treesit-sexp-type-regexp. If elixir-ts-mode could get this done, maybe treesit-forward-sexp can be a more reliable forward-sexp.

[andreyorst]

@andreyorst[https://github.com/andreyorst] Yes, I can confirm that. I don't know elixir so I just played with some example from the internet. From what I see, *forward-sexp* in *elixir-ts-mode* is doing very werid things, and I can't see how to fix that (or fix Puni for that).

Same for lua-ts-mode. Puni-convolute, puni-splice-killing-backward and others don't work properly.

Emacs master now provides *treesit-forward-sexp*, which requires the major mode to set *treesit-sexp-type-regexp*. If *elixir-ts-mode* could get this done, maybe *treesit-forward-sexp* can be a more reliable *forward-sexp*.

Yeah, I've mentioned this in this thread some time ago. However, simply setting forward-sexp-function to treesit-forward-sexp doesn't fix puni in these modes.

[alternateved]

I've noticed some issues with typescript-ts-mode as well (using unreleased Emacs 30). I often use my command +puni-kill (as I don't enable puni-global-mode globally, I just bind commands to keys and use them explicitly):

  (defun +puni-kill ()
    "Kill until end of a sexp."
    (interactive)
    (puni-delete-region
     (point)
     (cdr (puni-bounds-of-list-around-point))
     'kill))

and when normally that command kills until end of parentheses, then in typescript-ts-mode it would kill until the end of file which is pretty jarring.