Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pipe operator support #146

Open
JeanMertz opened this issue Dec 7, 2020 · 5 comments
Open

Add pipe operator support #146

JeanMertz opened this issue Dec 7, 2020 · 5 comments
Labels
type: feature A value-adding code addition that introduce new functionality. vrl: syntax Changes to the syntax

Comments

@JeanMertz
Copy link
Contributor

JeanMertz commented Dec 7, 2020

Remap revolves around expressions, most expressions are implemented as functions, which take zero or more input arguments, and provide output.

For example:

trim(uppercase(to_string(.foo)))

The above will eventually (#4905) also work in string templates in Vector itself:

my_config_field = "{{ trim(uppercase(to_string(.foo))) }}"

The above works, but can be hard to read.

I wonder if we want to add support for the pipe operator (similar to Elixir).

.foo |> to_string |> uppercase |> trim
my_config_field = "{{ .foo |> to_string |> uppercase |> trim }}"

The gist of this syntax would be that given an expression, if the expression is followed by the pipe operator (|>), then the expression after that operator would receive the result of the previous expression as its first argument.

The precedence of the pipe operator would be higher than the other operators. This allows one to write:

.foo = .bar |> split |> contains("baz") || .baz |> to_string |> contains("qux") # bool

# similar to
.foo = ((.bar |> split |> (contains("baz")) || (.baz |> to_string |> contains("qux"))) # bool

You can still supply more arguments if need be:

"foo bar,baz" |> split(delimiter = ",") |> join(delimiter = "|") # "foo bar|baz"

This would be purely syntactic sugar to make the code easier to read, especially in string templates, but also in general.


There are still some design decisions to make (for example, what happens if you provide a non-function expression to the rhs of the pipe operator? Compile error?), but I wanted to put this issue up to gauge interest from others.

Also, some languages that focus specifically on templating use the | pipe syntax to do this:

.foo | to_string | uppercase | trim

We could do this, but then we'd lose the ability to implement bitwise operators in the future.

cc @binarylogic @FungusHumungus

@JeanMertz JeanMertz added the type: feature A value-adding code addition that introduce new functionality. label Dec 7, 2020
@binarylogic
Copy link
Contributor

I'm a fan. Although, I wouldn't describe Elixir's chaining syntax a solution for templating. For our use case I tend to lean towards:

.foo | to_string | uppercase | trim

But I'm just as happy with |>. And since this requires that the first argument take a path, how well do we currently conform to this? I'm getting the sense that we should formalize language design guidelines so that we can include details like this.

@JeanMertz
Copy link
Contributor Author

JeanMertz commented Dec 7, 2020

I wouldn't describe Elixir's chaining syntax a solution for templating

Sure 🙂 it's the same syntax (and the same semantics), it's just extra useful for our use-case when it comes to having single-line programs such as in string templates.

For our use case I tend to lean towards

Yes, as I mentioned in my proposal, I also like that more, but it would preclude us from introducing bitwise or operations, I'm not really sure how useful that is to have in Remap, though? Maybe @bruceg has some examples that would be helped by having bitwise operators? (we can of course always add functions to provide this instead of having explicit grammar)

since this requires that the first argument take a path

That's the example I gave, but it doesn't require a path as the lhs expression, it just requires any expression on the lhs, and a function expression on the rhs.

The grammar would be something like:

pipeline = { expression ~ "|" ~ function_call }

how well do we currently conform to this? I'm getting the sense that we should formalize language design guidelines so that we can include details like this.

I believe we have a very flexible and composable remap grammar. You should definitely take a look at our grammar.pest file, it's an easy read. We have a few more restrictions in the parser itself, but mostly this grammar file shows what's possible and what's not.

most of the time, we take an expression as input wherever we can (in function arguments, if-statements, etc), given how the pipe operator works (e.g. passing in the lhs value as the first argument of the rhs function), we would have an exception in this case, and only allow function calls for the rhs value (with a clear error message to help people out).

Other than that, 👍 on "formalize language design guidelines".

@bruceg
Copy link
Member

bruceg commented Dec 7, 2020

I'm a fan too. My only hesitation for the use of | is as mentioned, the ambiguity between its use as "bitwise or" in many languages, and || as "logical or" in TRL. This is a flow-control operation, not an "or". I don't have any examples where I know bitwise operators would be useful in TRL, though.

@lukesteensen
Copy link
Member

Do we not use | for coalescing? If we do this (I'm personally inclined to wait and see a bit longer), I'd lean towards |> for that reason alone.

@JeanMertz
Copy link
Contributor Author

Do we not use | for coalescing?

We do, but we only support path coalescing, and it requires the paths to be surrounded by parenthesis, so we're still free to use | in other contexts for other reasons.

Having said that, I could still see us deciding against using | for that reason, I agree.

I'm personally inclined to wait and see a bit longer

Agreed, there's no rush. It'll probably become more of a priority once we support TRL in template strings, but even then it's mostly to improve readability, not to unlock missing features, so there's no rush here.

@binarylogic binarylogic changed the title remap-lang: pipe operator support Add pipe operator support to Remap Jan 1, 2021
@JeanMertz JeanMertz added the vrl: syntax Changes to the syntax label Jun 7, 2022
@fuchsnj fuchsnj changed the title Add pipe operator support to Remap Add pipe operator support Mar 28, 2023
@fuchsnj fuchsnj transferred this issue from vectordotdev/vector Mar 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: feature A value-adding code addition that introduce new functionality. vrl: syntax Changes to the syntax
Projects
None yet
Development

No branches or pull requests

4 participants