RFC: renderer plugins

[eyelidlessness]

This may be premature, but Astro offers almost everything I’ve wanted in a web [meta-]framework:

• Component oriented dev experience
• Supports JSX
• Built with established/emerging industry standard tooling
• Trivial partial hydration
• Many batteries included for varying build preferences
• Component library-agnostic
• At a glance observation: great defaults, lots of clear thought went into providing wonderful DX without imposing build tool config hell

I also think it’s well positioned to check the one large-ish remaining checkbox on my list:

• Actually bring your own framework

I already mentioned this to Nate: I want to try this with Solid (I’d love to @ Ryan here but I’m not sure how that would work in a private repo, but this project should definitely be on his radar), another performance-obsessed library supporting JSX, but currently lacking support for partial hydration.

I thought I was going to spend most of my initial early access time tinkering with using this clever bit of machinery. But I did what I always do and read through most of the source. (It’s some of the best I’ve seen in a web context by the way!)

As I would’ve hoped, the current BYOF offering works by abstracting each respective library’s interfaces behind a common one. This is an excellent design for the current set of supported component libraries.

My naïve thought seeing that was “oh, adding support for Solid would likely be trivial-ish and a great first contribution!” Taking a little hammock time led to: this design also provides a good foundation for a plugin architecture.

Proposal

1. Define the existing renderer abstractions as an explicit interface.
2. Identify any other renderer-specific internal behavior.
3. Design clear, consistent interfaces for known-common patterns.
4. Extract anything identified in this process into renderer plugins, included in Astro by default (or as optional peer dependencies).
5. Provide and document an experimental/alpha API for external renderers.
6. Implement support for Solid (or any other library the Skypack team might want to include) on that API as an example/exercise of the initial design.
7. Red green refactor 🚀

Risks

This is a big ask for a young project and a small team with a lot going on. It feels like a good fit for the project, but I’d be remiss if I didn’t recognize the risks I see upfront:

• Like I said: young project, small team. Taking on any major unplanned feature work likely takes away from planned feature work. (That said, y’all have a great reputation at this point of shooting for the stars)
• Supporting arbitrary renderers might be a bad API. New libraries might come along with novel concepts that don’t fit and either the API grows a bunch of hard to rest ? types or those libraries get disappointing second-class treatment.
• A “contrib”-first ecosystem can put a huge burden on maintainers. Lots of projects have abandoned this model because it was too onerous and caused a lot of confusion when third party dependencies varied in quality.
• The intersection of the last two: supporting arbitrary renderers might be burdensome for something likely to be niche.

That said:

✋ I wanna help

This project is something I want to use, support and promote. I’m big on contributing to open source. I’d be happy to participate in this effort (and others as they come to my attention) from a personal investment perspective alone.

[eyelidlessness]

Shamelessness

I’m also currently unspoken for in an employment sense—exploring some prospects but no one has yet made an honest man/person of me—and while frontend problems isn’t especially one of my passions, boldly working on problems that could impact millions of people in countless thankless ways is definitely one of them. If I can pay my bills, keep my pup healthy and fed, and work on stuff like this, I’d be pleased as pie to devote more than weekend hours to it.

---

Wow that was the weirdest, most unexpected cover letter I’ve ever written and published semi-publicly.

[matthewp]

Thanks for filing the issue. Our thoughts so far on how to allow other frameworks have fallen into 2 places:

1. Just build in as many frameworks as possible. If the cost to adding one is low and maintenance isn't a big issue, this could be the way to go. Still too early to say. Issues with dependency versions is worrisome for this idea.
2. Some sort of API like you're suggesting. I think we aren't currently 100% sure if the way we do frameworks now is the long term solution. But it's probably good enough for now.

I'm going to propose something like:

export default {
  experimentalFrameworkPlugins: [ { type: 'solid', Renderer } ],
  extensions: {
    '.solid': 'solid'
  }
}

The extensions config already exists, so it would just be a matter of allowing externally defined renderers. Easy technically, if people are ok with us saying "this is experimental, we might change it", then I think it's a good path for now.

Also note that for something like Solid the plugin would need to provide a Snowpack plugin to compile the files to JS.

[eyelidlessness]

That's pretty much the interface I had in mind, give or take some naming. I'm gonna poke around and see if I can get something like this working.

[matthewp]

@eyelidlessness I'm going to confirm with the rest of the core team that this API is fine with them. One thing to note with your experimentation is that you need a Solid snowpack plugin as well. You can bring your own snowpack config file in Astro and we'll load it. Something like this might work: https://github.com/ravupad/solid-snowpack-starter/blob/master/snowpack.config.json

However this probably means you can't use other jsx frameworks side-by-side. It would be nice if we could get something figured out in the future so that you can. You currently can't use a React and Preact component in the same app for the same reason.

[eyelidlessness]

@matthewp I should push up the stuff I have so far when I get to my desk. Even if it’s all throwaway prototyping it’s mostly types, but I had expanded the current interface to include a compile method for experimental renderers rather than couple plugins. This might at least help inform that discussion.

That said I’d also recommend bringing up previous discussions with @natemoo-re about making Microsite itself a Snowpack plugin, and whether Astro could be implemented that way too. There’s a certain amount of build tool config sprawl fatigue, and consolidating on one tool could be a big DX improvement. That might also be a better model for BYOF.

However this probably means you can't use other jsx frameworks side-by-side. It would be nice if we could get something figured out in the future so that you can. You currently can't use a React and Preact component in the same app for the same reason.

I haven’t tried but couldn’t you just choose custom extensions for each? This is something I saw in one of the screenshots.

I’m not sure how much of a use case there is for mixing React/Preact, they’re so nearly compatible, but I could definitely see one for mixing Solid with one of the others for compatibility reasons.

[natemoo-re]

Regarding Snowpack integration, we're using Astro as a chance to dogfood Snowpack's JS API. Astro also does significantly more internal work than Microsite. We're keeping tool fatigue in mind but Astro is shaping up to be it's own thing at the moment for many reasons.

Regarding React/Preact can we currently set .react.jsx and .preact.jsx as the extensions? Not sure if compound extensions are supported. Having a "React" mode that is just preact/compat seems like a good option as well.

I built the current renderer interface as a first pass but I haven't gotten the time to dig into this proposal just yet! I'll add more thoughts when I do.

This is tangential, but part of the issue with the current approach is requiring every framework to be a direct dependency of Astro. I wonder if we completely sidestep that with remote-next mode?

[natemoo-re]

I hesitate to recommend peerDependencies given the upcoming auto-installation in npm@7.

[matthewp]

@eyelidlessness That's a good idea! Using .preact.jsx and .react.jsx is something that presumably a user can do. I'm not sure if that works today or not, worth investigating to see if it does. I'm on board with that being a temporary solution.

[eyelidlessness]

Regarding Snowpack integration, we're using Astro as a chance to dogfood Snowpack's JS API. Astro also does significantly more internal work than Microsite. We're keeping tool fatigue in mind but Astro is shaping up to be it's own thing at the moment for many reasons.

I don’t want to overly push back on a project that’s new to me but you’ve lived and breathed for much longer, but if I may push back a bit... this may be a good way to dogfood and identify where the Snowpack config the rest of us use doesn’t serve use cases like this.

One of my biggest frustrations with all of the current popular build tools is that configuring them for anything moderately complex can require jamming together a lot of tools with opaque or sprawling APIs spread out over many files. This is not only mentally taxing, it also makes it hard to understand what’s happening, and when. On the bright side this is what led me to get neck deep in Microsite and start contributing, but a lot of people understandably won’t want to do that.

If there were one place to configure the whole build, including explicitly specifying which steps the user expects to perform before and after Astro does its thing, this would go a long way toward making all of that much easier to understand and maintain. And that kind of flexibility would also probably improve both Astro and Snowpack.

Regarding React/Preact can we currently set .react.jsx and .preact.jsx as the extensions? Not sure if compound extensions are supported. Having a "React" mode that is just preact/compat seems like a good option as well.

I had actually been thinking more along the lines of .react etc, but that probably doesn’t play well with all the tooling one might encounter. This definitely feels safer.

This is tangential, but part of the issue with the current approach is requiring every framework to be a direct dependency of Astro. [...] I hesitate to recommend peerDependencies given the upcoming auto-installation in npm@7.

This sort of suggests to me that it might be better to make all of the renderers use whatever the ultimate plugin interface turns out to be, installed separately from the Astro core.

Then each such plugin could specify its peerDependencies with the confidence that any user installing the plugin explicitly wants to use its associated framework. It would also help decouple Astro from renderer-specific logic in the few places where it’s not already decoupled.

[matthewp]

We definitely don't want to force the first step of installing Astro to be installing some other framework plugins. React, Vue, Svelte, and Preact being builtin is pretty crucial. I'd rather add more frameworks to core rather than remove any we currently have.

[amoutonbrady]

Happy to see that there are already some Astro X Solid talk going. The two sound definitely like they could be a nice match together.

As per the Snowpack plugin requirement, I can probably work that out, based on the work we've done for vite. That being said, I maintain a somewhat popular solid template for Snowpack over here and the beauty of it, is that it depends mostly on the Snowpack integration of Babel since that's what compiles down the JSX to regular JS for Solid via babel-preset-solid.

@eyelidlessness If you have any question regarding the Solid part of the integration, feel free to poke us around on our Discord, you can find the link in our Github repo. Would a snowpack plugin that wraps Babel help you out?

As per the general consensus of that discussion, it seems pretty alright so far with the little I know about Astro so far. Having a .solid.jsx or .solid.tsx doesn't sound too bad though a bit redundant. I guess if we can make that opt-in, only for those that want to mess with different JSX components.

That's my two cents at the moment.

/cc @davedbase

[eyelidlessness]

@matthewp

We definitely don't want to force the first step of installing Astro to be installing some other framework plugins.

Fair enough. For what it’s worth, this doesn’t have to be an additional step. Many projects providing optional functionality include single-step instructions for installing the core and whichever optional things in one command, or a create script which does this for you.

If that’s still not the desired DX, I think there’s still benefits to implementing the built in frameworks against the same plugin interface that third party plugins would use, particularly in terms of decoupling and maintaining a clear interface between Astro and the frameworks.

[eyelidlessness]

@amoutonbrady

If you have any question regarding the Solid part of the integration, feel free to poke us around on our Discord, you can find the link in our Github repo.

Thanks! I joined the other day and was happy to see some discussion of Astro there too! It really does feel like a good fit.

Would a snowpack plugin that wraps Babel help you out?

Possibly! I think it depends on the interface design here. In the proposed design above I think it will need to call Babel programmatically. I have some prototype code (which I really should push up) that may be most of the way there. It’s pretty straightforward, the only complexity really is a second step where I use the Babel AST to get the compiled imports, which Astro needs to know about (presumably to prevent compiling them out).

[matthewp]

@amoutonbrady For clarity, the discussion here is scoped towards unblocking people who want to use a non-builtin framework. In no way do I think that this proposal is the ideal API. It's being proposed because it's close to how Astro internally handles frameworks. The .solid.jsx thing would lonly be needed if you wanted to use multiple jsx frameworks.

Otherwise it would probably be something like this:

import SolidRenderer from 'astro-solid';

export default {
  experimentalFrameworkPlugins: [
    {
      type: 'solid',
      Renderer: SolidRenderer
    }
  ],
  extensions: {
    '.jsx': 'solid'
  }
}

Then a snowpack.config.js that handles running Solid's babel plugin. That's all we need here. For now the user would bring their own snowpack.config.js, so using your starter app is perfect. In the future we'll figure out how to have the plugin provide that.

[amoutonbrady]

Yeah ok that's what I understood as well, thanks for clarifying. That config sounds really solid and easy to process from a user perspective actually. I totally dig it!

[eyelidlessness]

Then a snowpack.config.js that handles running Solid's babel plugin. That's all we need here. For now the user would bring their own snowpack.config.js, so using your starter app is perfect. In the future we'll figure out how to have the plugin provide that.

Oh. In my prototype WIP I’ve defined a plugin interface extending the renderer interface to add a compile method. If the snowpack approach is preferred that makes for a smaller change. But I do still think a single config would be a better DX long term.

[natemoo-re]

Just catching up on this one!

@eyelidlessness Leveraging Snowpack plugins to handle compilation is what we'd prefer! The "Astro renderer plugin" interface could possibly define which Snowpack plugin(s) the renderer relies on and Astro could automatically inject those plugins.

Not a final decision, but we've been leaning towards Snowpack being an invisible implementation detail. So the "configuration" DX would be to configure Astro if necessary.

Fair enough. For what it’s worth, this doesn’t have to be an additional step. Many projects providing optional functionality include single-step instructions for installing the core and whichever optional things in one command, or a create script which does this for you.

This is a good idea. I think at this point we're more comfortable building renderers into core because we want to make it really easy for everyone to try everything out. When we launch publicly, I imagine our implementation will look much more like what you're suggesting—a nice "start project" CLI and less built-in to core with a formal Renderer plugin interface.

If you're just hoping to kick the tires on Solid support, does the current (not ideal) interface work? If it's missing something, I'm very open to suggestions. Hopefully building this out can inform what the official renderers should look like.

[eyelidlessness]

Leveraging Snowpack plugins to handle compilation is what we'd prefer! The "Astro renderer plugin" interface could possibly define which Snowpack plugin(s) the renderer relies on and Astro could automatically inject those plugins.

This makes sense. One nit (and it’s one that already applies to Snowpack), I’d greatly prefer to be able to provide a plugin instance rather than string | [ string, object ]. The latter makes it so much easier to provide invalid options.

Not a final decision, but we've been leaning towards Snowpack being an invisible implementation detail. So the "configuration" DX would be to configure Astro if necessary.

This is definitely good to hear. I’ll reiterate that I think it would be good if the interface for this is an extension of the Snowpack config. A lot of newer tooling is converging on similar config semantics, a very welcome development. It would be great to be able to transfer my familiarity with snowpack.config to this project from the same team built with the same tooling.

This is a good idea. I think at this point we're more comfortable building renderers into core because we want to make it really easy for everyone to try everything out. When we launch publicly, I imagine our implementation will look much more like what you're suggesting—a nice "start project" CLI and less built-in to core with a formal Renderer plugin interface.

For what it’s worth, I’m less invested in the idea of the built in renderers being extracted into separate dependencies, and more invested in first party and third party renderers having the same interfaces.

The former would benefit anyone sensitive to dependency bloat by alleviating the peerDependencies concerns you brought up before. But I’m not especially sensitive to that, I only brought up that possibility because the concern was raised.

The latter would benefit everyone involved:

• It would ensure Astro is strictly a solution for partial hydration of arbitrary frameworks, totally decoupled from those frameworks and their implementation details. This benefits your team the most, as the explicit boundaries between Astro and BYOF help ensure framework-specific details don’t find their way into core. If you or anyone on the team haven’t seen the linked talk I highly recommend it, it’s a wonderful foundation for thinking about how to design interfaces for things that are complementary, paired, and could someday be separated. (It’s also just great overall for thinking about abstraction design.)
• It would ensure any plugin ecosystem that might emerge is just as empowered and just as limited as the built-in/contrib frameworks. This benefits your team and third party plugins roughly equally. For your team, you get the benefit of expanding the set of third parties invested in the platform’s strengths and contributing to the framework integration story. For third parties, we get to follow the same path for our preferred framework without having to work around or reverse engineer whatever privileged built in behavior might not be exposed to plugins.
• It makes errors and lapses in documentation much less likely. This benefits your team especially as it might expand, but it benefits the ecosystem (and gives it room to grow) immensely. One of the biggest pain points of integrating with pluggable frontend tooling and providing anything of value is that getting to know how to start means learning your way around stuff that assumes you know the internals and can be just plain out of date or wrong because documentation is a secondary consideration. Dogfooding the plugin API safeguards against that.

If you're just hoping to kick the tires on Solid support, does the current (not ideal) interface work? If it's missing something, I'm very open to suggestions. Hopefully building this out can inform what the official renderers should look like.

No, it’s actually quite good! And this is coming from someone who’s obviously a stickler on the topic and spends more time than I’d like quietly or privately complaining about interfaces I have to deal with. My goal starting to kick the tires was “can I bring Solid?” My goal now is “can I help shape a solution where the next weirdo after me wants to bring something else, and it’s good for everyone involved?”

I think it’s clear there’s buy in for BYOF with a plugin system. That’s awesome. Mostly I’m looking for some clarity on the interface out of the gate (which is pretty clear now, at least to start it can match the internal interface with compilation handled by a Snowpack plugin), and some understanding of interest in the core/contrib renderers using the same mechanisms as third party ones will.

[drwpow]

Thank you all for the thoughts and feedback! We are launching this soon, with documentation being written now. Would love to continue getting feedback on this API as we move forward.