[Experiment] WASM IPLD Codecs and ADLs #9016

aschmahmann · 2022-06-07T15:42:25Z

This PR is an experiment exploring how go-ipfs could leverage IPLD Codecs and ADLs written in WebAssembly.

Some pieces that are here that could be extracted having nothing to do with WASM:

Plugin interface for specifying new ADLs
Prototype of leveraging selectors on the HTTP gateway endpoint to render any IPLD node presenting as the Bytes kind as a file

This allows loading codecs and ADLs via the config file, e.g.

  "Plugins": {
    "Plugins": {
		"ipld-wasmipld" : {
			"Config" : {
				"Codecs" : [
					{
						"Code" : "bencode",
						"Encode" : true,
						"Decode" : true,
						"WasmPath" : "/foo/bar/bencode.wasm"
					}
				],
				"ADLs" : [
					{
						"Name" : "bittorrentv1-directory",
						"WasmPath" : "/baz/bt_dirv1.wasm"
					}
				]
			}
		}
	}
  }

The work describing how to make WASM code that's compatible is being explored in https://github.com/aschmahmann/wasm-ipld. ipld/wasm-ipld#2 is the latest draft.

You can just do something like cargo build --target wasm32-unknown-unknown --release in the wasmlib folder to generate the wasm blobs for inclusion and play around with it.

If you want to see custom codecs/ADLs in action it can be a bit of a pain since you have to actually write the data somewhere and then import it to go-ipfs (e.g. ipfs block put/ipfs dag put/ipfs dag import). If you want to see a koala picture inside a BitTorrent folder load over a go-ipfs HTTP gateway then:

Add each of the blocks in the fixtures folder to go-ipfs with SHA-1
Add the animals.infodict block to go-ipfs with SHA-1
Go to http://localhost:8080/ipfs/f01631114d55f4390b4e6f5c980ff06340beda9bddd6ff926?selector=bafyqanvbmf7keyj6ufqwnilcmy7kc2kln5qwyyjonjygpilbf2qgeyltozrgs5dun5zhezloor3dcllenfzgky3un5zhs
Profit (PS: you can also do ipfs dag get on that selector to see what it looks like in dag-json)

� Conflicts: � core/corehttp/gateway_handler.go � go.mod � go.sum

willscott

It'll be interesting to see how this setup performs when we end up with a wasm ADL of something like UnixFS where we have multiple recursive iterations of reification as we traverse down a directory tree. I'm worried that the jumping in/out of wasm for each ADL evaluation may add up pretty quick, but i bet in practice this setup works fine for a lot of IPFS use cases.

Very excited for this future! 🚀

willscott · 2022-06-09T10:11:43Z

core/coreapi/coreapi.go

+		if cl, ok := link.(cidlink.Link); !ok {
+			return nil, fmt.Errorf("cannot process link: %v", link)
+		} else {
+			block, err := api.blocks.GetBlock(linkContext.Ctx, cl.Cid)


I think there's regret around the structure we ended up here - with links in practice needing to be cidlink.Link but doing this check of it every time to pull out the Cid.

Instead, I think the pattern ipld-prime has been hoping to transition to is to use cid.Cast(link.Binary())

makes sense, I thought this pattern is what we've got at the moment. Also, wouldn't cid.Cast do parsing again?

willscott · 2022-06-09T10:14:01Z

plugin/plugins/wasmipld/spec.md

@@ -0,0 +1,209 @@
+---
+title: "WAC Specification"


what is 'wac'?

This will get extracted (and maybe a better name?) and made into an spec PR to the IPLD repo, but basically I created a new IPLD codec called WAC (webassembly codec). The name isn't really great, just named after the first use case I had in mind.

The idea is basically to have a simple codec that fits the IPLD data model, which both:
a) Gives us something to discuss about what the IPLD data model is
b) Means I can get data in/out of WASM losslessly without requiring tons of calls to extract each integer, string, byte array, etc. manually. I can copy entire nested maps of nested maps of lists and it's fine

ipld/ipld#226

willscott · 2022-06-09T10:15:22Z

plugin/plugins/wasmipld/wasmipld.go

+	registry = &wasmRegistry{}
+
+	for _, c := range cfg.Codecs {
+		wasm, err := ioutil.ReadFile(c.WasmPath)


how much more annoying would it be to reference the wasm as a cid in the block store rather than an on-disk path?

Wouldn't that lead to some kind of weird attacks, where you could create a format shows up differently based on some unrepeatable sideeffects ?

(I guess we could configure the WASM interpreter to be deterministic and repeatable (no random, or io available, ...)).

how much more annoying would it be to reference the wasm as a cid in the block store rather than an on-disk path?

Not that much more, I'd like to be able to do that. Some things I think we'd want to do to enable this include:

A way to protect the WASM blocks from getting cleared by GC (should be pretty easy)

At least for now assert a 2MiB limit on WASM blobs make that they must be Raw blocks

Let the plugin get access to a blockservice

willscott · 2022-06-09T10:15:39Z

plugin/plugins/wasmipld/wasmipld.go

+	return nil
+}
+
+const fuelPerOp = 10_000_000


maybe make configurable?

ya, makes sense. Just left it there for now, not sure what makes sense for limits. Could certainly allow limits to be per-module or something if people want to mess with things. Could also remove the limits by default 🤷

None of our codecs or ADLs have any sort of limits like this at the moment. It's worth considering what this means in terms of helping users make their code predictably deployable. Wouldn't want people to be surprised that some of their data loaded with implementation A, but not B due to limits here. Obviously people can do whatever they want as with block size limits, but we probably want some reasonably large safe boundaries to let people work with.

Jorropo · 2022-08-16T17:48:21Z

core/corehttp/gateway_handler.go

+		var fnBuildSel func(comps []string) (builder.SelectorSpec, error)
+		fnBuildSel = func(comps []string) (builder.SelectorSpec, error) {


Suggested change

var fnBuildSel func(comps []string) (builder.SelectorSpec, error)

fnBuildSel = func(comps []string) (builder.SelectorSpec, error) {

fnBuildSel := func(comps []string) (builder.SelectorSpec, error) {

Pretty sure the code won't compile if you do that since the function is recursive

let me check

Ok you are right, it's cursed to have a recursive virtual call to a function pointer if you aren't doing DP.

I would move this out as a private function on the global scope.

BigLep · 2022-09-01T01:56:05Z

While @aschmahmann is out, I'm not imagining anyone will be driving it forward. I think relevant parties are aware of the code in case it is relevant in the short term.

aschmahmann added 11 commits June 7, 2022 08:18

hack: load bittorrent files over gateways

eab36cc

� Conflicts: � core/corehttp/gateway_handler.go � go.mod � go.sum

ipldwasm

bcc3771

wasm with wac codec

27c9731

wac some adls

13122f9

wasm fuel + cleanup

3b51106

draft wac spec

6a2aaec

fix wac encoding, and switch for now use an already reserved code name

9bf4dd2

switch to separate wasmlib

61259ad

remove bittorrent specifics

b2ad3f5

make adls and codecs configurable from the config file

1d39402

point at public commit

b402ea4

willscott reviewed Jun 9, 2022

View reviewed changes

BigLep assigned aschmahmann Jun 14, 2022

aschmahmann mentioned this pull request Jun 28, 2022

spec: initial WAC spec ipld/ipld#226

Open

feat: use draft IPLD URI

89af104

aschmahmann mentioned this pull request Jul 22, 2022

Support CGO cross building in docker container ipfs/distributions#546

Open

Jorropo reviewed Aug 16, 2022

View reviewed changes

aschmahmann mentioned this pull request Aug 28, 2023

Plugings using wasm #10054

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Experiment] WASM IPLD Codecs and ADLs #9016

[Experiment] WASM IPLD Codecs and ADLs #9016

aschmahmann commented Jun 7, 2022

willscott left a comment

willscott Jun 9, 2022

aschmahmann Jun 17, 2022

willscott Jun 9, 2022

aschmahmann Jun 17, 2022

aschmahmann Jun 28, 2022

willscott Jun 9, 2022

Jorropo Jun 9, 2022 •

edited

Loading

aschmahmann Jun 17, 2022

willscott Jun 9, 2022

aschmahmann Jun 17, 2022

Jorropo Aug 16, 2022

aschmahmann Aug 16, 2022

Jorropo Aug 16, 2022

Jorropo Aug 16, 2022

BigLep commented Sep 1, 2022

		var fnBuildSel func(comps []string) (builder.SelectorSpec, error)
		fnBuildSel = func(comps []string) (builder.SelectorSpec, error) {

	var fnBuildSel func(comps []string) (builder.SelectorSpec, error)
	fnBuildSel = func(comps []string) (builder.SelectorSpec, error) {
	fnBuildSel := func(comps []string) (builder.SelectorSpec, error) {

[Experiment] WASM IPLD Codecs and ADLs #9016

Are you sure you want to change the base?

[Experiment] WASM IPLD Codecs and ADLs #9016

Conversation

aschmahmann commented Jun 7, 2022

willscott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jorropo Jun 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BigLep commented Sep 1, 2022

Jorropo Jun 9, 2022 •

edited

Loading