Is it possible to just decode specific parts of a message?

[oliveryasuna]

Our API works by sending a serialized protobuf in the request/response body. No gRPC involved. We have a mobile app that is mainly a web view (using Capacitor). One of our endpoints has begun to return a large response (~30 MB). In turns out that iOS' WebKit has a longtime bug where, if the memory usage spikes, it can crash. So, when we attempt to deserialize the response body, the web view crashes.

We're discussing many potential solutions. One solution could be, if we could decode certain parts of a message at a time, we can avoid a large memory spike.

For example, consider the following message:

message MyResponse {
  repeated Foo foos = 1;
  repeated Bar bars = 2;
  repeated Baz bazs = 3;
}

Is there a way to just decode one-or-more fields at a time?

[stephenh]

Hi @oliveryasuna , no, not using ts-proto's MyResponse.decode methods.

However you might be able to pull this off using the lower-level @buf/protobufjs library, if you know the tags/schema/etc of the message you're wanting to read. 🤔

I.e. it has the lower-level methods like readInt / readMessage / etc. that would give the [tag, value] pairs.

Good luck!

[oliveryasuna]

Thank you.

[stephenh]

Hi @oliveryasuna sorry, I just read the first part of your message, and didn't see it the first time...

It kinda sounds like an "async" decode method would be useful for you, where instead of decode blocking, it would "do a little work", "yield to the JS engine", "do a little work", "yield to the JS engine".

Ironically this is basically what React's internals have started being able to do, is "break up" the "one big render of the DOM", into chunks of work that yield back to the browser every so often...

I think it'd be quite a big change to make a new "async decode", and have ts-proto do internal bookkeeping about how long it's blocked the browser thread...

But, just mentioning the idea in case you're sufficiently motivated to try it out :-)