Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ feat: Add streaming support #125

Merged
merged 47 commits into from
Dec 8, 2024
Merged

✨ feat: Add streaming support #125

merged 47 commits into from
Dec 8, 2024

Conversation

wanjohiryan
Copy link
Member

@wanjohiryan wanjohiryan commented Nov 3, 2024

This adds:

  • Keyboard and mouse handling on the frontend
  • Video and audio streaming from the backend to the frontend
  • Input server that works with Websockets

Update - 17/11

  • Master docker container to run this
  • Steam runtime
  • Entrypoint.sh

Copy link

cloudflare-workers-and-pages bot commented Nov 3, 2024

Deploying nestri with  Cloudflare Pages  Cloudflare Pages

Latest commit: 67569dd
Status: ✅  Deploy successful!
Preview URL: https://09539f22.nestri.pages.dev
Branch Preview URL: https://feat-stream.nestri.pages.dev

View logs

wanjohiryan and others added 10 commits November 11, 2024 16:31
I've finished cooking 🍝

---------

Co-authored-by: DatCaptainHorse <[email protected]>
…ging (#129)

Adds a functional NESTRI_PARAMS env variable to container image,
allowing usage like:
```docker run -e NESTRI_PARAMS='--no-audio=true --verbose=true'```

Also adds a no-audio option to help debug stream issues.
Also also, changes the container builder portion to use the actual main nestri repo.

---------

Co-authored-by: DatCaptainHorse <[email protected]>
@wanjohiryan
Copy link
Member Author

wanjohiryan commented Nov 14, 2024

I just added the first part of handling websockets thru a relay, partykit

It works locally, i have yet to deploy the server side end of things

FIXME: I think I overwrote the changes done by DatHorse in the prior branch :(
TODO:

  • Make sure it works with the remote Cloudflare partykit server(s)
  • Test the latency, whether it has improved or dropped

UPDATE: This was scraped, too much overhead :/

wanjohiryan and others added 4 commits November 15, 2024 13:23
… WHIP output (#130)

I feel things can be improved still, however rather than making perfect
CLI parameter handling, gonna PR to help move on.

WHIP output option added for starters, we may need to write our own
WebRTC element based off examples and API from:
https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/tree/main/net/webrtc

---------

Co-authored-by: DatCaptainHorse <[email protected]>
Co-authored-by: Wanjohi <[email protected]>
wanjohiryan and others added 11 commits November 17, 2024 10:31
- Adds heartbeat to keep WS alive on nestri-server side
- Fixes inputs getting stuck on repeated keys, while also allowing
modifiers to be sent after.
- For now treat Home keys as Esc to make games escapable.
- Add button to start stream, to fix browser preventing play issue.

Co-authored-by: DatCaptainHorse <[email protected]>
This adds functionality to:
1. Broadcast input to the server using WebRTC Data Channels
2. Change some terms for easy understanding `stream -> room`, `viewer =>
participant`
3. Remove input functionality currently relying on Cloudflare's D3 and
move it to our relay

---------

Co-authored-by: Kristian Ollikainen <[email protected]>
Co-authored-by: DatCaptainHorse <[email protected]>
@wanjohiryan
Copy link
Member Author

Update

  • We are using WebRTC for streaming video and audio from the server-side to the client-side. (Yeap, we dropped QUIC)
  • We are using WebRTC data channels to share everything else - input, docker commands etc etc

NOTE: No ports were opened during this operation B)

@wanjohiryan
Copy link
Member Author

TODO

It has been reported that input has a lot of latency, there might be many causes of this but here's my two cents:

  • JSON might be "too heavy" to send, so maybe use ArrayBuffers or protobuf
  • We could optimise data-channels to use unordered streams
  • Handle input much better in the Golang relay, need to investigate on how many milliseconds we lose there
  • Check nestri-server and adjust buffers accordingly - need to investigate the latency here as well

DatCaptainHorse and others added 8 commits November 28, 2024 13:53
…tor connection (#137)

Using the same WebRTC connection as DataChannel inputs, though switched
it to WHIP etc.

Draft since I'm still looking into some stuff.

---------

Co-authored-by: DatCaptainHorse <[email protected]>
Add latency trackers, tidy things into separate files, improve code
behaviour all around but especially in `nestri-server`, removal of
buffering and usage of mutex locking allows to push 120FPS easily again.

Time to see if DOOM is now playable with the hugely increased
responsiveness..

---------

Co-authored-by: DatCaptainHorse <[email protected]>
@wanjohiryan wanjohiryan merged commit 379db1c into main Dec 8, 2024
1 check passed
@wanjohiryan wanjohiryan deleted the feat/stream branch December 8, 2024 11:54
@Blast545
Copy link

@wanjohiryan Can I ask you why QUIC was dropped? I'm just curious, I was taking a look to Rust projects using any kind of moq

@wanjohiryan
Copy link
Member Author

Hello 👋🏾 @Blast545

We dropped MoQ temporarily because it was slowing us down - it is not mature yet.

For example; audio is not working as expected. It is choppy, and when it is not choppy - it is mono channel.

Also, it has very limited support on Firefox and Safari. (Webcodecs and Webtransport have very limited browser support)... WebRTC on the other hand is more widely supported.

I am not against MoQ, and I hope when we get Nestri out of the door, we will come back and help out where we can.

But for now we will be going with the more mature WebRTC

@Blast545
Copy link

Thanks for your answer! I was getting the same feeling when I tried MoQ a week ago

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants