Conjurer is a web app for designing audiovisual experiences for the Canopy of Luminous Conjury, a large LED art piece by The Servants of the Secret Fire.
You can think of Conjurer as an in-browser Digital Audio Visual Workstation, similar to a Digital Audio Workstation (DAW). Whereas a DAW is used to arrange and produce audio compositions, Conjurer lets you arrange audio and visuals into an "experience" which can be saved and played at a later time.
Note: see Onsite Setup for more detailed instructions if preparing for an event.
# use the correct version of node
nvm use
# install dependencies
yarn
# run the app with hot reloading on save
yarn dev
Conjurer should be running locally at http://localhost:3000.
- In this repo, patterns/effects at their core are just fragment shaders. They may seem scary at first, but with a proper introduction like in The Book of Shaders, you too could wield their considerable power!
- The shaders page contains more useful links for learning about shaders.
- See the How to make a pattern page if you are interested in creating a pattern or effect of your own!
- We use Chakra for our UI in this repo. Check out the available components here as well as the default theme
- We use MobX for state management. It's not Redux!
- We use ThreeJS and React Three Fiber to render the shaders/3D canopy.
- We use
react-icons
. Just search for what you want and import the icon from the correct place using the 2-letter prefix. - We use
recharts
to do some simple graphs. - We use
wavesurfer.js
for all of our audio needs.
- Pattern
- A fragment shader that generates a texture (an image) based purely on parameters (uniforms)
- This texture can either be rendered directly to the canopy or passed to an effect
- Effect
- A fragment shader that accepts a texture and applies an effect based purely on parameters, outputting a new texture
- Just like a pattern, this texture can either be rendered directly to the canopy or passed to an effect
- Note: Identical to patterns, except that effects accept a texture as an input
- Parameter
- This is a value that tweaks what is being generated by a pattern/effect
- "Color", "Fuzziness", "Radius" for example
- Parameter variations
- Changes over time applied to a pattern/effect parameter
- "Change the color from blue to green over 5 seconds"
Here is the zoomed out view of the architecture. Frame data is sent over websocket to the Unity app's websocket server. Ultimately this data is piped to the canopy and the canopy displays that frame.
graph
1(Conjurer frontend client)--websocket-->2(Unity websocket server)
2--magic-->3(Canopy)
- / - main page where experiences can be edited and viewed
- /viewer - view-only page
- /portal - view-only page that tells a story
- /playground - page for tinkering with patterns+effects, can be used to VJ
- /controller - controller page can control playground page (requires running controllerServer)
- /beatMapper - page for constructing a beat map for a song (work in progress)
- /test - test page for the embedded Conjurer viewer
Generates boilerplate for a new pattern called PatternName. Choose your own unique PatternName. It prints out the filepaths it writes, including the fragment shader and typescript pattern definition.
Starts the server that passes messages between conjurer and conjurer controllers.
Generates canopy geometry data and stores it in src/data/canopyGeometry.json
.
Starts a websocket server at port 8080 on localhost. For development use only, to mock the websocket server that the Unity app would run. Writes src/scripts/output.png
once per second.
Downloads all of the experience and audio files from s3 into the folder public/cloud-assets
. Conjurer can then read from these files when in "local asset mode", useful for situations when internet is not available. See section below for more details.
Use webpack analyzer to analyze the bundle. Will launch three tabs in your browser with bundle size details.
- Find your local IP address, and set
CONTROLLER_SERVER_WEBSOCKET_HOST
(websocketHost.ts
) to that address. - Run
yarn dev
, oryarn build && yarn start
. - Run
yarn controllerServer
. - Open http://localhost:3000/playground.
- On any device on the network, visit http://<IP_ADDRESS>:3000/controller.
You are good to go - when you change things with the controller, you should see the playground page update.
This is a big React app, hastily designed, that's doing a lot of expensive CPU/GPU things, so it has been difficult to keep it running smoothly. Here are some random thoughts associated with performance:
- You may encounter some memory leaks related to the hot module reloading/fast refresh when running the app locally. Just reloading or even hard reloading the tab in firefox doesn't free all of the used memory in my experience, so for the best results, if you have been changing code and the app has been hot reloading, you should periodically close the tab and open a new one.
- Memory leaks are apparently much easier to accomplish in React than I realized: https://schiener.io/2024-03-03/react-closures
- Careful with setTimeout chains/setInterval. Make sure there is a way to clean them up on hot reloads. useEffect is a good way to do this.
- If the app slows down a bunch, use the browser profiler to identify where it's spending time. If it's spending time in the garbage collector/cycle collector, it's likely a memory leak issue. At the time of writing, running locally with devtools open the app should use about 1GB of memory.
To dos are captured in the wiki, and occasionally are captured as issues.
Please do! This is a group effort, and any help is welcome.