Lanterna magica

Operators and wires. As in TouchDesigner, from whose vocabulary the names are borrowed, a patch is a network of operators. Image operators (TOPs, in Derivative’s words) make or change a picture: noise, shapes, text, the camera, blur, feedback, a kaleidoscope. Wire one’s output, on its right, into another’s input, on its left; the last goes into Out, which the large viewer shows. Channel operators (CHOPs) are numbers that move: a slow wave, a tempo, the mouse, the loudness of the room, a MIDI knob, the pinch of your fingers.

Formulas instead of wires. Channels are not wired to images: any number of any operator can become a formula with ƒ, such as 0.2 + audio1.low * 0.3 or sin(time) * 0.5, and the editor draws a dashed line from the channel to what it moves. Or drag a channel operator’s green dot onto another operator and choose the parameter. TouchDesigner uses Python for this; here the formulas are parsed by a small grammar, never evaluated as code, so a shared network cannot run anything on your computer.

On the graphics card. Every image operator is a WGSL shader, the language of WebGPU, run once a frame into its own half-float texture, so light above white survives for glow and feedback. A Feedback operator outputs the last frame of a later operator: the loop behind tunnels, trails and smoke. Particles are moved by a compute shader, half a million at once, along curl noise, a field without sources or sinks, so they swirl rather than bunch (Bridson, Hourihan and Nordenstam, 2007). Reaction–diffusion is Gray and Scott’s pair of chemicals, which John Pearson showed in 1993 could grow spots, stripes and mazes from almost nothing; it runs a dozen steps a frame. Noise is Ken Perlin’s gradient noise; Cells is Steven Worley’s.

No limits but the card’s. Resolution goes from 640 × 360 to 4K and beyond, in any shape; the output records straight to a video file, plays full screen, and a whole network fits in a link.

Everything that makes sound here is computed on the visitor’s own graphics card, like the images; nothing is streamed. Press ♪, or click anywhere in the window, and the network plays.

Cellular automata as scores. The Cellular automaton operator runs a Life-like rule on the GPU, one cell per block of pixels: Conway’s Game of Life (B3/S23, Gardner’s column in Scientific American, October 1970), HighLife with its replicator, the symmetric Day & Night, Seeds, Maze. The Cell sequencer cuts any image into a grid and sweeps a playhead across it in time; every bright cell under the playhead sounds a note of a scale, higher up the grid, higher in pitch. Its grid is computed on the GPU and read back each frame; the notes are scheduled a little ahead on the audio clock, so they land on the sixteenth. Iannis Xenakis composed with cellular automata in Horos (1986); Eduardo Miranda’s CAMUS (1993) turned Life into counterpoint. Here the automaton is also what you see.

Pictures as sound. In 1957 Yevgeny Murzin built the ANS synthesizer in Moscow: a composer scratched marks on a glass plate covered in mastic, and light through the scratches drove hundreds of pure tones, one per height on the plate; Eduard Artemiev scored Tarkovsky’s Solaris with it. The Spectral synth does the same with any image in the network: a line sweeps across it, and each of 256 rows, spaced logarithmically from low to high, is a sine whose loudness is the brightness under the line. All 256 are summed for every one of 48,000 samples a second in a WebGPU compute shader, each sample independent of the others, which is what a graphics card is for.

And back. The Spectrum operator turns sound into an image, Lanterna’s own or the microphone’s: a scrolling spectrogram, bars, a waveform. So the music can be fed back into the picture that makes it. TheAudio channel operator gives its loudness, lows, mids, highs and onsets as numbers for any formula.

The Track operators use Google’s MediaPipe (Lugaresi et al., 2019), small neural networks that run in the page on the graphics card, about thirty times a second, on the camera’s image. Nothing is sent anywhere, and nothing costs.

• Hand: twenty-one joints per hand, from a palm detector and a landmark model (Zhang et al., 2020). Lanterna derives the centre, the index tip, the pinch and how open the hand is.
• Face: 478 points on the face’s surface (Kartynnik et al., 2019) and 52 blendshapes, the weights an animator would use: a smile, an open jaw, closed eyes, raised brows.
• Body: BlazePose’s 33 joints (Bazarevsky et al., 2020): the head, both wrists, their spread.
• Person: the selfie segmenter, which says for every pixel how likely it belongs to a person, to cut you from the room.

Each becomes channels, to move anything, and Landmarks draws what the trackers see. The models load the first time an operator needs them, a few megabytes each.

Make turns a sentence into a network: the model is given the catalogue of operators, their parameters and ranges, and the rules of wiring, and answers with the network as data, which Lanterna checks (known operators, values in range, no wires into loops) before building it. Change sends the network on screen with your request.New shader asks for WGSL and compiles it; if the compiler objects, its messages go back to the model, up to three times, a loop of the kind Chen and colleagues found makes models markedly better at code (“Teaching Large Language Models to Self-Debug”, 2023).

Three ways to reach a model, all free:

• Through this site. A small server on Heroku holds the site’s key to OpenRouter, which routes a request to whichever of its free models is available. It accepts only free models, so it costs nothing; it does have a daily share, which visitors divide. For its first month the same route also offers GPT-5.4 mini, on Microsoft Azure in the EU, paid from the site’s Azure credit, a hundred and twenty requests a day.
• With your own OpenRouter key, free to make, kept in this browser and sent only to OpenRouter. Free models allow about fifty requests a day, a thousand once an account has bought ten dollars of credit. The list of free models changes; Lanterna reads it live.
• In this browser, with WebLLM (Ruan et al., 2024), which runs a language model on WebGPU: Qwen2.5-Coder (Hui et al., 2024) at 1.5, 3 or 7 billion parameters, downloaded once (one to five gigabytes) and then private and offline. Smaller than the hosted models, so better at shaders than at whole networks.

If a paid model is ever wanted, OpenRouter charges what each provider charges, without a markup on the tokens; a whole network is a few thousand tokens, a fraction of a cent on the cheaper models.

TouchDesigner, by Derivative in Toronto, is the instrument of most installations and stage visuals of the last twenty years. Its free licence is for non-commercial use, at up to 1280 × 1280; an Educational licence is $300, Commercial $600, Pro $2,200. It runs on Windows and macOS.

So: better where a browser is better, in price, reach, sharing, and in an AI that knows its operators; not yet in depth. The 3D, Python and the professional connections are what would come next.