Learning from Shader Program Traces

Eurographics, to appear, April 2022

Yuting Yang, Connelly Barnes, Adam Finkelstein

Learning to extrapolate from the partial computation of a procedural shader called Venice. The reference solution (a) results from a full computation at 1000 samples per pixel (SPP). A simplified version of the shader provides an approximate solution (b) using only 1 SPP and less computation per sample (72% of the original compute; 1400x speedup overall). Our RGBx baseline method (c) learns to approximate the reference well for much of the image, based on only the RGB output of the simplified shader as well as a few hand-picked auxiliary features – but exhibits artifacts in the distance (obvious in the zooms boxed in green). This paper shows that difficult learning tasks like this can benefit from relying on not just the RGBx features but also the program trace (a record of the intermediate values computed at every pixel, in this case that of the simplified shader) – producing a more faithful approximation of the reference. Percent (%) denotes mean perceptual error relative to that of the RGBx baseline, averaged over the test set.

Deep learning for image processing typically treats input imagery as pixels in some color space. This paper proposes instead to learn from program traces of procedural fragment shaders -- programs that generate images. At each pixel, we collect the intermediate values computed at program execution, and these data form the input to the learned model. We investigate this learning task for a variety of applications: our model can learn to predict a low-noise output image from shader programs that exhibit sampling noise; this model can also learn from a simplified shader program that approximates the reference solution with less computation, as well as learn the output of postprocessing filters like defocus blur and edge-aware sharpening. Finally we show that the idea of learning from program traces can even be applied to non-imagery simulations of flocks of boids. Our experiments on a variety of shaders show quantitatively and qualitatively that models learned from program traces outperform baseline models learned from RGB color augmented with hand-picked shader-specific features like normals, depth, and diffuse and specular color. We also conduct a series of analyses that show certain features within the trace are more important, and even learning from a small subset of the trace outperforms the baselines.

Paper (preprint)
Code (github)
Video: Short (5 min)
Video: Long (20 min)

Yuting Yang, Connelly Barnes, and Adam Finkelstein.
"Learning from Shader Program Traces."
Eurographics, to appear, April 2022.

@inproceedings{Yang:2022:LFS,
   author = "Yuting Yang and Connelly Barnes and Adam Finkelstein",
   title = "Learning from Shader Program Traces",
   booktitle = "Eurographics, to appear",
   year = "2022",
   month = apr
}