Echoes of Somewhere

For scenes that have fluffy grass, I felt that the grass could be extra fluffy. Some sort of “fur” setup.

Because the grass is basically a post process effect, capturing the color of the rendered image and “smearing” it upwards, it matches the background image perfectly. Also the custom shadow shader that uses the hand painted shadow texture is replicated perfectly on the grass blades.

So the idea was that I would do the usual pipeline of modeling the location based on the fSpy camera solve of the 2D image. I would then paint grass on this mesh and apply my custom gbuffer reading material on the grass for capturing the underlying color and also whip up some nice wind vertex animations on it while I am at it.

For this to work I would need a shader that blends in seamlessly with the background image and occludes the characters and also receives shadow that matches perfectly with my custom hand painted shadow setup.

In my mind I imagined this to be easy! Just whip up an unlit material that reads the scene color buffer at the mesh pivot point! How hard can it be! It turns out it can be VERY hard.

The problems with grass world to screen space transform

The first issue to solve was to get a material to read the scene color from the opaque geometry of the scene – like a color picker in photoshop. To my horror, there was no transform node for world space position to screen space. I would have to write my own! This was a daunting task. It took me the whole day of trial and error to figure it out.

The camera projection is not flat. It is curved. To get the projected texture to scale properly, I would have to figure out a way to undo the fish-eye effect of the projected UVs.

The grass would need to sample a single point in the image on the ground level, If it sampled the pixel position, it would just be invisible. It would need to work like the smear brush in photoshop.

Scattering. I would have to scatter hundreds of individual grass blades on the location meshes.

The Solutions

And with the power of editing, I have now solved all of the shader issues and have a working shader here:

I am not going to be writing too much of the process of figuring out the very problematic world space to screen space position transformation, but I was pretty hopeless while working on it. This sort of stuff is way above my pay grade!

But I kept at it. Instead of working with the pivot positions, I made my life easier and just tried to create a material that displays the rendered image, pixel perfect, on 3D a mesh. I was so happy and exited every time I got some aspect of it working. First just to be able to read the opaque pass in the material, then to get the image in correct aspect ratio, make it scale based on the mesh position in z-space and finally removing the fish eye effect!

Color

To get the color of the ground plane, I modelled the grass with individual blades. I would then sample the object world space position, transformed to screen space, to get the color for the grass. I just bade sure the pivot of the grass blade was at the middle of the root. The grass cross section is triangular so that I would not have to care about the rotation of the grass in relation to the camera.

I figured as the material will be unlit, it would be pretty fast to render, so making the grass blades individually would not be too much of an issue.

I needed individual blades to get an accurate root position for each of the grass blades. The carpeted color will be the same for the whole mesh object, so if there were many blades of grass, they would all be the same color. To have each of the blades collared individually, I would need individual pivots for all grass blades.

To be honest, I could create a setup that samples the position coordinate of the grass blade at local 0 height, but it might add some unwanted streaking. If the individual blades are an issue, this is an optimisation I can take.

Scattering

First I tried to use Polybrush (Unity’s tool for painting prefabs on meshes) to paint these individual grass blades on the terrain. This was not ideal. To get a better coverage, I created a bundle of grass blades I could paint.

As the game is seen from one angle only, the grass does not need to have 100% coverage, as long as the game view looks appropriately fluffy.

Because the grass is basically a post process effect, capturing the color of the rendered image and “smearing” it upwards, it matches the background image perfectly. Also the custom shadow shader that uses the hand painted shadow texture is replicated perfectly on the grass blades.

Using the 3D projected texture of location as a guide, it is pretty easy to paint the grass on areas where the grass is in the 2D image.

Wind

I created a very basic noise based wind setup for the grass. The idea is to read 2 different noise maps using the world space X,Z position as an UV. This UV is then rotated using time. 2 noises are sampled to get high frequency wind and slower wind swells. Both of the noise maps are transformed so that they range from -0.5 to 0.5 so that the grass bends in both directions.

The wind speed parameter also generally bends the wind in it’s direction so that the wind has more of a direction visible in the grass.

In motion, the grass looks nice and windy. The nice part is that this is a completely unlit shader that only does some vertex deformations!