As the game will be 2.5D, it is very important to make the 3D characters blend “seamlessly” with the 2D background. Here is how I did it.
The main contributing factors to the blend is the perspective, occlusion and lighting. Especially the shadows.
Shadows
For the shadows to render on the scene at all, we need to have the painting turned into 3D. The process was already explained in the previous post but here is some views from inside Unity.
When the scene is not viewed trough the camera, it is crazy stretched. This 3D version of the scene is very important. It will act as a shadow-catcher and naturally occlude the player when he is behind objects. It will also ensure that the characters are correctly scaled in the scene when they move in depth.
The 3D scene will also make it very easy to place lights in the space to light the character convincingly to match the painting.
URP pipeline in unity does not have a shadow catcher shader built in, but luckily it was not too difficult to create. Few hours of Googling and tenacious trial and error later, I had working shadows!
The Problem
But as the lighting is baked in to the AI generated 2D image, we can not simply render 3D shadows on top. As this looks odd. The specular highlights are darkened under the shadow. The already shadowed areas are shadowed even more.
This looks plain bad. But I had an idea. I already had to create a custom shadow rendering solution, so armed with this new skill I figured I could write a shader that has none of these issues.
The Solution
The solution is to hand paint a shadowed version of the background image. Then I can take the 3D shadow and use it to reveal the hand painted shadows on top of the background.
Overpainting the shadows is very fast in photo editing software like Photoshop. I just created a new layer, changed my brush to Darken-mode and used the color picker to select colors from near the highlights on the floor and painted away. The painting does not need to be perfect as the shadowed area usually is pretty small. Only paint the shadows where they need to be, do not darken the already shadowed areas. This yields the best results.
Fixing the shader to work with 2 images instead of darkening the surface was pretty trivial. Pasted here is my complete shader code. It still has some kinks I need to iron out: sometimes the shadow attenuation areas get confused and the shadows are rendered for the wrong lights. I intend to fix this soon.
Unity URP unlit shadow mask HLSL shader code:
Shader "URP Unlit Shadow masked"
{
Properties
{
_BaseMap("Base Map", 2D) = "white"
_ShadowMap("Shadow Map", 2D) = "white"
_ShadowRange("Shadow Range", float) = 50.0
}
SubShader
{
Tags { "RenderType" = "Opaque" "RenderPipeline" = "UniversalRenderPipeline" }
Pass
{
//Name "ForwardLit"
//Tags { "LightMode" = "UniversalForward" }
//Blend DstColor Zero, One One // multiply alpha source value by One instead of Zero to preserve alpha info
//Cull Back
//ZTest LEqual
//ZWrite Off
HLSLPROGRAM
#pragma vertex vert
#pragma fragment frag
#pragma prefer_hlslcc gles
#pragma exclude_renderers d3d11_9x
#pragma target 2.0
#pragma multi_compile _ _MAIN_LIGHT_SHADOWS
#pragma multi_compile _ _MAIN_LIGHT_SHADOWS_CASCADE
#pragma multi_compile _ _SHADOWS_SOFT
#pragma multi_compile _ _ADDITIONAL_LIGHTS_VERTEX _ADDITIONAL_LIGHTS
#pragma multi_compile _ _ADDITIONAL_LIGHT_SHADOWS
#pragma multi_compile _ LIGHTMAP_SHADOW_MIXING // v10+ only, renamed from "_MIXED_LIGHTING_SUBTRACTIVE"
#pragma multi_compile _ SHADOWS_SHADOWMASK // v10+ only
#pragma multi_compile_fog
#include "Packages/com.unity.render-pipelines.universal/ShaderLibrary/Lighting.hlsl"
#include "Packages/com.unity.render-pipelines.universal/ShaderLibrary/Core.hlsl"
struct Attributes
{
float4 positionOS : POSITION;
float2 uv : TEXCOORD0;
UNITY_VERTEX_INPUT_INSTANCE_ID
};
struct Varyings
{
float4 positionCS : SV_POSITION;
float3 positionWS : TEXCOORD3;
//float fogCoord : TEXCOORD1;
float2 uv : TEXCOORD0;
UNITY_VERTEX_INPUT_INSTANCE_ID
UNITY_VERTEX_OUTPUT_STEREO
};
// This macro declares _BaseMap as a Texture2D object.
TEXTURE2D(_BaseMap);
TEXTURE2D(_ShadowMap);
// This macro declares the sampler for the _BaseMap texture.
SAMPLER(sampler_BaseMap);
SAMPLER(sampler_ShadowMap);
CBUFFER_START(UnityPerMaterial)
// The following line declares the _BaseMap_ST variable, so that you
// can use the _BaseMap variable in the fragment shader. The _ST
// suffix is necessary for the tiling and offset function to work.
float4 _BaseMap_ST;
float4 _ShadowMap_ST;
float _ShadowRange;
CBUFFER_END
Varyings vert (Attributes input)
{
Varyings output;
UNITY_SETUP_INSTANCE_ID(input);
UNITY_TRANSFER_INSTANCE_ID(input, output);
UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(output);
VertexPositionInputs vertexInput = GetVertexPositionInputs(input.positionOS.xyz);
//output.positionCS = vertexInput.positionCS;
output.positionCS = TransformObjectToHClip(input.positionOS.xyz);
output.positionWS = vertexInput.positionWS;
//output.fogCoord = ComputeFogFactor(vertexInput.positionCS.z);
output.uv = TRANSFORM_TEX(input.uv, _BaseMap);
return output;
}
half4 frag (Varyings input) : SV_Target
{
UNITY_SETUP_INSTANCE_ID(input);
UNITY_SETUP_STEREO_EYE_INDEX_POST_VERTEX(input);
half4 color = half4(1,1,1,1);
half4 mainColor = SAMPLE_TEXTURE2D(_BaseMap, sampler_BaseMap, input.uv);
half4 shadowColor = SAMPLE_TEXTURE2D(_ShadowMap, sampler_ShadowMap, input.uv);
/*#ifdef _MAIN_LIGHT_SHADOWS
VertexPositionInputs vertexInput = (VertexPositionInputs)0;
vertexInput.positionWS = input.positionWS;
float4 shadowCoord = GetShadowCoord(vertexInput);
half shadowAttenutation = MainLightRealtimeShadow(shadowCoord);
// lerp from alpha 0 instead of 1 to have the mesh surface be fully transparent:
color = lerp(half4(1,1,1,0), _ShadowColor, (1.0 - shadowAttenutation) * _ShadowColor.a);
color.rgb = MixFogColor(color.rgb, half3(1,1,1), input.fogCoord);
#endif*/
#ifdef _ADDITIONAL_LIGHT_SHADOWS
VertexPositionInputs vertexInput = (VertexPositionInputs)0;
vertexInput.positionWS = input.positionWS;
float4 shadowCoord = GetShadowCoord(vertexInput);
int lightAmount = GetAdditionalLightsCount();
half shadowAttenutation = 1;
half product = 0;
half combined = 1;
half radius = 1;
half shadow = 1;
for (int i = 0; i < lightAmount; i++) {
Light light = GetAdditionalLight(i, vertexInput.positionWS);
//shadowAttenutation = AdditionalLightRealtimeShadow(i, vertexInput.positionWS, light.direction);
shadowAttenutation = (1 - AdditionalLightRealtimeShadow(i, vertexInput.positionWS, light.direction))*clamp(_ShadowRange * light.distanceAttenuation, 0, 1);
//shadow = clamp(1-shadowAttenutation,0,1);
//radius = clamp(50 * light.distanceAttenuation, 0, 1);
//combined = clamp(radius*shadow,0,1);
product = clamp(product + shadowAttenutation,0,1);
}
// lerp from alpha 0 instead of 1 to have the mesh surface be fully transparent:
color = lerp(mainColor, shadowColor, product);
//color.rgb = MixFogColor(color.rgb, half3(1, 1, 1), input.fogCoord);
#endif
return color;
}
ENDHLSL
}
}
}
This shader is not compatible with Unity 2022. I might upgrade the project at some point and try to fix it.
Black levels
In addition to all this, using the fog and the ambient scene lighting settings in Unity we try to match the scene black on the character as well as possible. This makes the character feel more like part of the scene.
As the background image is using an unlit material, it will not be affected by the fog or the ambient color at all. Only the realtime 3D characters are affected.
And there you have it! Next time, we might talk about animated interactions between the character and the painting.
Leave a Reply