With the pipeline I currently had set up, it is easy to add 3D characters on top of 2D images with shadows and interactive lighting. But what about reflective surfaces? Let’s go!

Final results with reflections on the 2D background image

The workflow for reflective locations is by and large the same as for the other locations. With some added texture requirements and steps.

The creation of this 4 room spa location took me around 6 hours from scratch, including the prompt process. So the speed gains that can be had using AI for the locations are absolutely insane.

Breakdown

Scene workflow breakdown video

Table of contents:

  • Prompting
  • Preparing the image layers
  • 3D modeling
  • Planar reflection shader

Prompting

As always, we start with prompting in Midjourney. II try to keep the prompts simple and just do multiple generations to get something that works. For this bath scene, I generated 40 different pool areas to land one that had my requirements met: pool with walkable area behind, but not extending all the way to the camera and multiple doors to walk trough.

I also generated other locations, such a a lobby, exterior and a pump room.

cyberpunk dirty abandoned outdoors pool, multiple doors, wide angle, hdr masterpiece adventure game –ar 5:3

cyberpunk dirty abandoned spa front entrance exterior wide angle, hdr masterpiece adventure game –ar 5:3 –seed 4046812885

cyberpunk dirty abandoned spa front entrance exterior wide angle, hdr masterpiece adventure game –ar 5:3 –seed 4046812885

cyberpunk dirty abandoned spa pump engine utility room, broken, garbage, wide angle, hdr masterpiece adventure game –ar 5:3 –seed 4046812885 –no pool

The original image

Preparing the image layers

For the reflections and the lighting to work, we would need 4 versions of the background image.

The first version is the “base pass” a version of the original image with all of the reflections more or less painted away. As we will be adding realtime reflections on top of the surface, we meed this version to avoid having double reflections. I also want the character’s reflection to properly occlude bright areas from the reflection, so this pass is a must.

The second pass is the good old shadow pass. This is a version of the image that will be masked by the realtime shadow of the character. This does not need to be perfect, as only a small portion of it will be visible at all times.

New layers that are required by the reflection pass are simple black and white masks. One mask for telling the floor which areas are actually reflective. And one optional mask for driving different features of the reflection, such as water ripples and reflection blurring.

In this location, I needed to have the pool surface fully reflective, and most of the floor almost fully reflective. I went in by hand and created a selection over every small rock and piece of paper on the floor I did not want the reflections to appear on and added some vignetting and faked fresnel curve.

For the special features mask I needed to separate the pool surface from the floor for adding water ripples on the pool and adding blurred reflections on the floor.

Masks like this were also created for the lobby area. The exterior and the pump doom did not have any reflections. The lobby, however, did not require a water ripple mask. As everything reflective is water.

Lobby area original image and masks

3D modeling

For the 3D modeling, as always, the first step is to reverse engineer the AI image camera data. Even though the images are hallucinated by an AI, the perspective in them is nearly perfect each time.

For ths I am using an app called fSpy. Here is their handy tutorial on how it works: https://fspy.io/tutorial/

Sometimes the camera data Dows not quite work out and while modeling, or even after building the scene in Unity you see that the perspective is simply wrong, it is best to just redo the fSpy step and model the location from scratch. These locations are mostly inverted boxes anyway and the workflow in Blender is super fast!

A finished form pass of a scene

I usually work in some of the quad views, but look at the camera projection to see what I am doing. Blender is superb as it allows you to zoom and pan around the perspective doing a crop, not actually moving the camera and messing up the projection. This is absolutely amazing!

After the fist simple pass of getting the scene for in, it is time to whip out the Loop Cut -tool and add a ton more geometry to the image. If the geometry is not relatively dense, the UV projection will cause mapping artefacts on the surface akin to an old PSX game.

Modeling a location in blender

Here is a screenshot of a finished location in blender. The mesh is very simple. Usually for indoor locations with nothing for the player to go behind, it is a simple box that you can just slice and extrude as required.

UV Projection

To get the UV for the scene, in edit mode, select all of the faces of the mesh and press “u” while the mouse is hovering on the camera view and select “Project from View”

You are now ready to export the FBX to Unity (or Unreal).

Planar reflection shader

For the reflections I chose to use a plugin from the asset store called Fast Mobile Planar Reflections.

The materials in the asset are no longer compatible with the latest URP, but I only need one script from the package. A script called “PlanarURP”. This little thing renders the mirror texture for the scene from the origo of the game object it is assigned to.

This render is then send to the _ReflectionTex parameter of materials in the mesh renderer of the gameobject.

With this information on hand, we can now create any kind of shader graph that has reflections. I made mine an additive material that assigns the reflection texture (in screen space) and multiplies it with the reflectance mask.

I also distort the reflection texture screen space UVs with some normals maps and mask that with another texture.

As the final step, I also added some overly complex blurring on the reflection for smooth surfaces.

Planar reflection shader (with blurred reflections & ripples) graph
A simpler version of the planar reflection shader (no blur)

The reflection material is applied on a copy of the room floor and placed slightly above the real floor. With the masks and all of the different tricks the reflections look pretty good.

Sharp and blurry reflections

I especially like that now the character properly occludes the bright reflections from the room geometry. Also, the reflective surfaces now have accurate reflections from the room, as it is actually reflecting the 3D space, not just an approximation made by the AI.

6 responses to “AI Assisted Gamedev: Adding reflections”

  1. Avatar

    We find your work truly fascinating, and we can’t wait to see how far you will take the experience with the game you’re developing. Anyway, we are following you through the magazine, well done!

  2. Avatar
    Ted

    This is amazing work, thanks for sharing the process.

  3. […] This scene has a simple shadow and reflection setup for compositing the characters into the scene. The scene has 3 textures: normal, shadow pass and reflection mask. If you want to read more about how the reflections work head on over to AI Assisted Gamedev: Adding reflections! […]

  4. Avatar
    Andrey

    Your breakdown is super insightful and thorough, kudos on the fantastic work! I noticed you mentioned using Fast Mobile Planar Reflections, but it seems like the plugin has been deprecated. Do you have any alternatives in mind? Thanks!

    1. Jussi Kemppainen

      Oh that is a shame! I have the plugin from a previous project and I just simply rewrote it to work with the latest Unity. I wonder if I could share it for free in the blog 🤔
      EDIT: if you google for the asset, you will find it from a couple of locations.

  5. […] As always, the next step is fSpy. If you have not been following the project, fSpy is a free software that allows you to reverse engineer camera position and focal length from a photo. I use the camera from fSpy in Bender to help with the modeling process and also in Unity to render the final game image. When the camera perspective is correct, the realtime 3D elements like characters match the scene a lot better! […]

Leave a Reply

Your email address will not be published. Required fields are marked *

Recent Posts


Archive


Social Links