Echoes of Somewhere

There have been multiple advancements in the AI space that I have found very interesting but there has not been a use for them in the game project yet. So I set up a news section to cover more things.

The first tool I am covering is Blockade Labs’s Skybox generation tool.

It has been out for a while now and people have been creating some very cool things with it!

Most of the videos I have seen of this, have been third or first person perspective, and it begins to break down quite soon after you begin to move deep into the scene. My hypothesis was to create a carousel camera that orbits the scene, allowing only the character to move in depth, but never breaking the illusion of a fully 3d scene.

I hopped on over to their website and generated a quick test scenario to test. The prompt for this was “Cyberpunk city center wide asphalt street corner with multiple shops and restaurants adventure game” in Digital painting style. I then downloaded the image for depth map generation.

I then take this generation and pop it over to automatic1111 and generate a depth map from it using ControlNet.

In Automatic1111 you need to pop ovet to img2img tab and put in the panorama image to the source image. Then turn on tiling and set the image size to 2:1 aspect ratio. Denoising strength can be set very low.

After that, enable ControlNet, set the preprocessor to depth and the model to depth. Remember to set the aspect ratio to 2:1 again.

You are now ready to generate the image. If you do not get a depth mask generated, try tweaking some of the settings (like denies strength) and try again. I had this fail on me a couple of times.

To get a correct displacement, I loaded up the image in photoshop and overlaid a black color fill on the map with 50% opacity. As in blender the displacement outward happens with luminosity values 0 – 0.5.

There is a great video tutorial on how to walk trough these steps, so you can take a look at that to get even better and more detailed tutorial on this.

After following the tutorial somewhat I ended up with this mesh. (If you are interested in the steps in Blender, check the video tutorial, he does way better job at it than me!)

I was also informed of a free blender add-on that does a spherical mesh with depth displacement automatically. So you can also give that a go if it interests you.

The bottom of the blob was all messed up but I flattened it to get rid of the artefacts. I would be needing a flat floor for what I had planned to do! I also split the mesh in 2: one intact mesh and one of just the floor. The floor mesh would be used as a navmesh for the character to walk on and as a plane for the planar reflections.

As you can see, the mesh I ended up with is far from perfect. It would not matter as I was going to not be allowing free camera movement in the scene.

My setup is going to be a null object at the center of the panorama, always pointing at the player character. The camera would be offset backwards from this point to get some parallax and depth to the scene when the camera is moving. The look at script for the camera is a simple component I copied from Unity manual.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class CamLookAt : MonoBehaviour
{
    public Transform target;
    public float speed = 0.01f;
    Vector3 targetPos;
    Vector3 currentPos;
    private Vector3 velocity = Vector3.zero;

    private void Start()
    {
        targetPos = target.position;
        currentPos = targetPos;
    }

    void Update()
    {
        targetPos = target.position;
        currentPos = Vector3.SmoothDamp(currentPos, targetPos, ref velocity, speed);

        transform.LookAt(currentPos);
    }
}

I put in all the little tricks like hand painted shadows, planar reflections and some AI driven smoke effects to make the scene pop a bit more.

I also used the panorama image as a reflection probe material and ambient lighting for the 3D characters to marry them to the scene better.

Also some spot lights were added to add some light influence from the windows on the characters.

This technique was very fun to work with. I can definitely see a game done using panoramic backgrounds like these!