I had planned to keep the characters far away from the camera, as they were not the best quality. But as I have been working on these scenes the main character creeps closer and closer. So I needed to make him look better.
The idea for my character pipeline was to use as much AI generated pixels in the final character as possible. I achieved this by prompting a character turnaround from Midjourney and projecting the AI generated imagery on the mesh UVs after modeling. A full breakdown can be read in this blog post, but I will do a TL:DR
This is the source image for the main character with the prompt “very old and weak man, cyberpunk point and click adventure game character, full body, model sheet turnaround, full color, front::4 view, side view and back view –v 4 –ar 3:2“
The character stayed coherent because all of the angles were in the same image. I used this as a reference for modeling and then posed the character in 3 different poses (morph) for projecting the image on the geometry.
These 3 projections were then combined in photoshop to create the final texture.
My initial idea indeed was to never let the character get too close to the camera. But as the project went along and I generated more and more locations, I found that the character was quite often was up close to the camera.
The low resolution, artefact ridden, flat texture begun to make me feel more and more uncomfortable.
Generative Fill and overpainting to the rescue
Around this time the Adobe incorporated the Firefly generative AI engine in the latest AI and I saw it as a way to easily get rid of most of the artefacts. I used a combination of generative fill and overpainting to fix the original UV.
With the use of generative fill I was able to very easily fix most of the issues of the texture. It works so well for removing unwanted features from an image while retaining the original style!
I did though do good old fashioned overpaint for most of the face, as generative fill did some pretty insane facial features! I also panted out the eyeglasses from the texture, as they would be modelled in later.
To get rid of the overly cartoony shadows I used the lighten blend mode to only affect the shadow areas.
As a last touch, I copied the document, flattened it and used Adobe’s AI upscale to upscale the texture 2x.
in addition to the edits on the base color map, I also added a metallic mask and a roughness mask. I wanted the few metal parts of the costume to appear metallic in game.
Adding more details
Now it was time to hop on over to Modo for some old fashioned 3D modeling for additional details.
Separating the Eyeglasses
First, I modelled a set of eyeglasses. They really looked horrible as this low res texture only garbage.
The mesh was not that complicated, but it really makes a difference.
Normal map scuplting + baking
One big issue I had with the game character was how little actual modelled details there were. The clothing for example had all these wrinkles, but the 3D lighting did not react to these in any way.
The easiest solution for this was to bake UV texture for the low detail character from a very high polygon, sculpted mesh.
To get a quick base for the high res sculpt, I simply took the in-game mesh and subdivided it a couple of times. I altered between faceted and smooth subdivision to get topology I was happy with.
Once I had the high resolution mesh. I changed to a textured view in order to see all the cloth details on the mesh.
Then using a wrinkle brush I painted over all of the wrinkles on the character clothing to add actual 3D deformation. I was not too careful with this sculpt. It was not meant to look pretty, just to provide some additional surface detail for the game lighting to grab on to. The video is in an untextured mode to show the sculpting happening. But I used a textured view throughout the sculpt process to see the texture details.
The final sculpt looked quite horrible when untextured. I pretty much only used the wrinkle brush and did not even try to do any real sculpting. For the beard and hair I created some interesting criss-crossing wrinkles to mimic hair strands. I think it worked decently.
Now that I had the high resolution mesh I used the low resolution mesh to bake the normals from the high polygon mesh to a normal map on the low polygon mesh. This map can now be used in Unity to make the game character appear higher resolution.
This is really basic process used in every game for decades.
Even though the high polygon sculpt looks absolutely awful in the modeling software, used as a normal texture in game it looks pretty good! The game lighting now has much more detail to hit and the absence of the clipped shadows makes the character look a lot smoother and match the background style better.
I personally also like the added specularity on the metal part of the clothing and the eyeglasses.
I am not too happy about how the collar mesh looks though, so I am very likely to completely remodel it. That part of the mesh does not really hold up in closeups at all.
But I am really happy that this work got so much faster with the new version of photoshop. Updating the old character texture took very little time.
I am now much more confident with showing the character up close to the camera!
I spend some time remodelling the character’s collar and I added a bunch of polygon detail to the shirt area.
I did a bunch of hand painting on top of the character texture as well and touched up on the normal map. The AI generated bits of the character are always less and less. It still has a lot of the telltale AI signs in the jacket for example. But I am not going to get rid of all of them as all side characters will not get the same treatment.
I also touched up on the skinning as some of the weights Mixamo made for the character were not that great. Slowly the character is getting better and better. I like this workflow where I get passable results real quick and then can do additional passes on the character as I grow tired of its issues.