Building Custom Lighting Material Shader to Style Guide Spec
building a toon shader according to specifications for ease of use and supporting documentation for artists
CONTENT
process overview
specifications
translating specs to functional requirements
drafting and testing functional behavioral modules
testing in engine
specification feedback and iteration (Art Director)
asset pipeline feedback and iteration (Lead Artists)
optimization and performance feedback integration (Dev Team)
preparation for initial deployment
PROCESS OVERVIEW
My general process for shader design is based on the principles of rapid digital prototyping and modular separation of concerns for functional modules. To that end, I personally prefer node based shader editors because its real time feedback makes debugging faster, and the visualizations allow the development process to be more collaborative cross-functionally. Specialized functionality and HLSL code can be injected on an as-needed basis during drafting with code modules. And I will draft shaders in text only if desired behaviors cannot be replicated otherwise, or if some meaningful performance benefit could be had by restructuring low level behaviors.
In my experience, designing shaders requires a balance of cross-disciplinary concerns and needs. However, it’s easier to critique and improve an existing draft of a solution than it is to theory craft in the abstract.
The two are at odds since the most rapid approaches benefit from “messy code” and experimentation, and the latter benefits from considered and deliberate planning. I prefer to work on a module by module basis, working quickly to draft major anticipated modules in a quick and dirty way, and stitch those modules together quickly for feedback.
In subsequent passes as features and approaches solidify, modules internal functions can be refactored, and the contents of the shader that stitch those modules together can be considered for modularization if it improves legibility or iteration.
The final step is to take a final design pass to simplify the shader for use by artists and simultaneously draft art pipeline documentation to capture the assumptions, requirements, and known edge cases present in the draft.
SPECIFICATIONS
In the preceding step (make hyperlink), recommendations were drafted for consideration by the Art Director. Assuming that those recommendations were green lit, the resulting shader spec sheet might look like the following:
CUSTOM LIGHTING MATERIAL SHADER - BUILT-IN RENDER PIPELINE - ANIME STYLE
Core Behavior Specifications:
Material shader with diffuse lighting, AO based shading effects, and abstracted specular controls.
The shader must allow for per-material definition of a diffuse light tint, and a real-time shadow color. The shader will not make use of scene defined global illumination and baked lighting data.
Conceptually, the shader will support a distinct behaviors between “lit” and “unlit” regions.
To replicate the three step shading of the style guide, the shader will allow for the primary definition of an albedo color for the middle step, a highlight region for the first step, and a shaded region as the third step.
Unlike the albedo region which will be set, the highlight and shaded regions will be derived.
The shader should allow for both stepped and gradient transition between highlighted, albedo, and shaded regions.
Highlighted regions should allow for controls to interpolate between scene light components’ colors and an artist definable tint. The entire highlighted region should render flat except where ambient occlusion applies and the albedo color. Transition between the highlighted region and albedo region should be definable on a per material basis to be stepped or gradient.
Albedo and shaded regions will define the bulk of visible surface. Ambient occlusion and specular highlight will stack on the base diffuse contribution. Transition between the albedo region and shaded region should be definable on a per material basis to be stepped or gradient.
AO and specular effects will vary between “lit” and “unlit” regions.
Ambient occlusion masks should permit the darkening of highlighted regions to the albedo color.
The ambient occlusion mask in both albedo and shaded regions should be able to both impact the diffuse value, as well as apply a stacked tint to darken to a (HSV value) value below the shadow tint.
Specular reflections in the highlighted region will be ignored.
Specular reflections in the albedo and shaded regions will be visible.
Specular reflections will be calculated based off of a light vector equal to the view direction, and not from scene lights.
Shader Inputs and Controls:
An albedo texture input and secondary albedo tint color input to determine the albedo color
A normal map input with normal “power” slider input for light responsive fine geometric details
A diffuse reflection tint color input, a light color to tint color slider, and a diffuse light color intensity slider to tune the derived highlight color
A real time shadow tint color input, and a shadow tint strength slider to tune the derived shaded color
A set of four sliders to assign transition control
For all diffuse reflection strength values above slider 1 (Max Lit Value) will be rendered in the derived highlighted region color
For all diffuse reflection strength values below slider 4 (Min Shadow Value) will be calculated relative to this derived shaded region color
For all diffuse reflection strength values between slider 2 (Min Lit Value) and slider 3 (Max Shadow Value) will be calculated relative to base albedo color
For values between slider 1 and slider 2, the result will be an interpolation between the highlighted and albedo colors
For values between slider 3 and slider 4, the result will be an interpolation between the albedo and shaded colors
Wider deltas between sliders 1 and 2, as well as slider 3 and 4 will allow for a gradient transition, and smaller delta will appear as discrete steps
A specular intensity slider and shininess slider to control specular highlights in the “unlit” region — the former controlling tint and the latter for diffusion (size) of the region
A “maps” texture input
R-channel - Ambient Occlusion Map
G-channel - Shininess Map
B-channel - Grime/Highlight Map
Ambient occlusion controls
AO tightness slider - uses exponential power to exponentially tighten or suppress the affected area
AO map value scale and offset sliders - to linearly spread and scale the affected area that will be rolled into diffuse reflection intensity calculation
AO shadow tint slider to tune AO region shading between HSV value-based darkening, or real time shadow color tinting
Expected variants’ behaviors and inputs
Broadly for gameplay purposes, assets are visually differentiated by whether the player can mechanically interact with them or not. Interactive elements include avatars, enemies, collectables, and interactable elements that the player can mechanically affect. Environmental elements include non interactive assets like vegetation, terrain, and various mise en scene.
[Interactive] Inverted hull outline variant - interactive element variant that uses inverted hull outline technique to create outlines and exterior linework effect
Requires outline color input and outline scale value input
[Interactive] Mesh deformation animated variant - requires a variant to account for per use needs (e.g. cyclic sinusoidal mesh scaling for jellyfish, spiral mesh rotation for tentacles…)
Requires rate, magnitude, input parameter scaling inputs (e.g. object space vertex distance for jellyfish)
“Maps” texture B-channel if texture based masking required
Requires inverted hull outline functionality
[Environmental] Static variant - effectively the variant of the based spec
[Environmental] Foliage variant - requires configurable noise based deformation variants to create the impression of wind
Requires rate, magnitude, and world space input into simplex noise scaling control inputs
Requires opacity clipping for foliage card, with opacity encoded into the albedo texture, and an exposed clipping threshold
[Environmental] Terrain variant - requires Unity Terrain System pattern splat map inputs for use with Unity’s native terrain system
Requires potentially per splat layer controls for all controls. For first pass only requires albedo tinting
Maps texture inputs for each splat layer needs to be reworked to Unity Terrain convention pattern
R-channel - metallic
G-channel - AO
B-channel - height
A-channel - smoothness
TRANSLATING SPECS TO FUNCTIONAL REQUIREMENTS
-
I personally prefer Amplify’s node-based shader editor (ASE) over Unity’s native implementation (Shader Graph) in most cases, but it is the only option for Unity’s Built-In Render Pipelines.
ASE is a mature product that is stable with some key advantages.
ASE supports the Built-In Render Pipeline
ASE made shaders compile to a standard ShaderLab/HLSL/Cg file, meaning shaders can be easily developed outside of a project respository and imported into a project.
ASE’s noise node is more robust with simplex 2D and 3D options, which Shader Graph doesn’t natively
ASE’s UI requires fewer actions to configure properties, default values, and configuration of sliders.
ASE provides examples and templates with community developed features that are better documented and expansive than Unity’s offering.
-
Shader Function - ASE allows users to create their own nodes which they call Shader Functions. These nodes are compiled into code at the compile time of shaders that include them, so they are not required dependencies for ASE-made shaders.
-
The following is how diffuse lighting is calculated in a Blinn-Phong lighting model.
Compute fragment normal - take interpolated vertex normal values and modify them based on normal map values.
Compute “facing” factor - take dot product of fragment normal vector and light direction vector; the output after clamping is a 0 to 1 output where 1 represent a surface pointed directly at a light, and 0 represents surfaces that are perpendicular to or facing away from a light.
Tint and scale - the facing factor is multiplied against the light color, scaled based on light attenuation/falloff, and occluded by the shadow mask.
Global illumination contribution - the product of the last operation has the global illumination color added to it to account for ambient bounce lighting.
Albedo contribution - the diffuse contribution is multiplied against the albedo color.
-
The following is how specular lighting is calculated in a Blinn-Phong lighting model.
Compute fragment normal - take interpolated vertex normal values and modify them based on normal map values.
Compute “reflection” factor - take a dot product between fragment normal vector and the half vector where the clamped output would be a 0 to 1 value where 1 represent a surface perfectly at a reflection angle between the view point and the light source, and 1 represents being perpendicular to facing directly away from the reflection angle.
Suppress/diffuse the specular region - the “reflection” factor has a power operation applied to it with an exponent representing the “shininess” of a surface. Shinier (smoother) surfaces would have a tighter and more intense specular reflection, where as more matte and textured surfaces would have a larger and less intense specular reflection area.
Apply light color - the last step multiplies the previous output by the light’s color. Because a specular reflection assumes negligible refraction and absorption, the reflected light should be the color of the incident light source.
Based on the specs outlined above, only a few essential “shader functions” need to be developed.
Diffuse Value Function
Unlike the diffuse calculation in Blinn-Phong (BPL) like diffuse lighting calculations, a simplified single diffuse intensity value is necessary. As opposed to a color value output, a single floating point value is needed to be able to drive interpolations and define thresholds.
Unlike the BPL approach, the light color value can be converted to grayscale to evaluate luminosity independent of color channels.
Unlike the BPL approach, this diffuse lighting value cannot be used to tint values down from a theoretical purely illuminated max albedo color.
Instead, it will be used as a value to evaluate for thresholds and to drive interpolations.
Camera Source Specular Function
Unlike the specular calculation in a BPL-like lighting calculation, light position and color are ignored. Instead, a light source is assumed to originate from the camera like the samples from the style guide, and so a dot product is taken of the fragment normal against the view direction.
Since the shader spec requires a per-material diffuse light tint input, the value of that expected calculation can be used in place of the light source color.
To mimic the anisotropic effect of the specular reflection, the y-axis value of the dot-product evaluations can be ignored to create long vertical reflections.
Normal Angle “Power”
To give artists control over the impact of normal map contributions to lighting, a normal power function is necessary.
Since a normal map encodes a direction using a texture, the fragment normal deflection can effectively be intensified by scaling down the z-axis value or scaling up the x-axis and y-axis values before re-normalizing the vector.
The shader will need to be able to do the following
Take a given surface’s normal value, and deflect it on a per-fragment basis using a controllable deflection intensity from any normal map value
Using the resulting normal vector, calculate a diffuse value
lit/unlit -> highlight/flat/shaded
