MAYA_PRESET_PATH is an abortion of shame.

Today the animation lead asked me to make a set of MentalRay render settings presets available for everyone.

One would think that this:

MAYA_PRESET_PATH

Defines the location for Maya presets. Each entry in the path points to the directory above the attrPresets directory.

Would mean what you think it means.

But, that's for a completely different preset API, one defined purely for the attribute editor and actually implented in MEL. The nodePreset built-in command used by the render settings window to build its preset list completely ignores MAYA_PRESET_PATH. So instead of setting an environment variable, I have to copy a preset into everyone's home maya/2009-x64/presets directory.

The butterfly effect

Autodesk has been trumpeting Maya's Python API since 8.5. It's improved, but it's still buggy compared to the MEL API. This week we discovered a truly insidious bug.

userSetup.mel is a standard entry point for many studios. On startup, Maya looks in MAYA_SCRIPT_PATH and executes the first instance of this script it finds. We use this to initialize various proprietary plugins and tools. Starting with 2009, we've switched to using Python to develop the majority of our tools. So, it makes sense that Maya would also have a Python entry point similar to userSetup.mel. It does. Maya scans PYTHONPATH for userSetup.py and loads the first instance it finds.

Wait, PYTHONPATH? Why not MAYA_PYTHON_PATH? We don't want to necessarily expose modules compiled for Maya in other python interpreters. So, we decided to use userSetup.mel to wedge a set of paths from MAYA_PYTHON_PATH into PYTHONPATH and invoke the first userSetup.py we find from there.

This worked fine for nearly a year. Suddenly, last week, scenes began failing to open with errors about not finding required plugins. We traced the problem down to Maya somehow saving scenes without the "required" lines that indicate which plugins the scene needs. WTF?

After a few days of trial-and-error, we traced it further to a single line of code that invoked a method from the Python OpenMaya API. Without producing any errors or warnings, a standard API method call caused Maya to produce invalid output on save.

Broken references, revisited.

Today's episode of Maya=FAIL is brought to you by the number 8.

I was called over to a lighter's desk today. One of the assets in her shot refused to allow itself to be added to a render layer. Closer inspect revealed that it was somehow named wrong. This particular asset was part of a numbered series: asset1, asset2, asset3, etc.. In this particular case, asset8 was in the shot, but the namespace was called asset9.

What?

Nowhere in our workflow do we change the name of an asset. I tracked through all other assets that made up the scene, and found a few lingering nodes related to asset9, probably from nodes that were imported then removed or renamed. Thinking that somehow, they were responsible, I cleaned up the assets in question, rebuilt the lighting scene and... nope, still there.

Where was asset9 coming from? There was no longer any asset9 in ANY of the maya files that comprised the scene!

Maya's automatic naming clash handling was somehow kicking in, erroneously, and renaming asset8's namespace to asset9 on a few nodes. That was causing connections to fail (silently, of course) and weirdness to ensue..

The solution? In the lighting sync scene (which is then referenced into the lighter's scene) I changed the namespace from "asset8" to "asset8A".

And now the lighting scene works. Why? No idea.

Hey Autodesk, if ending a namespace in a number can cause mysterious node renaming when those assets are later referenced, WHY is it even possible to do so?

FAIL.

The fail that is render layers

Building a lighting pipeline for anything beyond simple CG characters is complex. Compositing wants specific layers, with secondary outputs of diffuse, specular, shadow, depth, uv, world position, and many, many others. Different layers need different shader assignments and different rendering attributes. Maya's solution to this appeared in Maya 7 and is called "Render Layers".

Render Layers works like this: there is one layer by default. It's a node, called defaultRenderLayer1. Confusingly, it ends up called "Master Layer" in the Render Layers window, but no matter. The Master Layer contains all geometry in the scene, the default settings. Now, if you create a new render layer and make it current, you can create a per-layer override for any attribute on any object.

"That sounds great!", I thought to myself. And merrily the lighters went about creating render layers, assigning different shaders and... wait, why won't that shader attach? What does that obscure error message mean? Why are all my shader assignments reset when I re-open my scene?

Turns out that Render Layers do not play well with references. Meaning, they don't work. They break. The only solution is to import all the references in your scene, cross your fingers, and hope that it all works in the ends.

In the pipeline for a large show, we often want to reuse work wherever possible. So, if we have a sequence with 30 or so nearly identical shots, we'd ideally like to reuse the lighting, layers, and per-layer overrides from a single shot (the "key" shot) as a starting point for all the other shots. So, of course, Aliasdesk must have included a simple way to export a render layer as a MEL script? Right? Right?

No.

FAIL!

Bugs in the node soup

Yet another day wasted (and by day I mean 14 hours) trying to work around yet another fucking Maya bug.

Today, I tried to trace the source of this message:

Warning: line 1: Node 'test_JR02004_layout:shot04_v07_pCubeShape12.instObjGroups[0]': cannot make assignment to 'initialShadingGroup' shader.

This warning is misleading. It's actually hiding a more serious error: shader assignments don't stick. When a Maya scene gets into this state, changes you make to shader assignments are not persistent. Why this isn't an error is beyond me.

So, a lighter creates a bunch of shading nodes, assigns them to holdout objects from animation, saves the scene, and launches a render. But when the scene is opened and RIB is generated, no shaders are attached to the objects -- they render in plasticene grey. The lighter comes back the next morning, re-opens their scene, and BAM! Why aren't the shaders attached? Damn, must have forgot. Re-attach, re-launch. Still bad frames.

How does a Maya scene get into this state? I have no idea. Maya adds crap to a scene every time you export or import, and lighting is where all the crap comes together. If anything is going to break, it's going to break in lighting.

Why is this so hard to track down? I blame Maya's complete lack of encapsulation. Any Maya node can (and will) connect to any other Maya node. There is no abstraction or protection. It's a bit like the old days of DOS and Windows 3.1, where any application could corrupt any other application's memory space. Compare that to Windows XP, where each application is isolated in its own space, preventing a single badly behaved program from taking down the whole system.

Maya is the Windows 3.1 of 3D applications.

And worse than that, the problem is essentially that Maya cannot correctly read files it itself generated! Unacceptable.

And how are we going to solve this problem? By working around it. This is production -- we can't afford to wait 6 months for Autodesk to fix a bug. So, we're doing a complete end-run around shader attachment and exporting compiled shaders. Thank fuck we have that option -- if we were using Mental Ray we'd be truly buggered.

So now I have to write a tool to export and manage individual shaders. Those tools will inevitably have bugs, too, so we're stuck supporting a hack around something that software celebrating its 10th anniversary should have figured out long ago.

A few words on global namespaces

Stupid. Stupid stupid stupid stupid stupid.

That is all.

FAIL of the day

Since Maya's asset management is so piss-poor, studios end up building scripts and plug-ins to do the job. Let me backtrack a bit and describe how Maya's asset management works.

A Maya scene is a collection of nodes, attributes on nodes, and connections between attributes. There are a few other little details, such as plug-ins, DAG paths, references and namespaces, but that's essentially it.

There are two ways to split work between two scene files. The first is to just import the scene into your scene. This loads all the nodes (and connections and attributes) and prepends a namespace in front of all the node names to prevent name clashes. The second is called a reference. It does the same thing as an import, but links all of the nodes to a reference node as well. A reference node is a special node that tells Maya to load a bunch of nodes from another file. That way, you can refresh the reference. But, since Maya allows connections between any node, referenced nor not, provision must be made for changes to referenced nodes. It handles this by adding another, invisible blob of data called reference edits. These store changes to referenced nodes. References can be nested as well, so you can imagine how complex reference edits can get.

But I digress.

This morning I'm trying to build a scene that is basically a collection of references to other scenes. I'm using a simple MEL script to references a bunch of textured models, apply some geocached animation to them, and load the camera and set from layout. Seems simple enough. But, but, but... it fails. It spits out the usual screenfuls of dubious warnings, then segfaults.

Thinking that maybe some of our custom plug-ins were responsible, I ran the same process through gdb. Nope:

Saving Generated Scene: /dfs1/net/prod/pipeline/working/markv/clean2/projects/dot/shots/test/JR02003/light/synch/test_JR02003_light_synch.ma

[crap omitted]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 47512146590080 (LWP 14527)]
0x0000000018ed0d10 in ?? ()
(gdb) where
#0 0x0000000018ed0d10 in ?? ()
#1 0x00002b363f6833ff in TDGtypeFilter::matches ()
from /usr/autodesk/maya2009-x64/lib/libDependEngine.so
#2 0x00002b363f685af4 in TDGanyFilter::matches ()
from /usr/autodesk/maya2009-x64/lib/libDependEngine.so
#3 0x00002b3642806c03 in TglobalTranslator::callNodeCallbacks ()
from /usr/autodesk/maya2009-x64/lib/libShared.so
#4 0x00002b3642806e27 in TglobalTranslator::callNodeCallbacks ()
from /usr/autodesk/maya2009-x64/lib/libShared.so
#5 0x00002b3642806ea8 in TglobalTranslator::callNodeCallbacks ()
from /usr/autodesk/maya2009-x64/lib/libShared.so
#6 0x00002b36428048cb in Tscene::saveScene ()
from /usr/autodesk/maya2009-x64/lib/libShared.so
#7 0x00002b3642804aa4 in Tscene::doSave ()
from /usr/autodesk/maya2009-x64/lib/libShared.so
#8 0x00002b36427d68d1 in TsceneOperator::saveScene ()
from /usr/autodesk/maya2009-x64/lib/libShared.so
#9 0x00002b36427d0647 in TfileCmd::handleFileSaveFlag ()
from /usr/autodesk/maya2009-x64/lib/libShared.so
#10 0x00002b36427d3e60 in TfileCmd::handleFlags ()
from /usr/autodesk/maya2009-x64/lib/libShared.so
#11 0x00002b36427d61dd in TfileCmd::doCommand ()
from /usr/autodesk/maya2009-x64/lib/libShared.so
---Type to continue, or q to quit---
#12 0x00002b363f3b3406 in Mel_Command_Dispatch ()
from /usr/autodesk/maya2009-x64/lib/libCommandEngine.so
#13 0x00002b363f3d8152 in node_exec ()
from /usr/autodesk/maya2009-x64/lib/libCommandEngine.so
#14 0x00002b363f3d832f in sophia_call_executable ()
from /usr/autodesk/maya2009-x64/lib/libCommandEngine.so
#15 0x00002b363f3f4f2e in SophiaExecutable::evaluate ()
from /usr/autodesk/maya2009-x64/lib/libCommandEngine.so
#16 0x00002b363f3b9e42 in TcommandEngine::sourceFile ()
from /usr/autodesk/maya2009-x64/lib/libCommandEngine.so
#17 0x000000000040f4a1 in TmayaApp::initAfter ()
#18 0x000000000040d9c9 in TmayaApp::initBatch ()
#19 0x00002b36401fe43c in Tapplication::start ()
from /usr/autodesk/maya2009-x64/lib/libExtensionLayer.so
#20 0x000000000041060b in appmain ()
#21 0x000000000041e93b in main ()
(gdb) quit

Here's the kicker: the same script, run from interactive Maya does not crash.

How in the hell does a piece of software that's 10 years old still have a bug that causes it to segfault?

FAIL.