mip_matteshadow and how I have used it
In this post I will explain the way I have been using the mip_matteshadow shader in maya 2008 (along with some of the other mentalray "production" shaders). It is not going to be a step-by-step tutorial - and I'm certainly no "production" shader expert. I started by reading Zap's mental ray tips, and the Production Shader Wiki and the PDF manual. Then I did some experimenting to see how I could include these new methods in my work which, unlike Zap's examples, usually involves rendering CG characters in CG environments rather than live-action or photographic backgrounds.
Here is a simple example of how I have recently started using mip_matteshadow as a better alternative to maya's useBackground shader... (click here for the rest of the post)
Let's say I need to render a CG scene consisting of some kind of background and an animated character. If I want to achieve certain special fx in post (motion blur maybe, or even just to have more control over lighting and color corection) then I need to render my scene and seperate the character, the shadows, the reflections, the occlusion and the background into different render layers which I will composite afterwards. I use after effects by the way.
Here is a wireframe showing a very simple scene that I will use as my example.
The grey sphere will be my "character" and the green stair case is a background object. I have illuminated the scene with final-gather and mia_physicalsky, tone-mapped with mia_exposure_photographic. Both objects are shaded with mia_material_x and it looks like this when I hit render (low quality settings).
First I create a render layer called _bgnd which I use to render out just the stair-case and the sky. Nothing special. It looks like this.
Since the alpha channel is just the stairs I can use it as a mask in afx to apply color changes and other filters to the sky or the stairs independantly.
Next I have a layer called _characterSolo. This is just the sphere on its own with alpha channel over a black background but it is illuminated exactly like in the master layer and reflecting the other objects in the scene (in this case the sky and the stairs). To hide the stairs I just create a layer override for Primary Visibility on the stairs object. To hide the sky I turn on the "Use Background" flag in mia_physicalsky, and leave the "Background" slot empty. This makes the sky black but leaves the illumination unchanged. And here is the render.
There is actually another way to deal with hiding the sky. It's more complicated, but I will explain it here to demonstrate some other production shader tricks.
mip_rayswitch_environment is a shader that lets you specify two inputs, namely "Background" and "Environment". If you make the "Background" black then thats what you get. Objects in the scene are illuminated by what ever you connect to "Environment". So in my example scene, for the _characterSolo render layer, I have inserted a mip_rayswitch_environment between the mia_physicalsky and the camera, so that mia_physicalsky still works as the "Environment". For the "Background" I have used a trick to make it black with zero alpha channel - which is simply connecting a default multiplyDivide node. This exploits a quirk in the way maya's utility shaders are translated to mentalray. Here is a snapshot of the hypershade network (have a look at the example scene to see the connections).
Remember it is probably easier to just do this using the first method I suggested, but now you know two ways.
Next I created render layers where I used mip_matteshadow assigned to the stairs to catch the occlusion, reflection and shadow. The sphere now has a Primary Visibility layer override. And the sky is hidden as shown above. These three layers differ only in the flags that are enabled in the mip_matteshadow shader. Here is a snapshot of the attribute editor with all three turned on (but for my render layers I used layer overrides to render each seperately).
One important thing to note here is the use of the "Ambient" attribute. This controls the shadow density and needs to be set to very bright if you are using high dynamic range illumination. So in the example scene I have set it to a value of 15000 (considerably beyond the default range of the slider).
The shadows layer renders a black image with the shadow information held in the alpha channel, which looks like this
Heres the occlusion layer (which is also contained in the alpha channel).
And the reflections of the sphere.
To achieve "correct" illumination of the sphere in these reflections I used some more production shaders. I needed the stairs to contribute to the illumination of the sphere, but to appear black in the render, with no self-reflection. For this I used a mip_rayswitch node connected to the background slot of mip_matteshadow.
mip_rayswitch is like mip_rayswitch_environment except that it has more inputs. It lets you choose what shader contributes to the many different types of rays that bounce around the scene. In this case I used the same multiplyDivide node trick to give a black background with zero alpha. Then I connected the original green stairs shader to the Reflection, Finalgather and Shadow slots. This must be done with the connection editor (drag and drop doesnt work). You need to connect the mia_material_x "result" output. Now the sphere "sees" the colored stairs for the purpose of reflection and finalgather color, but the camera just sees black stairs, with the sphere reflections.
Well thats pretty much it. Actually compositing these layers is a whole other story.
To finish up I would suggest that there are many ways to achieve similar results to what I have shown here. The occlusion render layer might be easier to do using the old mib_amb_occlusion, but you would need to deal with self occlusion of the stairs. The reflection layer could be done using the multiple output capabilities of mia_material_x, but that has issues of its own that make it a bit complicated. The shadows done using mip_matteshadow lack the color imparted by the indirect illumination. It can be faked in post, but it doesnt look quite the same.
Please feel free to download my example scene and to experiment with the connections yourself. I'd be pleased to hear your own methods either as comments here or via email.
Here's a quick comp I did in after effects, with a sky photo dropped in, some DOF and some color changes.
Hi David, amazing tutorial, i was just wondering is there a simpler way too catch the shadows from physical sun, or would mip_matteshadow be the easier solution?
thanks in advance
Comment by drone3 — August 5, 2008 @ 6:39 am
Hey thanks for the compliment. Once you get familiar with mip_matteshadow it's actually quite simple to use. I probably made it sound more complicated because I was trying to show some of the little tricks I use to catch more than just shadows. I don't know of any alternative (other than the less versatile useBackground shader - which does not work with some mentalray setups).
Comment by david — August 5, 2008 @ 11:36 pm
As suspected, i'll give it another whirl, keep up the ausome blog, lots of gems in this lab.
Comment by drone3 — August 7, 2008 @ 12:30 pm
Hi David!
first off, brilliant post! I have been looking for days for a solution to this! Being a Max user, i think you would find it interesting to know that this entire setup in Max is a single switch (leave it to Maya to spice things up!). Now to my question: when applied, this solution renders out a black shadow. How would i get the original shadow color/illumination color?
Anyhow, your blog is fantastic and i keep coming back for more all the time!
Comment by loganai — August 29, 2008 @ 4:11 pm
Thanks for your nice words. I'm glad to share what I know. I don't have a good answer for rendering colored shadows as a separate pass. If I need this I usually do something simple like just rendering with primary visibility turned off on the shadow casting objects. But more often I just fake it in after effects (sometimes with the indirect lighting captured separately and added in).
Maybe 2009 will make all this much easier.
Comment by david — August 29, 2008 @ 11:04 pm
Ok! Well i guess there isn't a cure for everything yet! Thanks for you reply in any case. I did my composite in AE as well.
Comment by loganai — August 30, 2008 @ 5:08 am
thanks so help full
Comment by dican — January 19, 2009 @ 1:12 pm
thanks, thanks, thanks!!!!!!!!
This was lifesaving. I have deadline tomorrow (ooooops, actualy that's today:)
My problem was to get shadow from VUE trees in maya. And I only needed trees (+shadow in alpha) pass for compositing.
And I hardly figure it out. I didn't realize (at first) that I can, actually must, assign mip_matteshadow as a material to object that is catching shadows (lack of concentration after several working hours).
live long and prosper
Comment by sheldor — February 24, 2009 @ 4:03 pm
Hi DJ,
First off, thanks so much for all your help ... I've been following your blog for awhile and your tools and posts have saved me tons of time and taught me alot about mental ray.
I've been having trouble getting a shadow pass to refract in mental ray ... first I tried a useBackgroundShader with a phong sphere in front of it, then i tried mip_matte_shadow with mia_material glass sphere in front of it (propagate alpha turned on), trace depths set high enough ... problem is, i still keep getting solid mattes of what should be just a shadow pass (need to refract / distort a shadow pass for compositing). I also tried pumping the useBackgroundShader's matteOpacity attr into the color in of a surface shader, and it works fine up until it goes through any geometry with a glass shader, at which point it just becomes solid white.
any tips? i'm at the end of my rope.
Thanks again for all your help, please keep it up!
best,
m
Comment by agenesis — June 27, 2009 @ 5:33 am
agenesis: I'm glad you found my blog useful. I appreciate the comment.
Unfortunately I have no experience with render passes. Using render layers, like in this post, are how I still do it. Even then refraction is tricky when it comes to alpha channels. I think I would probably try to set up a render layer with white objects casting shadows, but it would not be accurate like a true shadow pass.
Hopefully someone else can help you with this. Good luck.
Comment by david — June 28, 2009 @ 12:22 am
Hey Dave, its me again onto this topic now. As you know, I was just asking you about the HDR and IBL stuff earlier on. And now I have came to this point with the shadows not showing up in my scene and have researched a bit into the mip_MatteShadow. Sounds and looks kind of confusing, probably cause its new me. But anyway, I have my scene setup with the rayswitch, and now what? Do I just connect the MatteShadow into the shadow slot on the rayswitch? Or is there more to that? Or is it suppose to go into say on my grounds SG node? Thank you.
Comment by smokedogg — December 16, 2009 @ 2:23 pm
smokedogg: This post kind of assumes you already know how mip_matteshadow works. If not then have a look at the link to zap's blog (in the first paragraph). There you will find some set up hints and example scenes too.
I'm guessing (from your other comments) that you are going to be using mip_matteshadow with your sIBL setups. If that is the rayswitch that you are asking about, then probably the answer is no. You dont connect mip_matteshadow to that rayswitch. You were right in the last question - mip_matteshadow would be assigned to your floor geometry.
As I have shown in my examples in this post, I often do use rayswitch nodes with mip_matteshadow, but not the one created with the sIBL loader.
Comment by david — December 16, 2009 @ 10:19 pm
OK, I am understanding what you are saying but the 1 thing that is puzzling me is that when I go to connect anything from or to the matteshadow, I keep getting the connection editor and am not sure what attr. to connect to where. e.g.- I have my sIBL all setup, my building, and my ground plane. Now when I render I get no shadows. Thats where the mattashadow comes into play and now where I get lost. From here, I am not sure where and what goes what. I have downloaded your example scene and tried following what you have on your setup but no resulting shadows. I feel like a newbie again. Sorry. Maybe if I could somehow email or post my test scene and you could take a look at for me to see what I am doing wrong would be greatly honored. Thank you.
Comment by smokedogg — December 17, 2009 @ 5:53 am
Hello David, well after my last post I did some more researching on hdr labs and re-read the sIBL GUI manual i believe after the creation and import/export, supposedly a mip_matteshadow should be connected to where ever and however the script loads. Im not sure cause im at work now and wont be able to give it a try for about another 9 hours. So hopefully now everything will just ok. Thanks for your time as your probably sick of hearing from me. Although you have become a saint to me for all the help you have brought to me thus far since I found your A+ blog!
Comment by smokedogg — December 17, 2009 @ 8:25 am
smokedogg: I hope it is working out for you. If not send me a scene and I'll take a look. I emailed you, so you have my address. Thanks for you kind words too!
Comment by david — December 17, 2009 @ 10:57 pm