Update 2012-07-28: The script has been updated with new features, which include a user interface replacing the command-line stuff. The leaf layout details explained here are still relevant and correct, except for the command to run the script. The new command can be found at the end of the new post.
There is a long, long thread on cgTalk called "Forests in maya mental ray". The discussion is all about techniques for creating and rendering the vast numbers of polygons needed to make convincing grass and trees in large numbers.
Maya's paintFX can be used to make trees which can be converted to polygons with leaves that consist of just a few polys per leaf. Texture maps (color, opacity, bump etc) are then applied. The default output from the paintFX conversion has each leaf mapped to fill the standard uv quadrant, so each leaf gets a copy of the texture. This can look pretty good, but the repetition is usually very obvious. You could break the leaf mesh into a few parts and assign variations of the leaf textures to each part. But a more efficient way is to layout the uv's so that groups of leaves are mapped to different parts of the uv quadrant in a 4x4 tiled patternfor example. Then you can make a texture map with leaf variations arranged in the same 4x4 tile pattern. Now you can have 16 different looking leaves on the tree, almost as easily as you can have one.
Default paintFX conversion gives you this
But what you really want is something like this 4x4 tile pattern (or maybe 2x2 or 3x3)
The problem is that you might have 50,000 leaves on your tree. It would be very difficult to select and edit the uv's for different leaves by hand. This is something that needs to be automatedAround page 12 of the Forests thread Roberto Fdez. de Gamboa (aka tostao_wayne) posted a mel script that was able to loop through the leaves and edit each one to fit in one of the randomly selected tiles. It got the job done, but could easily take over an hour to process, depending on the number of leaves.
I've recently been teaching myself to write python scripts so I decided to see if I could rewrite Roberto's script in a way that would cut down on processing. I knew this would mean leveraging the power of the maya API so as well as reading the manual I looked around for code examples to learn from.
As luck would have it, I discovered a recent post at Maya Station by Owen Burgess that showed how to extract information about uv shells from a poly mesh. This was a big head start, and after some study I wrote djPFXUVs.py
djPFXUVs.py can process 50,000 leaves in approx 30 seconds on my old quad core. It arranges leaves into 2x2, 3x3 or 4x4 tiles. You can select multiple meshes to process one after the other.
Install it by placing djPFXUVs.py in your local scripts folder.
Run it by first selecting a poly mesh. Then execute the following command in a python tab in the maya script editor window. import djPFXUVs; djPFXUVs.leafLayout(3) You can change the number in the parentheses to 2, 3 or 4 to get 2x2, 3x3 or 4x4 tiles. And you might want to drag the code to your shelf for future use.
The script seems pretty robust, but I would recommend saving your scene prior to running it - just in case. The script runs with undo turned off, so you will not be able to go back.
Get djPFXUVs.py from my downloads page.
UPDATE 15 March 2011:
A few script limitations that have been brought to my attention.
- No multiple uv set support. Script only operates on the uv's in map1 (the default uv set).
- Meshes with history may revert to old uv's after script finishes.
- Very little error checking is performed
When I get some more time I'll address some of these issues, but for now you'll just have to live with them.
User interface and grass layout and multiple uv-set support added. New command to load the ui.