Skin/Flesh Simulation

Ideas, enhancements, feature requests and development related discussion.

Skin/Flesh Simulation

Postby John VanSickle » Fri Jul 18, 2008 5:56 pm

Ideas for the realistic simulation of semi-rigid structures (skin, muscle, fat, sinew, hair, and cloth) keep coming to mind as I work on my far less capable modeler. There are advantages and drawbacks to starting work on a modeler than can handle this stuff:

Advantages: It's an excuse to start a new project, and if it works, it will be a good time to park my old modeler in a bad neighborhood with the keys in the ignition and walk away from it. If it's really good, I can get a better job than the non-IT job I have now.

Disadvantages: Much of the math hasn't been figured out yet by anybody.

Issue #1: The data structures involved will of course have to be more complex than what we're dealing with for two-dimensional manifolds. I suspect that I will be venturing into areas where there isn't even a Wikipedia page yet, although I did read a PDF about subdivision volumes some time back...

Issue #2: Finding a way to efficiently deal with collisions is going to be a major computative challenge.

Issue #3: I should probably learn Java so that the results can be easily shared across platforms.
John VanSickle
 
Posts: 189
Joined: Sat Feb 16, 2008 2:17 am

Re: Skin/Flesh Simulation

Postby dcuny » Fri Jul 18, 2008 9:29 pm

So you're potentially giving up on LionSnake? :?

Muscle, hair and cloth simulation is already part of most major 3D packages, such as Maya. One thing that Pixar found out was that for cloth to behave correctly, you actually need to tailor the clothing the same way that clothing is made. Similarly, to create a convincing musculature, you need to know the underlying anatomy. It's challenging enough to come create a model, without having to create the underlying muscle system. Pixar got around this by having a standard rig that all characters inherited.

As you noted, dealing efficiently with the problem is a different matter. In The Incredibles, Pixar pre-calculated the deformations:
Essentially, they put a character through a set of exercises— a representative sample of poses — and then used the muscle positions for those poses to train a mathematical representation of the internal coefficients; they’d create a compressed memory of what the muscles looked liked and implement it as an algorithm. "The system would know that when a character looks like this, the muscles look like that," says Rick Sayre, supervising technical director. Thus, the animators could see the result of a dynamic muscle-and-skin system without running a simulation.

Then the tools group spun that idea into clothing. "We trained a statistical model of the cloth," says Anderson. An evolution of the simulation engine developed by Andy Witkin, David Baraff, and Michael Kass for Monsters, Inc. moved the cloth during the training exercise. Once trained, the statistical model took over. That made it possible not only to do cloth sims for multiple characters who appear throughout the film, but also to do so while giving animators control of each major character’s silhouette.
I didn't see a specific paper for it on the Pixar paper page, but I suspect they are using harmonic coordinates to compress the information.
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Skin/Flesh Simulation

Postby John VanSickle » Sat Jul 19, 2008 6:34 am

dcuny wrote:So you're potentially giving up on LionSnake? :?

I often think about the stuff that needs doing and say to myself, "man I'm getting tired of this..." It generally happens when I can't think of elegant or efficient solutions to the problem at hand...

Muscle, hair and cloth simulation is already part of most major 3D packages...

None of which, to my knowledge, are free. That's the gap I wanted to fill.
John VanSickle
 
Posts: 189
Joined: Sat Feb 16, 2008 2:17 am

Re: Skin/Flesh Simulation

Postby dcuny » Sat Jul 19, 2008 9:42 am

John VanSickle wrote:I often think about the stuff that needs doing and say to myself, "man I'm getting tired of this..." It generally happens when I can't think of elegant or efficient solutions to the problem at hand...

Fair enough. It's your time, and you pretty much need a real passion to survive the long slog that coding a large application entails.

None of which, to my knowledge, are free. That's the gap I wanted to fill.

Ah, that clarifies things.

I'd had a go at cloth and hair simulation some time back. Given my lack of higher math skills, they were a bit limited.

You can read about my attempts to write a hair simulator [url=http://forum.jpatch3d.org/viewtopic.php?f=3&t=788[/url]. I didn't get around to writing any collision detection code - I ended up getting bogged down trying to write a zbuffer renderer that would be able to handle hair well. The goal was tp use [url=http://forum.jpatch3d.org/viewtopic.php?f=3&t=788]opacity maps[/url] for handling deep shadowing in hair, and incorporate it into my raytracer with compositing.

Once I got hair working, it was pretty slick. Most papers I had seen discussed using proxies to handle collisions, such as spheres. It's a lot faster to use proxy objects than real geometry.

I never got deep enough into the simulation to deal with hair clumping, which is important when simulating long hair.

Most of the articles I've seen have noted that the hair simulation itself is fairly simple. What's complex is being able to control what the hair does during an animation - for example, ensure that it doesn't fall in front of the character's face, or have it move in a particular direction.

You can also check out my attempts to write a cloth simulator, which was far less successful. I had a lot of trouble handling collisions and stretching. Edges of objects were particularly difficult.

Reading the difficulties with cloth simulation in programs like Animation:Master convinced me that cloth simulation is good to have, but given how difficult it is to set up and use, you probably only want to deal with it in very small amounts.
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Skin/Flesh Simulation

Postby John VanSickle » Tue Jul 22, 2008 2:17 pm

I downloaded the JDK and netBeans from the Sun site the other day, and got HelloWorldApp.java to compile and run. This summarizes my total experience with Java to date. I hope to start coding during my days off (I work nights, so my off time takes place when the rest of the house is asleep).

The only real question I have now is, JOGL or Java3d?
John VanSickle
 
Posts: 189
Joined: Sat Feb 16, 2008 2:17 am

Re: Skin/Flesh Simulation

Postby dcuny » Tue Jul 22, 2008 4:34 pm

John VanSickle wrote:I downloaded the JDK and netBeans from the Sun site the other day, and got HelloWorldApp.java to compile and run. This summarizes my total experience with Java to date.

It's been a while since I've looked at netBeans, I'll have to give it a try. You might also want to check out Eclipse.

The only real question I have now is, JOGL or Java3d?

I was under the impression that Java3d was primarily a scene graph implementation, which made it easier to use, but less flexible than JOGL, which is just a wrapper around OpenGL. I could be wrong - Sascha would know better.
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Skin/Flesh Simulation

Postby sascha » Tue Jul 22, 2008 8:35 pm

David is right, Java3D is a scene-graph API, a bit like Open-Inventor or even VRML - it's quite inflexible and pretty much dead.
JOGL on the other hand is the official implementation of JSR-231 - "Java™ Binding for the OpenGL® API" - it's updated frequently (currently it supports the full OpenGL 2.0 API) and will eventually become part of the Java platform (maybe in 7.0).
It provides a Java method for each OpenGL C function, the syntax is pretty much identical for most functions. There's a set of tools too, check out the API doc here. There are also some demos with source available, but if you're used to OpenGL programming everything should be very familiar anyway.
So, to answer your question, JOGL is the way to go.

Two side notes: 1) Netbeans is cool, but you might want to take a look at Eclipse too. 2) Effective Java is the best book you'll find about Java once your beyond the beginner level (or have experience with another object oriented language) - I can only recommended it.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Skin/Flesh Simulation

Postby John VanSickle » Wed Jul 23, 2008 1:11 pm

sascha wrote:David is right, Java3D is a scene-graph API, a bit like Open-Inventor or even VRML - it's quite inflexible and pretty much dead.

Is there any benefit from importing it, or part of it? I peeked at it a bit and noticed that a number of classes that may be useful are defined in it.

There are also some demos with source available, but if you're used to OpenGL programming everything should be very familiar anyway.

I grok OpenGL, more or less; it's how the gfx are done in LS.

Two side notes: 1) Netbeans is cool, but you might want to take a look at Eclipse too. 2) Effective Java is the best book you'll find about Java once your beyond the beginner level (or have experience with another object oriented language) - I can only recommended it.

Both suggestions are noted. Many thanks.
John VanSickle
 
Posts: 189
Joined: Sat Feb 16, 2008 2:17 am

Re: Skin/Flesh Simulation

Postby sascha » Wed Jul 23, 2008 3:33 pm

Is there any benefit from importing it, or part of it? I peeked at it a bit and noticed that a number of classes that may be useful are defined in it.

Ah, now I see - sorry, my fault. Java3D consists of two packages:
javax.media.j3d - the actual Java3D API (contains the Canvas3D, Node, Shape, TriangleArray, etc. classes).
javax.vecmath - a vector-math package (contains classes like Point, Vector, Matrix3d, Matrix4d, etc).

The latter is actually quite useful - it's lightweight Java (no native code), and I'm using it a lot. It's now a standalone project, hosted here (CVS only) - and it's licensed under the GPLv2 (with classpath exception).

If you're interested in the scene-graph, go ahead and take a look at Java3D. The source is open, and I'm sure that there's a lot of useful stuff in there. I just think that the Java3D API itself is not really suited for my purposes - immediate mode rendering is very, very limited.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Skin/Flesh Simulation

Postby John VanSickle » Thu Jul 24, 2008 1:29 pm

sascha wrote:Java3D consists of two packages:
javax.media.j3d - the actual Java3D API (contains the Canvas3D, Node, Shape, TriangleArray, etc. classes).
javax.vecmath - a vector-math package (contains classes like Point, Vector, Matrix3d, Matrix4d, etc).

The latter is actually quite useful - it's lightweight Java (no native code), and I'm using it a lot. It's now a standalone project, hosted here (CVS only) - and it's licensed under the GPLv2 (with classpath exception).

I'll probably use it. No need to reinvent the wheel.
John VanSickle
 
Posts: 189
Joined: Sat Feb 16, 2008 2:17 am

Re: Skin/Flesh Simulation

Postby dcuny » Fri Jul 25, 2008 11:23 pm

I picked up Cloth Modeling and Animation the other day. It's terribly out of date, but I thought there were a couple of interesting points in it.

The first, reinforced by virtually every article, is that in order to get cloth to behave like cloth (and unlike a sheet of rubber), it's important to understand how real cloth behaves. This is echoed in the Pixar behind the scenes videos, where they bring in seamstresses to create the virtual clothing.

The other interesting article was "Clothing in Disney's 'Fantasia 2000'", which I had seen some time ago. At the time, I didn't think much of it, because the technology seemed pretty pedestrian. But the point was brought up multiple times that cloth is a secondary action which needs to be under the animator's control. Their example was the ballerina's skirt in "The Steadfast Tin Soldier". They cited various issues, such as the skirt rising too high, or having to fix interpenetration problems. The ability of the animator to control or correct the simulation was always of prime importance.

Cloth, hair and muscle simulation seems to me a very invisible sort of task: it works best when no one notices it. :)
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Skin/Flesh Simulation

Postby John VanSickle » Sat Jul 26, 2008 8:54 pm

dcuny wrote:Cloth, hair and muscle simulation seems to me a very invisible sort of task: it works best when no one notices it. :)

The producer's commentary on The Incredibles goes into this. They went around many times on cloth sim and probably only hobbyists like ourselves noticed the amazing technical accomplishments involved.

As for what I've decided: Aside from the topology of the models, the user will define all of the properties of the objects involved (namely, how many data fields are attached to each object to described things like mass, stiffness, etc.). That way I can shove a lot of work onto the user while making it look like I'm allowing for flexibility. As matters stand, there will be control points, edges between control points, faces connecting simple circuits of edges, and cells contained by simple manifolds of faces. The bones will be a special kind of edge. I suppose.
John VanSickle
 
Posts: 189
Joined: Sat Feb 16, 2008 2:17 am

Re: Skin/Flesh Simulation

Postby dcuny » Sun Jul 27, 2008 6:01 am

John VanSickle wrote:The producer's commentary on The Incredibles goes into this.

I was a bit disappointed by the commentary on Ratatouille - there was very little technical discussion at all. :(

While Googling around for some information on simulations, I found out some things about The Incredibles that I wasn't aware of: it had a lot of "cheats." Brad Bird complained that the Pixar folk tried to do solve everything with RenderMan, instead of "thinking outside the box." So (for example) the shot of water in the sink was composited from live action from a kiddie wading pool. There were a number of shots using mattes instead of CGI renderings, and they composited some shots (like the stretching suit) which were rendered with other software.

That way I can shove a lot of work onto the user while making it look like I'm allowing for flexibility.

I can imagine the following tasks would be necessary:
  • You need to create clothing for your character. It would great to have some pre-defined clothing - shirts, pants, dresses - to choose from. Of course, some editing should be allowed, so you can alter the clothing to make it fit the character, or slightly modify the style.
  • Dressing the character is a specialized task. I've seen programs where you create clothing in halves, and the stitch it together slowly, running the simulation and relaxing it as it settles around the character. Again, I could see thing being done with a stand-alone module.
  • Running the simulator is also a specialized task. Once a character is fitted for clothing, I'd like to be able to animate them using a tool of my choice, and then import the animation into the simulator using some common format (perhaps .obj). Only the character and any collision objects would be exported, and the simulator would export the simulated clothing out in some common format.
  • I really like the idea of being able to "bake" the results of the cloth simulator out as morphs. This would be a lot faster than running the simulation each time. You wouldn't get true secondary actions, but you'd get nice wrinkling an stuff, which is what you're probably after most of the time. So once you fit the character, you'd run the simulator for hours and hours, and it would eventually come up with a set of morphs that could be driven from bone angles.
It seems to me that you can divide clothing into two parts: stuff that "wrinkles" (around the shoulders, waist, knees) and stuff that "flops" (skirts, pant legs). Most of the wrinkly stuff can probably be baked, leaving only a smaller part that requires actual "live" simulation.

Of the stuff that needs "live" simulation, since it's less tightly fitting, you should be able to use proxy objects such as cylinders rather than actual geometry.

One of the hardest bits in cloth simulation is getting cloth that's squished together to behave properly. The Untangling Cloth paper covers that. (It also covers self-intersections, another problematic bit). The coolest cheat I ran across for this was from the Animation:Master forum: they just got rid of the geometry in the character's armpits, so there wasn't anything to pinch up against. :)
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Skin/Flesh Simulation

Postby John VanSickle » Sun Jul 27, 2008 7:14 am

dcuny wrote:While Googling around for some information on simulations, I found out some things about The Incredibles that I wasn't aware of: it had a lot of "cheats." Brad Bird complained that the Pixar folk tried to do solve everything with RenderMan, instead of "thinking outside the box."

I'm tempted to think of it as "colliding visions." The long-time Pixar people wanted to stick to one tool as much as possible (which, admittedly, does make life simpler if the tool is robust). Brad wanted the movie done with high quality and efficiency.

John VanSickle wrote:That way I can shove a lot of work onto the user while making it look like I'm allowing for flexibility.

I can imagine the following tasks would be necessary:
  • You need to create clothing for your character. It would great to have some pre-defined clothing - shirts, pants, dresses - to choose from. Of course, some editing should be allowed, so you can alter the clothing to make it fit the character, or slightly modify the style.

Designing the clothes is, from the computer's viewpoint, a task of modeling the topology, and adding the various properties to the various materials.

I hope to make clothing that can re-size itself to match the model (which means fitting control points for each model, and model control points that can be input automatically from a third model instead of being defined at model creation). The people who use Poser go round and round with getting clothes to fit and move with the models. As matters stand, if you want to dress up Michael in the French Maid outfit, you have to download the special model they released as an April Fool's joke.

  • Dressing the character is a specialized task. I've seen programs where you create clothing in halves, and the stitch it together slowly, running the simulation and relaxing it as it settles around the character. Again, I could see thing being done with a stand-alone module.

Yes, to make clothes fit right in the first place requires the sim to have a whack at them.

  • Running the simulator is also a specialized task. Once a character is fitted for clothing, I'd like to be able to animate them using a tool of my choice, and then import the animation into the simulator using some common format (perhaps .obj). Only the character and any collision objects would be exported, and the simulator would export the simulated clothing out in some common format.

The sim module really has to be quite separate from the modeling, animation, and rendering modules, and perhaps will probably run best as a separate application. As long as the different modules can communicate, that shouldn't be a problem.

I can imagine the sim being handed a set of control points with properties and topology, and told to run such-and-such a sim, and return the results either as new control points or as an OpenGL window with geometry displayed. The phrase "such-and-such a sim" involves many details (time, gravity, etc). I think the sim module will be a script runner combined with something to pass data, with a simple renderer thrown in.

  • I really like the idea of being able to "bake" the results of the cloth simulator out as morphs. This would be a lot faster than running the simulation each time.

Simply write a sim output script to generate a morph data file. "I have reduced it to a problem already solved (in theory)."

One of the hardest bits in cloth simulation is getting cloth that's squished together to behave properly. The Untangling Cloth paper covers that.

Self-intersection is definitely a problem that requires a more efficient solution than what we have now. Pixar and AM both resort to a "hide it" strategy, although somewhere I did see a Pringle Chip solution, where two objects that the animator knew were going to collide would have an invisible surface placed between them, with the two objects pressing against that; that way the objects were only tested against a small set of geometry, instead of everything in the whole model (or scene :roll: ). If you can't get away from O(N^2), at least make N as small as possible...
John VanSickle
 
Posts: 189
Joined: Sat Feb 16, 2008 2:17 am

Re: Skin/Flesh Simulation

Postby dcuny » Sun Jul 27, 2008 7:46 am

John VanSickle wrote:I hope to make clothing that can re-size itself to match the model (which means fitting control points for each model, and model control points that can be input automatically from a third model instead of being defined at model creation).

Wow, you're pretty ambitious. I was thinking that you might select scales and sizes, or pull points around. But automatically fitting clothing? Impressive.

The people who use Poser go round and round with getting clothes to fit and move with the models. As matters stand, if you want to dress up Michael in the French Maid outfit, you have to download the special model they released as an April Fool's joke.

Poser is probably a good point of reference, since making clothing for characters there is a big thing.

"I have reduced it to a problem already solved (in theory)."

Yeah, the textbox limit is just a bit to small, or I'd post this really elegant code that I've got. :P

...although somewhere I did see a Pringle Chip solution, where two objects that the animator knew were going to collide would have an invisible surface placed between them, with the two objects pressing against that; that way the objects were only tested against a small set of geometry, instead of everything in the whole model (or scene :roll: ). If you can't get away from O(N^2), at least make N as small as possible...

I like that cheat. :)
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Next

Return to Development

Who is online

Users browsing this forum: No registered users and 2 guests

cron