GlueFace

This is a new forum that can be used to discuss anything that's not directly related to JPatch.

GlueFace

Postby dcuny » Sun Mar 08, 2009 9:54 am

I've been working on a program tentatively titled GlueFace for a Lego film maker. It's a specialized program that takes an existing X Sheet (like Yolo generates) and maps it to a set of mouths. You can then graphically position the mouths onto frames of an .avi file and generate a new .avi file. Here's a screenshot:
glueface.png
The timeline allows simple linear interpolation between frames. Most of the heavy lifting is performed by Java's Graphics2D class, although I've mapped the mouth to a cylinder, so you can "spin" the face around in 3D. When the user suggested it, I figured that was one of those impossible tasks. But it turned out to be simpler than I thought, even for someone as math challenged as myself. :)

I had promised that it would be done around Christmas of last year, and I'm just now finishing it up. I tracked down about 4 bugs tonight that were all based in precision errors. The arccos function gets very unhappy with values even just a little bit out of range. :?
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: GlueFace

Postby sascha » Sun Mar 08, 2009 12:46 pm

Cool! :)
When the user suggested it...

You mean, you've already released it without telling us? :shock:
Where can it be downloaded?
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: GlueFace

Postby pndragon » Sun Mar 08, 2009 4:39 pm

That looks like a fun tool!
"We're so sorry, Uncle Albert,
But we haven't done a bloody thing all day."
--- Paul McCartney
pndragon
 
Posts: 591
Joined: Sun Dec 05, 2004 1:27 am
Location: North Carolina

Re: GlueFace

Postby dcuny » Sun Mar 08, 2009 7:32 pm

sascha wrote:You mean, you've already released it without telling us? :shock:

Relax, it's still in beta. ;)

Without a video file, and X Sheet and mouth shapes, it's worthless. It's really a "one trick pony" kind of utility which is only useful it you're making brickfilms.

Well, I guess you could turn off the cylinder mapping and add the mouths to existing animation...

Where can it be downloaded?

The beta is probably coming out in a day or so. I thought I tracked down those bugs last night, but the user reported he's still seeing them. :( Hopefully I'll get a test case so I can duplicate it.

I'm going to put up a SourceForge page at some point, but I haven't decided on the license for it yet. I haven't looked at any licensing in a while - it GPL still a good option?
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: GlueFace

Postby sascha » Sun Mar 08, 2009 7:58 pm

...but the user reported he's still seeing them.

Ah, now I get it. Could it be that we witness the dawn of a new generation of film-makers here?
The beta is probably coming out in a day or so.

Can't wait to play with it...
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: GlueFace

Postby dcuny » Mon Mar 09, 2009 6:12 am

My (only) user is Mario Baumgartner, who's put together a number of Brickfilms. He ran across the JLipSync program and wondered if I could make some "minor" adjustments to it to support what he needed. What got his attention was the ability to generate .avi files, which is perhaps the weakest part of the program. :roll:

He's got a Brickfilm website, so he'll be hosting the program. But it'll show up on SourceForge once I figure out what license I'm using.

The program is just part of a larger toolchain. To use it, you first need to:
  • Create an uncompressed .avi file.
  • Create an X Sheet of the lipsync with something like Yolo or Papagayo.
  • Create a set of mouths for the character.
Once you've got that, you load up the .avi file, load the lipsync file, and load a mouths file which describes the mouth set you're using (it's a simple .txt file). You then use the various tools to manipulate the faces into position. You can scrub through the timeline, but if you want to hear the result, you have to create an .avi file and play it with another program.

I sort of punted on a number of things. For example, you can use the mouse controls to rotate things around, but I track the change in mouse position rather than the actual angle. I went through a number of iterations, starting out with it being rather clever. Eventually, I added more functions, and it made sense to use the traditional toolbar. It's still simple enough that all the features can fit into the top toolbar instead of a menu system.

I think I tracked down the last known major bug. I'm sure there are others to be discovered, though. ;)
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: GlueFace

Postby sascha » Mon Mar 09, 2009 1:01 pm

Ok, I thought that maybe one of your kids would try to make a Lego-animation :)

About avi-export: I see that the (uncompressed?) AVI export makes sense, I would, however, focus on 3rd party tools to encode and watch animations. mencoder (the encoder part of mplayer) comes to mind. Sure it's a command-line utility and the sheer number of command-line options scares away most users, but if it can be controlled from a GUI-application, the only scary part for the users is to somehow get the binaries on the disk and point the application to the executable.
It's pretty simple to start an external application from within Java, it's easy to set the environment and all command-line options, and you get an input- and an output-stream to communicate with the application during runtime (so you could e.g. read the mencoder progress by reading the input-stream, or control mplayer playback by sending keyboard commands to the output-stream).

I think this would make sense for a lot of applications (I mean remote-controlling mencoder and mplayer from a Java GUI), so it would also make a lot of sense to make this a standalone-package that could then be used by JPatch, JLipsying, GlueFace, and whatever. If you're OK with mplayer/mencoder I'd volunteer to put some time into this. I think they're available for all three major platforms (Windows, OSX and Linux) and are quite powerful tools. mplayer can playback about just anything (including raw streams and sequences of image-files), and mencoder can encode to a wide variety of formats. A distribution could contain the Java packages and the mplayer/mencoder executables, together with a number of standard codecs (there could be patent issues in the U.S., so it should only contain free codecs, but make it easy to install the patent-encumbered ones as an option).

What do you think?

Ah, and about your license question: Personally I'm happy with the GPLv2 and v3. I am not a lawyer, but as far as I understand you can (as long as you own the copyright for the entire code, i.e. have written it all by yourself) do whatever you want with your code at any time. If you're not happy with the GLP, you can even sell a later version under some proprietary license. Things get complicated as soon as others contribute - as they contribute under the terms of the (for example) GPL. If you'd later change the license, you'd need their approval or you'd have to remove their contributions. This is easy as long as only code is concerned, but I think it would even be unfair to use user-feedback and bug-reports. I'm not sure about the legal issues here, but I'd be quite angry if someone would use my feedback (which I gave for a FLOSS application), use it to make her or his application better and then sell it under some proprietary license.
The GPL is quite strict, it even prevents proprietary code to link against your binaries. To some companies this is too radical, that's the reason for some more business-friendly licenses like the LGPL, BSD- and Apache-licenses, and this is also the reason for some companies to use dual-licensing (Qt and MySql are the most prominent examples, I think.) JPatch is non-commercial though, I don't want any commercial software to even link against it, so I'm perfectly happy with the GPL (and I also won't change the license at some point in the future, don't panic!)
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: GlueFace

Postby dcuny » Tue Mar 10, 2009 12:52 am

sascha wrote:Ok, I thought that maybe one of your kids would try to make a Lego-animation :)

I think they'd be just as happy to hold a Lego figure in front of a camera and move it. ;)

The older boy had done some animation, but it's all 2D at this point.

I would, however, focus on 3rd party tools to encode and watch animations. mencoder (the encoder part of mplayer) comes to mind.

The main question I didn't want to deal with was having to handle multiple platforms. Since the animation was shot frame by frame in the first place, I'm surprised that wasn't suggested as an option. There was another utility on his site, and it dealt directly with the frames instead of video.

Java has some native support for .avi files, but going through the documentation, it looked like a real pain to code. I'm not sure it supported compressed files, though.

I had played with mencoder for my faux video editor, but it seemed a bit slow. So I wasn't that motivated to go that direction, especially since I had code on hand which already worked.

But it would be nice for JPatch to have this sort of functionality, so yes, it would be nice to have at some point.

Ah, and about your license question: Personally I'm happy with the GPLv2 and v3.

I'll go that direction, then. I'll have to check where the .avi code came from - I believe it was public domain.
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: GlueFace

Postby sascha » Tue Mar 10, 2009 11:42 am

I had played with mencoder for my faux video editor, but it seemed a bit slow.

AFAIK mencoder itself is just a kind of frontend, and it uses several different codecs (like ffmpeg) under the hood - so in theory you should get state of the art encoding and decoding with mencoder and mplayer, you just have to find the right commandline-switches ;-)
What I think would be cool is some way to bypass temporary image files. This won't work for larger 3D projects (where you need some sort of compositor), but it should for simple animations. I think it should be possible to use stdin/stdout to pass the frames to the encoder, so there's no need for some 1000 temporary image files. Mencoder would simply read the images from stdin (where the Java frontend would send them to) and encode them to any format the user wants. I agree that cross-platform compatibility is an issue, but fortunately I'm able to test in on all three platforms. I'll give it a shot when I have time...
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: GlueFace

Postby dcuny » Tue Mar 10, 2009 5:41 pm

Coincidentally, ffmpeg just announced their 0.5 release.
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: GlueFace

Postby sascha » Wed Mar 25, 2009 3:52 pm

I've started a new thread about video encoding/decoding in Java here
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: GlueFace

Postby dcuny » Wed Mar 25, 2009 5:28 pm

There's now a forum thread for GlueFace users to post their bugs to on the Brick Filmer's forum. There's a link to GlueFace at the top of the thread.
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: GlueFace

Postby sascha » Thu Mar 26, 2009 2:43 pm

I've just watched the video tutorial and... WOW, seems like you've created some cool piece of software here. Now I simply have to try and make a short brick-film.
One question thoug: since both JLipsync and Yolo seemed pretty complete and usable, why the extra Papagayo step? I think an integrated dope-sheet editor would especially be helpful when animating the eyes too, since you can then spare the second Papagayo pass.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: GlueFace

Postby dcuny » Thu Mar 26, 2009 6:07 pm

sascha wrote:since both JLipsync and Yolo seemed pretty complete and usable, why the extra Papagayo step?

When I went to do lipsync for the Moai video, I ended up using Papagayo because of issues with Yolo. I should fix them at some point. But it seems to me better to have each tool do one thing, and do it well, instead of integrate everything together.

Mario (the person who asked for the software) thinks otherwise.

I think an integrated dope-sheet editor would especially be helpful when animating the eyes too, since you can then spare the second Papagayo pass.

I'm thinking of adding some editing to the timeline to allow editing the eyes. I'd originally had an "eye" track, but removed it. It'll probably appear in the next major release.
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: GlueFace

Postby sascha » Thu Mar 26, 2009 7:57 pm

Another interesting option would be vector graphics mouth shapes. This way they could be interpolated smoothly. It's a matter of taste, I know that you prefer the "switching" mouth shapes, while I'd prefer a more smooth way of blending between them.

SVG would be one option (there's an Apache project called Batik, which is an excellent Java SVG renderer) but I think it's overkill.
Something moch more simple could use Java2D "GeneralPath" objects - the can consist of line segments and (cubic) curves. They could be rendered as outline (with a specified line-thickness and color), filled (with a color or gradient), or both. A GUI to draw with these and allow to add morphs (by moving control-points of the "path" objects) should be relatively easy to write.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Next

Return to Off topic

Who is online

Users browsing this forum: No registered users and 1 guest

cron