Video encoding/decoding in Java

This is a new forum that can be used to discuss anything that's not directly related to JPatch.

Re: Video encoding/decoding in Java

Postby dcuny » Thu Aug 20, 2009 12:58 am

I've gotten a request to create an easy-to-use greenscreen application.

Have you already implemented this sort of thing with your video compositor? If not, do you think it's something that I could put together with this tool chain? I suspect that doing a pixel-by-pixel comparison in Java without graphic acceleration would be rather slow.
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Video encoding/decoding in Java

Postby pndragon » Fri Aug 21, 2009 12:47 pm

dcuny wrote:I've gotten a request to create an easy-to-use greenscreen application.


I don't have anything to add so you can skip this if you want. But I do I have ask for more info
about the who-what-why of the project or my curiosity will go crazy and drive me to drink and loose
women. Please respond; I have my family to think about.

--- Jim
"We're so sorry, Uncle Albert,
But we haven't done a bloody thing all day."
--- Paul McCartney
pndragon
 
Posts: 591
Joined: Sun Dec 05, 2004 1:27 am
Location: North Carolina

Re: Video encoding/decoding in Java

Postby dcuny » Fri Aug 21, 2009 7:24 pm

Mario makes "Brick Films" - a generic terms for "Lego" films. He's got a website where he promotes this sort of thing. I'd written the GlueFace program for him, and he tells me he's getting quite a number of requests from people who want to be able to greenscreen their characters into scenes.

I had suggested Blender, because it's got some really nice compositing tools. But he says it's too complicated for his users. So they're looking for a free alternative to some of the commercial programs.

This basic idea isn't that hard, and I've got some code for doing that sort of thing lying around. Basically, you convert a color from RGB space to HSV space, and then compare to see if how close the color is to the "greenscreen". If it's close enough, you make that pixel transparent. You'll probably need "garbage" mattes as well - areas of the screen that you manually mark as transparent. Overlay the image against another frame, and output the frame. Voila!

I could probably do this as a batch process (like Glueface) without much trouble, but using OpenGL means I could probably do it in realtime.

Did that make sense? :?
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Video encoding/decoding in Java

Postby sascha » Sat Aug 22, 2009 9:28 pm

Yes ;-)

With SD resolutions (e.g. 640x480) this should work in realtime on the CPU as well.
Keep in mind that you'd need a shader, so OpenGL 2.0 would be a minimum requirement if you want to do it in hardware.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Video encoding/decoding in Java

Postby dcuny » Sun Aug 23, 2009 6:38 pm

sascha wrote:Yes ;-)


So is your program available anywhere? :P
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Video encoding/decoding in Java

Postby sascha » Sun Aug 23, 2009 7:00 pm

Well, I have something that loads frame via my JNI ffmpeg adapter and displays them via OpenGL. The raw frame data is in YUV format, and my shader converts it to RGB - you'd need to modify it to convert it to HSV as an intermediate step and it should be straightforward to compute the alpha value by looking it the HSV values in the same step, and then blend the resulting texture over another frame. Alternatively the shader could read both textures (the blue-box frame and the target) and do the mixing too.

The shader code for YUV2RGB is trivial, and I guess using HSV won't be much more complicated:
Code: Select all
uniform sampler2D Ytex, Utex, Vtex;

void main(void) {
    vec2 txCoord = vec2(gl_TexCoord[0].xy);
     
    float y,u,v;
    float r,g,b;
   
    y = 1.1643 * (texture2D(Ytex, txCoord).r - 0.0625);
    u = texture2D(Utex, txCoord).r - 0.5;
    v = texture2D(Vtex, txCoord).r - 0.5;

    r = y + 1.5958 * v;
    g = y - 0.39173 * u - 0.81290 * v;
    b = y + 2.017 * u;
   
    gl_FragColor=vec4(r, g, b, 1.0);
}


You'd need to compile and link the JNI part though.

If you're interested, I can make the files available for you within the next few days.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Video encoding/decoding in Java

Postby dcuny » Mon Aug 24, 2009 4:59 am

Yes, I'd like to get the code some time.

There's no huge rush on this - next week pretty busy at work, so I don't expect to have any spare time.

Thanks! :)
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Video encoding/decoding in Java

Postby dcuny » Mon Aug 24, 2009 6:50 pm

I found a free program called CineGobs that looks like a very slick solution. Unfortunately, it's written in .Net, so it'll only work on the Windows platform. :(

I played around with some of the algorithms last night, and "native" Java is surprisingly fast - at least, under Windows. So I might go that route, since it wouldn't require that I learn something new, and possibly have to deal with OpenGL across multiple platforms.

I've already got the tools to read and write uncompressed .avi files. GlueFace seems to work well with that constraint, but I've been pressured to make it support other formats as well. When working with things like mattes and green screen, compression is not your friends, since it increases the number of artifacts. So I may have a real excuse after all.

Being a lazy coder, I'm trying to figure out the simplest way around this problem. :P

One solution that comes to mind is to use ffmpeg from the command line to generate a "raw" .avi file. The "simplest" solution is to have the user run the ffmpeg themselves, and not deal with it at all. But that's likely to be off-putting for the users. So my question is whether ffmpeg can be embedded into a .jar and then called from the command line to convert a video file.

Of course, this also has to work with multiple operating systems. :|

Is this a feasible approach? Another option would be to install the files into a directory, which probably wouldn't be that difficult, since they could just be zipped together. That's probably a cleaner approach, but having everything in a single .jar has appeal to it.
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Video encoding/decoding in Java

Postby sascha » Mon Aug 24, 2009 9:36 pm

:?

When using the ffmpeg commandline, I strongly recommend piping the frames through stdin/stdout. It's not elegant, but reasonably fast, and it's completely transparent, the user won't even notice that you create an ffmpeg progess.

Then, you could use JNI or JNA to access the ffmpeg library direct. JNI requires writing, compiling and linking the C/C++ glue code (writing and compiling is straightforward, I always struggle with the linking part.) JNA on the other hand seems to be buggy, I get random segfaults when running in a 64 bit environment :(

The best option at this point might be to look into Xuggle. If they deliver precompiled binaries it should be as easy to use as JOGL.

The code I have written so far (for video editing) is quite modular, so it should be easy to replace JNA'd ffmpeg access with JNI'd one, or processes with pipes, of Xuggle, etc.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Video encoding/decoding in Java

Postby dcuny » Mon Aug 24, 2009 10:48 pm

sascha wrote:The best option at this point might be to look into Xuggle. If they deliver precompiled binaries it should be as easy to use as JOGL.
Well, if you figure out how to auto-install Xuggle nicely, I'll consider it. Otherwise, I'm just too stupid/lazy/tired to figure out how to make that sort of thing work cross-platform. :(
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Video encoding/decoding in Java

Postby sascha » Tue Aug 25, 2009 7:15 am

Well, they say they've made it a lot easier to install Xuggle now, but I had no time to check it.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Previous

Return to Off topic

Who is online

Users browsing this forum: No registered users and 2 guests

cron