Video encoding/decoding in Java

This is a new forum that can be used to discuss anything that's not directly related to JPatch.

Re: Video encoding/decoding in Java

Postby John VanSickle » Thu Apr 02, 2009 12:52 pm

dcuny wrote:Of course, don't let John see it, or he'll argue that it's proof IRTC videos are better without sound. :wink:

Already seen it. I've never said that they were better without sound; what I've said is that I do my own part of the judging with the sound turned off. The only reason I haven't done any sound in my own work is because I haven't found tools I like for editing sound in an MPG.
John VanSickle
 
Posts: 189
Joined: Sat Feb 16, 2008 2:17 am

Re: Video encoding/decoding in Java

Postby sascha » Thu Apr 02, 2009 1:40 pm

Hmmm...
I think that the rule said that you should not judge based on the quality of the audio-part of any submitted animation. Turing the sound off is consequent, but (for my taste) a bit radical.
I also think that the intention of the "no sound" rule was that some of the judges might not have sound-playback capabilities on their Sun-Workstations, and so the animations should not rely on a soundtrack - this might have been true in times where computers with an internet connection used to be Sun-Workstations, but certainly isn't true anymore for at least a decade now.
Some judges (IMHO) even penalized submissions for using a soundtrack, something that's clearly never been intended by the IRTC admins.

Anyway, there's been enough heated discussion about this on the POV-Ray newsgroups.

What I'd second is that, in general, animated shorts without a spoken dialog are better than those that rely on dialog. The problem with this one was that one of the very few things JPatch could do at that time was lip-syncing, so I've been trying to show off with it ;-)
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Video encoding/decoding in Java

Postby dcuny » Thu Apr 02, 2009 6:43 pm

sascha wrote:Here's a list of all the effects:

I thought there might have been a playback change (or dropped frames), but it's hard to tell with YouTube some times.

The vertical scratch positions actually don't use random but a sine-wave plus some harmonics.

Clever.

The nice thing about the filter is that it runs in realtime (at least for low resolution material) - I'll put it online soon.

I suspect there'd be some interest in something like this, especially if it was glued together with the ffmpeg code or something.

I've also posted a "confession" to the IRTC newsgroups here
Heh. :)

it's definitely time to continue JPatch development and make some new animations with it.

Yay :mrgreen:
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Video encoding/decoding in Java

Postby sascha » Thu Apr 02, 2009 6:54 pm

You can click on "HD" on youtube now. There's an MPEG-1 version here, and I'll also upload an MPEG-4 version.

Right now it uses mencoder as input, but it's modular, so once I get xuggler to work, I'll add ffmpeg sources and destinations as well.
The funny thing is still that you can patch it between mencoder and mplayer, and most effects will run in realtime (unless it's full HD, but then chances are that mplayer can't play it in realtime anyway). Right now sound is bypassed though - but there should be ways around that. One would be to use fifos (aka named pipes) on linux, or simply abuse stdin for the video stream (that's what I do now) and stderr in parallel for the audio.
The other option is to use uncompressed avi as "in between" format and just one stream...
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Video encoding/decoding in Java

Postby sascha » Thu Jun 04, 2009 12:01 pm

I spent some time fiddling about video encoding/decoding with Java, and found out quite a few interesting things:

Using Runtime.exec(...) to start commandline processes (like mplayer, mencoder or ffmpeg) and piping video frames through their stdin/stdout worked surprisingly well, but it has a few shortcomings. While performance is good, it doesn't feel really "cool" to do it that way, and it certainly is a waste of resources. And after all, the ffmpeg commandline program is a lot less flexible than ffmpeg's libavcodec. Long story short, I was looking for a way to directly call libavcodec functions from Java.

* JNA

First I ran across Java Native Access (JNA), which uses a thin native-code wrapper and allows Java to directly call any native functions without the need to compile your own JNI glue-code. This looks very promising, and I was able to write a short test code to use libavcodec to decode mpeg-frames: My program uses pure Java to parse an AVI file (including index) and read frames, passes them to livabvodec's avcodec_decode_video() function and receives a decodec YUV4:2:0 frame for further processing. There's even a prject called ffmpeg-java, which is a Java wrapper around livavcodec that uses JNA to call the native functions.

Very nice, but:
1.) On OSX (I wasn't at home and had the mac-book with me, so I haven't tried it on Linux or Windows yet), everything works fine with 32bit, but when using a 64bit JVM, I got random segfaults when calling livavcodec functions. Ugh!
2.) While you don't need to write JNI-glue code, the Java code becomes cluttered with wrappers around C pointers and other ugly stuff. It's never been easier to crash the machine from within pure Java code.

* JNI

So after a while I decided to give JNI a try, and after some infights with the ld linker I finally got my first JNI library. It also allows my Java programm to access some of the libavcodec funtions and so far works like a charm. Sure, you have to compile the native libraries for each target platform - but since I'd limit them to i386 and amd64 on Windows, Linux and OSX, I think this is doable (and I guess it should be possible to cross-compile the JNI code for all platforms with a single make file, but I'll have to look into this).
The result is clean Java and C code (calling Java methods from C using JNI looks a lot less ugly then calling C functions from Java using JNA, and when it segfaults at least I'm sure it's in the C code ;-)), well worth going through the hassle of writing and compiling the glue-code - it's not that hard after all, and if you're interested, here's the code:
VideoCodec.java:
Code: Select all
import java.nio.ByteBuffer;
public class VideoCodec {
   static {
      /* load JNI glue */
      System.load("/Users/sascha/Documents/VideoCodec/build/Debug/libVideoCodec.dylib");
   }
   static int count = 0;
   
   private native boolean init(String codecName, int width, int height, int bufferSize);
   private native void freeResources();
   
   /* returns a ByteBuffer (of size bufferSize) to write raw (e.g. mpeg-compressed) frames into
   public native ByteBuffer getRawFrameBuffer();

   /* returns the ByteBuffer containing the decoded frame (has to be called after decodeFrame)
   public native ByteBuffer getDecodedFrameBuffer(int plane);
   
   public native int decodeFrame(int size);
   
   private final int uid = count++; // we use this to map native resources to instances of Java VideoCodec objects.
      
   public VideoCodec(String codecName, int width, int height, int bufferSize) {
      if (!init(codecName, width, height, bufferSize)) {
         throw new RuntimeException("could not initialize ffmpeg codec");
      }
   }

   protected void finalize() {
      freeResources(); // clean up
   }
}

javah will generate the corresponging VideoCodec.h file, and here's the implementation:
VideoCodec.cp:
Code: Select all
#include <JavaVM/jni.h>
#include "VideoCodec.h"
extern "C" {
   #include "libavcodec/avcodec.h"
   #include "libavformat/avformat.h"
}
#include <map>
#include <iostream>

using namespace std;


typedef struct AVParams {
   AVCodec *avCodec;
   AVCodecContext *codecCtx;
   AVFrame *avFrame;
   AVPacket *avPacket;
   int rawSize;
   int height;
};


map <jint, AVParams *> objectMap;

jint get_av_uid(JNIEnv * env, jobject obj) {
   jclass cls = env->GetObjectClass(obj);
   jfieldID fid = env->GetFieldID(cls, "uid", "I");
   return env->GetIntField(obj, fid);
}

JNIEXPORT void JNICALL Java_VideoCodec_freeResources (JNIEnv * env, jobject obj) {
   AVParams *avParams = objectMap[get_av_uid(env, obj)];
   av_free(avParams->avCodec);
   av_free(avParams->codecCtx);
   av_free(avParams->avFrame);
   av_free(avParams->avPacket);
}

JNIEXPORT jboolean JNICALL Java_VideoCodec_init (JNIEnv * env, jobject obj, jstring codecName, jint width, jint height, jint bufferSize) {   
   av_register_all();
   
   const char *c_string = env->GetStringUTFChars(codecName, 0);

   cout << c_string << "\n";
   
   AVCodec *avCodec = avcodec_find_decoder_by_name(c_string);
   if (avCodec == 0) {
      return JNI_FALSE;
   }
   
   AVCodecContext *codecCtx = avcodec_alloc_context();
   if (codecCtx == 0) {
      return JNI_FALSE;
   }

   if (avcodec_open(codecCtx, avCodec) != 0) {
      av_free(avCodec);
      av_free(codecCtx);
      return JNI_FALSE;
   }

   AVFrame *avFrame = avcodec_alloc_frame();
   if (avFrame == 0) {
      return JNI_FALSE;
   }   

   AVPacket *avPacket = new AVPacket();
   avPacket->data = (uint8_t*) av_malloc(bufferSize);
   
   AVParams *avParams = new AVParams();
   avParams->avCodec = avCodec;
   avParams->codecCtx = codecCtx;
   avParams->avFrame = avFrame;
   avParams->avPacket = avPacket;
   avParams->rawSize = bufferSize;
   avParams->height = height;

   objectMap[get_av_uid(env, obj)] = avParams;

   return JNI_TRUE;
}

JNIEXPORT jobject JNICALL Java_VideoCodec_getDecodedFrameBuffer(JNIEnv *env, jobject obj, jint plane) {
   AVParams *avParams = objectMap[get_av_uid(env, obj)];
   int height = plane == 0 ? avParams->height : avParams->height / 2;
   return env->NewDirectByteBuffer(avParams->avFrame->data[plane], (jlong) (avParams->avFrame->linesize[plane] * height));
}

JNIEXPORT jobject JNICALL Java_VideoCodec_getRawFrameBuffer(JNIEnv *env, jobject obj) {
   AVParams *avParams = objectMap[get_av_uid(env, obj)];
   return env->NewDirectByteBuffer(avParams->avPacket->data, (jlong) (avParams->rawSize));
}

JNIEXPORT jint JNICALL Java_VideoCodec_decodeFrame(JNIEnv *env, jobject obj, jint size) {
   AVParams *avParams = objectMap[get_av_uid(env, obj)];
   avParams->avPacket->size = size;
   int got_picture;
   avcodec_decode_video2(avParams->codecCtx, avParams->avFrame, &got_picture, avParams->avPacket);
    return got_picture;
}


I don't need to expose all libavcodec funtions to Java - actually I need just a few, clean Java methods to encode and decode video frames, and perhaps to read/write to container files. So I think the best method to do this is to write some short JNI code that handles libavcodec under the hood and exports some high-level functions to Java.

What do you think?

But back to JNA - although it seems to be a bit unstable (the crashes on 64bit OSX) and Java code using JNA looks horrible, it has a cool solution for the problem of installing/loading native libraries - there is a helper method that detects the platform, extracts the needed libraries from the jar-file into a temporary folder and loads them via System.load(). As this might cause problems with some paranoid system configurations (SELinux) it's only done if the native libs can't be found on the system. This means that the user can install the native libraries, and if he does, they will be used. If he doesn't, there's a 99% chance that they can be extracted and loaded on the fly - so everything is in a single, self-contained .jar archive, and the novice user doesn't have to install anything. I remember that I have tried something like that before, but didn't get it to work - but I'll use this method of loading native libraries for JPatch as well...

* A Java video-editor
So, using libavcodec functions in Java finally enables me to proceed with my long-term sub-project, a video editor in Java:
Here's what I have in mind:

* Simple user interface (like iMovie, but perhaps featuring a shot->scene->sequence->movie hierarchy for larger projects)
* Use either sequences of raw frames (e.g. png) or any video files libavcodec can read as input (avi, mov, mpeg1/2/4, etc.)
* Output to anything libavcodec can output to (avi, mov, mpeg1/2/4, etc.)
* Adding transitions (crossfade in the first version, more later)
* Adding postprocessing effects (brightness, contrast, color - more later)
* A simple audio editor (in a later version)

Everything should be done in realtime - and since the source material could be 1080p HD frames (or even 4k), possible mpeg encoded (so you don't have random access to each frame, but need to decode them in sequence (P and B frames), here are a few ideas:
* When a clip is opened, ffmpeg is used to downscale it to preview size (e.g. 640x480) and stores it as mpeg file with I-frames only (this way each frame is self-contained)
* When working on the video, these preview mpeg files are used - their size is limited and frames can be accessed in random order, so this should be quite fast.
* All processing (colorspace conversion, chroma subsampling, blending and effects) is done in OpenGL (using GLSL fragment shaders). This should be very fast, so for the preview all effects can be computed in realtime.
* Once finished, all edits are applied to the original clips (like with AVID) - it's still using OpenGL so this should be quite fast too, but doesn't have to be realtime.

This also allows collaboration features: The original movie (e.g. several gigabytes of uncompressed 1920x1080 frames) could be anywhere, all internet users would work with the relatively small preview files. This way svn or cvs could be used to edit the final film. Once it's done, the one with the original files would check out the edit-decision-list and apply it to the original clips, thus producing the final, edited HD output.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Video encoding/decoding in Java

Postby dcuny » Fri Jun 05, 2009 11:06 am

Sounds good!

As an end user, I only really care about a couple things, and it looks like you've taken care of them:

  • Is it easy to install? The more automatic, the better. The more I have to fiddle with it and set environment variables, the less likely I am to install it, no matter how great it is.
  • Will it run on my hardware? I've recently upgraded my 10 year old machine, so I can now run stuff that I couldn't before. But in general, I dislike programs that require I have the latest and greatest of anything, and lock me out of the software promised land because I can't afford cutting-edge hardware.
  • Will it run acceptably fast on my machine? This is a variation on the above theme.
As I said, it looks like you've got those bases covered. :)
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Video encoding/decoding in Java

Postby sascha » Fri Jun 05, 2009 11:49 am

Unless you're on SE-Linux, it should start up and load all required libraries by simply doubleclicking on the jar-file. If this doesn't work, I'll add my old JPatch "native library helper" that will guide you through the library installation (in the worst case, you'll need to restart it as root/Adminisrator and try it again).

I'd bundle precompiled native libs for Linux(i386 and amd64), Windows(i386) and OSX (i386 and amd64). Note that amd64 is equivalent to x86_64, so it will work on both, 64 bit Intel and AMD chips. If you use a different platform/architecture you'll have to compile it yourself, but I think the above should cover 99% of all users - perhaps I'll include PPC Macs too.

So the only showstopper would be OpenGL 2.0. On Linux, type
Code: Select all
glxinfo | grep "OpenGL version"
. Anything above 2.0 is fine (in theory 1.5 would also provide the shader extensions, but if the hardware supports programmable shaders, there should be a driver update to at least 2.0 available).
It would be possible to add a software path that uses pure Java (or libavcodec) for colorspace conversion, chroma subsampling/reconstruction and the effects, but it would make the desing of the application much more complicated, and each and every new effect would need an OpenGL and a Java implementation. And given the fact that my 5 year old Laptop at work supports OpenGL 2.0 I think it's save to assume that 99% of all users will have OpenGL 2.0 available - every graphics card today supports it (the current version is 3.0), even old AGP cards. So I'd prefer to make OpenGL 2.0 support a minimum requirement.

But as a C/C++ newbie I've got three more questions:
1. How does cross compilation work, and what is required (do I need access to a Windows installation from my Linux box to cross-compile to a Windows DLL?)
2. If I use the GNU compiler (gcc) on Windows, is the end user required to install mingw or cygwin, or will the resulting DLLs work on any Windows box?
3. What I still don't get is how these dynamic libraries (.dll, .so, .dylib) files actually work. What exactly is the compiler output, what does the linker do, and how are the libraries loaded? For example, I wrote my JNI functions and compiled and liked it into a "libvideocodec" file. Of course it requires ffmpeg's "libavcodec", so I had to include the avcodec.h headers from ffmpeg during compilation, and had to point the linker to the libavcodec library file. Now if a program loads my libvideocodec library, how does it know that it needs to load libavcodec as well, and more important, how does it know where to find it?

Sorry if these are stupid questions, I'm just really new to C/C++ - perhaps you could point me to some docs on the Internet where these things are explained in detail, or recommend a good book?
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Video encoding/decoding in Java

Postby dcuny » Fri Jun 05, 2009 6:44 pm

sascha wrote:But as a C/C++ newbie I've got three more questions:

I don't really have expertise here. John's probably the best resource.

...do I need access to a Windows installation from my Linux box to cross-compile to a Windows DLL?

I don't believe so. The compiler should be able to target Windows without Windows actually being installed.

If I use the GNU compiler (gcc) on Windows, is the end user required to install mingw or cygwin, or will the resulting DLLs work on any Windows box?

I've only compiler using the old Borland compiler, so I don't know. I have a suspicion that you do need the mingw or cygwin dlls.

What I still don't get is how these dynamic libraries (.dll, .so, .dylib) files actually work.

I believe they work the same way under Linux as they do in Windows. That is, references are made to external .dlls, and they aren't resolved until you try to run the program.

You can call a dll dynamically from a program, which is what most interpreters do. I'm guessing that compiled programs attempt to link to the dlls when they are loaded, but I don't know for sure. I believe they use various environment variables to track down the locations of the dlls.

Of course, I could be completely wrong about this. Dealing with issues like this is one of the reasons I started using Java in the first place. :|
dcuny
 
Posts: 2902
Joined: Fri May 21, 2004 6:07 am

Re: Video encoding/decoding in Java

Postby sascha » Sat Jun 06, 2009 8:58 am

Dealing with issues like this is one of the reasons I started using Java in the first place.

Full circle :)
I'll have a shot at it, and if it tends to be too complicated, I can fall back to JNA (which means ugly Java code, but no need to compile any native code.) I'd have to resolve the segfault issues on 64bit OSX though, but perhaps they're gone in the dev-branch of JNA.
But since I need to provide native libs for JOGL anyway (ok, I don't have to compile them) and ffmpeg (I'd have to compile them at least for OSX, since I couldn't find any precompiled binaries.)
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Video encoding/decoding in Java

Postby sascha » Tue Jun 09, 2009 12:52 am

My JNI adapter to ffmpeg's libavcodec works fine on Linux too (I still need to test this on Windows.)

I've also written a simple glsl shader that converts from yuv4:2:0 to rgb and used it to display the frames.
This way there's almost no Java code in the "pipeline", and performance is great: at 1280x720 @ 60fps (!) it uses just about 40% (of a single core) for mpeg-2 decoding (libavcodec), chroma reconstruction, colorspace conversion and image scaling (all OpenGL) - so the Java part can focus on the user-interface, while all the hard work is done by ffmpeg and the graphics hardware (the latest ffmpeg versions even supports nVidia's VPDAU to use hardware h.264 decoding.)

Since typical previews (for editing) will be smaller (e.g. 640x480, etc.) and framerates are typically 24, 25 or 30 fps, this should work very well on slower machines too.

Here's the shader code:
Code: Select all
uniform sampler2D Ytex, Utex, Vtex;

void main(void) {
    vec2 txCoord = vec2(gl_TexCoord[0].xy);
     
    float y,u,v;
    float r,g,b;
   
    y = 1.1643 * (texture2D(Ytex, txCoord).r - 0.0625);
    u = texture2D(Utex, txCoord).r - 0.5;
    v = texture2D(Vtex, txCoord).r - 0.5;

    r = y + 1.5958 * v;
    g = y - 0.39173 * u - 0.81290 * v;
    b = y + 2.017 * u;
   
    gl_FragColor=vec4(r, g, b, 1.0);
}

The nice thing about it is that chroma-reconstruction (the inverse of chroma-subsampling) is done implicitly by the texture2D(...) calls.
If the U- and V-textures happens to be smaller than the Y-texture, the shader automatically takes care of it, the subsampling mode (4:4:4, 4:2:2, 4:1:1, 4:2:0, etc.) doesn't matter at all.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Video encoding/decoding in Java

Postby xuggle » Wed Jun 24, 2009 4:34 am

Hi guys,

Just to chime in from earlier in this thread, we (Xuggle) hear you loud and clear about things being too complex with our advanced API, and so to (hopefully) simplify things, we've introduce a new simpler API:
http://blog.xuggle.com/2009/06/05/intro ... ediatools/

And, on our build server, we've started to introduce pre-compiled bundles that you can just install. We'll do more on the installers on (look for it in July), but for now you can find the latest Linux (Ubuntu 9 binaries) here:
http://build.xuggle.com/view/Stable/job ... k5_stable/ (look for the "*-all-bin*" binaries

And for Windows:
http://build.xuggle.com/job/xuggle_java ... 386_winxp/ (look for the setup*.exe binaries for a Windows installer).

- Art (co-founder at Xuggle)
xuggle
 
Posts: 3
Joined: Wed Jun 24, 2009 4:29 am

Re: Video encoding/decoding in Java

Postby sascha » Thu Jun 25, 2009 9:56 am

Cool, thanks a lot for the info, I'll have a look at it.

For my little video-editor sub-project I'd like to keep the part that does the video encoding/decoding separate (and thus interchangeable) - so I can use ffmpeg natively (via JNI), per commandline or xuggle and see which works best (or keep all three as an option and let the end-user decide which one to use.)

I browsed through the xuggle java-docs (just briefly, I have to admit) and learned that you use AWT images to access the frames from Java (which is great if you want to use Java2D to alter the image or draw something into it,) but I wonder if you also can access the raw (chroma subsampled) YUV planes (e.g. via a direct ByteBuffer) in xuggle? For the majority of the transitions and effects I have in mind I don't need any YUV-to-RGB conversion or chroma reconstruction, and I'd also like to do as much of the processing work as possible on the GPU, so direct access to the raw YUV frames would speed up things a lot.
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

Re: Video encoding/decoding in Java

Postby xuggle » Fri Jun 26, 2009 1:57 am

Yes. See IMediaData.getByteBuffer(...)

- Art
xuggle
 
Posts: 3
Joined: Wed Jun 24, 2009 4:29 am

Re: Video encoding/decoding in Java

Postby xuggle » Fri Jun 26, 2009 5:01 am

More specifically:
http://build.xuggle.com/view/Stable/job ... cReference)

This returns a ByteBuffer that points to the direct native memory for the image (in YUV or other format, as determined by the IStreamCoder object). In that way, you can manipulate YUV data directly. See the other IVideoPicture API elements for how to parse the data, but we (at xuggler) use this method to manage pictures without ever converting from YUV to RGB and back.

The result is pretty *ucking fast.

- Art
xuggle
 
Posts: 3
Joined: Wed Jun 24, 2009 4:29 am

Re: Video encoding/decoding in Java

Postby sascha » Sun Jun 28, 2009 10:33 am

Ok, thanks, I'll have a shot at it...
sascha
Site Admin
 
Posts: 2792
Joined: Thu May 20, 2004 9:16 am
Location: Austria

PreviousNext

Return to Off topic

Who is online

Users browsing this forum: Google [Bot] and 1 guest

cron