Expert Linux developer olegk0 has made a framebuffer driver for the RK30 HDMI with Mali acceleration (rk30fb_drv.so). You can just download and use his .so to have 3D acceleration, or you can follow these instructions to compile it yourself and help debug it.
Among other things, like 2D/3D, it allows playing fullscreen videos with far less CPU usage, leaving the processor just for what your app requires to decode the videos. With "app" I mean Parole, VLC or, my strong recommendation, the excellent commandline utility: mplayer (which immediately takes advantage of the HW acceleration).
The downside is that it still has some bugs, like turning everything in the screen black (except the played video). Just move the mouse or a window around and everything gets redrawn again.
Another bug is that the video output becomes mangled when the video window touches the left or right sides of the screen.
In any case I love to be able to watch movies/videos on this stick with Linux, so what I do is open a terminal and type:
mplayer whatever-video.aviAnd screen turns black except the video, I move the mouse a bit around (which causes the black parts to be redrawn) to find the corners of the mplayer video window and make it big enough to cover most of the screen, just without touching either left or right sides of the screen.
In any case, if you would like to give us a helping hand debugging the XVideo implementation (my suspect is ./xf86-video-fbdev/src/video.c) just follow these steps to tinker with the code and compile the driver to see the changes:
git clone git://github.com/olegk0/xf86-video-fbdev.git
sudo apt-get install xserver-xorg-dev
We must then fix a dangling symlinked folder at "./xf86-video-fbdev/src/ump" pointing to "../../../../../r3p1/ump/include/ump"
So go to http://malideveloper.arm.com/develop-for-mali/drivers/open-source-mali-gpus-ump-user-space-drivers-source-code-2/ and download the same revision (r3p1, until we get a first working driver; then we can try with a newer version), that is: Linux Kernel Device Driver r3p1-01rel1
At the time of writing this is located at: http://malideveloper.arm.com/downloads/DX910-SW-99006-r3p1-01rel1.tgz
Copy or symlink the uncompressed folder "./DX910-SW-99006-r3p1-01rel1/driver/src/ump/include/ump" to "./xf86-video-fbdev/src/ump"
And now just do:
./configure --prefix=/usr
make
sudo make install
There is a lot of debug information printed on /var/log/syslog every time you play a video with mplayer, which should help in the debugging.