Wednesday, July 30, 2014

Android Camera Orientation


Due to developing an android application for camera or it's hardware library (libcamera.so or camera.<hw_name>.so) used by mediaserver through libcameraservice some obscurities about camera orientation might happen. In order to clarify them lets consider code snippets responsible for camera orientation on different levels.


Camera HAL module

On this level a vendor of certain Android device must implement a shared library working with camera hardware and implementing Camera API (CameraHardwareInterface.h). For each specific camera it's necessary to define two parameters: orientation and facing presented by corresponding fields of camera_info structure. In orther words a code snippet setting orientation and facing might look something like this:
switch (cameraId) {
    case 0:
        cameraInfo->facing = CAMERA_FACING_BACK;
        cameraInfo->orientation = 0;
        break;
    case 1:
        cameraInfo->facing = CAMERA_FACING_FRONT;
        cameraInfo->orientation = 180;
        break;
    default:
        break;
}
Both parameters are quite obvious. The first one specifies a view point direction relatively to the device screen (front or back) and the second one specifies camera orientation in degrees.

Mediaserver

Mediaserver is an android process having several services (audioflinger, mediaplayerservice, cameraservice and audiopolicyservice). The general purpose of cameraservice is to organise communication between an android application and camera HAL module. As a default cameraservice ignores orientation field and takes care only about facing. 
CameraClient::CameraClient(const sp& cameraService,
        const sp& cameraClient,
        int cameraId, int cameraFacing, int clientPid, int servicePid):
        Client(cameraService, cameraClient,
                cameraId, cameraFacing, clientPid, servicePid)
{
    int callingPid = getCallingPid();
    LOG1("CameraClient::CameraClient E (pid %d, id %d)", callingPid, cameraId);

    mHardware = NULL;
    mMsgEnabled = 0;
    mSurface = 0;
    mPreviewWindow = 0;
    mDestructionStarted = false;

    // Callback is disabled by default
    mPreviewCallbackFlag = CAMERA_FRAME_CALLBACK_FLAG_NOOP;
    mOrientation = getOrientation(0, mCameraFacing == CAMERA_FACING_FRONT);
    mPlayShutterSound = true;
    LOG1("CameraClient::CameraClient X (pid %d, id %d)", callingPid, cameraId);
}
That's why even if you set the orientation to 180 degrees in your camera module a camera preview of android application won't be rotated without additional call setDisplayOrientation(). This method of android.hardware.Camera class calls sendCommand() where the orientation is applied passed as an argument .
status_t CameraClient::sendCommand(int32_t cmd, int32_t arg1, int32_t arg2) {
    LOG1("sendCommand (pid %d)", getCallingPid());
    int orientation;
    Mutex::Autolock lock(mLock);
    status_t result = checkPidAndHardware();
    if (result != NO_ERROR) return result;

    if (cmd == CAMERA_CMD_SET_DISPLAY_ORIENTATION) {
        // Mirror the preview if the camera is front-facing.
        orientation = getOrientation(arg1, mCameraFacing == CAMERA_FACING_FRONT);
        if (orientation == -1) return BAD_VALUE;

        if (mOrientation != orientation) {
            mOrientation = orientation;
            if (mPreviewWindow != 0) {
                ALOGI("orientation=%d, sendCommand", mOrientation);
                native_window_set_buffers_transform(mPreviewWindow.get(),
                        mOrientation);
            }
        }
        return OK;
    } 
    ...
}
If you look at getOrientation() method prototype you will see that the second parameter is named as mirror and it's initialized in all calls by the expression mCameraFacing == CAMERA_FACING_FRONT. It means that from cameraservice point of view the facing is nothing more than mirroring. Cameraservice mirrors camera preview (native window) if facing is FRONT. For 0 degrees it's done by horizontal flip:
int CameraClient::getOrientation(int degrees, bool mirror) {
    if (!mirror) {
        if (degrees == 0) return 0;
        else if (degrees == 90) return HAL_TRANSFORM_ROT_90;
        else if (degrees == 180) return HAL_TRANSFORM_ROT_180;
        else if (degrees == 270) return HAL_TRANSFORM_ROT_270;
    } else {  // Do mirror (horizontal flip)
        if (degrees == 0) {           // FLIP_H and ROT_0
            return HAL_TRANSFORM_FLIP_H;
        } else if (degrees == 90) {   // FLIP_H and ROT_90
            return HAL_TRANSFORM_FLIP_H | HAL_TRANSFORM_ROT_90;
        } else if (degrees == 180) {  // FLIP_H and ROT_180
            return HAL_TRANSFORM_FLIP_V;
        } else if (degrees == 270) {  // FLIP_H and ROT_270
            return HAL_TRANSFORM_FLIP_V | HAL_TRANSFORM_ROT_90;
        }
    }
    ALOGE("Invalid setDisplayOrientation degrees=%d", degrees);
    return -1;
}

Application level

Since cameraservice sets the orientation to 0 degrees while initialising a camera an android application have to take care about the orientation configured in camera HAL module to properly display camera preview by it self. Here there is an example how to make the camera image show in the same orientation as the display:
 public static void setCameraDisplayOrientation(Activity activity,
         int cameraId, android.hardware.Camera camera) {
     android.hardware.Camera.CameraInfo info =
             new android.hardware.Camera.CameraInfo();
     android.hardware.Camera.getCameraInfo(cameraId, info);
     int rotation = activity.getWindowManager().getDefaultDisplay()
             .getRotation();
     int degrees = 0;
     switch (rotation) {
         case Surface.ROTATION_0: degrees = 0; break;
         case Surface.ROTATION_90: degrees = 90; break;
         case Surface.ROTATION_180: degrees = 180; break;
         case Surface.ROTATION_270: degrees = 270; break;
     }

     int result;
     if (info.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
         result = (info.orientation + degrees) % 360;
         result = (360 - result) % 360;  // compensate the mirror
     } else {  // back-facing
         result = (info.orientation - degrees + 360) % 360;
     }
     camera.setDisplayOrientation(result);
 }

Friday, February 7, 2014

ffmpeg – converting video file to raw format

In this post you’ll see how to covert a specific video file format to YUV and play this raw video file. In addition the video will be resized and clipped.
Original file: ARDrone.mp4 (dimensions: 480×270, video codec: H.264)

Converting to YUV

ffmpeg -i ARDrone.mp4 -f rawvideo -vcodec rawvideo -pix_fmt yuv420p ARDrone.yuv
All flags after -i ARDrone.mp4 are intented for ouput:
  • -f rawvideo – raw output file format 
  • -vcodec rawvideo – video codec for output stream (not used for rawvideo) 
  • -pix_fmt yuv420p – pixel format of video fram


Playing YUV using mplayer

mplayer ARDrone.yuv -demuxer rawvideo -rawvideo w=480:h=270:format=i420
To play a raw video properly it’s necessary to specify frame size and pixel format since this information is not presented in the file.

Converting to YUV and resizing video

ffmpeg -i ARDrone.mp4 -f rawvideo -vcodec rawvideo -pix_fmt yuv420p -s 176x144 ARDrone_176x144.yuv
You may also use abbreviations instead of specifing a format as WxH. Fox example, -s 176×144 can be replaced by -s qcif.

Resizing raw video

If you want to resize existing raw video you must add information about your input file such as video file format, video codec, frame size and pixel format.
ffmpeg -s 480x270 -f rawvideo -vcodec rawvideo -pix_fmt yuv420p -i ARDrone.yuv -f rawvideo -vcodec rawvideo -pix_fmt yuv420p -s qcif ARDrone_qcif.yuv

Converting to YUV and clipping video

ffmpeg -i ARDrone.mp4 -f rawvideo -vcodec rawvideo -pix_fmt yuv420p -ss 5 -t 10 ARDrone_10s.yuv
This command converts the video file to raw format and extract a clip from 5th second (-ss 5) to 15th. Flag -t sets video duration.