Android Camera JNI NDK NDK_vendor介绍

• 前言
• 主要有哪几种interface？
• Android SDK
• • Camera API 1
  • Camera API 2
  • 小结
• Android NDK
• • NDK Interface
  • NDK Vendor Interface
  • 小结
• Camera VTS Testcase
• 总结
• Reference

前言

本篇博客是想介绍Android camera从application layer到camera service layer整个框架中，能够使用的所有接口方式。

主要有哪几种interface？

分为两大类：Android SDK，Android NDK。
Android NDK分为NDK和NDK vendor。对于camera module Android SDK可以分为camera API 1和camera API 2。最后有一种特殊的方式是用过HIDL interface来实现camera APP，给出总体架构图：

Android SDK

Camera API 1

• API 1作为最原始的一套camera API现在已经被Google启用，已经不在有功能上的升级，可以提供基础capture pictures and videos in your applications。API 1流程非常的特殊，他是通过JNI layer调用到libcameraservice，如果想了解JNI移步：链接: Camera Java Native Interface(JNI)介绍。
• 接下来通过camera open代码分析JNI调用过程

frameworks/base/core/java/android/hardware/Camera.java
public static Camera open() {int numberOfCameras = getNumberOfCameras();CameraInfo cameraInfo = new CameraInfo();for (int i = 0; i < numberOfCameras; i++) {getCameraInfo(i, cameraInfo);if (cameraInfo.facing == CameraInfo.CAMERA_FACING_BACK) {return new Camera(i);}}return null;
}

如果在camera APP中调用API 1的open function去打开默认后置camera，一开始会新建一个camera device object，直接调用camera构造函数。

frameworks/base/core/java/android/hardware/Camera.java
Camera(int cameraId) {if(cameraId >= getNumberOfCameras()){throw new RuntimeException("Unknown camera ID");}int err = cameraInit(cameraId);…initAppOps();
}

在camera 构造函数中调用cameraInit。

frameworks/base/core/java/android/hardware/Camera.java
private int cameraInit(int cameraId) {…//调用到JNI中android_hardware_Camera_native_setupreturn native_setup(new WeakReference(this), cameraId,ActivityThread.currentOpPackageName());
}

在camera Init中通过native_setup interface就会调用到JNI中android_hardware_Camera_native_setup function。

frameworks/base/core/jni/android_hardware_Camera.cpp
static jint android_hardware_Camera_native_setup(JNIEnv *env, jobject thiz,jobject weak_this, jint cameraId, jstring clientPackageName)
{const char16_t *rawClientName = reinterpret_cast(env->GetStringChars(clientPackageName, NULL));jsize rawClientNameLen = env->GetStringLength(clientPackageName);String16 clientName(rawClientName, rawClientNameLen);env->ReleaseStringChars(clientPackageName,reinterpret_cast(rawClientName));int targetSdkVersion = android_get_application_target_sdk_version();sp camera = Camera::connect(cameraId, clientName, Camera::USE_CALLING_UID,Camera::USE_CALLING_PID, targetSdkVersion);…sp context = new JNICameraContext(env, weak_this, clazz, camera);context->incStrong((void*)android_hardware_Camera_native_setup);camera->setListener(context);return NO_ERROR;
}

• android_hardware_Camera_native_setup 中connect是Camera C/S架构的客户端，调用connect函数向服务器发送连接请求。
• JNICameraContext这个类是一个监听类，用于处理底层Camera回调函数传来的数据和消息。

frameworks/av/camera/Camera.cpp
sp Camera::connect(int cameraId, const String16& clientPackageName,int clientUid, int clientPid, int targetSdkVersion)
{return CameraBaseT::connect(cameraId, clientPackageName, clientUid,clientPid, targetSdkVersion);
}

JNI调用进入C++ framework camera file connect function。

frameworks/av/camera/CameraBase.cpp
template 
sp CameraBase::connect(int cameraId,const String16& clientPackageName,int clientUid, int clientPid, int targetSdkVersion)
{ALOGV("%s: connect", __FUNCTION__);//创建一个camera对象，调用Camera和CameraBase构造函数sp c = new TCam(cameraId);sp cl = c;//获取camera service对象const sp<::android::hardware::ICameraService> cs = getCameraService();binder::Status ret;if (cs != nullptr) {TCamConnectService fnConnectService = TCamTraits::fnConnectService;ret = (cs.get()->*fnConnectService)(cl, cameraId, clientPackageName, clientUid,clientPid, targetSdkVersion, /*out*/ &c->mCamera);}…return c;
}

CameraBase中通过获得到server端（camera service）代理对象ICameraService。
然后ICameraService通过fnConnectService链接camera service。

frameworks/av/camera/CameraBase.cpp
// establish binder interface to camera service
template 
const sp<::android::hardware::ICameraService> CameraBase::getCameraService()
{Mutex::Autolock _l(gLock);if (gCameraService.get() == 0) {if (CameraUtils::isCameraServiceDisabled()) {return gCameraService;}//通过binder获取camera servicesp sm = defaultServiceManager();sp binder;do {binder = sm->getService(String16(kCameraServiceName));if (binder != 0) {break;}ALOGW("CameraService not published, waiting...");usleep(kCameraServicePollDelay);} while(true);if (gDeathNotifier == NULL) {gDeathNotifier = new DeathNotifier();}binder->linkToDeath(gDeathNotifier);gCameraService = interface_cast<::android::hardware::ICameraService>(binder);}ALOGE_IF(gCameraService == 0, "no CameraService!?");return gCameraService;
}

getCameraService function中主要是展示通过binder的方法获取到camera service代理对象。
为什么fnConnectService可以链接camera device呢？因为fnConnectService就等于connect，如下代码：

frameworks/av/camera/Camera.cpp
CameraTraits::TCamConnectService CameraTraits::fnConnectService =&::android::hardware::ICameraService::connect;

而connect本身定义为AIDL interface，就是通过binder实现跨进程调用到camera service，在camera service中进一步处理。

frameworks/av/camera/aidl/android/hardware/ICameraService.aidl
AIDL 文件中定义的connect interface，目的为跨进程调用
ICamera connect(ICameraClient client,int cameraId,String opPackageName,int clientUid, int clientPid,int targetSdkVersion);

到此camera API 1的流程就梳理完毕，JNI在其中做一个桥梁的作用，API 1通过JNI连接到C++ framework，也展示JNI的目的就是为让java field可以调用到C++ field。

Camera API 2

• 为更好的控制camera device，Google抛弃camera API 1重新设计camera API 2接口。
• Camera2 provides in-depth controls for complex use cases, but requires you to manage device-specific configurations。
• Camera API2 框架为应用提供更接近底层的相机控件，包括高效的零复制连拍/视频流以及曝光、增益、白平衡增益、颜色转换、去噪、锐化等方面的每帧控件。
• camera API 2调用流程相对简单，直接从API 2 interface通过AIDL调用到camera service，接下来也是举例openCamera流程进行分析。

frameworks/base/core/java/android/hardware/camera2/CameraManager.java
public void openCamera(@NonNull String cameraId,@NonNull final CameraDevice.StateCallback callback, @Nullable Handler handler)throws CameraAccessException {openCameraForUid(cameraId, callback, CameraDeviceImpl.checkAndWrapHandler(handler),USE_CALLING_UID);
}public void openCameraForUid(@NonNull String cameraId,@NonNull final CameraDevice.StateCallback callback, @NonNull Executor executor,int clientUid, int oomScoreOffset) throws CameraAccessException {…openCameraDeviceUserAsync(cameraId, callback, executor, clientUid, oomScoreOffset);
}

APP通过API 2 open camera操作可以调用到camera manager中的openCamera function，进而调用到openCameraForUid、openCameraDeviceUserAsync。

frameworks/base/core/java/android/hardware/camera2/CameraManager.java
private CameraDevice openCameraDeviceUserAsync(String cameraId,CameraDevice.StateCallback callback, Executor executor, final int uid,final int oomScoreOffset) throws CameraAccessException {CameraCharacteristics characteristics = getCameraCharacteristics(cameraId);CameraDevice device = null;Map physicalIdsToChars =getPhysicalIdToCharsMap(characteristics);synchronized (mLock) {ICameraDeviceUser cameraUser = null;android.hardware.camera2.impl.CameraDeviceImpl deviceImpl =new android.hardware.camera2.impl.CameraDeviceImpl(cameraId,callback,executor,characteristics,physicalIdsToChars,mContext.getApplicationInfo().targetSdkVersion,mContext);ICameraDeviceCallbacks callbacks = deviceImpl.getCallbacks();try {ICameraService cameraService = CameraManagerGlobal.get().getCameraService();if (cameraService == null) {throw new ServiceSpecificException(ICameraService.ERROR_DISCONNECTED,"Camera service is currently unavailable");}cameraUser = cameraService.connectDevice(callbacks, cameraId,mContext.getOpPackageName(),  mContext.getAttributionTag(), uid,oomScoreOffset, mContext.getApplicationInfo().targetSdkVersion);} catch (ServiceSpecificException e) {…}return device;
}

openCameraDeviceUserAsync function中通过getCameraService获取到camera service代理ICameraService，这部分操作是在class CameraManagerGlobal中实现。

frameworks/base/core/java/android/hardware/camera2/CameraManager.java
class CameraManagerGlobal {public ICameraService getCameraService() {synchronized(mLock) {connectCameraServiceLocked();if (mCameraService == null && !sCameraServiceDisabled) {Log.e(TAG, "Camera service is unavailable");}return mCameraService;}}private void connectCameraServiceLocked() {// Only reconnect if necessaryif (mCameraService != null || sCameraServiceDisabled) return;Log.i(TAG, "Connecting to camera service");IBinder cameraServiceBinder = ServiceManager.getService(CAMERA_SERVICE_BINDER_NAME);if (cameraServiceBinder == null) {// Camera service is now down, leave mCameraService as nullreturn;}try {cameraServiceBinder.linkToDeath(this, /*flags*/ 0);} catch (RemoteException e) {// Camera service is now down, leave mCameraService as nullreturn;}//直接通过AIDL 获取camera serviceICameraService cameraService = ICameraService.Stub.asInterface(cameraServiceBinder);…}
}

• global manger camera class主要作为保持和camera service的链接，并且distribute camera device callback notice。
• getCameraService调用到connectCameraServiceLocked，可以很清楚的看到通过binder获取到cameraService。
• 因为connectDevice interface本身定义为AIDL，跨进程的方式从java framework调用到camera service中。

frameworks/av/camera/aidl/android/hardware/ICameraService.aidlICameraDeviceUser connectDevice(ICameraDeviceCallbacks callbacks,String cameraId,String opPackageName,@nullable String featureId,int clientUid, int oomScoreOffset,int targetSdkVersion);

以上就讲解完通过 API 2方式从APP调用到camera service。

小结

• 虽然Android SDK中的API 1和API 2都是通过AIDL调用到camera service，但是分别是在C++ code和java code中调用，这里就体现AIDL这种接口描述性语言的优势，不需要考虑binder service和client两端的编程语言，只需要按照要求编写好AIDL interface，编译之后自动生成代码，在使用的时候按接口标准就可以实现。

Android NDK

NDK Interface

• NDK的全拼是：Native Develop Kit，Android NDK 是一套允许您使用原生代码语言(例如C和C++)实现部分应用的工具集。在开发某些类型应用时，这有助于您重复使用以这些语言编写的代码库。
• 我们来分析要是使用NDK的方式来编写camera APP进程的代码，open camera的流程是怎么样的。

frameworks/av/camera/ndk/NdkCameraManager.cpp
camera_status_t ACameraManager_openCamera(ACameraManager* mgr, const char* cameraId,ACameraDevice_StateCallbacks* callback,/*out*/ACameraDevice** device) {return mgr->openCamera(cameraId, callback, device);
}

在system分区中通过使用C++代码编写的camera APP，open camera调用的是NDK中的ACameraManager_openCamera，进一步进入camera manager openCamera function。

frameworks/av/camera/ndk/NdkCameraManager.cpp
camera_status_t
ACameraManager::openCamera(const char* cameraId,ACameraDevice_StateCallbacks* callback,/*out*/ACameraDevice** outDevice)
{sp cs = CameraManagerGlobal::getInstance().getCameraService();sp callbacks = device->getServiceCallback();sp deviceRemote;int targetSdkVersion = android_get_application_target_sdk_version();//调用CameraService::connectDevicebinder::Status serviceRet = cs->connectDevice(callbacks, String16(cameraId), String16(""), {},hardware::ICameraService::USE_CALLING_UID, /*oomScoreOffset*/0,targetSdkVersion, /*out*/&deviceRemote);...
}

openCamera function中先通过getCameraService获取到camera service代理对象（其中代码流程同上，就不分析了），再通过connectDevice AIDL interface 跨进程连接到camera service。

NDK Vendor Interface

• VNDK的全称是Vendor Native Development Kit，是Android 8.0引入的一种新技术。它表现一系列库的合集，用于让供应商开发自己的HALs。VNDK 包含在 system.img 中，并在运行时与供应商代码动态关联。
• 使用vndk的原因是自Android8.0以来，Google引入了Treble架构，希望对vendor和system分区进行解耦处理，期待实现：framwork进程不加载vendor共享库，vendor进程仅加载vendor共享库（和部分framework共享库），而framework进程和vendor进程之间通过HIDL和hwbinder来通信
• 这里也是通过open camera function，来分析在vendor分区中编写C++ 代码的camera APP 调用到camera service。

frameworks/av/camera/ndk/ndk_vendor/impl/ACameraManager.cpp
camera_status_t
ACameraManager::openCamera(const char* cameraId,ACameraDevice_StateCallbacks* callback,/*out*/ACameraDevice** outDevice) {sp rawChars;camera_status_t ret = getCameraCharacteristics(cameraId, &rawChars);Mutex::Autolock _l(mLock);if (ret != ACAMERA_OK) {ALOGE("%s: cannot get camera characteristics for camera %s. err %d",__FUNCTION__, cameraId, ret);return ACAMERA_ERROR_INVALID_PARAMETER;}ACameraDevice* device = new ACameraDevice(cameraId, callback, std::move(rawChars));sp cs = CameraManagerGlobal::getInstance().getCameraService();if (cs == nullptr) {ALOGE("%s: Cannot reach camera service!", __FUNCTION__);delete device;return ACAMERA_ERROR_CAMERA_DISCONNECTED;}sp callbacks = device->getServiceCallback();sp deviceRemote_2_0;//通过调用HidlCameraService中connectDevice，hidl方式Status status = Status::NO_ERROR;auto serviceRet = cs->connectDevice(callbacks, cameraId, [&status, &deviceRemote_2_0](auto s, auto &device) {status = s;deviceRemote_2_0 = device;});…sp deviceRemote = castResult;device->setRemoteDevice(deviceRemote);device->setDeviceMetadataQueues();*outDevice = device;return ACAMERA_OK;
}

这里直接给出的是NDK Vendor中的openCamera function，但是这个函数也是被ACameraManager_openCamera调用。
其中mgr->openCamera是调用NDK还是NDK Vendor中的interface呢？需要通过mCameraManager来确定。

ACameraManager_openCamera(mCameraManager, mCameraId, &mDeviceCb, &mDevice);
mCameraManager = ACameraManager_create();
ACameraManager* ACameraManager_create() {ATRACE_CALL();return new ACameraManager();
}

• ACameraManager_openCamera调用时会传入mCameraManager，mCameraManager是由ACameraManager_create创建出来。ACameraManager_create 创建过程直接new ACameraManager对象。

NDK ACameraManager：
struct ACameraManager {ACameraManager() :mGlobalManager(&(android::acam::CameraManagerGlobal::getInstance())) {}~ACameraManager();camera_status_t getCameraIdList(ACameraIdList** cameraIdList);static void     deleteCameraIdList(ACameraIdList* cameraIdList);camera_status_t getCameraCharacteristics(const char* cameraId, android::sp* characteristics);camera_status_t openCamera(const char* cameraId,ACameraDevice_StateCallbacks* callback,/*out*/ACameraDevice** device);private:enum {kCameraIdListNotInit = -1};android::Mutex         mLock;android::sp mGlobalManager;
};NDK Vendor ACameraManager：
struct ACameraManager {ACameraManager() :mGlobalManager(&(android::acam::CameraManagerGlobal::getInstance())) {}~ACameraManager();camera_status_t getCameraIdList(ACameraIdList** cameraIdList);static void     deleteCameraIdList(ACameraIdList* cameraIdList);camera_status_t getCameraCharacteristics(const char* cameraId, android::sp* characteristics);camera_status_t openCamera(const char* cameraId,ACameraDevice_StateCallbacks* callback,/*out*/ACameraDevice** device);camera_status_t getTagFromName(const char *cameraId, const char *name, uint32_t *tag);private:enum {kCameraIdListNotInit = -1};android::Mutex         mLock;android::sp mGlobalManager;
};

• new ACameraManager需要区分是调用NDK还是NDK Vendor创建对象，不同的对象进而实现不同的调用接口逻辑。
• 再回来看openCamera function中的connectDevice interface，其中传递的参数和NDK代码中明显不同，因为NDK Vendor调用的是HIDL 定义connectDevice interface。

frameworks/hardware/interfaces/cameraservice/service/2.0/ICameraService.hal
connectDevice(ICameraDeviceCallback callback, string cameraId)generates (Status status, ICameraDeviceUser device);

• ICameraService.hal文件中定义的HIDL connectDevice interface。
• cs->connectDevice通过binder跨进程可以调用到HidlCameraService::connectDevice中。

小结

• 以上分析了通过NDK和NDK Vendor方式，connect device连接到camera service的代码流程，区别在于NDK使用AIDL interface而NDK Vendor使用 HIDL interface，并且在libcameraservice中对接的接口也不一样。
• System分区中new ACameraManager实例调用NDK interface，Vendor 分区中new ACameraManager实例调用NDK Vendor interface。
• HidlCameraService对上连接NDK Vendor，但是最终实现功能还是依赖于CameraService。

Camera VTS Testcase

• HIDL interface提供的接口是可以直接实现camera device所有基础的功能，但是由于没有 camera service的介入对于管理camera device是非常困难的。使用这种方式构建camera APP非常的特殊，没有经过整个Android framework，可以直接对HAL进行测试。
• Camera VTS testcase 是基于HDIL interface来实现功能的测试程序，毕竟是Google提供给HAL 厂商稳定的接口，肯定需要测试保证其稳定性。
• 对于如果是camera service process没有启动的情况下，可以使用HDIL interface来写camera APP，感兴趣可以移步: Camera HIDL接口实现camera preview功能

总结

• 这篇博客主要是对Android framework接口方式进行介绍，其中包括Android SDK和Android NDK，Android SDK可以分为API 1和API 2，其中API 1通过JNI的方式调用实现。
• JNI调用的cameraBase是和NDK、NDK vendor处于同一目录下，framework中的audio和video服务层（av目录名字含义）。
• CameraBase、API 2和NDK都是通过AIDL跨进程调用到libcameraservice，而只有NDK Vendor独特的使用HIDL跨进程调用到libcameraservice。

Reference

链接: Android JNI(一)——NDK与JNI基础