| OpenCV 5.0.0alpha Open Source Computer Vision

Loading...
Searching...
No Matches
Using Orbbec 3D cameras (UVC)

Prev Tutorial: Using Orbbec Astra 3D cameras
Next Tutorial: Using Creative Senz3D and other Intel RealSense SDK compatible depth sensors

Original author | Jinyue Chen
Compatibility | OpenCV >= 4.10

## Introduction

This tutorial is devoted to the Orbbec 3D cameras based on UVC protocol. For the use of the older Orbbec 3D cameras which depends on OpenNI, please refer to the previous tutorial.

Unlike working with the OpenNI based Astra 3D cameras which requires OpenCV built with OpenNI2 SDK, Orbbec SDK is not required to be installed for accessing Orbbec UVC 3D cameras via OpenCV. By using `cv::VideoCapture` class, users get the stream data from 3D cameras, similar to working with USB cameras. The calibration and alignment of the depth map and color image are done internally.

## Instructions

In order to use the 3D cameras with OpenCV. You can refer to Get Started to install OpenCV.

Note since 4.11 on, Mac OS users need to compile OpenCV from source with flag `-DOBSENSOR_USE_ORBBEC_SDK=ON` in order to use the cameras:

cmake -DOBSENSOR_USE_ORBBEC_SDK=ON ..
make
sudo make install

# Code

Python

This tutorial code's is shown lines below. You can also download it from here

#!/usr/bin/env python
import numpy as np
import sys
import cv2 as cv
def main():
# Open Orbbec depth sensor
orbbec_cap = cv.VideoCapture(0, cv.CAP_OBSENSOR)
if orbbec_cap.isOpened() == False:
sys.exit("Fail to open camera.")
while True:
# Grab data from the camera
if orbbec_cap.grab():
# RGB data
ret_bgr, bgr_image = orbbec_cap.retrieve(None, cv.CAP_OBSENSOR_BGR_IMAGE)
if ret_bgr:
cv.imshow("BGR", bgr_image)
# depth data
ret_depth, depth_map = orbbec_cap.retrieve(None, cv.CAP_OBSENSOR_DEPTH_MAP)
if ret_depth:
color_depth_map = cv.normalize(depth_map, None, 0, 255, cv.NORM_MINMAX, cv.CV_8UC1)
color_depth_map = cv.applyColorMap(color_depth_map, cv.COLORMAP_JET)
cv.imshow("DEPTH", color_depth_map)
else:
print("Fail to grab data from the camera.")
if cv.pollKey() >= 0:
break
orbbec_cap.release()
if __name__ == '__main__':
main()
cv::VideoCapture
Class for video capturing from video files, image sequences or cameras.
Definition videoio.hpp:727
cv::normalize
void normalize(InputArray src, InputOutputArray dst, double alpha=1, double beta=0, int norm_type=NORM_L2, int dtype=-1, InputArray mask=noArray())
Normalizes the norm or value range of an array.
cv::imshow
void imshow(const String &winname, InputArray mat)
Displays an image in the specified window.
cv::pollKey
int pollKey()
Polls for a pressed key.
cv::applyColorMap
void applyColorMap(InputArray src, OutputArray dst, int colormap)
Applies a GNU Octave/MATLAB equivalent colormap on a given image.
main
int main(int argc, char *argv[])
Definition highgui_qt.cpp:3
C++

This tutorial code's is shown lines below. You can also download it from here

#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <iostream>
using namespace cv;
int main()
{
VideoCapture obsensorCapture(0, CAP_OBSENSOR);
if(!obsensorCapture.isOpened()){
std::cerr << "Failed to open obsensor capture! Index out of range or no response from device";
return -1;
}
double fx = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_FX);
double fy = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_FY);
double cx = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_CX);
double cy = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_CY);
std::cout << "obsensor camera intrinsic params: fx=" << fx << ", fy=" << fy << ", cx=" << cx << ", cy=" << cy << std::endl;
Mat image;
Mat depthMap;
Mat adjDepthMap;
// Minimum depth value
const double minVal = 300;
// Maximum depth value
const double maxVal = 5000;
while (true)
{
// Grab depth map like this:
// obsensorCapture >> depthMap;
// Another way to grab depth map (and bgr image).
if (obsensorCapture.grab())
{
if (obsensorCapture.retrieve(image, CAP_OBSENSOR_BGR_IMAGE))
{
imshow("RGB", image);
}
if (obsensorCapture.retrieve(depthMap, CAP_OBSENSOR_DEPTH_MAP))
{
depthMap.convertTo(adjDepthMap, CV_8U, 255.0 / (maxVal - minVal), -minVal * 255.0 / (maxVal - minVal));
applyColorMap(adjDepthMap, adjDepthMap, COLORMAP_JET);
imshow("DEPTH", adjDepthMap);
}
// depth map overlay on bgr image
static const float alpha = 0.6f;
if (!image.empty() && !depthMap.empty())
{
depthMap.convertTo(adjDepthMap, CV_8U, 255.0 / (maxVal - minVal), -minVal * 255.0 / (maxVal - minVal));
cv::resize(adjDepthMap, adjDepthMap, cv::Size(image.cols, image.rows));
for (int i = 0; i < image.rows; i++)
{
for (int j = 0; j < image.cols; j++)
{
cv::Vec3b& outRgb = image.at<cv::Vec3b>(i, j);
uint8_t depthValue = 255 - adjDepthMap.at<uint8_t>(i, j);
if (depthValue != 0 && depthValue != 255)
{
outRgb[0] = (uint8_t)(outRgb[0] * (1.0f - alpha) + depthValue * alpha);
outRgb[1] = (uint8_t)(outRgb[1] * (1.0f - alpha) + depthValue * alpha);
outRgb[2] = (uint8_t)(outRgb[2] * (1.0f - alpha) + depthValue * alpha);
}
}
}
imshow("DepthToColor", image);
}
image.release();
depthMap.release();
}
if (pollKey() >= 0)
break;
}
return 0;
}
cv::Mat
n-dimensional dense array class
Definition mat.hpp:951
cv::Mat::at
_Tp & at(int i0=0)
Returns a reference to the specified array element.
cv::Mat::cols
int cols
Definition mat.hpp:2425
cv::Mat::empty
bool empty() const
Returns true if the array has no elements.
cv::Mat::rows
int rows
the number of rows and columns or (-1, -1) when the matrix has more than 2 dimensions
Definition mat.hpp:2425
cv::Mat::convertTo
void convertTo(OutputArray m, int rtype, double alpha=1, double beta=0) const
Converts an array to another data type with optional scaling.
cv::Mat::release
void release()
Decrements the reference counter and deallocates the matrix if needed.
cv::Size_
Template class for specifying the size of an image or rectangle.
Definition types.hpp:338
cv::Vec
Template class for short numerical vectors, a partial case of Matx.
Definition matx.hpp:379
CV_8U
#define CV_8U
Definition interface.h:76
cv::resize
void resize(InputArray src, OutputArray dst, Size dsize, double fx=0, double fy=0, int interpolation=INTER_LINEAR)
Resizes an image.
highgui.hpp
imgproc.hpp
cv
Definition core.hpp:107
videoio.hpp

## Code Explanation

### Python

- Open Orbbec Depth Sensor: Using `cv.VideoCapture(0, cv.CAP_OBSENSOR)` to attempt to open the first Orbbec depth sensor device. If the camera fails to open, the program will exit and display an error message.
- Loop to Grab and Process Data: In an infinite loop, the code continuously grabs data from the camera. The `orbbec_cap.grab()` method attempts to grab a frame.
- Process BGR Image: Using `orbbec_cap.retrieve(None, cv.CAP_OBSENSOR_BGR_IMAGE)` to retrieve the BGR image data. If successfully retrieved, the BGR image is displayed in a window using `cv.imshow("BGR", bgr_image)`.
- Process Depth Image: Using `orbbec_cap.retrieve(None, cv.CAP_OBSENSOR_DEPTH_MAP)` to retrieve the depth image data. If successfully retrieved, the depth image is first normalized to a range of 0 to 255, then a false color image is applied, and the result is displayed in a window using `cv.imshow("DEPTH", color_depth_map)`.
- Keyboard Interrupt: Using `cv.pollKey()` to detect keyboard events. If a key is pressed, the loop breaks and the program ends.
- Release Resources: After exiting the loop, the camera resources are released using `orbbec_cap.release()`.

### C++

- Open Orbbec Depth Sensor: Using `VideoCapture obsensorCapture(0, CAP_OBSENSOR)` to attempt to open the first Orbbec depth sensor device. If the camera fails to open, an error message is displayed, and the program exits.
- Retrieve Camera Intrinsic Parameters: Using `obsensorCapture.get()` to retrieve the intrinsic parameters of the camera, including focal lengths (`fx`, `fy`) and principal points (`cx`, `cy`).
- Loop to Grab and Process Data: In an infinite loop, the code continuously grabs data from the camera. The `obsensorCapture.grab()` method attempts to grab a frame.
- Process BGR Image: Using `obsensorCapture.retrieve(image, CAP_OBSENSOR_BGR_IMAGE)` to retrieve the BGR image data. If successfully retrieved, the BGR image is displayed in a window using `imshow("BGR", image)`.
- Process Depth Image: Using `obsensorCapture.retrieve(depthMap, CAP_OBSENSOR_DEPTH_MAP)` to retrieve the depth image data. If successfully retrieved, the depth image is normalized and a false color image is applied, then the result is displayed in a window using `imshow("DEPTH", adjDepthMap)`. The retrieved depth values are in millimeters and are truncated to a range between 300 and 5000 (millimeter). This fixed range can be interpreted as a truncation based on the depth camera's depth range, removing invalid pixels on the depth map.
- Overlay Depth Map on BGR Image: Convert the depth map to an 8-bit image, resize it to match the BGR image size, and overlay it on the BGR image with a specified transparency (`alpha`). The overlaid image is displayed in a window using `imshow("DepthToColor", image)`.
- Keyboard Interrupt: Using `pollKey()` to detect keyboard events. If a key is pressed, the loop breaks and the program ends.
- Release Resources: After exiting the loop, the camera resources are released.

## Results

### Python

BGR And DEPTH frame

### C++

BGR And DEPTH And DepthToColor frame

## Note

Mac users need `sudo` privileges to execute the code.