| OpenCV 5.0.0alpha Open Source Computer Vision

Loading...
Searching...
No Matches
Interactive Visual Debugging of Computer Vision applications

What is the most common way to debug computer vision applications? Usually the answer is temporary, hacked together, custom code that must be removed from the code for release compilation.

In this tutorial we will show how to use the visual debugging features of the cvv module (opencv2/cvv.hpp) instead.

# Goals

In this tutorial you will learn how to:

- Add cvv debug calls to your application
- Use the visual debug GUI
- Enable and disable the visual debug features during compilation (with zero runtime overhead when disabled)

# Code

The example code

- captures images (videoio), e.g. from a webcam,
- applies some filters to each image (imgproc),
- detects image features and matches them to the previous image (features).

If the program is compiled without visual debugging (see CMakeLists.txt below) the only result is some information printed to the command line. We want to demonstrate how much debugging or development functionality is added by just a few lines of cvv commands.

1// system includes
2#include <iostream>
3
4// library includes
5#include <opencv2/imgproc.hpp>
6#include <opencv2/features.hpp>
7#include <opencv2/videoio.hpp>
8
9#define CVVISUAL_DEBUGMODE
10#include <opencv2/cvv/debug_mode.hpp>
11#include <opencv2/cvv/show_image.hpp>
12#include <opencv2/cvv/filter.hpp>
13#include <opencv2/cvv/dmatch.hpp>
14#include <opencv2/cvv/final_show.hpp>
15
16 using namespace std;
17 using namespace cv;
18
19 template<class T> std::string toString(const T& p_arg)
20{
21 std::stringstream ss;
22
23 ss << p_arg;
24
25 return ss.str();
26}
27
28
29
30
31 int
32 main(int argc, char** argv)
33{
34 // parser keys
35 const char *keys =
36 "{ help h usage ? | | show this message }"
37 "{ width W | 0| camera resolution width. leave at 0 to use defaults }"
38 "{ height H | 0| camera resolution height. leave at 0 to use defaults }";
39
40 CommandLineParser parser(argc, argv, keys);
41 if (parser.has("help")) {
42 parser.printMessage();
43 return 0;
44 }
45 int res_w = parser.get<int>("width");
46 int res_h = parser.get<int>("height");
47
48 // setup video capture
49 cv::VideoCapture capture(0);
50 if (!capture.isOpened()) {
51 std::cout << "Could not open VideoCapture" << std::endl;
52 return 1;
53 }
54
55 if (res_w>0 && res_h>0) {
56 printf("Setting resolution to %dx%d\n", res_w, res_h);
57 capture.set(cv::CAP_PROP_FRAME_WIDTH, res_w);
58 capture.set(cv::CAP_PROP_FRAME_HEIGHT, res_h);
59 }
60
61
62 cv::Mat prevImgGray;
63 std::vector<cv::KeyPoint> prevKeypoints;
64 cv::Mat prevDescriptors;
65
66 int maxFeatureCount = 500;
67 Ptr<ORB> detector = ORB::create(maxFeatureCount);
68
69 cv::BFMatcher matcher(cv::NORM_HAMMING);
70
71 for (int imgId = 0; imgId < 10; imgId++) {
72 // capture a frame
73 cv::Mat imgRead;
74 capture >> imgRead;
75 printf("%d: image captured\n", imgId);
76
77 std::string imgIdString{"imgRead"};
78 imgIdString += toString(imgId);
79 cvv::showImage(imgRead, CVVISUAL_LOCATION, imgIdString.c_str());
80
81 // convert to grayscale
82 cv::Mat imgGray;
83 cv::cvtColor(imgRead, imgGray, COLOR_BGR2GRAY);
84 cvv::debugFilter(imgRead, imgGray, CVVISUAL_LOCATION, "to gray");
85
86 // detect ORB features
87 std::vector<cv::KeyPoint> keypoints;
88 cv::Mat descriptors;
89 detector->detectAndCompute(imgGray, cv::noArray(), keypoints, descriptors);
90 printf("%d: detected %zd keypoints\n", imgId, keypoints.size());
91
92 // match them to previous image (if available)
93 if (!prevImgGray.empty()) {
94 std::vector<cv::DMatch> matches;
95 matcher.match(prevDescriptors, descriptors, matches);
96 printf("%d: all matches size=%zd\n", imgId, matches.size());
97 std::string allMatchIdString{"all matches "};
98 allMatchIdString += toString(imgId-1) + "<->" + toString(imgId);
99 cvv::debugDMatch(prevImgGray, prevKeypoints, imgGray, keypoints, matches, CVVISUAL_LOCATION, allMatchIdString.c_str());
100
101 // remove worst (as defined by match distance) bestRatio quantile
102 double bestRatio = 0.8;
103 std::sort(matches.begin(), matches.end());
104 matches.resize(int(bestRatio * matches.size()));
105 printf("%d: best matches size=%zd\n", imgId, matches.size());
106 std::string bestMatchIdString{"best " + toString(bestRatio) + " matches "};
107 bestMatchIdString += toString(imgId-1) + "<->" + toString(imgId);
108 cvv::debugDMatch(prevImgGray, prevKeypoints, imgGray, keypoints, matches, CVVISUAL_LOCATION, bestMatchIdString.c_str());
109 }
110
111 prevImgGray = imgGray;
112 prevKeypoints = keypoints;
113 prevDescriptors = descriptors;
114 }
115
116 cvv::finalShow();
117
118 return 0;
119}
CVVISUAL_LOCATION
#define CVVISUAL_LOCATION
Creates an instance of CallMetaData with the location of the macro as value.
Definition call_meta_data.hpp:65
cv::BFMatcher
Brute-force descriptor matcher.
Definition features.hpp:995
cv::CommandLineParser
Designed for command line parsing.
Definition utility.hpp:890
cv::Mat
n-dimensional dense array class
Definition mat.hpp:951
cv::Mat::empty
bool empty() const
Returns true if the array has no elements.
cv::VideoCapture
Class for video capturing from video files, image sequences or cameras.
Definition videoio.hpp:727
debug_mode.hpp
dmatch.hpp
features.hpp
filter.hpp
final_show.hpp
cv::NORM_HAMMING
@ NORM_HAMMING
Definition base.hpp:133
cv::Ptr
std::shared_ptr< _Tp > Ptr
Definition cvstd_wrapper.hpp:23
cv::noArray
InputOutputArray noArray()
Returns an empty InputArray or OutputArray.
cvv::finalShow
void finalShow()
Passes the control to the debug-window for a last time.
Definition final_show.hpp:23
cvv::debugDMatch
static void debugDMatch(cv::InputArray img1, std::vector< cv::KeyPoint > keypoints1, cv::InputArray img2, std::vector< cv::KeyPoint > keypoints2, std::vector< cv::DMatch > matches, const impl::CallMetaData &data, const char *description=nullptr, const char *view=nullptr, bool useTrainDescriptor=true)
Add a filled in DMatch <dmatch> to debug GUI.
Definition dmatch.hpp:49
cvv::showImage
static void showImage(cv::InputArray img, impl::CallMetaData metaData=impl::CallMetaData(), const char *description=nullptr, const char *view=nullptr)
Add a single image to debug GUI (similar to imshow <>).
Definition show_image.hpp:38
cvv::debugFilter
static void debugFilter(cv::InputArray original, cv::InputArray result, impl::CallMetaData metaData=impl::CallMetaData(), const char *description=nullptr, const char *view=nullptr)
Use the debug-framework to compare two images (from which the second is intended to be the result of ...
Definition filter.hpp:36
cv::cvtColor
void cvtColor(InputArray src, OutputArray dst, int code, int dstCn=0, AlgorithmHint hint=cv::ALGO_HINT_DEFAULT)
Converts an image from one color space to another.
cv::CAP_PROP_FRAME_WIDTH
@ CAP_PROP_FRAME_WIDTH
Width of the frames in the video stream.
Definition videoio.hpp:138
cv::CAP_PROP_FRAME_HEIGHT
@ CAP_PROP_FRAME_HEIGHT
Height of the frames in the video stream.
Definition videoio.hpp:139
main
int main(int argc, char *argv[])
Definition highgui_qt.cpp:3
imgproc.hpp
cv
Definition core.hpp:107
std
STL namespace.
show_image.hpp
videoio.hpp
cmake_minimum_required(VERSION 2.8)
project(cvvisual_test)
SET(CMAKE_PREFIX_PATH ~/software/opencv/install)
SET(CMAKE_CXX_COMPILER "g++-4.8")
SET(CMAKE_CXX_FLAGS "-std=c++11 -O2 -pthread -Wall -Werror")
# (un)set: cmake -DCVV_DEBUG_MODE=OFF ..
OPTION(CVV_DEBUG_MODE "cvvisual-debug-mode" ON)
if(CVV_DEBUG_MODE MATCHES ON)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DCVVISUAL_DEBUGMODE")
endif()
FIND_PACKAGE(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})
add_executable(cvvt main.cpp)
target_link_libraries(cvvt
opencv_core opencv_videoio opencv_imgproc opencv_features
opencv_cvv
)

# Explanation

1. We compile the program either using the above CmakeLists.txt with Option CVV_DEBUG_MODE=ON (cmake -DCVV_DEBUG_MODE=ON) or by adding the corresponding define CVVISUAL_DEBUGMODE to our compiler (e.g. g++ -DCVVISUAL_DEBUGMODE).
2. The first cvv call simply shows the image (similar to imshow) with the imgIdString as comment.
   cvv::showImage(imgRead, CVVISUAL_LOCATION, imgIdString.c_str());
   The image is added to the overview tab in the visual debug GUI and the cvv call blocks.

image

The image can then be selected and viewed

image

Whenever you want to continue in the code, i.e. unblock the cvv call, you can either continue until the next cvv call (Step), continue until the last cvv call (*>>*) or run the application until it exists (Close).

We decide to press the green Step button.

1. The next cvv calls are used to debug all kinds of filter operations, i.e. operations that take a picture as input and return a picture as output.
   cvv::debugFilter(imgRead, imgGray, CVVISUAL_LOCATION, "to gray");
   As with every cvv call, you first end up in the overview.

image

We decide not to care about the conversion to gray scale and press Step.

cvv::debugFilter(imgGray, imgGraySmooth, CVVISUAL_LOCATION, "smoothed");

If you open the filter call, you will end up in the so called "DefaultFilterView". Both images are shown next to each other and you can (synchronized) zoom into them.

image

When you go to very high zoom levels, each pixel is annotated with its numeric values.

image

We press Step twice and have a look at the dilated image.

cvv::debugFilter(imgEdges, imgEdgesDilated, CVVISUAL_LOCATION, "dilated edges");

The DefaultFilterView showing both images

image

Now we use the View selector in the top right and select the "DualFilterView". We select "Changed Pixels" as filter and apply it (middle image).

image

After we had a close look at these images, perhaps using different views, filters or other GUI features, we decide to let the program run through. Therefore we press the yellow *>>* button.

The program will block at

cvv::finalShow();

and display the overview with everything that was passed to cvv in the meantime.

image

1. The cvv debugDMatch call is used in a situation where there are two images each with a set of descriptors that are matched to each other.

   We pass both images, both sets of keypoints and their matching to the visual debug module.

   cvv::debugDMatch(prevImgGray, prevKeypoints, imgGray, keypoints, matches, CVVISUAL_LOCATION, allMatchIdString.c_str());

   Since we want to have a look at matches, we use the filter capabilities (*#type match*) in the overview to only show match calls.

image

We want to have a closer look at one of them, e.g. to tune our parameters that use the matching. The view has various settings how to display keypoints and matches. Furthermore, there is a mouseover tooltip.

image

We see (visual debugging!) that there are many bad matches. We decide that only 70% of the matches should be shown - those 70% with the lowest match distance.

image

Having successfully reduced the visual distraction, we want to see more clearly what changed between the two images. We select the "TranslationMatchView" that shows to where the keypoint was matched in a different way.

image

It is easy to see that the cup was moved to the left during the two images.

Although, cvv is all about interactively seeing the computer vision bugs, this is complemented by a "RawView" that allows to have a look at the underlying numeric data.

image

1. There are many more useful features contained in the cvv GUI. For instance, one can group the overview tab.

image

# Result

- By adding a view expressive lines to our computer vision program we can interactively debug it through different visualizations.
- Once we are done developing/debugging we do not have to remove those lines. We simply disable cvv debugging (cmake -DCVV_DEBUG_MODE=OFF or g++ without -DCVVISUAL_DEBUGMODE) and our programs runs without any debug overhead.

Enjoy computer vision!