| OpenCV 5.0.0alpha Open Source Computer Vision

Loading...
Searching...
No Matches
Back Projection

Prev Tutorial: Histogram Comparison
Next Tutorial: Template Matching

Original author | Ana Huamán
Compatibility | OpenCV >= 3.0

# Goal

In this tutorial you will learn:

- What is Back Projection and why it is useful
- How to use the OpenCV function cv::calcBackProject to calculate Back Projection
- How to mix different channels of an image by using the OpenCV function cv::mixChannels

# Theory

## What is Back Projection?

- Back Projection is a way of recording how well the pixels of a given image fit the distribution of pixels in a histogram model.
- To make it simpler: For Back Projection, you calculate the histogram model of a feature and then use it to find this feature in an image.
- Application example: If you have a histogram of flesh color (say, a Hue-Saturation histogram ), then you can use it to find flesh color areas in an image:

## How does it work?

- We explain this by using the skin example:
- Let's say you have gotten a skin histogram (Hue-Saturation) based on the image below. The histogram besides is going to be our model histogram (which we know represents a sample of skin tonality). You applied some mask to capture only the histogram of the skin area:

- Now, let's imagine that you get another hand image (Test Image) like the one below: (with its respective histogram):

- What we want to do is to use our model histogram (that we know represents a skin tonality) to detect skin areas in our Test Image. Here are the steps
  1. In each pixel of our Test Image (i.e. \(p(i,j)\) ), collect the data and find the correspondent bin location for that pixel (i.e. \(( h_{i,j}, s_{i,j} )\) ).
  2. Lookup the model histogram in the correspondent bin - \(( h_{i,j}, s_{i,j} )\) - and read the bin value.
  3. Store this bin value in a new image (BackProjection). Also, you may consider to normalize the model histogram first, so the output for the Test Image can be visible for you.
  4. Applying the steps above, we get the following BackProjection image for our Test Image:

1. In terms of statistics, the values stored in BackProjection represent the probability that a pixel in Test Image belongs to a skin area, based on the model histogram that we use. For instance in our Test image, the brighter areas are more probable to be skin area (as they actually are), whereas the darker areas have less probability (notice that these "dark" areas belong to surfaces that have some shadow on it, which in turns affects the detection).

# Code

- What does this program do?
  - Loads an image
  - Convert the original to HSV format and separate only Hue channel to be used for the Histogram (using the OpenCV function cv::mixChannels )
  - Let the user to enter the number of bins to be used in the calculation of the histogram.
  - Calculate the histogram (and update it if the bins change) and the backprojection of the same image.
  - Display the backprojection and the histogram in windows.

C++

- Downloadable code:
  - Click here for the basic version (explained in this tutorial).
  - For stuff slightly fancier (using H-S histograms and floodFill to define a mask for the skin area) you can check the improved demo
  - ...or you can always check out the classical camshiftdemo in samples.
- Code at glance:
  #include "opencv2/imgproc.hpp"
  #include "opencv2/imgcodecs.hpp"
  #include "opencv2/highgui.hpp"
  #include <iostream>
  using namespace cv;
  using namespace std;
  Mat hue;
  int bins = 25;
  void Hist_and_Backproj(int, void* );
  int main( int argc, char* argv[] )
  {
  CommandLineParser parser( argc, argv, "{@input |Back_Projection_Theory0.jpg| input image}" );
  samples::addSamplesDataSearchSubDirectory("doc/tutorials/imgproc/histograms/back_projection/images");
  Mat src = imread(samples::findFile(parser.get<String>( "@input" )) );
  if( src.empty() )
  {
  cout << "Could not open or find the image!\n" << endl;
  cout << "Usage: " << argv[0] << " <Input image>" << endl;
  return -1;
  }
  Mat hsv;
  cvtColor( src, hsv, COLOR_BGR2HSV );
  hue.create(hsv.size(), hsv.depth());
  int ch[] = { 0, 0 };
  mixChannels( &hsv, 1, &hue, 1, ch, 1 );
  const char* window_image = "Source image";
  namedWindow( window_image );
  createTrackbar("* Hue bins: ", window_image, &bins, 180, Hist_and_Backproj );
  Hist_and_Backproj(0, 0);
  imshow( window_image, src );
  // Wait until user exits the program
  waitKey();
  return 0;
  }
  void Hist_and_Backproj(int, void* )
  {
  int histSize = MAX( bins, 2 );
  float hue_range[] = { 0, 180 };
  const float* ranges[] = { hue_range };
  Mat hist;
  calcHist( &hue, 1, 0, Mat(), hist, 1, &histSize, ranges, true, false );
  normalize( hist, hist, 0, 255, NORM_MINMAX, -1, Mat() );
  Mat backproj;
  calcBackProject( &hue, 1, 0, hist, backproj, ranges, 1, true );
  imshow( "BackProj", backproj );
  int w = 400, h = 400;
  int bin_w = cvRound( (double) w / histSize );
  Mat histImg = Mat::zeros( h, w, CV_8UC3 );
  for (int i = 0; i < bins; i++)
  {
  rectangle( histImg, Point( i*bin_w, h ), Point( (i+1)*bin_w, h - cvRound( hist.at<float>(i)*h/255.0 ) ),
  Scalar( 0, 0, 255 ), FILLED );
  }
  imshow( "Histogram", histImg );
  }
  cv::CommandLineParser
  Designed for command line parsing.
  Definition utility.hpp:890
  cv::Mat
  n-dimensional dense array class
  Definition mat.hpp:951
  cv::Mat::size
  MatSize size
  Definition mat.hpp:2448
  cv::Mat::create
  void create(int rows, int cols, int type)
  Allocates new array data if needed.
  cv::Mat::depth
  int depth() const
  Returns the depth of a matrix element.
  cv::Mat::at
  _Tp & at(int i0=0)
  Returns a reference to the specified array element.
  cv::Mat::empty
  bool empty() const
  Returns true if the array has no elements.
  cv::Point_< int >
  cv::Scalar_< double >
  cv::normalize
  void normalize(InputArray src, InputOutputArray dst, double alpha=1, double beta=0, int norm_type=NORM_L2, int dtype=-1, InputArray mask=noArray())
  Normalizes the norm or value range of an array.
  cv::String
  std::string String
  Definition cvstd.hpp:151
  CV_8UC3
  #define CV_8UC3
  Definition interface.h:101
  cvRound
  int cvRound(double value)
  Rounds floating-point number to the nearest integer.
  Definition fast_math.hpp:200
  MAX
  #define MAX(a, b)
  Definition cvdef.h:531
  cv::imshow
  void imshow(const String &winname, InputArray mat)
  Displays an image in the specified window.
  cv::rectangle
  void rectangle(InputOutputArray img, Point pt1, Point pt2, const Scalar &color, int thickness=1, int lineType=LINE_8, int shift=0)
  Draws a simple, thick, or filled up-right rectangle.
  cv::calcBackProject
  void calcBackProject(const Mat *images, int nimages, const int *channels, InputArray hist, OutputArray backProject, const float **ranges, double scale=1, bool uniform=true)
  Calculates the back projection of a histogram.
  cv::calcHist
  void calcHist(const Mat *images, int nimages, const int *channels, InputArray mask, OutputArray hist, int dims, const int *histSize, const float **ranges, bool uniform=true, bool accumulate=false)
  Calculates a histogram of a set of arrays.
  highgui.hpp
  main
  int main(int argc, char *argv[])
  Definition highgui_qt.cpp:3
  imgcodecs.hpp
  imgproc.hpp
  cv
  Definition core.hpp:107
  std
  STL namespace.

Java

- Downloadable code:
  - Click here for the basic version (explained in this tutorial).
  - For stuff slightly fancier (using H-S histograms and floodFill to define a mask for the skin area) you can check the improved demo
  - ...or you can always check out the classical camshiftdemo in samples.
- Code at glance:
  import java.awt.BorderLayout;
  import java.awt.Container;
  import java.awt.Image;
  import java.util.Arrays;
  import java.util.List;
  import javax.swing.BoxLayout;
  import javax.swing.ImageIcon;
  import javax.swing.JFrame;
  import javax.swing.JLabel;
  import javax.swing.JPanel;
  import javax.swing.JSlider;
  import javax.swing.event.ChangeEvent;
  import javax.swing.event.ChangeListener;
  import org.opencv.core.Core;
  import org.opencv.core.CvType;
  import org.opencv.core.Mat;
  import org.opencv.core.MatOfFloat;
  import org.opencv.core.MatOfInt;
  import org.opencv.core.Point;
  import org.opencv.core.Scalar;
  import org.opencv.highgui.HighGui;
  import org.opencv.imgcodecs.Imgcodecs;
  import org.opencv.imgproc.Imgproc;
  class CalcBackProject1 {
  private Mat hue;
  private Mat histImg = new Mat();
  private JFrame frame;
  private JLabel imgLabel;
  private JLabel backprojLabel;
  private JLabel histImgLabel;
  private static final int MAX_SLIDER = 180;
  private int bins = 25;
  public CalcBackProject1(String[] args) {
  if (args.length != 1) {
  System.err.println("You must supply one argument that corresponds to the path to the image.");
  System.exit(0);
  }
  Mat src = Imgcodecs.imread(args[0]);
  if (src.empty()) {
  System.err.println("Empty image: " + args[0]);
  System.exit(0);
  }
  Mat hsv = new Mat();
  Imgproc.cvtColor(src, hsv, Imgproc.COLOR_BGR2HSV);
  hue = new Mat(hsv.size(), hsv.depth());
  Core.mixChannels(Arrays.asList(hsv), Arrays.asList(hue), new MatOfInt(0, 0));
  // Create and set up the window.
  frame = new JFrame("Back Projection 1 demo");
  frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
  // Set up the content pane.
  Image img = HighGui.toBufferedImage(src);
  addComponentsToPane(frame.getContentPane(), img);
  // Use the content pane's default BorderLayout. No need for
  // setLayout(new BorderLayout());
  // Display the window.
  frame.pack();
  frame.setVisible(true);
  }
  private void addComponentsToPane(Container pane, Image img) {
  if (!(pane.getLayout() instanceof BorderLayout)) {
  pane.add(new JLabel("Container doesn't use BorderLayout!"));
  return;
  }
  JPanel sliderPanel = new JPanel();
  sliderPanel.setLayout(new BoxLayout(sliderPanel, BoxLayout.PAGE_AXIS));
  sliderPanel.add(new JLabel("* Hue bins: "));
  JSlider slider = new JSlider(0, MAX_SLIDER, bins);
  slider.setMajorTickSpacing(25);
  slider.setMinorTickSpacing(5);
  slider.setPaintTicks(true);
  slider.setPaintLabels(true);
  slider.addChangeListener(new ChangeListener() {
  @Override
  public void stateChanged(ChangeEvent e) {
  JSlider source = (JSlider) e.getSource();
  bins = source.getValue();
  update();
  }
  });
  sliderPanel.add(slider);
  pane.add(sliderPanel, BorderLayout.PAGE_START);
  JPanel imgPanel = new JPanel();
  imgLabel = new JLabel(new ImageIcon(img));
  imgPanel.add(imgLabel);
  backprojLabel = new JLabel();
  imgPanel.add(backprojLabel);
  histImgLabel = new JLabel();
  imgPanel.add(histImgLabel);
  pane.add(imgPanel, BorderLayout.CENTER);
  }
  private void update() {
  int histSize = Math.max(bins, 2);
  float[] hueRange = {0, 180};
  Mat hist = new Mat();
  List<Mat> hueList = Arrays.asList(hue);
  Imgproc.calcHist(hueList, new MatOfInt(0), new Mat(), hist, new MatOfInt(histSize), new MatOfFloat(hueRange), false);
  Core.normalize(hist, hist, 0, 255, Core.NORM_MINMAX);
  Mat backproj = new Mat();
  Imgproc.calcBackProject(hueList, new MatOfInt(0), hist, backproj, new MatOfFloat(hueRange), 1);
  Image backprojImg = HighGui.toBufferedImage(backproj);
  backprojLabel.setIcon(new ImageIcon(backprojImg));
  int w = 400, h = 400;
  int binW = (int) Math.round((double) w / histSize);
  histImg = Mat.zeros(h, w, CvType.CV_8UC3);
  float[] histData = new float[(int) (hist.total() * hist.channels())];
  hist.get(0, 0, histData);
  for (int i = 0; i < bins; i++) {
  Imgproc.rectangle(histImg, new Point(i * binW, h),
  new Point((i + 1) * binW, h - Math.round(histData[i] * h / 255.0)), new Scalar(0, 0, 255), Imgproc.FILLED);
  }
  Image histImage = HighGui.toBufferedImage(histImg);
  histImgLabel.setIcon(new ImageIcon(histImage));
  frame.repaint();
  frame.pack();
  }
  }
  public class CalcBackProjectDemo1 {
  public static void main(String[] args) {
  // Load the native OpenCV library
  System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
  // Schedule a job for the event dispatch thread:
  // creating and showing this application's GUI.
  javax.swing.SwingUtilities.invokeLater(new Runnable() {
  @Override
  public void run() {
  new CalcBackProject1(args);
  }
  });
  }
  }
  cv::Point
  Point2i Point
  Definition types.hpp:209
  cv::Scalar
  Scalar_< double > Scalar
  Definition types.hpp:712

Python

- Downloadable code:
  - Click here for the basic version (explained in this tutorial).
  - For stuff slightly fancier (using H-S histograms and floodFill to define a mask for the skin area) you can check the improved demo
  - ...or you can always check out the classical camshiftdemo in samples.
- Code at glance:
  from __future__ import print_function
  from __future__ import division
  import cv2 as cv
  import numpy as np
  import argparse
  def Hist_and_Backproj(val):
  bins = val
  histSize = max(bins, 2)
  ranges = [0, 180] # hue_range
  hist = cv.calcHist([hue], [0], None, [histSize], ranges, accumulate=False)
  cv.normalize(hist, hist, alpha=0, beta=255, norm_type=cv.NORM_MINMAX)
  backproj = cv.calcBackProject([hue], [0], hist, ranges, scale=1)
  cv.imshow('BackProj', backproj)
  w = 400
  h = 400
  bin_w = int(round(w / histSize))
  histImg = np.zeros((h, w, 3), dtype=np.uint8)
  for i in range(bins):
  cv.rectangle(histImg, (i*bin_w, h), ( (i+1)*bin_w, h - int(np.round( hist[i]*h/255.0 )) ), (0, 0, 255), cv.FILLED)
  cv.imshow('Histogram', histImg)
  parser = argparse.ArgumentParser(description='Code for Back Projection tutorial.')
  parser.add_argument('--input', help='Path to input image.', default='home.jpg')
  args = parser.parse_args()
  src = cv.imread(cv.samples.findFile(args.input))
  if src is None:
  print('Could not open or find the image:', args.input)
  exit(0)
  hsv = cv.cvtColor(src, cv.COLOR_BGR2HSV)
  ch = (0, 0)
  hue = np.empty(hsv.shape, hsv.dtype)
  cv.mixChannels([hsv], [hue], ch)
  window_image = 'Source image'
  cv.namedWindow(window_image)
  bins = 25
  cv.createTrackbar('* Hue bins: ', window_image, bins, 180, Hist_and_Backproj )
  Hist_and_Backproj(bins)
  cv.imshow(window_image, src)
  cv.waitKey()
  cv::mixChannels
  void mixChannels(const Mat *src, size_t nsrcs, Mat *dst, size_t ndsts, const int *fromTo, size_t npairs)
  Copies specified channels from input arrays to the specified channels of output arrays.
  cv::samples::findFile
  cv::String findFile(const cv::String &relative_path, bool required=true, bool silentMode=false)
  Try to find requested data file.
  cv::waitKey
  int waitKey(int delay=0)
  Waits for a pressed key.
  cv::namedWindow
  void namedWindow(const String &winname, int flags=WINDOW_AUTOSIZE)
  Creates a window.
  cv::createTrackbar
  int createTrackbar(const String &trackbarname, const String &winname, int *value, int count, TrackbarCallback onChange=0, void *userdata=0)
  Creates a trackbar and attaches it to the specified window.
  cv::imread
  CV_EXPORTS_W Mat imread(const String &filename, int flags=IMREAD_COLOR_BGR)
  Loads an image from a file.
  cv::cvtColor
  void cvtColor(InputArray src, OutputArray dst, int code, int dstCn=0, AlgorithmHint hint=cv::ALGO_HINT_DEFAULT)
  Converts an image from one color space to another.

# Explanation

- Read the input image:

  C++
  CommandLineParser parser( argc, argv, "{@input |Back_Projection_Theory0.jpg| input image}" );
  samples::addSamplesDataSearchSubDirectory("doc/tutorials/imgproc/histograms/back_projection/images");
  Mat src = imread(samples::findFile(parser.get<String>( "@input" )) );
  if( src.empty() )
  {
  cout << "Could not open or find the image!\n" << endl;
  cout << "Usage: " << argv[0] << " <Input image>" << endl;
  return -1;
  }
  Java
  if (args.length != 1) {
  System.err.println("You must supply one argument that corresponds to the path to the image.");
  System.exit(0);
  }
  Mat src = Imgcodecs.imread(args[0]);
  if (src.empty()) {
  System.err.println("Empty image: " + args[0]);
  System.exit(0);
  }
  Python
  parser = argparse.ArgumentParser(description='Code for Back Projection tutorial.')
  parser.add_argument('--input', help='Path to input image.', default='home.jpg')
  args = parser.parse_args()
  src = cv.imread(cv.samples.findFile(args.input))
  if src is None:
  print('Could not open or find the image:', args.input)
  exit(0)

- Transform it to HSV format:

  C++
  Mat hsv;
  cvtColor( src, hsv, COLOR_BGR2HSV );
  Java
  Mat hsv = new Mat();
  Imgproc.cvtColor(src, hsv, Imgproc.COLOR_BGR2HSV);
  Python
  hsv = cv.cvtColor(src, cv.COLOR_BGR2HSV)

- For this tutorial, we will use only the Hue value for our 1-D histogram (check out the fancier code in the links above if you want to use the more standard H-S histogram, which yields better results):

  C++
  hue.create(hsv.size(), hsv.depth());
  int ch[] = { 0, 0 };
  mixChannels( &hsv, 1, &hue, 1, ch, 1 );
  Java
  hue = new Mat(hsv.size(), hsv.depth());
  Core.mixChannels(Arrays.asList(hsv), Arrays.asList(hue), new MatOfInt(0, 0));
  Python
  ch = (0, 0)
  hue = np.empty(hsv.shape, hsv.dtype)
  cv.mixChannels([hsv], [hue], ch)
- as you see, we use the function cv::mixChannels to get only the channel 0 (Hue) from the hsv image. It gets the following parameters:
  - &hsv: The source array from which the channels will be copied
  - 1: The number of source arrays
  - &hue: The destination array of the copied channels
  - 1: The number of destination arrays
  - ch[] = {0,0}: The array of index pairs indicating how the channels are copied. In this case, the Hue(0) channel of &hsv is being copied to the 0 channel of &hue (1-channel)
  - 1: Number of index pairs

- Create a Trackbar for the user to enter the bin values. Any change on the Trackbar means a call to the Hist_and_Backproj callback function.

  C++
  const char* window_image = "Source image";
  namedWindow( window_image );
  createTrackbar("* Hue bins: ", window_image, &bins, 180, Hist_and_Backproj );
  Hist_and_Backproj(0, 0);
  Java
  JPanel sliderPanel = new JPanel();
  sliderPanel.setLayout(new BoxLayout(sliderPanel, BoxLayout.PAGE_AXIS));
  sliderPanel.add(new JLabel("* Hue bins: "));
  JSlider slider = new JSlider(0, MAX_SLIDER, bins);
  slider.setMajorTickSpacing(25);
  slider.setMinorTickSpacing(5);
  slider.setPaintTicks(true);
  slider.setPaintLabels(true);
  slider.addChangeListener(new ChangeListener() {
  @Override
  public void stateChanged(ChangeEvent e) {
  JSlider source = (JSlider) e.getSource();
  bins = source.getValue();
  update();
  }
  });
  sliderPanel.add(slider);
  pane.add(sliderPanel, BorderLayout.PAGE_START);
  Python
  window_image = 'Source image'
  cv.namedWindow(window_image)
  bins = 25
  cv.createTrackbar('* Hue bins: ', window_image, bins, 180, Hist_and_Backproj )
  Hist_and_Backproj(bins)

- Show the image and wait for the user to exit the program:

  C++
  imshow( window_image, src );
  // Wait until user exits the program
  waitKey();
  Java
  // Use the content pane's default BorderLayout. No need for
  // setLayout(new BorderLayout());
  // Display the window.
  frame.pack();
  frame.setVisible(true);
  Python
  cv.imshow(window_image, src)
  cv.waitKey()

- Hist_and_Backproj function: Initialize the arguments needed for cv::calcHist . The number of bins comes from the Trackbar:

  C++
  int histSize = MAX( bins, 2 );
  float hue_range[] = { 0, 180 };
  const float* ranges[] = { hue_range };
  Java
  int histSize = Math.max(bins, 2);
  float[] hueRange = {0, 180};
  Python
  bins = val
  histSize = max(bins, 2)
  ranges = [0, 180] # hue_range

- Calculate the Histogram and normalize it to the range \([0,255]\)

  C++
  Mat hist;
  calcHist( &hue, 1, 0, Mat(), hist, 1, &histSize, ranges, true, false );
  normalize( hist, hist, 0, 255, NORM_MINMAX, -1, Mat() );
  Java
  Mat hist = new Mat();
  List<Mat> hueList = Arrays.asList(hue);
  Imgproc.calcHist(hueList, new MatOfInt(0), new Mat(), hist, new MatOfInt(histSize), new MatOfFloat(hueRange), false);
  Core.normalize(hist, hist, 0, 255, Core.NORM_MINMAX);
  Python
  hist = cv.calcHist([hue], [0], None, [histSize], ranges, accumulate=False)
  cv.normalize(hist, hist, alpha=0, beta=255, norm_type=cv.NORM_MINMAX)

- Get the Backprojection of the same image by calling the function cv::calcBackProject

  C++
  Mat backproj;
  calcBackProject( &hue, 1, 0, hist, backproj, ranges, 1, true );
  Java
  Mat backproj = new Mat();
  Imgproc.calcBackProject(hueList, new MatOfInt(0), hist, backproj, new MatOfFloat(hueRange), 1);
  Python
  backproj = cv.calcBackProject([hue], [0], hist, ranges, scale=1)
- all the arguments are known (the same as used to calculate the histogram), only we add the backproj matrix, which will store the backprojection of the source image (&hue)

- Display backproj:

  C++
  imshow( "BackProj", backproj );
  Java
  Image backprojImg = HighGui.toBufferedImage(backproj);
  backprojLabel.setIcon(new ImageIcon(backprojImg));
  Python
  cv.imshow('BackProj', backproj)

- Draw the 1-D Hue histogram of the image:

  C++
  int w = 400, h = 400;
  int bin_w = cvRound( (double) w / histSize );
  Mat histImg = Mat::zeros( h, w, CV_8UC3 );
  for (int i = 0; i < bins; i++)
  {
  rectangle( histImg, Point( i*bin_w, h ), Point( (i+1)*bin_w, h - cvRound( hist.at<float>(i)*h/255.0 ) ),
  Scalar( 0, 0, 255 ), FILLED );
  }
  imshow( "Histogram", histImg );
  Java
  int w = 400, h = 400;
  int binW = (int) Math.round((double) w / histSize);
  histImg = Mat.zeros(h, w, CvType.CV_8UC3);
  float[] histData = new float[(int) (hist.total() * hist.channels())];
  hist.get(0, 0, histData);
  for (int i = 0; i < bins; i++) {
  Imgproc.rectangle(histImg, new Point(i * binW, h),
  new Point((i + 1) * binW, h - Math.round(histData[i] * h / 255.0)), new Scalar(0, 0, 255), Imgproc.FILLED);
  }
  Image histImage = HighGui.toBufferedImage(histImg);
  histImgLabel.setIcon(new ImageIcon(histImage));
  Python
  w = 400
  h = 400
  bin_w = int(round(w / histSize))
  histImg = np.zeros((h, w, 3), dtype=np.uint8)
  for i in range(bins):
  cv.rectangle(histImg, (i*bin_w, h), ( (i+1)*bin_w, h - int(np.round( hist[i]*h/255.0 )) ), (0, 0, 255), cv.FILLED)
  cv.imshow('Histogram', histImg)

# Results

Here are the output by using a sample image ( guess what? Another hand ). You can play with the bin values and you will observe how it affects the results: