envelopmenuskypeburger-menulink-externalfacebooktwitterlinkedin2crossgithub-minilinkedin-minitwitter-miniarrow_rightarrow_leftphonegithubphone-receiverstack-overflow

Traffic analysis with OpenCV: Part 1

part1 header

Introduction

Correlating with the uptick of interest in Artificial Intelligence, machine vision has received increased attention in recent years. Vivid technology demonstrations such as FaceApp and Google’s Deep Learning "paintings" are being prominently mentioned even in mainstream, non-technical media.

Flashy such things may be, they are not that useful to the average Joe - beyond a moment of amusement at least. However, they do demonstrate that with today’s availability of various libraries and APIs, as well as relative cheapness of processing power, machine vision approaches can be used at scale ubiquitously.

Series' Concept

This series of blog entries will describe, discuss and analyze various approaches to a particular machine-vision-related problem, namely road traffic analysis.

Specifically, we’ll look into the issue of identifying local road congestion in time and space. When considering this use case, we’ll be using the venerable OpenCV library.

One important thing to note, however, is that these blog entries will not be merely a series of tutorials on how to use OpenCV on the JVM - there’s enough of such content available via a simple Google search. Instead, I will attempt to emulate a genuine problem-solving process, including a discussion of sub-optimal approaches. The intention here is to demonstrate how to identify various pitfalls when tackling these kinds of challenges, and how to verify one’s reasoning by procuring Solid Data™.

The Problem Setting

Our primary dataset will be composed of recordings of small-traffic, local city roads. We will be looking for signs of traffic congestion, both during day and night. Our goal is to, at minimum, identify and demonstrate core traffic flow problems, and, building on that, suggest automated solutions for mitigation.

Our focus will therefore be two-pronged:

  • obtaining relatively accurate positioning of cars, especially those that cause a traffic congestion;

  • representing the aforementioned positioning data in a human-readable way.

The Tools

As already said, our primary tool will be the OpenCV library, specifically the JavaCV JVM port by bytedeco. Apart from that, we’ll take advantage of various other JVM libs on an as-needed basis. The code will be written in Scala.

Note

There are actually at least two popular JVM ports of OpenCV - apart from the aforementioned JavaCV, there’s also the one by OpenPnP. JavaCV was chosen here for the following reasons:

  • JavaCV contains bindings for several other machine vision libraries apart from OpenCV;

  • there’s a large base of examples, written in Scala, for reference;

  • JavaCV’s API contains some additional JVM-specific utility classes (and subjectively, appears to be friendlier to polyglot programming on the JVM),

  • JavaCV appeared to be better-optimized in preliminary tests - in a small benchmark of simple OpenCV calls [1] JavaCV was around 3 times faster than the OpenPnP bindings, apparently, due to better resource usage [2].

Identyfying cars

Initial approach

Let’s start with tackling the problem of finding cars on a still picture of a road. We will initially work on the simplest case - a limited, fixed-angle series of picture of a small stretch of road, seen from a high angle above (and therefore mitigating perspective-related distortion).

Simple Car Detection

We now want a general method to find whenever we have a car on our still. One approach is to employ an edge detection algorithm, and then calculate closed shapes (hulls) composed by the detected edges. And "edge" in this sense is simply an area of a picture where the image’s characteristics (color, lightness, etc.) change in some way.

An often-recommended approach for this is the usage of OpenCV’s implementation of the Canny edge detection algorithm. For our purposes, we need to know that:

  • the Canny detector, as with most edge detectors, is somewhat sensitive to noise (in the visual/signal sense), therefore we need to smooth our picture out with an appropriate smoothing filter (usually Gaussian smoothing is employed) ;

  • the detector requires "low-pass filter" and "high-pass filter" parameters which determine the "sensitivity" of what counts as an edge;

  • the three parameters need to be tweaked manually per picture to achieve an ideal result,

  • although there is some interaction between them that thankfully constrains our search space for the optimal values.

Without further ado, here’s the basic outline of a program that executes Canny edge detection on a still:

import org.bytedeco.javacpp.opencv_core._
import org.bytedeco.javacpp.{opencv_imgproc => ImgProc}
import org.bytedeco.javacpp.{opencv_imgcodecs => ImgCodecs}



case class ContouredCars(img: Mat, cars: MatVector) (1)

object CarCounter {
  implicit class IterativeMatVector(val mV: MatVector) extends AnyVal { (2)
    def iter() = (0L until mV.size()).view.map(mV.get)
  }
}

case class CannyParameters(low: Int, high: Int, blur: Int) { (3)
  require(blur % 2 == 0)
}

class SimpleCarCounter {

  import CarCounter._

  private def toGray(img: Mat) = { (4)
    val gray = img.clone()
    ImgProc.cvtColor(img, gray, ImgProc.COLOR_BGR2GRAY) (5)
    gray
  }

  def classifyCars(road: Mat)(params: CannyParameters): Try[ContouredCars] = {
    import resource._
    (for (gray <- managed(toGray(road))) yield { (6)

      val blurred = gray
      ImgProc.GaussianBlur(gray, blurred, new Size(params.blur, params.blur), 0) (7)

      val contours = new MatVector()

      for (edges <- managed(blurred.clone())) {
        ImgProc.Canny(blurred, edges, params.low, params.high) (8)
        ImgProc.findContours(edges, contours, ImgProc.RETR_EXTERNAL, ImgProc.CHAIN_APPROX_SIMPLE) (9)
      }

      val hulls = contours
        .iter()
        .map(m => {
          val hull = new Mat()
          ImgProc.convexHull(m, hull, false, true) (10)
          hull
        })

      val hullVector = new MatVector(hulls: _*)
      ImgProc.drawContours(road, hullVector, -1, new Scalar(255, 255, 0, 0)) (11)

      ContouredCars(road, hullVector)
    }).tried
  }

}
  1. Result utility type. Mat is a matrix type in OpenCV that contains information from an image.

  2. Utility class to obtain an iterable view of a MatVector, which is a sequence of Mat s.

  3. Parameter container for making method definitions etc. more compact

  4. Conversion to grayscale here - since the Canny algorithm operates only on one "dimension" (e.g. color component) of the image.

  5. Note that most operations/conversions are performed in-place, both due to OpenCV being a C project, and for memory efficiency’s sake.

  6. Usage of the scala-arm library [3].

  7. Reducing noise via Gaussian blur.

  8. The core call that executes Canny edge detection algorithm on the blurred image.

  9. Convert the edge information into a sequence of actual lines, or contours, of objects.

  10. Join the lines obtained in the previous step into closed hulls.

  11. Draw the contours on the original image (in GBR ffff00, i.e. cyan).

A call to obtain the contours on an image under path would be:

new SimpleCarCounter(CannyParameters(50, 150, 7))
       .classifyCars(ImgCodecs.imread(path))

And here’s some example results:

canny initial
Figure 1. Initial result (Image is additionally blurred compared to actual input).

As you can see, the output varies wildly depending on the parameter values, so we need to employ some way to tweak them.

Pre-tweaking Cleanup

However, before we do that, there are some observations we can make about the pictures:

  • there’s a large number of objects identified from the edges, way more than can be present on the given road segment;

  • some of the objects are way too large to be cars, and others way too small.

Therefore, we can:

  • filter out spurious small and large objects by simple thresholds on their sizes;

  • if necessary, include only the first N largest objects after the filtering.

Here’s how we’d go about regarding the first cleanup modification:

case class DoubleRange(low: Double, high: Double) { (1)
  def contains(d: Double) = d >= low && d <= high
}

object CarCounter {

  val WidthProportions  = (0.2, 0.6) (2)
  val HeightProportions = (0.3, 0.8)
  ...

class CarCounter {

  import CarCounter._

  ...

  private def filterOutlier(img: Mat) = { (3)
    val imgSize = img.size()

    val widthRange  = DoubleRange(imgSize.width * WidthProportions._1,
                                  imgSize.width * WidthProportions._2)
    val heightRange = DoubleRange(imgSize.height * HeightProportions._1,
                                  imgSize.height * HeightProportions._2)

    (outline: Mat) =>
      {
        val rect = ImgProc.boundingRect(outline)
        widthRange.contains(rect.width) && heightRange.contains(rect.height)
      }
  }

  ...

  def classifyCars(road: Mat)(params: CannyParameters): Try[ContouredCars] = {
  ...
  val hulls = contours
    .iter()
    .filter(filterOutlier(gray)) (3)
    .map(m => {
      val hull = new Mat()
      ImgProc.convexHull(m, hull, false, true)
      hull
    })
   ...
  1. Utility class for non-integer ranges.

  2. Size constraints - we want to exclude objects that are both too small and too big to be cars.

  3. We simply apply our size restriction function as a filtering step.

And here’s the result for identical parameter values:

canny improved
Figure 2. Improved result (image is additionally blurred compared to actual input).

As you can see, we’ve got a pretty-well cleaned-up picture already, so there’s no need to apply the second restriction.

Next steps

The approach of using Canny’s edge detection algorithm (combined with contour detection), even when selecting parameters via guesswork, has already shown some promising results.

However, there are also problems. As you can see, while the count of cars being present is roughly preserved, their shapes are occasionally inexactly matched. Also, while the examples shown here look fine, in practice every time of day/type weather conditions/etc. requires slightly different parameter values.

Next time, we’ll look into alternatives to the method discussed here that do not necessitate special tweaking.


1. The user code was basically identical - all it took to switch between the OpenPnP version and JavaCV was changing the imports plus minor collection modifications.
2. Specifically, while threading was quite similar and the OpenPnP version spent less time blocking, JavaCV actually took advantage of the available heap space, while the OpenPnP bindings never increased the heap beyond Xms.
3. There’s an efficiency problem with scala-arm and AutoCloseable currently, however it’s not relevant for illustrations purposes.

scala times
Interested in Scala news?

Subscribe to Scala Times Newspaper delivered weekly by SoftwareMill straight to your inbox.