Object tracking with a novel visual-thermal sensor fusion method in template matching

Published Online July 2019 in MECS DOI: 10.5815/ijigsp.2019.07.03

The combined use of thermal and visible sensors in the tracking problem is motivated by the fact that each imagery has its limitations and advantages. The visible imagery though provides a foremost hint of colour of the object, but thermal imagery has got its advantages in case of night or improper vision conditions, illumination variations like the shadow, etc. and in case of the object being camouflaged.

Earlier, many efforts have been formulated to utilize the advantages of these imaging modalities in different surveillance applications. These techniques are discussed in detail in the section 2. The proposed method advances the field by working for the betterment of tracking related issues like time constraints, accuracy, and reliability.

Our method carves out a statistical correlation value based template match framework into an effective fusion approach. Initially, individual imagery track is obtained by template matching, and it is modified with the track object's dimension based haphazard transporting controlling (HTC) process in each frame. Then, the outputs retrieved from individual imagery tracker are fed into the proposed merger algorithm. This algorithm firstly makes adaption in the weight factors to be provided to each imagery and later adjusts the positional coordinates of the final track estimate to remove the offset caused by least weighing modality. The rest organization of the paper is detailed in the following paragraph.

• II.    Related Work

Due to exciting advantages of thermal-visual fusion, various uses of visual-thermal fusion are found in the literature.

Face recognition inhibiting thermal and visible sensor fusion using background subtraction had been proposed earlier in the literature [1, 2, 3, and 4]. It has also been actively utilized in robotics-based vision applications [5, 6, and 7]. Besides this, the use of visible and thermal imagery in fusion has also been hosted in spacecraft proximity operations [8] to overcome the effect of different lighting conditions in the orbital environment. [9, 10] have advanced the field of image fusion of multisensory data for the improved vision applications.

Notably, the use of said bi-modal information has been increased for object tracking under challenging conditions of vision in recent times. The significance of the field has even made researchers incorporate new intelligent techniques like deep learning and convolutional neural networks [11] in thermo-visual tracking. A two-stream convolution net was formed independently for each domain, and the fusion was achieved by forming a fusion net. Despite a new approach, it is to be noted that it is dependent on a rigorous training process and requires complex hardware processing systems.

It is interesting to note that several methods like [15, 16, and 17] make use of the particle filter [18] based approach to aggregate the information from each sensing domain and formulate a sound decision of track. However, in general, the particle filter based approaches have demerit of particle degeneration and an associated unpredictability in its output estimate, and it generally fails to track in severe visual conditions.

To address this issue, [15] proposed a basic fusion that involved multiplication of the individual specialists that used cues from thermal as well as visible imagery to form their opinions. However, simple multiplication only serves the purpose of video sequences with no veiling conditions. In case of false tracking in one mode of imagery, it may lead to overall false result due to the absence of a procedure to verify this.

Further, the techniques employing fusion to overcome demerits of particle filter [16, 17] make use of fusion step per single frame of the track as many times as the no. of particle chosen. This mechanism takes many computational efforts since most of times background calculations are involved in performing a single step fusion. In addition, the computation cost rises as if no. of the particle is increased to increase the accuracy.

More insight into the advances in this field can be reviewed in [12, 13, and 14]. Table 1 lists the description of some methods that incorporated the visible sensor and thermal sensor conjunctively. The fusion strategy adopted, along with the applicative aim of these, are covered under this table.

• III.    Proposed Work Methodology

The algorithm is formulated into a three-step approach. First is the original track estimate using correlation-based template matching, second the proposed HTC procedure and third is the said thermo-visual fusion process overview. Fig. 1 presents the overall illustration of the workflow of the complete process. The complete section is divided into the following subsections:

• A.    Template matching based Tracking

For tracking in a particular domain, a fast template matching process was incorporated. The template matching process used the value of the correlation coefficient between the template region and current frame image regions. The formulae for this is given by:

c⁽x,y) =

1.г^ш+1,у+п-П(т(и)-т)

J^S i Z j Cltx+Ly+fl-iy^S i Z j Cr’tMj-T)²}

In (1), I stands for the image and T for the template and resembles the pixel position in the template region of the object and the image frame, respectively. Values, and represents the mean values of pixels in the template region and the corresponding image region with a size equivalent to the template size. The output c(x, y) represents the template matching result of a rectangular region in the image with (x, y) as the starting pixel location.

Table 1. Type Sizes for Camera-Ready Papers