Bridge deck image stitching and monitoring
Bridge deck image stitching
For the ease of bridge deck inspection and monitoring, we combine taken photos into a single large image as shown in Figure 6. This is a specific case of the general image stitching problem. In image stitching problem, camera motion is unknown and not constrained and intrinsic camera parameters can change between the given images. In our specific problem of bridge deck surface image stitching, we benefit from constraints we know to exist due to the nature of the problem and the setup of the hardware system. We have two identical cameras that simultaneously take images of different but overlapping areas of the bridge. Also the robot’s estimated position each time a photo is taken is known with the help of onboard sensor fusion based EKF (La et al. 2013a).
The two facing-down surface cameras (Canon EOS Rebel T3i, 16 MPixel) are mounted on two computer-controlled pneumatic rods (Figure 5). The resolution of the cameras is up to 5184 ×3456 pixels. These two surface cameras are extended out of the robot footprint area when the robot starts the data collection. Each of the cameras covers an area of a size of 1.83m×0.6m. The images simultaneously collected by these two cameras have a about 30 % overlap area that is used for image stitching as shown in Figure 7. Use of flash can be necessary to obtain shadow free and well-exposed photos and in our system cameras are set to auto-exposure and auto-flash modes. Intrinsic calibration of the cameras is made separately and the camera parameters are used to undistort the acquired images. Extrinsic calibration of the camera pair consists of finding the relative location of left camera with respect to right camera.
Motion estimation
Based on the constraints imposed by the setup, we estimate the motion as a 2D rigid motion model; translation on the x−y plane and rotation around z axis. Robot and image coordinate systems can be mapped to each other by -90 degrees rotation. Robot x−a
x
i
s corresponds to negative y−a
x
i
s of image coordinates, robot y−a
x
i
s corresponds to negative x−a
x
i
s of image coordinates (7), and the constant factor resolution is the pixels per meter ratio (R
im
).
$$ \left\{ \begin{array}{l} x_{im} = -y_{r} R_{im} \\ y_{im} = -x_{r} R_{im} \end{array} \right. $$
((7))
Sparse feature-matching and image-to-image matching procedures (Forsyth and Ponce 2003; Brown and Lowe 2007) are used to estimate the camera motion incrementally. We pose the problem as a template-matching problem that tries to find the location of the overlapping area of the images inside the other image. This way we perform left-to-right and frame-to-frame matching. Robot motion estimate gives us the rough location of overlapping area for consecutive frames. Rough overlapping area for left-to-right images matching is fixed since the camera locations on the platform are fixed. Knowing the overlapping area, appearance-based template matching can give finer estimation of the camera motion. If the robot motion estimation is not accessible or not accurate enough, overlapping area can be searched over the whole image, which is a more time consuming process.
To reduce the tremendous amount of data to be processed, we resort to multi-resolution pyramidal search method (Forsyth and Ponce 2003), where we search for a larger motion range in lower resolution image and reduce the possible motion range for higher resolution image. Due to possible large illumination and reflection changes between different frames, we use image comparison method Normalized Correlation Coefficient (8) that is less illumination independent. In Equ. (8) correlation coefficient for each location x,y is denoted by R(x,y), where search image region is I, template image that is searched is T and normalized versions of them are I
′ and T
′ respectively. We compare the grayscale versions of the images to get rid of any white-balance effects in different images.
$$ \left\{ \begin{array}{l} R(x,y) = \frac{\sum_{x^{\prime}, y^{\prime}}[T^{\prime}(x^{\prime}, y^{\prime})I^{\prime}(x+x^{\prime}, y+y^{\prime})]}{\sqrt{\sum_{x^{\prime}, y^{\prime}}T^{\prime}(x^{\prime}, y^{\prime})^{2}\sum_{x^{\prime}, y^{\prime}}I^{\prime}(x+x^{\prime}, y+y^{\prime})^{2}}}\\ I^{\prime}(x+x^{\prime}, y+y^{\prime}) = I(x+x^{\prime}, y+y^{\prime}) - \\ \frac{\sum_{x^{\prime\prime}, y^{\prime\prime}}I(x+ x^{\prime\prime}, y+ y^{\prime\prime})}{w.h}\\ T^{\prime}(x^{\prime}, y^{\prime}) = T(x^{\prime}, y^{\prime}) - \frac{\sum_{x^{\prime\prime}, y^{\prime\prime}}T(x^{\prime\prime}, y^{\prime\prime})}{w.h}. \end{array} \right. $$
((8))
here, w and h are the width and height of the image I, respectively.
Exposure compensation and blending
Exposure compensation step obtains the most blending exposures for each image by selecting the suitable brightness ratio of overlapping area between images. Then when combining existing image and the new arrived image, we are performing an image-blending step to remove the shadows in the image (9). If the new arriving pixel is considerably brighter than the existing pixel in the same location, we replace the pixel with the new one. A threshold value of 0.7 is used for th to indicate being considerable is brighter than corresponding pixel. Gaps in the region formed by the pixels to be used from new image are filled using a 2D median filter of size 7×7 pixels. This ensures the completeness of the shadow removal region.
$$ I(x,y)=f(x)\left\{ \begin{array}{l} I_{2}(x,y), I_{2}(x,y)*th > I_{1}(x,y) \\ I_{1}(x,y), else. \end{array} \right. $$
((9))
Bridge deck monitor
The bridge deck viewer (BDV) software is developed using Java language to support the bridge engineer to monitor the bridge decks in an efficient way. The stitched images are first loaded and then calibrated to map to the bridge coordinate as Figure 8. The BDV software can find the crack locations on the surface of bridge in the viewing image and allows to mark them for the next view or any purpose by left mouse click on that locations. The details of the crack detection algorithm is reported in (La et al.). The BDV also shows the notification about the position of the cracks. As can be seen in Figure 9, the flags appear at the crack locations corresponding with coordinates (x, y) on the bridge deck.
Additionally, the BDV software allows to measure the distance of crack on the deck by right mouse click on the starting point and drag the hold right mouse to the last point. A line that connects the starting point and ending point appears to show the length of the crack as shown in Figure 9. Figure 10 shows the image stitching results of two bridges in Virginia and Illinois states, respectively. The stitched image is calibrated to the bridge deck coordinate to allow the ease of condition assessments. Each stitched image has very high resolution of more than 3 Gigapixel. This allows the bridge engineer to zoom in at every specific locations to monitor the cracks even with millimeter size on the deck.
NDE methods and analysis
This section presents NDE methods including electrical resistivity (ER), impact-echo (IE) and ultrasonic surface waves (USW). The robot is equipped with four ER probes (Figure 11) and two acoustic arrays, and each array can produce 8 IE and 6 USW data set as shown in Figure 12. These raw data sets are collected by the robot at every two feet (60 cm) on the bridge deck.
Electrical resistivity (ER) data analysis
The corrosive environment of concrete and thus potential for corrosion of reinforcing steel can be well evaluated through measurement of ER of concrete. Dry concrete will pose a high resistance to the passage of current, and thus will be unable to support ionic flow. On the other hand, presence of water and chlorides in concrete, and increased porosity due to damage and cracks, will increase ion flow, and thus reduce resistivity. It has been observed that a resistivity less than 5 k Ω can support very rapid corrosion of steel. In contrast, dry concrete may have resistivity above 100 k Ω. Research has shown in a number of cases that ER of concrete can be related to the corrosion rates of reinforcing steel. The ER surveys are commonly conducted using a four-electrode Wenner probe, as illustrated in Figure 11-Left. Electrical current is applied through two outer electrodes, while the potential of the generated electrical field is measured using two inner electrodes. From the two, ER is calculated as indicated in Figure 11-Left. The robot carries four electrode Wenner probes and collects data at every two feet (60 cm) on the deck. To create a conducted environment between the ER probe and the concrete deck, the robot is integrated with the water tank and to spray water on the target locations before deploying the ER probes for measurements as shown in Figure 11-Right.
Impact-echo (IE) data analysis
Impact-Echo (IE) is a widely used NDT method that has demonstrated to be effective in identifying and characterizing delaminations in concrete structures. Impact-Echo (IE) is an elastic-wave based method to identify and characterize delaminations in concrete structures. This method uses the transient vibration response of a plate-like structure subjected to a mechanical impact. The mechanical impact generates body waves (P-waves or longitudinal waves and S-waves or transverse waves), and surface-guided waves (e.g. Lamb and Rayleigh surface waves) that propagate in the plate. The multiple-reflected and mode-converted body waves eventually construct infinite sets of vibration resonance modes within the plate. In practice, the transient time response of the solid structure is commonly measured with a contact sensor (e.g., a displacement sensor or accelerometer) coupled to the surface close to the impact source. The fast Fourier transform (amplitude spectrum) of the measured transient time-signal will show maxima (peaks) at certain frequencies, which represent particular resonance modes as show in Figure 13.
There are different ways of interpreting the severity of the delamination in a concrete deck with the IE method. One of the ways used in this study is shown in Figure 14. A test point is described as solid if the dominant frequency corresponds to the thickness stretch modes (Lamb waves) family. In that case, the frequency of the fundamental thickness stretch mode (the zero-group-velocity frequency of the first symmetric (S
1) Lamb mode, or also called the IE frequency (f
IE
). The frequency can be related to the thickness of a plate H for a known P-wave velocity C
p
of concrete by
$$ H = \frac{\beta_{1} C_{p}}{f_{IE}} $$
((10))
where β
1 is a correction factor that depends on Poisson’s ratio of concrete, ranging from 0.945 to 0.957 for the normal range of concrete. A delaminated point in the deck will theoretically demonstrate a shift in the thickness stretch mode toward higher values because the wave reflections occur at shallower depths. Depending on the extent and continuity of the delamination, the partitioning of the wave energy reflected from the bottom of the deck and the delamination may vary. The initial or incipient delamination, described as occasional separation within the depth of the slab, can be identified through the presence of dominant frequencies associated with the thickness stretch modes from both the bottom of the deck and the delamination. Progressed delamination is characterized by a single peak at a frequency corresponding to the depth of the delamination. Finally, in case of wide or shallow delaminations, the dominant response of the deck to an impact is characterized by a low frequency response of flexural-mode oscillations of the upper delaminated portion of the deck.
Ultrasonic surface waves (USW) data analysis
The ultrasonic surface waves (USW) technique is an offshoot of the spectral analysis of surface waves (SASW) method used to evaluate material properties (elastic moduli) in the near-surface zone. The SASW uses the phenomenon of surface wave dispersion (i.e., velocity of propagation as a function of frequency and wave length, in layered systems to obtain the information about layer thickness and elastic moduli) as shown in Figure 15. A SASW test consists of recording the response of the deck, at two receiver locations, to an impact on the surface of the deck (Figure 16). The surface wave velocity can be obtained by measuring the phase difference Δ
ϕ between two different sensors (sensor 1 and sensor 2) as follows,
$$ C = 2\pi f\frac{d}{\Delta \phi} $$
((11))
where f is frequency, d is distance between two sensors. The USW test is identical to the SASW, except that the frequency range of interest is limited to a narrow high-frequency range in which the surface wave penetration depth does not exceed the thickness of the tested object. In cases of relatively homogeneous materials, the velocity of the surface waves does not vary significantly with frequency. The surface wave velocity can be precisely related to the material modulus, or concrete modulus in the case of bridge decks, using either the measured or assumed mass density, or Poisson’s ratio of the material. In the case of a sound and homogenous deck, the velocity of the surface waves will show little variability. An average velocity is used to correlate it to the concrete modulus. Significant variation in the phase velocity will be an indication of the presence of a delamination or other anomaly.
Ground penetrating radar
Ground penetrating Radar (GPR) is a geophysical method that uses radar pulses to image the subsurface. GPR can provide a qualitative condition assessment of bridge decks and is used as a diagnostic tool to detect apparent or suspected deterioration in an existing deck (e.g., delamination or corrosive environment), or quality assurance tool for new construction or rehabilitation. The GPR system uses high-frequency (varying from 10 MHz to 2.5 GHz) radio waves and transmits into the ground. When the wave hits a buried object such as rebars in the bridge, the receiving antenna records variations in the reflected return signal. GPR data usually consist of changes of reflection strength and arrival time of specific reflections, source wave distortion, and signal attenuation. The rebar detections are marked by hyperbolic shapes in the GPR image (see Figure 4). These hyperbolas are obtained due to the reason that the antenna transmits energy in a spatially varying pattern which can be approximated to a cone. Then, the antenna receives the reflections from the rebars.
Two IDS Hi-Bright GPR arrays are attached on the rear side of the robot (Figure 2). Each of the arrays has sixteen antennas, or two sets of eight antennas with dual polarization to obtain high spatial resolution signals. For high-efficiency inspection, the GPR system can omit the wave and collect reflective signals even the robot is in motion. It is required that the GPR antenna has to be close (1-2 cm) to the deck surface to obtain good signals. The GPR attenuation map of a bridge deck is presented in Figure 17. The color plots in this condition map demonstrates deterioration grades of the bridge deck. The GPR condition map indicates a large cluster of serious deterioration around locations at longitudinal/lateral positions (45, 10) ft and (90, 10) ft.