- Open Access
Visual monitoring of civil infrastructure systems via camera-equipped Unmanned Aerial Vehicles (UAVs): a review of related works
Visualization in Engineering volume 4, Article number: 1 (2016)
Over the past few years, the application of camera-equipped Unmanned Aerial Vehicles (UAVs) for visually monitoring construction and operation of buildings, bridges, and other types of civil infrastructure systems has exponentially grown. These platforms can frequently survey construction sites, monitor work-in-progress, create documents for safety, and inspect existing structures, particularly for hard-to-reach areas. The purpose of this paper is to provide a concise review of the most recent methods that streamline collection, analysis, visualization, and communication of the visual data captured from these platforms, with and without using Building Information Models (BIM) as a priori information. Specifically, the most relevant works from Civil Engineering, Computer Vision, and Robotics communities are presented and compared in terms of their potential to lead to automatic construction monitoring and civil infrastructure condition assessment.
The application of Unmanned Aerial Vehicles (UAVs), also known as drones, on project sites has exponentially grown in the past few years (ENR 2015). The rapid advances in sensing, battery, and aeronautics technologies, together with autonomous navigation methods and equipped low-cost digital cameras have helped make UAVs more affordable, reliable, and easy to operate (Liu et al. 2014). Today, large numbers of Architecture/Engineering/Construction and Facility Management (AEC/FM) firms and relevant service companies use these platforms to visually monitor construction and operation of buildings, bridges, and other types of civil infrastructure systems. By capturing very large collections of images and videos, along with methods that process the visual data into 3D models, these platforms frequently survey construction sites, monitor work-in-progress, create documents for safety, and inspect existing structures, particularly for hard-to-reach areas (Liu et al. 2014). The purpose of this paper is to provide a concise review of (1) the most recent methods that streamline the collection and analysis of the visual data captured from these platforms, and (2) methods that transform and visualize actionable performance information from the collected data, with and without using Building Information Models (BIM) as a priori information. The relevant works from different research communities are presented and their potential for automating each of the processes mentioned above for both construction monitoring and condition assessment in civil infrastructure systems are discussed.
Relation to other fields
Generating as-is 3D models of the built environment using visual data collected via UAVs, transforming the outcome into useful information about the scene, and guaranteeing the accuracy and completeness during the data collection process, have all received significant attentions from both Computer Vision and Robotics communities. However, direct application of these methods for construction monitoring and civil infrastructure condition assessment purposes is challenging due to (1) the enormous amounts of visual data that need to be frequently captured and rapidly processed (e.g. more than thousands of images are collected per flight on a construction site, which results in 3D point clouds with more than hundred million points); and (2) a large number of civil infrastructure elements that needs to be detected and characterized that all of these elements and their relevant visual data need to be analyzed in less than a few hours to be meaningful for real-time project controls. In construction and civil infrastructure domains, BIM can provide strong a priori information about geometry and appearance of scenes. Therefore, collection of visual data, and detection and analysis of individual elements would be simpler problems compared to the generic problems encountered in Computer Vision and Robotics fields. This makes the problem domain of UAV-driven construction monitoring and civil infrastructure condition assessment interesting, as it can lead to domain-specific findings which can be generalized to solve the generic problems. Such problems being closely related to several research fields encourage interdisciplinary and multi-disciplinary efforts, and it is our hope that this paper contributes to highlighting such open areas of research.
Organization of this paper
This paper starts with problem statements on (1) collection of visual data via UAVs, (2) synthesizing and processing the visual data into actionable performance information, and (3) visualizing and communicating the outcome with practitioners involved in the project (Section 2). The subsequent sections review the most relevant works on each procedure and present the open areas of research. Section 6 concludes the paper.
UAV-driven visual monitoring for construction and civil infrastructure systems
The goal of a UAV-driven visual performance monitoring procedure is to (1) collect images or videos from the most informative views on a project site, (2) analyze them with or without a priori BIM to reason about performance deviations during construction (e.g. progress and quality), (3) monitor ongoing operations for productivity and safety, (4) characterize the as-is conditions of existing civil infrastructure systems, and (5) quickly and frequently visualize and communicate the most updated state of work-in-progress with onsite and offsite project participants. Figure 1 illustrates an example of next generation construction site where camera-equipped UAVs autonomously monitor the construction performance. The following sections describe the most recent research in each procedure, their challenges, and future direction for research.
Collecting informative visual data
Providing accurate performance information about the state of construction or existing conditions of civil infrastructures requires UAVs to collect visual data in form of images and videos (e.g., digital: RGB; thermal: T; depth: D; digital + depth: RGB + D) from the most relevant locations and views on a project site. To streamline this process, research in UAV-driven visual data collection need to address the following challenges: (1) autonomous or semi-autonomous path planning, navigation, and take-off and landing procedures; (2) characterization of the criteria necessary for data collection, including the configurations among the images to guarantee complete as-built information; and (3) identification of the most informative views for data collection (e.g., canonical view, top down view, etc. for appearance-based recognition of work-in-progress as in (Han et al. 2015)). Table 1 categorizes prior works based on their level of autonomy and with respect to their specific applications.
UAV-driven data collection in practice still relies on experienced pilots navigating the UAVs on and around project sites although there have been recent efforts on Simultaneous Localization and Mapping (SLAM) techniques by the research community. Recent studies (Fernandez Galarreta et al. 2015; Kerle 2999; Zollmann et al. 2014) and commercial systems (DJI 2015) provide autonomous navigation and data collection capabilities using GPS waypoints and pre-determined flight path. These methods are beneficial in land surveying or monitoring low-structures with large footprint. Nevertheless, a GPS-driven flight planning method that builds on existing maps has the following limitations (Lin, Han, Fukuchi et al. 2015a): (1) does not account for dynamics on a construction site and their impact on safety (e.g. the location and orientation of temporary resources such as cranes); (2) can be negatively affected by loss/interference in GPS signal at interiors or due to shadowing effects caused by nearby buildings or other structures in densely populated metropolitan areas and high-rise buildings; and (3) can be subject to navigational hazards due to the loss of calibration on magnetometer sensors in proximity to structural and non-structural steel components. In the meantime, research on SLAM techniques for UAVs and ground robots has been mainstream in the Robotics community. Using high resolution laser scanners (Zhang & Singh, S. “LOAM: Lidar Odometry and Mapping in Real-time.” Proc., Robotics: Science and Systems Conference (RSS 2014), monocular cameras (Blo, x, sch, M et al. 2010; Lui et al. 2015), and RGB-D cameras (Loianno et al. 2015), these techniques generate 3D maps of unknown scenes and localize the robot in that environment. The latest efforts such as (Michael et al. 2014) have focused on experimenting a variant of SLAM using RGD-B for scanning post-disaster buildings, yet there is almost little to no work reported on actively testing these algorithms for producing 3D maps on civil infrastructure systems for construction monitoring or condition assessment purposes. Also, methods that account for evolving structures (i.e. large numbers of shoring and frames in room partitions) and dynamic objects (i.e. equipment and human) are not reported in the literature.
As can be seen from the literature, research is mainly focused on challenge 1, and challenge 2 and 3 still lacks investigation in terms of the number of studies conducted. For example, identifying the most informative views for observing different tasks with or without BIM, and collecting visual data to monitor locations, activities of equipment, and craft workers on a jobsite are not well studied. There is an opportunity for leveraging a priori information about geometry and appearance of a site via BIM. When integrated with schedules, BIM can also report on the most likely locations for expected changes on the site to steer data collection. A BIM-driven data collection method has potential to overcome some of these challenges mentioned above such as collision avoidance with structures. Together with 4D (3D + time) point clouds, BIM definitely has potential to minimize many technical challenges facing fully autonomous navigation and data collection for the built environment.
Visual data analytics
The analytics of visual data has been a research subject for more than a decade. Many image processing, computer vision, and geometrical processing techniques are developed that can (1) generate semantically-rich 3D models from collections of overlapping images; (2) manually, semi-automatically, or automatically conduct progress monitoring, surveying, safety inspection, quality monitoring and activity analysis during construction, and (3) streamline condition assessment in existing buildings and infrastructure systems. (Cho et al. 2015; Pătrăucean et al. 2015; Son et al. 2015; Teizer 2015; Yang et al. 2015) provide thorough reviews of these techniques. These all methods can be applied to visual data collected from UAVs, yet only a few studies have validated them in such contexts. Tables 2 and 3 summarize the most recent literature from the last few years that focus on methods that are exclusively developed or applied to images and videos from UAVs.
Figure 2 categorizes these methods based on their level of automation and with respect to specific applications. As seen, prior works are mainly based on image-based 3D reconstruction procedures to analyze unordered and uncalibrated image collections. While these methods generally work better on ordered and sequential images, yet without a rigid control on the location and viewpoint of the captured images with respect to onsite construction elements, the visual content of the images may not be most suitable for certain applications. For example, for appearance-based construction progress monitoring, leveraging material recognition techniques such as Han et al. (Han et al. 2015) benefits from images that are taken in a canonical view to identify the most updated status of construction for onsite elements. This requires research that identifies the most informative views, and then techniques that can leverage such information as a priori information for data collection. In addition, a detailed assessment of work-in-progress, quality, and existing conditions requires the analysis of geometry, appearance, and interconnectivity among the construction elements. It also requires techniques for as-built and as-damaged information modeling; nevertheless, techniques that address these issues are not well studied. Ongoing efforts such as the Flying Superintendents project at the University of Illinois (Lin, Han & Golparvar-Fard 2015), and the ARIA project at Carnegie Melon University (ARIA (Team 2015)) are geared towards providing specific frameworks together with methods for construction monitoring and civil infrastructure condition assessment to address some of these gaps-in-knowledge.
Achieving effective flow of information both to and from project sites and conducting actionable analytics for construction monitoring and condition assessment require intuitive visualization of the information produced throughout the process on top of the UAV visual data - images and point clouds. While attention to visual sensing and analytics has been the mainstream of the literature, less work is conducted on interactive visualization. A recent work by (Zollmann et al. 2014) introduces an interactive multi-layer 4D visualization of information captured and analyzed through a UAV-driven procedure in form of mobile augmented reality. Their system, similar to (Karsch 2999) overlays color-coded 3D construction progress information on the physical world and adopts filtering methods to avoid information clutter and issues associated with displaying detailed augmented information. Other recent examples by (Han et al. 2015; Lin, Han, Fukuchi et al. 2015a) introduce web-based tools with scalable system architectures for visualizing and manipulating large scale 4D point cloud data, large collections of images, 4D BIM, and other project information. These methods account for level of details in data representation and dynamically consider the limited computational power and connection bandwidth for visualizing data on commodity smartphones. Since BIM is hosted on the server side of their system architecture, these tools can support pull and push of the geometry and other semantic information from BIM. This provides access to the most updated information and does not require storing BIM locally on the client device (Fig. 3). More research is still needed to map, visualize, and explore modalities of user interaction with operation-level construction performance data (e.g. locations and activities of workers and equipment) and condition assessment information (e.g. the location and characteristics of defects) through these models, both on and off site.
While the application of camera-equipped UAVs provides an unprecedented mechanism for inexpensive, easy, and quick documentation of the as-built data, there are still numerous open problems for further research. Nevertheless, given the editorial limitations on the scope, this paper only reviews the most recent and relevant UAV-driven research on automating construction monitoring and civil infrastructure condition assessment. It presents high level gaps-in-knowledge and opportunities for further research on data collection, analytics, and visualization techniques. Conducting interdisciplinary and multi-disciplinary research to address such technical challenges can make an impact on enhancing the current practices of project controls and condition assessment in the AEC/FM industry. It is our hope that this manuscript can highlight some of the emerging opportunities for multi-disciplinary and interdisciplinary research on autonomous vision-based systems for construction performance monitoring and condition assessment purposes.
ARIA Team (2015). "The Aerial Robotic Infrastructure Analyst (ARIA) project." <http://aria.ri.cmu.edu/>.
Blo, x, sch, M, Weiss, S, Scaramuzza, D, and Siegwart, R. “Vision based MAV navigation in unknown and unstructured environments.” Proc., Robotics and Automation (ICRA), 2010 IEEE International Conference on, Piscataway, NJ. 21-28.
Cho, YK, Ham, Y, & Golpavar-Fard, M. (2015). 3D as-is building energy modeling and diagnostics: A review of the state-of-the-art. Advanced Engineering Informatics, 29(2), 184–195.
DJI (2015). “DJI Pilot version 1.2.0.”, <http://www.dji.com/product/phantom-3/pilot-app>.
Dobson, RJ., Brooks, C, Roussi, C, and Colling, T. “Developing an unpaved road assessment system for practical deployment with high-resolution optical data collection using a helicopter UAV.” Proc., Unmanned Aircraft Systems (ICUAS), 2013 International Conference on, Piscataway, NJ. 235-243.
ENR (2015). “Drones: A Gateway Technology to Full Site Automation.”
Eschmann, C, Kuo, C-M, Kuo, C-H, and Boller, C. “Unmanned aircraft systems for remote building inspection and monitoring.” Proc., 6th European workshop on structural health monitoring.
Fernandez Galarreta, J, Kerle, N, & Gerke, M. (2015). UAV-based urban structural damage assessment using object-based image analysis and semantic reasoning. Natural Hazards and Earth System Sciences, 15(6), 1087–1101.
Fiorillo, F, Jiménez, B., Remondino, F, & Barba, S. (2012). 3D surveying and modeling of the archaeological area of Paestum, Italy. Virtual Archaeol Rev, 4, 55–60.
Gao, J, Yan, Y, and Wang, C. “Research on the Application of UAV Remote Sensing in Geologic Hazards Investigation for Oil and Gas Pipelines.” Proc., ICPTT 2011, Beijing, China. 381-390.
Gheisari, M, Irizarry, J, and Walker, B. “UAS4SAFETY: The Potential of Unmanned Aerial Systems for Construction Safety Applications.” Proc., Construction Research Congress 2014, Reston, VA. 1801-1810.
Han, K, Lin, J, and Golparvar-Fard, M. “A Formalism for Utilization of Autonomous Vision-Based Systems and Integrated Project Models for Construction Progress Monitoring.” Proc., 2015 Conference on Autonomous and Robotic Construction of Infrastructure.
Irizarry, J, Gheisari, M, & Walker, B. (2012). Usability Assessment of Drone Technology as Safety Inspection Tools. Journal of Information Technology in Construction, 17, 194–212.
Karsch, K., Golparvar-Fard, M., and Forsyth, D. “ConstructAide: Analyzing and visualizing construction sites through photographs and building models.” Proc., ACM Transactions on Graphics.
Kerle, N, Fernandez Galarreta, J, and Gerke, M. “Urban structural damage assessment with oblique UAV imagery, object-based image analysis and semantic reasoning.” Proc., 35th Asian conference on remote sensing.
Kluckner, S, Birchbauer, J-A, Windisch, C, Hoppe, C, Irschara, A, Wendel, A, Zollmann, S, Reitmayr, G, and Bischof, H. “AVSS 2011 demo session: Construction site monitoring from highly-overlapping MAV images.” Proc., Advanced Video and Signal-Based Surveillance (AVSS), 2011 8th IEEE International Conference on, Piscataway, NJ. 531-532.
Lin, J, Han, K, Fukuchi, Y, Eda, M, and Golparvar-Fard, M (2015a). “Model based Monitoring of work in progress via Images Taken by Camera equipped UAV and BIM.” Proc., 2015 International Conference on Civil and Building Engineering Informatics.
Lin, J, Han, K, & Golparvar-Fard, M. (2015). 2015. Proc, Computing in Civil Engineering, 2015, Reston, VA. 156–164.
Liu, P, Chen, AY, Huang, YN, Han, JY, Lai, JS, Kang, SC, et al. (2014). A Review of Rotorcraft Unmanned Aerial Vehicle (UAV) Developments and Applications in Civil Engineering. Smart Structures and Systems. Smart Structures and Systems, 13(6), 1065–1094.
Loianno, G, Thomas, J, and Kumar, V. “Cooperative localization and mapping of MAVs using RGB-D sensors.” Proc., Robotics and Automation (ICRA), 2015 IEEE International Conference on, Piscataway, NJ. 4021-4028.
Lui, V, and Drummond, T. “Image based optimisation without global consistency for constant time monocular visual SLAM.” Proc., Robotics and Automation (ICRA), 2015 IEEE International Conference on, Piscataway, NJ. 5799-5806.
Matsuoka, R, Nagusa, I, Yasuhara, H, Mori, M, Katayama, T, Yachi, N, et al. (2012). Measurement of large-scale solar power plant by using images acquired by non-metric digital camera on board UAV. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XXXIX-B1, 435–440.
Michael, N, Shen, S, Mohta, K, Kumar, V, Nagatani, K, Okada, Y, Kiribayashi, S, Otake, K, Yoshida, K, Ohno, K, Takeuchi, E, and Tadokoro, S. (2014). “Collaborative Mapping of an Earthquake Damaged Building via Ground and Aerial Robots.” Field and Service Robotics, K. Yoshida, and S. Tadokoro, eds., Toronto, Canada: Springer Berlin Heidelberg, 33-47.
Oskouie, P, Becerik-Gerber, B, & Soibelman, L. (2015). A Data Quality-Driven Framework for Asset Condition Assessment Using LiDAR and Image Data. Computing in Civil Engineering, 2015, 240–248.
Pătrăucean, V, Armeni, I, Nahangi, M, Yeung, J, Brilakis, I, & Haas, C. (2015). State of research in automatic as-built modelling. Advanced Engineering Informatics, 29(2), 162–171.
Qin, R. (2014). An Object-Based Hierarchical Method for Change Detection using georeferenced UAV images. Remote Sensing, 6, 7911–7932.
Siebert, S, & Teizer, J. (2014). Mobile 3D mapping for surveying earthwork projects using an Unmanned Aerial Vehicle (UAV) system. Automation in Construction, 41, 1–14.
Son, H, Bosché, F, & Kim, C. (2015). As-built data acquisition and its use in production monitoring and automated layout of civil infrastructure: A survey. Advanced Engineering Informatics, 29(2), 172–183.
Teizer, J. (2015). Status quo and open challenges in vision-based sensing and tracking of temporary resources on infrastructure construction sites. Advanced Engineering Informatics, 29(2), 225–238.
Vetrivel, A, Gerke, M, Kerle, N, and Vosselman, G. “Segmentation of UAV based images incorporating 3D point cloud information.” Proc., PIA15 + HRIGI15: Joint ISPRS conference
Wefelscheid, C, Hansch, R, & Hellwich, O. (2011). Three-dimensional building reconstruction using images obtained by unmanned aerial vehicles. ISPRS – Int Arch Photogramm Remote Sens Spatial Inform Sci, XXXVIII-1/C22, 183–188.
Wen, M-C, and Kang, S-C. “Augmented Reality and Unmanned Aerial Vehicle Assist in Construction Management.” Proc., Computing in Civil and Building Engineering (2014), Reston, VA. 1570-1577.
Xie, F, Lin, Z, Gui, D, & Lin, H. (2012). Study on construction of 3D building based on UAV images. ISPRS – Int Arch Photogramm Remote Sens Spatial Inform Sci, XXXIX-B1, 469–473.
Yamamoto, T, Kusumoto, H, and Banjo, K. “Data Collection System for a Rapid Recovery Work: Using Digital Photogrammetry and a Small Unmanned Aerial Vehicle (UAV).” Proc., Computing in Civil and Building Engineering (2014), Reston, VA. 875-882.
Yang, J, Park, M-W, Vela, P A, & Golparvar-Fard, M (2015). Construction performance monitoring via still images, time-lapse photos, and video streams: Now, tomorrow, and the future. Advanced Engineering Informatics, 29(2), 211–224.
Ye, S, Nourzad, S, Pradhan, A, Bartoli, I, and Kontsos, A. “Automated Detection of Damaged Areas after Hurricane Sandy using Aerial Color Images.” Proc., Computing in Civil and Building Engineering (2014), Reston, VA. 1796-1803.
Zhang, C, & Elaksher, A. (2012). An Unmanned Aerial Vehicle-Based Imaging System for 3D Measurement of Unpaved Road Surface Distresses1. Computer-Aided Civil and Infrastructure Engineering, 27(2), 118–129.
Zhang, J, and Singh, S. “LOAM: Lidar Odometry and Mapping in Real-time.” Proc., Robotics: Science and Systems Conference (RSS 2014).
Zollmann, S, Kalkofen, D, Hoppe, C, Kluckner, S, Bischof, H, and Reitmayr, G. “Interactive 4D Overview and Detail Visualization in Augmented Reality.” Proc., IEEE International Symposium on Mixed and Augmented Reality (2012).
Zollmann, S, Hoppe, C, Kluckner, S, Poglitsch, C, Bischof, H, & Reitmayr, G. (2014). Augmented Reality for Construction Site Monitoring and Documentation. Proceedings Of the IEEE, 102(2), 137–154.
This material is in part based upon work supported by the National Science Foundation under Grants #1427111 and #1446765. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
The authors declare that they have no competing interests.
YH, KH, JL, and MGF carried out the literature review, explore gaps-in-knowledge in underlying theories and practices, and drafted the manuscript. YH led the entire processes of this study and MGF edited the manuscript. All authors read and approved the final manuscript.