This case study describes a drone-based system for automated building façade inspection that utilizes AIoT for drone management and image-analytics. The system was developed by the Real Estate & Infrastructure Division of TÜV SÜD.
Building Façades and Related Challenges
Building façades are an important aspect of buildings, both from an architectural as well as from an engineering perspective. Building façades have a huge impact not only on aesthetics but also on energy efficiency and safety. Especially in high-rise buildings, the façade can be quite complex, combining a number of different materials, including concrete, glass, steel, polymers and complex material mixes.
Problems with building façades can arise during construction as well as during the building operations phase. Typical problems include cracks in different materials, concrete spalling, corrosion, delamination, decolorization, efflorescence, peeling and flaking, chalking, hollowness, sealant deterioration, and so on. While some of these problems only have an impact on the optics of the buildings, others can have a quite severe impact on safety, e.g., because of façade elements falling down from high heights, increased risk of fires, or even complete collapses.
Façade inspection is an integral part of building maintenance, especially for high-rise buildings. It helps to verify the integrity of the building structure and ensures safety for its occupants and people passing by. However, conventional manual façade inspection can be time, labor and cost intensive, and disruptive for building occupants, and dangerous for inspectors due to difficult access at height. Finally, the results of manual façade inspection can be subjective, depending on the expertise of the inspector.
In some countries, regular façade inspections are required by regulators. Regulations usually differ depending on building size and age. For example, in Singapore, buildings older than 20 years old and over 13 meters in height have to undergo façade inspections every 7 years. In other countries, the requirements for periodic façade inspections are driven more by building insurance companies.
Automated Façade Inspection
Automated façade inspection solutions must accurately scan the exterior of buildings, e.g., utilizing drones to carry high-resolution cameras. The Smart Façade Inspection service of TÜV SÜD caters to building owners and operators of large high-rise buildings and helps construction companies ensure façade quality and monitor construction progress.
The customer journey of the automated façade inspection solution starts with the customer request for the service. Based on the customer information provided, the service operator (TÜV SÜD) will prepare the required documentation and apply with the required authorities for drone flight approvals. On-site inspection will be carried out by a specialized drone operations team. The data, inspection results and a 3D model of the façade will be made available via a specialized cloud platform.
Customer benefits include:
- The results are available in a fraction of the time compared to conventional inspection
- Digital representation of the façade and whole building facilitates building operation
- Automated digital workflow and data benchmarking improve service quality and interoperability
- Domain experts for standards and best practice, ensuring up-to-date compliance to continually evolving regulations
Implementation with AIoT
At the core of the operational system is a smart piloting system for the drone, which ensures both operational safety and high-quality visual inspection. The acquired data are securely managed by TÜV SÜD’s inspection platform, which automatically masks any private information to protect your privacy.
The AI-based solution assists professional engineers in delivering detailed, accurate and compliant inspection reports. The software constructs a 3D model of the building façade, which helps to better understand the building structure and automatically locate the detected defects on the building.
The TÜV SÜD Drone Façade Inspection application provides access to all the data, report findings, and 3D model at any time. Repairs and follow-ups can be seamlessly managed through the platform to improve efficiency and save costs.
Principle stakeholders for operations of the drone in the field include the drone pilot, safety officer, and domain expert. Professional engineers are supporting in the backend. Customer stakeholders include building owners, facility managers, and regulators.
The drone is equipped with a number of sensors to support both flight operations and building façade scanning. These flight support sensors include IMU, UWB, Lidar and stereo cameras. The drone carries thermal sensors and a visual camera as the main payload for building façade data capture. On the drone, AI is mainly used for drone positioning, collision avoidance and path planning. This is supported by a smart controller device used by the drone pilot on the ground.
A number of backend applications support the management, processing, analysis and visualization of the captured data. Domain experts and professional engineers can add their domain expertise as well.
A key feature of the solution is advanced drone control, which provides semi-automated path control for scanning the building surface, supporting complex urban environments. Multimodal sensor fusion is used for navigation. Auto path planning supports inspection and obstacle avoidance and operational safety of the drone and ensures high-quality image capture for visual inspection.
To support this, the drone carries a miniature, high-performance Inertial Measurement Unit (IMU) and Attitude Heading Reference System (AHRS). The Lidar sensor provides stereo data for dense short range on path obstacle detection (30 m). The system also has two stereo cameras for sparse long-range obstacle detection (120 m).
Drone Data Analysis: Façade Inspection
Another key application of AI is drone data analysis, which is used for creating façade inspection reports. First, the raw façade data are preprocessed, e.g., anonymizing the captured data. Second, an AI-enhanced image analysis tool is applied to visual and thermal data. Finally, the meta-data are analyzed, utilizing AI to identify individual façade elements, different types of defects, and even detailed defect attributes.
The following discussion will provide insights into the TÜV SÜD Drone-based Building Façade Inspection project from Marc Grosskopf (Business Unit Manager, Building Lifecycle Services, TÜV SÜD, Germany) and Martin Saerbeck (CTO Digital Services, TÜV SÜD, Singapore).
Dirk Slama: Marc, what were -- or are -- some of the biggest challenges in this project?
Marc Grosskopf: Only opportunities, no challenges! However, all kidding aside: of course this is an iterative process, from the initial pilots to the global roll-out which we are currently preparing. In the early stages, challenges tend to be more common on the technology and sourcing side. Then, you are quickly getting into regulatory aspects, customer acceptance, data quality, internal acceptance and processes, regional differences, etc. So it is never getting boring.
DS: Martin, from the CTO perspective, what were some of the initial challenges?
Martin Saerbeck: On the technology side, we have two main aspects: Drone-based image capturing and the data platform. For drones, it is very much about striking a good balance between cost, flight capabilities, and the quality of the sensors, and of course establishing a supply chain that can support us globally. For the data platform, we need to be able to support stakeholders with different backgrounds, roles and responsibilities. The user interface must be intuitive even if backend AI algorithms can be quite complex.
MG: Yes, do not forget that we have quite a complex constituency -- drone operators, data scientists, domain experts, customers, and so on -- all need to be supported by the central Façade Inspection Platform.
DS: Let's start by looking at the drones and drone operations...
MS: Of course you need to get the initial platform setup correctly. There are many powerful drone platforms available, but we need to adapt them to our needs, and not the other way around. One example is implementing automated flight path control to ensure façade coverage and high quality images. But perhaps the greatest challenge is keeping up with the constant flux of technology and changing regulatory requirements in different regions. Take, just as an example, free-flying vs. tethered (i.e., cable-bound drones). There are many different opinions on what should happen if the tether fails: Are we allowed to automatically switch over to the drone battery for safe landing or not? How much time do we to have until we need to trigger an emergency routine? What exactly constitutes a tether failure? The list goes on. For us, it is important to be directly involved in standardization committee work, both locally and globally.
MG: Technical people tend to focus on the "sexy" stuff first: AI, automation, image analysis, and so on. However, we also need to look at drone maintenance, firmware updates and battery management. Of course, on-site support such as system setup, traffic management, etc. At the end of the day, this process needs to be so efficient and effective that the overall process is cheaper than the manual process. We need to ensure that we have enough in-house knowledge before we can source this regionally. We cannot take any shortcuts because we need a solid foundation and have to avoid building up technical debt because of cost-driven supplier situations.
DS: Let's talk about the backend platform. What does this look like?
MG: It depends on who you talk to. For drone pilots and on-site staff, the platform mainly needs to support the management of image uploads. For domain experts, we need an efficient way of reviewing and labeling the image material. This process is now increasingly supported by AI. Finally, we have end customers who access the platform to obtain the final results and reports. As an added challenge, they want to use the platform to monitor and manage building defects in the current project and for future comparison of quality development."
DS: Does this mean the platform is not magically smart and fully automates the inspection process from the start via image analysis?
MG: It gives us a fully digital process from the beginning, since we now have a process for efficiently capturing and managing the image data. This is already an important step. We are now gradually using our huge network of building façade domain experts to label relevant data and then use this to train the system. This means over time we get more and more automation. Initially, by prefiltering huge amounts of data, domain experts only have to review relevant image data. So this leads to more automated classification.
DS: How does this look like?
MS: Based on the labeled data from domain experts, our data scientists are accessing the platform via standard developer tools to build a library of defect detection algorithms. These algorithms vary depending on the defect type and the façade materials. For example, cracks need different detection algorithms than spallings; glass façades are different than metal or concrete.
DS: When can you retrain, and when do you have to develop new algorithms?
MS: It depends. For example, for cracks in different concrete types, we can use transfer learning to a certain extent. However, detecting and evaluating cracks in glass requires models that we essentially train from scratch.
DS: What about privacy?
MS: This is a very important point. Especially if the drone is likely to inadvertently capture people throughout the scanning process (e.g., standing behind windows), we need to automatically identify and anonymize this. Privacy preservation is key. We spent considerable effort on this portion.
DS: So how are your scaling this up for global roll-out?
MG: First, we have to ensure regional support for the drone service. This means dealing with local regulations, finding local service partners, suppliers and so on. Then, we have to ensure that our processes can be easily replicated: how do you execute a drone-based building scan, how are our domain experts working with the data, how can our central competence center in Singapore best support the regions with reusable fault detection algorithms, and how do we best onboard and support our customers in the regions?
DS: Your current focus is on building façades. Can you apply your lessons learned also to other use cases?
MS: Sure. Let us take, for example, building construction progress monitoring. There are many similarities here. This is an area where we are following a similar approach, together with our partner Contillio, which is focusing on Lidar and AI for analyzing the construction progress and mapping this back to the original BIM models. Of course you can also take a similar approach to inspection of power plants, bridges, solar panels, etc. A lot is happening, but we have to take it step by step!
DS: Thank you, Marc and Martin!