More...More...More...More...More...Hardware.exeMore...More...More...More...More...More...Ignite AIoT - Artificial Intelligence

Quality Management (QM) is responsible for overseeing all activities and tasks needed to maintain a desired level of quality. QM in Software Development traditionally has three main components: quality planning, quality assurance, and quality control. In many agile organizations, QM is becoming closely integrated with the DevOps organization. Quality Assurance (QA) is responsible for setting up the organization and its processes to ensure the desired level of quality. In an agile organization, this means that QA needs to be closely aligned with DevOps. Quality Control (QC) is responsible for the output, usually by implementing a test strategy along the various stages of the DevOps cycle. Quality Planning is responsible for setting up the quality and test plans. In a DevOps organization, this will be a continuous process.

QM for AIoT-enabled systems must take into consideration all the specific challenges of AIoT-development, including QM for combined hardware / software development, QM for highly distributed systems (including edge components in the field), as well as any homologation requirements of the specific industry. Verification & Validation (V&V) usually plays an important role as well. For safety relevant systems (e.g. in transportation, aviation, energy grids), Independent Verification & Validation (IV&V) via an independent third party can be required.

Verification & Validation

Verification and validation (V&V) are designed to ensure that a system meets the requirements and fulfills its intended purpose. Some widely used Quality Management Systems such as ISO 9000 are building on verification and validation as key quality enablers. Validation is sometimes defined as the answer to the question "Are you building the right thing?" since it checks that the requirements are correctly implemented. Verification can be expressed as "Are you building the product right?", since it relates to the needs of the user. Common verification methods inlude unit tests, integration tests and test automation. Validation methods include user acceptance tests and usability tests. Somewhere in between verification and validation we have regression tests, system tests and beta test programs. Verification is usually linking back to requirements. In an agile setup, this can be supported by linking verification tests to the Definition of Done and the Acceptance Criteria of the user stories.

Quality Control

Quality Assurance and AIoT DevOps

So how does Quality Assurance fit in with our holistic AIoT DevOps approach? First, we need to understand the quality-related challenges, including functional and non-functional. Functional challenges can be derived from the agile story map and sprint backlogs. Non-functional challenges in an AIoT system will related to AI, cloud and enterprise systems, network, and IoT/edge devices. In addition, previously executed tests, as well as input from the ongoing system operations must be taken into consideration. All of this must serve as input to the Quality Planning. During this planning phase, concrete actions for QA-related activities in development, integration, testing and operations will be defined.

QA-tasks during development must be supported both by the development team, as well as any dedicated QA engineers. The developers usually perform tasks such as manual testing, code reviews, and development of automated unit tests. The QA engineers will work on the test suite engineering and automation setup.

During the CI phase (Continuous Integration), basic integration tests can be performed, as well as automated unit tests (before the check-in of the new code), and automatic code quality checks.

During the CT phase (Continuous Testing), many automated tests can be performed, including API testing, integration testing, system testing, automated UI tests, and automated functional tests.

Finally, during Continuous Delivery (CD) and operations, User Acceptance Test (UATs) and lab tests can be performed. For an AIoT system, digital features of the physical assets can be tested with test assets in the field.

Quality Assurance and AIoT DevOps

Verification and Validation for AIoT

V&V for AI

From the perspective of the integrated system, verification and validation of the AI-related services usually focuses on functional testing, considering the AI-based services a black box ("Black Box Testing"), which is tested in the context of the other services which make up the complete AIoT system. However, it will usually be very difficult to ensure a high level of quality if this the only test approach. Consequently, V&V for the AI services in an AIoT system also requires a "white box" approach, which specifically focuses on the AI-based functionality.

In AI, important techniques for Training Data Validation include the validation of data quality, completeness and representativeness. Techniques for Model Testing & Optimization include K-fold Cross-Validation, Holdout Method, Hyper Parameter Tuning, Statistical Validation Methods and AI Model Simulations. Field tests and production tests can also provide critical feedback to the V&V team of the AI services.

QA for AI

Integrated V&V for AIoT

On the service-level, AI services can usually be tested using the methods outlined in the previous section. After the initial tests performed by the AI service team, it is important that the AI services will be integrated into the overall AIoT product for real-world integration tests. This means that the AI services are integrated with the remaining IoT services, to build the full AIoT system. This is shown in the figure following. The fully integrated system can then be used for User Acceptance Tests, load and scalability tests, and so on.

QA for AIoT

Homologation

AIoT and Homologation

Authors and Contributors

Dirk Slama.jpeg
DIRK SLAMA
(Editor-in-Chief)

AUTHOR
Dirk Slama is VP and Chief Alliance Officer at Bosch Software Innovations (SI). Bosch SI is spearheading the Internet of Things (IoT) activities of Bosch, the global manufacturing and services group. Dirk has over 20 years experience in very large-scale distributed application projects and system integration, including SOA, BPM, M2M and most recently IoT. He is representing Bosch at the Industrial Internet Consortium and is active in the Industry 4.0 community. He holds an MBA from IMD Lausanne as well as a Diploma Degree in Computer Science from TU Berlin.
Eric Schmidt.jpg
ERIC SCHMIDT, BOSCH CENTER FOR ARTIFICIAL INTELLIGENCE
CONTRIBUTOR
Eric Schmidt is an AI consultant and data scientist at the Bosch Center for Artificial Intelligence (BCAI). He initiates and implements data-driven solutions in various areas, ranging from engineering to manufacturing and supply chain management. Eric holds degrees in both computer science and business administration.
Martin Lubisch.jpg
MARTIN LUBISCH, BOSCH CENTER FOR ARTIFICIAL INTELLIGENCE
CONTRIBUTOR
As an AI Consultant and Product Manager at the Bosch Center for Artificial Intelligence Martin Lubisch identifies solutions that help customers in their daily business and brings those to scale e.g. Vivalytic: a Covid-19 testing device enhanced with DL based optical inspection. Martin has co-founded a start-up and worked along the lines of Industry 4.0, IoT and AI in various locations such as Berlin, Stuttgart and Silicon Valley.