Ignite AIoT: Verification & Validation
Verification and validation (V&V) are important elements of a quality management system (QMS), implemented to ensure that the product meets the requirements and specifications. Verification typically addresses the question "Am I building the product right?", while validation is addressing the question "Am I building the right product?".
V&V for AIoT-based products has to take the specifics of AI and IoT into consideration, which is why Ignite AIoT is focusing on V&V, assuming that most organizations have overarching QM Systems which can be adapted to fit the needs of an AIoT-specific V&V approach.
Quality Assurance for AIoT
Software Quality Assurance (SQA) is a widely established practice for monitoring the software engineering processes to ensure the quality of software. SQA typically covers the entire software development process, including requirements management, software design, coding, code quality, source code control, software configuration management, testing, as well as release and rollout management. Verficication & Validation is an important subset of SQA. SQA again is an element of Software Quality Management (SQM), which includes Quality Assurance, Quality Planning, and Quality Control. In this part of the Ignite AIoT Framework the focus is on Verfication and Validation, since this has many aspects which are highly specific to AIoT. Some other aspects of SQA/SQM have also been addressed as part of the AIoT DevOps discussion.
The challenge for any AIoT product is now that it needs to combine the well well established concepts of software quality assurance with the less well established concepts of QA for embedded hardware and software, as well as Artificial Intelligence - creating an integrated QA concept for AIoT. The V-Model - as outlined in the following - provides a good foundation for this.
Foundation: V-Model
The V-model is a systems development lifecycle which has verification and validation "built in". It is often used for development of mission critical systems, e.g. in automotive, aviation, energy and military applications. It also tends to be used in hardware-centric domains. The V-model - not surprisingly - is using a v-shaped visual representation, where the left side of the "V" represents the decomposition of requirements, as well as the creation of system specifications ("definition and decomposition"). The right side of the "V" represents the integration and testing of components. Moving up on the right side, testing usually starts with the basic verification (e.g. unit tests, then integration tests), followed by validation (e.g. user acceptance tests).
When applying the V-model to AIoT, it needs to take different dimensions into consideration, usually including hardware, software, AI and networking. In addition to the standard verification tests (unit tests, integration tests) and validation tests (user acceptance and usability tests), the V-model for AIoT also needs to address interoperability testing, performance testing, scalability testing and reliability testing. The highly distributed nature of AIoT systems of will pose specific challenges here.
Test automation is key to ensure a high level of test efficiency and test coverage. On the software-side, there are many tools and techniques available to support this. In the AI-world, these kinds of tools and techniques are only just beginning to emerge, which makes it likely that a more custom approach will be required. In the embedded and hardware world, simulation techniques like Hardware-in-the-Loop (HIL), Software-in-the-Loop (SIL) and Model-in-the-Loop (MIL) are well established. However, most AIoT products will also require testing of the actual physical product, and how it performs in the field in different types of environments. This again will be a challenge, and some ingenuity will be required to automate testing of physical products wherever possible.
Agile V-Model
Ignite AIoT is trying to strike a good balance between the agile software world and the less agile world of often safety-critical, complex and large(r)-scale AIoT product development with hardware and potentially even manufacturing elements to it. Therefore it is important to understand how an agile method works well together with a V+V-centric approach like the V-model. The logical consequence is the Agile V-model. Combining agile development with the V-model is no contradiction. They both can work very well together, as shown in the figure below:
- Agile methods use story maps including epics, themes and backlogs for logical decomposition. This maps well to the left side of the V
- Continuous Integration / Continous Test / Continuous Delivery are inherently agile methods, which map well to the right side of the V
- The key assumption is that the V-model is not used like one large waterfall approach. Instead, the Agile V-model must ensure that the sprints themselves will become Vs according to the V-model
There are two options to implement the latter:
- Each sprint becomes a complete V, including development and integration / test
- The agile schedule introduces the concept of dedicated integration sprints.
- One V becomes 2 sprints: One development sprint, one integration sprint
- There are pros and cons to both approaches
- Complexity and scale of the project will surely play a role in determining the best setup
Ignite AIoT is assuming in the following that the Agile V-model will be applied.
V&V for IoT
Given the complexity and highly distributed native of an AIoT product - driven mainly by the IoT-side of things - applying the V-model approach makes a lot of sense. As discussed in the Data/Functional Viewpoint of the Product Architecture, it is recommended to apply a service architecture design to the product. This design is the first step in the decomposition of the system, as proposed by the V-model. Applying an API-first approach will help making sure that this top-level layer of system decomposition can be properly managed.
Each team responsible for an individual AIoT service will be able to develop and test this service autonomously by using simple dummy implementations for the other side of the interfaces which are not in their control. In some cases, even more sophisticated service simulation tools can be used. The next step is usually the integration of key components, to test how they are interacting. This is particularly important for services which are functionally tightly coupled. Next - and constantly moving up the right side of the "V" - will be integration tests which combine a larger number of the services which will eventually make up the complete systems. Especially after the Start of Production, it will be more and more difficult to test individual, smaller changes in the context of the complete system. This is why it is so important that the test environment utilizes the capabilities enabled by API-based decomposition.
The following table provides an overview of the key test areas that can usually be found in an IoT system:
| IoT Test Area | Description | Tools / Methods |
|---|---|---|
| Functional Testing | Verification: Does the IoT functionality match the requirements specification? | Test robots for UI/HMI; simulation for hardware |
| Usability Testing | Validation of User Acceptance | Regular User Acceptance Tests |
| Localization/Environment Testing | How are IoT devices/assets behaving in different geographic locations and environments? | Field tests |
| Device/Asset Interoperability | Are all devices/assets (own and from partners) interoperating as planned? | Test lab with physical assets |
| Performance and Load Testing | How is the system reacting to large numbers of connected devices/assets? How does it react to bursts of activities from many devices/assets? | Simulation |
| Network Testing | Connectivity and interoperability testing in different geo locations | Simulation, field tests |
| Security Testing | End-to-end security testing, from the asset/device to the cloud backend | Penetration Tests |
V&V for AI
From the perspective of the integrated system, verification and validation of the AI-related services is usually focusing on functional testing, considering the AI-based services a black box ("Black Box Testing"), which is tested in the context of the other services which make up the complete AIoT system. However, it will usually be very difficult to ensure a high level of quality if this the only test approach. Consequently, V&V for the AI services in an AIoT system also requires a "white box" approach, which specifically focuses on the AI-based functionality.
In a recent paper published by the European Union Aviation Safety Agency (EASA), the authors propose a W-shaped model for the "Design Assurance for Neural Networks". This model (see below) assigns data management, learning process management and model training to the left side of the W. The middle part is the learning process verification. On the right side, model implementation, inference model verfication and data verification can be found.
The W-model proposed by EASA could be an interesting approach for addressing some of the common challenges found in verification and validation of AI models, including poor matching quality, data bias, over- and underfitting, model decay and adverserial attacks.
In AI, important techniques for Training Data Validation include the validation of data quality, completeness and representativeness. Techniques for Model Testing & Optimization include K-fold Cross-Validation, Holdout Method, Hyper Parameter Tuning, Statistical Validation Methods and AI Model Simulations. Field tests and production tests can also provide critical feedback to the V&V team of the AI services.
V&V for AIoT
On the service-level, AI services can usually be tested using the methods outlined in the previous section. After the initial tests performed by the AI service team, it is important that the AI services will be integrated into the overall AIoT product for real-world integration tests. This means that the AI services are integrated with the remaining IoT services, to build the full AIoT system. This is shown in the figure below. The fully integrated system can then be used for User Acceptance Tests, load and scalability tests, and so on.
Agile V-Model for AIoT
Taking everything together that was discussed so far for IoT and AI, as well as Agile V-Model, and combining it with the product organization discussion from the chapter on Product Organization, the complete picture looks like in the diagram below: The AIoT workstreams are synchronized by applying the Agile V-model accross all AIoT workstreams. Development and integration sprints alternate (or are integrated into single sprints), in order to ensure that at the end of each agile V-sprint, a potentially shippable increment is achieved (although in reality this will not be achievable for the hardware components - some ingenious approach still has to be applied here).
Agile V-Model & SOP
Finally, this discussion should conclude by mentioning the probably biggest differentiator between a "normal" cloud project and an AIoT project: The Start of Production, or SOP. This is the point in time when the mass production of the smart, connected products is starting, and they are leaving the factory to be deployed in the field. Any required changes to the hardware configuration of the product will now be extremely difficult to be achieved (usually involving costly product recalls, which nobody wants). In a world where manufacturers want to utilize the AIoT to constantly stay in contact with their products after they have reached the customer, and provide the customer with new digital services and applications - rolled out via OTA - it becomes extremely important to understand that the V&V process does not stop with the SOP - at least not for anything that is software. And with software defined vehicles, software defined robots and software defined everything, this is fast becoming the new normal.
Authors and Contributors
DIRK SLAMA
(Editor-in-Chief) AUTHOR Dirk Slama is VP and Chief Alliance Officer at Bosch Software Innovations (SI). Bosch SI is spearheading the Internet of Things (IoT) activities of Bosch, the global manufacturing and services group. Dirk has over 20 years experience in very large-scale distributed application projects and system integration, including SOA, BPM, M2M and most recently IoT. He is representing Bosch at the Industrial Internet Consortium and is active in the Industry 4.0 community. He holds an MBA from IMD Lausanne as well as a Diploma Degree in Computer Science from TU Berlin. |