Automated Embedded Testing with Robot Framework and Tinkerforge

Introduction

In software engineering, testing is an essential phase for the development of a product, necessary to ensure quality, in terms of robustness to failures and reliability. Embedded systems are no exception, but testing their firmware is particularly tricky because software and hardware are tightly coupled, hence these two components have to be taken into account as a whole by test cases. This poses some extra challenges compared to other software engineering areas, but it is anyway possible to run many kinds of tests, even in an automated way (mostly with the aid of dedicated software frameworks) and include them in the product delivery pipeline. The goal of this article is showing an example of how this can be achieved, presenting the solution delivered by Concept Reply to a customer to ensure that their device firmware gets extensively tested upon each new release.

Testing embedded systems

As mentioned, most of the general principles of software engineering can be applied to embedded systems too.
When it comes to defining the scope of tests, it is indeed possible to identify the same levels as for the testing of any other piece of software. Moving from the lowest level to the highest one, namely from the smallest units of the system up to the system itself, the following type of tests are performed: unit tests, integration tests, system tests, acceptance tests and regression tests.

Not just the levels of testing, but also the testing methods applied to embedded systems are valid for most of the sectors in the software/hardware industry. Methods vary in terms of the point of view from which the tested object is observed, and they essentially resemble to two main approaches: black box testing and white box testing. Tests following the former methodology are meant to verify that the output of the product is the expected one in all foreseeable circumstances, i.e. for the highest possible number of combinations of inputs; such types of tests are also referred to as functional tests. The diametrically opposite approach consists in considering only the internal mechanisms of the system and verifying their correct behaviour in the greatest possible number of conditions, irrespective of the status of the surrounding environment. Because the ultimate goal is to cover the biggest possible portion of lines of code, tests in this category are also known as coverage tests.

The intrinsic nature of embedded systems though, with a tight bound between software and hardware, makes testing them trickier compared to other engineering domains.

For example, it is often the case that not just the firmware, but also the hardware must be created from scratch to fulfil the requirements of a product. This means that some of the software tests must be postponed until the physical components are available, thus increasing the risk that major problems affecting the interaction between software and hardware are discovered only at a very late stage of development. That is also the reason why integration tests in the embedded field usually include an additional specific step where the interface between firmware and physical components is assessed.
Also, many times the embedded software is used for safety critical applications, where human lives are in danger in case of any malfunctioning, so tests must be rigorous in order to minimize the risk for hazards.

Since most of the devices are supposed to operate without interruptions for long periods of time, sometimes in areas where they are hardly accessible, fixing any kind of error after their deployment can become very expensive. Reducing the chances of such situations requires the testing of these products to be as meticulous as possible, and hence more challenging.

Being the hardware resources normally very limited on embedded systems, functional tests must consider performances too, ensuring that they always remain above a minimum threshold. Such tests must verify that the device is robust against temporary or prolonged stress conditions, caused for example by unexpected fluctuations of the input signals which end up overloading the channels or saturating the memory buffers. These tests should also help to spot the portions of firmware code that can be improved or optimized, thus offering a better way of coping with insufficient system performances than simply replacing existing hardware components with more powerful ones.

The last major aspect which makes testing them unique, is that most embedded devices are real- time systems, so tests must additionally verify that the tight time constraints associated with this property are respected. In such cases, integration tests cannot be limited to software and hardware, it must rather involve the real time operating system (RTOS) too. To this aim it is quite common to support testing with simulators, which reproduce critical environmental conditions and allow to check whether the system always operates within the real-time boundaries. [1], [2],

[3], [4].

Testing automation

Given the aforementioned importance of thoroughly verifying the correct behaviour of embedded products before their deployment, it does not come as a surprise that it is nowadays regarded as a best practice in the whole software industry to automate the testing phase. As a matter of fact, all the different ways of implementing such automation offer the same advantages over manual testing:

• integration with Continuous Integration (CI) and Delivery (CD): before new code is added to the code base and then released, it must be tested in depth, so automating this step is of paramount importance to perform the code integration and delivery procedures upon each new commit
• shorter duration: automatically running tests is faster than performing them manually one by one, so efficiency increases and the time to market is reduced
• cheaper: removing the need for human intervention also reduces the number of required testers and makes testing overall less expensive; moreover, chances are greater compared to manual tests that bugs are detected at an early stage and not after a product has already been released, making the fix less expensive
• less tedious: many test cases imply repetitive tasks, which would make testing boring, had they to be executed by a person
• less error-prone: limiting the interaction with testers contributes to reducing the chances of mistakes, augmenting reliability and accuracy
• wider code coverage: especially white box testing benefits from the implementation of automation, because it allows to expand the number and the complexity of tests carried out and thus to inspect larger portions of code

Automating a test means creating and running a testing tool, a piece of software or a script which carries out one after the other all the single steps included in the manual test. Even though it would be possible to write this tool from scratch, it is more convenient to base the implementation on an automation framework. Such frameworks provide the developers with basic objects to build the tests upon (often in the form of a software library, with APIs for multiple programming languages) as well as an execution environment to run the test cases. Moreover, they include functions to automatically inject data into the simulations and, last but not least, produce execution reports. An automation framework helps to augment the modularity of the test harness, making test cases more easily maintainable and re-usable, since they are all built upon some common fundamental elements offered by the framework itself. The adoption of a framework to automate tests may result in a more expensive development process at first, but this is compensated by reduced maintenance costs on the long-term, so it is overall advisable to rely on automated framework for testing [5], [6], [7], [8], [9].

Robot Framework

Many automation frameworks are available on the market and the solution presented in this article was based on one of the most famous: Robot Framework. It belongs to the category of the so-called keyword-driven frameworks: test cases are expressed as a composition of human-readable keywords, each abstracting one of the actions which are performed during a test to verify that the device, the so-called system under test (SUT), behaves as expected. The usage of keywords gives such frameworks the great advantage of being easily accessible even to testers without programming skills. Indeed, because of the strong distinction between the definition of test cases and their implementation, testers can just focus on keywords and ignore the underlying technical details. Another useful feature of keywords is that they can be re-used across multiple tests, which speeds up the setup of the testing phase, above all in such scenarios where many test cases are similar to one another and differ by few steps only. Nevertheless, it must be said that even for not so complicated SUTs the number of keywords can grow quite quickly, making it not so obvious for testers to fully grasp the test actions associated with each of them.

What follows is a closer look at Robot Framework, starting from the strong points that motivated the application in the project described below:

• it’s a free of use, open source library, released under the Apache License 2.0
• test script functions are abstracted through multiple levels of keywords, where lower-level keywords can be combined into higher-level ones to ease the creation of a large number of test cases
• automatic reporting (in HTML format) at the end of tests
• complete design of the testing tool with the creation of new test libraries, beside the built-in libraries ready available in the framework
• along with keyword-driven one, offers data-driven and behaviour-driven testing styles
• suitable for acceptance testing, acceptance test driven development (ATDD), behaviour driven development (BDD) and robotic process automation (RPA)

The following image illustrates the architecture of the framework:

Clearly emerging from this picture is the modularity of the framework, resulting in complete decoupling of all the components, with a clear separation of the core of the framework from the test data and the test libraries. This adds flexibility to Robot Framework and makes it suitable for testing many different categories of systems, from web sites to mobile applications, obviously including embedded systems.

Next is a description of the main blocks in the diagram:

• Test Data: provided using a syntax in tabular fashion, they include the basic settings of the tests, like the connection to external data sources (resources and variables files), the test cases themselves, and possibly user keywords (see below). Most importantly, they contain the list of the Test Libraries (see below) to be imported. In Robot Framework, tests are identified by a name and contain a list of keywords, either imported from test libraries or defined in the test data itself. An interesting feature of the framework is the possibility of specifying a Test setup and a Test teardown, namely a list of actions (formulated again as keywords) to be performed respectively before and after the test. They are very useful to define setups which are common to different tests, as well as clean-up operations to be carried out as final step of the tests, possibly to handle failure conditions. The framework also facilitates the organization of tests through Test suites, namely lists of tests to be executed one after the other in a certain order.

• Robot Framework: tests are started providing the path to the corresponding test data, and the core of the framework is in charge of running them until full completion, unless an error occurs. In both cases, some output files reporting the details of the execution and the results of the tests are generated. The framework is written in Python, so the associated runtime environment (or its implementation in another programming language, like Jython for Java, or IronPython for .NET) is required.
• Test Libraries: these form an abstraction layer for the framework to be agnostic about the underlying SUT, making it suitable for the testing of practically any type of software product. The core already comprises some libraries, but in most of the cases new ones have to be created, tailored to the particular test cases and devices. Robot Framework conveniently can adapt to test libraries written using any programming language, besides of course natively being compatible with Python [10], [11], [12], [13], [14], [15].

Hardware In the Loop: Tinkerforge

The approach followed in the project for testing the firmware resembles a quite common one used for the validation of embedded systems, referred to as Hardware In the Loop (HIL). It is based on a hardware stand (or bench) featuring sensors and actuators which get connected respectively to the inputs and outputs of the device to be tested. A piece of software, the HIL simulator, executed on the stand, generates the signals to be sent to the system under test and monitors the control commands reported by its actuators. Thanks to the HIL bench it is possible to automatically verify the functioning of the whole system in a simulated, yet realistic, environment, thus providing a highly accurate proof of whether the product behaves as expected in all conditions. As a matter of fact, HIL testing allows to simulate corner cases and extreme environmental conditions too, including safety critical ones, which would otherwise be left untested because the life of the tester would be put in danger.

For the hereafter described setup, the test stand was assembled connecting several building blocks from the Tinkerforge construction set. This is a collection of hardware modules, each providing a specific minimal functionality for the simulations of the I/Os of an embedded system.

In particular, two types of modules are available:

• Bricks: for any Tinkerforge system it is mandatory to include at least one of these blocks, because they carry out fundamental tasks like sensing physical properties or controlling servo engines. The communication with the HIL simulator is granted by the master bricks, by default connected via USB-C (although serial, WLAN and Ethernet interfaces can be added through master extensions too). The master brick routes the commands from the simulation software to other bricks (master bricks can also be stacked) and/or to the bricklets attached to them.

• Bricklets: these elements extend the sensing and controlling capabilities of the bricks to which they are connected. Examples of such pieces of hardware features light sensors to detect colours or relays to implement switches or to measure voltage and electric current

The HIL simulator in the presented project controls the test stand via the Python API (bindings are available for most common programming languages), allowing a seamless integration with Robot Framework [16].

HIL testing solution architecture

Before delving into the details of the tool created for testing new firmware releases, it is worth first giving a quick overview of the customer’s product for which the software was commissioned. As visible in the figure, the device is equipped with sensors to measure temperature, pressure and water flow. It also features an actuator by means of which a valve can be closed or opened. Upon the initial commissioning, the device is supposed to be coupled with the user’s domestic WIFI so as to enable the communication with cloud services, which in turn allows to monitor, configure and send commands to the device itself with the use of a proprietary mobile application. The interface to the user is represented on one side by two hard buttons: one for starting the integration with an access point and one for directly issuing a valve opening or closure. On the other side, the product communicates its status by means of the visual feedback offered by LEDs and a buzzer (active only in alert conditions).

The second figure shows the setup of the HIL test stand, that leverages the Robot Framework and Tinkerforge to automate the testing of the firmware of the customer’s product. The comparison of the two images clarifies how the automation of the embedded testing is realised:

• the user intervention is no longer required because it is simulated by the HIL simulator, represented in this case by the Tinkerforge master bricks and the connected bricklets
• the simulator also replicates the electrical signals generated by the environmental physical inputs into the sensors of the device
• the Human Machine Interface (HMI) is monitored to check the feedback given to the user by the device

The HIL test bench is accessible and controllable through a Raspberry Pi Model 4B+, where the testing tool software runs, and to which is connected via USB cable. The choice of such kind of host is justified by the fact that it offers many communication interfaces, including a Wi-Fi adapter (mandatory for the tests involving cloud functionalities), sufficient processing power and extensive documentation for a relatively cheap price. It only takes to setup an SSH server to add the possibility of even logging into the machine, starting and monitoring the tests remotely, which might be particularly convenient for time-consuming tests.

Testing tool

The testing tool developed for the customer is meant to automatically execute system, acceptance and regression tests upon each new firmware release. Applying a black box methodology, the goal is to ensure that the final product meets the user requirements and that new code changes do not break existing functionalities.

The software of the tool is built upon the Robot Framework, that is why in the picture below, showing the architectural organization, it is possible to find the already mentioned elements which characterize any solution for automated testing which adopts this framework.

• the Test Data are represented by a collection of Test Suites, text files with the .robot extension where a number of Test Cases are defined, together with a few user keywords, defined combining multiple keywords from the test libraries. Some suites are meant to check the behaviour of the device, while others provide an interface to the device and allow the testing phase to be automated. For instance, some tests automatically switch the SUT on and off, simulate a button pressed by the user, flash the firmware and integrate the device with the user’s access point. A few test suites also contain the definition of some variables to specify a particular configuration of the device and are passed as arguments to the library keywords to tune the tests accordingly
• nearly all keywords listed in the Test Cases are implemented in a Test Library and imported in (the Settings section of) the Test Data. The Robot Framework core connects the test library with the test data (ensuring the already mentioned decoupling of the tests description from their implementation), allowing the tests to run when the tester starts them. The library comprises many Python scripts whose scope ranges from the library keywords (the highest level) to the interaction with the SUT (the lowest level)
• the interface between the test library and the device is provided by some modules wrapped around the Python API of the Tinkerforge platform. Under the hood, the function calls are redirected via a local TCP/IP socket to a Linux daemon, running on the Raspberry Pi, which forwards them back and forth to the Master Brick of the construction set. Besides this daemon, the test tools also include an ST-Link flasher for the deployment of the firmware to the PCB, the AWS SDK for Python (Boto3) for the triggering of Lambdas when testing the Firmware Over The Air (FOTA) updates and, more in general, for the handling of the cloud services. Finally, the Pyhton library Pyserial is used to monitor the internal serial communication between the chips inside the device.

Last, but not least, the presented software solution features a logging system, built on top of the Robot Framework. At the end of the execution of each test case, two HTML files are automatically created:

• a log, reporting details about the single test steps
• a report, with statistics about the overall test execution

The framework offers another useful feature related to reporting, namely the automatic generation of a high-level HTML documentation of the test cases (with a tool called Testdoc) and of the libraries (with another built-in tool named Libdoc).

Examples of test cases automation

This section presents some paradigmatic examples of automation of test cases from the tool delivered to the customer: considering that all the actions included in each of tests can be executed with one simple command, even remotely, represents a strong proof of the benefits associated with testing automation for embedded systems.

• Flash and integrate. The goal of this test is to verify that the device can be successfully “integrated” , meaning that it is possible to associate it with the user’s domestic AP , a mandatory step of the initial product setup, necessary to establish a communication with the cloud services and enable the control of the device via smartphone.
  — at the beginning, both the appliance and the communication module of the board are flashed with the desired firmware, using the STM-32 and ESP-IDF command-line flashing tools
  — the successive two steps require the user to press the two buttons present: first to remove all the information from preceding integrations from the storage and then to bring the device into the so-called “setup mode”. Thanks to the relays bricklets, these events are triggered without any human intervention. Once in setup mode, the device starts its own access point and a starts an HTTP server, to which the testing tool can connect to set the SSID and password of the user’s domestic AP (pre-configured in settings of the test suite)
• Trigger FOTA update. This test verifies the correct behaviour of one crucial feature of the product, namely the Firmware Over the Air Update (FOTA Update), by which the control software can be updated, to solve bugs or introduce enhancements, without having physical access to the device, rather remotely. Except for one button press, all the steps do not involve the user, so the automatization of this test is achieved by simply triggering the AWS Lambdas which start and monitor the FOTA update.
• Interruption and re-establishment of Wi-Fi connection. Given the importance of the cloud services for the proper usage of the product, this test verifies that the user is promptly notified as soon as Internet connectivity is missing.
  — a few seconds after the device has completed the boot procedure, the UART line is monitored to verify that the cloud servers are reachable, in particular by checking that the corresponding message reporting the connectivity status is sent by communication module to the appliance
  — the status of the LEDs is assessed too, to verify that the expected optical feedback is provided to the user
  — a mandatory precondition for this test is that the product is connected to a virtual access point created by the Raspberry Pi, so that it is possible to simulate a Wi-Fi disconnection during the test. Again, by means of the Tinkerforge API, it is verified that the colour of the LEDs and the frequency of their blinking are the expected one in case of an issue, to warn the user
  — the last step consists in re-enabling the access point and verifying that the device is capable of reconnecting to the Internet. This is done again by monitoring the UART frames transmitted by the communication module and ensuring that the visual feedback to the user does no longer indicate an error condition
• Immediate shut off. The last test worth mentioning is a fundamental one, because it inspects a safety property of the product, namely the automatic closure of the valve in presence of alarms, aimed at preventing dangerous consequences. This is also a significant demonstration of one of the great advantages associated with HIL-testing, i.e. the possibility of accurately verifying the behaviour of the SUT in unsafe circumstances, without having to really reproduce such dangerous conditions. In this specific case, high levels of flow and pressure can be simulated, and it can be checked that the product adopts the expected safety measures.
  — first, it is verified that the valve gets closed by measuring, thanks to the Tinkerforge digital counter, the PWM signal from the actuator controlling the valve. It is also ensured that a UART system state message with the bit corresponding to an error condition set to 1 is sent by the appliance
  — the test checks that the user is informed about the serious condition by turning on the LEDs and starting an acoustic alarm, the latter being inspected again analysing the PWM signal driving the buzzer

Conclusions

The choice of basing the software to control the HIL test bench on the Robot Framework revealed itself to overall be convenient, for several reasons:

• to begin with, because of the possibility of implementing the test library with various layers of abstraction on top of each other, resulting in very easily readable test-cases with high- level keywords only. The reusability of such keywords contributes to making the formulation of test cases quite simple
• the total independence of the test data from the test library helps to keep the HIL software neat and tidy
• since the test library is ultimately a collection of Python scripts, tests are more easily extendible, readable and maintainable than if they had to be expressed using other programming languages, like C or Java
• the automatic generation of detailed reports is of great help for the documentation of the tests results
• the framework is open source, free of charge and has good community-based support

Beside these positive aspects, it must be said that the keyword driven approach can bring some disadvantages too:

• trying to parametrize the keywords for coding re-usage might not be that straightforward and, on the other hand, having too specific keywords can lead to code duplication and to a significantly growing number of functions to maintain
• the maintenance of keywords for the testing tool in large projects is challenging, because the adaptation of many keywords is required whenever a function of the test library is modified
• although in theory it should be possible even for roles without a technical background to understand and perform the tests, in practice this is only the case when the functions mapped to the keywords are simple. In more complex projects, instead, single keywords tend to be associated with many test steps, because otherwise too many keywords would be needed, and their maintenance would be too demanding. Therefore, the more actions are included in a keyword, the harder it gets to clearly express them with simple words, so testers eventually have to trace the high-level keywords back to their lower-level testing functions to properly use them. Of course, this calls for technical knowledge though and it eliminates the benefits of abstracting scripts with the keywords themselves

Moreover, the creation of the entire test library from scratch inevitably requires some additional effort not just for the initial implementation, but also and above all for the maintenance and bug-fixing in the long term.

Having made use of the bricks and bricklets drastically simplified the design of the HIL simulator, thanks to the wide selection of hardware components, ease of use of the API and clarity of the documentation of the construction set. Though Tinkerforge definitely fit this particular project, this might not be absolutely valid in cases where a high number of hardware components are involved, because costs would rise quickly, or applications are particularly demanding, given the limited computational power of some bricks [17].
Despite of the usual difficulties in testing embedded systems, it can be stated in conclusion that the testing tool presented in this article helped to reduce the duration of the testing phase, removing manual and repetitive tasks. At the same time, accuracy and efficiency were increased thanks to the automation of some procedures through software (Robot Framework) and hardware (Tinkerforge) support, which allowed us to monitor and take into account more variables and system conditions. In the end, this translated to costs-saving and enhanced reliability of the testing phase for the customer.

References

[1] Think Tank QA , “What is Embedded Testing? Types and Challenges”,

https://www.thinktankqa.com/blog/what-is-embedded-testing/

[2} Niedźwiedź , Krzysztof, “Common Challenges in Embedded Software Testing and How to

Overcome Them”, 2024, https://intechhouse.com/blog/common-challenges-in-embedded-

software-testing-and-how-to-overcome-them

[3] Lepistö, Juho, “Embedded Software Testing Methods”, 2012,

https://core.ac.uk/download/pdf/38063214.pdf

[4] A. Berger, “Embedded Software Testing Basics”, 2011, https://www.embedded.com/the-

basics-of-embedded-software-testing-part-1

[5] Nagabushanam, Durga & Dharinya, Sree & Vijayasree, Dasari & Sai Roopa, Nadendla & Arun,

Anugu. (2022). “A Review on the Process of Automated Software Testing”.

10.48550/arXiv.2209.03069.

[6] Ali, Hussein & Hamza, Mahmood & Rashid, Tarik. (2023). “A Comprehensive Study on

Automated Testing with The Software Lifecycle”. The Journal of Duhok University. 26. 613–620.

10.26682/csjuod.2023.26.2.55.

[7] Martinez, Dennis, “5 Benefits of Automated Testing and Why It Is Important”, 2024,

https://www.telerik.com/blogs/5-benefits-automated-testing-why-important

[8] Umar, Mubarak Albarka & Chen, Zhanfang. (2019). “A Study of Automated Software Testing:

Automation Tools and Frameworks”. 8. 217–225. 10.5281/zenodo.3924795.

[9] Wu, Dante, “Automated Software Testing: A Comprehensive Guide for Software Testers “,

https://www.headspin.io/blog/what-is-test-automation-a-comprehensive-guide-on-automated-

testing[10] Figueira, Paulo Jorge Sena, “Test Automation Framework for Embedded Systems”, 2018,

https://run.unl.pt/handle/10362/108739

[11] Robot Framework Foundation, “Robot Framework User Guide”,

https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html

[12] Jaspers, Thomas, “Robot Framework Tutorial — Overview”, 2012,

https://www.codecentric.de/wissens-hub/blog/robot-framework-tutorial-overview

[13] Jaspers, Thomas, “Robot Framework Tutorial — Closing the Gap”, 2012,

https://www.codecentric.de/wissens-hub/blog/robot-framework-closing-the-gap

[14] Imbus, “Testautomatisierung mit Robot Framework“,

https://www.imbus.de/softwaretest/testautomatisierung-mit-robot-framework

[15] ELSYS Design, “Python for embedded systems testing”, https://www.elsys-

design.com/en/python-embedded-systems-testing/#robot_framework

[16] Tinkerforge, “Documentation”, https://www.tinkerforge.com/en/doc/

[17] Sven Ruppert, “IoT with TinkerForge and Java”, 09/04/2024,

https://svenruppert.com/category/java/tinkerforge/