LII:Medical Device Software Development with Continuous Integration/Validation

-----Return to the beginning of this guide-----

Overview of unit tests

In computer programming, unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual function or procedure. In object-oriented programming a unit is usually a method. Unit tests are created by programmers or occasionally by white box testers during the development process. In the world of Java, we have a number of popular options for the implementation of unit tests, with JUnit and TestNG being, arguably, the most popular. Examples provided in this article will use TestNG syntax and annotations.

Traditionally (and by traditionally, I mean in their relatively brief history), unit tests have been thought of as very simple tests to validate basic inputs and outputs of a software method. While this can be true, and such simple tests can serve of some amount of value, it is possible to achieve much more with unit tests. In fact, it is not only possible but recommended that we implement much of our user acceptance, functional, and possibly even some non-functional tests within a unit test framework.

To further enhance quality, we can augment the acceptance with unit tests. While I personally have never been a fan of test-driven development (I feel that the assumptions required by test-driven development do not allow for a true iterative approach), I do believe that creation of unit tests in parallel with development leads to much more quality software. In the world of Agile, this means that no functional requirement (or user story) is considered fully implemented without a corresponding unit test. This strict view of unit tests may be a bit extreme, but it is not without merit.

The first unit test a developer may ever write is likely so simple that it's nearly useless. It may go something like this.

Given a method:

public int doSomething (int a, int b) {    …    return c;}

A simple unit test may look something like this:

public class MyUnitTests {    @Test    public void testDoSomething() {        assertEquals(doSomething(1, 2), expectedResult);    }}

Given a very simple method the developer is able to assert that, essentially, a + b = c. This is easy to write, and there is little overheard involved, but it really isn’t a very useful unit test.

Early attempts to automate functional testing

Long ago I was involved with a project in which management invested a significant amount of time and training in an attempt to implement automated testing. The chosen tool was Rational Robot (now an IBM product). The idea behind tools such as Robot was that a test creator could record test macros, note points of verification, and replay the macros later with test results noted. Tools such as Rational Robot and WinRunner attempted to replace the human tester with record scripts. These automated scripts could be written using a scripting language or, more commonly, by recording mouse movements, clicks, and keyboard actions. In this regard, these tools of test automation allowed black-box testing through a user interface.

In this over-simplified view of automated testing, there were simply too many logistical problems with test implementation to make them practical. Any minor changes to the user interface would result in a broken test script. Those responsible for maintaining these automated scripts often found themselves spending more time maintaining the tests than using them for actual application testing.

Rational Robot and tools like it are alive and well, but I refer to them in the past tense because such tools, in my experience, have proven themselves to be a failure. I say this because I have personally spent significant amounts of time creating automated scripts in such tools, and I have been frustrated to learn later that they would not be used because of the substantial amount of interface code that changes as a project progresses. Such changes are absolutely expected, and yet, a recorded automated test does not lend itself well to an iterative development environment or an ongoing project.

Automating functional tests using unit test framework

Most software projects, especially in any kind of Agile environment, undergo frequent changes and refactoring. If the traditional single-flow waterfall model worked, recorded test scripts such as those noted previously would probably work just fine as well, albeit with little benefit.

But it should be well known by know that the traditional single-flow waterfall model has failed, and we live in an iterative/Agile world. As such, our automated tests must be equally equipped for ongoing change. And because the functional unit tests are closely related to requirements at both a white-box and black-box level, developers, not testers, have an integral role in the creation of automated tests.

To achieve this level of unit testing, a test framework must be in place. This requires a bit of up-front effort, and the details of creating such a framework go well beyond the scope of this article. Additionally, the needs of a test framework will vary depending on the project.

Test fixtures become an important part of complex functional unit testing. A test fixture is a class that incorporates all of the setup necessary for running such unit tests. It provides methods that can create common objects (for example, test servers and mock interfaces). The details included in a test fixture are specific to each project, but some common methods include test setup, simulation, and mock object creation and destruction, as well as declaration of any common functionality to be used across unit tests. To provide further detail on test fixture creation would require much more detail than can be provided here.

Given what may seem like extreme overhead in the creation of complex unit tests, we may begin to question the value. There is, no doubt, a significant up-front cost to the creation of a versatile and useful unit test framework (including a test fixture, which includes all the necessary objects and setup needed to simulate a running environment for the sake of testing). And given the fact that manual function and user acceptance testing remains a project necessity, it seems like there may be an overlap of effort.

But this is not the case.

With a little up-front creation of a solid unit test framework, we can make efforts to create unit tests simple. We can even go as far as requiring a unit test for any functional requirement implementation prior to allowing that requirement (or ticket) to be considered complete. Furthermore, as we discover potential functionality problems, we have the opportunity to introduce a new test right then and there! The hardware system, software program, and general quality assurance system controls discussed below are essential in the automated manufacture of medical devices. The systematic validation of software and associated equipment will assure compliance with the QS regulation and reduce confusion, increase employee morale, reduce costs, and improve quality. Further, proper validation will smooth the integration of automated production and quality assurance equipment into manufacturing operations. Medical devices and the manufacturing processes used to produce them vary from the simple to the very complex. Thus, the QS regulation needs to be and is a flexible quality system. This flexibility is increasingly valuable as more device manufacturers move to automated production, test/inspection, and record-keeping systems.

What is a good unit test?

In his book Safe and Sound Software, Thomas H. Farris describes the unit test as such:

Software testing may occur on software modules or units as they are completed. Unit testing is effective for testing units as they are completed, when other units are components have not yet been completed. Testing still remains to be completed to ensure that the application will work as intended when all software units or components are executed together.

This is a start, but unit tests can achieve so much more! Farris goes on to describe a number of different categories of software:

• Black box test
• Unit test
• Integration test
• System test
• Load test
• Regression test
• Requirements-based test
• Code-based test
• Risk-based test
• Clinical test

Traditionally this may be considered a fair list. Used wisely, and with the proper frame work, however, we can perform black box, integration, system, load, regression, requirements, code-based, risk-based, and clinical tests with efficient unit tests that simulate a true production environment. The purpose of this article is not to go into the technical details of how (to explain unit test frameworks, fixtures, mock objects and simulations would require much more space). Rather, I simply want to point out the benefits that result. To achieve these benefits, your software team will need to develop a deep understand of unit tests. It will take some time, but it will be time very well spent.

It’s a good idea to have unit tests that go above and beyond what we traditionally think of as unit tests, and go several steps further, automating functional testing. This is another one of those areas where team members often (incorrectly) feel that there is not sufficient time to do all the work. As Harris goes on to state:

Software testing and defect resolution are very time-consuming, often draining more than one-half of all effort undertaken by a software organization ... Testing need not wait until the entire product is completed; iteratively designed and developed code may be tested as each iteration of code is completed. Prior to beginning of verification or validation, the project plan or other test plan document should discuss the overall strategy, including types of tests to be performed, specific functional tests to be performed, and a designation of test objectives to determine when the product is sufficiently prepared for release and distribution.

Harris is touching on something that is very important in our FDA-regulated environment, and this is the fact that we must document and describe our tests. For our unit tests to be useful we must provide documentation of what each test does (that is, what specifically it is testing) and what the results are. The beauty of unit tests and the tools available (incorporation into our continuous integration environment) is that this process is streamlined in a way that makes the traceability and re-creation of test conditions required for our 510(k) extremely easy!