LII:Medical Device Software Development with Continuous Integration/Introduction

-----Return to the beginning of this guide-----

Definition

To start with, we need to define continuous integration. In August 2011, Wikipedia offered the following definition:

In software engineering, continuous integration (CI) implements continuous processes of applying quality control — small pieces of effort, applied frequently. Continuous integration aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development.

The term first emerged at the turn of the century out of Martin Fowler and Kent Beck's writings on the topic. Fowler's 2000 paper is an often-cited source of information on the subject. Beck's 1999 book Extreme Programming Explained, the original reference for Extreme Programming, also introduced the term.

The working definition given by Wikipedia above is good because it helps to eliminate the mistaken idea that continuous integration is a task within the larger picture (i.e., when people think of continuous integration, they tend to think of continuous integration builds and not the entire process). Continuous integration is a process that encompasses software development efforts and not simply the act of integrating and building code. This is very important to take note of!

Continuous integration on regulated projects

How can continuous integration be utilized in the tightly regulated world of software medical devices? Let me begin with a story.

When it comes to regulated environments, such as the regulation enforced by the FDA on medical device software, we tend to focus on standard operating procedures (SOPs). Often these SOPs become so rigid that they tie our hands, preventing agile (lowercase a) software implementation and introducing bureaucracy for bureaucracy's sake. Ultimately, those standard operating procedures that took so long to create and (attempt to) enforce become so cumbersome that all other work takes secondary attention to the almighty process. The good news is that someone was trying to think of the process (if there is any good news to be found). The bad news is that nobody (including the author) remembers exactly what those procedures say. This is where an intelligent continuous integration approach to all software development can be a silver bullet.

My first experience working on medical device software came in 2002. My previous employer, an exciting venture of the dot-com era, failed to provide me the millions of dollars that I expected. Still somewhat fresh out of college, I was hit by reality: those 100,000 shares of stock weren’t going to be worth anything, and I would continue driving a 1998 Dodge Stratus for a few more years.

Faced with my first humbling experience in the unemployment line, I was eager to accept the first job that came along, and soon enough I found myself working on a blood bank project performing software quality assurance. I’m not sure what was more humbling to an arrogant young software developer: the unemployment line or being forced to write test scripts. But it was a job, and being one of thousands of software developers looking for work, I took it.

On day one at my new job I learned something bizarre: the software we wrote would be audited and controlled by a set of standards defined by the FDA. "The FDA," I asked, "as in The Food and Drug Administration?" Yep. That FDA. I was told to sit in my cubicle and read something called the CFR, focusing specifically on parts 11 and 820. It was long and boring and strange sounding, and I was annoyed. I yawned incessantly as I forced my way through it.

My next task was to read a bunch of standard operating procedures (SOPs). Upon reading an SOP I took a test answering a few simple questions about it. It was graded and signed and placed in a permanent file somewhere as proof that I knew the SOP. At this point I promptly forgot what the SOP said.

My job was simple. I was to review use cases, write test scripts for those use cases (both manual and automated), and run the scripts. When I finished one task, I moved on to the next (that task being whatever my boss told me to work on). Soon enough the SOPs I had read during my first few weeks of employment were a distant memory. I had no reason to revisit them.

But perhaps I should have. Remember how I signed a test saying that I read and understood the SOP? About six months into my employment, it came time for an internal audit of the project. The purpose of this audit was to perform activities similar to those which might occur during an actual FDA audit for a 510k. I was pulled into a room with the auditors and interrogated.

The auditor asked me the first question: "So Matt, when you do your work, how do you know what to do?"

The question seemed odd, but I responded politely: "Uh, well, I guess I do whatever my boss tells me to do."

Wrong answer.

The auditor continued: "No, what I mean is, how do you know what you are supposed to create?"

I began to squirm a bit: "Um, because my boss tells me what to do… I read the use case and requirements, and I write the test."

This was not going well.

The bizarre questioning continued for an uncomfortably long time until finally the auditor gave up and told me that I could return to my cubicle. For a brief period of time, I was relieved. Then the project manager caught wind of my inarticulate question and answer session.

I was in trouble.

Continuous integration is the process

Are we talking about standard operating procedures or continuous integration? We're talking about both! Continuous integration is much more than simply contributing to a shared repository and ensuring software build integrity. Continuous integration is a process that leads to a cohesive team with logical procedures. Continuous integration means that no member of the team, from management to development, may work in a vacuum. The process by which high-quality software is engineered is a process that involves all roles, and it requires more than simple memorization of some procedures that were read once time. The procedures are an integral part of the daily work activities. More importantly, the procedures make sense.