Video Transcript:
I’m Carlos Almeida, VP of Engineering at SPK and Associates. I’ve spent most of my professional life doing software engineering. 25 years plus in the world of high tech in the electronic design automation space, tools for chip designers. And in the last 10 years since joining SPK, I’ve been mostly working in regulated companies still driving all sorts of software projects, Development, quality process, all those things. I thought it would be fun to share some insights and lessons in my journey from non-regulated to regulated industries. My high tech world was all about speed. First to market wins the game. Innovation at a rapid pace. I worked a lot with chip manufacturers and fabs focusing on tools, thinking about design capture and layout, simulation software. In joining SPK over 10 years ago, I started working with companies that had extensive regulatory overlay. Everyone had SOPs, standards and rigorous procedures. Many of my customers shifted to medical device companies, life science partners and other regulated companies. So I thought it’d be fun to share three insights or best practices that I’ve learned in navigating this industry and of course let’s have some fun.
But before we get into the real lessons, I have to step back and say learn the language. I remember having meetings where folks would say stuff like “…after you’ve hit ALARP for your residual risk mitigations, please make sure you’ve updated your FMEA test protocols and generated a new traceability matrix. Thank you”. What’s ALARP? What’s a traceability matrix? ALARP is as low as reasonably possible but we’ll get into that. So lots of new acronyms. Tons of new processes it was alphabet soup at best. Klingon at worst.
So let’s set the stage. Each industry has its own set of standards and processes. These are comprehensive documents that cover all their activities. If you’re working in these verticals: aerospace, automotive, medical device; you’ll need to spend time to learn them. I think it took me about a year to get somewhat comfortable in my first vertical. The good news is that even though each industry does have differences, there are themes that appear.
Every industry has quality documents which define how requirements are captured architected and implemented. They also cover testing activities and producing data for auditability. They share an ISO 9000-like “say what you do do what you say” feel. There’s also tons of attention paid to risk management. It’s all about identifying the risks and harms putting in the proper mitigations. Lots of FMEAS and DFMEAS, more alphabet soup. And within the last decade or so, much more attention has been paid to security especially cybersecurity, if you have a connected product. There’s too much to learn but the good news is that once you’ve dug into one vertical the next one is a little easier to ramp up on.
Alright let’s dig in a little bit into one of these standards. The IE62304 standard is the bible for how to do software development in medical device companies. It has all the things you would expect to see. You know you have to have requirements management, have a defect tracking system, have a configuration management system. You know you have to do a lot of planning and detail how you’re going to design, implement, test and release your product. You know you need to fold in risk management and other processes. But to my surprise, there was nothing written about what kind of life cycle model you would have. No specific directive! You could have waterfall, iterative, Agile. It’s all up to you the most important thing at the end of the day is that you need to satisfy the quality management system – n this case FDA’s CFR 820 or Europe’s ISO 3485 – yes you need to do all of the risk management class 1, 2 and 3 work. And the biggie – all of my design inputs have to match my design outputs. But then again the life cycle model wasn’t mandated so why would I care? What I hear from many customers is that it takes too long to get my product out the door. This is common in regulated companies. “My customers are beating me to the punch”
So lesson one for today is do what the non-regulated world has been doing for years and adopt an Agile methodology into your development model. And as recently as 2012, the FDA did finally say it was okay to do so via TIR 45. So I learned that although Agile is commonplace in non-regulated industries is still really struggling to gain a foothold in the compliance world like medical device. Many companies I’ve encountered have maintained a very waterfall-feel to their development model. For them it’s comfortable. It’s tried and true. But you don’t have to settle for that. And if you really want to go faster you really need to consider doing agile. For me another notable benefit of Agile is about improving the quality of your product, which is the most important thing when dealing with safety critical systems. You have to be safe. You can’t have failures in the field and since with Agile you’re able to identify and correct issues early you’ll be releasing a safer product. You have more time to invest in quality. And finally, Agile forces you to check in with your key stakeholders throughout the entire process to make sure you’re still building the right product. It allows you to make course corrections to adapt to competitive changes. With waterfall you’re willing to dice. “Will another player beat me to the market? Will what I’m making now still be what the customer wants when it finally goes out the door?”
Now there’s tons of information about Agile online but just to give a very, very quick summary on some key concepts let’s go over a few of these. Agile values individuals and actions over processes and tools. So smaller cross-functional teams working together with less bureaucracy and organizational structures. Agile values a working product over comprehensive documentation. So product requirements are prioritized and documented into bite size increments. No exhaustive planning up front. These increments go through daily development and test iterations in short two to three week cycles called sprints. And the output of this is a working product containing just those features. The goal is to mature the product through a series of sprints rather than doing one big development cycle as seen in a waterfall model. Agile values customer collaboration over contract negotiation. This means getting customer feedback very early after a sprint on those limited features. The customer is a participant in the process making sure we’re building what they want as opposed to waiting for the product to be completely done which is often the case with waterfall. And finally, Agile values responding to change over following a plan. This is about being able to make course corrections during the development cycle. You can introduce new features or throw out features. It’s okay, if the customer changes his or her mind. It’s better to reset after a two to three week sprint than after a year-long product release cycle. So let’s see how we actually apply this in a medical device scenario. As mentioned before IE 62304 is the medical device standard for software development. It doesn’t specify what type of life cycle model to use. So back in 2012, another guidance document was created called Tier 45. It describes how Agile practices can be mapped back into 62304. At the highest level, your design inputs have to match your design outputs. This is wrapped around the planning activities from requirements, to design, to verification and system testing to release. Again most companies do this in a waterfall manner. Tier 45 breaks this further down into four activity buckets. The first one is your user story. This is your single feature. Your bug fix! You’re required to do what I call a mini but full v model on it. You update your requirements. Your designs you run your vmv including system testing and do risk analysis. Risk analysis is done throughout the entire process.
The next bucket is called an increment, which looks like the traditional sprint. It’s a collection of features. Typically two to four weeks. The next comes a release. Think of it as a major milestone or a collection of sprints which may be sent off for additional internal testing, internal customer feedback etc.
And lastly you have the project, which is your finished product ready to go out the door to customers.
So doing all this involves updating your documents with every iteration. They feed back into other systems like your QMS. It’s a crazy amount of work and difficult to do if you don’t have good tools in place to support you. So I’ll highlight a couple of “gotchas” here. If your documents are sitting in a file cabinet or loosely organized with Excel and Word, you’ll die you won’t be able to accurately and efficiently update them. It will not be easy to do Agile the solution to this problem is to use a good requirements management system.
A good requirements management system treats document content as database objects, with that you have the ability to create links and relationships between those objects. And that will give you the ability to set a baseline for the collection of requirements in your story to do impact risk analysis before making your changes. Do what if scenarios. “What if I make a change? What is going to be affected downstream. Control who can make the change. Establish roles and workflows with sign-offs and after the change is made you have complete traceability to demonstrate that your design inputs do in fact match your design outputs. Another common best practice is to leverage a good requirements management to help you with your risk analysis process. The same tool can help document your functional analysis sourcing of your hazards and risks, capturing your control measures and mitigations and tracing them down to your verification and validation test cases. It can also dynamically generate for you your FMEAs and DFMEA-style reports. And another key item is this one in the center. This one labeled CI for continuous integration. This is your development engine room. It needs to be highly efficient and automated in order for all this to work. So what we’re left with is a series of iterations, each adding features, each going through all the document updates risk analysis and testing. And at the end ensuring all your design inputs are traced to your design outputs and ultimately updating all your artifacts as part of your design history file( DHF). I call these running your mini v’s. So a common question is “what if I have both software and hardware in my product? Can I still do Agile?” The answer is yes, but it’s going to be a little bit different. Hardware development requires longer lead times. A hardware build can take months. So sometimes hardware teams utilize what is called an adaptive approach, which is somewhere in between waterfall and Agile. They’ll create these longer “uber sprints” and plan to use them as alignment points with the software team. Where possible, the software folks in between these uber sprints will try to focus their smaller sprints on changes that can be tested on their existing revision of the hardware. So they re-prioritize their backlog. In parallel to that, hardware engineers can also try to cut down their long lead times by using quick turn boards, programmable logic devices, field programmable gate arrays and quick prototype units etc to play with. And also really cool and becoming more common is the use of 3D printing for quick prototypes. And of course, there are companies that create software modeling and simulation tools for prototyping. It’s not perfect and every company is different but there’s always a way to move groups toward agile or something between called adaptive.
Another common business challenge that I hear from my customers is “my development environment is too slow for Agile.” So how do we get around that? You have to optimize it piece by piece. In order to be able to run those TIR sprints with those mini V’s, you need to have it running like a well-tuned race car. There are several things you can do to get there. Here are a few of my recommendations – my top six
You need to remove every manual step possible. Anything not automated can introduce errors to slow you down. The tools you use must work together well. If they don’t you’ll add more delays plus get unhappy engineers. The CI (continuous integration) which is the engine of your race car, has to be optimized for daily and multiple per day automated builds and test runs. It has to be fast. The faster the better. And as mentioned before you want to drive that database driven requirements management system, which is also connected to your risk management system. And you’ll want to be able to string together more complex workflows, which may include additional testing, provisioning, and releasing in your product. That’s called pipelining
And lastly as a reminder you can’t just grab any tool and throw it into your development environment. The FDA requires you to validate your tools for its “intended use” with protocols and objective evidence that’s called a CSV (computer system validation). We do a whole lot of those at SPK.
So let’s look at an implementation example from one of our customers who’s doing Agile in their environment. We start with a requirements management system in PLM. In this case we’re leveraging PTC requirements validation and source platform. also known as Integrity (now renamed to PTC Windchill RV&S) and PLM Windchill. This is our source of truth for documents and our risk analysis hub and home for our DHF files. We use Jira to drive our Agile planning, backlog and sprints as well as our defect tracking. And the developers work in their IDE’s doing code reviews with Crucible and configuration management with Git.
And then we use Cloudbees CD to execute a workflow pipeline. This is to run additional testing through dev QA stage and prod environments.
When digging into our Cloudbees CD workflow we can start our pipeline. The pipeline is organized into four buckets: Dev (Development), QA (Quality Assurance), Staging and Prod (Production). The idea is that each bucket will go through a list of activities in any order you want as part of the workflow. What’s cool is you can put just about anything into those pipelines. You can image devices, add more tests, add packaging, spin off custom builds, and performance runs. You can establish pass/fail criteria at any point. You can require formal signatures between activities if you want. And of course, throughout your entire process you should be getting metrics on your code, your tests and all of your activities. My last business challenge is something I hear customers say often. “Making sure you’ve covered all your design inputs and design outputs is too hard and takes too long”. I’ve seen companies take weeks massaging a document especially if they have an audit coming up. The solution for that is to create an automated traceability report – one with customizable formatting. Often when I start a project with a new company the compliance and QA folks approach me with what they’re currently using for their traceability report. It’s usually an Excel spreadsheet that has been highly customized. This document is very, very important to them. They’ve used it for years. They have passed audits with it. It’s the Torah of Excel docs. Then they say to me “please make this document. Please make this document out of your new system or we may not be able to pass go. Yes there may be some flexibility later on but you need to show us this first.” So after many repeats of this scenario with different companies we (SPK) decided to create a customizable engine.
Presenting the SPK Report Engine. It was developed as an add-on to the PTC Windchill RV&S system aka PTC Integrity. It was also part of the sample customer infrastructure implementation that I showed you earlier. It provides end-to-end, with formatting, customization it’s a fully supported product which we have delivered to many of our medical device accounts. Select click and done! Happy customers!
And that’s it I’ve covered a lot of ground really quickly. If you want any more information on anything discussed some help with your infrastructure setting up Agile or automated trace reporting, please drop us a line we’d love to hear from you. And again thank you so much for sharing your time with us today.
Thank you.
