Discussing Software Validation: Learning from Our Users

Cognition Corporation’s 2017 User Conference was a chance for current and prospective users to learn about how other organizations utilize the Cockpit™ platform. It was also a chance for Cognition to listen to existing customers and obtain feedback on the products we offer. We typically hold an Advisory Board before the opening of the conference to learn from our users, but this year we took it one step further with a discussion about our Validation Kit.

With Software-as-a-Service (SaaS) tools replacing traditional data management models, organizations are adjusting their focus. Now that more SaaS solutions handle important product development functions, there is an insistence on the FDA’s part to ensure that those solutions function in a consistent, reliable manner, so validation of these tools is a hot topic right now in the medical device world.

FDA and other regulatory bodies do not prescribe one particular software validation approach; rather, they issue guidance on what software validation should look like. FDA does recommend—in its General Principles of Software Validation; Final Guidance for Industry and FDA Staff—that any third-party SaaS solution should:

  • Utilize system requirements
  • Utilize a validation process
  • Utilize the results of validation provided by the third-party vendor/developer of the software as the starting point for required internal validation, including testing protocols and results

This is the base from which all product development teams work to determine how to validate a SaaS solution. To help clients leverage FDA’s third recommendation, Cognition has developed the Cockpit Validation Kit (CVK). Designed to baseline Cockpit in its out-of-the-box state, the CVK provides a framework to achieve FDA compliance, resulting in a comprehensive validation plan. Well-structured and modular by design, the Validation Kit provides robust, efficient, and maintainable validation exercises.

In our ongoing efforts to make the kit as efficient and user-oriented as possible, Cognition’s CEO, David Cronin, led a discussion during our 2017 User Conference about the CVK and what users are looking for in a viable validation kit for Cockpit.

What We Discussed

Within our customer base, there were major differences in approaching how validation should be executed and on what processes it should be done. While all agreed that validation efforts are necessary, there was no clear consensus as to what the CVK should be validating.

FDA vs. Business Requirements

One of the things many Cockpit users agreed on was that business requirements for validation are often stricter than what FDA requires. There are discrepancies between what an organization requires versus what the FDA is looking for. This comes down to the fact that validation exercises for third-party software are one of the better risk-aversion methods an organization can implement. Ensuring that the software is working as intended is paramount to the business’s bottom line; if a SaaS solution fails, it could mean anything from time lost in production to potential remediation woes. Product development teams need to be sure that the SaaS solution, at its base level, is functioning as intended and will meet business requirements.

What to Validate: Signatures or Functionality?

This was a hotly debated topic. Many organizations struggle with determining what should be validated within a SaaS solution: Should it be the core functionality and the automated processes, or electronic signatures on all reviewed data?

In discussion, this broke into three camps. There were those who believe that if the software behaves in a consistent manner and users can trust its results, then just signatures need to be validated. What drives this approach is the inherent trust users give Cockpit and software like it to perform consistently, which is ensured by Cognition’s software validation prior to release. As well, there is the driving force of business requirements and the need to watch for the bottom line in any validation activity; not worrying about validating core processes with each release saves time and money.

Conversely, there were those who pushed back on trusting the software due to the amount of data and the automatic calculations made within it. Likening Cockpit to a calculator, many remarked that trusting that it functioned correctly when signing off on its results was a potentially risky move. Validating accurate and reproducible results is as important as validating the appropriate reviewer’s signature.

The third opinion stemmed from both the first and second opinions. There was consensus around the idea that the basic functions of the software could be trusted to perform the necessary exercises consistently. Because Cockpit is validated prior to each release, users feel confident they can trust its performance, and therefore only signatures have to be validated. However, the problems come when a SaaS solution is configured. Comparing the issue to Excel, users agreed that the basic functions that Excel already has built in can be relied upon to produce a trusted result. If Excel is being used as software for an organization’s product development process, they don’t have to worry about validating the existing Excel functions. However, once custom functions are written or built on top of other ones, then there is a need to validate the new configuration and the reviewer’s signature.

There was no full consensus on what should be validated within a software solution. Yet most agreed that it is up to each organization to decide what is worth validating and what functionality they can trust.

The Takeaway

What we learned, coming out of our discussion about the CVK and validation efforts, is that there is a lot of gray area still around software validation and determining the best path forward. Each organization has its own approach to validating the software they use in product development, but everyone agrees that those efforts are necessary.

To maximize software validation, product development teams need tools and resources to ensure the baseline functionality of the software. They also need to confirm that those tools can conform to their business needs and deliver consistent results for all business requirements. To learn how the Cockpit Validation Kit meets these needs, read our Validation Kit White Paper.

About the Author

Nick Schofield is a content creator for Cognition Corporation. A graduate of the University of Massachusetts Lowell, he has written for newspapers, the IT industry, and cybersecurity firms. In his spare time, he is writing, hanging out with his girlfriend and his cats, or geeking out over craft beer. He can be reached at nick.schofield@cognition.us.