Transforming your audit program with advanced analytics and AI

I recently spoke at the Pharmaceutical Compliance Congress with Fabien Saint-Gerard, the head of internal audit R&D, strategy and growth at Novartis. We discussed how R&D audit teams can get started with advanced analytics and AI, and covered its capabilities, benefits, challenges and more. Please see these condensed highlights of our conversation. 

Michael Shaw: To start us off, can you summarize how you’re using data and advanced analytics to improve the audit process at Novartis?    

Fabien Saint-Gerard: Sure, thanks Michael. We’re using thousands of real-time risk reports from clinical teams across all levels—global, regional and so on—to develop data analyses for leadership. This data enables them to, for example, identify specific issues like patient drop-off. Our data-driven approach removes bias, and I believe it allows for a more effective response to potential risks.

MS: What are the first steps a company should pursue if it wants to use advanced analytics and AI in R&D?

FSG: You’ll first want to sit down as a leadership team and define your goals for using advanced analytics, because most organizations have enough data to deliver some valuable insights. Sure, there may be some imperfect data pockets, but there’s a vast amount of internal data to leverage. One key early step is to partner with analytical teams to define specific use cases. These use cases don’t have to be incredibly impactful—you can start small, experiment and apply key learnings as you uncover them.

You’ll also want to consider the makeup of your team. Does it have the skill set and experience to help you achieve these goals? Over the past several years, Novartis has built a large ecosystem of hundreds of data scientists across various domains, creating hubs in locations like Barcelona and India. This team of data experts can rapidly build models, understand the data landscape and provide advisory support to the business. It’s important to have access to these capabilities by either bringing them in-house or working with an external partner.

Another important learning: Make sure to involve IT early. I’ve found a cautious, collaborative approach is essential when introducing innovative capabilities within a traditional corporate IT environment. Plus, your IT team can help you align the entire ecosystem, provide preferred vendors and accomplish much more.

MS: I know you’re passionate about the importance of quality data. What are some strategies organizations can execute to ensure their data is high quality?

FSG: There are several steps you can take to make sure your data is useful:

• Leverage existing data: It’s crucial to use data that’s already generated by ongoing processes within the organization, rather than relying on manually entered data sets.
• Define data quality standards: Set realistic expectations for model accuracy. Focus on your desired level of confidence and a realistic acceptable error rate.
• Improve iteratively: Start with a baseline model accuracy and improve it gradually by incorporating more data and techniques.
• Embrace data governance: Collaborate with the data governance team to ensure the quality of enterprise data.
• Educate: Train your team on the importance of data quality and updating data for better model performance.

MS: Relatedly, how can AI and advanced analytics reduce risk around inaccurate R&D data?

FSG: AI can analyze vital signs, labs and other patient data to reveal patterns that suggest problems such as poorly calibrated equipment or manipulated data entries. These findings shouldn’t be considered conclusive proof of a problem but can serve as red flags for further investigation. Of course, it’s critical to make sure the business is on the same page about using AI and analytics for this purpose.  

MS: That’s well said, thank you Fabien. What’s an R&D function that you’ve seen benefit from advanced analytics and AI?

FSG: Let’s talk about the thousands of clinical research associate (CRA) reports generated each year. This wealth of information offers an opportunity to identify new risks at various levels, from individual sites to global trends. Thankfully, advanced analytics can help us transform these unstructured reports into a structured database.

Traditionally, risk identification has been more focused on the global perspective, but using advanced analytics to analyze structured CRA reports provides a high-volume, detailed view of risk across different levels of a clinical trial. This approach is similar to analyzing lab performance reports, but with a significantly larger data set.

MS: As more R&D organizations use AI, what risks should they consider and work to mitigate?

FSG: Obviously hallucinations are a risk that receives a lot of attention. Overreliance is also a risk, but it can be countered by educating teams across the organization about the model’s constraints. With that said, the risk I would be most concerned about is the risk of not using AI—I believe focusing on the hazards of not using AI can encourage using AI responsibly. While AI and advanced analytics won’t replace audit teams, those who use it are more likely to succeed.

Watch Michael and Fabien’s full conversation.