06. Will the FDA accept data and analyses generated with solutions developed and available as open source?

[MichaelRimler]

Will the FDA accept data and analyses generated with solutions developed and available as open source?

[MichaelRimler]

FDA Statistical Software Clarifying Statement (May 6, 2015)

"FDA does not require use of any specific software for statistical analyses, and statistical software is not explicitly discussed in Title 21 of the Code of Federal Regulations [e.g., in 21CFR part 11]. However, the software package(s) used for statistical analyses should be fully documented in the submission, including version and build identification."

https://www.fda.gov/media/109552/download

Examples of submissions:

• Novo Nordisk: https://www.youtube.com/watch?v=t33dS17QHuA
• Roche: https://posit.co/blog/roche-shifting-to-an-open-source-backbone-in-clinical-trials/
• FDA: https://www.r-consortium.org/using-r-in-regulatory-review

[rossfarrugia]

https://rconsortium.github.io/submissions-wg/

[bms63]

The FDA does! but what about all the others? The R Consortium Submissions WG has only explored FDA.

Perhaps the title should change to be focused on all Regulatory agencies?

[rossfarrugia]

makes sense! we're going to do a webinar with Posit post-summer to share our experience so far submitting in R also including China and EMA (as part of the new data pilot). i heard at least anecdotally that Japan are also open to R.

[bms63]

YEah!! I think having any info on submissions to these agencies would be very helpful. Maybe in a table format. This is just a quick silly mockup.

Sponsor/Agency	FDA	EMA	PMDA	UK one?
Public Pilots	Submitted	Not done
NN	Submitted
Roche	Not done

Several people at Pharmasug were shocked that the FDA took R code after my R-based Submission presentation. They were under the impression that only SAS code was permitted!! So we have a lot of work to do!!

[rossfarrugia]

MHRA, close enough though

[geoskiad]

This is an excellent point and discussion. Very happy to see it raised here. If not mistaken, in the last R WG for Submissions it was also mentioned that SwissMedic (Health Authority from Switzerland) are interested for greater exposure to R. Based on my experience Health Canada and ANVISA (Brazil) also come often back with questions or have specific rules/requirements related to statistical analysis. But ok. Step by step.

[bms63]

@geoskiad you leading Pilot 5 on the R based Submission to SwissMedic?

[geoskiad]

@geoskiad you leading Pilot 5 on the R based Submission to SwissMedic?

No, not at all. Not even a member of it. I am just trying to stay informed of the latest R developments in Pharma world.

[bms63]

I was implying you should propose and lead Pilot 5!!

[KatjaGlassConsulting]

Thanks to all who participated in the OSTCDA call yesterday. We discussed this topic and would love to get additional input here. Specifically, it would be excellent to share submission experiences, challenges and links. @bms63 , @geoskiad and @rossfarrugia - following your suggestions, here is the table we started yesterday and hopefully we get more input.

Sponsor/Agency	FDA	EMA	NMPA	PMDA	Health Canada
RConsortium Pilot	Submitted - Accepted	Not done
Novo Nordisk Presentation	Submitted Big data ADaM programmed in R and pre-announced this during Type C and B meeting packages Remaining ADaM in SAS and double programmed in R (where applicable) All TFL in R Following requests from FDA all TFL programs and internal R packages submitted. Packages packed using pkglite. Additional guidance and informal meeting with FDA statisticians on restoring R environment was held	Submitted content using R	Same as PMDA	Big data ADaM programmed in R and pre-announced during additional eSubmission meeting Remaining ADaM in SAS and double programmed in R (where applicable) All TFL in R programs submitted	Submitted content using R
Roche Presentation	Submitted content using R Initial pushback from the FDA reviewer to accept R, they wanted SAS instead. Roche pushed back and submitted in R (based on Technical Conformance Guidance) key message - acceptance of R may vary between different agency reviewers key message - have the content & format discussions early with the agency	Submitted content using R (no comments received from the agency so far)	Submitted content using R (no comments received from the agency so far)
SwissMedic
General		Don’t reanalyse, might be easer		Don’t reanalyse, might be easer
Merck	Submitted Multiple successful submissions GxP platform available Reproducible environment
Astellas Pharma	- (2021) for using R markdown to submit programs - Example

[bms63]

This is a very cool table!! Be great post to put on linkedin asking for other companies inputs

[vijethamokkapati]

Thanks to all who participated in the OSTCDA call yesterday. We discussed this topic and would love to get additional input here. Specifically, it would be excellent to share submission experiences, challenges and links. @bms63 , @geoskiad and @rossfarrugia - following your suggestions, here is the table we started yesterday and hopefully we get more input.

Sponsor/Agency FDA EMA NMPA PMDA
RConsortium Pilot Submitted

• Accepted Not done
  Novo Nordisk Submitted - Double programmed with SAS?
• Didn’t pre announce using R
• Maybe data preparation in SAS, but TLF in R
  Roche Submitted - Pushed back from reviewers (SAS)
• Depending on reviewer
• Content wise no request
• Asked upfront if it‘s ok to do so Submitted content from R Submitted content from R
  SwissMedic
  General Don’t reanalyse, might be easer Don’t reanalyse, might be easer
  Merck Submitted - Multiple successful submissions- These were not purely R submissions. R only covered one or more TLFs, so the eCTD package contained both SAS and R content.
• GxP platform available
• Reproducible environment
  Astellas Pharma - (2021) for using R markdown to submit programs
• Example

[BhupendraMistry]

The Roche information in the table is not correct. Please amend as follows

[KatjaGlassConsulting]

Thanks, the table for Roche is updated

[ari-stotle-max]

For Novo Nordisk it is:

FDA:

• Big data ADaM programmed in R and pre-announced this during Type C and B meeting packages
• Remaining ADaM in SAS and double programmed in R (where applicable)
• All TFL in R
• Following requests from FDA all TFL programs and internal R packages submitted. Packages packed using pkglite.
• Additional guidance and informal meeting with FDA statisticians on restoring R environment was held

EMA:
Submitted content using R

Health Canada:
Submitted content using R

PMDA:

• Big data ADaM programmed in R and pre-announced during additional eSubmission meeting
• Remaining ADaM in SAS and double programmed in R (where applicable)
• All TFL in R programs submitted

NMPA:
Same as PMDA

MFDS (Korea):
Same as PMDA

Basically everywhere:
Submitted content using R

[KatjaGlassConsulting]

Following our group discussions yesterday (thanks to all who participated) - please continue discussion:

FDA Experiences:

• SAP might mention programming language, e.g. using R -> use what we said that we do
• How specific and where does the information go with R, e.g. which packages, which versions etc.
• Who owns reconciliation
  • Sponsor followed what they said they do (using R)
• Important to point out in early authority meeting to use R
• ADRG used for documentation, packages, package versions, not executable, but readable
• Validation of packages has been asked, provided validation records (provided upfront)
• Fall back SAS option
• Other authorities seem to be a bit eager for

EMA:

• Likely follow what is delivered on FDA
• Pilots running – submission of data, independent of SAS/R

Python:

• Interactive dashboards -> exported to figures
• Less conversation, but it’s within the scope
• Likely used internally for ML, dashboards similar
• Data transformations, graph modelling
• https://github.com/philbowsher/Open-Source-in-New-Drug-Applications-NDAs-FDA

Clinical Pharmacology

• Using other tools & OS for submissions

[bms63]

The FDA reviewers we worked with in the Pilot 3 were okay with ADRG having screenshots for installing renv and using the renv.lock file to install all package versions from CRAN. The screenshots helped reduce a lot of verbiage! Hopefully, opens possibilities for companies to have instructions on installing container infrastructure!

[KatjaGlassConsulting]

Another related topic (thanks for all who participated yesterday in the discussions):

What has been used for OS Submissions:

• PDFs for TLF
• Interactive app
• Dataset creation in XPT?
• Pilot4 -> HTML
  • Container
  • webR stuff (quarto)
• FDA accepts RMD -> did someone so?
• Specifications for File Format Types Using eCTD Specifications has rmd files for M5 https://www.fda.gov/media/85816/download

[bms63]

We should make a push to get .qmd files included in the list!

Also - this is very cool blog post coming about being careful when using SAS time versus Unix time epochs and xpts pharmaverse/blog#192

[MikeKSmith-Pfizer]

Following discussion at PhUSE EU Connect, some comments captured from the OSTCDA workshop:

• Some regulatory authorities definitely accept R
• Be prepared and learn how to submit
• Be ready to prove that you have validated the software. And be ready for questions (MikeKSmith note: likely questions for clarification about environment, packages, package versions etc.)
• How to have conversations with regulatory authorities?
• Be ready to discuss things like numerical matching.
• Do organisations (sponsor companies) have documentation of why SAS is approved (for regulatory submission)?