General

Included in the subscription, we have Major Releases every 6 months (March and September), which include a new R runtime version, updated package versions, and additional packages. For an additional cost we have Minor Releases in between Major Releases (June and December) which only add additional packages.  

Atorus provides a customer specific Installation Guide detailing every step of the installation process including links to downloadable binaries or containers depending on your installation method (direct server/container).  

As part of the Product Customer Success Process, our customers can provide stacked ranked prioritization of packages to be included as candidates for each future release as well as input on the software feature road map. 

Based on what’s been seen through efforts like the R Consortium’s pilot submissions and the experience that has been publicized like Novo Nordisk’s submission, Atorus can provide instructions for a reviewer to recreate the environment based on the OpenValTM version so that the reviewer can follow those instructions. Based on security policies at the agency and transparency necessary for the environment, this is the industry standard necessary at this time. 

If there are issues arising from the R runtime and packages installed with OpenValTM, and the issues are not user error of the packages, this falls under Product Support included in the subscription. We also have additional Managed Services arrangements we can discuss to provide more holistic support to the R environments. This could include additional support as well as performing the installation and change management of OpenValTM in the customer environment.  

Package Validation

Packages are qualified through additional testing not provided as unit tests and qualified for multiple operating systems and the most current version of R. The installation of OpenValTM provides a self-documenting Validation Report (IQ/OQ) of a successful installation of OpenValTM in a customer’s environment.  

Our risk assessment follows many of the ideas outlined here: https://www.pharmar.org/white-paper/. Our initial programmatic risk assessment process uses some of the ideas of the risk metric package, namely for calculating the number of downloads. Beyond the initial programmatic risk, a validator looks at the package and its documentation to determine the final assigned risk. The validator will look at the quality of documentation of the package, code coverage of the unit tests, how well maintained the package is, and any other details they can find before declaring the final risk category. 

The purpose of giving a package a risk is to determine the level of remediation/testing required to mitigate the risk. Low risk packages will test the main use cases of the package. Medium risk packages will test the main use cases and edge cases of the package. High risk packages will test the majority of functions users of the package would interact with, as well as important helper functions. 

Our validation strategy has been developed in line with best practices recommended by organizations like the R Validation Hub and developed alongside pharmaceutical partners for use within GxP systems. In addition, OpenValTM has been audited by our customers for the following regulations, but not limited to: 

  • FDA Guidance: Use of Electronic Records and Electronic Signatures in Clinical Investigations Under 21 CFR Part 11 – Questions and Answers, Jun 2017 
  • ISPE GAMP 5 Guide: A Risk-Based Approach to Compliant GxP CSV, 2/08 
  • 21 CFR Part 11/ EU Annex 11 – Electronic Records and Electronic Signatures 
  • ISO 9001:2015 – Quality Management System 
  • ICH E6 – Guideline for Good Clinical Practice 
  • ICH E9 – Statistical Principles for Clinical Trials 

The requirements and test cases are detailed in the Validation Report to indicate what has been tested. Test code, if needed, can be viewed through a request to Atorus Quality Assurance. 

Our testers have a minimum of 2 years’ experience writing tests for package development and testing. Within OpenValTM our testers have spent over 30,000 hours writing more than 11,000 test scripts.

To work on statistical packages, testers must have at least a master’s degree as well as some experience with hands-on statistical applications.  

System and Privacy

OpenValTM can be installed on a direct server or containerized deployments:  

  • OpenValTM requires Python 3 and Git 
  • Minimum specifications for server and the workstation include CPU = 2, Ram = 8G, Storage = 10GB 
  • OpenValTM currently qualifies packages for Ubuntu 20.04+ (Focal), Ubuntu 22.04+ (Jammy), and RHEL 8 

Atorus can work with customers to determine how OpenValTM will be installed. Customers can conduct the installation themselves. Some customers leverage Managed Services to perform the installation and change management within their environment. 

No. We distribute our own binaries. 

Note, for a non-GxP system we could include the binaries of R packages within your installation of Posit Package Manager. This would provide you with our evaluated and compiled versions of R packages, but not include the on-system testing.  

OpenValTM is accessible from whatever IDE is chosen and has no reliance on an IDE in general. OpenValTM replaces your R runtime itself.  

User libraries are by default disabled. This ensures the stability of the environment by preventing users from being able to override the current environment for themselves. Through our Managed Services we can assist companies in customizing the system solution to support their needs if their requirements vary. For example, if a customer wants to maintain a library for internally developed and maintained packages, which sits on top of the OpenValTM installation, this can be configured. 

OpenValTM is not a user-based license. User capacity will be limited by the system which OpenValTM is installed into, such as Posit Workbench. 

The only information Atorus maintains about a customer is for the license, packages installed or requested, method of installation (direct server/container), and operating systems in use. 

Customers can conduct the installation themselves, without granting Atorus access to the environment. Some customers leverage Managed Services to perform the installation and change management within their environment. 

Package Selection

Atorus develops a candidate package list for upcoming releases through compiled prioritization requests from customers, as well as leveraging our industry involvement to identify relevant packages.  

Tidyverse packages are part of OpenValTM. OpenValTM contains many packages from the pharmaverse such as admiral, Tplyr, pharmaRTF, metacore, metatools, logrx, and xportr. The pharmaverse packages are high priority for us, but several pharmaverse packages are still in an experimental state. These packages will be added as they mature, but we target our support at packages achieving stability. 

A snapshot date eliminates versioning problems. Choosing a date to pull packages from ensures all packages in OpenValTM work together and with the R version, and there are no dependency issues. 

We anchor our versions of OpenValTM against specific dates where we can determine all necessary packages to be in compliance. If dplyr 1.1.4 isn’t passing CRAN checks, it’s likely that a patch would be published within a short period of time, so we’ll select the snapshot date accordingly.  

In the case of a less supported package, this would require some investigation to determine the best course of action. If a package is pulled from CRAN, it’s important to understand why. Perhaps the author didn’t get updates in on time, maybe their package was pulled because of a dependency, or maybe the author no longer intends to support the package. These become contributing risk factors.  

If a package becomes unsupported, we may keep it in OpenValTM for a certain period of time using the latest version that still passes our validation checks, but flag that the package could soon be deprecated from OpenVal™ since in the future it may no longer meet compliance.

Customization

OpenValTM does not let you choose the R version. The R version is bundled with OpenValTM packages, upon which we do extensive testing to make sure that the underlying operating system is compatible through R and all included packages.  

Yes. One goal of OpenValTM is to aid in the reproducibility of results.  

With a container installation, one container for each OpenValTM release can be created so study teams can access the necessary version of their study. The limit to the number of versions of OpenValTM would be dictated by your company retention policies for containers made available to users, and as such there is no limit.  

With a direct server installation, multiple versions of OpenValTM can be installed in parallel, but we would expect that a maximum of 3 installations can co-exist together as OpenValTM has no control over the required system dependencies of the open-source packages within it.  

Atorus can help with any step of the process of implementing OpenValTM including but not limited to: 

  • Installation of a statistical computing environment prior to installation of OpenValTM 
  • Installation and change management of OpenValTM 
  • Consulting on best practices for using and maintaining OpenValTM 

Additionally, for individual users of OpenValTM and leveraging R, we have Atorus Academy as our training solution, for which we have course tracks focused on helping SAS® programmers learn to adopt R. 

Customers have options for Package Validation Services. We’d discuss the logistics of this with the business unit including additional mitigations that focus on the business process of the system.