Why use AnVIL?

The NHGRI AnVIL (Genomic Data Science Analysis, Visualization, and Informatics Lab-space) is a project powered by Terra for biomedical researchers to access data, run analysis tools, and collaborate. Both biology researchers and educators can benefit from using AnVIL (anvil.terra.bio) for their research and in the classroom.

This guide acts as a resource answering the question "why use AnVIL?". It will discuss the research, classroom, and general benefits of using AnVIL and point to related resources throughout.

Why use AnVIL? There are research, classroom, and general benefits presented as sections within this guide

Benefits of using AnVIL for research

The research benefits for using the AnVIL include ease of platform access, variety of analysis solutions, data options, data and analysis in the same place, scalability, renting needed resources, role-based permissions, shareable workspaces, and being a repository compliant with DMS policy

Ease of platform access

The primary means of accessing the AnVIL platform (anvil.terra.bio) is through a web browser - users do not need to download data or install software.

Accessing AnVIL beyond the web browser:

The platform (anvil.terra.bio) provides a variety of graphical user interfaces (GUIs) to interact with data, analysis solutions, and workflows. Bioconductor packages (AnVIL, AnVILGCP, AnVILWorkflow) offer additional methods for users to programmatically interact with and access AnVIL resources from within AnVIL, or stand-alone computing environments such as a user's laptop.

Variety of analysis solutions

Compute options available on the AnVIL include Jupyter notebooks, workflow description languages, RStudio, and Galaxy

AnVIL supports an assortment of frameworks and tools. Researchers can use their favorite tool to work with data interactively or through non-interactive batch processing. Due to this variety and interoperability with other platforms, researchers can stay within a single environment for their analysis without having to shift between platforms.

Interactive sessions:

Interactive sessions are available with Jupyter, RStudio/Bioconductor, and Galaxy.

Use of Galaxy on AnVIL enables even more customizability with the ability to install specific tools/versions with Toolshed.

For more information on the AnVIL project portal: Interactive analyses

Non-interactive batch processing:

Workflows can be user-supplied/written or imported with Dockstore and used to steer non-interactive pipelines and batch processing of data.

For more information on the AnVIL project portal: Analysis Workflows

Data: yours or cloud-hosted open & controlled access

Data options available through AnVIL include several consortia datasets including GTEx

AnVIL securely stores diverse, open and controlled access, cloud-hosted datasets with a browsable summary catalog so researchers can identify relevant datasets they may need to request access to.

Consortia Data on the AnVIL:

Data on the AnVIL includes data from consortia such as

Additional consortia or upcoming planned data ingestion can be found on the AnVIL project portal. By inverting the typical method of genomic analysis, AnVIL brings a computing environment to the data rather than moving the data from storage to the computing environment or storing copies of the data across institutional high performance computing clusters.

Amount of data on AnVIL:

As discussed in the flagship AnVIL paper, the AnVIL hosts data from >280,000 human genomes from >240 different cohorts spanning multiple consortia and major NHGRI projects. The AnVIL offers a browsable catalog of summary information about all of the datasets so that even if a user isn't authorized to access the data itself, they can better determine if the data will be helpful for their research if they need to apply for authorization for access. AnVIL is working to facilitate data harmonization across studies, ensuring consistency and interoperability, which is critical for large-scale analyses. These efforts will increase the value of the AnVIL data and maximize its utility to the researcher community.

AnVIL hosts over seven petabytes of data across 113 dbGaP accessions and across several large consortia

For more information on the AnVIL project portal: Finding and accessing data

AnVIL is a FedRAMP Moderate compliant platform:

As a FedRAMP Moderate compliant platform, AnVIL maintains FedRAMP authorization of compliance to ensure as a cloud service provider, minimum security requirements are met for data processing, storage, and transmission of Protected Health Information (PHI) and Personally Identifiable Information (PII) where loss of confidentiality, integrity, and availability would result in serious adverse effect or non-life threatening harm. All steps necessary to maintain compliance, such as robust logging of access to data, periodic audits by third-party analysts, and monitoring for abnormal use patterns are managed and guaranteed by AnVIL.

For more information on the AnVIL project portal: Platform security and NIH's Data Management and Sharing policy requirements

Data & analysis in same place

The AnVIL portal is an entry point for all parts of the AnVIL with documentation and announcements. The home page of the portal is shown

AnVIL is a unified computing environment for data storage, management, and analysis. The AnVIL portal serves as an entry point to access all parts of the AnVIL system as well as training materials and announcements.

Scalability

AnVIL is conducive to analysis at massive scale and for data exploration and training. Researchers get access to dedicated compute resources, avoiding queue time and lack of access at some institutions. Researchers can also launch light environments or run test analyses without incurring much cost or spending a lot of time to configure.

Rent needed resources

Since AnVIL handles the support and maintenance of the platform (including the hardware and software), you can focus on performing your work on AnVIL rather than setting up and maintaining the platform, freeing up effort for your science. This is immensely valuable for researchers who do not have deep institutional IT and system administrator support for research infrastructure.

Role-based permissions

A workspace is the fundamental unit of activity in AnVIL with role-based access. An example workspace is shown with the access level or role highlighted

Group management can be utilized to control who can access specific data, analysis workspaces, and your billing resources. Workspaces provide a collaborative environment with role-based permissions. These permission include reading, writing, or owning with additional permissions for running compute and sharing. Especially within the contexts of working with sensitive data or large amounts of data, AnVIL's role-based group management permission structure is instrumental.

For more information on the AnVIL project portal: Terra Workspaces

Shareable workspaces

A workspace can be a shareable record of analyses. An example workspace is shown with options to share the workspace through its permalink highlighted

Workspaces can contain data, metadata, and analysis tools, as well as documentation and history of workflow runs, additionally displaying important information such as when the workspace was created and last modified. AnVIL workspaces on the web can serve as shareable, reproducible records of analyses. Research conducted on the AnVIL platform has contributed to over 115 scientific publications citing the AnVIL paper, demonstrating its role in advancing genomic and biomedical research.

Examples of AnVIL workspaces shared in publications:

AnVIL workspaces have previously been shared in publications to demonstrate reproducible science.

Repository compliant with DMS Policy

The AnVIL serves as a cloud data repository compliant with the Data Management and Sharing (DMS) Policy. Data access controls can be specified to limit data access and use.

By submitting their data to AnVIL, not only can researchers meet the requirements of DMS Policy, they can also contribute to the expanding network of NIH funded data housed in the AnVIL, furthering scientific discovery.

What is DMS Policy?

The National Institutes of Health (NIH) Data Management and Sharing (DMS) policy requires that all NIH-supported research which generates scientific data (barring ethical, legal, or technical factors limiting data sharing) must create a plan and budget for data management and sharing.

While some funding announcements for some research may specify which repository should be used to comply with DMS policy, mostly the NIH does not specify where data should be stored.. The NIH does provide an interactive table listing NIH supported data repositories than can be used and suggests that researchers use a repository most appropriate for the data generated from their research.

How is data access managed and requests granted?

AnVIL utilizes the Data Use and Oversight System (DUOS) to efficiently expedite data access and management while maintaining security. Researchers can explore datasets hosted in the AnVIL cloud and request access using DUOS. Data use limitations can be set if necessary with Data Access Committees reviewing access requests.

For more information on the AnVIL project portal: NIH's Data Management and Sharing policy requirements

Benefits of using AnVIL in the classroom

The classroom benefits of using the AnVIL are outlined including that it provides an unified computing system, authentic experience with cloud computing, a varity of tools, relevant datasets, and has prepared exercises

AnVIL provides all the advantages of a cloud computing environment, such as version control and offering a unified computing system without providing physical computers with certain specifications. Additionally, AnVIL provides students with authentic experience working in the cloud -- which is becoming common in today's research environment. Students can also gain experience with a variety of tools (e.g., Galaxy, RStudio, Jupyter notebooks, WDL workflows) all in one place while working with relevant datasets and prepared exercises.

Instructor Guide Available:

See more in our instructor guide on why AnVIL is a good choice for your classroom.

Four example books and mini Cures are shown. These include a demo on single cell with bioconductor, a book on SARS with Galaxy on AnVIL, the BioDIGS R Data Package, and the RNA-Seq mini cure

Overall benefits of AnVIL

The general benefits of using the AnVIL are shown. These include the ability to control costs, work with protected data safely, maintenance is handled, training and support is available, and there is a collaborative community

Ability to control costs

Cloud computing is not free and estimating costs may seem daunting to those considering use of the AnVIL. However, Terra provides thorough, transparent documentation explaining data storage and cloud computing costs and has been working to improve transparency and management of costs for AnVIL users through cost reporting, cost controls and estimates, and cost optimizations. Additionally, in order to debug or benchmark your work, analyses or workflows can be tested with smaller scale test datasets or light environments without incurring much cost or spending a lot of time to configure environments.

For more information on the AnVIL project portal: Controlling cloud costs

Work with protected data safely

Due to AnVIL maintaining compliance with FedRAMP policies, clinical data containing PHI and PII can be safely and securely stored and analyzed on AnVIL. This includes the ability to export data from clinical data collection and management tools like REDCap and import it into AnVIL Terra Tables.

For more information on the AnVIL project portal: Platform security and NIH's Data Management and Sharing policy requirements

Maintenance is handled

Training and support is available

To equip researchers and students to work on the AnVIL, the AnVIL team

provides and maintains training materials and documentation in multiple formats (ex: Getting Started on AnVIL),
moderates a support forum
hosts demos (ex: https://anvilproject.org/events/anvil2023-december-demos)
hosts workshops (ex: https://anvilproject.org/events/anvil2024-nhgri-intramural-workshop)

Many AnVIL support options or books can be found in the AnVIL Collection. One such option is the AnVIL Getting Started book. Other options include the support forum and AnVIL Demos which are held monthly live, but also recorded and posted to YouTube

For more information on the AnVIL project portal: Getting Started Instructions and Videos & Tutorials

Collaborative community

Community examples and resources such as the getting started book, the portal, and AnVIL on Terra

AnVIL has begun hosting community conferences to collaboratively innovate during CoFests! and to discuss research performed with the platform. The community can work directly with the AnVIL team to understand current development, feature requests, and a roadmap or future directions for the platform.

Past and Upcoming AnVIL Community Conferences:

Additionally, AnVIL values and routinely solicits user feedback to improve the user experience and provide the most beneficial features and enhancement for biomedical research. Feedback is gathered:

at the community conference
through State of the AnVIL community polls
through voluntary user interviews
24/7 at the support forum help.anvilproject.org

For more information on the AnVIL project portal: Events and News

Conclusion

All of this together describes how the AnVIL provides secure, cost-effective genomic analysis at scale, and is a useful cloud-based platform for training and research.

Help us make these docs great!

All AnVIL docs are open source. See something that’s wrong or unclear? Submit a pull request.

Make a contribution