Software is fundamental to modern scientific research (Hey et al., 2009; Wilson, 2014). It can accelerate the pace of research, formalize algorithmic logic, and support reproducibility (Piccolo and Frampton, 2016). In a 2014 survey, 92% of academic scientists reported using software in their research (Hong, 2014). Our article focuses on command-line tools, which scientists use in many disciplines (Kumar and Dudley, 2007) and which provide advantages over point-and-click tools. In particular, the process of executing command-line tools can be formalized (Kumar and Dudley, 2007; Piccolo and Frampton, 2016). Our article also focuses on computational workflows, which are defined series of steps for processing or analyzing data (Leipzig, 2017). Each step in a workflow applies one or more command-line tools to the data with specific inputs, outputs, and configuration settings. A workflow engine is a software system for executing workflows. For example, the Snakemake and Nextflow workflow engines facilitate execution of workflows and are used widely for scientific research (Di Tommaso et al., 2017; Köster and Rahmann, 2012). These and other workflow engines provide flexibility regarding the computing environment in which a workflow is executed, allowing researchers to use local, cluster-, or cloud-based computers. In many cases, workflow steps can be executed in parallel. With this flexibility comes the challenge of ensuring that operating system and tool configurations are consistent across all computers used. This process is made easier when workflows provide instructions for executing the steps within software containers (Boettiger, 2015).

In scientific research, the use of workflows can be classified into two main categories. One category includes orchestration systems, in which workflow engines repeatedly process data of a given type. For example, a genomics core facility might use a workflow to ingest DNA-sequencing data, align reads to a reference genome, and call DNA variants, as well as other steps in between. The core facility might have other workflows for processing RNA-sequencing or bisulfite-sequencing data. In some cases, a modular design might be useful; for example, some command-line tools could be reused in multiple workflows. In this scenario, data throughput is a key concern, so the workflow engine might be connected to cluster- or cloud-based computer environments, allowing throughput to scale as needs fluctuate. Arvados and Tibanna are examples of cloud-based orchestration systems (Amstutz, 2015; Lee et al., 2019). Scientists might also take advantage of error handling provided by some workflow engines. For example, if a power outage occurred while a workflow was being executed, it would be possible to restart the workflow at the point of failure rather than needing to rerun the entire workflow. A second way that workflows are used in scientific research is to support reproducibility of a particular analysis. When performing research studies, scientists often use existing software and write custom code to process data, apply statistical tests, create graphics, etc. Inherently, the particular combination of computational tasks used in a given study should be unique. Therefore, there is no need to orchestrate the tasks for repeated execution. However, much benefit can come from sharing the code with the scientific community and ensuring that others can easily re-execute the code (Piccolo and Frampton, 2016). Workflows aid in this process because they provide a way to encapsulate the logic for executing code in a fully specified computational environment with all necessary software dependencies installed. For many studies, a relatively simple workflow engine like cwltool (RRID:SCR_015528) can be used to re-execute the analyses.

Different workflow engines use different methodologies and vocabularies for defining workflows and for interfacing with software within those workflows (Leipzig, 2017). For example, to use Snakemake or Nextflow, a researcher can specify a workflow using a custom programming language that is specific to either workflow engine. These languages are extensions of Python and Groovy, respectively. Python and Groovy are familiar to many researchers, making them relatively easy for these researchers to adopt. However, if a researcher does not have programming expertise, is not familiar with these programming languages, or wants to switch between workflow engines, workflow creation might be difficult or time-consuming. Such challenges motivated creation of the Common Workflow Language (CWL), a formal specification for describing command-line tools and workflows (Amstutz et al., 2016). The CWL project is a community-based, collaborative effort driven by individuals and institutions across diverse disciplines; participation is open to anyone. These efforts should help to ensure the project’s longevity and acceptance among researchers. Indeed, CWL documents are already recognized by many workflow engines, including Snakemake (Köster and Rahmann, 2012), cwltool, Toil (Vivian et al., 2017), Apache Airflow (Kotliar et al., 2019), Tibanna (Lee et al., 2019), and Arvados (Amstutz, 2015).

By creating CWL documents, scientists can describe tools and workflows in a way that is standards-based and agnostic to the workflow engine(s) on which they are executed. CWL documents are text-based files and thus can be created via a text editor without doing any computer programming. Despite this simplicity, many researchers have yet to adopt CWL. Useful tutorials are available online (Common Workflow Language working group, 2021), but the CWL specification provides great flexibility, so researchers face a learning curve to ensure that documents are specified correctly. Some software applications are available to aid in this process. For example, Rabix Composer is a desktop application that enables researchers to create and edit CWL documents (Amstutz et al., 2021). This application supports many features in the CWL specification and provides text-based as well as visual editors. However, researchers unfamiliar with nuanced details of the CWL specification may find Rabix Composer too advanced for their needs, and it does not support the latest versions of CWL. Alternatively, CWL plugins exist for many code editors (Common Workflow Language working group, 2021). In addition, developers have created application programming interfaces that enable researchers to build CWL documents. However, many researchers who could benefit from CWL lack the programming expertise to use these resources.

In this article, we illustrate how to use CWL to describe command-line tools and workflows and to perform reproducible research analyses. We provide 27 examples of CWL documents for completing diverse types of research tasks, ranging from simple (e.g., printing custom messages to the console) to advanced (identifying differentially expressed genes or calling somatic mutations in cancer genomes). In addition, we introduce ToolJig, a Web application that enables researchers to create CWL documents. ToolJig provides a simple, interactive interface that requires no installation and includes prompts to guide the user. Via ToolJig, a researcher can specify details about a tool’s expected inputs and outputs, operating-system environment, and auxiliary files (e.g., scripts, configuration files). Researchers can also create workflows that integrate these tools. ToolJig checks the information provided by the user to ensure it is complete and valid. After successfully describing a tool or workflow, the researcher can download CWL files and use ToolJig to create ‘input-object’ files, which store input values for a particular invocation of a tool or workflow. In addition, ToolJig provides examples of how to execute the tool or workflow via a workflow engine.

CWL tool examples for printing simple command-line messages

In the GitHub repository that accompanies this article, we have provided example tool descriptions, formatted in YAML according to the CWL specification (https://github.com/srp33/ToolJig/tree/master/examples, copy archived at swh:1:rev:ae8d3b358ccc44e45604125257c5361d20c26832, Stephen, 2021). The first series of examples is stored in the hello subdirectory.

The first example, 01_hello.cwl, requires two inputs: (1) a person’s given name and (2) the person’s surname. It uses the baseCommand field to construct a command based on ordinal positions specified for the inputs; the resulting command prints a greeting for that person. In this simple example, the only output is the greeting sent to standard output, which is redirected to a file called 01_output.txt. In the GitHub repository, hello_objects.yml is an example input-object file for this tool. It species a person’s given name (‘Fernanda’) and surname (‘Dantas’). The user could execute the tool via the cwl workflow engine using this command: cwltool 01_hello.cwl hello_objects.yml. The output would be, ‘Hello, Fernanda Dantas’.

Suppose we wished to alter the greeting to include an exclamation point and to indicate the person’s age. First, we would add an input for the person’s age (an integer). Second, we would update the command that will be executed. However, CWL’s baseCommand field provides limited flexibility for constructing commands. Instead, our example provides a command template using the ShellCommandRequirement and arguments fields and uses placeholders within the template for each input. As shown in 02_hello.cwl (and Figure 1A), we use the following template: echo Hello, $(inputs.given_name) $(inputs.surname)! You are $(inputs.age) years old.

Figure 1

Download asset Open asset

We use a $ character and parentheses to indicate placeholders for input variables. We prefix each input variable with ‘inputs’ to indicate that they will be specified as inputs.

By default, the above two examples would be executed within the same operating-system environment as the workflow engine. Accordingly, these tools could only be executed on operating systems that support the echo command. Many commands, including echo, are only supported on particular operating systems—or their behaviors differ by operating system. So in 03_hello.cwl (and Figure 1B), we use a Docker image based on the ‘buster’ release (version 10.3) of the Debian Linux operating system. The DockerRequirement field is added (in this case, two lines of text). Before executing the tool, a researcher would install a container engine such as Docker Desktop (RRID:SCR_016445). Then, when executing the tool, the workflow engine would integrate itself with the container engine, which would identify any input files or directories and create container volumes so that the files or directories could be accessed from within the container. When a command-line tool executes within a Docker container, it becomes portable—it can be executed on any computer that supports the container and workflow engines.

CWL tool examples for performing simple data analyses

The second series of examples is stored in the examples/bmi subdirectory on the GitHub site. The 01_bmi.cwl tool provides a simple example of a reproducible, quantitative analysis that could be performed using CWL. It accepts as input a tab-separated file containing names, weights, and heights of (fictional) individuals. A second input specifies the name of the column in the tab-separated file that contains weight information. The third input specifies the column name containing height information. The fourth input is the name of an output file that will be created. This example illustrates the use of an auxiliary file. Under the InitialWorkDirRequirement field, the contents of a Python script are stored. This script is used to calculate body mass index (BMI) values for each person in the input file and store those values in a BMI column in the output file. Below is the command template that we use.

python calculate_bmi.py "$(inputs.input_file.path)" "$(inputs.weight_column_name)" "$(inputs.height_column_name)" "$(inputs.output_file_name)"

The command template specifies the inputs as arguments to the Python script. When a file input is used, the workflow engine stores metadata about the file in an object with multiple attributes. Thus to reference the file’s path within the command template, we append ‘path’ to the input name. As the workflow engine executes the tool, it stores the auxiliary Python script inside the container, invokes the script, and collects the output file that the script generates.

If a researcher wished to ensure that others could reproduce the BMI calculations, they would need only to share the CWL file, the input-object file (01_bmi_objects.yml), and the data file (biometric_data.tsv). However, many analyses use data stored in online repositories. In such cases, it is convenient for a CWL tool to pull data directly from an online repository. The 02_bmi.cwl tool description and Figure 2 illustrate this approach. Similar to the previous example, it extracts names, weights, and heights from a tab-separated file and adds a BMI column. However, it pulls the file from a Web server (in this case, our GitHub repository). The command template is similar to the previous example. Again, we use a software container based on Debian Linux. But this time, we also use the NetworkAccess field to enable the container to connect to external computers. The tool emits messages to both standard output and standard error; these messages are stored in files called ‘02_output.txt’ and ‘02_error.txt’, respectively.

Figure 2

Download asset Open asset

CWL workflow example for identifying somatic variants in a cancer genome

The examples in the examples/workflows/somatic subdirectory demonstrate a process for calling somatic variants from Illumina sequencing reads. We use paired-end reads from tumor and normal cells for an individual from the Texas Cancer Research Biobank (Becnel et al., 2016). (Although these data are publicly available, they are subject to some data-use restrictions; Becnel et al., 2016.) To shorten execution times, we use a subset of the data: the first 10,000,000 reads from lane 2 of the sequencing run. Furthermore, our analysis is limited to essential steps for preparing the data and calling variants. Additional steps like annotation and filtering would improve sensitivity and specificity of the variant calls; accordingly, researchers should interpret our variant calls with caution.

In these examples, somatic-variant calling occurs in a series of steps (Figure 3):

1. Download and index a human reference genome (version hg38). We use the Linux wget and gunzip utilities to download and decompress a FASTA file from the UCSC genome repository (Haeussler et al., 2019). We also use bwa, samtools, and Picard Tools to index the FASTA file and create a sequence dictionary (Li et al., 2009; Li and Durbin, 2009; Picard, RRID:SCR_006525).

2. Preprocess reference files containing known polymorphic sites in preparation for base-quality score recalibration (BQSR). We download Variant Call Format files (Danecek et al., 2011) from the Genome Analysis Toolkit (GATK) resource bundle (Depristo et al., 2011) and use a custom Python script to adjust chromosome identifiers that may differ across reference genomes. We also use Picard Tools to sort the reference files.

3. Download the raw sequencing reads (FASTQ files). The files are stored in a publicly available, Open Science Framework repository (Foster and Deardorff, 2017).

4. Trim adapter sequences and low-quality bases using atropos (Didion et al., 2017).

5. Align the trimmed reads to the reference genome using bwa mem (Li and Durbin, 2009). A read-group string is also specified.

6. Sort and index the resulting BAM files (Li et al., 2009) using sambamba (Tarasov, 2015).

7. Mark duplicate reads and re-index the resulting BAM files using sambamba.

8. Derive a BQSR table from each BAM file using GATK.

9. Apply the BQSR table to the BAM files using GATK.

10. Call somatic single-nucleotide variants and small insertions/deletions using Mutect2 (Benjamin et al., 2019). This step produces a VCF file.

11. Call somatic structural variants using Delly (Rausch, 2012). This step produces a VCF file.

Figure 3

Download asset Open asset

This workflow executes many of the same steps for a normal DNA sample and a tumor DNA sample. These steps are independent of each other. Accordingly, when multiple computing cores are available, the workflow engine may execute these steps in parallel.

Most of the tools in this workflow use container images from the BioContainers project, which provides thousands of Docker images that encapsulate biology-related software (da Veiga Leprevost et al., 2017). In some cases, we used a BioContainers image that had been built for a specific version of a software package in the Bioconda project (Grüning et al., 2018). In cases where we used multiple packages for a given task, we started with a base image from BioContainers and used Bioconda to install the packages. In the case of GATK, we used container images that had been created by the Broad Institute and stored on Docker Hub.

In most cases, we followed the recommendation that a single container image use only a single software package (Gruening et al., 2019). However, in some cases, we determined that it was more sensible to use multiple software packages in a single CWL tool. For example, when preparing index files for the reference genome, we use three separate software packages. In contrast, sometimes we used the same software in multiple CWL tools. For example, the BQSR steps are computationally intensive; thus, we separated them into distinct CWL tools so that computational resources can be allocated at a more granular level. In this sense, each CWL tool represents a practical unit for data processing, not necessarily a particular software package.

Progress in biology research is hindered when software tools are difficult to install, when inputs and outputs are inadequately or inconsistently specified, and when it is difficult to combine tools into workflows. The CWL specifications—in combination with package managers and software containers—are helping to alleviate these longstanding challenges. Moreover, CWL tool and workflow descriptions can facilitate reproducible research. Rather than simply providing analysis code alongside journal manuscripts, researchers can provide CWL documents. As illustrated in our examples, CWL documents provide instructions for executing analyses in software containers that encapsulate all relevant dependencies (Figure 5), along with ancillary scripts and instructions for accessing data files. People who read (or review) the manuscripts can then repeat the analyses, without needing to install any software other than a relevant workflow engine and container engine, even if their operating system or configuration differs from the authors’.

Figure 5

Download asset Open asset

Multiple online repositories provide CWL documents, including the Dockstore tool registry (O’Connor et al., 2017) and GitHub. For example, a search on GitHub for CWL documents that use the FastQC software (Brown et al., 2017) resulted in 667 matches (August 30, 2021). Researchers can reuse and adapt these documents. However, in cases where reuse is infeasible or extensive adaptations are necessary, scientific progress may be accelerated as researchers, including non-bioinformaticians, gain greater efficiency in creating CWL documents. ToolJig aims to facilitate this process, enabling researchers to build CWL tools interactively, without needing to gain a deep understanding of the CWL specification or YAML syntax.

One advantage of CWL is that it can be used with diverse workflow engines. Whether or not they support CWL, most workflow engines provide custom languages or programming interfaces for creating workflows. Relatively little support is available for migrating from these engine-specific solutions to CWL in an automated manner. However, when these engines support execution within Docker-compatible containers, researchers can migrate these tools manually using ToolJig (or other means). Providing better support in existing workflow engines for exporting to CWL will be a positive step toward ensuring that CWL truly becomes a common language for command-line tools and workflows.

The CWL specification provides considerable flexibility for describing command-line tools and workflows. Our goal was to support common use cases for biology research. For the sake of simplicity and to reduce barriers of entry for new creators of CWL documents, ToolJig does not support some optional features within the CWL specification. These include input directories, dependent and exclusive parameters, process requirements, hints, and output directories. The CWL specifications provide details about these features.

ToolJig has no dependencies other than a modern Web browser. Accordingly, it can be used from virtually any computer with no installation process. When ToolJig is updated, the user simply needs to refresh their browser. A tradeoff to this simplicity is that ToolJig does not provide a direct means of testing tools or workflows. However, the cwltest utility provides a command-line testing framework, enabling researchers to compare tool and workflow outputs with expected results. In particular, researchers implementing CWL in production systems would benefit from using such a utility for validation.

We did not generate data for this manuscript. However, we did create software for this manuscript. The full code for the software are available on GitHub using a liberal, open-source license: https://github.com/srp33/ToolJig (copy archived at https://archive.softwareheritage.org/swh:1:rev:ae8d3b358ccc44e45604125257c5361d20c26832).

1. 1. Amstutz P

   (2015) Portable, reproducible analysis with Arvados

   F1000Research 4:114.

   https://doi.org/10.7490/f1000research.1110114.1

   • Google Scholar
2. Software

   1. Amstutz P
   2. Crusoe MR
   3. Tijanić N
   4. Chapman B
   5. Chilton J
   6. Heuer M
   7. Kartashov A
   8. Leehr D
   9. Ménager H
   10. Nedeljkovich M
   11. Scales M
   12. Soiland-Reyes S
   13. Stojanovic L

   (2016) Common workflow language, v1.0

   Figshare.

   https://figshare.com/articles/dataset/Common_Workflow_Language_draft_3/3115156/2
3. Software

   1. Amstutz P
   2. Soiland-Reyes S
   3. Crusoe MR

   (2021) Rabix: Power tools for the Common Workflow Language

   Seven Bridges.

   http://www.rabix.io
4. 1. Barrett T
   2. Troup DB
   3. Wilhite SE
   4. Ledoux P
   5. Evangelista C
   6. Kim IF
   7. Tomashevsky M
   8. Marshall KA
   9. Phillippy KH
   10. Sherman PM
   11. Muertter RN
   12. Holko M
   13. Ayanbule O
   14. Yefanov A
   15. Soboleva A

   (2011) NCBI GEO: archive for functional genomics data sets--10 years on

   Nucleic Acids Research 39:D1005–D1010.

   https://doi.org/10.1093/nar/gkq1184

   • PubMed
   • Google Scholar
5. 1. Becnel LB
   2. Pereira S
   3. Drummond JA
   4. Gingras MC
   5. Covington KR
   6. Kovar CL
   7. Doddapaneni HV
   8. Hu J
   9. Muzny D
   10. McGuire AL
   11. Wheeler DA
   12. Gibbs RA

   (2016) An open access pilot freely sharing cancer genomic data from participants in Texas

   Scientific Data 3:1–10.

   https://doi.org/10.1038/sdata.2016.10

   • PubMed
   • Google Scholar
6. Preprint

   1. Benjamin D
   2. Sato T
   3. Cibulskis K
   4. Getz G
   5. Stewart C
   6. Lichtenstein L

   (2019) Calling Somatic Snvs and Indels with Mutect2

   bioRxiv.

   https://doi.org/10.1101/861054

   • Google Scholar
7. 1. Boettiger C

   (2015) An introduction to Docker for reproducible research

   ACM SIGOPS Oper Syst Rev 49:71–79.

   https://doi.org/10.1145/2723872.2723882

   • Google Scholar
8. 1. Bottomly D
   2. Walter NAR
   3. Hunter JE
   4. Darakjian P
   5. Kawane S
   6. Buck KJ
   7. Searles RP
   8. Mooney M
   9. McWeeney SK
   10. Hitzemann R

   (2011) Evaluating Gene Expression in C57BL/6J and DBA/2J Mouse Striatum Using RNA-Seq and Microarrays

   PLOS ONE 6:e17820.

   https://doi.org/10.1371/journal.pone.0017820

   • PubMed
   • Google Scholar
9. 1. Brown J
   2. Pirrung M
   3. McCue LA

   (2017) FQC Dashboard: Integrates FastQC results into a web-based, interactive, and extensible FASTQ quality control tool

   Bioinformatics 33:3137–3139.

   https://doi.org/10.1093/bioinformatics/btx373

   • PubMed
   • Google Scholar
10. Software

    1. Charliecloud collaborators

    (2021) Charliecloud

    GitHub.

    https://github.com/hpc/charliecloud
11. Website

    1. Common Workflow Language working group

    (2021) Common Workflow Language User Guide

    Accessed September 10, 2021.

    https://www.commonwl.org/user_guide/index.html
12. 1. da Veiga Leprevost F
    2. Grüning BA
    3. Alves Aflitos S
    4. Röst HL
    5. Uszkoreit J
    6. Barsnes H
    7. Vaudel M
    8. Moreno P
    9. Gatto L
    10. Weber J
    11. Bai M
    12. Jimenez RC
    13. Sachsenberg T
    14. Pfeuffer J
    15. Vera Alvarez R
    16. Griss J
    17. Nesvizhskii AI
    18. Perez-Riverol Y

    (2017) BioContainers: An open-source and community-driven framework for software standardization

    Bioinformatics 33:2580–2582.

    https://doi.org/10.1093/bioinformatics/btx192

    • PubMed
    • Google Scholar
13. 1. Danecek P
    2. Auton A
    3. Abecasis G
    4. Albers CA
    5. Banks E
    6. DePristo MA
    7. Handsaker RE
    8. Lunter G
    9. Marth GT
    10. Sherry ST
    11. McVean G
    12. Durbin R

    (2011) The variant call format and VCFtools

    Bioinformatics 27:2156–2158.

    https://doi.org/10.1093/bioinformatics/btr330

    • PubMed
    • Google Scholar
14. 1. Depristo MA
    2. Banks E
    3. Poplin R
    4. Garimella KV
    5. Maguire JR
    6. Hartl C
    7. Del Angel G
    8. a RM
    9. Hanna M
    10. McKenna A
    11. Fennell TJ
    12. Kernytsky AM
    13. Sivachenko AY
    14. Cibulskis K
    15. Gabriel SB
    16. Altshuler D
    17. Daly MJ

    (2011) A framework for variation discovery and genotyping using next-generation DNA sequencing data

    Nature Genetics 43:491–498.

    https://doi.org/10.1038/ng.806

    • PubMed
    • Google Scholar
15. 1. Di Tommaso P
    2. Chatzou M
    3. Floden EW
    4. Barja PP
    5. Palumbo E
    6. Notredame C

    (2017) Nextflow enables reproducible computational workflows

    Nature Biotechnology 35:316–319.

    https://doi.org/10.1038/nbt.3820

    • PubMed
    • Google Scholar
16. 1. Didion JP
    2. Martin M
    3. Collins FS

    (2017) Atropos: Specific, sensitive, and speedy trimming of sequencing reads

    PeerJ 5:e3720.

    https://doi.org/10.7717/peerj.3720

    • PubMed
    • Google Scholar
17. 1. Foster ED
    2. Deardorff A

    (2017) Open Science Framework (OSF)

    Journal of the Medical Library Association 105:203–206.

    https://doi.org/10.5195/jmla.2017.88

    • Google Scholar
18. 1. Gomes J
    2. Bagnaschi E
    3. Campos I
    4. David M
    5. Alves L
    6. Martins J
    7. Pina J
    8. López-García A
    9. Orviz P

    (2018) Enabling rootless Linux Containers in multi-user environments: The udocker tool

    Computer Physics Communications 232:84–97.

    https://doi.org/10.1016/j.cpc.2018.05.021

    • Google Scholar
19. 1. Gruening B
    2. Sallou O
    3. Moreno P
    4. da Veiga Leprevost F
    5. Ménager H
    6. Søndergaard D
    7. Röst H
    8. Sachsenberg T
    9. O’Connor B
    10. Madeira F
    11. Angel DD
    12. Crusoe MR
    13. Varma S
    14. Blankenberg D
    15. Jimenez RC
    16. Community B
    17. Perez-Riverol Y

    (2019) Recommendations for the packaging and containerizing of bioinformatics software

    F1000Research 7:742.

    https://doi.org/10.12688/f1000research.15140.2

    • PubMed
    • Google Scholar
20. 1. Grüning B
    2. Dale R
    3. Sjödin A
    4. Chapman BA
    5. Rowe J
    6. Tomkins-Tinch CH
    7. Valieris R
    8. Köster J

    (2018) Bioconda: Sustainable and comprehensive software distribution for the life sciences

    Nature Methods 15:475–476.

    https://doi.org/10.1038/s41592-018-0046-7

    • PubMed
    • Google Scholar
21. 1. Haak LL
    2. Fenner M
    3. Paglione L
    4. Pentz E
    5. Ratner H

    (2012) ORCID: A system to uniquely identify researchers

    Learned Publishing 25:259–264.

    https://doi.org/10.1087/20120404

    • Google Scholar
22. 1. Haeussler M
    2. Zweig AS
    3. Tyner C
    4. Speir ML
    5. Rosenbloom KR
    6. Raney BJ
    7. Lee CM
    8. Lee BT
    9. Hinrichs AS
    10. Gonzalez JN
    11. Gibson D
    12. Diekhans M
    13. Clawson H
    14. Casper J
    15. Barber GP
    16. Haussler D
    17. Kuhn RM
    18. Kent WJ

    (2019) The UCSC Genome Browser database: 2019 update

    Nucleic Acids Research 47:D853–D858.

    https://doi.org/10.1093/nar/gky1095

    • PubMed
    • Google Scholar
23. Book

    1. Hey T
    2. Tansley S
    3. Tolle K

    (2009)

    The Fourth Paradigm: Data-Intensive Scientific Discovery

    Microsoft research.

    • Google Scholar
24. Website

    1. Holmes I

    (2013) You can download our code from the URL supplied. Good luck downloading the only postdoc who can get it to [Tweet]. Twitter

    Accessed September 10, 2021.

    https://twitter.com/ianholmes/status/288689712636493824
25. Software

    1. Hong NC

    (2014) We are the 92%

    Figshare.

    https://figshare.com/articles/presentation/We_are_the_92_/1243288/1
26. 1. Huber W
    2. Carey VJ
    3. Gentleman R
    4. Anders S
    5. Carlson M
    6. Carvalho BS
    7. Bravo HC
    8. Davis S
    9. Gatto L
    10. Girke T
    11. Gottardo R
    12. Hahne F
    13. Hansen KD
    14. a IR
    15. Lawrence M
    16. Love MI
    17. Macdonald J
    18. Obenchain V
    19. Oleś AK
    20. Pagès H
    21. Reyes A
    22. Shannon P
    23. Smyth GK
    24. Tenenbaum D
    25. Waldron L
    26. Morgan M

    (2015) Orchestrating high-throughput genomic analysis with Bioconductor

    Nature Methods 12:115–121.

    https://doi.org/10.1038/nmeth.3252

    • PubMed
    • Google Scholar
27. 1. Ison J
    2. Kalaš M
    3. Jonassen I
    4. Bolser D
    5. Uludag M
    6. McWilliam H
    7. Malone J
    8. Lopez R
    9. Pettifer S
    10. Rice P

    (2013) EDAM: An ontology of bioinformatics operations, types of data and identifiers, topics and formats

    Bioinformatics 29:1325–1332.

    https://doi.org/10.1093/bioinformatics/btt113

    • PubMed
    • Google Scholar
28. 1. Köster J
    2. Rahmann S

    (2012) Snakemakea scalable bioinformatics workflow engine

    Bioinformatics 28:2520–2522.

    https://doi.org/10.1093/bioinformatics/bts480

    • PubMed
    • Google Scholar
29. 1. Kotliar M
    2. Kartashov AV
    3. Barski A

    (2019) CWL-Airflow: A lightweight pipeline manager supporting Common Workflow Language

    GigaScience 8:giz084.

    https://doi.org/10.1093/gigascience/giz084

    • PubMed
    • Google Scholar
30. 1. Kumar S
    2. Dudley J

    (2007) Bioinformatics software for biologists in the genomics era

    Bioinformatics 23:1713–1717.

    https://doi.org/10.1093/bioinformatics/btm239

    • PubMed
    • Google Scholar
31. 1. Kurtzer GM
    2. Sochat V
    3. Bauer MW

    (2017) Singularity: Scientific containers for mobility of compute

    PLOS ONE 12:e0177459.

    https://doi.org/10.1371/journal.pone.0177459

    • PubMed
    • Google Scholar
32. 1. Lee S
    2. Johnson J
    3. Vitzthum C
    4. Kırlı K
    5. Alver BH
    6. Park PJ

    (2019) Tibanna: Software for scalable execution of portable pipelines on the cloud

    Bioinformatics 35:4424–4426.

    https://doi.org/10.1093/bioinformatics/btz379

    • PubMed
    • Google Scholar
33. 1. Leipzig J

    (2017) A review of bioinformatic pipeline frameworks

    Brief Bioinform 18:530–536.

    https://doi.org/10.1093/bib/bbw020

    • PubMed
    • Google Scholar
34. 1. Li H
    2. Durbin R

    (2009) Fast and accurate short read alignment with Burrows-Wheeler transform

    Bioinforma Oxf Engl 25:1754–1760.

    https://doi.org/10.1093/bioinformatics/btp324

    • PubMed
    • Google Scholar
35. 1. Li H
    2. Handsaker B
    3. Wysoker A
    4. Fennell T
    5. Ruan J
    6. Homer N
    7. Marth G
    8. Abecasis G
    9. Durbin R

    (2009) The Sequence Alignment/Map format and SAMtools

    Bioinforma Oxf Engl 25:2078–2079.

    https://doi.org/10.1093/bioinformatics/btp352

    • PubMed
    • Google Scholar
36. 1. Love MI
    2. Huber W
    3. Anders S

    (2014) Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2

    Genome Biology 15:550.

    https://doi.org/10.1186/s13059-014-0550-8

    • PubMed
    • Google Scholar
37. 1. Nüst D
    2. Sochat V
    3. Marwick B
    4. Eglen SJ
    5. Head T
    6. Hirst T
    7. Evans BD

    (2020) Ten simple rules for writing Dockerfiles for reproducible data science

    PLOS Computational Biology 16:e1008316.

    https://doi.org/10.1371/journal.pcbi.1008316

    • PubMed
    • Google Scholar
38. 1. O’Connor BD
    2. Yuen D
    3. Chung V
    4. Duncan AG
    5. Liu XK
    6. Patricia J
    7. Paten B
    8. Stein L
    9. Ferretti V

    (2017) The Dockstore: Enabling modular, community-focused sharing of Docker-based genomics tools and workflows

    F1000Research 6:52.

    https://doi.org/10.12688/f1000research.10137.1

    • PubMed
    • Google Scholar
39. 1. Piccolo SR
    2. Sun Y

    (2012) A single-sample microarray normalization method to facilitate personalized-medicine workflows

    Genomics 100:337–344.

    https://doi.org/10.1016/j.ygeno.2012.08.003

    • PubMed
    • Google Scholar
40. 1. Piccolo SR
    2. Frampton MB

    (2016) Tools and techniques for computational reproducibility

    GigaScience 5:30.

    https://doi.org/10.1186/s13742-016-0135-4

    • PubMed
    • Google Scholar
41. 1. Priedhorsky R
    2. Randles T

    (2017) Charliecloud: unprivileged containers for user-defined software stacks in HPC

    SC ’17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis pp. 1–10.

    https://doi.org/10.1145/3126908.3126925

    • Google Scholar
42. Conference

    1. Project Jupyter
    2. Bussonnier M
    3. Forde J
    4. Freeman J
    5. Granger B
    6. Head T
    7. Holdgraf C
    8. Kelley K
    9. Nalvarte G
    10. Osheroff A
    11. Pacer M
    12. Panda Y
    13. Perez F
    14. Ragan-Kelley B
    15. Willing C

    (2018) Python in Science Conference

    Binder 2.0 - Reproducible, interactive, sharable environments for science at scale. pp. 113–120.

    https://doi.org/10.25080/Majora-4af1f417-011

    • Google Scholar
43. 1. Rausch T

    (2012) DELLY: Structural variant discovery by integrated paired-end and split-read analysis

    Bioinformatics 28:i333–i339.

    https://doi.org/10.1093/bioinformatics/bts378

    • Google Scholar
44. Software

    1. Stephen P

    (2021) Tooljig: An app for building simplified common workflow language tool and workflow descriptions, version swh:1:rev:ae8d3b358ccc44e45604125257c5361d20c26832

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:3e7275dd8056207f3f9c2d7af2143ae59325b606;origin=https://github.com/srp33/ToolJig;visit=swh:1:snp:f8ea5fce17127ca3aace81ffa4e20f50a1d8a5d8;anchor=swh:1:rev:ae8d3b358ccc44e45604125257c5361d20c26832
45. 1. Tarasov A

    (2015) Sambamba: Fast processing of NGS alignment formats

    Bioinformatics 31:2032–2034.

    https://doi.org/10.1093/bioinformatics/btv098

    • Google Scholar
46. 1. Vivian J
    2. Rao AA
    3. Nothaft FA
    4. Ketchum C
    5. Armstrong J
    6. Novak A
    7. Pfeil J
    8. Narkizian J
    9. Deran AD
    10. Musselman-Brown A
    11. Schmidt H
    12. Amstutz P
    13. Craft B
    14. Goldman M
    15. Rosenbloom K
    16. Cline M
    17. O’Connor B
    18. Hanna M
    19. Birger C
    20. Kent WJ
    21. Patterson DA
    22. Joseph AD
    23. Zhu J
    24. Zaranek S
    25. Getz G
    26. Haussler D
    27. Paten B

    (2017) Toil enables reproducible, open source, big biomedical data analyses

    Nature Biotechnology 35:314–316.

    https://doi.org/10.1038/nbt.3772

    • PubMed
    • Google Scholar
47. Software

    1. Wickham H

    (2018a)

    Dplyr: A grammar of data manipulation

    Dplyr.
48. Software

    1. Wickham H

    (2018b)

    Readr: Read Rectangular Text Data

    Readr.
49. 1. Wilson G

    (2014) Best Practices for Scientific Computing

    PLOS Biology 12:e1001745.

    https://doi.org/10.1371/journal.pbio.1001745

    • Google Scholar

Author details

1. Stephen R Piccolo

   Department of Biology, Brigham Young University, Provo, United States

   Contribution

   Conceptualization, Funding acquisition, Software, Supervision, Visualization, Writing - original draft, Writing – review and editing

   For correspondence

   stephen_piccolo@byu.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2001-5640

2. Zachary E Ence

   Department of Biology, Brigham Young University, Provo, United States

   Contribution

   Conceptualization, Software, Writing – review and editing

   Competing interests

   No competing interests declared

3. Elizabeth C Anderson

   Department of Biology, Brigham Young University, Provo, United States

   Contribution

   Conceptualization, Software, Writing – review and editing

   Competing interests

   No competing interests declared

4. Jeffrey T Chang

   Department of Integrative Biology and Pharmacology, University of Texas Health Science Center at Houston, Houston, United States

   Contribution

   Conceptualization, Funding acquisition, Writing – review and editing

   Competing interests

   No competing interests declared

5. Andrea H Bild

   Department of Medical Oncology and Therapeutics, City of Hope Comprehensive Cancer Institute, Monrovia, United States

   Contribution

   Funding acquisition, Writing – review and editing

   Competing interests

   No competing interests declared

• 650

  views

• 63

  downloads

• 5

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.