Research Computing procured and installed Dogwood cluster
Research Computing procured a new high-performance compute cluster, Dogwood. Acquired from Lenovo, Dogwood initially comprises 8,052 cores on 183 nodes, each with 44x2.4GHz physical cores and 512GB RAM, on an Infiniband EDR low latency fabric.
Dogwood is designed to execute computational jobs routinely that would not have been practical, even possible, to run on Research Computing’s Kill Devil cluster: e.g., 4,224-way jobs; 1,056-way non-blocking jobs.
Research Computing plans to add in excess of another 2,000 cores. Among the many things Research Computing does, high-performance computing is its bread-and-butter.
Research Computing decommissioned the Kure cluster
Research Computing decommissioned the Kure cluster in March 2017. Kure coincided with the explosion of data produced by NextGen Sequencing technologies. The initial 500 TB Isilon file system has been expanded to nearly 4 PB. Kure processed 82 percent of the samples processed locally in The Cancer Genome Atlas (TCGA) effort. Processing 26 percent of the 13,665 samples processed under the V1 pipeline, Kure was selected as the main production for the V2 pipeline, processing 98 percent of the remaining 53,870 samples. TCGA is a multi-university, international effort with the stated mission: "to accelerate our understanding of the molecular basis of cancer." At UNC-Chapel Hill, the TCGA effort was a six-year endeavor that exceeded $20 million. It continues under the Leidos TCGA3 project.
Assumed provisioning and management of the infrastructure of ibiblio
As part of a new partnership between ITS and the School of Information and Library Science, Research Computing took over provisioning and management of the infrastructure of ibiblio.org.
Research Computing took on this work with ibiblio, a digital archive, in May 2017. People all over the world take advantage of ibiblio's hosting and sharing services. It is one of ITS' largest network customers, reaching millions of users every day.
Ibiblio was founded at UNC-Chapel Hill in September 2000 as one of the world’s first online libraries and a way to share and support all kinds of free software.
Leidos TCGA3 Pipeline work moves to Longleaf/Pine
In the wake of Kure’s decommissioning, much of the type workload that Kure so ably served is now being discharged on Longleaf/Pine. A successor to TCGA, the Leidos TCGA3 Pipeline is a large-scale sequencing project associated with Lineberger Bioinformatics Group and High Throughput Sequencing Facility. About 10,000 samples are expected over several years. The pipeline set up on Longleaf is sufficient for processing more than 500 samples per week. This is a sub-contract, with Leidos Corp. handling sequencing only. However, a quality control pipeline is necessary to guarantee quality of sequencing results. More than 500 samples were successfully processed this year.
Improving the performance of Amber with GPUs on Longleaf
Processing with Graphical Processing Units continues to grow. Longleaf included five GPU nodes, each with eight Nvidia 1080X CPUs. Research Computing plans to add more. Here’s an example of why: Amber is a popular molecular dynamics suite and its performance is done with Amber's standard benchmark job, dihydrofolate reductase (DHFR) in water with 23,558 atoms. These jobs are performed in the GPU partition of the Longleaf cluster. The speedup of using 1 GPU/1 CPU core with respect to 1 CPU core is 125 fold.
Runt core utilization (more like 30 percent, given the RAM requirement) running flat-out for about 11 days. See https://doi.org/10.1093/mnras/stx720, especially Figure 11.
By the Numbers
Two new cluster systems (Longleaf and Dogwood)
More than 5.8 million jobs on Longleaf
More than 15.7 million processor hours on Longleaf
More than 11 petabytes of research data storage