In April 2014, under the supervision of the Head of Research Computing, Dr Robert Esnouf, the Centre installed a new compute cluster with 3.5x more processing power and 8x more memory than the system it replaced. Recently OCF issued a press release about the development, which was seized on by the computing press: so far articles have appeared in Computer Weekly, Computer World UK, The Platform, and The Information Daily. Robert explains what they’re all so excited about.
With my deputy Jon Diprose, I have worked with the high-performance integrator OCF to design and install one of the most powerful university departmental high-performance computing (HPC) facilities in the UK – about 4000 compute cores and counting! The Centre now houses the second-largest next-generation sequencing facility in England, currently producing more than 500 genomes per year. Each processed and compressed genome is about 30GB on disk. Across our storage systems roughly 15,000-20,000 human genomes occupy about 0.5PB – that’s half a petabyte, or half a million gigabytes – although even that is only 10% of the total storage we currently manage.
However, simply processing data from sequencing machines is no longer that demanding in terms of processing power. What really stresses systems are ‘all-against-all’ analyses of hundreds or even thousands of genomes: lining up multiple genomes against each other and using sophisticated statistics to compare them and spot differences that might explain the genetic origin of diseases or susceptibility to diseases. Most of our users want to complete this type of research.
The more samples you can compare at once, the more statistical power you get and the more sensitive and reliable the answers are. For example, before we had the new cluster we managed to compare about 1000 genomes simultaneously. Now (at a push) I’d back us to process 10,000.
Each research group can use their own server to submit jobs to, and receive results from, the cluster. Users don’t need to logon directly to the cluster or be aware of other research groups using it. We designed the system to try to isolate groups so that they don’t slow each other down. And we’ve kept it as simple as possible: users do not need to be HPC experts to use the system safely and effectively.
The benefits for the Centre’s electron microscopy research are also clear, particularly for analyzing large particles such as whole viruses which is a specialization of STRUBI. The system has already produced some of the world’s highest-resolution electron microscopy reconstructions – revealing structural details vital to understanding processes such as infection and immunity. Co-head of STRUBI Dave Stuart is delighted by what his colleagues can do with the new system.
“Advances in detector design and processing algorithms over the past two years have revolutionized electron microscopy, making it the method of choice for studying the structure of complex biological processes such as infection’, he says. ‘However, we thought we could not get sufficient access to the necessary compute to exploit these advances fully. The new genetics cluster provided such a fast and cost-effective solution to our problems that we invested in expanding it immediately!”
Research is driving our adoption of HPC. Compared to this time last year, our researchers – and we have about 100 active users – can put through around five times more work and are doing so on a machine with the same energy footprint. With the support of OCF and hardware partners, like Fujitsu, Mellanox and DDN, we’re now fully armed to meet today’s challenges. But research never stands still; our current system may only be relevant for 3-5 years, and we’re already planning for the next phase!