STFC's Tom Griffin talks to SCW about how the growth of AI and the increasing capabilities of UK national resources, such as the Diamond Light Source, are posing significant challenges to research infrastructure.
Can you tell me about your role at the STFC?
Tom Griffin: My main job is Director of Scientific Computing. That is the department that delivers all of the scientific computing infrastructure that underpins the national labs, part of accelerators, the astronomy programme, particle physics programme, material science – all of that – and that's everything from computational science through to large-scale HPC platforms and 100 petabytes of data storage and access for all of those. I'm seconded at the moment into a role as Digital Transformation Director, which is related to all of STFC's transformation and prioritisation, which I'm sure you've heard all about.
What does digital transformation look like at STFC?
TG: It potentially means different things to different people. In this context, it's very specifically about the STFC prioritisation exercise and reshaping the organisation to be as efficient and effective as it can be. Part of that involves examining how we use AI and other tools to make the organisation more efficient and effective. It's also looking at how we deliver all of our scientific computing to meet our science's needs while being effective and efficient.
How much of an impact will AI have on workloads?
TG: That's the million-dollar question. If I could answer that, I'd solve all the problems. Within the facilities – and this goes back to questions we've heard before – we see AI. By facilities, I mean the Diamond Light Source, the ISIS neutron source, and the Central Laser Facility. We can already see AI touching pretty much every part of that facility's workflow. Facilities are big, football field-sized accelerators used to look at materials, viruses, and genomes, and every step of that workflow: from experiment design, the pre-modelling and simulation that we do, running the experiment, determining when to stop the experiment, driving the experiment and data analysis, to running all underlying support services like cooling systems, cryostats, particle accelerator operations, and fault diagnosis. We are already applying AI to those steps to a greater or lesser extent.
In the design phase, it's surrogate modelling. In particle accelerators, it's modelling for design, but also automated AI fault-finding, feeding the diagnostics of particle accelerators into an AI model (a digital twin, if you like) to speed up fault diagnosis and, even better, map blind faults before they occur, spotting subtle patterns in accelerator diagnostics so we can detect faults before they impact stability and operations. It really is everywhere. Across the wider organisation, it's much more about using turnkey AI tools, like Claude for coding, right through to Copilot to automate business processes within the organisation.
Where do you see STFC's role evolving within the wider research computing ecosystem?
TG: The key thing to remember about STFC is that we are a dual body: we are both the funder and the doer of research. As a funder, we have long supported data-intensive doctoral training centres (CDTs) and AI CDTs, investing in postdoc and PhD schemes to train and deploy AI in high-energy physics. Within the labs, as doers, we have a role in developing those tools and putting them in the hands of users of our experimental facilities. AI is many things to many people. In many ways, it will give us better tools, and users will still run those experiments; it's just that the control system will be AI-enhanced rather than manually programmed. STFC has published an AI strategy, and that clearly shows we have that dual role.
The other significant role we play is as a generator of high-quality data. Our facilities are data-generating machines, and if you look at examples like DeepMind and specifically AlphaFold, that model was developed because of data deposited over many years into repositories like the Protein Data Bank. A huge fraction of that data comes from synchrotron sources like the Diamond Light Source, which I'm looking at out of my window right now. So our role is to provide data, crucially, AI-ready data, and to ensure it is appropriately annotated and open, with a data policy that allows it to be reused by AI. STFC touches loads of different parts of the AI and data landscape.
Datasets need to be appropriately stored and managed for later use with AI. Are there best practices or data standards that you recommend for the wider community?
TG: It varies a lot by domain. If you look at particle physics, there are long-established open formats into which data in that domain is collected. Whether it's machine learning, AI, or statistics, these approaches have been used for a long time in those communities, and they are adapting to adopt more modern AI approaches.
Looking at multidisciplinary facilities, such as materials science or biology, the [degree of] variation varies a lot by scientific domain – probably more than it does by facility. Areas like bio in general and protein crystallography specifically have well-established data formats that are widely used and contain all the metadata. Others are behind the curve on that. We have established facility standards for raw data, but metadata quality and reusability vary widely across domains. Interestingly, we're definitely seeing a trend of users becoming aware that they need to provide higher-quality metadata to enable their own future reuse.
The other interesting side is that AI is driving how we need to store data. Beyond having the right metadata, we are seeing demand for different data storage styles. Because we store hundreds of petabytes of scientific data, we do a lot of that on tape because tape is extremely good value – it is a cheap and very energy-efficient way of storing large volumes of data. However, it is not compatible with most AI use cases where you want to scan over huge volumes of data at speed. So we are having to make new investments in different performance tiers of data storage to facilitate AI use cases, and that comes at a high cost, especially given the memory crisis.
How do you strategise around the huge price increases?
TG: In some cases, the use case requires that very high-end NVMe-type approach, and we have to buy what we can afford. In others, we benefit from scale-out features. We operate well over 50 petabytes of spinning disk storage, and the scale-out nature of platforms like Ceph offers massive parallelisability, which fits some use cases incredibly well.
How has demand for these types of resources changed?
TG: It has grown significantly, and in many cases is outpacing Moore's law, certainly outpacing our budgets. A lot of what we support within the national labs and multidisciplinary facilities is for close-to-real-time use: in-experiment HPC. We are seeing much more demand for during-experiment support, modelling, and data simulation. That doesn't fit traditional HPC batch models and is pushing us toward using more cloud-like technologies. We have a very large OpenStack instance and see different usage patterns on that.
That's a challenge for efficiency, because batch systems are inherently efficient; they operate at over 90% efficiency in most cases. With cloud workloads, we have to balance the cost-benefit of having resources sitting idle versus direct support for science.
The most obvious change we're seeing is the rapid, massive increase in data volume, driven by increased source brightness. The Diamond Synchrotron is upgrading to a new accelerator, which will lead to 10 to 100 times the brightness in many cases. With detector increases and bigger, faster detectors, single detectors will output data on the order of hundreds of gigabits per second in the next few years. A facility like Diamond has around 30 beamlines, so you can multiply these numbers up several times.
Are there negative impacts of AI?
TG: The AI bubble has led to price rises in memory and GPUs, which is a problem against relatively flat budgets. On the positive side, DSIT has made significant UK investments in initiatives such as the AI Research Resource (AIRR) for the Isambard-AI machine and the AIRR-2 heterogeneous computing resource. One of our challenges is how best to make use of those. As I mentioned earlier, there is a tension between our need for in-experiment use of AI versus national services that are based around traditional batch-based processes.
Another change in facilities, like Diamond and ISIS, is opening them up to a wider range of users. Twenty or thirty years ago, users were specialists in neutron or synchrotron facilities who knew how to do everything. Now we see a much broader range of science done on these facilities, which means providing different levels of user services, data analysis support, instrument scientists, and simpler interfaces and tools. People often come to use these facilities for just a couple of days a year or throughout their entire PhD, as one tool among many. We see much more diverse use, which changes how we need to provide data and systems to make them easier to use.
So you now have a more diverse user base?
TG: Very definitely. Scientists are pushing forward really exciting and innovative ways of using AI within their workflows; it has become a much easier tool to use. It is challenging for us when users arrive with a local LLM on their Mac, wanting data fed directly into it, which requires us to modify our systems to allow that outside the data centre, along with all the security and safety considerations that come with it.
There seems to be a lot of worry about the potential impact of AI on cybersecurity. How much of this is real fear about the here and now, and how much is based on the future of AI development?
TG: It is a tricky question to answer, but it does look pretty scary. It is the shrinking gap between when exploits are discovered and when they are exploited that makes the timeframe much shorter, meaning our time to patch is getting shorter. The classic answer is defence in depth; we cannot rely on any single method of protection. It requires network segmentation, protection, patching, and threat analysis coming together. The rate of change, or the time from discovery to announcement to exploit, is terrifyingly short.
The availability of easy-to-use hacking tools that have the latest vulnerabilities available to exploit is very noticeable.
Tom Griffin is the Director of STFC's Scientific Computing Department.
