When it comes to data centre operation, one of the greediest energy consumers is the cooling system. New solutions are now aiming to combat this fact. One of these – admittedly not really new - approaches is water-based cooling. Totally new, however, is the idea of MEGWARE and SCHÄFER IT-Systems for cooling the Leibniz data centre by doing without air completely and also cooling the mains adapters, network components and switches with water – resulting in a significant increase in energy efficiency. With this CoolMUC-3 system, they intend to demonstrate the technical feasibility of IT systems which are completely cooled with water.
The Leibniz data centre (LRZ) at the Bavarian Academy of Sciences and Humanities is the supercomputing centre for the Ludwig Maximilians University (LMU), the Technical University of Munich
(TUM), the Bavarian Academy of Sciences and Humanities as well as other scientific and educational institutions in the State of Bavaria. It provides the well over 100,000 students, professors, lecturers and employees with IT services such as e-mail, internet, Wi-Fi and VPN. In addition, the LRZ operates high performance computing systems for all those universities, as well as a national supercomputer, which is one of the most efficient in the whole world and is available to all public research institutions in Germany.
These high-performance computing systems (HPCs) form the core for the simulation of real problems. This cuts out the costs for laboratory experiments and results in unrealisable experiments, such as in the fields of astrophysics, aircraft development or disaster control, actually being made possible.
For such requirements, standardised architectures quickly reach their technological limits. In the special case of the LRZ, an additional computing capacity was needed. The intention behind this renewal was to
give users the possibility to prepare their applications for future computing architectures. With a view to saving energy, those in charge of the LRZ decided to use a 100 % water-based cooling system.
“Air cooling isn’t the best solution for our computers. Though we have up to now used systems with warm water cooling, there was room for improvement regarding their efficiency in cooling the components,
which was, of course, also due to their age”, says Herbert Huber, head of LRZ high performance systems, alluding to the fact that the LRZ, in its pioneering role, had already backed water cooling for its systems in 2011. However, not all components of the systems used were cooled with water, mainly the power supply units, network components and the switches. The resulting high energy loss to the ambient air could not be kept below 15 %. Huber continues: “With large systems like these, we had to use a great many kilowatts of electricity for cooling.”
The solution was provided by a company called MEGWARE Computer Vertrieb und Service GmbH, along with its partner enterprise, SCHÄFER IT-Systems. Though the systems used in 2011 also came from MEGWARE, the company’s water cooling system for high performance computing and IT solutions is now in its fourth generation. The waste heat from all components is now dissipated using direct water cooling. To achieve this, the existing solutions were developed further and retrofitted with cold plates, for example: all done in cooperation with SCHÄFER IT-Systems, a manufacturer of tailor-made network cabinet, server cabinet and data centre solutions. The system, which now operates completely with warm water cooling, is the only one of its kind worldwide. Its thermal insulation minimizes the heat loss to the environment to around 3 % waste heat in the room. The rest is done by room humidification and dehumidification systems. The temperature level at which the water cooling can work efficiently has also been raised. This means the hot water used for the cooling concept can also be used for secondary applications, such as heating the building. This is the case in the LRZ, too.
In the most recent Europe-wide tender in 2017, MEGWARE and SCHÄFER IT-Systems made the most convincing impression. What was required was the 100 % water cooling of all components with water heated to over 40 ° Celsius to cool the cluster all the year round without any energy-intensive mechanical refrigeration processes. During development, two aspects were given priority: thermal insulation of the racks to ensure less waste heat will enter the room, and achieving the highest possible temperature for the cooling water, so that the waste heat can be efficiently used for absorption cooling, for example. There were a total of only three offers. “That was no great surprise for us. Not many companies are capable of meeting our technical demands and MEGWARE was the only one that could provide an adequate cooling system,” says Huber. “We then concluded a supply and framework agreement and put the HPC system into operation in 2018.”
Mit dem Einsatz des Systems, geht das LRZ in seiner Rolle als Pionier erneut einen weiteren Schritt voraus. Schließlich wird die Auswahl der Lieferanten für IT-Infrastrukturen eingeschränkt, die ein derartiges Kühlungskonzept mit passenden Systemen bedienen können. Durch seine Expertise konnte SCHÄFER IT-Systems diese Lücke bei der technischen Umsetzung der Kühlung für MEGWARE schließen. Das Unternehmen stimmte mit MEGWARE in enger Zusammenarbeit zunächst das Konzept und anschließend die entsprechenden Lösungen ab. Hohe Ansprüche stellte es beispielsweise an den Wasserkreislauf und dessen gleichbleibende Qualität: Das Kühlsystem besteht aus relativ kleinen Strukturen, durch die das Wasser fließen muss. Deshalb muss es möglichst frei von Partikeln sein. Durch die Berührung mit dem Kühlmittel ist auch das genutzte Material der Komponenten entscheidend.
By using this system, the LRZ has moved another step ahead in its pioneering role. Ultimately, the choice of IT infrastructure suppliers that can provide systems suitable for such a cooling concept is limited. Thanks to its particular expertise, SCHÄFER IT-Systems was able to step in and realise the cooling system’s technical implementation for MEGWARE. In close cooperation with MEGWARE, the company first agreed on the concept and then coordinated the solutions accordingly. High demands were placed on the water cycle, for example, regarding its consistently high quality. The cooling system itself consists of relatively small structures, which the water has to flow through. This makes it essential that no particles must be inside them and, because it comes into contact with the coolant, the choice of material is also decisive. The result is a rack which accommodates water pipes, a heat exchanger and pump assembly, as well as a pressure compensation vessel. This allows the heat to be dissipated without it coming into contact with the data centre’s water circuit. A closed cooling water circuit or the separation of circuits in server cabinets and data centres is an essential step: circuits in data centres are subject to much less stringent requirements for water quality. Also integrated into the rack is a touch screen which displays all internal and external temperatures, which are transmitted by the appropriate sensors and interfaces. This replaces the current app-based cloud solutions which were never a genuine alternative for the LRZ for reasons of security. The intelligent rack incorporates these readings and values into management software which is programmed and administrated by MEGWARE. This enables the administrators to gain access at any time from other locations.
“After about a year, we can say that the HPC system has met all our expectations and initial requirements. We use the system actively in collaboration with students who are doing scientific work and conducting applied research in the field of energy efficiency, for example. Above all however, we are seeing significant energy savings”, Huber says in summary.