Here at Blackcore we have spent years perfecting the testing process for our industry leading overclocked servers. This process is crucial for ensuring that every single Blackcore server will perform reliably in real production environments.
We test every single server for at least 32 hours; only upon passing each set of our tests within our specifications can a server be prepared for shipping. You may be wondering - what tests do we run? Why do we run them? What is required from each server to pass our criteria?
Firstly, all servers will go through functional testing and configuration checks. Here we are ensuring that all the correct parts are installed in the server and working as expected. This includes checks such as all PCIe slots being enumerated, PSU failover is working, or that all the system fans are spinning correctly. At this point we will also flash the correct firmware images and load BIOS profiles, ensuring those apply successfully.
Once we’re satisfied, the server will move on to stress testing.
Stress testing involves pushing high synthetic workloads through various components to ensure that the components can run at the overclocked performance without failure. These stress tests will cause the components to heat up, which means they will also verify the server's cooling is working as intended under load. Additionally, the stress tests cause the components to draw peak power for prolonged periods of time, which can identify any issues with the component's ability to handle the overclocked profile that we have installed. While stress testing often focuses on CPU and RAM as key components, the effects of this increased power draw and temperature generation will also stress other components in the system, including the motherboard and the power supply. This is why we simply cannot test for a short term to verify the overclock is operational – we need to verify the server is completely prepared for production use.
Predominantly we use Prime95 (or ‘mprime’ in Linux) to stress test our servers. Prime95 uses FFT (Fast Fourier Transform) calculations of various sizes to find prime numbers – however, it turns out this workload is very good at putting maximum stress on a CPU, which will draw more power and generate more heat than many other stress tests that could be used.
Prime95 can be configured to run on any number of cores – for our testing, we run it on all cores in the server, so our end-users can have peace of mind when they are running their workloads on our servers.
Prime95 can run various sizes of calculation, which means that it is possible to adjust the size to control where the data is being held within the server, and thus which part of the server is being stressed – for example, smaller data sets might fit in L1 cache and larger data sets may have to be stored in RAM.
We run Prime95 in two modes:
The CPU executes software code based on a pre-defined list of instructions that the CPU can perform. At its most basic form, think of this like the instruction to add or subtract. The most common family of instruction sets in modern CPUs used by Blackcore servers today is often referred to as SSE (Streaming SIMD Extensions), which is a Single Instruction, Multiple Data (SIMD) CPU instruction set designed by Intel. Most developed software code will leverage these instructions at the hardware level due to its wide support and relative simplicity.
Over time, new instruction sets have been added to new CPU generations, one of the most frequently discussed is Advanced Vector Extensions (AVX). AVX has multiple generations, each adding new instructions or increasing the bus size of existing instructions – all of which is intended to speed up certain calculations. To utilise new instructions and benefit from the acceleration, software must be re-written and re-compiled. This may not be possible depending on the target hardware. Additionally, newer instructions may come with other downsides, like increased power draw or reduced frequency of operation.
AVX instructions are typically much harder on the CPU, and a single AVX instruction will draw more power, and generate more heat, than a single SSE instruction. Due to this it’s typical for AVX code to run at a lower clock speed than traditional SSE instructions - which the CPU will manage on its own when AVX instructions are used. Different CPU manufacturers handle this differently, but in general there will be power or thermal limits that are used to control or offset the overall clock speed of AVX instructions.
Historically we have found that Prime95 running in AVX mode is unrealistically hard for the CPU and is a workload which is not typically encountered by end users in production environments. Therefore, all our Prime95 testing is performed with AVX disabled.
It’s worth noting that all the clock speeds advertised by Blackcore are based on non-AVX workloads. For AVX workloads, we instead work directly with our end users on their specific use case and fine tune the offsets required and agree a viable stress test for validation pre-shipment.
In recent years however, CPUs have been getting better at performing AVX instructions with less impact on clock speeds. Additionally, we’ve seen AVX use become more important in certain electronic trading workloads. As such, all new products, starting with the ICON 3100-GM series, have AVX profiles and specific stress testing in place as standard.
For AVX stress testing we use YCruncher, as it allows us to specifically chose certain AVX instruction families to iterate through and test. YCruncher is primarily a tool to calculate Pi to extreme numbers of digits and will scale across all cores and various RAM sizes.
All servers must pass 16 hours of each stress test.
A pass means that the server performed the stress test without any failures – which could be an application error or crash, an OS crash such as kernel panic, or a hard crash such as the server rebooting. In addition to this, for each product we have strict thermal, power, and voltage ranges that the server components must operate within during the entirety of the test.
We log all of the servers' stats during the stress tests, which allows us to see what was happening at any point during the test cycle. This can help us understand why a server may have failed a test and help us to tune the overclock profile. Once a server passes our tests, the full report for each test is saved for future reference if needed.
If a server cannot pass our stress testing, even with additional tuning, then the testing data will be used to identify which is the weak component that needs to be replaced. At this point, all testing is reset and the process starts again.
Once a server has passed all the stress testing, any additional config is applied, this may include writing FRU data or applying client specific configuration like an operating system installation or configuring IP details for easier deployment in the data center.
Lastly, the server is sent to our dispatch team, where a final physical configuration check is performed, once again ensuring that the right components are installed and installed correctly.
All servers are then packaged in secure packaging and firmly attached to pallets to ensure that the server reaches the end users in the same state that it left our manufacturing facility.
We hope this article gave you an insight into the testing process at Blackcore.
If you’d like to learn more about our overclocked servers and our processes, please email [email protected]
