Monthly Archives: January 2014

Hacking Fan Speed on Dell PowerEdge Servers

A couple of weeks ago, I acquired three old Dell servers to play with: a PowerEdge 840, PowerEdge 830, and a PowerEdge SC 430. One thing I didn’t think about before I got them was how much noise they might make. I’m used to desktop machines; they’re designed to run quietly enough not to annoy someone trying to work in the same room. Dell doesn’t go to the same trouble when they design their servers. The SC 430 is reasonably quiet (Dell based it on their Precision platform), but the other two, the 830 and 840, are just loud enough to be annoying.

My first thought was that I might be able to control the fan speed, but fan speed is firmly under the control of the BMC (Baseboard Management Controller); I can’t control it from software. I determined after some Googling that most people solve the problem by replacing the system fans with slower, quieter, models. Unfortunately, I also found that the slower fans often trigger the “Lower Critical Threshold” (they spin too slow), causing the BMC to spin them up, which gets me back to the noise problem I had in the first place.

Of course, because this is the Internet and I’m not the only hacker who likes to play with hardware, someone else had already solved this problem.

TLDR; the thresholds in used by the BMC can only be changed by hacking the BMC firmware update package. I downloaded that guy’s Python script and ran it on my PE 830. The script was able to parse the BMC firmware update for the PE 830, so I went ahead and ordered a new fan.

new_old

Now, before I could replace the fan, I had to deal with Dell’s custom pinout (because their 4-pin arrangement is clearly superior to the standard 4-pin arrangement that carries the exact same signals). This is already documented in several places on the web, but just to get it up here one more time:

Signal Dell Color Standard Color
PWM Blue Blue
RPM Yellow Green
+12V Red Yellow
Ground Black Black

I lopped the connector off of the original fan (at least it’s not a non-standard pinout on a standard connector). I couldn’t shove the stranded wire into the new fan’s connector, so I soldered some solid wire from a bit of spare CAT 6 cable onto the leads. I pushed the wire all the way through the connector and bent it over, which should keep everything in place.

solid_copper

Before clamping and soldering.

wired

I plugged it in and started the machine. Good news: it was nearly silent. Bad news: loading the machine runs the core temp up to 60C (Intel says to keep it below 63). The fan I bought pushes 74 CFM at full speed; the OEM fan managed 150 CFM. That wouldn’t be a problem, except that Dell was cheap on all of their tower chassis in the mid-2000s and made the back case fan do double duty as the CPU fan.

I ran an experiment to try and determine how fast my fan needed to run to be effective. I wrote a script to collect fan speed and CPU temperature every 2 seconds. After collecting 5-10 minutes of data at idle, I started a program that fully loaded the CPU for several minutes, and continued collecting data until the system returned to a stable idle state. I ran this experiment on the PE 830 (Pentium D 940, 3.2 GHz, 130W TDP) and the PE 840 (Core 2 Duo E6400, 2.13 GHz, 65W TDP).

As it turns out, the BMC isn’t intelligent enough to vary the speed of the fan based on CPU temperature. On the new fan, it drops the fan speed in increments of 75 RPM until it gets below the threshold, then spins it back up to a much higher speed and repeats the process. This cycle is apparently unaffected by CPU temperature:

(Note: The label on the X-axis should read “Time (mm:ss)”, but I’m too tired to go back and change it now. Click on the plots for full-sized versions.)

nightcaller_ms

The CPU temp peaks around 60 C. The heating seems to be fastest when the fan runs below 1000 RPM.

Now, even at high speeds, the new fan is very quiet, so noise is no longer a problem. However, the fan doesn’t cool the CPU effectively at speeds below about 1000 RPM, causing the CPU to heat very quickly if it’s loaded at the lower part of the fan’s cycle. This problem was easy enough to solve though. Instead of hacking BMC firmware hack to lower the speed threshold, I hacked the firmware to raise the threshold to 1000 RPM (the Python script already allowed this).

nightcaller_mm

With the threshold set at just below 1000 RPM, the fan speed kicks back up before the CPU temp can rise too far.

For reference, here’s a plot of the fan behavior with the stock fan and stock firmware. The fan speed really doesn’t vary at all, regardless of CPU temperature.

nightcaller_ss

I also tested out the new fan in the PE 840, and gathered similar results. CPU temperature still doesn’t factor into fan speed.

korolev_ms

korolev_ss

Fan speed is flat. CPU temp isn’t.

I haven’t bought a second fan for the PE 840 yet, and I’m not sure if I will. For some reason, it doesn’t seem as loud as the 830, even though both run the fan at the same speed.

I really wish I knew why the firmware keeps trying to lower the speed on the new fan. My best guess is that Dell’s PWM fans don’t work quite the same way as standard PWM fans (because re-inventing PWM obviously makes sense…).