Hacking Fan Speed on Dell PowerEdge Servers

A couple of weeks ago, I acquired three old Dell servers to play with: a PowerEdge 840, PowerEdge 830, and a PowerEdge SC 430. One thing I didn’t think about before I got them was how much noise they might make. I’m used to desktop machines; they’re designed to run quietly enough not to annoy someone trying to work in the same room. Dell doesn’t go to the same trouble when they design their servers. The SC 430 is reasonably quiet (Dell based it on their Precision platform), but the other two, the 830 and 840, are just loud enough to be annoying.

My first thought was that I might be able to control the fan speed, but fan speed is firmly under the control of the BMC (Baseboard Management Controller); I can’t control it from software. I determined after some Googling that most people solve the problem by replacing the system fans with slower, quieter, models. Unfortunately, I also found that the slower fans often trigger the “Lower Critical Threshold” (they spin too slow), causing the BMC to spin them up, which gets me back to the noise problem I had in the first place.

Of course, because this is the Internet and I’m not the only hacker who likes to play with hardware, someone else had already solved this problem.

TLDR; the thresholds in used by the BMC can only be changed by hacking the BMC firmware update package. I downloaded that guy’s Python script and ran it on my PE 830. The script was able to parse the BMC firmware update for the PE 830, so I went ahead and ordered a new fan.

new_old

Now, before I could replace the fan, I had to deal with Dell’s custom pinout (because their 4-pin arrangement is clearly superior to the standard 4-pin arrangement that carries the exact same signals). This is already documented in several places on the web, but just to get it up here one more time:

Signal Dell Color Standard Color
PWM Blue Blue
RPM Yellow Green
+12V Red Yellow
Ground Black Black

I lopped the connector off of the original fan (at least it’s not a non-standard pinout on a standard connector). I couldn’t shove the stranded wire into the new fan’s connector, so I soldered some solid wire from a bit of spare CAT 6 cable onto the leads. I pushed the wire all the way through the connector and bent it over, which should keep everything in place.

solid_copper

Before clamping and soldering.

wired

I plugged it in and started the machine. Good news: it was nearly silent. Bad news: loading the machine runs the core temp up to 60C (Intel says to keep it below 63). The fan I bought pushes 74 CFM at full speed; the OEM fan managed 150 CFM. That wouldn’t be a problem, except that Dell was cheap on all of their tower chassis in the mid-2000s and made the back case fan do double duty as the CPU fan.

I ran an experiment to try and determine how fast my fan needed to run to be effective. I wrote a script to collect fan speed and CPU temperature every 2 seconds. After collecting 5-10 minutes of data at idle, I started a program that fully loaded the CPU for several minutes, and continued collecting data until the system returned to a stable idle state. I ran this experiment on the PE 830 (Pentium D 940, 3.2 GHz, 130W TDP) and the PE 840 (Core 2 Duo E6400, 2.13 GHz, 65W TDP).

As it turns out, the BMC isn’t intelligent enough to vary the speed of the fan based on CPU temperature. On the new fan, it drops the fan speed in increments of 75 RPM until it gets below the threshold, then spins it back up to a much higher speed and repeats the process. This cycle is apparently unaffected by CPU temperature:

(Note: The label on the X-axis should read “Time (mm:ss)”, but I’m too tired to go back and change it now. Click on the plots for full-sized versions.)

nightcaller_ms

The CPU temp peaks around 60 C. The heating seems to be fastest when the fan runs below 1000 RPM.

Now, even at high speeds, the new fan is very quiet, so noise is no longer a problem. However, the fan doesn’t cool the CPU effectively at speeds below about 1000 RPM, causing the CPU to heat very quickly if it’s loaded at the lower part of the fan’s cycle. This problem was easy enough to solve though. Instead of hacking BMC firmware hack to lower the speed threshold, I hacked the firmware to raise the threshold to 1000 RPM (the Python script already allowed this).

nightcaller_mm

With the threshold set at just below 1000 RPM, the fan speed kicks back up before the CPU temp can rise too far.

For reference, here’s a plot of the fan behavior with the stock fan and stock firmware. The fan speed really doesn’t vary at all, regardless of CPU temperature.

nightcaller_ss

I also tested out the new fan in the PE 840, and gathered similar results. CPU temperature still doesn’t factor into fan speed.

korolev_ms

korolev_ss

Fan speed is flat. CPU temp isn’t.

I haven’t bought a second fan for the PE 840 yet, and I’m not sure if I will. For some reason, it doesn’t seem as loud as the 830, even though both run the fan at the same speed.

I really wish I knew why the firmware keeps trying to lower the speed on the new fan. My best guess is that Dell’s PWM fans don’t work quite the same way as standard PWM fans (because re-inventing PWM obviously makes sense…).

15 thoughts on “Hacking Fan Speed on Dell PowerEdge Servers

  1. kevin

    Hi there, since this is a recent post of yours, I’ll explain how I got around the proprietary dell stuff on my poweredge 840.

    I chose to go with a non-pwm fan. I used an antec tricool. twist tied it in place. And ran the wires into an extra connector that I have (i believed I salvaged it from an old sound/audio cable that was 4 pin that was thin enough to fit into the dell motherboard headers).

    I also got an antec spot cool and mounted it inside.

    Now, I have the tricool on medium settings and the spot cool acts a s push/pull fan on the cpu cooler.

    seems to work ok .

    Reply
  2. Frank Roberts Post author

    I chose a PWM fan was because I assumed that the BIOS/BMC firmware controlled the speed in an intelligent way. Looking at the data, that’s clearly not true, so I guess a PWM fan isn’t really getting me anything.

    For some reason, it never occurred to me to attach a fan to the CPU cooler. I guess that takes care of the problem pretty directly. I still haven’t done anything to my 840 (just the 830), perhaps I’ll try that approach on the 840.

    I haven’t had any problems with my setup so far, but I also haven’t loaded it much.

    Reply
  3. Maren

    I have a 840 also and find it is quite responsive at turning the fans up and down when I put the some load on the CPU. That maybe another issue worth dealing with later…. but I digress.

    I’ve not done any testing on mine yet but what you say does not make sense. To me the heat sinks on your machine is not working.

    I can’t see how it is possible that the massive metalic tower grill in the picture on this page will not respond to higher air circulation. What you are saying violates the laws of thermo dynamics and or simply the heat sink is not big enough.
    http://store.flagshiptech.com/dell-poweredge-700-830-840-1800-1600sc-2600-processor-cpu-heatsink-8×169/

    That thing is one of the biggest things I’ve seen in the market and with the amount of air pumping out of the 840 it is impossible for it not be work.

    I suggest that you take the heat sink off and remove the heat sink grease and reapply it….

    Reply
  4. Frank Roberts Post author

    The plots clearly show that the temperature responds to increased fan speed. Watch what happens to the temperature when the fan speed kicks up after hitting the minimum.

    The problem isn’t that the heatsink doesn’t work, the problem is that, at least on my systems, the CPU temperature has no effect on fan speed. The stock fan runs at 1800 RPM all the time, but the aftermarket fan constantly tries to slow down until it hits the minimum speed.

    Reply
  5. Maren

    Only in one chart the RPM have a little effect….. for the rest when the RPM goes up, the temperature does what ever it does…. Even your PWN fan the RPMs increase drop do nothing to the temperature. Your stock fan the RPMs are constant and seem high to me, and have no impact on the temperature….

    why is that? COuld it be that the RPMs don’t provide any cooling what so ever? Could that be because your cpu is isolated from your heat sink?

    My machine was using Xeon 3040 1.86Ghz CPU turns on the fans very promptly…. and turns them off when the load drops…. (now this could be an annoyance dealing with in itself but e will leave that for later).

    I’ve upgraded the bios and the cpu to a Xeon 3060 put in new grease, I’ve now got a use power management under the CPU menu in the BIOS. Whilst maxing out the cpu, the heat sink is ice cold or the fans stay slow speed even though the cpu is 2.4Ghz

    Reply
  6. John

    Any idea if this concept works on the precision R5500? I know it has custom snap in fans so thinking they can’t be replaced. No actual wires to splice, so chances are would have to stick with th given modular fans. I have four of these units and thinking the noise in my house will be overwhelming.

    JR

    Reply
    1. Frank Roberts Post author

      You could try adjusting the thresholds, but I doubt that will help much with the noise. The site I link to has a note that you really need to replace the fans if you want to reduce the noise. The fans in server chassis aren’t build with noise in mind, as they’re often drowned out by the machine room’s air conditioning anyway.

      Reply
  7. Frank Roberts Post author

    It looks like you could probably remove the connector from one of the existing fans and attach it to a new fan, if you can find fans that are the same form factor.

    Reply
  8. John

    Have some BlackNoise PWM fans on the way. Hoping they will work. trying to find the, in my case, 4-pin Dell connectors with the locking clips on them. Was thinking instead of hacking fans and soldering them back together, would create an extension cable, one side the male 4 pin connector to connect to the BlackNoise fans, the other side to a new Dell 4-pin connector. I know all articles out there mention the 5-pin Dell connector that the last pin isn’t used. Mine only has 4. With this method if I get the pins wrong can swap them around easy by popping out the ends and adjusting. Even a really think connector may work if not the Dell specific one.

    JR

    Reply
  9. John

    Yes, I saw that adapter. Comes real close other than not sure how they have the wires pinned out. What. Is close is everyone talks about the Dell 5-pin connector with four wires used and the lock mechanism. Mine has 4 physical pins on the connector. So would need something like that, but with both sides having four pins. The 5 pin connection side wouldn’t fit into the motherboard connector on the power supply as it’s to wide.

    JR

    Reply
    1. Kevin Guerra

      You could get those at fry’s, but you’d have to cut wires and connect them as needed. Or if you can find a soundblaster to cd rom audio cable, but it only has 3 wires, so you need to add the fourth pin. That’s what I did to my rig.

      Reply
  10. Dan

    Hello!

    This helped me with quieting a T420 I had, thanks! The pins were actually lined up so I just had to make an extension cable!

    Dan

    Reply
  11. Maurice

    I investigated driving the pwm pins from an arduino with uln2003 driver it worked fine and gives the micro almost full control. What i have learned is very interesting the minimum fan speed that can be set is 3000rpm ie cutting the blue wire and connecting the blue wire from the fan to the black wire sets the fan speed at 3000rpm. Which is above the minimum required by the cpu. still alittle noisy but functional with no parts required.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *