Fixing ‘Clocksource tsc unstable’ warnings on older hardware that result in a system crash

10 March 2012 — 17 Comments

We still have some older hardware running for non-critical and lightweight operations. Since upgrading these old boxes to Debian Squeeze weird things started to happen. After some time, the system clock stops working and that results in processes to crash and new ones unable to start. Of course, nothing works properly on any computer without a working system clock. Here you see how that looks like:

First thought was to replace the CMOS battery, but that didn’t help and it occurred on multiple machines around the same time. Rebooting solves the problem for a while but 24-36 hours later the system clock stops again. Sometimes the problem stays away much longer. Also, some servers with the same hardware do not seem to have the problem. It seems hardware related. But is that really true?

Since the server crashes it is kinda hard to figure out what exactly happens. Logs aren’t properly written anymore and system processes are unreliable. Sometimes it is possible to access the server via SSH, sometimes it isn’t.

Today this happened again with one of our servers. Fortunately I was able to SSH into the machine and I found this warning in the logs:

Mar  14 13:00:05 server kernel: [97383.660485] Clocksource tsc unstable (delta = 4686838547 ns)

TSC is the Time Stamp Counter. Processors have dynamically changed clock speed (ofc to save power). The TSC is supposed to tick at the CPU rate so on frequency change, this ought to happen. The kernel will automatically switch to something else. This is true for modern hardware, this old CPU is set to constant_tsc (when using cat /proc/cpuinfo it’s one of the flags). It is thus supposed to run at the same frequency at all times.

The system might have multiple available clock sources. You can list the available clock sources in a system like this:

cat /sys/devices/system/clocksource/clocksource0/available_clocksource

On my system this returns:

tsc acpi_pm jiffies

To see which one is actually in use, issue:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

On my system this returns:

tsc

So this tells me I’m using ‘tsc’ as a Clock Source, and my system has two more options. Since I’m having trouble with ‘tsc’, let’s change the Clock Source from ‘tsc’ to ‘acpi_pm’.

echo acpi_pm > /sys/devices/system/clocksource/clocksource0/current_clocksource

Verify the new setting like this:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

It should now return ‘acpi_pm”.

This can be done at run time, when you reboot the setting is lost. It is a great way to test the setting. To make it permanent, add this line to the boot parameters in /boot/grub/menu.lst:

notsc clocksource=acpi_pm

I’ve just changed this on the server and I’m really curious whether this will be the solution to prevent the hangs! At least the time is still ticking.. 😉

If anyone has more suggestions or info, please let me know! I’ll keep you posted.

Update: Just reached 4 days of uptime!

Update 2: Today it’s 4 month’s after I wrote this blog. The machine is still up & running and has reached 106 days of uptime!

Update 3: After 7 months (222 days of uptime) I finally retired the machine since we migrated to our CloudStack cloud. The fix described above really works 🙂

17 responses to Fixing ‘Clocksource tsc unstable’ warnings on older hardware that result in a system crash

  1. 

    thank you remi, solved my problem as well 🙂

  2. 

    I am having similar problem but this time is tsc it jiffies an unfortunetley i can not change to acpi_pm. Any help?

    • 

      when the system is crashed then i got this message: clocksource tsc unstable (delta = 107974825 ns)
      Please can you advise any help? i’ll be appreciate it. Thanks
      Thomas

      • 

        Hi Thomas, when you know what clock sources are available on our system, I’d advise you to try them one-by-one and see which one works best. In my case the best one was ‘acpi_pm’. I’m not saying that one always works best. Let me know if one of the other options solves your problem. Good luck!

  3. 

    I can’t get into too much detail but I’m having this issue on some systems I maintain at work. I’m a bit of a noob when it comes to linux but I know they boot using PXE. The operating system they load is debian, but there’s no direct way to interact with them. I can plug in a monitor and keyboard, thus how I know this is the issue that’s been causing some of our issues. However, nothing I type is actually executed.

    With that in mind, where would I change the clocksource, if there were any others available?

    • 

      You can also supply this as a parameter when you boot Linux. Append ‘clocksource=acpi_pm’ to the cmd-line. Replace ‘acpi_pm’ with the clocksource you want to use. You probably need to edit this in the PXE environment you use to boot. Good luck!

  4. 

    Thanks! It helps me migrate centos from vmware to virtual pc.

  5. 

    VPC in XP is my favourite virtual machine. I wanted to look how centos works in VPC, but it worked with errors and slowly. Alas, I refuse from centos in VPC. I only start to study linux. Bad knowledge of English the big problem for me.

  6. 

    Is there a RedHat Bug ID for this issue?

  7. 

    hello! for me is not work to change in hpet or acpi_pm 😦
    bash: /sys/devices/system/clocksource/clocksource0/current_clocksource: Permission denied
    what can i do!! please help me! thank you!!

  8. 

    … similar problem.
    echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
    Then when i did
    cat /sys/devices/system/clocksource/clocksource0/current_clocksource it turns out that somehow i get acpi_pm
    Any solutions ??

    • 

      I came accoss this post while i was searching for a fix to my problem , im runing vertialbox with kali as a VM, after booting it stops at tsc like there is something wrong with ,, can you help please

    • 

      You probably don’t have the option of tsc from issuing the command:
      cat /sys/devices/system/clocksource/clocksource0/available_clocksource

      Just a guess that is.

What do you think?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s