You are here: Home > Publications > RIPE Labs > Colin Petrie > How to Convert your TTM Box to a Stratum-1 NTP Server

How to Convert your TTM Box to a Stratum-1 NTP Server

Colin Petrie — Jun 05, 2014 04:55 PM
As part of decommissioning the TTM system, we investigated the possibilities for converting existing TTM boxes to stand-alone NTP servers.

The main reasons for investigating this are listed below:

  • Lots of hosts found one of the most useful features of TTM was having a local stratum-1 NTP server
  • The hosts have already gone to the effort of having the antenna installed on the roof
  • The RIPE NCC card driver has already been added to the upstream NTP distribution
  • It will allow these hosts to take over control of the systems and run them as an NTP server

Hardware Replacement

If you want to replace the hardware, the only real requirement is that the device you choose has an available PCI slot (not PCIe), an available hardware serial port (do not use a USB serial adaptor), and has hardware compatible with either FreeBSD or your chosen Linux distribution.

The TTM card does not interface with the system via the PCI bus - it is only there to obtain power from the bus (via the 12V and 5V power rails on the PCI bus). Therefore the requirement is for a PCI slot with active power rails.

Although most servers today only have PCI-Express ports on them, you may be able to find a model with a PCI slot with your chosen server provider. Alternatively, you could construct an external enclosure for the card which provides a power supply but no data lines.

Software Instructions

FreeBSD

The testing was done using FreeBSD 9.2-RELEASE (i386) as the latest stable version at time of writing

It should be installed with (or added later using sysinstall) the 'src' package, as a kernel recompile should be performed.

Compile a new kernel to enable the kernel PPS_SYNC option:

[root@ttm-test ~]# cd /usr/src/sys/i386/conf
[root@ttm-test /usr/src/sys/i386/conf]# cat TTM-PPSSYNC
 include GENERIC
ident TTM-PPSSYNC
options          PPS_SYNC

Depending on what modules you need for your hardware, you may choose not to bother recompiling all the modules. You should ensure that you compile the 'pps' module, plus any others you need:

[root@ttm-test ~]# kldstat
Id Refs Address    Size     Name
  1    7 0xc0400000 7e2ce4   kernel
  2    1 0xc0be3000 b928     ipmi.ko
  3    2 0xc0bf1000 5c484    acpi.ko
  4    1 0xc0c4e000 4f98     ida.ko
 
[root@ttm-test ~]# cat /boot/loader.conf
 ipmi_load="YES"
ida_load="YES"
[root@ttm-test /usr/src]# cat /etc/make.conf
MODULES_OVERRIDE = pps ipmi acpi ida

Now compile and install your new kernel:

[root@ttm-test /usr/src]# make buildkernel KERNCONF=TTM-PPSSYNC
<snip build process>
[root@ttm-test /usr/src]# make installkernel KERNCONF=TTM-PPSSYNC
<snip install process>

Update your ports tree:

[root@ttm-test ~]# portsnap fetch
[root@ttm-test ~]# portsnap extract
 

Now configure the build for NTP:

[root@ttm-test ~]# cd /usr/ports/net/ntp
[root@ttm-test /usr/ports/net/ntp]# make config

Enable the 'RIPENCC specific Trimble driver' and the 'ATOM PPS interface':

│ │ [x] ATOM         Enable ATOM PPS interface                               │ │  
│ │ [x] RIPENCC      Enable RIPENCC specific Trimble driver                  │ │

Compile NTP:

[root@ttm-test /usr/ports/net/ntp]# make

Install NTP:

root@ttm-test /usr/ports/net/ntp]# make install

Create a log directory:

[root@ttm-test ~]# mkdir /var/log/ntpstats

Put the following config in place:

cat /etc/ntp.conf
[root@ttm-test ~]# cat /etc/ntp.conf
#
# $FreeBSD: release/9.2.0/etc/ntp.conf 239600 2012-08-23 00:39:08Z delphij $
#
# Default NTP servers for the FreeBSD operating system.
#
# Don't forget to enable ntpd in /etc/rc.conf with:
# ntpd_enable="YES"
#
# The driftfile is by default /var/db/ntpd.drift, check
# /etc/defaults/rc.conf on how to change the location.

# Enable this if you want statistics to be logged.
statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
logconfig =all
logfile /var/log/ntpstats/logfile

# RIPE NCC Trimble card driver to enable antenna to send PPS
# signal, and provide the current timecode, and collect PPS via
# the PPSAPI interface
# 43.(X) -> X will be used to define:
# the serial device to access to talk to the GPS receiver (/dev/gpsX)
server 127.127.43.0 prefer      # use RIPENCC driver
fudge 127.127.43.0 refid GPS                        # set the refid name
fudge 127.127.43.0 flag3 1                          # enable kernel PPS sync
# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

Configure the system to use the port version of NTP, not the included system version:

[root@ttm-test ~]# cat /etc/rc.conf 
 <snip>
ntpd_enable="YES"                    # Run ntpd Network Time Protocol (or NO).
ntpd_program="/usr/local/bin/ntpd"   # path to ntpd, if you want a different one.
ntpd_sync_on_start="YES"             # Sync time on ntpd startup, even if offset is high

Instruct devfs to put an appropriate symlink in place for your serial device on boot. Also, manually create this symlink just now:

[root@ttm-test ~]# tail -1 /etc/devfs.conf 
 link    cuau0    gps0
[root@ttm-test ~]# cd /dev/
[root@ttm-test /dev]# ln -s cuau0 gps0

Now you can start NTPd:

[root@ttm-test ~]# /etc/rc.d/ntpd start
Starting ntpd.

You should see antenna initialisation output, followed by periodic timecode updates in the clockstats log:

[root@ttm-test /var/log/ntpstats]# cat clockstats
56758 35170.036 127.127.43.0 FW Versions:  Nav Proc  3.06   3/11/105  Sig Proc 10.04   3/11/105
56758 35170.037 127.127.43.0 Rcvr status1: Doing position fixes (00h); status2: No BBRAM, Ant OK (01h)
56758 35170.038 127.127.43.0 Rcvr Machine ID: 97; Status3 = RTC OK, Alm OK (00h)
56758 35170.038 127.127.43.0 Product ID 8F42
    extension:            0
    case serial # prefix: 0
    case serial #:        82643833
    prod. #:              39091
    premium options:      0h
    machine ID:           97
    key:                  B79h
56758 35170.141 127.127.43.0   search range:          0
   board options:         0
   board serial #:        82643833
   build date/hour:       28/06/05 07:00
   osc offset:            0.000 PPM (0 Hz)
   test code:             0
56758 35170.141 127.127.43.0 I/O Options: 1C  0  5  A
     MSL altitude output (Geoid height)
     MSL altitude input
     Double precision
     Time tags in UTC
     Fixes sent only on request
     PPS output during fixes
     Code-phase smoothed before output
     Signal Strength Output as dBHz
56758 35170.142 127.127.43.0 Channel A Configuration
    Transmit speed: No output at  9600
    Receive speed: TSIP at  9600
    Character format (bits/char, parity, stop bits): 8-ODD-1
56758 35170.142 127.127.43.0 Auto-Report Mask: 00 00 00 00
Auto-Reports scheduled for Output:
Auto-Reports NOT scheduled for Output:
       Almanac  , Ephemeris, UTC      , Iono     , GPS Msg  , Alm Hlth
       Time Fix , SV Select, Ext Event, Pos Fix  , Raw Meas
 56758 35170.239 127.127.43.0    operating mode:      OverDetermined Time
    dynamics:            Static
    elev angle mask:     10 deg
    SNR mask:            4 AMU
    DOP mask:            8
    DOP switch:          6
56758 35170.239 127.127.43.0 PPS is         enabled
    timebase:   UTC
    polarity:   Positive
    offset:     0.0 ns,
    biasunc:    1000.0 ns
56758 35170.239 127.127.43.0 Max # of position fixes for self-survey : 2000
56758 35170.848 127.127.43.0 U1 11.4.2014 11:46:11 13 01
56758 35170.893 127.127.43.0 L1 0 16 16 -1.86265e-09 -3.55271e-15 61440 1788 1694 7
56758 35170.988 127.127.43.0 C1 11042014 114611 6  189255 110.2 556 6.7 16 5222.374603 N 0453.268272 E 57  -7 26 -21 8 5 -15 28 9
56758 35171.033 127.127.43.0 S1 07 1 2 02 34.8  61.8 24.8
56758 35171.060 127.127.43.0 S1 26 2 1 02 56.5 293.3 71.1
56758 35171.097 127.127.43.0 S1 21 3 1 02 41.1 314.4 13.5
56758 35171.133 127.127.43.0 S1 08 4 1 02 59.7  61.2 55.4
56758 35171.161 127.127.43.0 S1 05 5 1 02 58.6 207.9 52.8
56758 35171.198 127.127.43.0 S1 15 6 2 02 32.7 289.4 32.5
56758 35171.225 127.127.43.0 S1 28 7 1 02 53.2 125.2 48.0
56758 35171.258 127.127.43.0 S1 09 8 1 02 66.1  65.2 64.3

While the antenna locks onto the available satellites, it will output timecodes but it may take up to 1 hour before it has trained accurately enough to produce PPS pulses.

During this period, the NTP driver will fail to get a PPS signal, so you will see these messages in /var/log/ntpstats/logfile initially - these messages will stop once the PPS signal is received: 

11 Apr 11:46:09 ntpd[69466]: 0.0.0.0 c016 06 restart
11 Apr 11:46:09 ntpd[69466]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
11 Apr 11:46:10 ntpd[69466]: ripencc_receive(): ripencc_get_pps_ts returns failure
11 Apr 11:46:10 ntpd[69466]: ripencc_receive(): !refclock_process
11 Apr 11:46:26 ntpd[69466]: 0.0.0.0 c41c 0c clock_step +7200.865151 s
11 Apr 13:46:27 ntpd[69466]: 0.0.0.0 c414 04 freq_mode
11 Apr 13:46:27 ntpd[69466]: 0.0.0.0 c418 08 no_sys_peer
11 Apr 13:46:27 ntpd[69466]: ripencc_receive(): ripencc_get_pps_ts returns failure

When it is ready to do so, the PPS pulses will arrive on the DCD pin of the serial port. When this happens, you should see the PPS LED on the PCI card flashing once per second.

You should now see your NTP peer begin to converge on a stable time - the offset and jitter should begin to reduce over time. This may take a few hours to reach full precision:

[root@ttm-test /usr/local/etc]# ntpq -p
      remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*GPS_RIPENCC(0)  .GPS.           0 l    7   16  377    0.000    0.000   0.001

 

Linux distributions

CentOS

We did not develop instructions for CentOS 6 after we realised that some of the earlier TTM systems had only 128M RAM, which is not sufficient to load the Anaconda installer. The listed recommended minimum for a CLI CentOS6 install is 392M.

Debian

Debian was chosen to test the older hardware as it has much lower system requirements for a base installation - particularly memory. The instructions below were tested with Debian 7.4.0 (i386). Note that the earlier TTM hardware will not support the amd64/x86_64 distributions - you must use the i386/i686 variants.

Package needs rebuild to add RIPENCC driver:

root@ttmbuild:~# apt-get install build-essential devscripts
root@ttmbuild:~# apt-get build-dep ntp
root@ttmbuild:~# apt-get install pps-tools
root@ttmbuild:~# mkdir ripencc_build
root@ttmbuild:~# cd ripencc_build/
root@ttmbuild:~/ripencc_build# apt-get source ntp
root@ttmbuild:~/ripencc_build# cd ntp-4.2.6.p5+dfsg/
root@ttmbuild:~/ripencc_build/ntp-4.2.6.p5+dfsg# vi debian/rules

Edit the configure line, to add the flag to build the refclock_ripencc and the Atom/PPS driver:

                --enable-all-clocks --enable-parse-clocks --enable-SHM --enable-RIPENCC --enable-ATOM\

You must now apply a patch to the refclock_ripencc.c driver file, which adds support for the LinuxPPS API. The driver can be downloaded from the LinuxPPS website:

Click here to get the driver patch

Once you have patched the driver, you can now build the package:

root@ttmbuild:~/ripencc_build/ntp-4.2.6.p5+dfsg# debuild -i -us -uc -b
You should now have packages:
root@ttmbuild:~/ripencc_build# ls -al *.deb
-rw-r--r-- 1 root root  588592 Apr 10 10:10 ntp_4.2.6.p5+dfsg-2_i386.deb
-rw-r--r-- 1 root root   80606 Apr 10 10:10 ntpdate_4.2.6.p5+dfsg-2_i386.deb
-rw-r--r-- 1 root root 1142300 Apr 10 10:10 ntp-doc_4.2.6.p5+dfsg-2_all.deb

Copy these packages to your TTM server. Do not install them yet.

On your NTP server, you need to install the LinuxPPS tools:

root@ttm-test:~# apt-get install pps-tools

Next, install udev rules to create the appropriate symlinks, and to attach the PPS line discipline to the serial port (the one you connected the GPS loopback serial cable to):

root@ttm-test:~# cat /etc/udev/rules.d/09.pps.rules
KERNEL=="ttyS0", SYMLINK+="gps0", RUN+="/bin/setserial -v /dev/%k low_latency irq 4", RUN+="/usr/sbin/ldattach pps /dev/%k"
KERNEL=="pps0", OWNER="root", GROUP="tty", MODE="0660", SYMLINK+="gpspps0"

Activate the rules by calling a udev trigger, and check your kernel messages to ensure they are loaded correctly:

root@ttm-test:~# udevadm trigger
root@ttm-test:~# dmesg
[  365.664354] pps_core: LinuxPPS API ver. 1 registered
[  365.664380] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti _at_ linux _dot_ it>
[  365.665841] pps_ldisc: PPS line discipline registered
[  365.706785] pps pps0: new PPS source serial0
[  365.706846] pps pps0: source "/dev/ttyS0" added
root@ttm-test:~# ls -al /dev/*[gp]ps*
lrwxrwxrwx 1 root root      5 Apr 10 15:59 /dev/gps0 -> ttyS0
lrwxrwxrwx 1 root root      4 Apr 10 15:59 /dev/gpspps0 -> pps0
crw-rw---T 1 root tty  251, 0 Apr 10 15:59 /dev/pps0

Now put the following NTP configuration in place (in /etc/ntp.conf)

root@ttm-test:~# cat /etc/ntp.conf
 
driftfile /var/lib/ntp/ntp.drift
# Enable this if you want statistics to be logged.
statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
logconfig =all
logfile /var/log/ntpstats/logfile

# RIPE NCC Trimble card driver to enable antenna to send PPS
# signal, and provide the current timecode, and collect PPS via
# the LinuxPPS interface
# 43.(X) -> X will be used to define:
# the serial device to access to talk to the GPS receiver (/dev/gpsX)
# the PPS device which receives the timing pulses (/dev/gpsppsX)
server 127.127.43.0 prefer    # RIPENCC driver
fudge 127.127.43.0 refid GPS
# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1
 

You can now install the custom-build package - which will automatically start the ntp server:

root@ttm-test:~# dpkg -i ntp_4.2.6.p5+dfsg-2_i386.deb 
Selecting previously unselected package ntp.
(Reading database ... 32562 files and directories currently installed.)
Unpacking ntp (from ntp_4.2.6.p5+dfsg-2_i386.deb) ...
Setting up ntp (1:4.2.6.p5+dfsg-2) ...
[ ok ] Starting NTP server: ntpd.
Processing triggers for man-db ...

You should also pin the package so a system update does not overwrite the custom-compiled version - of course this means you must be aware of any future security updates to this package and have a process to rebuild a new custom-compiled version each time:

root@ttm-test:~# echo "ntp hold" | dpkg --set-selections
root@ttm-test:~# dpkg --get-selections | grep ntp
ntp                                                   hold

You should see antenna initialisation output, followed by periodic timecode updates in the clockstats log:

56757 51930.862 127.127.43.0 FW Versions:  Nav Proc  3.06   3/11/105  Sig Proc 10.04   3/11/105
56757 51930.867 127.127.43.0 Rcvr status1: Only 2 satellites usable (0Ah); status2: No BBRAM, Ant OK (01h)
56757 51930.871 127.127.43.0 Rcvr Machine ID: 97; Status3 = RTC OK, No Alm (08h)
56757 51930.899 127.127.43.0 Product ID 8F42
    extension:            0
    case serial # prefix: 0
    case serial #:        82643833
    prod. #:              39091
    premium options:      0h
    machine ID:           97
    key:                  B79h
56757 51930.926 127.127.43.0   search range:          0
   board options:         0
   board serial #:        82643833
   build date/hour:       28/06/05 07:00
   osc offset:            0.000 PPM (0 Hz)
   test code:             0
56757 51930.935 127.127.43.0 I/O Options: 1C  0  5  A
     MSL altitude output (Geoid height)
     MSL altitude input
     Double precision
     Time tags in UTC
     Fixes sent only on request
     PPS output during fixes
     Code-phase smoothed before output
     Signal Strength Output as dBHz
56757 51930.945 127.127.43.0 Channel A Configuration
    Transmit speed: No output at  9600
    Receive speed: TSIP at  9600
    Character format (bits/char, parity, stop bits): 8-ODD-1
56757 51930.954 127.127.43.0 Auto-Report Mask: 00 00 00 00
Auto-Reports scheduled for Output:
Auto-Reports NOT scheduled for Output:
       Almanac  , Ephemeris, UTC      , Iono     , GPS Msg  , Alm Hlth
       Time Fix , SV Select, Ext Event, Pos Fix  , Raw Meas
56757 51931.009 127.127.43.0    operating mode:      3-D
    dynamics:            Land
    elev angle mask:     10 deg
    SNR mask:            4 AMU
    DOP mask:            8
    DOP switch:          6
56757 51931.036 127.127.43.0 PPS is         enabled
    timebase:   UTC
    polarity:   Positive
    offset:     0.0 ns,
    biasunc:    1000.0 ns
56757 51931.043 127.127.43.0 Max # of position fixes for self-survey : 2000
56757 51931.633 127.127.43.0 U1 10.4.2014 14:25:31 9 01
56757 51931.688 127.127.43.0 L1 0 16 16 0 1.77636e-15 589824 1787 1694 7
56757 51947.635 127.127.43.0 U1 10.4.2014 14:25:47 9 01
56757 51963.636 127.127.43.0 U1 10.4.2014 14:26:03 9 01
56757 51979.634 127.127.43.0 U1 10.4.2014 14:26:19 9 01
56757 51995.635 127.127.43.0 U1 10.4.2014 14:26:35 9 01
56757 52011.637 127.127.43.0 U1 10.4.2014 14:26:51 9 01
56757 52027.635 127.127.43.0 U1 10.4.2014 14:27:07 9 01
56757 52043.633 127.127.43.0 U1 10.4.2014 14:27:23 9 01
56757 52059.633 127.127.43.0 U1 10.4.2014 14:27:39 9 01

While the antenna locks onto the available satellites, it will output timecodes but it may take up to 1 hour before it has trained accurately enough to produce PPS pulses.

During this period, the NTP driver will fail to get a PPS signal, so you will see these messages in /var/log/ntpstats/logfile initially - these messages will stop once the PPS signal is received:

10 Apr 16:25:31 ntpd[2243]: ripencc_receive(): ripencc_get_pps_ts returns failure
10 Apr 16:25:31 ntpd[2243]: ripencc_receive(): !refclock_process
10 Apr 16:25:47 ntpd[2243]: ripencc_receive(): ripencc_get_pps_ts returns failure
10 Apr 16:25:47 ntpd[2243]: ripencc_receive(): !refclock_process

When it is ready to do so, the PPS pulses will arrive on the DCD pin of the serial port, be received by the line discipline LinuxPPS driver, and be presented on /dev/pps0. When this happens, you should see the PPS LED on the PCI card flashing once per second. You can also check that PPS is being correctly received by the kernel by using the pps-tools commands.

Before PPS is available:

root@ttm-test:~# ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
time_pps_fetch() error -1 (Connection timed out)
time_pps_fetch() error -1 (Connection timed out)
time_pps_fetch() error -1 (Connection timed out)
time_pps_fetch() error -1 (Connection timed out)

In addition, your NTP server will not yet show a synchronised peer:

root@ttm-test:~# ntpq -p
      remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
  GPS_RIPENCC(0)  .GPS.           0 l    -   16    0    0.000    0.000   0.000

After PPS becomes available:

root@ttm-test:~# ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
time_pps_fetch() error -1 (Connection timed out)
time_pps_fetch() error -1 (Connection timed out)
time_pps_fetch() error -1 (Connection timed out)
time_pps_fetch() error -1 (Connection timed out)
time_pps_fetch() error -1 (Connection timed out)
time_pps_fetch() error -1 (Connection timed out)
source 0 - assert 1397140681.267546064, sequence: 1 - clear  0.000000000, sequence: 0
source 0 - assert 1397140682.267547022, sequence: 2 - clear  0.000000000, sequence: 0
source 0 - assert 1397140683.267551381, sequence: 3 - clear  0.000000000, sequence: 0
source 0 - assert 1397140684.267554021, sequence: 4 - clear  0.000000000, sequence: 0

Once the PPS inputs are being received, check your NTP server status - it may take some time to stabilise, but your stratum 0 peer should become active (as indicated by the '*' flag - this flags the main system peer):

root@ttm-test:~# ntpq -p
      remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*GPS_RIPENCC(0)  .GPS.           0 l    2   16  377    0.000    0.000   0.002

Congratulations. You now have a Stratum 1 NTP server up and running for you to use as you wish.

3 Comments

User Image
Rob Turner says:
Jul 08, 2014 03:37 PM
My PPS light never flashes. I think I have a hardware fault. I followed the Debian based instructions, all complied/patched ok but no signs of life, even after several hours. ppstest constantly returns the timeout message above.
As the PCI card is used for power only am I correct in assuming even with a configuration error given time the PPS light on the back should blink 1/s as it did prior to our ttm being decommissioned?
User Image
Colin Petrie says:
Jul 08, 2014 04:13 PM
Hi Rob,

Do you see the antenna initialisation output in the clockstats logfile as shown, followed by timecodes being logged every 15 seconds or so?

The PCI card provides power but also provides a pass-through for the serial cable - the serial interface is still required to program the antenna to tell it to send the PPS pulses.

So you need to look in the initialisation log messages for "PPS output during fixes" and "PPS is enabled". And you should receive the periodic timecodes in the log as well (even before the PPS signals start arriving)

Regards
Colin
User Image
Rob Turner says:
Jul 08, 2014 04:33 PM
Hi Colin, no I'm not seeing anything in Clockstats, which is why I feel it's a hardware issue. The three other lights come on and stay on (2x voltage ones, 1x GPS) but these are on regardless of the GPS receiver being plugged in.
Add comment

You can add a comment by filling out the form below. Only plain text is possible. Web and email addresses will be transformed into clickable links. Comments are moderated so they won't appear immediately.