Servidores NTP não sincronizando na inicialização

Backstory: Eu tenho alguns relógios NTP de inicialização interna 1 com receptores GPS e 2 servidores NTP públicos que são virtualizados em cima do VMware ESXi que tomam tempo dos relógios S1 e o distribuem. Caso contrário, esta configuração funciona muito bem e oferece um bom tempo quando comparado a outros servidores públicos.

Problema: quando reinicio as máquinas virtuais, elas não iniciam a sincronização corretamente e ficam presas em um estado não sincronizado. Abaixo está a saída ntpq -p após uma reinicialização.

root@server:~$ ntpq -p
 remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.1.40    .GPS.            1 u   27   64    3    1.533  -258.43 5948.73
 192.168.2.40    .GPS.            1 u   24   64    3    1.118  -258.47 6138.19
 192.168.3.42    .GPS.            1 u   24   64    3    0.709  -258.42 5655.02
 194.100.49.151  194.100.49.134   2 u   22   64    3    8.124  -258.74 7131.65
 gbg1.ntp.se     .PPS.            1 u   26   64    3   21.856  -258.43 4876.90
 ntp2.sptime.se  .PPS.            1 u   23   64    3   19.991  -258.42 7764.97
 ntp1.sptime.se  .PPS.            1 u   27   64    3   20.489  -258.41 8574.46

Se eu executar a reinicialização do serviço ntp, recebo isso:

root@server:~$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.1.40    .GPS.            1 u    2   64    1    1.517  -258.45   0.065
 192.168.2.40    .GPS.            1 u    1   64    1    1.126  -258.46   0.025
 192.168.3.42    .GPS.            1 u    2   64    1    0.719  -258.42   0.020
 194.100.49.151  194.100.49.134   2 u    5   64    1    8.041  -258.72   0.000
 gbg1.ntp.se     .PPS.            1 u    6   64    1   21.839  -258.41   0.000
 ntp2.sptime.se  .PPS.            1 u    4   64    1   19.968  -258.41   0.000
 ntp1.sptime.se  .PPS.            1 u    3   64    1   20.418  -258.43   0.000

Um segundo depois ele pisa:

root@server:~$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.1.40    .STEP.          16 u    2   64    0    0.000    0.000   0.000
 192.168.2.40    .STEP.          16 u    2   64    0    0.000    0.000   0.000
 192.168.3.42    .STEP.          16 u    8   64    0    0.000    0.000   0.000
 194.100.49.151  194.100.49.134   2 u    -   64    1    7.976   -0.261   0.000
 gbg1.ntp.se     .PPS.            1 u    -   64    1   21.840    0.060   0.000
 ntp2.sptime.se  .STEP.          16 u    6   64    0    0.000    0.000   0.000
 ntp1.sptime.se  .STEP.          16 u    6   64    0    0.000    0.000   0.000

E depois disso voltamos ao funcionamento normal:

root@server:~$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.1.40    .GPS.            1 u    1   64    1    1.474    0.044   0.017
*192.168.2.40    .GPS.            1 u    1   64    1    1.102    0.030   0.005
 192.168.3.42    .GPS.            1 u    1   64    1    0.674    0.049   0.009
 194.100.49.151  194.100.49.134   2 u    8   64    1    7.976   -0.261   0.000
 gbg1.ntp.se     .PPS.            1 u    8   64    1   21.840    0.060   0.000
 ntp2.sptime.se  .PPS.            1 u    6   64    1   19.979    0.059   0.000
 ntp1.sptime.se  .PPS.            1 u    5   64    1   20.440    0.048   0.000

Portanto, parece que após a reinicialização o relógio do sistema está um pouco desligado, o que é esperado, mas por que o ntpd não entra em pânico e apenas avança o relógio é um pouco difícil de entender.

Aqui está o meu ntp.conf

tinker panic 0
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift


# Enable this if you want statistics to be logged.
statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable


# You do need to talk to an NTP server or two (or three).
#server ntp.your-provider.example

# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 192.168.1.40  iburst
server 192.168.2.40 iburst
server 192.168.3.42 iburst
server time1.mikes.fi
server ntp1.gbg.netnod.se
server ntp2.sptime.se
server ntp1.sptime.se

# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
# details.  The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
# might also be helpful.
#
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

# Clients from this (example!) subnet have unlimited access, but only if
# cryptographically authenticated.
#restrict 192.168.123.0 mask 255.255.255.0 notrust


# If you want to provide time to your local subnet, change the next line.
# (Again, the address is an example only.)
#broadcast 192.168.123.255

# If you want to listen to time broadcasts on your local subnet, de-comment the
# next lines.  Please do this only if you trust everybody on the network!
#disable auth
#broadcastclient