x86/nmi: start NMI watchdog on CPU0 after SMP bootstrap

We're noticing a reproducible system boot hang on certain Skylake platforms where the BIOS is configured in legacy boot mode with x2APIC disabled. The system stalls immediately after writing the first SMP initialization sequence into APIC ICR. The cause of the problem is watchdog NMI handler execution - somewhere near the end of NMI handling (after it's already rescheduled the next NMI) it tries to access IO port 0x61 to get the actual NMI reason on CPU0. Unfortunately, this port is emulated by BIOS using SMIs and this emulation for some reason takes more time than we expect during INIT-SIPI-SIPI sequence. As the result, the system is constantly moving between NMI and SMI handler and not making any progress. To avoid this, initialize the watchdog after SMP bootstrap on CPU0 and, additionally, protect the NMI handler by moving IO port access before NMI re-scheduling. The latter should also help in case of post boot CPU onlining. Although we're running watchdog at much lower frequency at this point, it's neveretheless possible we may trigger the issue anyway. Signed-off-by: Igor Druzhinin <[email protected]> Reviewed-by: Jan Beulich <[email protected]>
microslicedcore · Feb 20, 2018 · a44f169 · a44f169
1 parent 682b13c
commit a44f169
Show file tree

Hide file tree

Showing 3 changed files with 15 additions and 3 deletions.
diff --git a/xen/arch/x86/apic.c b/xen/arch/x86/apic.c
@@ -684,7 +684,7 @@ void setup_local_APIC(void)
         printk("Leaving ESR disabled.\n");
     }
 
-    if (nmi_watchdog == NMI_LOCAL_APIC)
+    if (nmi_watchdog == NMI_LOCAL_APIC && smp_processor_id())
         setup_apic_nmi_watchdog();
     apic_pm_activate();
 }

diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c
@@ -1293,7 +1293,10 @@ int __cpu_up(unsigned int cpu)
 void __init smp_cpus_done(void)
 {
     if ( nmi_watchdog == NMI_LOCAL_APIC )
+    {
+        setup_apic_nmi_watchdog();
         check_nmi_watchdog();
+    }
 
     setup_ioapic_dest();
 

diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
@@ -1706,22 +1706,31 @@ static nmi_callback_t *nmi_callback = dummy_nmi_callback;
 void do_nmi(const struct cpu_user_regs *regs)
 {
     unsigned int cpu = smp_processor_id();
-    unsigned char reason;
+    unsigned char reason = 0;
     bool handle_unknown = false;
 
     ++nmi_count(cpu);
 
     if ( nmi_callback(regs, cpu) )
         return;
 
+    /*
+     * Accessing port 0x61 may trap to SMM which has been actually
+     * observed on some production SKX servers. This SMI sometimes
+     * takes enough time for the next NMI tick to happen. By reading
+     * this port before we re-arm the NMI watchdog, we reduce the chance
+     * of having an NMI watchdog expire while in the SMI handler.
+     */
+    if ( cpu == 0 )
+        reason = inb(0x61);
+
     if ( (nmi_watchdog == NMI_NONE) ||
          (!nmi_watchdog_tick(regs) && watchdog_force) )
         handle_unknown = true;
 
     /* Only the BSP gets external NMIs from the system. */
     if ( cpu == 0 )
     {
-        reason = inb(0x61);
         if ( reason & 0x80 )
             pci_serr_error(regs);
         if ( reason & 0x40 )