ソースを参照

Add ACPI/MADT SMP with a big kernel lock

- Discover CPUs through RSDP -> RSDT/XSDT -> MADT and start the APs
  with INIT-SIPI-SIPI through a trampoline at 0x6000.  There is no
  MP-table fallback; a machine without ACPI, a usable MADT, the CPUID
  APIC bit or a LAPIC stays UP with the 8259 PIC + PIT untouched.
- Give each AP its own GDT/TSS/IDT, SSE CR4 state, LAPIC timer and idle
  stack.  The BSP keeps the global clock and device IRQs; APs mask their
  LINT inputs and use only timer/IPI vectors.
- Protect the global run/sleep/wait queues with an IRQ-safe scheduler
  lock and keep it held across the switch.S stack switch, so a process
  is never picked by another CPU while its kernel stack is still in use.
  Add per-CPU timer entries, reschedule IPIs (with LAPIC EOI) and
  cross-CPU kill/wake handling.
- Serialize kernel execution with the sleeping big kernel lock: every
  syscall and kernel thread takes it, voluntary blocks release it, and
  long serial/network waits use bkl_pause/bkl_resume.  Kernel threads
  stay pinned to the BSP; ring-3 processes may run on any CPU.
- Let APs take user work only after boot via smp_enable_sched(), and
  INIT-park them before kexec so the old kernel image can be overwritten
  safely; the new kernel starts them again.
- Add make run-smp, update AGENTS.md, and make the sched/top idle
  percentages aggregate across all CPUs.
Johannes Zellner 2 週間 前
親
コミット
747a3423ea
27 ファイル変更、1776 行追加、150 行削除
  1. 69 6
      AGENTS.md
  2. 36 9
      Makefile
  3. 314 0
      acpi.c
  4. 22 0
      acpi.h
  5. 86 0
      ap_tramp.S
  6. 16 0
      ap_tramp.ld
  7. 16 0
      cpu.c
  8. 5 0
      cpu.h
  9. 40 2
      kernel.c
  10. 6 0
      kexec.c
  11. 184 0
      lapic.c
  12. 24 0
      lapic.h
  13. 5 0
      paging.c
  14. 4 0
      paging.h
  15. 175 75
      process.c
  16. 10 0
      process.h
  17. 296 49
      sched.c
  18. 40 4
      sched.h
  19. 13 3
      serial.c
  20. 237 0
      smp.c
  21. 22 0
      smp.h
  22. 54 0
      spinlock.h
  23. 69 0
      switch.S
  24. 5 0
      sys_net.c
  25. 21 1
      syscall.c
  26. 2 1
      ucmd_top.c
  27. 5 0
      wm.c

+ 69 - 6
AGENTS.md

@@ -11,6 +11,7 @@ Toolchain: `gcc` (32-bit multilib), `as`/`ld`/`objcopy`, `mtools` (`mcopy`/`mkfs
 ```sh
 ```sh
 make all        # floppy.img usb.img final.bin root.img
 make all        # floppy.img usb.img final.bin root.img
 make run        # boot floppy.img + root.img in QEMU (fallback: qemu-system-x86_64)
 make run        # boot floppy.img + root.img in QEMU (fallback: qemu-system-x86_64)
+make run-smp    # same with -smp 4 (BSP + 3 APs, the Atom 330 topology)
 make run-usb    # boot usb.img as an IDE disk
 make run-usb    # boot usb.img as an IDE disk
 make run-uhci   # like `make run` but with a UHCI controller + USB mouse
 make run-uhci   # like `make run` but with a UHCI controller + USB mouse
 make run-net    # like `make run` + a PCI NE2000 NIC on QEMU's user network (slirp; DHCP/DNS/NAT)
 make run-net    # like `make run` + a PCI NE2000 NIC on QEMU's user network (slirp; DHCP/DNS/NAT)
@@ -33,10 +34,17 @@ make clean
    picks best VBE mode (≥16 bpp, LFB, fits VRAM, ≤1280px), writes mode info to
    picks best VBE mode (≥16 bpp, LFB, fits VRAM, ≤1280px), writes mode info to
    `0x600`, enables A20, enters protected mode, jumps to `kernel_entry`
    `0x600`, enables A20, enters protected mode, jumps to `kernel_entry`
    (`entry.S`, re-points ESP to the current stack top, then `kernel_main`).
    (`entry.S`, re-points ESP to the current stack top, then `kernel_main`).
-3. `kernel.c` — `kernel_main` zeroes BSS then inits serial → I/O → idle → timer
-   (PIT 1 kHz + IDT) → mouse → processes → VGA shadow buffer → terminal.
+3. `kernel.c` — `kernel_main` zeroes BSS then inits serial → memory/paging →
+   ACPI/MADT → I/O → idle → processes → timer (PIT 1 kHz + IDT) → SMP AP
+   bring-up → mouse → VGA shadow buffer → terminal.
 4. `mbr.S` — MBR (USB/HDD boot); loads partition boot sector at LBA 1.
 4. `mbr.S` — MBR (USB/HDD boot); loads partition boot sector at LBA 1.
 
 
+SMP is ACPI-only: RSDP → RSDT/XSDT → MADT, then `LAPIC` + `INIT-SIPI-SIPI`.
+There is deliberately no Intel MP-table parser.  If ACPI, a usable MADT, the
+CPUID APIC bit or the LAPIC is missing, `cpu_count` stays 1 and the machine
+boots exactly like before (8259 PIC + PIT, no APIC MSR/MMIO access).  This keeps
+a Pentium 1 (and a Pentium II board without ACPI) on the UP path.
+
 ## Hardcoded memory map (fragile — keep in sync when editing)
 ## Hardcoded memory map (fragile — keep in sync when editing)
 
 
 | Address    | Purpose |
 | Address    | Purpose |
@@ -49,6 +57,9 @@ make clean
 | `0x1000`  | E820 memory map (`struct e820_map`, written by `vesa_stage.S`, parsed by `mm.c`); also the kexec trampoline's private stack top (grows down, below the map) |
 | `0x1000`  | E820 memory map (`struct e820_map`, written by `vesa_stage.S`, parsed by `mm.c`); also the kexec trampoline's private stack top (grows down, below the map) |
 | `0x3000`  | UHCI frame list (1024 × 4 B, 4 KB-aligned; `uhci.c`) |
 | `0x3000`  | UHCI frame list (1024 × 4 B, 4 KB-aligned; `uhci.c`) |
 | `0x4000`  | ISA DMA buffer, one sector (`disk.c`) |
 | `0x4000`  | ISA DMA buffer, one sector (`disk.c`) |
+| `0x6000`  | AP real/protected-mode trampoline (`ap_tramp.S`/`ap_tramp.ld`; SIPI vector 0x06, image ends below `0x6C00`) |
+| `0x6C00`  | AP trampoline real-mode stack top (grows down) |
+| `0x6F00`  | AP startup info block (CR3, stack top, C entry, CPU index) |
 | `_end`    | Kernel end (linker symbol; `kernel.ld` ASSERTs it stays below `0x7F000`) |
 | `_end`    | Kernel end (linker symbol; `kernel.ld` ASSERTs it stays below `0x7F000`) |
 | (in-image) | Bootstrap/protected-mode stack: 8 KB `.stack` NOLOAD section, `__boot_stack_top`/`__boot_stack_bottom` (travels with the kernel) |
 | (in-image) | Bootstrap/protected-mode stack: 8 KB `.stack` NOLOAD section, `__boot_stack_top`/`__boot_stack_bottom` (travels with the kernel) |
 | `0x7FF00` | Program API struct (`PROGRAM_API_ADDR`, `run_api.h`) |
 | `0x7FF00` | Program API struct (`PROGRAM_API_ADDR`, `run_api.h`) |
@@ -68,11 +79,39 @@ below the run area), so adding code no longer moves any address.
   (no polling scans), stable per-PCB objects in a sparse `PROC_MAX` pointer
   (no polling scans), stable per-PCB objects in a sparse `PROC_MAX` pointer
   table, per-CPU `struct cpu`/GDT/TSS with `%gs`-based `this_cpu()`, and a
   table, per-CPU `struct cpu`/GDT/TSS with `%gs`-based `this_cpu()`, and a
   per-CPU `current` (no global `current_pid`).  Idle is not a process slot; a
   per-CPU `current` (no global `current_pid`).  Idle is not a process slot; a
-  normal `kworker` process does the housekeeping and the zombie reaper.  Ring 0
-  is preemptible at tick boundaries unless `preempt_count` is set (FAT and the
-  shadow flush), and voluntary blocks/yields go through the `int 0x81` gate.
+  normal `kworker` process does the housekeeping and the zombie reaper.
+  `sched_lock` (IRQ-safe spinlock) protects the global run/sleep/wait queues
+  and the process-table state touched by timers; `schedule()` keeps it across
+  the stack switch and `sched_finish_switch()` (called from `switch.S` after
+  loading the new ESP) drops it, so another CPU can never pick a process whose
+  kernel stack this CPU is still unwinding.  Excess wake-up races are handled
+  by skipping a queued process that is still marked running on another CPU.
+  The **big kernel lock** (`bkl_*`) is a sleeping recursive mutex: every
+  ring-3 syscall and every kernel thread takes it, voluntary block/sleep
+  points drop it (`sched_yield`), and long serial/network waits release it via
+  `bkl_pause()`/`bkl_resume()`.  A BKL holder is not timer-preempted, so it can
+  safely be in the middle of kernel data structures.  Kernel threads are pinned
+  to the BSP; user processes may run on any CPU.  The idle fallback uses
+  `bkl_trylock()` (never blocks); the kworker and kernel commands take the BKL
+  normally and release it around their sleeps.
+  APs use their LAPIC timer (vector 0x30); the BSP keeps the 8259 PIC/PIT and
+  is the only global-tick source.  APs do not start taking user work until
+  `smp_enable_sched()`, i.e. after device/FS setup.  LAPIC IPIs (vector 0x31)
+  request a reschedule; they must be EOIed on the target LAPIC.
   `SYS_PROCINFO` exposes weight/state/CPU/ticks for the ring-3 `top` command,
   `SYS_PROCINFO` exposes weight/state/CPU/ticks for the ring-3 `top` command,
   and `sched`/`renice` plus `SYS_SCHEDINFO` expose the scheduler counters.
   and `sched`/`renice` plus `SYS_SCHEDINFO` expose the scheduler counters.
+  Ring 0 is preemptible at tick boundaries unless `preempt_count` is set (FAT
+  and the shadow flush) or the BKL is held; voluntary blocks/yields go through
+  the `int 0x81` gate.
+- **SMP / ACPI**: `acpi.c/h` (RSDP → RSDT/XSDT → MADT; CPU/LAPIC list and
+  LAPIC base; 4 MB table page mapping; malformed tables are ignored), `lapic.c/h`
+  (xAPIC MMIO access, BSP/AP init, INIT/SIPI/IPI, calibrated periodic timer),
+  `smp.c/h` (trampoline copy, AP stacks via `process_init_ap`, INIT-SIPI-SIPI
+  with a second SIPI retry, per-CPU `smp_ap_main`, `smp_stop_aps`, lazy
+  `smp_enable_sched`), `ap_tramp.S` + `ap_tramp.ld` (real → protected-mode
+  trampoline at `0x6000`), and `spinlock.h` (the IRQ-safe spinlocks).
+  `cpu_enable_sse()` runs on every AP: an AP starts with firmware CR4, and the
+  SSE2 drawing fast paths #UD until OSFXSR/OSXMMEXCPT are set.
 - **Memory**: `mm.c/h` — INT 15h E820 detection (real-mode in `vesa_stage.S`,
 - **Memory**: `mm.c/h` — INT 15h E820 detection (real-mode in `vesa_stage.S`,
   map at `0x1000`) + a bitmap page-frame allocator (4 KB pages). Reserves low
   map at `0x1000`) + a bitmap page-frame allocator (4 KB pages). Reserves low
   1 MB and an 8 MB VGA-shadow region. `kmalloc.c/h` — a power-of-two heap
   1 MB and an 8 MB VGA-shadow region. `kmalloc.c/h` — a power-of-two heap
@@ -233,6 +272,11 @@ below the run area), so adding code no longer moves any address.
   acks every datagram with the next expected offset, verifies the checksum and
   acks every datagram with the next expected offset, verifies the checksum and
   jumps. While idle the same loop answers ARP/ICMP for our IP, so the board is
   jumps. While idle the same loop answers ARP/ICMP for our IP, so the board is
   reachable (and MAC-resolvable) without running any command.
   reachable (and MAC-resolvable) without running any command.
+  `kexec_jump()` calls `smp_stop_aps()` first: every AP gets an INIT and is
+  parked in the wait-for-SIPI state before the trampoline overwrites the kernel
+  they were executing.  It runs with interrupts off, resets `sched_lock` (an AP
+  may have been inside a scheduler critical section when INIT arrived) and then
+  jumps; the new kernel starts the APs again from scratch.
 - **Mouse**: `mouse.c/h` — cursor move/draw shared by the PS/2 path
 - **Mouse**: `mouse.c/h` — cursor move/draw shared by the PS/2 path
   (IntelliMouse, Cirrus/Radeon hardware cursor) and the USB backend;
   (IntelliMouse, Cirrus/Radeon hardware cursor) and the USB backend;
   `usbhid.c/h` — USB HID boot devices: find + configure a mouse and/or
   `usbhid.c/h` — USB HID boot devices: find + configure a mouse and/or
@@ -354,7 +398,9 @@ Wire protocol (host → target, sent immediately; no handshake):
 `[magic "KXEC"][entry u32 LE][size u32 LE][data][checksum u8]`.
 `[magic "KXEC"][entry u32 LE][size u32 LE][data][checksum u8]`.
 The command pack has its own frame (`make cmds-dev`), the same layout with a
 The command pack has its own frame (`make cmds-dev`), the same layout with a
 `"KXNP"` magic (the entry field is ignored), which replaces /ramfs in place
 `"KXNP"` magic (the entry field is ignored), which replaces /ramfs in place
-(see below).
+(see below).  On SMP the APs are INIT-parked just before the jump, and the
+serial receive drops the BKL while it waits for bytes so the WM and the other
+CPU's syscalls stay responsive during a `make dev`.
 
 
 **Every iteration:**
 **Every iteration:**
 ```sh
 ```sh
@@ -686,3 +732,20 @@ and the WM falls back to polling (`io_keyboard_irq_masked()`).
    offset 0x1000 (the linker's page alignment), so ~4 KB of each ~8 KB blob is
    offset 0x1000 (the linker's page alignment), so ~4 KB of each ~8 KB blob is
    padding: 25 commands carry ~100 KB of it.  Linking them with
    padding: 25 commands carry ~100 KB of it.  Linking them with
    `-z max-page-size=0x100` would remove most of that.
    `-z max-page-size=0x100` would remove most of that.
+10. SMP is **ACPI/MADT only**: no Intel MP-table fallback, so a Pentium II SMP
+    board that exposes only MP tables stays UP (deliberate; there is no test
+    hardware).
+11. There is no IOAPIC driver.  The BSP keeps the 8259 PIC in LAPIC ExtINT mode;
+    APs mask LINT0/LINT1 and only use their LAPIC timer/IPIs.  Device IRQs never
+    reach an AP, and `pit_irq_stub` should be treated as BSP-only.
+12. The BKL serializes all kernel execution: a long FAT/disk operation, a
+    non-network driver wait or a big shadow-buffer operation can stall syscalls
+    on the other CPUs for its duration.  `bkl_pause()`/`bkl_resume()` is used
+    for serial/network waits, not yet for disk I/O.
+13. Kernel threads (kworker, kexec/payload, kernel commands, the kernel WM
+    fallback) are pinned to the BSP.  Only ring-3 processes use the APs, so a
+    single CPU-bound kernel driver still limits throughput.
+14. Page-table updates for another CPU's running process (e.g. an MMIO mapping)
+    rely on not-present TLB entries not being cached; there is no cross-CPU
+    TLB shootdown.  No process shares an address space with another, so normal
+    process teardown is unaffected.

+ 36 - 9
Makefile

@@ -13,14 +13,14 @@ ASFLAGS  = --32 -I.
 BOOT_LDFLAGS   = -T linker.ld -nostdlib -m elf_i386
 BOOT_LDFLAGS   = -T linker.ld -nostdlib -m elf_i386
 KERNEL_LDFLAGS = -T kernel.ld -nostdlib -m elf_i386
 KERNEL_LDFLAGS = -T kernel.ld -nostdlib -m elf_i386
 
 
-.PHONY: all clean run run-floppy run-usb run-uhci run-net run-ahci test-usb run-serial floppy usb flash space prog bootstrap dev net-dev cmds-dev net-cmds netmon monitor test-crypto
+.PHONY: all clean run run-smp run-floppy run-usb run-uhci run-net run-ahci test-usb run-serial floppy usb flash space prog bootstrap dev net-dev cmds-dev net-cmds netmon monitor test-crypto
 
 
 all: floppy.img usb.img final.bin root.img
 all: floppy.img usb.img final.bin root.img
 
 
 boot.o: boot.S
 boot.o: boot.S
 	$(AS) $(ASFLAGS) -o $@ $<
 	$(AS) $(ASFLAGS) -o $@ $<
 
 
-kernel.o: kernel.c idle.h io.h mouse.h panic.h process.h sched.h terminal.h timer.h serial.h tss.h usbhid.h version.h vga.h
+kernel.o: kernel.c acpi.h cpu.h idle.h io.h lapic.h mouse.h panic.h process.h sched.h smp.h terminal.h timer.h serial.h tss.h usbhid.h version.h vga.h
 	$(CC) $(CFLAGS) -o $@ $<
 	$(CC) $(CFLAGS) -o $@ $<
 
 
 screen.o: screen.c screen.h
 screen.o: screen.c screen.h
@@ -71,7 +71,7 @@ run.o: run.c fat.h io.h fbtext.h run_api.h timer.h vga.h vfs.h
 rtc.o: rtc.c rtc.h
 rtc.o: rtc.c rtc.h
 	$(CC) $(CFLAGS) -o $@ $<
 	$(CC) $(CFLAGS) -o $@ $<
 
 
-serial.o: serial.c serial.h serial_ring_buf.h pci.h idle.h timer.h
+serial.o: serial.c serial.h serial_ring_buf.h pci.h idle.h sched.h timer.h
 	$(CC) $(CFLAGS) -o $@ $<
 	$(CC) $(CFLAGS) -o $@ $<
 
 
 serial_ring_buf.o: serial_ring_buf.c serial_ring_buf.h
 serial_ring_buf.o: serial_ring_buf.c serial_ring_buf.h
@@ -206,7 +206,7 @@ mouse.o: mouse.c mouse.h cirrus.h radeon.h timer.h usbhid.h vga.h
 idle.o: idle.c idle.h
 idle.o: idle.c idle.h
 	$(CC) $(CFLAGS) -o $@ $<
 	$(CC) $(CFLAGS) -o $@ $<
 
 
-kexec.o: kexec.c kexec.h io.h serial.h
+kexec.o: kexec.c kexec.h io.h serial.h smp.h
 	$(CC) $(CFLAGS) -o $@ $<
 	$(CC) $(CFLAGS) -o $@ $<
 
 
 kexec_tramp.o: kexec_tramp.S
 kexec_tramp.o: kexec_tramp.S
@@ -233,10 +233,31 @@ switch.o: switch.S irqret.inc
 entry.o: entry.S
 entry.o: entry.S
 	$(AS) $(ASFLAGS) -o $@ $<
 	$(AS) $(ASFLAGS) -o $@ $<
 
 
-sched.o: sched.c sched.h process.h paging.h sys_fs.h timer.h tss.h vga.h
+sched.o: sched.c sched.h process.h paging.h smp.h spinlock.h sys_fs.h timer.h tss.h vga.h
 	$(CC) $(CFLAGS) -o $@ $<
 	$(CC) $(CFLAGS) -o $@ $<
 
 
-process.o: process.c process.h sched.h io.h kmalloc.h mm.h serial.h idle.h timer.h
+acpi.o: acpi.c acpi.h paging.h serial.h
+	$(CC) $(CFLAGS) -o $@ $<
+
+lapic.o: lapic.c lapic.h timer.h
+	$(CC) $(CFLAGS) -o $@ $<
+
+smp.o: smp.c smp.h acpi.h idle.h lapic.h paging.h process.h sched.h serial.h timer.h tss.h
+	$(CC) $(CFLAGS) -o $@ $<
+
+ap_tramp.o: ap_tramp.S
+	$(AS) $(ASFLAGS) -o $@ $<
+
+ap_tramp.elf: ap_tramp.o ap_tramp.ld
+	$(LD) -T ap_tramp.ld -nostdlib -m elf_i386 -o $@ $<
+
+ap_tramp.bin: ap_tramp.elf
+	$(OBJCOPY) -O binary $< $@
+
+ap_tramp_embed.o: ap_tramp.bin
+	$(LD) -m elf_i386 -r -b binary -o ap_tramp_embed.o ap_tramp.bin
+
+process.o: process.c process.h sched.h io.h kmalloc.h mm.h serial.h smp.h idle.h timer.h
 	$(CC) $(CFLAGS) -o $@ $<
 	$(CC) $(CFLAGS) -o $@ $<
 
 
 mm.o: mm.c mm.h
 mm.o: mm.c mm.h
@@ -263,13 +284,13 @@ sys_fs.o: sys_fs.c sys_fs.h syscall.h fat.h vfs.h
 sys_info.o: sys_info.c sys_info.h syscall.h ahci.h blkdev.h mm.h mouse.h usbhid.h vfs.h
 sys_info.o: sys_info.c sys_info.h syscall.h ahci.h blkdev.h mm.h mouse.h usbhid.h vfs.h
 	$(CC) $(CFLAGS) -o $@ $<
 	$(CC) $(CFLAGS) -o $@ $<
 
 
-sys_net.o: sys_net.c sys_net.h syscall.h dhcp.h dns.h net.h netdev.h netlog.h
+sys_net.o: sys_net.c sys_net.h syscall.h dhcp.h dns.h net.h netdev.h netlog.h process.h sched.h
 	$(CC) $(CFLAGS) -o $@ $<
 	$(CC) $(CFLAGS) -o $@ $<
 
 
 syscall_stub.o: syscall_stub.S irqret.inc
 syscall_stub.o: syscall_stub.S irqret.inc
 	$(AS) $(ASFLAGS) -o $@ $<
 	$(AS) $(ASFLAGS) -o $@ $<
 
 
-kernel.elf: vesa_stage.o entry.o kernel.o timer.o switch.o sched.o process.o mm.o kmalloc.o paging.o elf.o tss.o syscall.o sys_wm.o syscall_stub.o screen.o io.o tty.o rtc.o terminal.o wm.o reboot.o poweroff.o disk.o fat.o run.o serial.o serial_ring_buf.o serial_loop.o serial_recv.o bin_recv.o vga.o blit.o cpu.o vga_copy.o gfxbench.o gpu.o ati.o cirrus.o nvidia.o nv50.o radeon.o mtrr.o font_8x8.o fbtext.o gfx.o sys_gfx.o display.o mouse.o uhci.o ohci.o usb.o usbhid.o lsusb.o panic.o blkdev.o ata.o ahci.o fatdetect.o fatformat.o format.o vfs.o sys_fs.o sys_info.o sys_net.o idle.o kexec.o kexec_tramp.o dbg.o lspci.o peek.o ne2000.o r8168.o netdev.o net.o netkexec.o netlog.o dhcp.o dns.o tcp.o prog_embed.o space_embed.o cmds_pack.o ramfs.o payload.o
+kernel.elf: vesa_stage.o entry.o kernel.o timer.o switch.o sched.o process.o mm.o kmalloc.o paging.o elf.o tss.o syscall.o sys_wm.o syscall_stub.o screen.o io.o tty.o rtc.o terminal.o wm.o reboot.o poweroff.o disk.o fat.o run.o serial.o serial_ring_buf.o serial_loop.o serial_recv.o bin_recv.o vga.o blit.o cpu.o vga_copy.o gfxbench.o gpu.o ati.o cirrus.o nvidia.o nv50.o radeon.o mtrr.o font_8x8.o fbtext.o gfx.o sys_gfx.o display.o mouse.o uhci.o ohci.o usb.o usbhid.o lsusb.o panic.o blkdev.o ata.o ahci.o fatdetect.o fatformat.o format.o vfs.o sys_fs.o sys_info.o sys_net.o idle.o kexec.o kexec_tramp.o dbg.o lspci.o peek.o ne2000.o r8168.o netdev.o net.o netkexec.o netlog.o dhcp.o dns.o tcp.o acpi.o lapic.o smp.o ap_tramp_embed.o prog_embed.o space_embed.o cmds_pack.o ramfs.o payload.o
 	$(LD) $(KERNEL_LDFLAGS) -o $@ $^
 	$(LD) $(KERNEL_LDFLAGS) -o $@ $^
 
 
 MBR_LDFLAGS = -T mbr.ld -nostdlib -m elf_i386
 MBR_LDFLAGS = -T mbr.ld -nostdlib -m elf_i386
@@ -493,6 +514,12 @@ root.img: prog.bin space.bin
 run: image.bin root.img
 run: image.bin root.img
 	$(MAKE) run-floppy
 	$(MAKE) run-floppy
 
 
+# Four logical CPUs (the Atom 330 topology).  UP is still the default so a
+# machine without ACPI/MADT boots exactly as before.
+run-smp: floppy.img root.img
+	qemu-system-i386 -M pc -smp 4 -device pci-ohci -device usb-mouse -drive file=floppy.img,format=raw,if=floppy -drive file=root.img,format=raw,if=floppy 2>/dev/null || \
+	qemu-system-x86_64 -M pc -smp 4 -device pci-ohci -device usb-mouse -drive file=floppy.img,format=raw,if=floppy -drive file=root.img,format=raw,if=floppy
+
 floppy: floppy.img
 floppy: floppy.img
 
 
 usb: usb.img
 usb: usb.img
@@ -630,4 +657,4 @@ net-dev: kernel.bin netkexec_send
 	./netkexec_send $(BOARD_IP) $(NETKEXEC_PORT) 0x$$entry kernel.bin
 	./netkexec_send $(BOARD_IP) $(NETKEXEC_PORT) 0x$$entry kernel.bin
 
 
 clean:
 clean:
-	rm -f boot.o vesa_stage.o entry.o kernel.o timer.o switch.o sched.o process.o mm.o kmalloc.o paging.o elf.o tss.o syscall.o sys_wm.o syscall_stub.o screen.o io.o tty.o rtc.o terminal.o wm.o hello.o reboot.o poweroff.o fat.o run.o disk.o serial.o serial_ring_buf.o serial_loop.o serial_recv.o bin_recv.o vga.o blit.o cpu.o vga_copy.o gfxbench.o gpu.o ati.o cirrus.o nvidia.o nv50.o radeon.o mtrr.o font_8x8.o font_8x16.o fbtext.o gfx.o sys_gfx.o display.o mouse.o uhci.o ohci.o usb.o usbhid.o lsusb.o panic.o blkdev.o ata.o ahci.o fatdetect.o fatformat.o format.o vfs.o sys_fs.o sys_info.o idle.o kexec.o kexec_tramp.o dbg.o lspci.o peek.o ne2000.o r8168.o netdev.o net.o netkexec.o netlog.o dhcp.o dns.o tcp.o mbr.o boot.elf kernel.elf mbr.elf boot.bin mbr.bin kernel.bin boot.bin.tmp boot.bin.patched image.bin floppy.img usb.img final.bin root.img prog.o prog.elf prog.bin space.o space.elf space.bin hello.elf hello_embed.o prog_embed.o space_embed.o cmds.pack cmds_pack.o ramfs.o payload.o mkpack uwm_lib.o *.boot.tmp $(addsuffix .o,$(UCMDS)) $(addsuffix .elf,$(UCMDS)) $(TLSLIB_OBJS) tls_test bootstrap.o bootstrap.elf bootstrap.bin send_serial kexec_send netkexec_send netmonitor
+	rm -f boot.o vesa_stage.o entry.o kernel.o timer.o switch.o sched.o process.o mm.o kmalloc.o paging.o elf.o tss.o syscall.o sys_wm.o syscall_stub.o screen.o io.o tty.o rtc.o terminal.o wm.o hello.o reboot.o poweroff.o fat.o run.o disk.o serial.o serial_ring_buf.o serial_loop.o serial_recv.o bin_recv.o vga.o blit.o cpu.o vga_copy.o gfxbench.o gpu.o ati.o cirrus.o nvidia.o nv50.o radeon.o mtrr.o font_8x8.o font_8x16.o fbtext.o gfx.o sys_gfx.o display.o mouse.o uhci.o ohci.o usb.o usbhid.o lsusb.o panic.o blkdev.o ata.o ahci.o fatdetect.o fatformat.o format.o vfs.o sys_fs.o sys_info.o idle.o kexec.o kexec_tramp.o dbg.o lspci.o peek.o ne2000.o r8168.o netdev.o net.o netkexec.o netlog.o dhcp.o dns.o tcp.o acpi.o lapic.o smp.o ap_tramp.o ap_tramp.elf ap_tramp.bin ap_tramp_embed.o mbr.o boot.elf kernel.elf mbr.elf boot.bin mbr.bin kernel.bin boot.bin.tmp boot.bin.patched image.bin floppy.img usb.img final.bin root.img prog.o prog.elf prog.bin space.o space.elf space.bin hello.elf hello_embed.o prog_embed.o space_embed.o cmds.pack cmds_pack.o ramfs.o payload.o mkpack uwm_lib.o *.boot.tmp $(addsuffix .o,$(UCMDS)) $(addsuffix .elf,$(UCMDS)) $(TLSLIB_OBJS) tls_test bootstrap.o bootstrap.elf bootstrap.bin send_serial kexec_send netkexec_send netmonitor

+ 314 - 0
acpi.c

@@ -0,0 +1,314 @@
+/* Minimal ACPI RSDP/RSDT/XSDT/MADT parser.  See acpi.h.
+ *
+ * ACPI tables are usually below the 128 MB identity map, but on larger
+ * machines they can sit above it; every table page is mapped through
+ * paging_map_ram() before it is read.  Anything malformed or unsupported is
+ * treated as "no SMP", never as a fatal error.
+ */
+
+#include "acpi.h"
+#include "paging.h"
+#include "serial.h"
+
+struct acpi_rsdp {
+	char signature[8];
+	unsigned char checksum;
+	char oem_id[6];
+	unsigned char revision;
+	unsigned int rsdt;
+	unsigned int length;
+	unsigned long long xsdt;
+	unsigned char ext_checksum;
+	unsigned char reserved[3];
+} __attribute__((packed));
+
+struct acpi_sdt {
+	char signature[4];
+	unsigned int length;
+	unsigned char revision;
+	unsigned char checksum;
+	char oem_id[6];
+	char oem_table_id[8];
+	unsigned int oem_revision;
+	unsigned int creator_id;
+	unsigned int creator_revision;
+} __attribute__((packed));
+
+#define RSDP_SIG "RSD PTR "
+#define MADT_SIG "APIC"
+#define LAPIC_DEFAULT 0xFEE00000u
+
+static unsigned int ids[ACPI_MAX_CPUS];
+static unsigned int ncpu;
+static unsigned int bsp_apic;
+static unsigned int lapic_base;
+static int acpi_ok;
+
+static void *phys_ptr(unsigned int addr)
+{
+	return (void *)addr;
+}
+
+static int range_ok(unsigned int addr, unsigned int len)
+{
+	if (addr >= 0xFFFFF000u || len > 0x100000u)
+		return 0;
+	return addr + len >= addr;
+}
+
+/* Map every 4 MB page touched by [addr, addr+len). */
+static void map_table(unsigned int addr, unsigned int len)
+{
+	unsigned int p = addr & 0xFFC00000u;
+	unsigned int last = (addr + len - 1u) & 0xFFC00000u;
+
+	for (;;) {
+		paging_map_ram(p, 0x400000u);
+		if (p >= last)
+			break;
+		if (p > 0xFFC00000u - 0x400000u)
+			break;
+		p += 0x400000u;
+	}
+}
+
+static int sum_ok(const unsigned char *p, unsigned int len)
+{
+	unsigned char sum = 0;
+	unsigned int i;
+
+	for (i = 0; i < len; i++)
+		sum = (unsigned char)(sum + p[i]);
+	return sum == 0;
+}
+
+static int rsdp_valid(const struct acpi_rsdp *r)
+{
+	unsigned int i;
+
+	for (i = 0; i < 8; i++)
+		if (r->signature[i] != RSDP_SIG[i])
+			return 0;
+	if (!sum_ok((const unsigned char *)r, 20))
+		return 0;
+	if (r->revision >= 2) {
+		if (r->length < 36 || r->length > 4096)
+			return 0;
+		if (!sum_ok((const unsigned char *)r, r->length))
+			return 0;
+	}
+	return 1;
+}
+
+static const struct acpi_rsdp *find_rsdp(void)
+{
+	volatile unsigned short *ebda_seg = (volatile unsigned short *)0x40E;
+	unsigned int ebda, i;
+
+	/* Opaque asm keeps -Warray-bounds quiet about the fixed BIOS pointer. */
+	__asm__ volatile ("" : "+r"(ebda_seg));
+	ebda = (unsigned int)(*ebda_seg) << 4;
+
+	if (ebda) {
+		for (i = 0; i < 1024; i += 16) {
+			const struct acpi_rsdp *r =
+				(const struct acpi_rsdp *)(ebda + i);
+			if (rsdp_valid(r))
+				return r;
+		}
+	}
+	for (i = 0xE0000; i < 0x100000; i += 16) {
+		const struct acpi_rsdp *r =
+			(const struct acpi_rsdp *)(i);
+		if (rsdp_valid(r))
+			return r;
+	}
+	return 0;
+}
+
+static const struct acpi_sdt *find_madt(const struct acpi_rsdp *rsdp)
+{
+	unsigned int addr, len, i, step;
+	const struct acpi_sdt *sdt;
+
+	/* Prefer the 64-bit XSDT when it is reachable from the 32-bit
+	 * identity map; otherwise fall back to the ACPI 1.0 RSDT. */
+	if (rsdp->revision >= 2 && rsdp->xsdt &&
+	    (unsigned int)(rsdp->xsdt >> 32) == 0) {
+		addr = (unsigned int)rsdp->xsdt;
+		step = 8;
+	} else {
+		addr = rsdp->rsdt;
+		step = 4;
+	}
+	if (!addr || !range_ok(addr, sizeof(struct acpi_sdt)))
+		return 0;
+	map_table(addr, sizeof(struct acpi_sdt));
+	sdt = (const struct acpi_sdt *)phys_ptr(addr);
+	len = sdt->length;
+	if (len < sizeof(struct acpi_sdt) || len > 0x100000u)
+		return 0;
+	map_table(addr, len);
+	if (!sum_ok((const unsigned char *)sdt, len))
+		return 0;
+
+	for (i = sizeof(struct acpi_sdt); i + step <= len; i += step) {
+		unsigned int taddr;
+		const struct acpi_sdt *t;
+		unsigned int tlen;
+
+		if (step == 8) {
+			unsigned long long a =
+				*(const unsigned long long *)((const unsigned char *)sdt + i);
+			if ((unsigned int)(a >> 32) != 0)
+				continue;
+			taddr = (unsigned int)a;
+		} else {
+			taddr = *(const unsigned int *)((const unsigned char *)sdt + i);
+		}
+		if (!taddr || !range_ok(taddr, sizeof(struct acpi_sdt)))
+			continue;
+		map_table(taddr, sizeof(struct acpi_sdt));
+		t = (const struct acpi_sdt *)phys_ptr(taddr);
+		tlen = t->length;
+		if (tlen < sizeof(struct acpi_sdt) || tlen > 0x100000u)
+			continue;
+		map_table(taddr, tlen);
+		if (t->signature[0] == 'A' && t->signature[1] == 'P' &&
+		    t->signature[2] == 'I' && t->signature[3] == 'C' &&
+		    sum_ok((const unsigned char *)t, tlen))
+			return t;
+	}
+	return 0;
+}
+
+static unsigned int cpuid_apic_id(void)
+{
+	unsigned int eax, ebx, ecx, edx;
+
+	eax = 1;
+	__asm__ volatile ("cpuid"
+			  : "+a"(eax), "=b"(ebx), "=c"(ecx), "=d"(edx));
+	if (!(edx & (1u << 9)))       /* CPUID.1:EDX.APIC */
+		return 0xFFFFFFFFu;
+	return ebx >> 24;             /* initial local APIC ID */
+}
+
+static void parse_madt(const struct acpi_sdt *madt)
+{
+	const unsigned char *p = (const unsigned char *)madt;
+	unsigned int len = madt->length;
+	unsigned int i;
+
+	lapic_base = LAPIC_DEFAULT;
+	if (len >= 44) {
+		unsigned int base = *(const unsigned int *)(p + 36);
+		if (base)
+			lapic_base = base;
+	}
+
+	for (i = 44; i + 2 <= len; ) {
+		unsigned char type = p[i];
+		unsigned char elen = p[i + 1];
+
+		if (elen < 2 || i + elen > len)
+			break;
+		if (type == 0 && elen >= 8) {
+			/* Processor Local APIC: flags bit 0 = enabled. */
+			if ((p[i + 4] & 1) && ncpu < ACPI_MAX_CPUS)
+				ids[ncpu++] = p[i + 3];
+		} else if (type == 5 && elen >= 12) {
+			/* Local APIC Address Override (64-bit). */
+			unsigned int lo = *(const unsigned int *)(p + i + 4);
+			unsigned int hi = *(const unsigned int *)(p + i + 8);
+			if (!hi && lo)
+				lapic_base = lo;
+		}
+		/* x2APIC (type 9), IOAPIC (1), NMI (4) and overrides are not
+		 * needed for the first SMP step. */
+		i += elen;
+	}
+}
+
+int acpi_init(void)
+{
+	const struct acpi_rsdp *rsdp;
+	const struct acpi_sdt *madt;
+	unsigned int i;
+
+	acpi_ok = 0;
+	ncpu = 0;
+	bsp_apic = cpuid_apic_id();
+	if (bsp_apic == 0xFFFFFFFFu)
+		return 0;
+
+	rsdp = find_rsdp();
+	if (!rsdp) {
+		serial_puts(SERIAL_COM1, "acpi: no RSDP, staying UP\n");
+		return 0;
+	}
+	madt = find_madt(rsdp);
+	if (!madt) {
+		serial_puts(SERIAL_COM1, "acpi: no MADT, staying UP\n");
+		return 0;
+	}
+	parse_madt(madt);
+
+	/* Put the BSP first.  If firmware omitted it, keep the discovered list
+	 * but make sure entry 0 is a real CPU. */
+	for (i = 0; i < ncpu; i++)
+		if (ids[i] == bsp_apic)
+			break;
+	if (i < ncpu) {
+		unsigned int t = ids[0];
+		ids[0] = ids[i];
+		ids[i] = t;
+	} else if (ncpu) {
+		ids[0] = bsp_apic;
+	}
+	if (!ncpu)
+		return 0;
+
+	acpi_ok = 1;
+	return 1;
+}
+
+unsigned int acpi_cpu_count(void)
+{
+	return acpi_ok ? ncpu : 0;
+}
+
+unsigned int acpi_cpu_apic_id(unsigned int index)
+{
+	return (acpi_ok && index < ncpu) ? ids[index] : 0;
+}
+
+unsigned int acpi_bsp_apic_id(void)
+{
+	return bsp_apic;
+}
+
+unsigned int acpi_lapic_base(void)
+{
+	return lapic_base;
+}
+
+void acpi_dump(void)
+{
+	unsigned int i;
+
+	if (!acpi_ok) {
+		serial_puts(SERIAL_COM1, "acpi: unavailable, 1 cpu\n");
+		return;
+	}
+	serial_puts(SERIAL_COM1, "acpi: lapic ");
+	serial_puthex(SERIAL_COM1, lapic_base);
+	serial_puts(SERIAL_COM1, ", ");
+	serial_putdec(SERIAL_COM1, ncpu);
+	serial_puts(SERIAL_COM1, " cpus:");
+	for (i = 0; i < ncpu; i++) {
+		serial_puts(SERIAL_COM1, " ");
+		serial_putdec(SERIAL_COM1, ids[i]);
+	}
+	serial_puts(SERIAL_COM1, "\n");
+}

+ 22 - 0
acpi.h

@@ -0,0 +1,22 @@
+/* Minimal ACPI table discovery: RSDP -> RSDT/XSDT -> MADT.
+ *
+ * Only the CPU/LAPIC information needed to start APs is parsed.  Absence of
+ * ACPI or of a usable MADT is not an error: the kernel then stays UP.
+ */
+#ifndef ACPI_H
+#define ACPI_H
+
+#define ACPI_MAX_CPUS 8
+
+/* Returns 1 when a usable MADT with at least one local APIC entry was found.
+ * acpi_cpu_count() then includes the BSP as entry 0. */
+int acpi_init(void);
+
+unsigned int acpi_cpu_count(void);
+unsigned int acpi_cpu_apic_id(unsigned int index);
+unsigned int acpi_bsp_apic_id(void);
+unsigned int acpi_lapic_base(void);
+
+void acpi_dump(void);
+
+#endif

+ 86 - 0
ap_tramp.S

@@ -0,0 +1,86 @@
+/*
+ * AP startup trampoline, linked at 0x6000 and copied there by the BSP.
+ *
+ * The BSP fills the fixed info block at 0x6F00 before sending SIPI:
+ *   0x6F00 cr3        (physical kernel page directory)
+ *   0x6F04 stack_top  (AP idle stack top, identity mapped)
+ *   0x6F08 entry      (C entry point, protected-mode address)
+ *   0x6F0C cpu_index
+ *
+ * The AP starts in real mode at 0x6000 (SIPI vector 0x06), builds a tiny flat
+ * GDT, enters protected mode, enables PSE + paging and jumps to the C entry.
+ */
+.code16
+.section .text.ap, "ax"
+
+.equ AP_INFO,      0x6F00
+.equ AP_INFO_CR3,  AP_INFO + 0
+.equ AP_INFO_STACK, AP_INFO + 4
+.equ AP_INFO_ENTRY, AP_INFO + 8
+.equ AP_INFO_CPU,  AP_INFO + 12
+
+.globl ap_tramp_start
+ap_tramp_start:
+	cli
+	cld
+	xorw  %ax, %ax
+	movw  %ax, %ds
+	movw  %ax, %es
+	movw  %ax, %fs
+	movw  %ax, %gs
+	movw  %ax, %ss
+	movw  $0x6C00, %sp
+
+	lgdt  ap_gdt_desc
+
+	movl  %cr0, %eax
+	orl   $1, %eax
+	movl  %eax, %cr0
+
+	/* 32-bit far jump into the protected-mode part. */
+	.byte 0x66, 0xea
+	.long ap_pm32
+	.word 0x08
+
+.code32
+ap_pm32:
+	movw  $0x10, %ax
+	movw  %ax, %ds
+	movw  %ax, %es
+	movw  %ax, %fs
+	movw  %ax, %gs
+	movw  %ax, %ss
+	movl  AP_INFO_STACK, %esp
+
+	movl  AP_INFO_CR3, %eax
+	movl  %eax, %cr3
+
+	movl  %cr4, %eax
+	orl   $0x10, %eax             /* CR4.PSE */
+	movl  %eax, %cr4
+
+	movl  %cr0, %eax
+	orl   $0x80010000, %eax       /* CR0.PG | CR0.WP */
+	movl  %eax, %cr0
+	jmp   1f
+1:
+	movl  AP_INFO_CPU, %eax
+	pushl %eax
+	movl  AP_INFO_ENTRY, %eax
+	call  *%eax
+
+2:	cli
+	hlt
+	jmp   2b
+
+.align 8
+ap_gdt:
+	.quad 0
+	.word 0xFFFF, 0x0000
+	.byte 0x00, 0x9A, 0xCF, 0x00
+	.word 0xFFFF, 0x0000
+	.byte 0x00, 0x92, 0xCF, 0x00
+ap_gdt_end:
+ap_gdt_desc:
+	.word ap_gdt_end - ap_gdt - 1
+	.long ap_gdt

+ 16 - 0
ap_tramp.ld

@@ -0,0 +1,16 @@
+/* AP startup blob, linked at the physical SIPI vector page 0x6000. */
+ENTRY(ap_tramp_start)
+SECTIONS
+{
+	. = 0x6000;
+	.text : {
+		*(.text.ap)
+		*(.text*)
+	}
+	.rodata : { *(.rodata*) }
+	.data : { *(.data*) }
+	.bss : { *(.bss*) *(COMMON) }
+	ap_tramp_end = .;
+}
+ASSERT(ap_tramp_end <= 0x6C00,
+       "AP trampoline grew into its stack area")

+ 16 - 0
cpu.c

@@ -16,6 +16,22 @@ static int cpuid_supported(void)
 	return ((a ^ b) & (1u << 21)) != 0;
 	return ((a ^ b) & (1u << 21)) != 0;
 }
 }
 
 
+void cpu_enable_sse(void)
+{
+	unsigned int eax, ebx, ecx, edx, cr4;
+
+	if (!cpuid_supported())
+		return;
+	eax = 1;
+	__asm__ volatile ("cpuid"
+			  : "+a"(eax), "=b"(ebx), "=c"(ecx), "=d"(edx));
+	if (!(edx & (1u << 24)))        /* FXSR */
+		return;
+	__asm__ volatile ("mov %%cr4, %0" : "=r"(cr4));
+	cr4 |= (1u << 9) | (1u << 10);  /* OSFXSR | OSXMMEXCPT */
+	__asm__ volatile ("mov %0, %%cr4" :: "r"(cr4) : "memory");
+}
+
 int cpu_has_sse2(void)
 int cpu_has_sse2(void)
 {
 {
 	unsigned int eax, ebx, ecx, edx;
 	unsigned int eax, ebx, ecx, edx;

+ 5 - 0
cpu.h

@@ -9,4 +9,9 @@
 /* 1 when the CPU supports SSE2 (the ION does, the Pentium targets do not). */
 /* 1 when the CPU supports SSE2 (the ION does, the Pentium targets do not). */
 int cpu_has_sse2(void);
 int cpu_has_sse2(void);
 
 
+/* Enable CR4.OSFXSR/OSXMMEXCPT on the calling CPU when it has FXSR.  The BSP
+ * does this lazily in cpu_has_sse2(); an AP starts with firmware CR4 and must
+ * call this before it can run the SSE2 drawing fast paths. */
+void cpu_enable_sse(void);
+
 #endif
 #endif

+ 40 - 2
kernel.c

@@ -1,7 +1,10 @@
 /* Kernel: entry point at 0x7E00, timer (PIT 1 kHz), process init, terminal. */
 /* Kernel: entry point at 0x7E00, timer (PIT 1 kHz), process init, terminal. */
 
 
+#include "acpi.h"
+#include "cpu.h"
 #include "idle.h"
 #include "idle.h"
 #include "io.h"
 #include "io.h"
+#include "lapic.h"
 #include "panic.h"
 #include "panic.h"
 #include "kmalloc.h"
 #include "kmalloc.h"
 #include "mm.h"
 #include "mm.h"
@@ -18,6 +21,7 @@
 #include "usbhid.h"
 #include "usbhid.h"
 #include "wm.h"
 #include "wm.h"
 #include "serial.h"
 #include "serial.h"
+#include "smp.h"
 #include "version.h"
 #include "version.h"
 #include "vga.h"
 #include "vga.h"
 #include "vfs.h"
 #include "vfs.h"
@@ -50,6 +54,9 @@ extern void kbd_irq_stub(void);
 extern void fdc_irq_stub(void);
 extern void fdc_irq_stub(void);
 extern void default_irq_stub(void);
 extern void default_irq_stub(void);
 extern void resched_stub(void);
 extern void resched_stub(void);
+extern void lapic_timer_stub(void);
+extern void lapic_resched_stub(void);
+extern void lapic_spurious_stub(void);
 extern void syscall_stub(void);
 extern void syscall_stub(void);
 extern unsigned int exc_stub_table[];
 extern unsigned int exc_stub_table[];
 
 
@@ -147,6 +154,12 @@ static void timer_init(void)
 	set_idt_gate(0x21, (unsigned int)kbd_irq_stub);
 	set_idt_gate(0x21, (unsigned int)kbd_irq_stub);
 	set_idt_gate(0x26, (unsigned int)fdc_irq_stub);
 	set_idt_gate(0x26, (unsigned int)fdc_irq_stub);
 
 
+	/* Local-APIC vectors: the AP timer and reschedule IPI, plus the
+	 * spurious vector the LAPIC can deliver at any time. */
+	set_idt_gate(LAPIC_TIMER_VECTOR, (unsigned int)lapic_timer_stub);
+	set_idt_gate(LAPIC_RESCHED_VECTOR, (unsigned int)lapic_resched_stub);
+	set_idt_gate(LAPIC_SPURIOUS_VECTOR, (unsigned int)lapic_spurious_stub);
+
 	/* Syscall gate (ring-3 can int 0x80). */
 	/* Syscall gate (ring-3 can int 0x80). */
 	set_idt_gate_user(0x80, (unsigned int)syscall_stub);
 	set_idt_gate_user(0x80, (unsigned int)syscall_stub);
 	/* Kernel-only voluntary reschedule gate (int 0x81). */
 	/* Kernel-only voluntary reschedule gate (int 0x81). */
@@ -163,6 +176,12 @@ static void timer_init(void)
 	__asm__ volatile ("sti");
 	__asm__ volatile ("sti");
 }
 }
 
 
+/* The BSP built the IDT once; APs only need to load the same pointer. */
+void idt_load_ap(void)
+{
+	__asm__ volatile ("lidt %0" : : "m"(idtr));
+}
+
 extern unsigned char __bss_start[], __bss_end[];
 extern unsigned char __bss_start[], __bss_end[];
 
 
 static int kworker_pid = -1;
 static int kworker_pid = -1;
@@ -200,8 +219,14 @@ static void kernel_idle_loop(void)
 {
 {
 	for (;;) {
 	for (;;) {
 		if (kworker_pid <= 0 ||
 		if (kworker_pid <= 0 ||
-		    !process_is_alive((unsigned int)kworker_pid))
-			system_housekeeping();
+		    !process_is_alive((unsigned int)kworker_pid)) {
+			/* Idle must never sleep on the BKL: if a user process on
+			 * another CPU owns it, just keep idling and retry. */
+			if (bkl_trylock()) {
+				system_housekeeping();
+				bkl_unlock();
+			}
+		}
 		cpu_idle();
 		cpu_idle();
 	}
 	}
 }
 }
@@ -229,6 +254,13 @@ void kernel_main(void)
 	mm_dump();
 	mm_dump();
 	kmalloc_init();
 	kmalloc_init();
 	paging_init();
 	paging_init();
+	/* paging_init only ORs in PSE; make sure the BSP's SSE state is on
+	 * before any drawing path can select an SSE2 helper. */
+	cpu_enable_sse();
+	/* ACPI needs paging for table access; a missing or unusable MADT simply
+	 * leaves the machine at one CPU. */
+	acpi_init();
+	acpi_dump();
 	io_init();
 	io_init();
 	idle_init();
 	idle_init();
 
 
@@ -237,6 +269,9 @@ void kernel_main(void)
 	 * the current process, its CR3 and the TSS, so those must be valid. */
 	 * the current process, its CR3 and the TSS, so those must be valid. */
 	process_init();
 	process_init();
 	timer_init();
 	timer_init();
+	/* The PIT (and its sti) is live, so the AP bring-up can use it as a
+	 * timeout clock.  APs idle until smp_enable_sched() after boot. */
+	smp_init();
 	mouse_init();
 	mouse_init();
 	/* USB HID backend (mouse + keyboard): controller bring-up + first
 	/* USB HID backend (mouse + keyboard): controller bring-up + first
 	 * scan; devices plugged in later are picked up by the hotplug
 	 * scan; devices plugged in later are picked up by the hotplug
@@ -279,5 +314,8 @@ void kernel_main(void)
 		process_set_weight((unsigned int)kworker_pid, 1);
 		process_set_weight((unsigned int)kworker_pid, 1);
 		process_set_affinity((unsigned int)kworker_pid, 1);
 		process_set_affinity((unsigned int)kworker_pid, 1);
 	}
 	}
+	/* Boot-time device/FS setup is complete; APs may now take user work.
+	 * Kernel threads stay pinned to the BSP and serialize on the BKL. */
+	smp_enable_sched();
 	kernel_idle_loop();
 	kernel_idle_loop();
 }
 }

+ 6 - 0
kexec.c

@@ -25,6 +25,7 @@
 #include "io.h"
 #include "io.h"
 #include "mm.h"
 #include "mm.h"
 #include "serial.h"
 #include "serial.h"
+#include "smp.h"
 
 
 #define KEXEC_MAGIC 0x4345584B /* "KXEC" little-endian */
 #define KEXEC_MAGIC 0x4345584B /* "KXEC" little-endian */
 
 
@@ -195,6 +196,11 @@ void program_kexec_main(void)
  * jump to it with the staging/load/size/entry arguments.  Never returns. */
  * jump to it with the staging/load/size/entry arguments.  Never returns. */
 void kexec_jump(unsigned int entry, unsigned int size)
 void kexec_jump(unsigned int entry, unsigned int size)
 {
 {
+	/* Park the APs first: the new kernel image is about to overwrite the
+	 * kernel they are executing.  INIT puts them back in real mode where
+	 * they wait for the new kernel's SIPI. */
+	smp_stop_aps();
+
 	/* Relocate the trampoline to 0x800 so it survives the kernel overwrite. */
 	/* Relocate the trampoline to 0x800 so it survives the kernel overwrite. */
 	{
 	{
 		const void *s = (const void *)kexec_trampoline;
 		const void *s = (const void *)kexec_trampoline;

+ 184 - 0
lapic.c

@@ -0,0 +1,184 @@
+/* Local APIC driver.  The MMIO window must already be mapped by smp.c. */
+
+#include "lapic.h"
+#include "timer.h"
+
+#define LAPIC_ID        0x020
+#define LAPIC_TPR       0x080
+#define LAPIC_EOI       0x0B0
+#define LAPIC_SVR       0x0F0
+#define LAPIC_ICR_LO    0x300
+#define LAPIC_ICR_HI    0x310
+#define LAPIC_LVT_TIMER 0x320
+#define LAPIC_LVT_LINT0 0x350
+#define LAPIC_LVT_LINT1 0x360
+#define LAPIC_TIMER_INIT 0x380
+#define LAPIC_TIMER_CUR  0x390
+#define LAPIC_TIMER_DIV  0x3E0
+
+#define IA32_APIC_BASE_MSR 0x1B
+
+static volatile unsigned int *lapic_mmio;
+static unsigned int lapic_phys;
+
+static inline unsigned int lapic_read(unsigned int off)
+{
+	return lapic_mmio[off / 4];
+}
+
+static inline void lapic_write(unsigned int off, unsigned int value)
+{
+	lapic_mmio[off / 4] = value;
+}
+
+static inline unsigned long long rdmsr(unsigned int msr)
+{
+	unsigned int lo, hi;
+	__asm__ volatile ("rdmsr" : "=a"(lo), "=d"(hi) : "c"(msr));
+	return ((unsigned long long)hi << 32) | lo;
+}
+
+static inline void wrmsr(unsigned int msr, unsigned long long value)
+{
+	unsigned int lo = (unsigned int)value;
+	unsigned int hi = (unsigned int)(value >> 32);
+	__asm__ volatile ("wrmsr" : : "c"(msr), "a"(lo), "d"(hi));
+}
+
+void lapic_set_base(unsigned int base)
+{
+	lapic_phys = base;
+	lapic_mmio = (volatile unsigned int *)base;
+}
+
+unsigned int lapic_base(void)
+{
+	return lapic_phys;
+}
+
+static void lapic_enable(void)
+{
+	unsigned long long msr = rdmsr(IA32_APIC_BASE_MSR);
+	unsigned int base = (unsigned int)(msr & 0xFFFFF000u);
+	unsigned int keep = (unsigned int)(msr & 0x100u); /* BSP bit */
+
+	if (base != lapic_phys) {
+		msr = (msr & ~0xFFFFF000ull) | lapic_phys;
+		wrmsr(IA32_APIC_BASE_MSR, msr);
+	}
+	/* Keep the BSP bit, clear x2APIC (we drive the xAPIC MMIO window), and
+	 * set the global enable bit. */
+	msr = (msr & ~0xF00ull) | keep | 0x800ull;
+	wrmsr(IA32_APIC_BASE_MSR, msr);
+}
+
+static int lapic_wait_icr(void)
+{
+	unsigned int n;
+
+	for (n = 0; n < 1000000u; n++) {
+		if (!(lapic_read(LAPIC_ICR_LO) & 0x1000u))
+			return 1;
+	}
+	return 0;
+}
+
+void lapic_eoi(void)
+{
+	if (lapic_mmio)
+		lapic_write(LAPIC_EOI, 0);
+}
+
+unsigned int lapic_id(void)
+{
+	return lapic_mmio ? (lapic_read(LAPIC_ID) >> 24) : 0xFFFFFFFFu;
+}
+
+int lapic_init_bsp(void)
+{
+	if (!lapic_mmio)
+		return 0;
+	lapic_enable();
+	lapic_write(LAPIC_TPR, 0);
+	/* Spurious vector: the handler must not send a PIC EOI. */
+	lapic_write(LAPIC_SVR, 0x1FF);
+	/* Keep the 8259 PIC on LINT0 (ExtINT); LINT1 stays NMI. */
+	lapic_write(LAPIC_LVT_LINT0, 0x700);
+	lapic_write(LAPIC_LVT_LINT1, 0x400);
+	lapic_timer_stop();
+	return 1;
+}
+
+int lapic_init_ap(void)
+{
+	if (!lapic_mmio)
+		return 0;
+	lapic_enable();
+	lapic_write(LAPIC_TPR, 0);
+	lapic_write(LAPIC_SVR, 0x1FF);
+	lapic_write(LAPIC_LVT_LINT0, 0x10000); /* masked */
+	lapic_write(LAPIC_LVT_LINT1, 0x10000);
+	lapic_timer_stop();
+	return 1;
+}
+
+void lapic_send_ipi(unsigned int apic_id, unsigned int vector)
+{
+	if (!lapic_mmio)
+		return;
+	lapic_wait_icr();
+	lapic_write(LAPIC_ICR_HI, (apic_id & 0xFFu) << 24);
+	lapic_write(LAPIC_ICR_LO, (vector & 0xFFu) | 0x00000000u);
+	lapic_wait_icr();
+}
+
+void lapic_send_init(unsigned int apic_id)
+{
+	if (!lapic_mmio)
+		return;
+	lapic_wait_icr();
+	lapic_write(LAPIC_ICR_HI, (apic_id & 0xFFu) << 24);
+	/* delivery mode INIT (5), level assert. */
+	lapic_write(LAPIC_ICR_LO, 0x00004500u);
+	lapic_wait_icr();
+}
+
+void lapic_send_sipi(unsigned int apic_id, unsigned int vector)
+{
+	if (!lapic_mmio)
+		return;
+	lapic_wait_icr();
+	lapic_write(LAPIC_ICR_HI, (apic_id & 0xFFu) << 24);
+	/* delivery mode startup (6); vector is the real-mode page number. */
+	lapic_write(LAPIC_ICR_LO, 0x00004600u | (vector & 0xFFu));
+	lapic_wait_icr();
+}
+
+int lapic_timer_start(unsigned int vector)
+{
+	unsigned int t0, count, per_ms;
+
+	if (!lapic_mmio)
+		return 0;
+	lapic_write(LAPIC_TIMER_DIV, 0x3); /* divide by 16 */
+	lapic_write(LAPIC_LVT_TIMER, 0x10000); /* masked while calibrating */
+	lapic_write(LAPIC_TIMER_INIT, 0xFFFFFFFFu);
+
+	t0 = get_ticks_ms();
+	while ((unsigned int)(get_ticks_ms() - t0) < 10)
+		__asm__ volatile ("pause");
+	count = lapic_read(LAPIC_TIMER_CUR);
+	per_ms = (0xFFFFFFFFu - count) / 10u;
+	if (per_ms == 0)
+		return 0;
+
+	lapic_write(LAPIC_LVT_TIMER, (vector & 0xFFu) | 0x00020000u);
+	lapic_write(LAPIC_TIMER_INIT, per_ms);
+	return 1;
+}
+
+void lapic_timer_stop(void)
+{
+	if (lapic_mmio)
+		lapic_write(LAPIC_LVT_TIMER, 0x10000);
+}

+ 24 - 0
lapic.h

@@ -0,0 +1,24 @@
+/* Local APIC access: BSP/AP bring-up, IPIs and the per-CPU timer. */
+#ifndef LAPIC_H
+#define LAPIC_H
+
+#define LAPIC_TIMER_VECTOR   0x30
+#define LAPIC_RESCHED_VECTOR 0x31
+#define LAPIC_SPURIOUS_VECTOR 0xFF
+
+void lapic_set_base(unsigned int base);
+unsigned int lapic_base(void);
+
+int  lapic_init_bsp(void);
+int  lapic_init_ap(void);
+void lapic_eoi(void);
+unsigned int lapic_id(void);
+
+void lapic_send_ipi(unsigned int apic_id, unsigned int vector);
+void lapic_send_init(unsigned int apic_id);
+void lapic_send_sipi(unsigned int apic_id, unsigned int vector);
+
+int  lapic_timer_start(unsigned int vector);
+void lapic_timer_stop(void);
+
+#endif

+ 5 - 0
paging.c

@@ -159,6 +159,11 @@ unsigned int paging_current_cr3(void)
 	return cr3 & 0xFFFFF000u;
 	return cr3 & 0xFFFFF000u;
 }
 }
 
 
+unsigned int paging_kernel_cr3(void)
+{
+	return (unsigned int)page_dir;
+}
+
 unsigned int paging_new_directory(void)
 unsigned int paging_new_directory(void)
 {
 {
 	unsigned int *pd = (unsigned int *)mm_alloc_page();
 	unsigned int *pd = (unsigned int *)mm_alloc_page();

+ 4 - 0
paging.h

@@ -17,6 +17,10 @@ void paging_map_ram(unsigned int base, unsigned int size);
  * is per-CPU; on one CPU it is the running process's directory. */
  * is per-CPU; on one CPU it is the running process's directory. */
 unsigned int paging_current_cr3(void);
 unsigned int paging_current_cr3(void);
 
 
+/* Physical address of the bootstrap/kernel page directory that new process
+ * directories clone. */
+unsigned int paging_kernel_cr3(void);
+
 /* Allocate a fresh page directory that clones the kernel identity map and
 /* Allocate a fresh page directory that clones the kernel identity map and
  * return its physical address (0 on out-of-memory). */
  * return its physical address (0 on out-of-memory). */
 unsigned int paging_new_directory(void);
 unsigned int paging_new_directory(void);

+ 175 - 75
process.c

@@ -13,6 +13,7 @@
 #include "mm.h"
 #include "mm.h"
 #include "paging.h"
 #include "paging.h"
 #include "serial.h"
 #include "serial.h"
+#include "smp.h"
 #include "sys_fs.h"
 #include "sys_fs.h"
 #include "sys_gfx.h"
 #include "sys_gfx.h"
 #include "sys_net.h"
 #include "sys_net.h"
@@ -24,7 +25,7 @@
 
 
 struct process *procs[PROC_MAX];
 struct process *procs[PROC_MAX];
 
 
-static struct process idle_proc;
+static struct process idle_procs[MAX_CPUS];
 static unsigned int next_pid_counter = 1; /* monotonic, wraps */
 static unsigned int next_pid_counter = 1; /* monotonic, wraps */
 static unsigned int fallback_stack_next;  /* 0..PROC_FALLBACK_STACK_MAX */
 static unsigned int fallback_stack_next;  /* 0..PROC_FALLBACK_STACK_MAX */
 static struct process *zombie_head, *zombie_tail;
 static struct process *zombie_head, *zombie_tail;
@@ -135,7 +136,8 @@ static struct process *proc_alloc(int *slot_out)
  * disabled and with parent = current_proc() (possibly idle). */
  * disabled and with parent = current_proc() (possibly idle). */
 static void proc_commit(struct process *p, int slot, const char *name,
 static void proc_commit(struct process *p, int slot, const char *name,
 			unsigned int esp, int owned, unsigned int stack_base,
 			unsigned int esp, int owned, unsigned int stack_base,
-			unsigned int cr3, unsigned int kstack_top)
+			unsigned int cr3, unsigned int kstack_top,
+			void (*kentry)(void), int kernel)
 {
 {
 	struct process *parent = current_proc();
 	struct process *parent = current_proc();
 
 
@@ -149,7 +151,11 @@ static void proc_commit(struct process *p, int slot, const char *name,
 	p->start_ticks = get_ticks_ms();
 	p->start_ticks = get_ticks_ms();
 	p->on_cpu = PROC_NO_CPU;
 	p->on_cpu = PROC_NO_CPU;
 	p->last_cpu = PROC_NO_CPU;
 	p->last_cpu = PROC_NO_CPU;
-	p->affinity = 0xFFFFFFFFu;
+	/* Kernel threads touch global kernel state; keep them on the BSP.
+	 * Ring-3 processes may use every CPU. */
+	p->affinity = kernel ? 1u : 0xFFFFFFFFu;
+	p->kentry = kentry;
+	p->kernel_thread = (unsigned int)kernel;
 	p->weight = 0;
 	p->weight = 0;
 	p->stride = 0;
 	p->stride = 0;
 	p->vruntime = 0;
 	p->vruntime = 0;
@@ -162,7 +168,9 @@ static void proc_commit(struct process *p, int slot, const char *name,
 		p->cwd[0] = '\0';
 		p->cwd[0] = '\0';
 	name_copy(p->name, name);
 	name_copy(p->name, name);
 	procs[slot] = p;
 	procs[slot] = p;
-	sched_add_ready(p);
+	/* The caller publishes the process as runnable only after applying any
+	 * per-process overrides (cwd/tty): an AP can pick it as soon as it is
+	 * on the run queue. */
 }
 }
 
 
 void process_init(void)
 void process_init(void)
@@ -175,28 +183,57 @@ void process_init(void)
 	for (i = 0; i < PROC_MAX; i++)
 	for (i = 0; i < PROC_MAX; i++)
 		procs[i] = 0;
 		procs[i] = 0;
 
 
-	clear_proc(&idle_proc);
-	idle_proc.state      = PROC_RUNNING;
-	idle_proc.pid        = 0;
-	idle_proc.slot       = PROC_MAX;
-	idle_proc.esp        = 0;
-	idle_proc.screen     = (void *)0;
-	idle_proc.stack_base = 0;  /* bootstrap stack, never freed */
-	idle_proc.cr3        = paging_current_cr3();
-	idle_proc.kstack_top = (unsigned int)__boot_stack_top;
-	idle_proc.cpu_ticks  = 0;
-	idle_proc.start_ticks = 0;
-	idle_proc.on_cpu     = 0;
-	idle_proc.last_cpu   = 0;
-	idle_proc.affinity   = 1;
-	idle_proc.weight     = 0;
-	idle_proc.stride     = 0;
-	idle_proc.vruntime   = 0;
-	idle_proc.on_runq    = 0;
-	name_copy(idle_proc.name, "idle");
-
-	cpus[0].idle = &idle_proc;
-	cpus[0].current = &idle_proc;
+	clear_proc(&idle_procs[0]);
+	idle_procs[0].state      = PROC_RUNNING;
+	idle_procs[0].pid        = 0;
+	idle_procs[0].slot       = PROC_MAX;
+	idle_procs[0].esp        = 0;
+	idle_procs[0].screen     = (void *)0;
+	idle_procs[0].stack_base = 0;  /* bootstrap stack, never freed */
+	idle_procs[0].cr3        = paging_current_cr3();
+	idle_procs[0].kstack_top = (unsigned int)__boot_stack_top;
+	idle_procs[0].cpu_ticks  = 0;
+	idle_procs[0].start_ticks = 0;
+	idle_procs[0].on_cpu     = 0;
+	idle_procs[0].last_cpu   = 0;
+	idle_procs[0].affinity   = 1;
+	idle_procs[0].weight     = 0;
+	idle_procs[0].stride     = 0;
+	idle_procs[0].vruntime   = 0;
+	idle_procs[0].on_runq    = 0;
+	name_copy(idle_procs[0].name, "idle");
+
+	cpus[0].idle = &idle_procs[0];
+	cpus[0].current = &idle_procs[0];
+}
+
+/* Prepare an AP idle process and its kernel stack.  Called by the BSP before
+ * SIPI; the AP itself publishes cpus[cpu].current in smp_ap_main(). */
+unsigned int process_init_ap(unsigned int cpu)
+{
+	struct process *p;
+	unsigned int base;
+
+	if (cpu == 0 || cpu >= MAX_CPUS)
+		return 0;
+	base = mm_alloc_page();
+	if (!base)
+		return 0;
+	p = &idle_procs[cpu];
+	clear_proc(p);
+	p->state      = PROC_RUNNING;
+	p->pid        = 0;
+	p->slot       = PROC_MAX;
+	p->esp        = 0;
+	p->cr3        = paging_kernel_cr3();
+	p->kstack_top = base + PROC_STACK_SIZE;
+	p->on_cpu     = cpu;
+	p->last_cpu   = cpu;
+	p->affinity   = 1u << cpu;
+	p->start_ticks = 0;
+	name_copy(p->name, "idle");
+	cpus[cpu].idle = p;
+	return p->kstack_top;
 }
 }
 
 
 /*
 /*
@@ -216,8 +253,35 @@ void process_init(void)
  *   EAX  ECX  EDX  EBX  ESP_orig  EBP  ESI  EDI   (pusha frame)
  *   EAX  ECX  EDX  EBX  ESP_orig  EBP  ESI  EDI   (pusha frame)
  *   <- saved ESP
  *   <- saved ESP
  */
  */
+/* Kernel-thread entry wrappers: take the BKL for the thread's lifetime and
+ * hand the real entry point the argument the stack frame supplied. */
+static void process_kernel_start0(void)
+{
+	struct process *p = current_proc();
+	void (*fn)(void) = (void (*)(void))p->kentry;
+
+	bkl_lock();
+	if (fn)
+		fn();
+	bkl_unlock();
+	process_exit();
+}
+
+static void process_kernel_start_arg(const char *arg)
+{
+	struct process *p = current_proc();
+	void (*fn)(const char *) = (void (*)(const char *))p->kentry;
+
+	bkl_lock();
+	if (fn)
+		fn(arg);
+	bkl_unlock();
+	process_exit();
+}
+
 static int setup_process(const char *name, unsigned int entry,
 static int setup_process(const char *name, unsigned int entry,
-			 int has_arg, const char *arg)
+			 int has_arg, const char *arg,
+			 void (*kentry)(void), int kernel)
 {
 {
 	int slot, owned, pid;
 	int slot, owned, pid;
 	unsigned int base, top, *sp, arg_ptr = 0, cr3;
 	unsigned int base, top, *sp, arg_ptr = 0, cr3;
@@ -280,7 +344,9 @@ static int setup_process(const char *name, unsigned int entry,
 	*--sp = 0;              /* ESI */
 	*--sp = 0;              /* ESI */
 	*--sp = 0;              /* EDI */
 	*--sp = 0;              /* EDI */
 
 
-	proc_commit(p, slot, name, (unsigned int)sp, owned, base, cr3, top);
+	proc_commit(p, slot, name, (unsigned int)sp, owned, base, cr3, top,
+		    kentry, kernel);
+	sched_add_ready(p);
 	pid = (int)p->pid;
 	pid = (int)p->pid;
 	irq_restore(flags);
 	irq_restore(flags);
 	return pid;
 	return pid;
@@ -288,13 +354,15 @@ static int setup_process(const char *name, unsigned int entry,
 
 
 int process_create(const char *name, void (*entry)(void))
 int process_create(const char *name, void (*entry)(void))
 {
 {
-	return setup_process(name, (unsigned int)entry, 0, (void *)0);
+	return setup_process(name, (unsigned int)process_kernel_start0, 0,
+			     (void *)0, entry, 1);
 }
 }
 
 
 int process_create_arg(const char *name, void (*entry)(const char *),
 int process_create_arg(const char *name, void (*entry)(const char *),
 		       const char *arg)
 		       const char *arg)
 {
 {
-	return setup_process(name, (unsigned int)entry, 1, arg);
+	return setup_process(name, (unsigned int)process_kernel_start_arg, 1,
+			     arg, (void (*)(void))entry, 1);
 }
 }
 
 
 /* Create a process whose entry takes a raw pointer argument (no string copy). */
 /* Create a process whose entry takes a raw pointer argument (no string copy). */
@@ -334,11 +402,13 @@ int process_create_ptr(const char *name, void (*entry)(void *), void *arg)
 	*--sp = (unsigned int)process_exit;/* return address */
 	*--sp = (unsigned int)process_exit;/* return address */
 	*--sp = 0x202;                    /* EFLAGS: IF=1 */
 	*--sp = 0x202;                    /* EFLAGS: IF=1 */
 	*--sp = 0x08;                     /* CS */
 	*--sp = 0x08;                     /* CS */
-	*--sp = (unsigned int)entry;      /* EIP */
+	*--sp = (unsigned int)process_kernel_start_arg; /* EIP */
 	*--sp = 0; *--sp = 0; *--sp = 0; *--sp = 0;  /* EAX ECX EDX EBX */
 	*--sp = 0; *--sp = 0; *--sp = 0; *--sp = 0;  /* EAX ECX EDX EBX */
 	*--sp = 0; *--sp = 0; *--sp = 0; *--sp = 0;  /* ESP EBP ESI EDI */
 	*--sp = 0; *--sp = 0; *--sp = 0; *--sp = 0;  /* ESP EBP ESI EDI */
 
 
-	proc_commit(p, slot, name, (unsigned int)sp, owned, base, cr3, top);
+	proc_commit(p, slot, name, (unsigned int)sp, owned, base, cr3, top,
+		    (void (*)(void))entry, 1);
+	sched_add_ready(p);
 	pid = (int)p->pid;
 	pid = (int)p->pid;
 	irq_restore(flags);
 	irq_restore(flags);
 	return pid;
 	return pid;
@@ -448,11 +518,13 @@ int process_create_user_tty(const char *name, const void *image, unsigned int si
 		*--sp = 0; *--sp = 0; *--sp = 0; *--sp = 0;  /* ESP EBP ESI EDI */
 		*--sp = 0; *--sp = 0; *--sp = 0; *--sp = 0;  /* ESP EBP ESI EDI */
 	}
 	}
 
 
-	proc_commit(p, slot, name, (unsigned int)sp, owned, base, cr3, top);
+	proc_commit(p, slot, name, (unsigned int)sp, owned, base, cr3, top,
+		    0, 0);
 	if (cwd)
 	if (cwd)
 		cwd_copy(p, cwd);
 		cwd_copy(p, cwd);
 	if (tty)
 	if (tty)
 		p->tty = (int)tty;
 		p->tty = (int)tty;
+	sched_add_ready(p);
 	pid = (int)p->pid;
 	pid = (int)p->pid;
 
 
 	irq_restore(flags);
 	irq_restore(flags);
@@ -499,8 +571,8 @@ void process_map_mmio_all(unsigned int phys_base, unsigned int size)
 	/* Map the kernel directory (the one new process directories clone) and
 	/* Map the kernel directory (the one new process directories clone) and
 	 * every live process copy.  paging_current_cr3() is the *current* process
 	 * every live process copy.  paging_current_cr3() is the *current* process
 	 * directory, which may be one of these; mapping it twice is harmless.
 	 * directory, which may be one of these; mapping it twice is harmless.
-	 * The reaper frees PCBs, so hold off the PIT while iterating. */
-	flags = irq_save_disable();
+	 * The reaper frees PCBs, so hold the process lock while iterating. */
+	flags = sched_lock_acquire();
 	paging_map_mmio(phys_base, size);
 	paging_map_mmio(phys_base, size);
 	for (i = 0; i < PROC_MAX; i++) {
 	for (i = 0; i < PROC_MAX; i++) {
 		struct process *p = procs[i];
 		struct process *p = procs[i];
@@ -510,15 +582,23 @@ void process_map_mmio_all(unsigned int phys_base, unsigned int size)
 		if (p->cr3)
 		if (p->cr3)
 			paging_map_mmio_cr3(p->cr3, phys_base, size);
 			paging_map_mmio_cr3(p->cr3, phys_base, size);
 	}
 	}
-	irq_restore(flags);
+	sched_lock_release(flags);
 }
 }
 
 
-/* Unpublish a dead process and hand it to the kworker.  Called with interrupts
- * disabled (from schedule(), process_kill(), or process_exit()'s path). */
+/* Unpublish a dead process and hand it to the kworker.  Takes sched_lock
+ * itself; callers must have already switched away from its kernel stack (or
+ * never run it), so the reaper cannot free memory still in use. */
 void process_zombie_add(struct process *p)
 void process_zombie_add(struct process *p)
 {
 {
-	if (!p || p->zombie_queued)
+	unsigned int flags;
+
+	if (!p)
 		return;
 		return;
+	flags = sched_lock_acquire();
+	if (p->zombie_queued) {
+		sched_lock_release(flags);
+		return;
+	}
 	p->zombie_queued = 1;
 	p->zombie_queued = 1;
 	p->on_cpu = PROC_NO_CPU;
 	p->on_cpu = PROC_NO_CPU;
 	if (p->slot < PROC_MAX)
 	if (p->slot < PROC_MAX)
@@ -529,17 +609,26 @@ void process_zombie_add(struct process *p)
 	else
 	else
 		zombie_head = p;
 		zombie_head = p;
 	zombie_tail = p;
 	zombie_tail = p;
+	sched_lock_release(flags);
 }
 }
 
 
-unsigned int process_zombie_count(void)
+unsigned int process_zombie_count_locked(void)
 {
 {
-	unsigned int flags, n = 0;
+	unsigned int n = 0;
 	struct process *p;
 	struct process *p;
 
 
-	flags = irq_save_disable();
 	for (p = zombie_head; p; p = p->zombie_next)
 	for (p = zombie_head; p; p = p->zombie_next)
 		n++;
 		n++;
-	irq_restore(flags);
+	return n;
+}
+
+unsigned int process_zombie_count(void)
+{
+	unsigned int flags, n;
+
+	flags = sched_lock_acquire();
+	n = process_zombie_count_locked();
+	sched_lock_release(flags);
 	return n;
 	return n;
 }
 }
 
 
@@ -554,10 +643,10 @@ void process_reap_dead(void)
 	struct process *p, *list;
 	struct process *p, *list;
 
 
 	preempt_disable();
 	preempt_disable();
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	list = zombie_head;
 	list = zombie_head;
 	zombie_head = zombie_tail = 0;
 	zombie_head = zombie_tail = 0;
-	irq_restore(flags);
+	sched_lock_release(flags);
 
 
 	for (p = list; p; ) {
 	for (p = list; p; ) {
 		struct process *next = p->zombie_next;
 		struct process *next = p->zombie_next;
@@ -643,12 +732,12 @@ void process_ps(void)
 		unsigned int pid, state, weight, cpu, flags;
 		unsigned int pid, state, weight, cpu, flags;
 		char name[PROC_NAME_LEN];
 		char name[PROC_NAME_LEN];
 
 
-		/* The reaper frees PCBs; snapshot this slot with the PIT off and
-		 * print the copy outside the critical section. */
-		flags = irq_save_disable();
+		/* The reaper frees PCBs; snapshot this slot under the process
+		 * lock and print the copy outside the critical section. */
+		flags = sched_lock_acquire();
 		p = procs[i];
 		p = procs[i];
 		if (!p || p->state == PROC_DEAD) {
 		if (!p || p->state == PROC_DEAD) {
-			irq_restore(flags);
+			sched_lock_release(flags);
 			continue;
 			continue;
 		}
 		}
 		pid = p->pid;
 		pid = p->pid;
@@ -658,7 +747,7 @@ void process_ps(void)
 		for (j = 0; j < sizeof(name) - 1 && p->name[j]; j++)
 		for (j = 0; j < sizeof(name) - 1 && p->name[j]; j++)
 			name[j] = p->name[j];
 			name[j] = p->name[j];
 		name[j] = '\0';
 		name[j] = '\0';
-		irq_restore(flags);
+		sched_lock_release(flags);
 
 
 		io_putchar(' ');
 		io_putchar(' ');
 		put_dec(pid);
 		put_dec(pid);
@@ -714,7 +803,7 @@ int process_procinfo(unsigned int index, struct sys_procinfo *out)
 
 
 	if (!out)
 	if (!out)
 		return 0;
 		return 0;
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	for (i = 0; i < PROC_MAX; i++) {
 	for (i = 0; i < PROC_MAX; i++) {
 		struct process *p = procs[i];
 		struct process *p = procs[i];
 
 
@@ -734,21 +823,21 @@ int process_procinfo(unsigned int index, struct sys_procinfo *out)
 		out->weight     = p->weight;
 		out->weight     = p->weight;
 		out->cpu        = p->on_cpu;
 		out->cpu        = p->on_cpu;
 		name_copy(out->name, p->name);
 		name_copy(out->name, p->name);
-		irq_restore(flags);
+		sched_lock_release(flags);
 		return 1;
 		return 1;
 	}
 	}
-	irq_restore(flags);
+	sched_lock_release(flags);
 	return 0;
 	return 0;
 }
 }
 
 
 int process_kill(unsigned int pid)
 int process_kill(unsigned int pid)
 {
 {
 	struct process *target = 0;
 	struct process *target = 0;
-	unsigned int i, flags;
+	unsigned int i, flags, on_cpu;
 
 
 	if (pid == 0)
 	if (pid == 0)
 		return -1; /* cannot kill the kernel idle process */
 		return -1; /* cannot kill the kernel idle process */
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	for (i = 0; i < PROC_MAX; i++) {
 	for (i = 0; i < PROC_MAX; i++) {
 		struct process *p = procs[i];
 		struct process *p = procs[i];
 
 
@@ -758,21 +847,32 @@ int process_kill(unsigned int pid)
 		}
 		}
 	}
 	}
 	if (!target) {
 	if (!target) {
-		irq_restore(flags);
+		sched_lock_release(flags);
 		return -1;
 		return -1;
 	}
 	}
 	if (target == current_proc()) {
 	if (target == current_proc()) {
 		target->state = PROC_DEAD;
 		target->state = PROC_DEAD;
+		sched_lock_release(flags);
 		wait_wake_all(&exit_wq);
 		wait_wake_all(&exit_wq);
-		irq_restore(flags);
 		sched_yield();     /* never resumes */
 		sched_yield();     /* never resumes */
 		return 0;
 		return 0;
 	}
 	}
-	sched_unlink(target);
+	on_cpu = target->on_cpu;
+	sched_unlink_locked(target);
 	target->state = PROC_DEAD;
 	target->state = PROC_DEAD;
-	process_zombie_add(target);
+	sched_lock_release(flags);
+
 	wait_wake_all(&exit_wq);
 	wait_wake_all(&exit_wq);
-	irq_restore(flags);
+
+	if (on_cpu == PROC_NO_CPU) {
+		/* Not running anywhere: its page directory is no longer in any
+		 * CR3, so the reaper may take it now. */
+		process_zombie_add(target);
+	} else {
+		/* The target CPU's next schedule queues the zombie only after it
+		 * has switched away from it. */
+		smp_resched_cpu(on_cpu);
+	}
 	return 0;
 	return 0;
 }
 }
 
 
@@ -785,17 +885,17 @@ int process_set_weight(unsigned int pid, unsigned int weight)
 	if (weight > SCHED_WEIGHT_MAX)
 	if (weight > SCHED_WEIGHT_MAX)
 		weight = SCHED_WEIGHT_MAX;
 		weight = SCHED_WEIGHT_MAX;
 
 
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	for (i = 0; i < PROC_MAX; i++) {
 	for (i = 0; i < PROC_MAX; i++) {
 		struct process *p = procs[i];
 		struct process *p = procs[i];
 
 
 		if (p && p->state != PROC_DEAD && p->pid == pid) {
 		if (p && p->state != PROC_DEAD && p->pid == pid) {
 			sched_set_weight_locked(p, weight);
 			sched_set_weight_locked(p, weight);
-			irq_restore(flags);
+			sched_lock_release(flags);
 			return 0;
 			return 0;
 		}
 		}
 	}
 	}
-	irq_restore(flags);
+	sched_lock_release(flags);
 	return -1;
 	return -1;
 }
 }
 
 
@@ -805,17 +905,17 @@ int process_set_affinity(unsigned int pid, unsigned int mask)
 
 
 	if (!mask)
 	if (!mask)
 		return -1;
 		return -1;
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	for (i = 0; i < PROC_MAX; i++) {
 	for (i = 0; i < PROC_MAX; i++) {
 		struct process *p = procs[i];
 		struct process *p = procs[i];
 
 
 		if (p && p->state != PROC_DEAD && p->pid == pid) {
 		if (p && p->state != PROC_DEAD && p->pid == pid) {
 			p->affinity = mask;
 			p->affinity = mask;
-			irq_restore(flags);
+			sched_lock_release(flags);
 			return 0;
 			return 0;
 		}
 		}
 	}
 	}
-	irq_restore(flags);
+	sched_lock_release(flags);
 	return -1;
 	return -1;
 }
 }
 
 
@@ -845,9 +945,9 @@ int process_is_alive(unsigned int pid)
 	unsigned int flags;
 	unsigned int flags;
 	int alive;
 	int alive;
 
 
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	alive = process_alive_locked(pid);
 	alive = process_alive_locked(pid);
-	irq_restore(flags);
+	sched_lock_release(flags);
 	return alive;
 	return alive;
 }
 }
 
 
@@ -858,7 +958,7 @@ int process_is_sleeping(unsigned int pid)
 
 
 	if (pid == 0)
 	if (pid == 0)
 		return 0;
 		return 0;
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	for (i = 0; i < PROC_MAX; i++) {
 	for (i = 0; i < PROC_MAX; i++) {
 		struct process *p = procs[i];
 		struct process *p = procs[i];
 
 
@@ -867,7 +967,7 @@ int process_is_sleeping(unsigned int pid)
 			break;
 			break;
 		}
 		}
 	}
 	}
-	irq_restore(flags);
+	sched_lock_release(flags);
 	return sleeping;
 	return sleeping;
 }
 }
 
 
@@ -875,7 +975,7 @@ void process_set_tty(unsigned int pid, int tty)
 {
 {
 	unsigned int i, flags;
 	unsigned int i, flags;
 
 
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	for (i = 0; i < PROC_MAX; i++) {
 	for (i = 0; i < PROC_MAX; i++) {
 		struct process *p = procs[i];
 		struct process *p = procs[i];
 
 
@@ -884,7 +984,7 @@ void process_set_tty(unsigned int pid, int tty)
 			break;
 			break;
 		}
 		}
 	}
 	}
-	irq_restore(flags);
+	sched_lock_release(flags);
 }
 }
 
 
 /* Block the current process for `ms` milliseconds.  The sleep list wakes it
 /* Block the current process for `ms` milliseconds.  The sleep list wakes it
@@ -899,14 +999,14 @@ void process_sleep_ms(unsigned int ms)
 void process_wait(unsigned int pid)
 void process_wait(unsigned int pid)
 {
 {
 	for (;;) {
 	for (;;) {
-		unsigned int flags = irq_save_disable();
+		unsigned int flags = sched_lock_acquire();
 
 
 		if (!process_alive_locked(pid)) {
 		if (!process_alive_locked(pid)) {
-			irq_restore(flags);
+			sched_lock_release(flags);
 			return;
 			return;
 		}
 		}
 		sched_block_on_locked(&exit_wq);
 		sched_block_on_locked(&exit_wq);
-		irq_restore(flags);
+		sched_lock_release(flags);
 		sched_yield();
 		sched_yield();
 	}
 	}
 }
 }

+ 10 - 0
process.h

@@ -73,6 +73,12 @@ struct process {
 	struct wait_queue *waiting_on;
 	struct wait_queue *waiting_on;
 	struct process *zombie_next;    /* reaped by the kworker */
 	struct process *zombie_next;    /* reaped by the kworker */
 	unsigned int zombie_queued;
 	unsigned int zombie_queued;
+	/* Kernel-thread entry: set for processes created with
+	 * process_create*(), whose wrapper takes the BKL. */
+	void (*kentry)(void);
+	unsigned int kernel_thread;
+	/* BKL depth saved when the process blocks and must reacquire it. */
+	unsigned int bkl_saved;
 };
 };
 
 
 /* User-mode virtual address layout (ring-3 processes). */
 /* User-mode virtual address layout (ring-3 processes). */
@@ -89,6 +95,8 @@ struct process {
 extern struct process *procs[PROC_MAX];
 extern struct process *procs[PROC_MAX];
 
 
 void process_init(void);
 void process_init(void);
+/* Prepare an AP idle PCB and stack; returns its kernel stack top, or 0. */
+unsigned int process_init_ap(unsigned int cpu);
 int  process_create(const char *name, void (*entry)(void));
 int  process_create(const char *name, void (*entry)(void));
 int  process_create_arg(const char *name, void (*entry)(const char *),
 int  process_create_arg(const char *name, void (*entry)(const char *),
 			const char *arg);
 			const char *arg);
@@ -118,6 +126,8 @@ void process_exit(void);
 /* Called by the kworker: free switched-out dead processes. */
 /* Called by the kworker: free switched-out dead processes. */
 void process_reap_dead(void);
 void process_reap_dead(void);
 unsigned int process_zombie_count(void);
 unsigned int process_zombie_count(void);
+/* Caller holds sched_lock. */
+unsigned int process_zombie_count_locked(void);
 /* Called from schedule()/kill with interrupts disabled: unpublish a dead PCB
 /* Called from schedule()/kill with interrupts disabled: unpublish a dead PCB
  * and put it on the kworker's zombie list. */
  * and put it on the kworker's zombie list. */
 void process_zombie_add(struct process *p);
 void process_zombie_add(struct process *p);

+ 296 - 49
sched.c

@@ -9,14 +9,16 @@
  * interactive process is scheduled promptly without accumulating unbounded
  * interactive process is scheduled promptly without accumulating unbounded
  * credit.
  * credit.
  *
  *
- * The run queue is a single global list for now.  That is also the shape an
- * SMP version wants: one queue protected by a lock, with per-CPU current.
+ * The run queue is a single global list protected by sched_lock; per-CPU
+ * current lets every CPU pick the next eligible task from it.
  */
  */
 
 
 #include "sched.h"
 #include "sched.h"
 #include "io.h"
 #include "io.h"
 #include "paging.h"
 #include "paging.h"
 #include "process.h"
 #include "process.h"
+#include "smp.h"
+#include "spinlock.h"
 #include "syscall.h"
 #include "syscall.h"
 #include "timer.h"
 #include "timer.h"
 #include "tss.h"
 #include "tss.h"
@@ -33,25 +35,48 @@ static struct process *runq_head, *runq_tail;
 static struct process *sleep_head, *sleep_tail;
 static struct process *sleep_head, *sleep_tail;
 static unsigned int wakeups;
 static unsigned int wakeups;
 
 
-/* switch.S reads these two facts directly: current at %gs:0 and esp at
- * offset 0 of the process.  Keep them true. */
-_Static_assert(__builtin_offsetof(struct cpu, current) == 0,
-	       "switch.S reads current from %gs:0");
-_Static_assert(__builtin_offsetof(struct process, esp) == 0,
-	       "switch.S saves the interrupted esp at offset 0");
+/* Protects the global run/sleep/wait queues and the process-table state that
+ * the timer path touches.  This is a spinlock: it is held with interrupts
+ * off for short, non-blocking critical sections only. */
+static spinlock_t sched_lock;
 
 
-static inline unsigned int irq_save_disable(void)
+/* Sleeping big kernel lock.  One owner process at a time (recursive); the
+ * owner is never timer-preempted, and every voluntary block drops it so the
+ * other CPU can keep making progress.  See bkl_lock()/bkl_unlock(). */
+static struct {
+	struct process *owner;
+	unsigned int depth;
+	struct wait_queue wait;
+} bkl;
+
+unsigned int sched_lock_acquire(void)
 {
 {
-	unsigned int flags;
-	__asm__ volatile ("pushfl; popl %0; cli" : "=r"(flags));
+	unsigned int flags = spin_irq_save_disable();
+
+	spin_lock(&sched_lock);
 	return flags;
 	return flags;
 }
 }
 
 
-static inline void irq_restore(unsigned int flags)
+void sched_lock_release(unsigned int flags)
+{
+	spin_unlock(&sched_lock);
+	spin_irq_restore(flags);
+}
+
+void sched_lock_reset(void)
 {
 {
-	__asm__ volatile ("pushl %0; popfl" : : "r"(flags));
+	sched_lock.locked = 0;
 }
 }
 
 
+/* switch.S reads these two facts directly: current at %gs:0 and esp at
+ * offset 0 of the process.  Keep them true. */
+_Static_assert(__builtin_offsetof(struct cpu, current) == 0,
+	       "switch.S reads current from %gs:0");
+_Static_assert(__builtin_offsetof(struct process, esp) == 0,
+	       "switch.S saves the interrupted esp at offset 0");
+_Static_assert(__builtin_offsetof(struct cpu, need_resched) == 24,
+	       "switch.S sets need_resched at offset 24");
+
 static inline unsigned long long weight_stride(unsigned int weight)
 static inline unsigned long long weight_stride(unsigned int weight)
 {
 {
 	if (weight < 1)
 	if (weight < 1)
@@ -212,13 +237,15 @@ static void wake_locked(struct process *p)
 		p->vruntime = min_vruntime_locked();
 		p->vruntime = min_vruntime_locked();
 	runq_add_locked(p);
 	runq_add_locked(p);
 	wakeups++;
 	wakeups++;
-	if (cur && (cur == c->idle ||
-		    p->weight >= cur->weight))
+	if (cur && (p->affinity & (1u << c->id)) &&
+	    (cur == c->idle || p->weight >= cur->weight))
 		c->need_resched = 1;
 		c->need_resched = 1;
 }
 }
 
 
 void sched_add_ready(struct process *p)
 void sched_add_ready(struct process *p)
 {
 {
+	unsigned int flags = sched_lock_acquire();
+
 	if (!p->weight) {
 	if (!p->weight) {
 		p->weight = SCHED_WEIGHT_DEFAULT;
 		p->weight = SCHED_WEIGHT_DEFAULT;
 		p->stride = weight_stride(p->weight);
 		p->stride = weight_stride(p->weight);
@@ -226,9 +253,10 @@ void sched_add_ready(struct process *p)
 	}
 	}
 	p->state = PROC_READY;
 	p->state = PROC_READY;
 	runq_add_locked(p);
 	runq_add_locked(p);
+	sched_lock_release(flags);
 }
 }
 
 
-void sched_unlink(struct process *p)
+void sched_unlink_locked(struct process *p)
 {
 {
 	runq_del_locked(p);
 	runq_del_locked(p);
 	if (p->waiting_on)
 	if (p->waiting_on)
@@ -237,6 +265,14 @@ void sched_unlink(struct process *p)
 		sleepq_del_locked(p);
 		sleepq_del_locked(p);
 }
 }
 
 
+void sched_unlink(struct process *p)
+{
+	unsigned int flags = sched_lock_acquire();
+
+	sched_unlink_locked(p);
+	sched_lock_release(flags);
+}
+
 void sched_set_weight_locked(struct process *p, unsigned int weight)
 void sched_set_weight_locked(struct process *p, unsigned int weight)
 {
 {
 	if (!p)
 	if (!p)
@@ -271,9 +307,9 @@ void sched_block_on(struct wait_queue *wq)
 
 
 	if (this_cpu()->preempt_count)
 	if (this_cpu()->preempt_count)
 		return;
 		return;
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	sched_block_on_locked(wq);
 	sched_block_on_locked(wq);
-	irq_restore(flags);
+	sched_lock_release(flags);
 	sched_yield();
 	sched_yield();
 }
 }
 
 
@@ -292,13 +328,13 @@ void sched_block_timeout(struct wait_queue *wq, unsigned int ms)
 	}
 	}
 	if (this_cpu()->preempt_count)
 	if (this_cpu()->preempt_count)
 		return;
 		return;
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	p->wake_ms = get_ticks_ms() + ms;
 	p->wake_ms = get_ticks_ms() + ms;
 	p->state = PROC_SLEEPING;
 	p->state = PROC_SLEEPING;
 	runq_del_locked(p);
 	runq_del_locked(p);
 	waitq_add_locked(wq, p);
 	waitq_add_locked(wq, p);
 	sleepq_add_locked(p);
 	sleepq_add_locked(p);
-	irq_restore(flags);
+	sched_lock_release(flags);
 	sched_yield();
 	sched_yield();
 }
 }
 
 
@@ -315,19 +351,162 @@ static void wake_one_locked(struct wait_queue *wq)
 
 
 void wait_wake_one(struct wait_queue *wq)
 void wait_wake_one(struct wait_queue *wq)
 {
 {
-	unsigned int flags = irq_save_disable();
+	unsigned int flags = sched_lock_acquire();
 
 
 	wake_one_locked(wq);
 	wake_one_locked(wq);
-	irq_restore(flags);
+	sched_lock_release(flags);
 }
 }
 
 
 void wait_wake_all(struct wait_queue *wq)
 void wait_wake_all(struct wait_queue *wq)
 {
 {
-	unsigned int flags = irq_save_disable();
+	unsigned int flags = sched_lock_acquire();
 
 
 	while (wq->head)
 	while (wq->head)
 		wake_one_locked(wq);
 		wake_one_locked(wq);
-	irq_restore(flags);
+	sched_lock_release(flags);
+}
+
+/* ---- sleeping big kernel lock ---- */
+
+void bkl_init(void)
+{
+	bkl.owner = 0;
+	bkl.depth = 0;
+	wait_queue_init(&bkl.wait);
+}
+
+void bkl_lock(void)
+{
+	struct process *p = current_proc();
+	unsigned int flags;
+
+	if (!p)
+		return;
+	if (bkl.owner == p) {
+		bkl.depth++;
+		return;
+	}
+	for (;;) {
+		flags = sched_lock_acquire();
+		if (!bkl.owner) {
+			bkl.owner = p;
+			bkl.depth = 1;
+			sched_lock_release(flags);
+			return;
+		}
+		sched_block_on_locked(&bkl.wait);
+		sched_lock_release(flags);
+		sched_yield();
+	}
+}
+
+void bkl_unlock(void)
+{
+	struct process *p = current_proc();
+	unsigned int flags;
+
+	if (!p)
+		return;
+	flags = sched_lock_acquire();
+	if (bkl.owner == p) {
+		if (bkl.depth > 1) {
+			bkl.depth--;
+		} else {
+			bkl.owner = 0;
+			bkl.depth = 0;
+			wake_one_locked(&bkl.wait);
+		}
+	}
+	sched_lock_release(flags);
+}
+
+int bkl_trylock(void)
+{
+	struct process *p = current_proc();
+	unsigned int flags;
+
+	if (!p)
+		return 1;
+	if (bkl.owner == p) {
+		bkl.depth++;
+		return 1;
+	}
+	flags = sched_lock_acquire();
+	if (!bkl.owner) {
+		bkl.owner = p;
+		bkl.depth = 1;
+		sched_lock_release(flags);
+		return 1;
+	}
+	sched_lock_release(flags);
+	return 0;
+}
+
+/* Drop the BKL around a voluntary switch and take it back on resume.  The
+ * saved depth lives in the PCB: only the owner can be holding it, and it is
+ * the current process on this CPU. */
+static void bkl_yield_release(void)
+{
+	struct process *p = current_proc();
+	unsigned int flags;
+
+	if (!p)
+		return;
+	flags = sched_lock_acquire();
+	if (bkl.owner == p) {
+		p->bkl_saved = bkl.depth;
+		bkl.owner = 0;
+		bkl.depth = 0;
+		wake_one_locked(&bkl.wait);
+	}
+	/* Not the owner: leave bkl_saved alone.  A nested sleep inside a
+	 * bkl_pause() section must not forget that it still has to resume the
+	 * paused ownership. */
+	sched_lock_release(flags);
+}
+
+static void bkl_yield_reacquire(void)
+{
+	struct process *p = current_proc();
+	unsigned int depth;
+
+	if (!p)
+		return;
+	depth = p->bkl_saved;
+	if (!depth)
+		return;
+	p->bkl_saved = 0;
+	bkl_lock();
+	bkl.depth = depth;
+}
+
+void bkl_pause(void)
+{
+	bkl_yield_release();
+}
+
+void bkl_resume(void)
+{
+	bkl_yield_reacquire();
+}
+
+/* Run once on the new kernel stack: queue the process this CPU just left, so
+ * its stack/page directory cannot be freed while switch.S still uses them. */
+void sched_finish_switch(void)
+{
+	struct cpu *c = this_cpu();
+	struct process *p;
+	unsigned int flags;
+
+	if (!c->deferred_unlock)
+		return;
+	p = c->pending_zombie;
+	c->pending_zombie = 0;
+	flags = c->sched_flags;
+	c->deferred_unlock = 0;
+	sched_lock_release(flags);
+	if (p)
+		process_zombie_add(p);
 }
 }
 
 
 void sched_init(void)
 void sched_init(void)
@@ -338,6 +517,8 @@ void sched_init(void)
 	runq_head = runq_tail = 0;
 	runq_head = runq_tail = 0;
 	sleep_head = sleep_tail = 0;
 	sleep_head = sleep_tail = 0;
 	wakeups = 0;
 	wakeups = 0;
+	sched_lock.locked = 0;
+	bkl_init();
 
 
 	for (i = 0; i < MAX_CPUS; i++) {
 	for (i = 0; i < MAX_CPUS; i++) {
 		cpus[i].current = 0;
 		cpus[i].current = 0;
@@ -348,6 +529,8 @@ void sched_init(void)
 		cpus[i].preempt_count = 0;
 		cpus[i].preempt_count = 0;
 		cpus[i].idle_ticks = 0;
 		cpus[i].idle_ticks = 0;
 		cpus[i].switches = 0;
 		cpus[i].switches = 0;
+		cpus[i].pending_zombie = 0;
+		cpus[i].deferred_unlock = 0;
 	}
 	}
 }
 }
 
 
@@ -371,11 +554,17 @@ static void sched_switch_to(struct process *next)
 static struct process *pick_locked(void)
 static struct process *pick_locked(void)
 {
 {
 	struct process *p, *best = 0;
 	struct process *p, *best = 0;
-	unsigned int mask = 1u << this_cpu()->id;
+	struct cpu *c = this_cpu();
+	unsigned int mask = 1u << c->id;
 
 
 	for (p = runq_head; p; p = p->run_next) {
 	for (p = runq_head; p; p = p->run_next) {
 		if (!(p->affinity & mask))
 		if (!(p->affinity & mask))
 			continue;
 			continue;
+		/* A process still marked on another CPU is a wake-up race
+		 * victim: it is about to reach its own CPU's schedule().  Let
+		 * that CPU deal with it instead of running it twice. */
+		if (p->on_cpu != PROC_NO_CPU && p->on_cpu != c->id)
+			continue;
 		if (!best || p->vruntime < best->vruntime ||
 		if (!best || p->vruntime < best->vruntime ||
 		    (p->vruntime == best->vruntime && p->pid < best->pid))
 		    (p->vruntime == best->vruntime && p->pid < best->pid))
 			best = p;
 			best = p;
@@ -406,6 +595,7 @@ void schedule(void)
 	struct cpu *c = this_cpu();
 	struct cpu *c = this_cpu();
 	struct process *cur = c->current;
 	struct process *cur = c->current;
 	struct process *next;
 	struct process *next;
+	unsigned int flags = sched_lock_acquire();
 
 
 	if (cur && cur != c->idle) {
 	if (cur && cur != c->idle) {
 		if (cur->state == PROC_RUNNING) {
 		if (cur->state == PROC_RUNNING) {
@@ -416,9 +606,10 @@ void schedule(void)
 			/* Woken between setting BLOCKED/SLEEPING and entering
 			/* Woken between setting BLOCKED/SLEEPING and entering
 			 * the gate: already on the run queue. */
 			 * the gate: already on the run queue. */
 		} else if (cur->state == PROC_DEAD) {
 		} else if (cur->state == PROC_DEAD) {
-			/* Keep the PCB alive until the kworker can free it, but
-			 * unpublish it immediately. */
-			process_zombie_add(cur);
+			/* Defer the zombie until switch.S has loaded the new
+			 * stack: its kernel stack and page directory are still
+			 * live until then. */
+			c->pending_zombie = cur;
 		}
 		}
 	}
 	}
 
 
@@ -426,6 +617,7 @@ void schedule(void)
 	if (!next) {
 	if (!next) {
 		next = c->idle;
 		next = c->idle;
 		if (!next) {
 		if (!next) {
+			sched_lock_release(flags);
 			for (;;)
 			for (;;)
 				__asm__ volatile ("cli; hlt");
 				__asm__ volatile ("cli; hlt");
 		}
 		}
@@ -444,6 +636,11 @@ void schedule(void)
 	c->current = next;
 	c->current = next;
 	c->need_resched = 0;
 	c->need_resched = 0;
 	sched_switch_to(next);
 	sched_switch_to(next);
+	/* Keep the lock across the stack switch performed by switch.S; the
+	 * old process must not become runnable on another CPU while this CPU
+	 * is still using its kernel stack. */
+	c->sched_flags = flags;
+	c->deferred_unlock = 1;
 }
 }
 
 
 void sched_yield(void)
 void sched_yield(void)
@@ -457,7 +654,12 @@ void sched_yield(void)
 		c->need_resched = 1;
 		c->need_resched = 1;
 		return;
 		return;
 	}
 	}
+	/* Release the BKL before switching away and take it back after being
+	 * resumed; a process that never resumes (exit) simply never reacquires
+	 * it.  Blocking helpers therefore need no explicit BKL handling. */
+	bkl_yield_release();
 	__asm__ volatile ("int $0x81" ::: "memory");
 	__asm__ volatile ("int $0x81" ::: "memory");
+	bkl_yield_reacquire();
 }
 }
 
 
 void sched_sleep_ms(unsigned int ms)
 void sched_sleep_ms(unsigned int ms)
@@ -473,29 +675,32 @@ void sched_sleep_ms(unsigned int ms)
 	}
 	}
 	if (this_cpu()->preempt_count)
 	if (this_cpu()->preempt_count)
 		return;   /* sleeping inside a non-preemptible section is invalid */
 		return;   /* sleeping inside a non-preemptible section is invalid */
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	p->wake_ms = get_ticks_ms() + ms;
 	p->wake_ms = get_ticks_ms() + ms;
 	p->state = PROC_SLEEPING;
 	p->state = PROC_SLEEPING;
 	runq_del_locked(p);
 	runq_del_locked(p);
 	sleepq_add_locked(p);
 	sleepq_add_locked(p);
-	irq_restore(flags);
+	sched_lock_release(flags);
 	sched_yield();
 	sched_yield();
 }
 }
 
 
 /* Called from the PIT after ticks has been incremented.  Returns 1 when the
 /* Called from the PIT after ticks has been incremented.  Returns 1 when the
  * assembly should call schedule(). */
  * assembly should call schedule(). */
-int sched_tick_bsp(void)
+/* Common tail of the tick path: with the scheduler lock held, decide whether
+ * the assembly stub should call schedule(). */
+static int tick_decide_locked(struct cpu *c, struct process *p, int ap)
 {
 {
-	struct cpu *c = this_cpu();
-	struct process *p = c->current;
-
 	if (p) {
 	if (p) {
 		p->cpu_ticks++;
 		p->cpu_ticks++;
 		if (p == c->idle)
 		if (p == c->idle)
 			c->idle_ticks++;
 			c->idle_ticks++;
 	}
 	}
-
-	wake_sleepers_locked();
+	/* A process inside the kernel holding the big lock must not be
+	 * preempted: the lock is not preemptible, and its owner may be in the
+	 * middle of a kernel data structure update.  Voluntary block points
+	 * release it first. */
+	if (p && bkl.owner == p && p->state == PROC_RUNNING)
+		return 0;
 
 
 	if (c->preempt_count) {
 	if (c->preempt_count) {
 		if (p && p != c->idle && p->state == PROC_RUNNING &&
 		if (p && p != c->idle && p->state == PROC_RUNNING &&
@@ -505,10 +710,14 @@ int sched_tick_bsp(void)
 		return 0;
 		return 0;
 	}
 	}
 
 
+	/* APs only start taking work once the BSP has finished booting. */
+	if (ap && !smp_sched_enabled)
+		return 0;
+
 	if (!p)
 	if (!p)
 		return 0;
 		return 0;
 	if (p == c->idle) {
 	if (p == c->idle) {
-		if (runq_head) {
+		if (pick_locked()) {
 			c->need_resched = 1;
 			c->need_resched = 1;
 			return 1;
 			return 1;
 		}
 		}
@@ -523,6 +732,36 @@ int sched_tick_bsp(void)
 	return c->need_resched ? 1 : 0;
 	return c->need_resched ? 1 : 0;
 }
 }
 
 
+/* BSP PIT: also wakes the global sleep list (the PIT is the system clock). */
+int sched_tick_bsp(void)
+{
+	struct cpu *c = this_cpu();
+	struct process *p;
+	unsigned int flags = sched_lock_acquire();
+	int r;
+
+	p = c->current;
+	wake_sleepers_locked();
+	r = tick_decide_locked(c, p, 0);
+	sched_lock_release(flags);
+	return r;
+}
+
+/* AP local timer: no global tick counter and no sleeper wake-up (the BSP PIT
+ * owns both); only this CPU's scheduling decision. */
+int sched_tick_ap(void)
+{
+	struct cpu *c = this_cpu();
+	struct process *p;
+	unsigned int flags = sched_lock_acquire();
+	int r;
+
+	p = c->current;
+	r = tick_decide_locked(c, p, 1);
+	sched_lock_release(flags);
+	return r;
+}
+
 void preempt_disable(void)
 void preempt_disable(void)
 {
 {
 	this_cpu()->preempt_count++;
 	this_cpu()->preempt_count++;
@@ -543,7 +782,7 @@ int sched_snapshot(struct sys_schedinfo *out)
 
 
 	if (!out)
 	if (!out)
 		return -1;
 		return -1;
-	flags = irq_save_disable();
+	flags = sched_lock_acquire();
 	out->ncpus = cpu_count;
 	out->ncpus = cpu_count;
 	out->runnable = 0;
 	out->runnable = 0;
 	for (p = runq_head; p; p = p->run_next)
 	for (p = runq_head; p; p = p->run_next)
@@ -557,12 +796,16 @@ int sched_snapshot(struct sys_schedinfo *out)
 		if (p && p->state == PROC_BLOCKED)
 		if (p && p->state == PROC_BLOCKED)
 			out->blocked++;
 			out->blocked++;
 	}
 	}
-	out->zombies = process_zombie_count();
-	out->switches = cpus[0].switches;
+	out->zombies = process_zombie_count_locked();
+	out->switches = 0;
+	out->idle_ticks = 0;
+	for (i = 0; i < cpu_count && i < MAX_CPUS; i++) {
+		out->switches += cpus[i].switches;
+		out->idle_ticks += cpus[i].idle_ticks;
+	}
 	out->wakeups = wakeups;
 	out->wakeups = wakeups;
-	out->idle_ticks = cpus[0].idle_ticks;
 	out->slice_ticks = SCHED_SLICE_TICKS;
 	out->slice_ticks = SCHED_SLICE_TICKS;
-	irq_restore(flags);
+	sched_lock_release(flags);
 	return 0;
 	return 0;
 }
 }
 
 
@@ -586,22 +829,26 @@ static void sched_put_dec(unsigned int n)
 void sched_print(void)
 void sched_print(void)
 {
 {
 	struct sys_schedinfo s;
 	struct sys_schedinfo s;
-	unsigned int uptime, idle_pm, up_s, idle_s, idle_frac;
+	unsigned int uptime, idle_pm, up_s, total_s, idle_s, idle_frac, ncpu;
 
 
 	if (sched_snapshot(&s) != 0)
 	if (sched_snapshot(&s) != 0)
 		return;
 		return;
 	uptime = get_ticks_ms();
 	uptime = get_ticks_ms();
+	ncpu = s.ncpus ? s.ncpus : 1;
 
 
-	/* Avoid 64-bit division: scale to seconds first.  ticks wraps at ~49
-	 * days, so idle_s * 1000 stays within 32 bits for the useful range. */
+	/* Idle is a share of the total CPU capacity, so 4 idle CPUs at 50%
+	 * each report 50%, not 200%.  Avoid 64-bit division: scale to seconds
+	 * first (four CPUs overflow after ~12 days; good enough for the
+	 * display). */
 	up_s = uptime / 1000u;
 	up_s = uptime / 1000u;
-	if (up_s == 0) {
-		idle_pm = (s.idle_ticks >= uptime) ? 1000 : 0;
+	total_s = up_s * ncpu;
+	if (total_s == 0) {
+		idle_pm = (s.idle_ticks >= uptime * ncpu) ? 1000 : 0;
 	} else {
 	} else {
 		idle_s = s.idle_ticks / 1000u;
 		idle_s = s.idle_ticks / 1000u;
 		idle_frac = s.idle_ticks % 1000u;
 		idle_frac = s.idle_ticks % 1000u;
-		idle_pm = (idle_s * 1000u) / up_s;
-		idle_pm += (idle_frac * 1000u / up_s) / 1000u;
+		idle_pm = (idle_s * 1000u) / total_s;
+		idle_pm += (idle_frac * 1000u / total_s) / 1000u;
 		if (idle_pm > 1000u)
 		if (idle_pm > 1000u)
 			idle_pm = 1000u;
 			idle_pm = 1000u;
 	}
 	}

+ 40 - 4
sched.h

@@ -1,8 +1,8 @@
-/* Scheduler core: per-CPU state, the running process, and the tick entry.
+/* Scheduler core: per-CPU state, the running process, and the tick entries.
  *
  *
- * Only CPU 0 exists today, but everything is shaped for more CPUs later:
- * current is per-CPU, reached through a per-CPU GS segment, and the timer
- * path goes through sched_tick_cpu-style code rather than a global pid index.
+ * Every online CPU has its own current/idle/TSS via a per-CPU GS segment.
+ * The ready/sleep/wait queues are global and protected by sched_lock; the
+ * BSP PIT is the global clock, while APs run their own LAPIC timer.
  */
  */
 #ifndef SCHED_H
 #ifndef SCHED_H
 #define SCHED_H
 #define SCHED_H
@@ -33,6 +33,16 @@ struct cpu {
 	unsigned int    preempt_count;
 	unsigned int    preempt_count;
 	unsigned int    idle_ticks;
 	unsigned int    idle_ticks;
 	unsigned int    switches;
 	unsigned int    switches;
+	/* Dead process this CPU just switched away from.  It is queued only
+	 * after the new stack is live (sched_finish_switch), so the reaper can
+	 * never free a stack or page directory still in use. */
+	struct process *pending_zombie;
+	/* schedule() keeps the scheduler lock until switch.S has loaded the
+	 * new stack; sched_finish_switch() then drops it.  Without this the
+	 * old process could be picked by another CPU while this CPU still runs
+	 * on its kernel stack. */
+	unsigned int    deferred_unlock;
+	unsigned int    sched_flags;
 };
 };
 
 
 extern struct cpu cpus[MAX_CPUS];
 extern struct cpu cpus[MAX_CPUS];
@@ -62,6 +72,30 @@ static inline struct process *current_proc(void)
 void sched_init(void);
 void sched_init(void);
 void schedule(void);
 void schedule(void);
 int  sched_tick_bsp(void);
 int  sched_tick_bsp(void);
+int  sched_tick_ap(void);
+/* Queue the previous process after the assembly stub has loaded the new
+ * kernel stack (called from switch.S). */
+void sched_finish_switch(void);
+
+/* The global ready/sleep/wait/process locks.  Callers get the old interrupt
+ * state back and must pass it to sched_lock_release(). */
+unsigned int sched_lock_acquire(void);
+void sched_lock_release(unsigned int flags);
+/* Reinitialize the lock after the APs have been parked (an INIT can stop an
+ * AP while it holds the lock; the old run queue no longer matters). */
+void sched_lock_reset(void);
+
+/* Big kernel lock: a sleeping mutex serializing kernel execution.  Held for
+ * the duration of a syscall or kernel-thread work item, dropped around
+ * voluntary blocking so the other CPU can make progress. */
+void bkl_init(void);
+void bkl_lock(void);
+void bkl_unlock(void);
+int  bkl_trylock(void);
+/* Drop/reacquire the BKL around a long wait that does not touch kernel
+ * state (serial receive, device polling). */
+void bkl_pause(void);
+void bkl_resume(void);
 
 
 /* Voluntary reschedule through the int 0x81 gate.  Must not be called with
 /* Voluntary reschedule through the int 0x81 gate.  Must not be called with
  * preempt_count set; blocking helpers below handle that. */
  * preempt_count set; blocking helpers below handle that. */
@@ -72,6 +106,8 @@ void sched_yield(void);
 void sched_sleep_ms(unsigned int ms);
 void sched_sleep_ms(unsigned int ms);
 void sched_add_ready(struct process *p);
 void sched_add_ready(struct process *p);
 void sched_unlink(struct process *p);
 void sched_unlink(struct process *p);
+/* Caller holds sched_lock. */
+void sched_unlink_locked(struct process *p);
 
 
 /* Wait queues.  wait_wake_* may be called from any process context; the
 /* Wait queues.  wait_wake_* may be called from any process context; the
  * block helpers handle interrupts themselves. */
  * block helpers handle interrupts themselves. */

+ 13 - 3
serial.c

@@ -2,6 +2,7 @@
 #include "serial_ring_buf.h"
 #include "serial_ring_buf.h"
 #include "netlog.h"
 #include "netlog.h"
 #include "pci.h"
 #include "pci.h"
+#include "sched.h"
 #include "timer.h"
 #include "timer.h"
 
 
 void serial_init(unsigned int port)
 void serial_init(unsigned int port)
@@ -87,10 +88,19 @@ int serial_ring_getc(void)
 int serial_ring_wait(unsigned int timeout_ms, unsigned char *out)
 int serial_ring_wait(unsigned int timeout_ms, unsigned char *out)
 {
 {
 	unsigned int start = get_ticks_ms();
 	unsigned int start = get_ticks_ms();
+	int got = 0;
+
+	/* A kexec/payload receive can take minutes over 115200 baud.  Do not
+	 * keep the big kernel lock for that: the WM, terminals and the other
+	 * CPU's syscalls must keep running while we wait for bytes. */
+	bkl_pause();
 	while (get_ticks_ms() - start < timeout_ms) {
 	while (get_ticks_ms() - start < timeout_ms) {
-		if (serial_ring_pop(out))
-			return 1;
+		if (serial_ring_pop(out)) {
+			got = 1;
+			break;
+		}
 		__asm__ volatile ("hlt");
 		__asm__ volatile ("hlt");
 	}
 	}
-	return 0;
+	bkl_resume();
+	return got;
 }
 }

+ 237 - 0
smp.c

@@ -0,0 +1,237 @@
+/* SMP bring-up: ACPI CPU list, AP trampoline, INIT-SIPI and per-CPU idle.
+ *
+ * No MP-table fallback and no IOAPIC: this targets ACPI/MADT machines (QEMU
+ * and the Atom board).  If ACPI is absent or has one CPU, smp_init() leaves
+ * the system exactly as it was.
+ */
+
+#include "smp.h"
+#include "acpi.h"
+#include "cpu.h"
+#include "idle.h"
+#include "lapic.h"
+#include "paging.h"
+#include "process.h"
+#include "sched.h"
+#include "serial.h"
+#include "timer.h"
+#include "tss.h"
+
+#define AP_TRAMP_ADDR 0x6000u
+#define AP_INFO_ADDR  0x6F00u
+
+extern unsigned char _binary_ap_tramp_bin_start[];
+extern unsigned char _binary_ap_tramp_bin_end[];
+extern void idt_load_ap(void);
+
+static volatile unsigned int ap_started[MAX_CPUS];
+static unsigned int online_aps;
+unsigned int smp_sched_enabled;
+
+static void delay_ms(unsigned int ms)
+{
+	unsigned int t0 = get_ticks_ms();
+
+	while ((unsigned int)(get_ticks_ms() - t0) < ms)
+		cpu_idle();
+}
+
+static int copy_trampoline(void)
+{
+	unsigned int n = (unsigned int)(_binary_ap_tramp_bin_end -
+					_binary_ap_tramp_bin_start);
+	unsigned char *dst = (unsigned char *)AP_TRAMP_ADDR;
+	unsigned int i;
+
+	if (n == 0 || n > 0xC00)
+		return 0;
+	for (i = 0; i < n; i++)
+		dst[i] = _binary_ap_tramp_bin_start[i];
+	__asm__ volatile ("wbinvd" ::: "memory");
+	return 1;
+}
+
+static void start_one_ap(unsigned int cpu, unsigned int top)
+{
+	volatile unsigned int *info = (volatile unsigned int *)AP_INFO_ADDR;
+	unsigned int apic = acpi_cpu_apic_id(cpu);
+	unsigned int t0;
+
+	info[0] = paging_kernel_cr3();
+	info[1] = top;
+	info[2] = (unsigned int)smp_ap_main;
+	info[3] = cpu;
+	__sync_synchronize();
+
+	ap_started[cpu] = 0;
+	lapic_send_init(apic);
+	delay_ms(10);
+	lapic_send_sipi(apic, AP_TRAMP_ADDR >> 12);
+
+	/* The spec recommends a second SIPI if the first one was missed; some
+	 * firmware/boards drop it while the AP is still finishing INIT. */
+	t0 = get_ticks_ms();
+	while ((unsigned int)(get_ticks_ms() - t0) < 10) {
+		if (ap_started[cpu])
+			return;
+		cpu_idle();
+	}
+	if (!ap_started[cpu])
+		lapic_send_sipi(apic, AP_TRAMP_ADDR >> 12);
+
+	t0 = get_ticks_ms();
+	while ((unsigned int)(get_ticks_ms() - t0) < 500) {
+		if (ap_started[cpu])
+			return;
+		cpu_idle();
+	}
+
+	/* It did not check in; don't let it start later with stale info. */
+	lapic_send_init(apic);
+}
+
+void smp_init(void)
+{
+	unsigned int n = acpi_cpu_count();
+	unsigned int i;
+
+	online_aps = 0;
+	if (n <= 1) {
+		serial_puts(SERIAL_COM1, "smp: unavailable, 1 cpu\n");
+		return;
+	}
+	if (!acpi_lapic_base())
+		return;
+
+	lapic_set_base(acpi_lapic_base());
+	paging_map_mmio(acpi_lapic_base(), 0x1000);
+	/* The 4 MB MMIO page could share a PDE with the framebuffer; restore
+	 * the LFB's WC/PCD choice if so. */
+	paging_lfb_refresh();
+	if (!lapic_init_bsp()) {
+		serial_puts(SERIAL_COM1, "smp: BSP LAPIC init failed\n");
+		return;
+	}
+	cpus[0].lapic_id = lapic_id();
+	if (!copy_trampoline()) {
+		serial_puts(SERIAL_COM1, "smp: bad AP trampoline, staying UP\n");
+		return;
+	}
+
+	for (i = 1; i < n; i++) {
+		unsigned int top = process_init_ap(i);
+
+		if (!top) {
+			serial_puts(SERIAL_COM1, "smp: no stack for AP\n");
+			break;
+		}
+		start_one_ap(i, top);
+		if (!ap_started[i]) {
+			serial_puts(SERIAL_COM1, "smp: AP ");
+			serial_putdec(SERIAL_COM1, i);
+			serial_puts(SERIAL_COM1, " did not start\n");
+			break;
+		}
+		online_aps++;
+		cpu_count = online_aps + 1;
+	}
+
+	serial_puts(SERIAL_COM1, "smp: ");
+	serial_putdec(SERIAL_COM1, cpu_count);
+	serial_puts(SERIAL_COM1, " cpu online\n");
+}
+
+void smp_ap_main(unsigned int cpu)
+{
+	struct process *idle;
+
+	if (cpu == 0 || cpu >= MAX_CPUS) {
+		for (;;)
+			__asm__ volatile ("cli; hlt");
+	}
+
+	gdt_tss_init(cpu);
+	idt_load_ap();
+	/* An AP starts with the firmware's CR4; enable the SSE state before it
+	 * can reach the drawing fast paths (the BSP does this lazily). */
+	cpu_enable_sse();
+	cpus[cpu].id = cpu;
+	cpus[cpu].lapic_id = lapic_id();
+	idle = cpus[cpu].idle;
+	if (!idle)
+		for (;;)
+			__asm__ volatile ("cli; hlt");
+	cpus[cpu].current = idle;
+	idle->state = PROC_RUNNING;
+	idle->on_cpu = cpu;
+	idle->last_cpu = cpu;
+	cpus[cpu].tss->esp0 = idle->kstack_top;
+
+	lapic_init_ap();
+	if (!lapic_timer_start(LAPIC_TIMER_VECTOR))
+		for (;;)
+			__asm__ volatile ("cli; hlt");
+
+	ap_started[cpu] = 1;
+	__sync_synchronize();
+
+	__asm__ volatile ("sti");
+	for (;;)
+		cpu_idle();
+}
+
+void smp_resched_cpu(unsigned int cpu)
+{
+	if (cpu >= cpu_count || cpu >= MAX_CPUS)
+		return;
+	if (cpu == this_cpu()->id) {
+		cpus[cpu].need_resched = 1;
+		return;
+	}
+	if (cpus[cpu].lapic_id == 0xFFFFFFFFu)
+		return;
+	lapic_send_ipi(cpus[cpu].lapic_id, LAPIC_RESCHED_VECTOR);
+}
+
+void smp_enable_sched(void)
+{
+	unsigned int i;
+
+	if (cpu_count <= 1)
+		return;
+	smp_sched_enabled = 1;
+	for (i = 1; i < cpu_count; i++)
+		smp_resched_cpu(i);
+}
+
+void smp_stop_aps(void)
+{
+	unsigned int i, n;
+
+	/* The kexec trampoline is about to overwrite the kernel.  Stop taking
+	 * interrupts first (also on UP, so the final copy cannot be
+	 * preempted), park every AP, and wait long enough for INIT to take
+	 * effect.  An AP may have been inside a scheduler critical section, so
+	 * the lock is reinitialized afterwards; the old run queue is dead
+	 * anyway and the new kernel re-creates everything. */
+	__asm__ volatile ("cli");
+	if (cpu_count <= 1)
+		return;
+	for (i = 1; i < cpu_count; i++) {
+		if (cpus[i].lapic_id != 0xFFFFFFFFu)
+			lapic_send_init(cpus[i].lapic_id);
+	}
+	/* A few million PAUSE iterations is well over the 10 ms the INIT
+	 * protocol requires on the Atom/P5 range, and costs nothing on the
+	 * few occasions a kexec runs. */
+	for (n = 0; n < 4000000u; n++)
+		__asm__ volatile ("pause");
+	sched_lock_reset();
+}
+
+void smp_dump(void)
+{
+	serial_puts(SERIAL_COM1, "smp: online ");
+	serial_putdec(SERIAL_COM1, cpu_count);
+	serial_puts(SERIAL_COM1, "\n");
+}

+ 22 - 0
smp.h

@@ -0,0 +1,22 @@
+/* SMP boot: ACPI CPU discovery, AP trampoline and INIT-SIPI. */
+#ifndef SMP_H
+#define SMP_H
+
+extern unsigned int smp_sched_enabled;
+
+void smp_init(void);
+/* Let APs take user work, after boot-time device/FS setup is complete. */
+void smp_enable_sched(void);
+
+/* AP C entry (called by the trampoline). */
+void smp_ap_main(unsigned int cpu);
+
+/* Ask another CPU to run its scheduler soon (LAPIC IPI). */
+void smp_resched_cpu(unsigned int cpu);
+
+/* Park APs before kexec overwrites the kernel image. */
+void smp_stop_aps(void);
+
+void smp_dump(void);
+
+#endif

+ 54 - 0
spinlock.h

@@ -0,0 +1,54 @@
+/* Minimal x86 spinlocks for the SMP paths. */
+#ifndef SPINLOCK_H
+#define SPINLOCK_H
+
+typedef struct {
+	volatile unsigned int locked;
+} spinlock_t;
+
+static inline unsigned int spin_irq_save_disable(void)
+{
+	unsigned int flags;
+	__asm__ volatile ("pushfl; popl %0; cli" : "=r"(flags));
+	return flags;
+}
+
+static inline void spin_irq_restore(unsigned int flags)
+{
+	__asm__ volatile ("pushl %0; popfl" : : "r"(flags));
+}
+
+static inline void spin_pause(void)
+{
+	/* REP NOP is the PAUSE encoding and is harmless on older CPUs. */
+	__asm__ volatile ("rep; nop" ::: "memory");
+}
+
+static inline void spin_lock(spinlock_t *l)
+{
+	while (__sync_lock_test_and_set(&l->locked, 1u)) {
+		while (l->locked)
+			spin_pause();
+	}
+}
+
+static inline void spin_unlock(spinlock_t *l)
+{
+	__asm__ volatile ("" ::: "memory");
+	l->locked = 0;
+}
+
+/* Caller saves/restores the interrupt state around the critical section. */
+static inline void spin_lock_irqsave(spinlock_t *l, unsigned int *flags)
+{
+	*flags = spin_irq_save_disable();
+	spin_lock(l);
+}
+
+static inline void spin_unlock_irqrestore(spinlock_t *l, unsigned int flags)
+{
+	spin_unlock(l);
+	spin_irq_restore(flags);
+}
+
+#endif

+ 69 - 0
switch.S

@@ -20,7 +20,10 @@
 .extern ticks
 .extern ticks
 .extern schedule
 .extern schedule
 .extern sched_tick_bsp
 .extern sched_tick_bsp
+.extern sched_tick_ap
+.extern sched_finish_switch
 .extern serial_poll
 .extern serial_poll
+.extern lapic_eoi
 
 
 .include "irqret.inc"
 .include "irqret.inc"
 
 
@@ -63,6 +66,10 @@ pit_irq_stub:
 	movl  (%eax), %esp
 	movl  (%eax), %esp
 
 
 .no_switch:
 .no_switch:
+	/* We are on the resumed process's kernel stack now; safe to release a
+	 * zombie this CPU switched away from earlier. */
+	call  sched_finish_switch
+
 	/* Both paths must leave a usable data selector for the context being
 	/* Both paths must leave a usable data selector for the context being
 	 * resumed; ring-3 needs 0x23, ring-0 uses 0x10 + GS 0x28. */
 	 * resumed; ring-3 needs 0x23, ring-0 uses 0x10 + GS 0x28. */
 	IRQ_RETURN_DS 36
 	IRQ_RETURN_DS 36
@@ -70,6 +77,67 @@ pit_irq_stub:
 	popa
 	popa
 	iret
 	iret
 
 
+/*
+ * AP local-APIC timer: EOI the LAPIC (no PIC EOI), account the tick against
+ * the current process and switch when its slice expired.  Unlike the PIT
+ * path this never touches the global tick counter; the BSP PIT is the system
+ * clock.
+ */
+.globl lapic_timer_stub
+lapic_timer_stub:
+	pusha
+	movw  $0x10, %ax
+	movw  %ax, %ds
+	movw  %ax, %es
+	movw  %ax, %fs
+	movw  $0x28, %ax
+	movw  %ax, %gs
+
+	call  lapic_eoi
+	call  sched_tick_ap
+	testl %eax, %eax
+	jz    .lapic_no_switch
+
+	movl  %gs:0, %eax
+	movl  %esp, (%eax)
+	call  schedule
+	movl  %gs:0, %eax
+	movl  (%eax), %esp
+
+.lapic_no_switch:
+	call  sched_finish_switch
+	IRQ_RETURN_DS 36
+	popa
+	iret
+
+/*
+ * Reschedule IPI: request a reschedule on this CPU.  The next local timer
+ * tick picks it up.  A fixed-delivery IPI still has to be EOIed on the target
+ * LAPIC, otherwise it stays in service and blocks the timer vector.
+ */
+.globl lapic_resched_stub
+lapic_resched_stub:
+	pusha
+	movw  $0x10, %ax
+	movw  %ax, %ds
+	movw  %ax, %es
+	movw  %ax, %fs
+	movw  $0x28, %ax
+	movw  %ax, %gs
+
+	movl  %gs:0, %eax
+	movl  $1, 24(%eax)          /* struct cpu.need_resched */
+
+	call  lapic_eoi
+	IRQ_RETURN_DS 36
+	popa
+	iret
+
+/* Spurious local-APIC interrupt: no EOI, no accounting. */
+.globl lapic_spurious_stub
+lapic_spurious_stub:
+	iret
+
 /*
 /*
  * Voluntary reschedule gate: int 0x81.  Entered from ring 0 only, with the
  * Voluntary reschedule gate: int 0x81.  Entered from ring 0 only, with the
  * CPU having pushed EFLAGS/CS/EIP exactly like the PIT path.  No EOI and no
  * CPU having pushed EFLAGS/CS/EIP exactly like the PIT path.  No EOI and no
@@ -90,6 +158,7 @@ resched_stub:
 	call  schedule
 	call  schedule
 	movl  %gs:0, %eax
 	movl  %gs:0, %eax
 	movl  (%eax), %esp
 	movl  (%eax), %esp
+	call  sched_finish_switch
 
 
 	IRQ_RETURN_DS 36
 	IRQ_RETURN_DS 36
 	popa
 	popa

+ 5 - 0
sys_net.c

@@ -30,12 +30,17 @@ static struct tcp_conn tcp;
  * safe. */
  * safe. */
 static void net_enter(void)
 static void net_enter(void)
 {
 {
+	/* ARP/DNS/DHCP/TCP waits spin on the PIT and can take seconds.  Give
+	 * the other CPU's syscalls (and the WM) the kernel while we wait; the
+	 * NIC ownership still serializes every stack user. */
+	bkl_pause();
 	__asm__ volatile ("sti");
 	__asm__ volatile ("sti");
 }
 }
 
 
 static void net_exit(void)
 static void net_exit(void)
 {
 {
 	__asm__ volatile ("cli");
 	__asm__ volatile ("cli");
+	bkl_resume();
 }
 }
 
 
 static unsigned int caller_pid(void)
 static unsigned int caller_pid(void)

+ 21 - 1
syscall.c

@@ -30,7 +30,7 @@ static int copy_user_string(char *dst, unsigned int max, const char *src)
 	return 0;
 	return 0;
 }
 }
 
 
-int syscall_dispatch(int num, int a1, int a2, int a3)
+static int syscall_dispatch_locked(int num, int a1, int a2, int a3)
 {
 {
 	switch (num) {
 	switch (num) {
 	case SYS_PUTSTR:
 	case SYS_PUTSTR:
@@ -338,3 +338,23 @@ int syscall_dispatch(int num, int a1, int a2, int a3)
 		return -1;
 		return -1;
 	}
 	}
 }
 }
+
+/* All ring-3 syscalls run under the big kernel lock.  Blocking syscalls drop
+ * it through sched_yield(); the lock is taken back before the dispatcher
+ * resumes, so the rest of the kernel sees one syscall at a time. */
+int syscall_dispatch(int num, int a1, int a2, int a3)
+{
+	struct process *p = current_proc();
+	int r;
+
+	/* A process killed while it was in ring 3 may reach the gate before its
+	 * CPU notices the DEAD state.  Do not let it run one more syscall on
+	 * resources that may already be reused. */
+	if (p && p->state == PROC_DEAD)
+		process_exit();
+
+	bkl_lock();
+	r = syscall_dispatch_locked(num, a1, a2, a3);
+	bkl_unlock();
+	return r;
+}

+ 2 - 1
ucmd_top.c

@@ -335,7 +335,8 @@ void _start(const char *arg)
 		wall = t1 - t0;
 		wall = t1 - t0;
 		if (!wall)
 		if (!wall)
 			wall = 1;
 			wall = 1;
-		idle_pm = (s1.idle_ticks - s0.idle_ticks) * 1000u / wall;
+		idle_pm = (s1.idle_ticks - s0.idle_ticks) * 1000u /
+			(wall * (s0.ncpus ? s0.ncpus : 1));
 		if (idle_pm > 1000u)
 		if (idle_pm > 1000u)
 			idle_pm = 1000u;
 			idle_pm = 1000u;
 		uptime_s = t1 / 1000;
 		uptime_s = t1 / 1000;

+ 5 - 0
wm.c

@@ -1683,6 +1683,11 @@ void wm_run(void)
 		wm_handle_mouse();
 		wm_handle_mouse();
 		wm_redraw_if_needed();
 		wm_redraw_if_needed();
 		vga_fb_flush_budget(VGA_FLUSH_BUDGET_BYTES);
 		vga_fb_flush_budget(VGA_FLUSH_BUDGET_BYTES);
+		/* This is the kernel-WM fallback, i.e. a kernel thread holding
+		 * the BKL.  Do not sleep on it: release it around the idle wait
+		 * so ring-3 syscalls on the APs can still run. */
+		bkl_pause();
 		cpu_idle();
 		cpu_idle();
+		bkl_resume();
 	}
 	}
 }
 }