Просмотр исходного кода

mm: add a highmem zone and a kmap window for RAM up to 4 GB

Lowmem stays the 1 GB identity map (MM_MAX_PHYS) and serves every
allocation the kernel dereferences by physical address.  RAM from 1 GB
up to MM_HIGH_LIMIT (4 GB) is now tracked by the same bitmap as a
highmem zone that is never identity-mapped; mm_alloc_user_page() hands
it out (lowmem fallback) for ring-3 program, stack and heap frames in
elf.c, process.c and sys_mem.c.  Each zone keeps its own first-free
hint; mm_alloc_pages() stays lowmem-only.

The kernel reaches such a frame through kmap()/kunmap(): the 4 MB slot
at MM_KMAP_VA (0x4FC00000, in the gap below USER_WIN_BASE) has one
supervisor page table shared by every process directory, with 4 LIFO
slots per CPU.  A highmem kmap disables preemption until kunmap and
flushes only the local TLB entry.  The PDE mappers refuse to overwrite
the slot.  It is not the last lowmem slot (firmware puts the ACPI
tables at the top of RAM) nor the top PDE (a VBE LFB may sit at
0xFF000000).

The user-stack argument is now written through one kmap'd page, so it
must fit the top stack page (spawn args are at most VFS_PATH_MAX).
elf_load() frees its page when the mapping fails.

The UEFI loader synthesizes the E820 map for all RAM below 4 GB; its
rootfs buffer stays in lowmem.

Verified in QEMU with a temporary test that took every frame through
kmap (nested slots included) at 512M/2G/3G/4G with 4 CPUs (0 bad, no
leak), WM/term stacks landing in highmem at 2G, the keyboard QMP test
on a 3G/4-CPU guest (3 runs incl. xHCI), test-usb, and
test-uefi/test-amd with QEMU_MEM=3G (2040 MB highmem under OVMF).
Johannes Zellner 2 дней назад
Родитель
Сommit
bad87a48e8
15 измененных файлов с 343 добавлено и 114 удалено
  1. 53 24
      AGENTS.md
  2. 16 12
      boot/efi/main.c
  3. 1 1
      docs/boot.html
  4. 4 3
      docs/development.html
  5. 1 1
      docs/index.html
  6. 33 23
      docs/memory.html
  7. 2 1
      docs/userland.html
  8. 21 6
      include/phys.h
  9. 12 5
      kernel/elf.c
  10. 80 24
      kernel/mm.c
  11. 12 3
      kernel/mm.h
  12. 84 2
      kernel/paging.c
  13. 8 0
      kernel/paging.h
  14. 12 7
      kernel/process.c
  15. 4 2
      kernel/sys_mem.c

+ 53 - 24
AGENTS.md

@@ -141,8 +141,9 @@ make clean
   {wait}`, and `{click:X,Y}` for a left click at canvas coordinates).  It needs the multilib `lib32-sdl3`,
   `lib32-wayland` and `lib32-libxkbcommon`; host sockets replace the NebOS TCP
   stack and the kernel/WM paths are not exercised.
-- QEMU guests default to 1 GB (`QEMU_MEM`, e.g. `make run QEMU_MEM=512M`),
-  matching `MM_MAX_PHYS` in `phys.h`; all run and UEFI targets pass `-m`.
+- QEMU guests default to 1 GB (`QEMU_MEM`), the lowmem size (`MM_MAX_PHYS`
+  in `phys.h`); `make run QEMU_MEM=3G` exercises highmem.  All run and UEFI
+  targets pass `-m`.
 - The command pack is the system rootfs: `cmds.pack` (NBPAK1), gzip-compressed
   to `cmds_pack_gz.bin`, wrapped by `mkrootfs` with a one-sector `NBROOT1`
   header into `cmds.img` (the 8 MB USB partition image) and `BOOTX64.EFI`.
@@ -249,7 +250,8 @@ and `make net-dev`/`kernel-net-dev` reload over the network.  `bootlog save
 4. `mbr.S` — MBR (USB/HDD boot); loads the active partition's boot sector
    (partition 1 at LBA 63).
 5. **UEFI (`boot/efi/`)** — an x86-64 PE32+ `BOOTX64.EFI` loads from the ESP. It
-   picks a GOP mode, converts `GetMemoryMap` into the E820 map at `0x1000`,
+   picks a GOP mode, converts `GetMemoryMap` into the E820 map at `0x1000`
+   (all RAM below 4 GB, at most 32 runs),
    stages `kernel.bin` below 4 GB, allocates a loader-data buffer below the
    1 GB identity window (above the low 1 MB) for the embedded gz rootfs
    and copies it there, fills
@@ -353,19 +355,36 @@ it to `0x7E00`.  The kernel region itself only has one wall (`_end` below
   trampoline at `0x6000`), and `spinlock.h` (the IRQ-safe spinlocks).
   `cpu_enable_sse()` runs on every AP: an AP starts with firmware CR4, and the
   SSE2 drawing fast paths #UD until OSFXSR/OSXMMEXCPT are set.
-- **Memory**: `phys.h` — the shared 1 GB physical ceiling (`MM_MAX_PHYS`),
-  also used by the UEFI loader; it ends where ring-3 space starts
-  (`USER_BASE` = `0x40000000`), and `paging.c` statically asserts that the
-  identity map never reaches a user window.  RAM above 1 GB is ignored (no
-  highmem zone or PAE yet).  `mm.c/h` — INT 15h E820 detection (real-mode in
-  `vesa_stage.S`, map at `0x1000`) + a bitmap page-frame allocator (4 KB
-  pages). Reserves low 1 MB and an 8 MB VGA-shadow region.  The bitmap is
-  sized to the highest E820 RAM address (32 KB at 1 GB) and is not in BSS:
-  `mm_init` places it in the first RAM run at or above `0x900000` that
-  avoids the boot pack and the LFB and marks its own pages used (a 512-byte
-  static array covers the 16 MB no-E820 fallback).  Allocation is lowest
-  address first: a `first_free` word hint skips the full low words,
-  `mm_alloc_page()` finds the bit with `bsf`, and `mm_alloc_pages()` steps
+- **Memory**: `phys.h` — `MM_MAX_PHYS` (1 GB, the lowmem/identity-map
+  ceiling, also used by the UEFI loader for the pack buffer), `MM_KMAP_VA`
+  (`0x4FC00000`) and `MM_HIGH_LIMIT` (4 GB).  Lowmem ends where ring-3 space
+  starts (`USER_BASE` = `0x40000000`), and `paging.c` statically asserts that
+  the identity map never reaches a user window.  `mm.c/h` — INT 15h E820
+  detection (real-mode in `vesa_stage.S`, map at `0x1000`) + a bitmap
+  page-frame allocator (4 KB pages) over all RAM below 4 GB, in two zones:
+  **lowmem** `[0, 1 GB)` is identity-mapped and serves every kernel
+  allocation (`mm_alloc_page`, `mm_alloc_pages`, `kmalloc`: page tables,
+  DMA buffers, canvases, kernel stacks); **highmem** `[1 GB, 4 GB)` is never
+  identity-mapped and is handed out only by `mm_alloc_user_page()` (highmem
+  first, lowmem fallback) for ring-3 program, stack and heap frames
+  (`elf.c`, `process.c`, `sys_mem.c`).  The kernel touches such a frame only
+  through the owning process's mapping or `kmap(phys)`/`kunmap()`
+  (`paging.c`): the 4 MB slot at `MM_KMAP_VA` (in the gap below
+  `USER_WIN_BASE`) has one supervisor page table shared by every process
+  directory, with 4 nesting (LIFO) slots per CPU; a highmem `kmap` disables
+  preemption until `kunmap` and flushes only its local TLB entry, lowmem
+  returns the identity address.  Nothing may sleep between them.
+  `paging_map_ram`, `paging_map_mmio_cr3` and the LFB mapping refuse to
+  overwrite the kmap slot.  (It is not the last lowmem slot, where firmware
+  puts the ACPI tables, nor the top PDE, where a VBE LFB may sit.)  The
+  kernel reserves low 1 MB and an 8 MB VGA-shadow region.  The bitmap is
+  sized to the highest E820 RAM address (32 KB at 1 GB, 96 KB at 3 GB) and
+  is not in BSS: `mm_init` places it in the first lowmem RAM run at or above
+  `0x900000` that avoids the boot pack and the LFB and marks its own pages
+  used (a 512-byte static array covers the 16 MB no-E820 fallback); COM1
+  logs `mm: bitmap ... highmem free N MB`.  Allocation is lowest address
+  first within a zone: a per-zone first-free word hint skips the full low
+  words, single pages are found with `bsf`, and `mm_alloc_pages()` steps
   over full and empty words 32 pages at a time. `kmalloc.c/h` — a power-of-two heap
   allocator on top of the frame allocator (used for PCBs and command context).
   Process stacks (≥ `0x900000`) come from the frame allocator, with an 8-stack
@@ -1341,7 +1360,10 @@ it to `0x7E00`.  The kernel region itself only has one wall (`_end` below
   WM top bar and the `powerinfo`/`sensors` commands work on both.
 - **Ring-3 address space**: the ELF image and 128 KB (32-page) stack live in
   the 4 MB `USER_BASE` (`0x40000000`)..`USER_STACK_TOP` (`0x40400000`)
-  window, which is too small for a full-frame image.  `SYS_MEM_ALLOC`/`SYS_MEM_GROW`/`SYS_MEM_FREE`
+  window, which is too small for a full-frame image.  Their frames, and the
+  heap's, come from highmem when the machine has RAM above 1 GB.  The 4 MB
+  slot at `0x4FC00000`, just below `USER_WIN_BASE`, is the kernel's kmap
+  window (supervisor-only).  `SYS_MEM_ALLOC`/`SYS_MEM_GROW`/`SYS_MEM_FREE`
   (`sys_mem.c/h`) map/grow/delete a single anonymous arena at
   `USER_HEAP_BASE` (`0x60000000`), capped at 64 MB (`USER_HEAP_MAX`; the
   browser's DOM arenas plus the QuickJS heap need ~30 MB before images); the initial allocation is
@@ -2517,7 +2539,9 @@ and the WM falls back to polling (`io_keyboard_irq_masked()`).
   the in-tree `libc.a` when they need allocation or standard string/stdlib
   functions; the `SYS_MEM_*` syscalls are internal to the libc backend.
 - **Paging is on but flat** — identity-mapped 4 MB pages (`paging.c`), so
-  physical == virtual everywhere. Still no per-process address spaces or process
+  physical == virtual for lowmem (below 1 GB).  A frame from
+  `mm_alloc_user_page()` may be highmem: never dereference it by its
+  physical address, use `kmap()`/`kunmap()`. Still no per-process address spaces or process
   isolation: everything runs ring-0 and shares the address space.
 - **Fresh memory is not zeroed** — `mm_alloc_page`/`kmalloc` hand out whatever
   RAM contained; QEMU starts guest RAM cleared, real hardware does not, so a
@@ -2563,11 +2587,16 @@ and the WM falls back to polling (`io_keyboard_irq_masked()`).
 3. No per-process address spaces or process isolation (paging is identity-mapped).
 4. FAT32: FSInfo free-count not updated, no directory-chain growth.
 5. `disk.c` `fdc_reset()` and `fdc_init_once()` overlap/duplicate logic.
-6. Physical RAM above 1 GB is unused: the identity map stops at `USER_BASE`
-   (`0x40000000`).  Using it needs a highmem zone (user-only frames plus a
-   per-CPU temporary kernel mapping for zeroing/ELF loads) and, for RAM above
-   4 GB, PAE.  MMIO is still mapped at its physical address, which also
-   blocks growing ring-3 space above the PCI hole.
+6. Highmem limits: RAM above 4 GB is ignored (it would need PAE), and on a
+   4 GB board the PCI hole leaves ~3-3.5 GB usable.  Highmem only backs
+   ring-3 pages, so the 1 GB lowmem zone (kernel structures, canvases,
+   contiguous buffers) can still run out first.  One process is still capped
+   by `USER_HEAP_MAX` (64 MB) and the 4 MB image/stack window; raising that
+   needs demand paging and moving MMIO out of the identity layout, which
+   currently maps BARs at their physical address and so blocks ring-3 space
+   near the PCI hole.  A BIOS that puts ACPI tables, MMIO or the LFB in the
+   4 MB kmap slot at `0x4FC00000` cannot have that range identity-mapped
+   (the mappers log `refusing to map over the kmap window`).
 7. `display` (needs a GPU snapshot) and `lsusb` (a 45-byte wrapper around
    the USB tree printer) are the last non-privileged commands still living in
    the kernel, plus the privileged ones (`peek`, `poke`, `dbg`, `serial`,
@@ -2650,7 +2679,7 @@ and the WM falls back to polling (`io_keyboard_irq_masked()`).
     `websrv` cannot start while any TCP socket is open.  Kill the other
     command first.
 20. `/share` is RAM only: a reboot/kexec starts empty, and the 16 MB cap is
-    charged against the 1 GB identity-mapped window.
+    charged against the 1 GB lowmem zone.
 
 21. The TCP backlog's early-request buffers cost 8 KB of kernel BSS
     (8 slots x 1 KB), and the FAT listing cache another 5 KB.  The command

+ 16 - 12
boot/efi/main.c

@@ -19,7 +19,11 @@
 
 #define MAX_PHYS    ((UINT32)MM_MAX_PHYS)
 #define PAGE_SIZE_  0x1000u
-#define MAX_PAGES   (MAX_PHYS / PAGE_SIZE_)
+/* The synthesized E820 map covers RAM up to 4 GB: the kernel identity-maps
+ * the first MAX_PHYS (lowmem) and uses the rest as highmem for ring-3
+ * pages.  The rootfs buffer must still be lowmem. */
+#define MAP_LIMIT   ((UINT64)MM_HIGH_LIMIT)
+#define MAP_PAGES   ((UINT32)(MAP_LIMIT / PAGE_SIZE_))
 #define E820_ADDR   0x1000u
 
 /* The kernel image is linked to live at 0x7E00.  Its file image must stay
@@ -44,11 +48,11 @@ extern unsigned char tramp_blob_end[];
  * generous and keeps the retry loop simple. */
 static UINT8 memmap_buf[65536] __attribute__((aligned(8)));
 
-/* One bit per 4 KB page over the kernel's 1 GB identity window.  Starts
+/* One bit per 4 KB page below 4 GB (128 KB).  Starts
  * all-used; only EfiConventional/BootServices memory is freed.  EfiLoader*
  * (this image and the staging buffer) stays reserved, which is safe and
  * costs only a few hundred KB. */
-static UINT8 page_used[MAX_PAGES / 8];
+static UINT8 page_used[MAP_PAGES / 8];
 
 struct fb_saved {
 	int      valid;
@@ -343,11 +347,11 @@ static void free_range(UINT64 base, UINT64 len)
 	UINT64 end;
 	UINT32 p0, p1, p;
 
-	if (base >= MAX_PHYS)
+	if (base >= MAP_LIMIT)
 		return;
 	end = base + len;
-	if (end > MAX_PHYS)
-		end = MAX_PHYS;
+	if (end > MAP_LIMIT)
+		end = MAP_LIMIT;
 	p0 = (UINT32)((base + PAGE_SIZE_ - 1) / PAGE_SIZE_);
 	p1 = (UINT32)(end / PAGE_SIZE_);
 	for (p = p0; p < p1; p++)
@@ -360,11 +364,11 @@ static void use_range(UINT64 base, UINT64 len)
 	UINT64 end;
 	UINT32 p0, p1, p;
 
-	if (base >= MAX_PHYS)
+	if (base >= MAP_LIMIT)
 		return;
 	end = base + len;
-	if (end > MAX_PHYS)
-		end = MAX_PHYS;
+	if (end > MAP_LIMIT)
+		end = MAP_LIMIT;
 	p0 = (UINT32)(base / PAGE_SIZE_);
 	p1 = (UINT32)((end + PAGE_SIZE_ - 1) / PAGE_SIZE_);
 	for (p = p0; p < p1; p++)
@@ -399,11 +403,11 @@ static void build_e820(EFI_MEMORY_DESCRIPTOR *map, UINTN map_size,
 	}
 
 	/* The scanout is MMIO, not RAM; make sure it can never be handed out
-	 * even when its BAR happens to sit below the 1 GB ceiling. */
+	 * even when its BAR happens to sit below 4 GB next to RAM. */
 	if (fb->valid)
 		use_range(fb->base, fb->size);
 
-	for (p = 0; p < MAX_PAGES; ) {
+	for (p = 0; p < MAP_PAGES; ) {
 		UINT32 start;
 
 		if (page_is_used(p)) {
@@ -411,7 +415,7 @@ static void build_e820(EFI_MEMORY_DESCRIPTOR *map, UINTN map_size,
 			continue;
 		}
 		start = p;
-		while (p < MAX_PAGES && !page_is_used(p))
+		while (p < MAP_PAGES && !page_is_used(p))
 			p++;
 		if (count < 32) {
 			UINT64 len = (UINT64)(p - start) * PAGE_SIZE_;

+ 1 - 1
docs/boot.html

@@ -206,7 +206,7 @@
 		<ol>
 			<li>Selects a GOP mode with BGRX or RGBX 32bpp, width at most 1920, <code>pitch × height</code> at most 8 MB (1920×1080 fits at 8.29 MB), and a framebuffer below 4 GB. With 32bpp, that size limit also keeps a full-screen RGB565 canvas inside the 4 MB window slot; the VBE stage additionally checks 2 M pixels because a 24bpp 1920×1200 mode would otherwise pass the shadow check.</li>
 			<li>Calls <code>GetMemoryMap</code> and converts the firmware map into the kernel's E820 format at <code>0x1000</code>.</li>
-			<li>Locates the parent whole-disk Block I/O handle from the ESP device path, reads the P1 boot sector and P2 <code>NBROOT1</code> header, and stages the kernel below 4 GB and the gzip pack below the 1 GB identity window. The kernel/pack embedded in <code>BOOTX64.EFI</code> are a fallback only.</li>
+			<li>Locates the parent whole-disk Block I/O handle from the ESP device path, reads the P1 boot sector and P2 <code>NBROOT1</code> header, and stages the kernel below 4 GB and the gzip pack below the 1 GB identity window, and synthesizes the E820 map for all RAM below 4 GB. The kernel/pack embedded in <code>BOOTX64.EFI</code> are a fallback only.</li>
 			<li>Writes the boot-info and framebuffer blocks, calls <code>ExitBootServices</code>, then copies a 192-byte trampoline to <code>0x800</code>.</li>
 			<li>Leaves IA-32e mode (clears <code>CR0.PG</code>, <code>EFER.LME</code>, <code>CR4.PAE/PGE/PCIDE/SMEP/SMAP/LA57</code>), copies the kernel to <code>0x7E00</code>, and jumps to <code>kernel_entry</code>.</li>
 		</ol>

+ 4 - 3
docs/development.html

@@ -245,9 +245,10 @@
 			</table>
 		</div>
 		<p>
-			QEMU guests default to 1 GB, matching <code>MM_MAX_PHYS</code> in
-			<span class="file">phys.h</span>. Override the size with
-			<code>QEMU_MEM</code>, for example <code>make run QEMU_MEM=512M</code>;
+			QEMU guests default to 1 GB, the lowmem size (<code>MM_MAX_PHYS</code> in
+			<span class="file">phys.h</span>). Override the size with
+			<code>QEMU_MEM</code>, for example <code>make run QEMU_MEM=3G</code> to
+			exercise highmem;
 			all run and UEFI targets pass it through.
 		</p>
 	</section>

+ 1 - 1
docs/index.html

@@ -157,7 +157,7 @@
 			<li><strong>Freestanding C.</strong> The kernel has no libc, <code>malloc</code> or <code>printf</code>; ring-3 commands can link the in-tree <code>libc.a</code> (<code>malloc</code>, strings, stdlib). Formatting and allocation are in-tree either way.</li>
 			<li><strong>i586 baseline.</strong> Builds use <code>-march=i586 -mno-sse -mno-sse2 -mno-mmx</code>; SSE2 drawing paths are probed at runtime.</li>
 			<li><strong>Identity-mapped paging.</strong> Physical and virtual addresses are equal. There is no per-process address space or isolation.</li>
-			<li><strong>1 GB physical ceiling.</strong> The identity map ends where ring-3 space starts (<code>0x40000000</code>); RAM above it is ignored.</li>
+			<li><strong>1 GB lowmem, 4 GB highmem.</strong> The identity map ends where ring-3 space starts (<code>0x40000000</code>); RAM up to 4 GB backs ring-3 pages through per-CPU <code>kmap</code> slots.</li>
 			<li><strong>Hardcoded low memory.</strong> Boot info, the E820 map, the framebuffer block and the AP trampoline have fixed addresses that every boot path must honour.</li>
 			<li><strong>No firmware runtime.</strong> <code>reboot</code> and <code>poweroff</code> use legacy ports; UEFI runtime services are not called after <code>ExitBootServices</code>.</li>
 		</ul>

+ 33 - 23
docs/memory.html

@@ -4,7 +4,7 @@
 <meta charset="utf-8">
 <meta name="viewport" content="width=device-width, initial-scale=1">
 <title>Memory | NebOS Architecture</title>
-<meta name="description" content="NebOS memory architecture: fixed low-memory addresses, E820 parsing, bitmap frame allocator, identity paging, MTRR write-combining, user arenas and boot-pack lifetime.">
+<meta name="description" content="NebOS memory architecture: fixed low-memory addresses, E820 parsing, bitmap frame allocator with lowmem/highmem zones, identity paging, kmap, MTRR write-combining, user arenas and boot-pack lifetime.">
 <link rel="icon" href="favicon.svg" type="image/svg+xml">
 <link rel="stylesheet" href="style.css">
 </head>
@@ -35,13 +35,15 @@
 		<p class="kicker">Memory &amp; paging</p>
 		<h1>A fixed ladder and a 4 KB frame allocator.</h1>
 		<p class="lede">
-			NebOS identity-maps physical RAM with 4 MB pages and hands out
-			4 KB frames from a bitmap. A small set of low-memory addresses is
+			NebOS identity-maps the first 1 GB of RAM (lowmem) with 4 MB pages
+			and hands out 4 KB frames from a bitmap; RAM from 1 GB up to 4 GB
+			(highmem) backs ring-3 pages and is reached by the kernel through
+			per-CPU <code>kmap</code> slots. A small set of low-memory addresses is
 			fixed by the boot path and the ABI; everything else — process stacks,
 			window canvases, the ring-3 heap and the rootfs copy — is dynamic.
 		</p>
 		<div class="meta">
-			<span class="pill">1 GB physical ceiling</span>
+			<span class="pill">1 GB lowmem · 4 GB highmem</span>
 			<span class="pill">4 KB frames</span>
 			<span class="pill">4 MB PSE pages</span>
 			<span class="pill">identity map</span>
@@ -96,12 +98,13 @@
 				<text class="svg-tiny" x="40" y="150">The 8 MB reserve protects the shadow buffer; the allocator starts at 0x900000.</text>
 			</svg>
 			<figcaption>
-				<strong>1 GB is a hard ceiling</strong> (<code>MM_MAX_PHYS</code>,
+				<strong>Lowmem ends at 1 GB</strong> (<code>MM_MAX_PHYS</code>,
 				<code>include/phys.h</code>). The identity map covers
 				<code>[0, 0x40000000)</code> and ring-3 space starts at
-				<code>USER_BASE</code> = <code>0x40000000</code>, so RAM above 1 GB is
-				ignored until a highmem zone exists; <code>paging.c</code>
-				statically asserts the two never overlap. The VGA shadow reserve is conservative: the
+				<code>USER_BASE</code> = <code>0x40000000</code>; <code>paging.c</code>
+				statically asserts the two never overlap. RAM above 1 GB (up to
+				<code>MM_HIGH_LIMIT</code>, 4 GB) is highmem: tracked by the same
+				bitmap, never identity-mapped, used only for ring-3 pages. The VGA shadow reserve is conservative: the
 				actual buffer is smaller, but the gap keeps a future resolution bump
 				from colliding with allocated pages.
 			</figcaption>
@@ -138,6 +141,7 @@
 					<tr><td><code>0x900000</code></td><td>Boot pack (written directly by the legacy stage), then the frame bitmap right after it; dynamic stacks and kexec staging</td><td>The pack is freed after <code>ramfs_init</code> takes its copy; the bitmap stays</td></tr>
 					<tr><td><code>0x100000</code></td><td>VGA shadow buffer base</td><td><code>VGA_SHADOW_BASE</code>; 8 MB reserved</td></tr>
 					<tr><td><code>0x40000000</code></td><td>Ring-3 ELF link/load address</td><td><code>USER_BASE</code>, 4 MB window up to <code>0x40400000</code></td></tr>
+					<tr><td><code>0x4FC00000</code></td><td>kmap window (virtual)</td><td><code>MM_KMAP_VA</code>; supervisor page table shared by every directory, 4 slots per CPU</td></tr>
 					<tr><td><code>0x50000000</code></td><td>Ring-3 window canvas</td><td><code>USER_WIN_BASE</code>, 4 MB slot</td></tr>
 					<tr><td><code>0x58000000</code></td><td>WM shadow mapping</td><td><code>WM_SHADOW_VA</code>; ring-3 WM only</td></tr>
 					<tr><td><code>0x59000000</code></td><td>WM content slots</td><td><code>WM_CONTENT_VA_BASE</code> + index × 4 MB (<code>WM_CONTENT_SLOT</code>)</td></tr>
@@ -158,7 +162,8 @@
 			<code>EFER.LME</code>.
 		</p>
 		<ul>
-			<li><strong>Identity map.</strong> Physical and virtual addresses are equal. Page-directory entries are shared with every process.</li>
+			<li><strong>Identity map.</strong> Physical and virtual addresses are equal below 1 GB (lowmem). Page-directory entries are shared with every process.</li>
+			<li><strong>kmap window.</strong> The 4 MB slot at <code>MM_KMAP_VA</code> (<code>0x4FC00000</code>, in the unused gap below <code>USER_WIN_BASE</code>) has one supervisor page table that every process directory shares. <code>kmap(phys)</code> returns the identity address for lowmem, and for a highmem frame takes this CPU's next slot (4 per CPU, nesting LIFO) with preemption disabled until <code>kunmap()</code>; only the local TLB entry is flushed, because no other CPU uses that slot. <code>paging_map_ram</code>, <code>paging_map_mmio_cr3</code> and the LFB mapping refuse to overwrite that slot. It is not the last lowmem slot, because firmware puts the ACPI tables at the top of RAM, and not the top of the address space, where a VBE LFB may sit.</li>
 			<li><strong>Kernel stacks mapped everywhere.</strong> Each process gets a page directory cloned from the kernel identity map, so CR3 can switch while still running on the old kernel stack.</li>
 			<li><strong>LFB is uncacheable by default.</strong> If <code>mtrr_set_wc()</code> successfully covers the framebuffer, the LFB pages are mapped cacheable so the shadow-buffer flush can write-combine.</li>
 			<li><strong>No process isolation.</strong> There is one address space with per-process page directories only for mappings; a wild ring-0 or ring-3 pointer can reach any mapped physical page.</li>
@@ -183,26 +188,30 @@
 		<h2>Frame allocator</h2>
 		<p>
 			<span class="file">mm.c</span> keeps one bit per 4 KB page for the
-			RAM below the 1 GB ceiling: the bitmap is sized to the highest E820 RAM
-			address (32 KB at 1 GB) and is not part of the kernel image.
-			<code>mm_init</code> places it in the first E820 RAM run at or above
+			RAM below 4 GB: the bitmap is sized to the highest E820 RAM
+			address (32 KB at 1 GB, 96 KB at 3 GB) and is not part of the kernel
+			image. <code>mm_init</code> places it in the first lowmem E820 RAM run at or above
 			<code>0x900000</code> that avoids the boot rootfs pack and the LFB, and
 			marks its own pages used (legacy boots put it right after the pack;
 			a 16 MB static fallback covers a missing E820 map).  The allocators
 			disable interrupts around the bitmap and hand out the lowest free
-			frames: a <code>first_free</code> word hint skips the full low part,
-			<code>mm_alloc_page()</code> finds the bit with <code>bsf</code>, and
+			frames of a zone: each zone has a first-free word hint that skips its
+			full low part, single pages are found with <code>bsf</code>, and
 			<code>mm_alloc_pages(n)</code> steps over full and empty words 32 pages
-			at a time. The allocator returns:
+			at a time. Everything the kernel dereferences by physical address
+			(page tables, DMA buffers, canvases, kernel stacks, <code>kmalloc</code>)
+			comes from lowmem; ring-3 program, stack and heap frames prefer
+			highmem. The allocator returns:
 		</p>
 		<ul>
-			<li><code>mm_alloc_page()</code> — one zero-address-ambiguous page (physical address 0 means failure).</li>
-			<li><code>mm_alloc_pages(n)</code> — <code>n</code> contiguous pages.</li>
+			<li><code>mm_alloc_page()</code> — one lowmem page (physical address 0 means failure).</li>
+			<li><code>mm_alloc_user_page()</code> — one frame for a ring-3 mapping: highmem first, lowmem when highmem is exhausted. Touch it only through <code>kmap()</code> or the user mapping (<span class="file">elf.c</span>, the user stack in <span class="file">process.c</span>, the heap in <span class="file">sys_mem.c</span>).</li>
+			<li><code>mm_alloc_pages(n)</code> — <code>n</code> contiguous lowmem pages.</li>
 			<li><code>mm_free_page()</code> / <code>mm_free_range()</code> — release a page or a page-aligned boot reservation.</li>
 		</ul>
 		<p>
 			At boot, <code>mm_init</code> reserves the low 1 MB, the 8 MB VGA
-			shadow region, any LFB that happens to live below 1 GB, the bitmap itself and the
+			shadow region, any LFB that overlaps RAM, the bitmap itself and the
 			boot-time rootfs pack. The pack reservation is released by
 			<code>ramfs_init</code> after it copies/inflates the pack into fresh
 			pages.
@@ -279,9 +288,9 @@
 					<tr><th>Range</th><th>Size</th><th>Contents</th></tr>
 				</thead>
 				<tbody>
-					<tr><td><code>0x40000000</code>–<code>0x40400000</code></td><td>4 MB</td><td>ELF segments, user stack top at <code>0x40400000</code> (32 pages / 128 KB, grows down)</td></tr>
+					<tr><td><code>0x40000000</code>–<code>0x40400000</code></td><td>4 MB</td><td>ELF segments, user stack top at <code>0x40400000</code> (32 pages / 128 KB, grows down); highmem frames when available</td></tr>
 					<tr><td><code>0x50000000</code>–<code>0x50400000</code></td><td>4 MB</td><td>One window canvas mapping (RW, per window)</td></tr>
-					<tr><td><code>0x60000000</code>–<code>0x64000000</code></td><td>up to 64 MB</td><td>Anonymous heap; initial allocation zeroed, growth pages are not</td></tr>
+					<tr><td><code>0x60000000</code>–<code>0x64000000</code></td><td>up to 64 MB</td><td>Anonymous heap (highmem frames when available, zeroed through <code>kmap</code>); initial allocation zeroed, growth pages are not</td></tr>
 				</tbody>
 			</table>
 		</div>
@@ -323,7 +332,8 @@
 		<h2>Limits &amp; leaks</h2>
 		<ul>
 			<li><strong>No per-process address spaces.</strong> Paging is identity-mapped; there is no memory protection between processes.</li>
-			<li><strong>1 GB ceiling.</strong> Physical memory above <code>0x40000000</code> is not used: that address range is ring-3 space, so the kernel has no mapping for frames above it (a highmem zone with temporary kernel mappings, or PAE for RAM above 4 GB, would be needed). A wild kernel pointer between the identity map and the user windows is an instant <code>#PF</code>.</li>
+			<li><strong>4 GB ceiling, 64 MB per process.</strong> RAM above 4 GB is ignored (it would need PAE). Highmem raises the system total, but one process still has a 64 MB heap and a 4 MB image/stack window. MMIO is still identity-mapped at its physical address, which blocks growing ring-3 space toward the PCI hole. A wild kernel pointer between the identity map and the user windows is an instant <code>#PF</code>.</li>
+			<li><strong>Highmem is user-only.</strong> Contiguous kernel buffers and every kernel structure stay in the 1 GB lowmem zone, so a machine with 4 GB can still run out of lowmem first.</li>
 			<li><strong>Kernel heap fragmentation.</strong> <code>kmalloc</code>'s power-of-two size classes make long-running mixed-size kernel allocation wasteful; ring-3 commands use the separate first-fit libc heap described above.</li>
 			<li><strong>No TLB shootdown.</strong> Cross-CPU mapping changes rely on not-present entries not being cached.</li>
 		</ul>
@@ -333,11 +343,11 @@
 		<h2>Source map</h2>
 		<ul class="filelist">
 			<li><code>kernel/mm.c</code>, <code>kernel/mm.h</code> — E820 + bitmap allocator</li>
-			<li><code>include/phys.h</code> — shared 1 GB physical ceiling</li>
+			<li><code>include/phys.h</code> — lowmem ceiling, kmap window, highmem limit</li>
 			<li><code>kernel/kmalloc.c</code>, <code>kernel/kmalloc.h</code> — kernel heap</li>
 			<li><code>user/lib/libc/malloc.c</code>, <code>user/lib/libc/ucmd_mem.c</code> — ring-3 heap and <code>SYS_MEM_*</code> backend</li>
 			<li><code>user/lib/libc/string.c</code>, <code>user/lib/libc/stdlib.c</code> — ring-3 libc support</li>
-			<li><code>kernel/paging.c</code>, <code>kernel/paging.h</code> — identity map</li>
+			<li><code>kernel/paging.c</code>, <code>kernel/paging.h</code> — identity map, kmap window</li>
 			<li><code>kernel/mtrr.c</code> — write-combining MTRRs</li>
 			<li><code>kernel/process.c</code>, <code>kernel/process.h</code> — stacks and user layout</li>
 			<li><code>kernel/ramfs.c</code> — pack copy / inflate</li>

+ 2 - 1
docs/userland.html

@@ -70,7 +70,8 @@
 			A user process gets a page directory cloned from the kernel identity
 			map, a 4 MB image/stack window at <code>0x40000000</code>, an optional
 			4 MB canvas at <code>0x50000000</code> and a growable heap at
-			<code>0x60000000</code>. The kernel remains mapped because paging is
+			<code>0x60000000</code>. Program, stack and heap frames come from
+			highmem (above 1 GB) when the machine has it. The kernel remains mapped because paging is
 			identity-mapped; this is isolation in privilege level only, not a
 			separate address space.
 		</p>

+ 21 - 6
include/phys.h

@@ -1,11 +1,26 @@
-/* Physical address ceiling shared by the 32-bit kernel and the UEFI loader. */
+/* Physical address limits shared by the 32-bit kernel and the UEFI loader. */
 #ifndef PHYS_H
 #define PHYS_H
 
-/* 1 GB: the kernel identity-maps [0, MM_MAX_PHYS) with 4 MB pages, and ring-3
- * address space starts at USER_BASE (0x40000000), so the identity map cannot
- * grow past 1 GB without a highmem zone.  The frame bitmap (32 KB at 1 GB)
- * is placed in RAM by mm_init(), not in the kernel's BSS. */
-#define MM_MAX_PHYS      (1024u * 1024 * 1024)
+/* Lowmem: the kernel identity-maps [0, MM_MAX_PHYS) with 4 MB pages and
+ * dereferences physical addresses there directly.  Ring-3 address space
+ * starts at USER_BASE (0x40000000), so lowmem ends at 1 GB.  Everything the
+ * kernel touches through a physical pointer (page tables, DMA buffers,
+ * canvases, kernel stacks, kmalloc) must be lowmem. */
+#define MM_MAX_PHYS      0x40000000u
+
+/* The kmap window: one 4 MB page-directory slot in the unused gap between
+ * the ring-3 program/stack window and USER_WIN_BASE (0x50000000).  Its
+ * supervisor page table holds per-CPU temporary mappings of arbitrary frames
+ * and is shared by every process directory.  Not the last lowmem slot
+ * (firmware puts the ACPI tables at the top of RAM) and not the top of the
+ * address space (a VBE LFB may sit at 0xFF000000). */
+#define MM_KMAP_VA       0x4FC00000u
+
+/* Highmem: RAM in [MM_MAX_PHYS, MM_HIGH_LIMIT) is tracked by the frame
+ * allocator but never identity-mapped; it only backs ring-3 pages, which
+ * the kernel reaches through the owning process's mapping or kmap().
+ * RAM above 4 GB would need PAE. */
+#define MM_HIGH_LIMIT    0x100000000ull
 
 #endif

+ 12 - 5
kernel/elf.c

@@ -84,15 +84,21 @@ unsigned int elf_load(unsigned int cr3, const void *image, unsigned int size)
 			unsigned int pva = pbase + p * 0x1000;
 			unsigned int phys = paging_page_phys(cr3, pva);
 			unsigned int start, end, dst, src, n;
+			unsigned char *page;
 
 			if (!phys) {
-				phys = mm_alloc_page();
+				phys = mm_alloc_user_page();
 				if (!phys)
 					return 0;
-				memzero((void *)phys, 0x1000);
 				if (!paging_map_page(cr3, pva, phys,
-						     PTE_USER | PTE_WRITABLE | PTE_PRESENT))
+						     PTE_USER | PTE_WRITABLE | PTE_PRESENT)) {
+					mm_free_page(phys);
 					return 0;
+				}
+				page = kmap(phys);
+				memzero(page, 0x1000);
+			} else {
+				page = kmap(phys);
 			}
 
 			/* Copy the file-backed part that falls in this page. */
@@ -104,7 +110,7 @@ unsigned int elf_load(unsigned int cr3, const void *image, unsigned int size)
 				dst = start - pva;
 				src = off + (start - va);
 				n = end - start;
-				memcopy((void *)(phys + dst),
+				memcopy(page + dst,
 					(const char *)image + src, n);
 			}
 
@@ -116,8 +122,9 @@ unsigned int elf_load(unsigned int cr3, const void *image, unsigned int size)
 			if (end > pva + 0x1000)
 				end = pva + 0x1000;
 			if (end > start)
-				memzero((void *)(phys + (start - pva)),
+				memzero(page + (start - pva),
 					end - start);
+			kunmap(page);
 		}
 	}
 

+ 80 - 24
kernel/mm.c

@@ -7,12 +7,18 @@
 
 /* One bit per 4 KB page; bit set = in use.
  *
- * The bitmap covers [0, top of RAM) rounded up to 128 KB, at most
- * MM_MAX_PHYS.  It is not in BSS (1 GB needs 32 KB, and the kernel image
- * must end below 0x90000): mm_init() places it in the first E820 RAM run
- * above the VGA-shadow reserve that avoids the boot rootfs pack and the
- * LFB, then marks those pages used.  The small static array is only the
- * fallback when no E820 map exists (the 16 MB guess) or nothing fits. */
+ * The bitmap covers [0, top of RAM) rounded up to 128 KB, at most 4 GB
+ * (MM_HIGH_LIMIT).  It is not in BSS (4 GB needs 128 KB, and the kernel
+ * image must end below 0x90000): mm_init() places it in the first lowmem
+ * E820 RAM run above the VGA-shadow reserve that avoids the boot rootfs
+ * pack and the LFB, then marks those pages used.  The small static array is
+ * only the fallback when no E820 map exists (the 16 MB guess) or nothing
+ * fits.
+ *
+ * Two zones share the bitmap: lowmem [0, MM_MAX_PHYS) is identity-mapped
+ * and serves every kernel allocation; highmem above it is handed out only
+ * by mm_alloc_user_page() for ring-3 pages, which the kernel reaches
+ * through kmap() (paging.c).  Each zone keeps its own first-free hint. */
 #define MM_FALLBACK_PHYS   0x1000000u      /* 16 MB */
 #define MM_FALLBACK_WORDS  (MM_FALLBACK_PHYS / PAGE_SIZE / 32)
 #define MM_PLACE_MIN       (MM_SHADOW_BASE + MM_SHADOW_RESERVE)
@@ -22,7 +28,9 @@ static unsigned int *bitmap = fallback_bitmap;
 static unsigned int bitmap_words;   /* words covered by the bitmap */
 static unsigned int bitmap_bits;    /* bitmap_words * 32 */
 static unsigned int bitmap_bytes;   /* page-rounded size when placed in RAM */
-static unsigned int first_free;     /* lowest word that may hold a free bit */
+static unsigned int low_words;      /* bitmap words below MM_MAX_PHYS */
+static unsigned int first_free;     /* lowest lowmem word with a free bit */
+static unsigned int first_free_high; /* same for highmem (>= low_words) */
 static unsigned int free_pages;
 static unsigned int total_ram_kb;
 
@@ -124,7 +132,7 @@ static void find_boot_regions(void)
 	sz = (unsigned int)fb->pitch * fb->height;
 	if (!sz)
 		sz = 0x1000000;
-	if (fb->valid && fb->lfb_base && fb->lfb_base < MM_MAX_PHYS) {
+	if (fb->valid && fb->lfb_base) {
 		lfb_lo = fb->lfb_base;
 		lfb_hi = lfb_lo + sz;
 	}
@@ -193,8 +201,8 @@ void mm_init(void)
 
 			if (e->type != E820_RAM || e->length == 0)
 				continue;
-			if (end > MM_MAX_PHYS)
-				end = MM_MAX_PHYS;
+			if (end > MM_HIGH_LIMIT)
+				end = MM_HIGH_LIMIT;
 			if (end > e->base) {
 				total_ram_kb +=
 					(unsigned int)((end - e->base) >> 10);
@@ -225,6 +233,9 @@ void mm_init(void)
 		}
 	}
 	bitmap_bits = bitmap_words * 32u;
+	low_words = MM_MAX_PHYS / (32u * PAGE_SIZE);
+	if (low_words > bitmap_words)
+		low_words = bitmap_words;
 
 	/* Everything starts in-use. */
 	for (i = 0; i < bitmap_words; i++)
@@ -278,6 +289,7 @@ void mm_init(void)
 		}
 	}
 	first_free = 0;
+	first_free_high = low_words;
 
 	serial_puts(SERIAL_COM1, "mm: bitmap ");
 	serial_putdec(SERIAL_COM1, bitmap_words * 4u);
@@ -285,17 +297,18 @@ void mm_init(void)
 	serial_putdec(SERIAL_COM1, bitmap_bits / 256u);
 	serial_puts(SERIAL_COM1, " MB at ");
 	serial_puthex(SERIAL_COM1, (unsigned int)bitmap);
-	serial_puts(SERIAL_COM1, "\n");
+	serial_puts(SERIAL_COM1, ", highmem free ");
+	serial_putdec(SERIAL_COM1, mm_free_high_pages() / 256u);
+	serial_puts(SERIAL_COM1, " MB\n");
 }
 
-/* Lowest free page.  Words below first_free are known to be full, so a
- * steady stream of allocations does not rescan the low, busy part. */
-unsigned int mm_alloc_page(void)
+/* Scan words [from, to) for the lowest free bit; claim it. */
+static unsigned int claim_bit(unsigned int from, unsigned int to,
+			      unsigned int *hint)
 {
 	unsigned int w;
-	unsigned int flags = irq_disable_save();
 
-	for (w = first_free; w < bitmap_words; w++) {
+	for (w = from; w < to; w++) {
 		unsigned int word = bitmap[w];
 
 		if (word != 0xFFFFFFFFu) {
@@ -303,17 +316,56 @@ unsigned int mm_alloc_page(void)
 
 			bitmap[w] = word | (1u << i);
 			free_pages--;
-			first_free = w;
-			irq_restore(flags);
+			*hint = w;
 			return (w * 32u + i) * PAGE_SIZE;
 		}
 	}
-	first_free = bitmap_words;
-	irq_restore(flags);
+	*hint = to;
 	return 0;
 }
 
-/* Lowest run of n free pages.  Full words break a run and empty words
+/* Lowest free page.  Words below first_free are known to be full, so a
+ * steady stream of allocations does not rescan the low, busy part. */
+unsigned int mm_alloc_page(void)
+{
+	unsigned int flags = irq_disable_save();
+	unsigned int pa = claim_bit(first_free, low_words, &first_free);
+
+	irq_restore(flags);
+	return pa;
+}
+
+/* A frame for a ring-3 mapping: highmem first, lowmem when highmem is
+ * exhausted (or absent).  The result may be above MM_MAX_PHYS, so the
+ * kernel must not dereference it directly; use kmap(). */
+unsigned int mm_alloc_user_page(void)
+{
+	unsigned int flags = irq_disable_save();
+	unsigned int pa = claim_bit(first_free_high, bitmap_words,
+				    &first_free_high);
+
+	if (!pa)
+		pa = claim_bit(first_free, low_words, &first_free);
+	irq_restore(flags);
+	return pa;
+}
+
+unsigned int mm_free_high_pages(void)
+{
+	unsigned int w, n = 0;
+
+	for (w = low_words; w < bitmap_words; w++) {
+		unsigned int x = ~bitmap[w];
+
+		while (x) {
+			x &= x - 1;
+			n++;
+		}
+	}
+	return n;
+}
+
+/* Lowest run of n free lowmem pages.  Full words break a run and empty words
  * extend it 32 pages at a time; only mixed words are walked bit by bit. */
 unsigned int mm_alloc_pages(unsigned int n)
 {
@@ -326,7 +378,7 @@ unsigned int mm_alloc_pages(unsigned int n)
 
 	flags = irq_disable_save();
 	p = first_free * 32u;
-	while (p < bitmap_bits) {
+	while (p < low_words * 32u) {
 		if ((p & 31u) == 0) {
 			unsigned int word = bitmap[p / 32u];
 
@@ -382,8 +434,12 @@ void mm_free_page(unsigned int addr)
 	if (bit_test(idx)) {
 		bit_clear(idx);
 		free_pages++;
-		if (idx / 32u < first_free)
-			first_free = idx / 32u;
+		if (idx / 32u < low_words) {
+			if (idx / 32u < first_free)
+				first_free = idx / 32u;
+		} else if (idx / 32u < first_free_high) {
+			first_free_high = idx / 32u;
+		}
 	}
 	irq_restore(flags);
 }

+ 12 - 3
kernel/mm.h

@@ -18,7 +18,8 @@
 #define E820_MAGIC       0xE820E820
 #define E820_MAX_ENTRIES 32
 
-/* Frame allocator covers physical RAM [0, MM_MAX_PHYS).  Bitmap: 1 bit/page.
+/* Frame allocator covers physical RAM [0, MM_HIGH_LIMIT): lowmem below
+ * MM_MAX_PHYS, highmem above it.  Bitmap: 1 bit/page.
  * The ceiling lives in phys.h so the UEFI loader cannot drift; see
  * kernel.ld for the loaded-image bound. */
 
@@ -47,14 +48,22 @@ struct e820_map {
 
 void mm_init(void);
 
-/* Return the base address of a free 4 KB page, or 0 if none. */
+/* Return the base address of a free 4 KB lowmem page (identity-mapped,
+ * below MM_MAX_PHYS), or 0 if none. */
 unsigned int mm_alloc_page(void);
+
+/* A frame for a ring-3 mapping, preferring highmem; 0 if none.  It may lie
+ * above MM_MAX_PHYS: only touch it through kmap() or the user mapping. */
+unsigned int mm_alloc_user_page(void);
+
+/* Free pages above MM_MAX_PHYS (a slow count, for reporting). */
+unsigned int mm_free_high_pages(void);
 void mm_free_page(unsigned int addr);
 
 /* Free a page-aligned boot-time reservation (never low 1 MB / the shadow). */
 void mm_free_range(unsigned int base, unsigned int size);
 
-/* Return n contiguous pages (base address), or 0 if unavailable. */
+/* Return n contiguous lowmem pages (base address), or 0 if unavailable. */
 unsigned int mm_alloc_pages(unsigned int n);
 
 unsigned int mm_free_pages(void);

+ 84 - 2
kernel/paging.c

@@ -14,6 +14,7 @@
 #include "mm.h"
 #include "mtrr.h"
 #include "process.h"
+#include "sched.h"
 #include "serial.h"
 #include "vga.h"
 #include "wm_abi.h"
@@ -26,6 +27,26 @@ _Static_assert(MM_MAX_PHYS <= USER_WIN_BASE &&
 	       MM_MAX_PHYS <= WM_CONTENT_VA_BASE &&
 	       MM_MAX_PHYS <= USER_HEAP_BASE,
 	       "identity map overlaps a ring-3 window");
+_Static_assert(MM_KMAP_VA >= USER_STACK_TOP &&
+	       MM_KMAP_VA + 0x400000u <= USER_WIN_BASE &&
+	       (MM_KMAP_VA & 0x3FFFFFu) == 0,
+	       "kmap window must be a free 4 MB slot below USER_WIN_BASE");
+
+/* kmap: KMAP_SLOTS nested mappings per CPU in the shared window page table. */
+#define KMAP_PDE    (MM_KMAP_VA >> 22)
+#define KMAP_SLOTS  4
+static unsigned int *kmap_pt;
+static unsigned int kmap_depth[MAX_CPUS];
+
+/* Every PDE writer below leaves the kmap slot alone: a BIOS-placed ACPI
+ * table, MMIO BAR or LFB in that 4 MB range cannot be identity-mapped. */
+static int kmap_pde_guard(unsigned int i)
+{
+	if (i != KMAP_PDE)
+		return 0;
+	serial_puts(SERIAL_COM1, "paging: refusing to map over the kmap window\n");
+	return 1;
+}
 
 /* CR4 bits that must not survive a BIOS protected-mode call.  A BIOS that
  * returns with CR4.PAE or EFER.LME set makes the first CR0.PG enable either
@@ -140,6 +161,8 @@ static void lfb_apply_pages(unsigned int *pd)
 		unsigned int page = i << 22;
 		unsigned int flags = PDE_PS | PDE_WRITABLE | PDE_PRESENT;
 
+		if (kmap_pde_guard(i))
+			continue;
 		if (!lfb_wc_ok || page < lfb_wc_base ||
 		    page + 0x400000 > lfb_wc_base + lfb_wc_size)
 			flags |= PDE_PCD;
@@ -172,7 +195,9 @@ void paging_map_ram(unsigned int base, unsigned int size)
 	if (!page_dir)
 		return;
 	for (i = p0; i < p1 && i < 1024; i++)
-		page_dir[i] = (i << 22) | PDE_PS | PDE_WRITABLE | PDE_PRESENT;
+		if (!kmap_pde_guard(i))
+			page_dir[i] = (i << 22) | PDE_PS | PDE_WRITABLE |
+				      PDE_PRESENT;
 }
 
 void paging_init(void)
@@ -204,6 +229,18 @@ void paging_init(void)
 	for (i = 0; i < MM_MAX_PHYS / 0x400000; i++)
 		page_dir[i] = (i << 22) | PDE_PS | PDE_WRITABLE | PDE_PRESENT;
 
+	/* The kmap window: a supervisor page table every process directory
+	 * shares (paging_new_directory copies the PDE). */
+	kmap_pt = (unsigned int *)mm_alloc_page();
+	if (!kmap_pt) {
+		serial_puts(SERIAL_COM1, "paging: out of memory (kmap)\n");
+		for (;;)
+			__asm__ volatile ("cli; hlt");
+	}
+	for (i = 0; i < 1024; i++)
+		kmap_pt[i] = 0;
+	page_dir[KMAP_PDE] = (unsigned int)kmap_pt | PDE_WRITABLE | PDE_PRESENT;
+
 	/* The OHCI ED/TD/interrupt-buffer page is DMA'd by the SB950's legacy
 	 * USB controller, which does not snoop dirty CPU cache lines.  Split
 	 * the containing 4 MB page now, before paging is enabled, so every
@@ -424,7 +461,8 @@ void paging_map_mmio_cr3(unsigned int cr3, unsigned int phys_base, unsigned int
 	unsigned int i;
 
 	for (i = p0; i < p1 && i < 1024; i++)
-		pd[i] = (i << 22) | PDE_PS | PDE_PCD |
+		if (!kmap_pde_guard(i))
+			pd[i] = (i << 22) | PDE_PS | PDE_PCD |
 			PDE_WRITABLE | PDE_PRESENT;
 }
 
@@ -464,3 +502,47 @@ static void uc_page_cr3(unsigned int cr3, unsigned int va)
 	}
 	pt[pte] |= PDE_PCD;
 }
+
+/* Temporary kernel mapping of one frame.  Lowmem is returned through the
+ * identity map; a highmem frame gets this CPU's next window slot with
+ * preemption disabled, so the task cannot migrate or be switched out while
+ * the slot is live.  Nothing may sleep or yield before kunmap(). */
+void *kmap(unsigned int phys)
+{
+	struct cpu *c;
+	unsigned int slot, va;
+
+	phys &= 0xFFFFF000u;
+	if (phys < MM_MAX_PHYS)
+		return (void *)phys;
+	preempt_disable();
+	c = this_cpu();
+	if (c->id >= MAX_CPUS || kmap_depth[c->id] >= KMAP_SLOTS) {
+		serial_puts(SERIAL_COM1, "paging: kmap slots exhausted\n");
+		for (;;)
+			__asm__ volatile ("cli; hlt");
+	}
+	slot = c->id * KMAP_SLOTS + kmap_depth[c->id]++;
+	va = MM_KMAP_VA + slot * PAGE_SIZE;
+	kmap_pt[slot] = phys | PTE_WRITABLE | PTE_PRESENT;
+	__asm__ volatile ("invlpg (%0)" : : "r"(va) : "memory");
+	return (void *)va;
+}
+
+/* Undo the most recent kmap() on this CPU (mappings nest LIFO). */
+void kunmap(void *addr)
+{
+	unsigned int va = (unsigned int)addr & 0xFFFFF000u;
+	struct cpu *c;
+	unsigned int slot;
+
+	if (va < MM_MAX_PHYS)
+		return;
+	c = this_cpu();
+	slot = (va - MM_KMAP_VA) / PAGE_SIZE;
+	kmap_pt[slot] = 0;
+	__asm__ volatile ("invlpg (%0)" : : "r"(va) : "memory");
+	if (kmap_depth[c->id])
+		kmap_depth[c->id]--;
+	preempt_enable();
+}

+ 8 - 0
kernel/paging.h

@@ -37,6 +37,14 @@ unsigned int paging_new_directory(void);
 #define PTE_WRITABLE 0x002
 #define PTE_USER     0x004
 
+/* Temporary kernel access to any frame, including highmem (above
+ * MM_MAX_PHYS).  Lowmem returns the identity address; highmem takes a
+ * per-CPU slot in the MM_KMAP_VA window with preemption disabled until the
+ * matching kunmap().  Mappings nest (LIFO, 4 per CPU); do not sleep, yield
+ * or take the BKL in between. */
+void *kmap(unsigned int phys);
+void kunmap(void *addr);
+
 /* Map one 4 KB page (va -> phys) in page directory `cr3`, allocating a page
  * table if needed.  Returns 1 on success, 0 on out-of-memory. */
 int paging_map_page(unsigned int cr3, unsigned int va, unsigned int phys,

+ 12 - 7
kernel/process.c

@@ -446,14 +446,14 @@ int process_create_user_tty(const char *name, const void *image, unsigned int si
 	}
 
 	/* Map the user stack just below USER_STACK_TOP.  Keep the top page's
-	 * physical address: the argument string is written through it (identity
-	 * mapped) after it is mapped in `cr3`. */
+	 * physical address: the argument string is written through a kmap of
+	 * it (the frame may be highmem) after it is mapped in `cr3`. */
 	{
 		unsigned int top_phys = 0;
 		unsigned int user_esp = USER_STACK_TOP;
 
 		for (i = 0; i < USER_STACK_PAGES; i++) {
-			unsigned int phys = mm_alloc_page();
+			unsigned int phys = mm_alloc_user_page();
 			unsigned int va = USER_STACK_TOP - (unsigned int)(i + 1) * 0x1000;
 			if (!phys ||
 			    !paging_map_page(cr3, va, phys,
@@ -470,11 +470,12 @@ int process_create_user_tty(const char *name, const void *image, unsigned int si
 		/* One C string argument at the top of the user stack with a fake
 		 * return address below it, so `_start(const char *arg)` reads it
 		 * as its first parameter without needing a crt0.  The virtual
-		 * addresses go into the frame; the writes go through the identity
-		 * map at the same offset from the physical page top. */
+		 * addresses go into the frame; the writes go through a kmap of the
+		 * top stack page at the same offset from its end. */
 		{
 			unsigned int va_top = USER_STACK_TOP;
-			unsigned char *base_phys = (unsigned char *)(top_phys + 0x1000);
+			unsigned char *top_map = kmap(top_phys);
+			unsigned char *base_phys = top_map + 0x1000;
 			unsigned char *str_phys;
 			unsigned int *frame_phys;
 			unsigned int len = 0;
@@ -484,7 +485,10 @@ int process_create_user_tty(const char *name, const void *image, unsigned int si
 				while (arg[len])
 					len++;
 			va = (va_top - (len + 1)) & ~15u;
-			if (va - 8 < USER_STACK_TOP - USER_STACK_PAGES * 0x1000) {
+			/* The string and frame are written through one kmap'd
+			 * page, so they must fit the top stack page. */
+			if (va - 8 < USER_STACK_TOP - 0x1000) {
+				kunmap(top_map);
 				mm_free_page(cr3);
 				kfree(p);
 				irq_restore(flags);
@@ -497,6 +501,7 @@ int process_create_user_tty(const char *name, const void *image, unsigned int si
 			frame_phys[0] = 0;   /* fake return address */
 			frame_phys[1] = va;  /* the argument pointer (user VA) */
 			user_esp = va - 8;
+			kunmap(top_map);
 		}
 		base = stack_alloc(&owned);
 		if (!base) {

+ 4 - 2
kernel/sys_mem.c

@@ -10,13 +10,15 @@ static void invlpg_va(unsigned int va)
 	__asm__ volatile ("invlpg (%0)" : : "r"(va) : "memory");
 }
 
+/* Heap frames may be highmem, so clear them through a kmap slot. */
 static void zero_page(unsigned int phys)
 {
-	unsigned int *p = (unsigned int *)phys;
+	unsigned int *p = (unsigned int *)kmap(phys);
 	unsigned int i;
 
 	for (i = 0; i < PAGE_SIZE / 4; i++)
 		p[i] = 0;
+	kunmap(p);
 }
 
 /* Map new pages [from, to) of the arena.  On failure release only the pages
@@ -27,7 +29,7 @@ static int map_pages(struct process *p, unsigned int from, unsigned int to,
 	unsigned int i;
 
 	for (i = from; i < to; i++) {
-		unsigned int phys = mm_alloc_page();
+		unsigned int phys = mm_alloc_user_page();
 		unsigned int va = USER_HEAP_BASE + i * PAGE_SIZE;
 
 		if (!phys)