瀏覽代碼

mm: raise the physical ceiling to 1 GB, move the frame bitmap out of BSS

The frame bitmap was a 16 KB BSS array sized for 512 MB, and the kernel
image must end below 0x90000.  mm_init() now sizes the bitmap to the
highest E820 RAM address and places it in the first RAM run at or above
0x900000 that avoids the boot rootfs pack and the LFB (a 512-byte static
array remains for the 16 MB no-E820 fallback).  _end drops from ~0x8ef20
to ~0x8bf20.

The allocators keep handing out the lowest free frames, but no longer
scan from page 0 with interrupts off: a first_free word hint skips the
full low part, mm_alloc_page() finds the bit with bsf, and
mm_alloc_pages() steps over full and empty words 32 pages at a time.

MM_MAX_PHYS is now 1 GB, exactly USER_BASE; paging.c statically asserts
the identity map never reaches a ring-3 window.  The UEFI loader may
place the pack anywhere below 1 GB, and QEMU guests default to 1G.

Verified in QEMU with a temporary randomized self-test against a naive
lowest-fit scan (128M/512M/1G/2G: 0 mismatches, no leak), a post-paging
write/read of every free frame up to 0x3FFFF000, and test-uefi/test-amd.

Known issue 6 (user pages leaked on reap) was stale: the reaper already
frees them.  It now records the real limit, RAM above 1 GB.
Johannes Zellner 2 天之前
父節點
當前提交
f3def28d01
共有 10 個文件被更改,包括 321 次插入 和 120 次删除
  1. 31 17
      AGENTS.md
  2. 3 3
      Makefile
  3. 3 3
      boot/efi/main.c
  4. 1 1
      docs/boot.html
  5. 2 2
      docs/development.html
  6. 1 1
      docs/index.html
  7. 30 19
      docs/memory.html
  8. 5 3
      include/phys.h
  9. 234 71
      kernel/mm.c
  10. 11 0
      kernel/paging.c

+ 31 - 17
AGENTS.md

@@ -141,7 +141,7 @@ make clean
   {wait}`, and `{click:X,Y}` for a left click at canvas coordinates).  It needs the multilib `lib32-sdl3`,
   `lib32-wayland` and `lib32-libxkbcommon`; host sockets replace the NebOS TCP
   stack and the kernel/WM paths are not exercised.
-- QEMU guests default to 512 MB (`QEMU_MEM`, e.g. `make run QEMU_MEM=1G`),
+- QEMU guests default to 1 GB (`QEMU_MEM`, e.g. `make run QEMU_MEM=512M`),
   matching `MM_MAX_PHYS` in `phys.h`; all run and UEFI targets pass `-m`.
 - The command pack is the system rootfs: `cmds.pack` (NBPAK1), gzip-compressed
   to `cmds_pack_gz.bin`, wrapped by `mkrootfs` with a one-sector `NBROOT1`
@@ -251,7 +251,7 @@ and `make net-dev`/`kernel-net-dev` reload over the network.  `bootlog save
 5. **UEFI (`boot/efi/`)** — an x86-64 PE32+ `BOOTX64.EFI` loads from the ESP. It
    picks a GOP mode, converts `GetMemoryMap` into the E820 map at `0x1000`,
    stages `kernel.bin` below 4 GB, allocates a loader-data buffer below the
-   512 MB identity window (above the low 1 MB) for the embedded gz rootfs
+   1 GB identity window (above the low 1 MB) for the embedded gz rootfs
    and copies it there, fills
    `0x500` (boot info) and `0x600` (framebuffer), calls `ExitBootServices`,
    copies a 192-byte trampoline to `0x800`,
@@ -285,7 +285,7 @@ a Pentium 1 (and a Pentium II board without ACPI) on the UP path.
 | `0x60000` | No longer a rootfs staging area: the legacy stage copies each 16 KB chunk straight to `0x900000`.  The kernel BSS/boot stack may use this range |
 | `_end`    | Kernel end (linker symbol; `kernel.ld` keeps `_end <= 0x90000` so the top of the low-memory gap and the EBDA stay untouched).  BSS and the boot stack may extend past `0x60000` |
 | (in-image) | Bootstrap/protected-mode stack: 8 KB `.stack` NOLOAD section, `__boot_stack_top`/`__boot_stack_bottom` (travels with the kernel) |
-| `0x900000`| Legacy boot-time rootfs pack written directly by `vesa_stage.S` (reserved by `mm_init` until `ramfs_init` takes its copy, then freed; process stacks and kexec staging also start here) |
+| `0x900000`| Legacy boot-time rootfs pack written directly by `vesa_stage.S` (reserved by `mm_init` until `ramfs_init` takes its copy, then freed); `mm_init` places the frame bitmap in the first RAM after it (`0xAF3000` with today's pack); process stacks and kexec staging also start here |
 | `0x100000`| VGA shadow buffer (`VGA_SHADOW_BASE`, `vga.h`) |
 
 The legacy stage uses `0x3000..0x6FFF` as a 16 KB-chunk rootfs staging
@@ -295,7 +295,7 @@ later.
 
 The boot-time rootfs pack placement depends on the boot path: the legacy
 stage copies it directly to `0x900000`, the UEFI loader puts it at the top of
-RAM below 512 MB, and kexec leaves it in the staging buffer
+RAM below 1 GB, and kexec leaves it in the staging buffer
 (≥ `0x900000`).  `mm_init` reserves `[pack_addr, pack_addr+pack_size)` from
 boot info when it is above the low 1 MB, so the legacy pack is protected
 from the frame allocator too.  `ramfs_init` copies/inflates it into fresh
@@ -353,10 +353,20 @@ it to `0x7E00`.  The kernel region itself only has one wall (`_end` below
   trampoline at `0x6000`), and `spinlock.h` (the IRQ-safe spinlocks).
   `cpu_enable_sse()` runs on every AP: an AP starts with firmware CR4, and the
   SSE2 drawing fast paths #UD until OSFXSR/OSXMMEXCPT are set.
-- **Memory**: `phys.h` — the shared 512 MB physical ceiling (`MM_MAX_PHYS`),
-  also used by the UEFI loader. `mm.c/h` — INT 15h E820 detection (real-mode in
+- **Memory**: `phys.h` — the shared 1 GB physical ceiling (`MM_MAX_PHYS`),
+  also used by the UEFI loader; it ends where ring-3 space starts
+  (`USER_BASE` = `0x40000000`), and `paging.c` statically asserts that the
+  identity map never reaches a user window.  RAM above 1 GB is ignored (no
+  highmem zone or PAE yet).  `mm.c/h` — INT 15h E820 detection (real-mode in
   `vesa_stage.S`, map at `0x1000`) + a bitmap page-frame allocator (4 KB
-  pages). Reserves low 1 MB and an 8 MB VGA-shadow region. `kmalloc.c/h` — a power-of-two heap
+  pages). Reserves low 1 MB and an 8 MB VGA-shadow region.  The bitmap is
+  sized to the highest E820 RAM address (32 KB at 1 GB) and is not in BSS:
+  `mm_init` places it in the first RAM run at or above `0x900000` that
+  avoids the boot pack and the LFB and marks its own pages used (a 512-byte
+  static array covers the 16 MB no-E820 fallback).  Allocation is lowest
+  address first: a `first_free` word hint skips the full low words,
+  `mm_alloc_page()` finds the bit with `bsf`, and `mm_alloc_pages()` steps
+  over full and empty words 32 pages at a time. `kmalloc.c/h` — a power-of-two heap
   allocator on top of the frame allocator (used for PCBs and command context).
   Process stacks (≥ `0x900000`) come from the frame allocator, with an 8-stack
   fallback pool (a static BSS array; no fixed address).
@@ -2489,8 +2499,8 @@ and the WM falls back to polling (`io_keyboard_irq_masked()`).
   structure whose fields are read before being written must be cleared on
   allocation.  `clear_proc()` (`process.c`) exists for exactly this: a stale
   `procs[i].cr3` made the boot-time MMIO mapping (`paging_map_mmio_cr3`) fault
-  on the board while QEMU was fine.  Anything above `MM_MAX_PHYS` (512 MB) is
-  also *unmapped*, so a wild pointer there is an instant #PF.
+  on the board while QEMU was fine.  Anything between `MM_MAX_PHYS` (1 GB) and
+  the user windows is *unmapped*, so a wild pointer there is an instant #PF.
 - **All drawing goes into the current render target** (the shadow buffer by
   default; `vga_target()`, see the graphics bullet); the main loop flushes the
   pending damage rects to VRAM on every iteration (byte
@@ -2528,9 +2538,11 @@ and the WM falls back to polling (`io_keyboard_irq_masked()`).
 3. No per-process address spaces or process isolation (paging is identity-mapped).
 4. FAT32: FSInfo free-count not updated, no directory-chain growth.
 5. `disk.c` `fdc_reset()` and `fdc_init_once()` overlap/duplicate logic.
-6. Killing/reaping a ring-3 process frees its kernel stack and page directory
-   but not the user pages mapped into it (a few pages leak per command run;
-   needs a page-table walk/unmap on reap).
+6. Physical RAM above 1 GB is unused: the identity map stops at `USER_BASE`
+   (`0x40000000`).  Using it needs a highmem zone (user-only frames plus a
+   per-CPU temporary kernel mapping for zeroing/ELF loads) and, for RAM above
+   4 GB, PAE.  MMIO is still mapped at its physical address, which also
+   blocks growing ring-3 space above the PCI hole.
 7. `display` (needs a GPU snapshot) and `lsusb` (a 45-byte wrapper around
    the USB tree printer) are the last non-privileged commands still living in
    the kernel, plus the privileged ones (`peek`, `poke`, `dbg`, `serial`,
@@ -2613,15 +2625,17 @@ and the WM falls back to polling (`io_keyboard_irq_masked()`).
     `websrv` cannot start while any TCP socket is open.  Kill the other
     command first.
 20. `/share` is RAM only: a reboot/kexec starts empty, and the 16 MB cap is
-    charged against the 512 MB identity-mapped window.
+    charged against the 1 GB identity-mapped window.
 
 21. The TCP backlog's early-request buffers cost 8 KB of kernel BSS
     (8 slots x 1 KB), and the FAT listing cache another 5 KB.  The command
     pack is out of the kernel now; after the WiFi/netmenu/install code and the
     per-connection TCP queues moved to the frame allocator, `_end` is
-    ~0x8ef20 (with the 6200/6300 DVM layer, the RTL8821AU driver, the
+    ~0x8ef20 before (with the 6200/6300 DVM layer, the RTL8821AU driver, the
     8×16 boot-console font, the IOAPIC/AMD-thermal additions and the safe
-    AMD MTRR path, the 32x32 cursor save buffer and the mount-policy table), ~4 KB below the `0x90000` wall, and the remaining limit
+    AMD MTRR path, the 32x32 cursor save buffer and the mount-policy table)
+    ~0x8bf20 now that the frame bitmap moved out of BSS, ~16 KB below the
+    `0x90000` wall, and the remaining limit
     is kernel code/buffers only.
 
 22. Browser limits: JavaScript runs inline and external classic scripts on
@@ -3030,5 +3044,5 @@ and the WM falls back to polling (`io_keyboard_irq_masked()`).
     for every spinner frame; both paths were written from the Linux
     drivers and only Cirrus and the software cursor are tested in QEMU.
     The cursor shape follows the WM poll, so it can lag the pointer by
-    one WM loop.  The 4 KB save buffer and the cursor code (~8 KB in all) leave `_end` at
-    ~0x8ef20, ~4 KB below the `0x90000` wall.
+    one WM loop.  The 4 KB save buffer and the cursor code (~8 KB in all) are included in
+    `_end` ~0x8bf20, ~16 KB below the `0x90000` wall.

+ 3 - 3
Makefile

@@ -929,9 +929,9 @@ root.img: browser.elf gallery.elf testdata/browser/demo.html testdata/browser/js
 
 OVMF ?= $(firstword $(wildcard /usr/share/ovmf/x64/OVMF.4m.fd /usr/share/OVMF/OVMF.4m.fd))
 
-# Guest RAM for the QEMU targets.  The kernel ceiling is 512 MB (phys.h);
-# override with `make run QEMU_MEM=1G` for larger guests.
-QEMU_MEM ?= 512M
+# Guest RAM for the QEMU targets, matching the 1 GB kernel ceiling (phys.h);
+# override with e.g. `make run QEMU_MEM=512M` (RAM above 1 GB is ignored).
+QEMU_MEM ?= 1G
 
 # root.img is deliberately not attached: a running QEMU may hold its write
 # lock, and the UEFI boot test does not need a second filesystem.

+ 3 - 3
boot/efi/main.c

@@ -44,7 +44,7 @@ extern unsigned char tramp_blob_end[];
  * generous and keeps the retry loop simple. */
 static UINT8 memmap_buf[65536] __attribute__((aligned(8)));
 
-/* One bit per 4 KB page over the kernel's 512 MB identity window.  Starts
+/* One bit per 4 KB page over the kernel's 1 GB identity window.  Starts
  * all-used; only EfiConventional/BootServices memory is freed.  EfiLoader*
  * (this image and the staging buffer) stays reserved, which is safe and
  * costs only a few hundred KB. */
@@ -399,7 +399,7 @@ static void build_e820(EFI_MEMORY_DESCRIPTOR *map, UINTN map_size,
 	}
 
 	/* The scanout is MMIO, not RAM; make sure it can never be handed out
-	 * even when its BAR happens to sit below the 512 MB ceiling. */
+	 * even when its BAR happens to sit below the 1 GB ceiling. */
 	if (fb->valid)
 		use_range(fb->base, fb->size);
 
@@ -778,7 +778,7 @@ EFI_STATUS efi_main(EFI_HANDLE image, EFI_SYSTEM_TABLE *st)
 		fail_before(st, "cannot allocate kernel staging buffer");
 
 	/* Rootfs pack: allocate a writable loader-data buffer below the
-	 * kernel's 512 MB identity window (and above the low 1 MB it
+	 * kernel's 1 GB identity window (and above the low 1 MB it
 	 * reserves).  It must be below MAX_PHYS: the kernel only
 	 * identity-maps that much and ramfs_init() reads it after paging is
 	 * enabled. */

+ 1 - 1
docs/boot.html

@@ -206,7 +206,7 @@
 		<ol>
 			<li>Selects a GOP mode with BGRX or RGBX 32bpp, width at most 1920, <code>pitch × height</code> at most 8 MB (1920×1080 fits at 8.29 MB), and a framebuffer below 4 GB. With 32bpp, that size limit also keeps a full-screen RGB565 canvas inside the 4 MB window slot; the VBE stage additionally checks 2 M pixels because a 24bpp 1920×1200 mode would otherwise pass the shadow check.</li>
 			<li>Calls <code>GetMemoryMap</code> and converts the firmware map into the kernel's E820 format at <code>0x1000</code>.</li>
-			<li>Locates the parent whole-disk Block I/O handle from the ESP device path, reads the P1 boot sector and P2 <code>NBROOT1</code> header, and stages the kernel below 4 GB and the gzip pack below the 512 MB identity window. The kernel/pack embedded in <code>BOOTX64.EFI</code> are a fallback only.</li>
+			<li>Locates the parent whole-disk Block I/O handle from the ESP device path, reads the P1 boot sector and P2 <code>NBROOT1</code> header, and stages the kernel below 4 GB and the gzip pack below the 1 GB identity window. The kernel/pack embedded in <code>BOOTX64.EFI</code> are a fallback only.</li>
 			<li>Writes the boot-info and framebuffer blocks, calls <code>ExitBootServices</code>, then copies a 192-byte trampoline to <code>0x800</code>.</li>
 			<li>Leaves IA-32e mode (clears <code>CR0.PG</code>, <code>EFER.LME</code>, <code>CR4.PAE/PGE/PCIDE/SMEP/SMAP/LA57</code>), copies the kernel to <code>0x7E00</code>, and jumps to <code>kernel_entry</code>.</li>
 		</ol>

+ 2 - 2
docs/development.html

@@ -245,9 +245,9 @@
 			</table>
 		</div>
 		<p>
-			QEMU guests default to 512 MB, matching <code>MM_MAX_PHYS</code> in
+			QEMU guests default to 1 GB, matching <code>MM_MAX_PHYS</code> in
 			<span class="file">phys.h</span>. Override the size with
-			<code>QEMU_MEM</code>, for example <code>make run QEMU_MEM=1G</code>;
+			<code>QEMU_MEM</code>, for example <code>make run QEMU_MEM=512M</code>;
 			all run and UEFI targets pass it through.
 		</p>
 	</section>

+ 1 - 1
docs/index.html

@@ -157,7 +157,7 @@
 			<li><strong>Freestanding C.</strong> The kernel has no libc, <code>malloc</code> or <code>printf</code>; ring-3 commands can link the in-tree <code>libc.a</code> (<code>malloc</code>, strings, stdlib). Formatting and allocation are in-tree either way.</li>
 			<li><strong>i586 baseline.</strong> Builds use <code>-march=i586 -mno-sse -mno-sse2 -mno-mmx</code>; SSE2 drawing paths are probed at runtime.</li>
 			<li><strong>Identity-mapped paging.</strong> Physical and virtual addresses are equal. There is no per-process address space or isolation.</li>
-			<li><strong>512 MB physical ceiling.</strong> The frame bitmap covers 512 MB; anything above that is unmapped.</li>
+			<li><strong>1 GB physical ceiling.</strong> The identity map ends where ring-3 space starts (<code>0x40000000</code>); RAM above it is ignored.</li>
 			<li><strong>Hardcoded low memory.</strong> Boot info, the E820 map, the framebuffer block and the AP trampoline have fixed addresses that every boot path must honour.</li>
 			<li><strong>No firmware runtime.</strong> <code>reboot</code> and <code>poweroff</code> use legacy ports; UEFI runtime services are not called after <code>ExitBootServices</code>.</li>
 		</ul>

+ 30 - 19
docs/memory.html

@@ -41,7 +41,7 @@
 			window canvases, the ring-3 heap and the rootfs copy — is dynamic.
 		</p>
 		<div class="meta">
-			<span class="pill">512 MB physical ceiling</span>
+			<span class="pill">1 GB physical ceiling</span>
 			<span class="pill">4 KB frames</span>
 			<span class="pill">4 MB PSE pages</span>
 			<span class="pill">identity map</span>
@@ -76,29 +76,32 @@
 		<figure class="figure">
 			<svg viewBox="0 0 920 180" role="img" aria-labelledby="phys-title phys-desc">
 				<title id="phys-title">Physical memory composition</title>
-				<desc id="phys-desc">Of the 512 MB addressable window, the low 1 MB and an 8 MB VGA shadow reserve are unavailable, leaving about 503 MB for the frame allocator.</desc>
+				<desc id="phys-desc">Of the 1 GB addressable window, the low 1 MB and an 8 MB VGA shadow reserve are unavailable, leaving about 1015 MB for the frame allocator.</desc>
 				<text class="svg-text" x="20" y="28">0x00000000</text>
-				<text class="svg-text" x="900" y="28" text-anchor="end">0x20000000 · 512 MB</text>
+				<text class="svg-text" x="900" y="28" text-anchor="end">0x40000000 · 1 GB</text>
 
-				<rect class="svg-box blue" x="40" y="44" width="2" height="34" rx="1"/>
-				<rect class="svg-box accent" x="42" y="44" width="13.5" height="34" rx="1"/>
-				<rect class="svg-box green" x="55.5" y="44" width="844.5" height="34" rx="2"/>
+				<rect class="svg-box blue" x="40" y="44" width="1.5" height="34" rx="1"/>
+				<rect class="svg-box accent" x="41.5" y="44" width="6.7" height="34" rx="1"/>
+				<rect class="svg-box green" x="48.2" y="44" width="851.8" height="34" rx="2"/>
 
 				<text class="svg-tiny" x="40" y="100">low 1 MB</text>
 				<text class="svg-tiny" x="64" y="100">VGA shadow · 8 MB</text>
-				<text class="svg-tiny" x="477" y="100" text-anchor="middle">frame allocator · ~503 MB</text>
+				<text class="svg-tiny" x="477" y="100" text-anchor="middle">frame allocator · ~1015 MB</text>
 
 				<line class="svg-line" x1="41" y1="78" x2="41" y2="88"/>
-				<line class="svg-line" x1="48" y1="78" x2="48" y2="88"/>
+				<line class="svg-line" x1="45" y1="78" x2="45" y2="88"/>
 				<line class="svg-line" x1="490" y1="78" x2="490" y2="88"/>
 
 				<text class="svg-tiny" x="40" y="130">boot structures · kernel image · BIOS/UMA · DMA buffers</text>
 				<text class="svg-tiny" x="40" y="150">The 8 MB reserve protects the shadow buffer; the allocator starts at 0x900000.</text>
 			</svg>
 			<figcaption>
-				<strong>512 MB is a hard ceiling.</strong> The frame bitmap covers
-				<code>[0, 0x20000000)</code>; addresses above it are not mapped by
-				the kernel page tables. The VGA shadow reserve is conservative: the
+				<strong>1 GB is a hard ceiling</strong> (<code>MM_MAX_PHYS</code>,
+				<code>include/phys.h</code>). The identity map covers
+				<code>[0, 0x40000000)</code> and ring-3 space starts at
+				<code>USER_BASE</code> = <code>0x40000000</code>, so RAM above 1 GB is
+				ignored until a highmem zone exists; <code>paging.c</code>
+				statically asserts the two never overlap. The VGA shadow reserve is conservative: the
 				actual buffer is smaller, but the gap keeps a future resolution bump
 				from colliding with allocated pages.
 			</figcaption>
@@ -132,7 +135,7 @@
 					<tr><td><code>0x6F00</code></td><td>AP startup info block</td><td>CR3, stack top, C entry, CPU index</td></tr>
 					<tr><td><code>0x60000</code></td><td>Former rootfs staging</td><td>No longer used by the loader; kernel BSS/stack may use it</td></tr>
 					<tr><td><code>_end</code></td><td>Kernel end</td><td><code>kernel.ld</code> asserts <code>_end ≤ 0x90000</code></td></tr>
-					<tr><td><code>0x900000</code></td><td>Boot pack (written directly by the legacy stage); dynamic stacks and kexec staging</td><td>Freed after <code>ramfs_init</code> takes its copy</td></tr>
+					<tr><td><code>0x900000</code></td><td>Boot pack (written directly by the legacy stage), then the frame bitmap right after it; dynamic stacks and kexec staging</td><td>The pack is freed after <code>ramfs_init</code> takes its copy; the bitmap stays</td></tr>
 					<tr><td><code>0x100000</code></td><td>VGA shadow buffer base</td><td><code>VGA_SHADOW_BASE</code>; 8 MB reserved</td></tr>
 					<tr><td><code>0x40000000</code></td><td>Ring-3 ELF link/load address</td><td><code>USER_BASE</code>, 4 MB window up to <code>0x40400000</code></td></tr>
 					<tr><td><code>0x50000000</code></td><td>Ring-3 window canvas</td><td><code>USER_WIN_BASE</code>, 4 MB slot</td></tr>
@@ -180,8 +183,17 @@
 		<h2>Frame allocator</h2>
 		<p>
 			<span class="file">mm.c</span> keeps one bit per 4 KB page for the
-			whole 512 MB window. The bitmap is not IRQ-safe; callers save and
-			disable interrupts around allocations. The allocator returns:
+			RAM below the 1 GB ceiling: the bitmap is sized to the highest E820 RAM
+			address (32 KB at 1 GB) and is not part of the kernel image.
+			<code>mm_init</code> places it in the first E820 RAM run at or above
+			<code>0x900000</code> that avoids the boot rootfs pack and the LFB, and
+			marks its own pages used (legacy boots put it right after the pack;
+			a 16 MB static fallback covers a missing E820 map).  The allocators
+			disable interrupts around the bitmap and hand out the lowest free
+			frames: a <code>first_free</code> word hint skips the full low part,
+			<code>mm_alloc_page()</code> finds the bit with <code>bsf</code>, and
+			<code>mm_alloc_pages(n)</code> steps over full and empty words 32 pages
+			at a time. The allocator returns:
 		</p>
 		<ul>
 			<li><code>mm_alloc_page()</code> — one zero-address-ambiguous page (physical address 0 means failure).</li>
@@ -190,7 +202,7 @@
 		</ul>
 		<p>
 			At boot, <code>mm_init</code> reserves the low 1 MB, the 8 MB VGA
-			shadow region, any LFB that happens to live below 512 MB, and the
+			shadow region, any LFB that happens to live below 1 GB, the bitmap itself and the
 			boot-time rootfs pack. The pack reservation is released by
 			<code>ramfs_init</code> after it copies/inflates the pack into fresh
 			pages.
@@ -296,7 +308,7 @@
 		<h2>Rootfs pack lifetime</h2>
 		<ol>
 			<li>The compressed pack is at most 2 MB (<code>PACK_MAX</code>), so the legacy copy ends below <code>0xB00000</code>; inflated it is about 2 MB with the WiFi firmware (the 8 MB <code>RAMFS_MAX_DECOMPRESSED</code> cap applies).</li>
-			<li>The boot path places the gzip pack in RAM: the legacy stage copies it directly to <code>0x900000</code> with <code>INT 15h AH=87h</code>, UEFI into a loader-data buffer below 512 MB, kexec in its staging buffer.</li>
+			<li>The boot path places the gzip pack in RAM: the legacy stage copies it directly to <code>0x900000</code> with a protected-mode <code>rep movsb</code>, UEFI into a loader-data buffer below 1 GB, kexec in its staging buffer.</li>
 			<li><code>mm_init</code> reserves <code>[pack_addr, pack_addr + pack_size)</code> so the frame allocator cannot reuse it.</li>
 			<li><code>ramfs_init</code> takes a private page-allocated copy, validates the <code>NBROOT1</code> header and inflates the pack into it.</li>
 			<li><code>mm_free_range</code> releases the boot region when it is above the VGA shadow reserve. The private copy lives until <code>ramfs_replace</code> or reboot.</li>
@@ -311,8 +323,7 @@
 		<h2>Limits &amp; leaks</h2>
 		<ul>
 			<li><strong>No per-process address spaces.</strong> Paging is identity-mapped; there is no memory protection between processes.</li>
-			<li><strong>512 MB ceiling.</strong> Physical memory above <code>0x20000000</code> is unmapped. A wild pointer there is an instant <code>#PF</code>.</li>
-			<li><strong>User page leak on reap.</strong> Killing a ring-3 process frees its kernel stack and page directory but not all user pages mapped into it; a few pages leak per command run.</li>
+			<li><strong>1 GB ceiling.</strong> Physical memory above <code>0x40000000</code> is not used: that address range is ring-3 space, so the kernel has no mapping for frames above it (a highmem zone with temporary kernel mappings, or PAE for RAM above 4 GB, would be needed). A wild kernel pointer between the identity map and the user windows is an instant <code>#PF</code>.</li>
 			<li><strong>Kernel heap fragmentation.</strong> <code>kmalloc</code>'s power-of-two size classes make long-running mixed-size kernel allocation wasteful; ring-3 commands use the separate first-fit libc heap described above.</li>
 			<li><strong>No TLB shootdown.</strong> Cross-CPU mapping changes rely on not-present entries not being cached.</li>
 		</ul>
@@ -322,7 +333,7 @@
 		<h2>Source map</h2>
 		<ul class="filelist">
 			<li><code>kernel/mm.c</code>, <code>kernel/mm.h</code> — E820 + bitmap allocator</li>
-			<li><code>include/phys.h</code> — shared 512 MB physical ceiling</li>
+			<li><code>include/phys.h</code> — shared 1 GB physical ceiling</li>
 			<li><code>kernel/kmalloc.c</code>, <code>kernel/kmalloc.h</code> — kernel heap</li>
 			<li><code>user/lib/libc/malloc.c</code>, <code>user/lib/libc/ucmd_mem.c</code> — ring-3 heap and <code>SYS_MEM_*</code> backend</li>
 			<li><code>user/lib/libc/string.c</code>, <code>user/lib/libc/stdlib.c</code> — ring-3 libc support</li>

+ 5 - 3
include/phys.h

@@ -2,8 +2,10 @@
 #ifndef PHYS_H
 #define PHYS_H
 
-/* 512 MB: enough for the browser and its JavaScript heap, while the 4 KB frame bitmap
- * stays 16 KB and the 4 MB identity map needs only 128 PDEs. */
-#define MM_MAX_PHYS      (512u * 1024 * 1024)
+/* 1 GB: the kernel identity-maps [0, MM_MAX_PHYS) with 4 MB pages, and ring-3
+ * address space starts at USER_BASE (0x40000000), so the identity map cannot
+ * grow past 1 GB without a highmem zone.  The frame bitmap (32 KB at 1 GB)
+ * is placed in RAM by mm_init(), not in the kernel's BSS. */
+#define MM_MAX_PHYS      (1024u * 1024 * 1024)
 
 #endif

+ 234 - 71
kernel/mm.c

@@ -5,11 +5,24 @@
 #include "serial.h"
 #include "vga.h"
 
-/* One bit per 4 KB page; bit set = in use. */
-#define MM_BITMAP_BITS  (MM_MAX_PHYS / PAGE_SIZE)
-#define MM_BITMAP_WORDS (MM_BITMAP_BITS / 32)
-
-static unsigned int bitmap[MM_BITMAP_WORDS];
+/* One bit per 4 KB page; bit set = in use.
+ *
+ * The bitmap covers [0, top of RAM) rounded up to 128 KB, at most
+ * MM_MAX_PHYS.  It is not in BSS (1 GB needs 32 KB, and the kernel image
+ * must end below 0x90000): mm_init() places it in the first E820 RAM run
+ * above the VGA-shadow reserve that avoids the boot rootfs pack and the
+ * LFB, then marks those pages used.  The small static array is only the
+ * fallback when no E820 map exists (the 16 MB guess) or nothing fits. */
+#define MM_FALLBACK_PHYS   0x1000000u      /* 16 MB */
+#define MM_FALLBACK_WORDS  (MM_FALLBACK_PHYS / PAGE_SIZE / 32)
+#define MM_PLACE_MIN       (MM_SHADOW_BASE + MM_SHADOW_RESERVE)
+
+static unsigned int fallback_bitmap[MM_FALLBACK_WORDS];
+static unsigned int *bitmap = fallback_bitmap;
+static unsigned int bitmap_words;   /* words covered by the bitmap */
+static unsigned int bitmap_bits;    /* bitmap_words * 32 */
+static unsigned int bitmap_bytes;   /* page-rounded size when placed in RAM */
+static unsigned int first_free;     /* lowest word that may hold a free bit */
 static unsigned int free_pages;
 static unsigned int total_ram_kb;
 
@@ -44,16 +57,22 @@ static inline void irq_restore(unsigned int flags)
 	__asm__ volatile ("pushl %0; popfl" : : "r"(flags));
 }
 
+static unsigned long long bitmap_limit(void)
+{
+	return (unsigned long long)bitmap_bits * PAGE_SIZE;
+}
+
 /* Mark a physical byte range [base, base+len) as used in the bitmap. */
 static void mark_used(unsigned long long base, unsigned long long len)
 {
 	unsigned long long end = base + len;
+	unsigned long long lim = bitmap_limit();
 	unsigned int p0, p1, p;
 
-	if (base >= MM_MAX_PHYS)
+	if (base >= lim)
 		return;
-	if (end > MM_MAX_PHYS)
-		end = MM_MAX_PHYS;
+	if (end > lim)
+		end = lim;
 
 	p0 = (unsigned int)(base >> 12);
 	p1 = (unsigned int)((end + PAGE_SIZE - 1) >> 12);
@@ -65,45 +84,159 @@ static void mark_used(unsigned long long base, unsigned long long len)
 static void mark_free(unsigned long long base, unsigned long long len)
 {
 	unsigned long long end = base + len;
+	unsigned long long lim = bitmap_limit();
 	unsigned int p0, p1, p;
 
-	if (base >= MM_MAX_PHYS)
+	if (base >= lim)
 		return;
-	if (end > MM_MAX_PHYS)
-		end = MM_MAX_PHYS;
+	if (end > lim)
+		end = lim;
 
-	p0 = (unsigned int)(base >> 12);
+	p0 = (unsigned int)((base + PAGE_SIZE - 1) >> 12);
 	p1 = (unsigned int)(end >> 12);
 	for (p = p0; p < p1; p++)
 		bit_clear(p);
 }
 
+static int e820_valid(const struct e820_map *map)
+{
+	return map->magic == E820_MAGIC && map->count <= E820_MAX_ENTRIES;
+}
+
+static int overlaps(unsigned long long a0, unsigned long long a1,
+		    unsigned long long b0, unsigned long long b1)
+{
+	return b1 > b0 && a0 < b1 && b0 < a1;
+}
+
+/* Boot-time regions that must stay out of the allocator (and out of the
+ * bitmap's own placement): the LFB when a BIOS put it in RAM, and the
+ * rootfs pack until ramfs_init() has copied it. */
+static unsigned long long lfb_lo, lfb_hi, pack_lo, pack_hi;
+
+static void find_boot_regions(void)
+{
+	const volatile struct vga_boot_fb_info *fb = VGA_BOOT_FB;
+	unsigned int sz, pa, ps;
+
+	__asm__ volatile ("" : "+r"(fb));
+	lfb_lo = lfb_hi = 0;
+	sz = (unsigned int)fb->pitch * fb->height;
+	if (!sz)
+		sz = 0x1000000;
+	if (fb->valid && fb->lfb_base && fb->lfb_base < MM_MAX_PHYS) {
+		lfb_lo = fb->lfb_base;
+		lfb_hi = lfb_lo + sz;
+	}
+
+	pack_lo = pack_hi = 0;
+	pa = bootinfo_pack_addr();
+	ps = bootinfo_pack_size();
+	if (pa >= MM_RESERVE_LOW && pa < MM_MAX_PHYS && ps != 0 &&
+	    ps <= MM_MAX_PHYS && pa + ps > pa && pa + ps <= MM_MAX_PHYS) {
+		pack_lo = pa;
+		pack_hi = (unsigned long long)pa + ps;
+	}
+}
+
+static unsigned long long page_up(unsigned long long a)
+{
+	return (a + PAGE_SIZE - 1) & ~(unsigned long long)(PAGE_SIZE - 1);
+}
+
+/* First E820 RAM address >= MM_PLACE_MIN where `bytes` fit without
+ * touching the pack or the LFB; 0 if there is none. */
+static unsigned int place_bitmap(const struct e820_map *map,
+				 unsigned int bytes)
+{
+	unsigned int i;
+
+	for (i = 0; i < map->count; i++) {
+		const struct e820_entry *e = &map->entry[i];
+		unsigned long long lo, hi, c;
+
+		if (e->type != E820_RAM || e->length == 0)
+			continue;
+		lo = page_up(e->base);
+		hi = e->base + e->length;
+		if (hi > MM_MAX_PHYS)
+			hi = MM_MAX_PHYS;
+		c = lo < MM_PLACE_MIN ? MM_PLACE_MIN : lo;
+		while (c + bytes <= hi) {
+			if (overlaps(c, c + bytes, pack_lo, pack_hi))
+				c = page_up(pack_hi);
+			else if (overlaps(c, c + bytes, lfb_lo, lfb_hi))
+				c = page_up(lfb_hi);
+			else
+				return (unsigned int)c;
+		}
+	}
+	return 0;
+}
+
 /* Mark the whole bitmap as used, then walk the E820 map and free RAM. */
 void mm_init(void)
 {
 	const struct e820_map *map = (const struct e820_map *)E820_MAP_ADDR;
-	unsigned int i, n;
+	unsigned long long top = 0;
+	unsigned int i, n, at = 0;
+
+	find_boot_regions();
+	total_ram_kb = 0;
+
+	/* Size the bitmap to the highest RAM address below the ceiling. */
+	if (e820_valid(map)) {
+		n = map->count;
+		for (i = 0; i < n; i++) {
+			const struct e820_entry *e = &map->entry[i];
+			unsigned long long end = e->base + e->length;
+
+			if (e->type != E820_RAM || e->length == 0)
+				continue;
+			if (end > MM_MAX_PHYS)
+				end = MM_MAX_PHYS;
+			if (end > e->base) {
+				total_ram_kb +=
+					(unsigned int)((end - e->base) >> 10);
+				if (end > top)
+					top = end;
+			}
+		}
+	}
+	if (top == 0)
+		top = MM_FALLBACK_PHYS;
+	bitmap_words = (unsigned int)((top + 32ull * PAGE_SIZE - 1) /
+				      (32ull * PAGE_SIZE));
+
+	bitmap = fallback_bitmap;
+	bitmap_bytes = 0;
+	if (bitmap_words > MM_FALLBACK_WORDS) {
+		unsigned int bytes = (bitmap_words * 4u + PAGE_SIZE - 1) &
+				     ~(PAGE_SIZE - 1u);
+
+		at = place_bitmap(map, bytes);
+		if (at) {
+			bitmap = (unsigned int *)at;
+			bitmap_bytes = bytes;
+		} else {
+			serial_puts(SERIAL_COM1,
+				    "mm: no room for the frame bitmap, using 16 MB\n");
+			bitmap_words = MM_FALLBACK_WORDS;
+		}
+	}
+	bitmap_bits = bitmap_words * 32u;
 
 	/* Everything starts in-use. */
-	for (i = 0; i < MM_BITMAP_WORDS; i++)
+	for (i = 0; i < bitmap_words; i++)
 		bitmap[i] = 0xFFFFFFFFu;
 
-	total_ram_kb = 0;
-
-	if (map->magic == E820_MAGIC && map->count <= E820_MAX_ENTRIES) {
+	if (e820_valid(map)) {
 		n = map->count;
 		for (i = 0; i < n; i++) {
 			const struct e820_entry *e = &map->entry[i];
-			if (e->type == E820_RAM && e->length != 0) {
-				unsigned long long end = e->base + e->length;
 
+			if (e->type == E820_RAM && e->length != 0)
 				mark_free(e->base, e->length);
-				if (end > MM_MAX_PHYS)
-					end = MM_MAX_PHYS;
-				if (end > e->base)
-					total_ram_kb +=
-						(unsigned int)((end - e->base) >> 10);
-			}
 		}
 	} else {
 		/* No E820 map (very old BIOS, or first kexec onto an old build):
@@ -121,63 +254,67 @@ void mm_init(void)
 	 * low RAM.  Never hand framebuffer pages to the allocator: the boot
 	 * console draws into the LFB, so an allocated page directory there
 	 * would be overwritten by its own progress line. */
-	{
-		const volatile struct vga_boot_fb_info *fb = VGA_BOOT_FB;
-		unsigned int sz;
-
-		__asm__ volatile ("" : "+r"(fb));
-		sz = (unsigned int)fb->pitch * fb->height;
-		if (!sz)
-			sz = 0x1000000;
-		if (fb->valid && fb->lfb_base && fb->lfb_base < MM_MAX_PHYS)
-			mark_used(fb->lfb_base, sz);
-	}
+	if (lfb_hi > lfb_lo)
+		mark_used(lfb_lo, lfb_hi - lfb_lo);
 
 	/* The boot path copied the command-pack rootfs into high RAM before
 	 * the kernel ran.  Keep it away from the allocator until ramfs_init()
 	 * has taken its own copy. */
-	{
-		unsigned int pa = bootinfo_pack_addr();
-		unsigned int ps = bootinfo_pack_size();
-
-		if (pa >= MM_RESERVE_LOW && pa < MM_MAX_PHYS && ps != 0 &&
-		    ps <= MM_MAX_PHYS && pa + ps > pa &&
-		    pa + ps <= MM_MAX_PHYS)
-			mark_used(pa, ps);
-	}
+	if (pack_hi > pack_lo)
+		mark_used(pack_lo, pack_hi - pack_lo);
+
+	/* The bitmap's own pages. */
+	if (bitmap_bytes)
+		mark_used((unsigned int)bitmap, bitmap_bytes);
 
 	/* Count free pages. */
 	free_pages = 0;
-	for (i = 0; i < MM_BITMAP_BITS; i++) {
-		if (!bit_test(i))
+	for (i = 0; i < bitmap_words; i++) {
+		unsigned int w = ~bitmap[i];
+
+		while (w) {
+			w &= w - 1;
 			free_pages++;
+		}
 	}
+	first_free = 0;
+
+	serial_puts(SERIAL_COM1, "mm: bitmap ");
+	serial_putdec(SERIAL_COM1, bitmap_words * 4u);
+	serial_puts(SERIAL_COM1, " B covers ");
+	serial_putdec(SERIAL_COM1, bitmap_bits / 256u);
+	serial_puts(SERIAL_COM1, " MB at ");
+	serial_puthex(SERIAL_COM1, (unsigned int)bitmap);
+	serial_puts(SERIAL_COM1, "\n");
 }
 
+/* Lowest free page.  Words below first_free are known to be full, so a
+ * steady stream of allocations does not rescan the low, busy part. */
 unsigned int mm_alloc_page(void)
 {
-	unsigned int i, w;
+	unsigned int w;
 	unsigned int flags = irq_disable_save();
 
-	for (w = 0; w < MM_BITMAP_WORDS; w++) {
-		if (bitmap[w] != 0xFFFFFFFFu) {
-			unsigned int word = bitmap[w];
-			for (i = 0; i < 32; i++) {
-				if (!(word & (1u << i))) {
-					unsigned int idx = w * 32 + i;
-					bitmap[w] |= (1u << i);
-					free_pages--;
-					irq_restore(flags);
-					return idx * PAGE_SIZE;
-				}
-			}
+	for (w = first_free; w < bitmap_words; w++) {
+		unsigned int word = bitmap[w];
+
+		if (word != 0xFFFFFFFFu) {
+			unsigned int i = (unsigned int)__builtin_ctz(~word);
+
+			bitmap[w] = word | (1u << i);
+			free_pages--;
+			first_free = w;
+			irq_restore(flags);
+			return (w * 32u + i) * PAGE_SIZE;
 		}
 	}
-
+	first_free = bitmap_words;
 	irq_restore(flags);
 	return 0;
 }
 
+/* Lowest run of n free pages.  Full words break a run and empty words
+ * extend it 32 pages at a time; only mixed words are walked bit by bit. */
 unsigned int mm_alloc_pages(unsigned int n)
 {
 	unsigned int start = 0, run = 0;
@@ -188,23 +325,47 @@ unsigned int mm_alloc_pages(unsigned int n)
 		return 0;
 
 	flags = irq_disable_save();
-	for (p = 0; p < MM_BITMAP_BITS; p++) {
+	p = first_free * 32u;
+	while (p < bitmap_bits) {
+		if ((p & 31u) == 0) {
+			unsigned int word = bitmap[p / 32u];
+
+			if (word == 0xFFFFFFFFu) {
+				run = 0;
+				p += 32;
+				continue;
+			}
+			if (word == 0) {
+				if (run == 0)
+					start = p;
+				run += 32;
+				p += 32;
+				if (run >= n)
+					break;
+				continue;
+			}
+		}
 		if (!bit_test(p)) {
 			if (run == 0)
 				start = p;
 			run++;
-			if (run == n) {
-				unsigned int i;
-				for (i = start; i < start + n; i++)
-					bit_set(i);
-				free_pages -= n;
-				irq_restore(flags);
-				return start * PAGE_SIZE;
-			}
+			p++;
+			if (run == n)
+				break;
 		} else {
 			run = 0;
+			p++;
 		}
 	}
+	if (run >= n) {
+		unsigned int i;
+
+		for (i = start; i < start + n; i++)
+			bit_set(i);
+		free_pages -= n;
+		irq_restore(flags);
+		return start * PAGE_SIZE;
+	}
 	irq_restore(flags);
 	return 0;
 }
@@ -214,13 +375,15 @@ void mm_free_page(unsigned int addr)
 	unsigned int idx = addr / PAGE_SIZE;
 	unsigned int flags;
 
-	if (addr == 0 || idx >= MM_BITMAP_BITS)
+	if (addr == 0 || idx >= bitmap_bits)
 		return;
 
 	flags = irq_disable_save();
 	if (bit_test(idx)) {
 		bit_clear(idx);
 		free_pages++;
+		if (idx / 32u < first_free)
+			first_free = idx / 32u;
 	}
 	irq_restore(flags);
 }

+ 11 - 0
kernel/paging.c

@@ -13,8 +13,19 @@
 #include "cpu.h"
 #include "mm.h"
 #include "mtrr.h"
+#include "process.h"
 #include "serial.h"
 #include "vga.h"
+#include "wm_abi.h"
+
+/* The RAM identity map is made of supervisor 4 MB PDEs; the ring-3 windows
+ * (program/stack, canvases, WM slots, heap) are user page tables above it.
+ * They must never share a PDE. */
+_Static_assert(MM_MAX_PHYS <= USER_BASE, "identity map overlaps user space");
+_Static_assert(MM_MAX_PHYS <= USER_WIN_BASE &&
+	       MM_MAX_PHYS <= WM_CONTENT_VA_BASE &&
+	       MM_MAX_PHYS <= USER_HEAP_BASE,
+	       "identity map overlaps a ring-3 window");
 
 /* CR4 bits that must not survive a BIOS protected-mode call.  A BIOS that
  * returns with CR4.PAE or EFER.LME set makes the first CR0.PG enable either