Procházet zdrojové kódy

paging: map MMIO, firmware tables and the LFB through an ioremap window

Nothing above lowmem is mapped at its physical address any more.
ioremap(phys, size, IOREMAP_MMIO|IOREMAP_RAM) returns the identity
address for lowmem (MMIO there still gets uncacheable 4 MB pages) and
otherwise a kernel address in the window [IOMAP_BASE = 0xC0000000, 4 GB):
4 KB pages, bump-allocated, never torn down.  Window page tables are
allocated on first use and copied into every live process directory by
process_sync_kernel_pde(); later processes clone them.  A range covered
by a same-type mapping reuses it, and one running past the newest
mapping grows it in place (ACPI maps a table header, then the table).
MMIO pages are uncached except those the LFB's WC MTRR covers, so a GPU
BAR containing the framebuffer and the LFB mapping always agree; this
replaces paging_lfb_refresh().

This fixes ACPI tables on ~1.25-1.6 GB machines landing on a WM content
slot, frees [USER_BASE, IOMAP_BASE) for ring-3, and drops the kmap-slot
PDE guard.  paging_map_mmio/_cr3, paging_map_ram and
process_map_mmio_all are gone.

Callers: ahci, e1000e, r8168, sdhci, iwl_trans, ehci, ohci, xhci,
ioapic, lapic, intel, radeon, nvidia/nv50, backlight, touchpad, the ATI
Mach64 register probes (they dereferenced BAR2 and the LFB aperture by
physical address), acpi and tpec (SMBIOS).  Drivers keep the physical
BAR for logs and reports.  The LFB is paging_lfb_va(): vga_fb_lfb_addr()
returns it, vga_fb_lfb_phys() the physical address; bootcon switches to
it right after paging is enabled.  NVIDIA BAR1/BAR3 are no longer mapped
(nothing dereferenced them); the stolen-VRAM alias is an IOREMAP_RAM
mapping.  peek/poke/dbg reach any physical byte through
paging_debug_map() (highmem via kmap, MMIO via ioremap on demand).

Tested in QEMU (1G/3G, 4 CPUs, e1000 DHCP, AHCI, xHCI/EHCI, ACPI above
1 GB, Cirrus, test-usb, test-uefi and test-amd at 3G) and on the AMD
970A board and the HP Chromebook.  ION and X201 still to be tested.
Johannes Zellner před 3 dny
rodič
revize
c88469eec8

+ 38 - 15
AGENTS.md

@@ -375,9 +375,8 @@ it to `0x7E00`.  The kernel region itself only has one wall (`_end` below
   directory, with 4 nesting (LIFO) slots per CPU; a highmem `kmap` disables
   preemption until `kunmap` and flushes only its local TLB entry, lowmem
   returns the identity address.  Nothing may sleep between them.
-  `paging_map_ram`, `paging_map_mmio_cr3` and the LFB mapping refuse to
-  overwrite the kmap slot.  (It is not the last lowmem slot, where firmware
-  puts the ACPI tables, nor the top PDE, where a VBE LFB may sit.)  The
+  (It is not the last lowmem slot, where firmware puts the ACPI tables,
+  nor the top PDE, where a VBE LFB may sit.)  The
   kernel reserves low 1 MB and an 8 MB VGA-shadow region.  The bitmap is
   sized to the highest E820 RAM address (32 KB at 1 GB, 96 KB at 3 GB) and
   is not in BSS: `mm_init` places it in the first lowmem RAM run at or above
@@ -408,8 +407,32 @@ it to `0x7E00`.  The kernel region itself only has one wall (`_end` below
   skip the VBE-only ATI CRTC reset / NVIDIA stolen-VRAM alias under EFI;
   `mouse.c` forces the software cursor; `acpi.c` prefers the passed RSDP and
   maps its page before validating it.
-- **Paging**: `paging.c/h` — identity-maps RAM (4 MB pages via CR4.PSE) + the
-  LFB (uncacheable) and enables CR0.PG/WP. Before the switch it clears stale
+- **Paging**: `paging.c/h` — identity-maps lowmem RAM (4 MB pages via
+  CR4.PSE) and enables CR0.PG/WP.  Nothing above lowmem is mapped at its
+  physical address: device registers, firmware tables and the LFB go
+  through **`ioremap(phys, size, IOREMAP_MMIO|IOREMAP_RAM)`**, which returns
+  the identity address for lowmem (an MMIO range there gets its 4 MB pages
+  made uncacheable, as before) and otherwise a kernel address in the
+  ioremap window `[IOMAP_BASE` = `0xC0000000`, 4 GB): 4 KB pages, bump
+  allocated, never torn down, with page tables allocated on first use and
+  their PDEs copied into every live process directory
+  (`process_sync_kernel_pde()`; later processes clone them).  A request
+  covered by a same-type mapping reuses it, and one that runs past the end
+  of the newest mapping grows it in place (ACPI maps a table header, then
+  the whole table).  MMIO pages are uncached except those the LFB's WC MTRR
+  covers, which are cacheable in every alias (so a GPU BAR containing the
+  framebuffer and the LFB mapping agree); `IOREMAP_RAM` pages (ACPI/SMBIOS
+  tables, the ION stolen-VRAM alias) are write-back.  So ring-3 space can
+  use all of `[USER_BASE, IOMAP_BASE)`; `paging.c` statically asserts that
+  no user window reaches the ioremap window.  Drivers keep the physical BAR
+  for logging and comparisons and use the returned pointer for registers;
+  `paging_lfb_va()` is the LFB's kernel address (`vga_fb_lfb_addr()`
+  returns it, `vga_fb_lfb_phys()` the physical one), and `bootcon` switches
+  to it right after paging is enabled.  `paging_debug_map()` gives
+  `peek`/`poke`/`dbg` access to any physical byte: lowmem directly, highmem
+  RAM through `kmap`, MMIO through its existing mapping or a new 4 KB
+  uncached one.  The LFB in lowmem keeps its identity mapping with
+  WC/UC 4 MB pages (`lfb_apply_pages`). Before the switch it clears stale
   translation controls a BIOS protected-mode call can leave behind
   (`CR4.PAE/PGE/LA57/PCIDE/SMEP/SMAP` and `EFER.LME`); the X201's legacy
   stage historically returned from `INT 15h AH=87h` with `CR4` set, and the
@@ -926,7 +949,7 @@ it to `0x7E00`.  The kernel region itself only has one wall (`_end` below
   from DHCP (off-subnet traffic goes via the router).
 - **WiFi (Intel 7265D/3165, the HP Chromebook)**: `iwl_trans.c/h` — PCIe
   transport after Linux iwlwifi (7000 family): PCI scan for
-  `8086:095a/095b/3165/3166`, D0, BAR0 mapped with `process_map_mmio_all`,
+  `8086:095a/095b/3165/3166`, D0, BAR0 mapped with `ioremap`,
   APM init, one legacy RX queue (256 x 4 KB RBDs), gen1 128-byte TFD queues
   with dword byte-count tables, firmware TLV parse and section DMA through
   the FH service channel (128 KB bounce, `LMPM_CHICK` for the extended
@@ -2077,7 +2100,8 @@ current address) and falls back to the static default when no server answers.
 The board also answers ARP/ICMP for its IP while idle.
 `lspci` lists every PCI function (one line each); with a filter (`lspci 8168`,
 `lspci 02`) it also prints BAR details, type and size for matching functions.
-`peek <addr> [len]` hex-dumps memory (identity map, so MMIO registers too).
+`peek <addr> [len]` hex-dumps physical memory, MMIO registers too (through
+`paging_debug_map()`: highmem via `kmap`, MMIO via its ioremap mapping).
 `poke <addr> <val>` writes one 32-bit dword and prints the read-back (for
 driving MMIO experiments from the shell).
 
@@ -2299,7 +2323,7 @@ briefly reporting disconnected.
   external USB ports.  Downstream devices are enumerated recursively; a
   full-speed device behind a USB 2.0 hub would need split transactions (not
   implemented).
-- **OHCI**: BAR0 MMIO (`paging_map_mmio()`), HCCA from the page allocator,
+- **OHCI**: BAR0 MMIO (`ioremap()`), HCCA from the page allocator,
   static 16-byte-aligned ED/TD sets. Control transfers run on the control
   list (`HcControlHeadED` + CLF) and are polled via each TD's condition code
   — a NAKed TD stays queued and is retried by the controller. Both HID EDs
@@ -2568,8 +2592,8 @@ and the WM falls back to polling (`io_keyboard_irq_masked()`).
   RAM contained; QEMU starts guest RAM cleared, real hardware does not, so a
   structure whose fields are read before being written must be cleared on
   allocation.  `clear_proc()` (`process.c`) exists for exactly this: a stale
-  `procs[i].cr3` made the boot-time MMIO mapping (`paging_map_mmio_cr3`) fault
-  on the board while QEMU was fine.  Anything between `MM_MAX_PHYS` (1 GB) and
+  `procs[i].cr3` made the boot-time MMIO mapping (then `paging_map_mmio_cr3`,
+  now `process_sync_kernel_pde`) fault on the board while QEMU was fine.  Anything between `MM_MAX_PHYS` (1 GB) and
   the user windows is *unmapped*, so a wild pointer there is an instant #PF.
 - **All drawing goes into the current render target** (the shadow buffer by
   default; `vga_target()`, see the graphics bullet); the main loop flushes the
@@ -2613,11 +2637,10 @@ and the WM falls back to polling (`io_keyboard_irq_masked()`).
    ring-3 pages, so the 1 GB lowmem zone (kernel structures, canvases,
    contiguous buffers) can still run out first.  One process is still capped
    by `USER_HEAP_MAX` (64 MB) and the 4 MB image/stack window; raising that
-   needs demand paging and moving MMIO out of the identity layout, which
-   currently maps BARs at their physical address and so blocks ring-3 space
-   near the PCI hole.  A BIOS that puts ACPI tables, MMIO or the LFB in the
-   4 MB kmap slot at `0x4FC00000` cannot have that range identity-mapped
-   (the mappers log `refusing to map over the kmap window`).
+   needs demand paging (`[USER_BASE, IOMAP_BASE)` is free for it now that
+   MMIO lives in the ioremap window).  The window is 1 GB of 4 KB pages
+   with 96 mapping records and no unmap: a driver that keeps remapping
+   would exhaust it (ioremap logs `window full` and returns 0).
 7. `display` (needs a GPU snapshot) and `lsusb` (a 45-byte wrapper around
    the USB tree printer) are the last non-privileged commands still living in
    the kernel, plus the privileged ones (`peek`, `poke`, `dbg`, `serial`,

+ 1 - 1
docs/development.html

@@ -364,7 +364,7 @@ make net-cmds                   # replace /ramfs only</code></pre>
 			<li><strong>Keys.</strong> <code>keytest</code> drains the raw scancode tap in <span class="file">io.c</span> and prints every byte with its E0-prefix state, make/break kind and decoded <code>IO_KEY_*</code>/ASCII value; Ctrl+C stops it.  It sees releases and keys the decoder ignores, so an unknown action-row key is easy to identify.</li>
 			<li><strong>Panic.</strong> A ring-0 fault dumps vector, error code, registers, CRs, EIP bytes and stack candidates to the LFB and COM1. Resolve addresses with <code>nm kernel.elf</code> / <code>addr2line -e kernel.elf</code>.</li>
 			<li><strong>Profiling.</strong> <code>gfxbench</code> times fill/copy/cell/canvas/flush; <code>top</code>, <code>sched</code>, <code>sensors</code>, <code>powerinfo</code>, <code>power</code> and <code>lspci</code>/<code>lsusb</code> expose live state.</li>
-			<li><strong>Memory.</strong> <code>peek addr len</code> hex-dumps identity-mapped memory (MMIO too); <code>poke addr val</code> writes one dword and prints the read-back.</li>
+			<li><strong>Memory.</strong> <code>peek addr len</code> hex-dumps physical memory, MMIO registers too (highmem through <code>kmap</code>, MMIO through its <code>ioremap()</code> mapping, created on demand); <code>poke addr val</code> writes one dword and prints the read-back.</li>
 		</ul>
 	</section>
 

+ 4 - 1
docs/drivers.html

@@ -109,7 +109,10 @@
 			driven at address zero.
 		</p>
 		<p>
-			MMIO regions are mapped through <code>paging_map_mmio()</code>. PCI
+			MMIO regions are mapped through <code>ioremap()</code>, which returns
+			a kernel address in the shared window at <code>0xC0000000</code>
+			(see <a href="memory.html#paging">Memory → Paging</a>); drivers keep
+			the physical BAR only for logging. PCI
 			interrupts are disabled before touching MMIO, so every driver polls
 			its own status registers. <code>lspci</code> lists every function;
 			with a filter it prints BAR type, size and address details.

+ 8 - 4
docs/memory.html

@@ -146,6 +146,7 @@
 					<tr><td><code>0x58000000</code></td><td>WM shadow mapping</td><td><code>WM_SHADOW_VA</code>; ring-3 WM only</td></tr>
 					<tr><td><code>0x59000000</code></td><td>WM content slots</td><td><code>WM_CONTENT_VA_BASE</code> + index × 4 MB (<code>WM_CONTENT_SLOT</code>)</td></tr>
 					<tr><td><code>0x60000000</code></td><td>Ring-3 anonymous heap</td><td><code>USER_HEAP_BASE</code>, up to 64 MB</td></tr>
+					<tr><td><code>0xC0000000</code></td><td>ioremap window (virtual)</td><td><code>IOMAP_BASE</code> to 4 GB: MMIO, firmware tables, LFB; kernel-only, 4 KB pages</td></tr>
 				</tbody>
 			</table>
 		</div>
@@ -156,14 +157,16 @@
 		<p>
 			<span class="file">paging.c</span> builds an identity map of RAM with
 			4 MB pages through <code>CR4.PSE</code>, adds the framebuffer's LFB
-			range, and enables <code>CR0.PG</code> and <code>CR0.WP</code>. Before
+			range (in lowmem; otherwise through the ioremap window), and enables
+			<code>CR0.PG</code> and <code>CR0.WP</code>. Before
 			switching, it clears translation controls a BIOS protected-mode call
 			can leave behind: <code>CR4.PAE/PGE/LA57/PCIDE/SMEP/SMAP</code> and
 			<code>EFER.LME</code>.
 		</p>
 		<ul>
 			<li><strong>Identity map.</strong> Physical and virtual addresses are equal below 1 GB (lowmem). Page-directory entries are shared with every process.</li>
-			<li><strong>kmap window.</strong> The 4 MB slot at <code>MM_KMAP_VA</code> (<code>0x4FC00000</code>, in the unused gap below <code>USER_WIN_BASE</code>) has one supervisor page table that every process directory shares. <code>kmap(phys)</code> returns the identity address for lowmem, and for a highmem frame takes this CPU's next slot (4 per CPU, nesting LIFO) with preemption disabled until <code>kunmap()</code>; only the local TLB entry is flushed, because no other CPU uses that slot. <code>paging_map_ram</code>, <code>paging_map_mmio_cr3</code> and the LFB mapping refuse to overwrite that slot. It is not the last lowmem slot, because firmware puts the ACPI tables at the top of RAM, and not the top of the address space, where a VBE LFB may sit.</li>
+			<li><strong>kmap window.</strong> The 4 MB slot at <code>MM_KMAP_VA</code> (<code>0x4FC00000</code>, in the unused gap below <code>USER_WIN_BASE</code>) has one supervisor page table that every process directory shares. <code>kmap(phys)</code> returns the identity address for lowmem, and for a highmem frame takes this CPU's next slot (4 per CPU, nesting LIFO) with preemption disabled until <code>kunmap()</code>; only the local TLB entry is flushed, because no other CPU uses that slot. It is not the last lowmem slot, because firmware puts the ACPI tables at the top of RAM, and not the top of the address space, where a VBE LFB may sit.</li>
+			<li><strong>ioremap window.</strong> Nothing above lowmem is mapped at its physical address. Device registers, ACPI/SMBIOS tables and the LFB go through <code>ioremap(phys, size, type)</code>: lowmem returns its identity address (an MMIO range there gets uncacheable 4 MB pages), everything else gets 4 KB pages in the kernel-only window <code>[0xC0000000, 4 GB)</code> (<code>IOMAP_BASE</code>). Its page tables are allocated on first use and copied into every live process directory (<code>process_sync_kernel_pde</code>); a range already covered by a same-type mapping is reused, and one that runs past the newest mapping grows it in place. MMIO pages are uncached except the pages the LFB's WC MTRR covers, which are cacheable in every alias; <code>IOREMAP_RAM</code> pages are write-back. Drivers keep the physical BAR for logs and use the returned pointer. <code>peek</code>, <code>poke</code> and <code>dbg</code> reach any physical byte through <code>paging_debug_map()</code>.</li>
 			<li><strong>Kernel stacks mapped everywhere.</strong> Each process gets a page directory cloned from the kernel identity map, so CR3 can switch while still running on the old kernel stack.</li>
 			<li><strong>LFB is uncacheable by default.</strong> If <code>mtrr_set_wc()</code> successfully covers the framebuffer, the LFB pages are mapped cacheable so the shadow-buffer flush can write-combine.</li>
 			<li><strong>No process isolation.</strong> There is one address space with per-process page directories only for mappings; a wild ring-0 or ring-3 pointer can reach any mapped physical page.</li>
@@ -332,7 +335,8 @@
 		<h2>Limits &amp; leaks</h2>
 		<ul>
 			<li><strong>No per-process address spaces.</strong> Paging is identity-mapped; there is no memory protection between processes.</li>
-			<li><strong>4 GB ceiling, 64 MB per process.</strong> RAM above 4 GB is ignored (it would need PAE). Highmem raises the system total, but one process still has a 64 MB heap and a 4 MB image/stack window. MMIO is still identity-mapped at its physical address, which blocks growing ring-3 space toward the PCI hole. A wild kernel pointer between the identity map and the user windows is an instant <code>#PF</code>.</li>
+			<li><strong>4 GB ceiling, 64 MB per process.</strong> RAM above 4 GB is ignored (it would need PAE). Highmem raises the system total, but one process still has a 64 MB heap and a 4 MB image/stack window; <code>[USER_BASE, IOMAP_BASE)</code> is free for demand paging. A wild kernel pointer between the identity map and the user windows is an instant <code>#PF</code>.</li>
+			<li><strong>ioremap is append-only.</strong> 1 GB of window, 96 mapping records and no unmap: a driver that remaps repeatedly would exhaust it (<code>ioremap</code> logs <code>window full</code> and returns 0).</li>
 			<li><strong>Highmem is user-only.</strong> Contiguous kernel buffers and every kernel structure stay in the 1 GB lowmem zone, so a machine with 4 GB can still run out of lowmem first.</li>
 			<li><strong>Kernel heap fragmentation.</strong> <code>kmalloc</code>'s power-of-two size classes make long-running mixed-size kernel allocation wasteful; ring-3 commands use the separate first-fit libc heap described above.</li>
 			<li><strong>No TLB shootdown.</strong> Cross-CPU mapping changes rely on not-present entries not being cached.</li>
@@ -347,7 +351,7 @@
 			<li><code>kernel/kmalloc.c</code>, <code>kernel/kmalloc.h</code> — kernel heap</li>
 			<li><code>user/lib/libc/malloc.c</code>, <code>user/lib/libc/ucmd_mem.c</code> — ring-3 heap and <code>SYS_MEM_*</code> backend</li>
 			<li><code>user/lib/libc/string.c</code>, <code>user/lib/libc/stdlib.c</code> — ring-3 libc support</li>
-			<li><code>kernel/paging.c</code>, <code>kernel/paging.h</code> — identity map, kmap window</li>
+			<li><code>kernel/paging.c</code>, <code>kernel/paging.h</code> — identity map, kmap window, ioremap window, debug physical access</li>
 			<li><code>kernel/mtrr.c</code> — write-combining MTRRs</li>
 			<li><code>kernel/process.c</code>, <code>kernel/process.h</code> — stacks and user layout</li>
 			<li><code>kernel/ramfs.c</code> — pack copy / inflate</li>

+ 4 - 0
include/phys.h

@@ -17,6 +17,10 @@
  * address space (a VBE LFB may sit at 0xFF000000). */
 #define MM_KMAP_VA       0x4FC00000u
 
+/* The ioremap window: kernel-only virtual space for MMIO, firmware tables
+ * and the LFB above lowmem.  Ring-3 space may use [USER_BASE, IOMAP_BASE). */
+#define IOMAP_BASE       0xC0000000u
+
 /* Highmem: RAM in [MM_MAX_PHYS, MM_HIGH_LIMIT) is tracked by the frame
  * allocator but never identity-mapped; it only backs ring-3 pages, which
  * the kernel reaches through the owning process's mapping or kmap().

+ 25 - 36
kernel/acpi.c

@@ -1,8 +1,8 @@
 /* Minimal ACPI RSDP/RSDT/XSDT/MADT parser.  See acpi.h.
  *
- * ACPI tables are usually below the 128 MB identity map, but on larger
- * machines they can sit above it; every table page is mapped through
- * paging_map_ram() before it is read.  Anything malformed or unsupported is
+ * ACPI tables are usually in lowmem, but on larger machines they sit above
+ * it; every table is reached through ioremap() (the identity address in
+ * lowmem, the kernel window otherwise).  Anything malformed or unsupported is
  * treated as "no SMP", never as a fatal error.
  */
 
@@ -51,9 +51,10 @@ static int acpi_ok;
  * when the firmware points at a real CMOS register. */
 static unsigned int rtc_century_reg;
 
-static void *phys_ptr(unsigned int addr)
+/* Kernel address of a firmware table range (write-back RAM mapping). */
+static void *table_ptr(unsigned int addr, unsigned int len)
 {
-	return (void *)addr;
+	return ioremap(addr, len, IOREMAP_RAM);
 }
 
 static int range_ok(unsigned int addr, unsigned int len)
@@ -63,21 +64,6 @@ static int range_ok(unsigned int addr, unsigned int len)
 	return addr + len >= addr;
 }
 
-/* Map every 4 MB page touched by [addr, addr+len). */
-static void map_table(unsigned int addr, unsigned int len)
-{
-	unsigned int p = addr & 0xFFC00000u;
-	unsigned int last = (addr + len - 1u) & 0xFFC00000u;
-
-	for (;;) {
-		paging_map_ram(p, 0x400000u);
-		if (p >= last)
-			break;
-		if (p > 0xFFC00000u - 0x400000u)
-			break;
-		p += 0x400000u;
-	}
-}
 
 static int sum_ok(const unsigned char *p, unsigned int len)
 {
@@ -114,14 +100,13 @@ static const struct acpi_rsdp *find_rsdp(void)
 	unsigned int given = bootinfo_rsdp();
 
 	/* The UEFI loader passes the RSDP address from the system table.  UEFI
-	 * places ACPI tables outside the 128 MB identity map often enough that
-	 * the scan below would never see them, so map the page first. */
+	 * places ACPI tables outside lowmem often enough that the scan below
+	 * would never see them, so map it first (4 KB covers the largest
+	 * length rsdp_valid() accepts). */
 	if (given && given < 0xFFF00000u) {
-		const struct acpi_rsdp *r;
+		const struct acpi_rsdp *r = table_ptr(given, 0x1000);
 
-		paging_map_ram(given & 0xFFC00000u, 0x800000u);
-		r = (const struct acpi_rsdp *)given;
-		if (rsdp_valid(r)) {
+		if (r && rsdp_valid(r)) {
 			serial_puts(SERIAL_COM1, "acpi: EFI RSDP at ");
 			serial_puthex(SERIAL_COM1, given);
 			serial_puts(SERIAL_COM1, "\n");
@@ -169,13 +154,15 @@ static const struct acpi_sdt *find_madt(const struct acpi_rsdp *rsdp)
 	}
 	if (!addr || !range_ok(addr, sizeof(struct acpi_sdt)))
 		return 0;
-	map_table(addr, sizeof(struct acpi_sdt));
-	sdt = (const struct acpi_sdt *)phys_ptr(addr);
+	sdt = table_ptr(addr, sizeof(struct acpi_sdt));
+	if (!sdt)
+		return 0;
 	len = sdt->length;
-	if (len < sizeof(struct acpi_sdt) || len > 0x100000u)
+	if (len < sizeof(struct acpi_sdt) || len > 0x100000u ||
+	    !range_ok(addr, len))
 		return 0;
-	map_table(addr, len);
-	if (!sum_ok((const unsigned char *)sdt, len))
+	sdt = table_ptr(addr, len);
+	if (!sdt || !sum_ok((const unsigned char *)sdt, len))
 		return 0;
 
 	for (i = sizeof(struct acpi_sdt); i + step <= len; i += step) {
@@ -194,13 +181,15 @@ static const struct acpi_sdt *find_madt(const struct acpi_rsdp *rsdp)
 		}
 		if (!taddr || !range_ok(taddr, sizeof(struct acpi_sdt)))
 			continue;
-		map_table(taddr, sizeof(struct acpi_sdt));
-		t = (const struct acpi_sdt *)phys_ptr(taddr);
+		t = table_ptr(taddr, sizeof(struct acpi_sdt));
+		if (!t)
+			continue;
 		tlen = t->length;
-		if (tlen < sizeof(struct acpi_sdt) || tlen > 0x100000u)
+		if (tlen < sizeof(struct acpi_sdt) || tlen > 0x100000u ||
+		    !range_ok(taddr, tlen))
 			continue;
-		map_table(taddr, tlen);
-		if (!sum_ok((const unsigned char *)t, tlen))
+		t = table_ptr(taddr, tlen);
+		if (!t || !sum_ok((const unsigned char *)t, tlen))
 			continue;
 		if (t->signature[0] == 'F' && t->signature[1] == 'A' &&
 		    t->signature[2] == 'C' && t->signature[3] == 'P') {

+ 3 - 2
kernel/ahci.c

@@ -255,8 +255,9 @@ int ahci_init(void)
 	cmd = pci_cfg_read32(bus, dev, func, 0x04);
 	pci_cfg_write32(bus, dev, func, 0x04, cmd | 0x0006u);  /* mem + bus master */
 
-	process_map_mmio_all(bar, 0x2000);
-	ahci.abar = bar;
+	ahci.abar = (unsigned int)ioremap(bar, 0x2000, IOREMAP_MMIO);
+	if (!ahci.abar)
+		return -1;
 
 	cap = rd(H_CAP);
 	if (cap == 0 || cap == 0xFFFFFFFFu) {

+ 14 - 6
kernel/ati.c

@@ -16,6 +16,7 @@
  * the lower 16 bits match the PCI device ID.
  */
 
+#include "paging.h"
 #include "ati.h"
 #include "pci.h"
 #include "radeon.h"
@@ -44,6 +45,9 @@ static int ati_device_is_radeon(unsigned int device_id)
 	}
 }
 
+/* Map the 1 KB register block at physical `pa` and probe it. */
+static int try_mmio_at(unsigned int pa, unsigned int device_id);
+
 static int try_mmio(volatile unsigned int *mmio, unsigned int device_id)
 {
 	unsigned int chip_id = mmio[ATI_CONFIG_CHIP_ID_OFS / 4];
@@ -65,8 +69,7 @@ static void ati_mach64_fixup(unsigned int bus, unsigned int dev,
 
 	/* Method 1: BAR2 (separate MMIO aperture) */
 	if (!(bar2 & 1) && (bar2 & 0xFFFFFFF0)) {
-		if (try_mmio((volatile unsigned int *)(bar2 & 0xFFFFFFF0),
-			     device_id)) {
+		if (try_mmio_at(bar2 & 0xFFFFFFF0, device_id)) {
 			mmio_method = 1;
 			return;
 		}
@@ -75,13 +78,11 @@ static void ati_mach64_fixup(unsigned int bus, unsigned int dev,
 	/* Method 2-3: MMIO at end of LFB aperture (264VT/GT) */
 	if (!(bar0 & 1) && (bar0 & 0xFFFFFFF0)) {
 		unsigned int lfb = bar0 & 0xFFFFFFF0;
-		if (try_mmio((volatile unsigned int *)(lfb + 0x7FFC00),
-			     device_id)) {
+		if (try_mmio_at(lfb + 0x7FFC00, device_id)) {
 			mmio_method = 2;
 			return;
 		}
-		if (try_mmio((volatile unsigned int *)(lfb + 0x3FFC00),
-			     device_id)) {
+		if (try_mmio_at(lfb + 0x3FFC00, device_id)) {
 			mmio_method = 3;
 			return;
 		}
@@ -103,6 +104,13 @@ static void ati_mach64_fixup(unsigned int bus, unsigned int dev,
 	}
 }
 
+static int try_mmio_at(unsigned int pa, unsigned int device_id)
+{
+	volatile unsigned int *mmio = ioremap(pa, 0x400, IOREMAP_MMIO);
+
+	return mmio ? try_mmio(mmio, device_id) : 0;
+}
+
 void ati_init(unsigned int bus, unsigned int dev)
 {
 	unsigned int id = pci_cfg_read32(bus, dev, 0, 0x00);

+ 9 - 4
kernel/backlight.c

@@ -37,7 +37,8 @@
 #define BLC_STEP_PERCENT	10
 #define BLC_DEFAULT_PERIOD	6000u	/* 19.2 MHz / (200 Hz * 16) */
 
-static unsigned int blc_bar;
+static unsigned int blc_bar;      /* physical BAR (reported) */
+static unsigned int blc_regs;     /* its kernel mapping */
 static unsigned int blc_pipe;
 static unsigned int blc_period;
 static unsigned int blc_duty;
@@ -48,12 +49,12 @@ static unsigned int blc_seq;	/* bumped on every user-visible change */
 
 static unsigned int blc_read(unsigned int reg)
 {
-	return *(volatile unsigned int *)(blc_bar + reg);
+	return *(volatile unsigned int *)(blc_regs + reg);
 }
 
 static void blc_write(unsigned int reg, unsigned int val)
 {
-	*(volatile unsigned int *)(blc_bar + reg) = val;
+	*(volatile unsigned int *)(blc_regs + reg) = val;
 }
 
 void backlight_init(unsigned int bus, unsigned int dev, unsigned int device_id)
@@ -83,7 +84,11 @@ void backlight_init(unsigned int bus, unsigned int dev, unsigned int device_id)
 
 	/* The backlight registers live below 2 MB in the display MMIO BAR. */
 	blc_bar = bar0 & 0xFFFFFFF0u;
-	paging_map_mmio(blc_bar, 0x200000);
+	blc_regs = (unsigned int)ioremap(blc_bar, 0x200000, IOREMAP_MMIO);
+	if (!blc_regs) {
+		blc_bar = 0;
+		return;
+	}
 
 	for (i = 0; i < 2; i++) {
 		ctl2[i] = blc_read(VLV_BLC_PWM_CTL2(i));

+ 6 - 0
kernel/bootcon.c

@@ -320,3 +320,9 @@ void bootcon_fb_info(void)
 	serial_puthex(SERIAL_COM1, fb_base);
 	serial_puts(SERIAL_COM1, "\n");
 }
+
+void bootcon_set_lfb(void *va)
+{
+	if (va)
+		fb_lfb = (volatile unsigned char *)va;
+}

+ 3 - 0
kernel/bootcon.h

@@ -18,6 +18,9 @@ void bootcon_init(void);
 int  bootcon_ok(void);
 void bootcon_set_enabled(int on);
 void bootcon_puts(const char *s);
+/* Switch drawing to the LFB's kernel address once paging is on (the LFB may
+ * live in the ioremap window, not at its physical address); 0 is ignored. */
+void bootcon_set_lfb(void *va);
 void bootcon_dec(unsigned int v);
 void bootcon_hex(unsigned int v);
 /* One line with the cached mode geometry and LFB address. */

+ 23 - 3
kernel/dbg.c

@@ -13,6 +13,7 @@
  */
 
 #include "dbg.h"
+#include "paging.h"
 #include "serial.h"
 
 #define DBG_LINE_MAX 64
@@ -131,8 +132,18 @@ static int dbg_command(const char *line)
 			serial_puthex(SERIAL_COM1, addr);
 			serial_putc(SERIAL_COM1, ':');
 			for (i = 0; i < len; i++) {
+				volatile unsigned char *b;
+				void *km;
+
 				serial_putc(SERIAL_COM1, ' ');
-				dbg_putbyte(((volatile unsigned char *)addr)[i]);
+				b = paging_debug_map(addr + i, &km);
+				if (!b) {
+					serial_puts(SERIAL_COM1, "??");
+					continue;
+				}
+				dbg_putbyte(*b);
+				if (km)
+					kunmap(km);
 			}
 			serial_putc(SERIAL_COM1, '\n');
 		}
@@ -144,8 +155,17 @@ static int dbg_command(const char *line)
 		} else {
 			unsigned int addr = args[0];
 			unsigned int i;
-			for (i = 1; i < nargs; i++)
-				((volatile unsigned char *)addr)[i - 1] = (unsigned char)args[i];
+			for (i = 1; i < nargs; i++) {
+				void *km;
+				volatile unsigned char *b =
+					paging_debug_map(addr + i - 1, &km);
+
+				if (!b)
+					break;
+				*b = (unsigned char)args[i];
+				if (km)
+					kunmap(km);
+			}
 			serial_puts(SERIAL_COM1, "ok\n");
 		}
 		return 0;

+ 2 - 2
kernel/display.c

@@ -71,7 +71,7 @@ void program_display_main(void)
 	put_hex((unsigned int)(unsigned long)vga_fb_ptr());
 	io_putchar('\n');
 	io_putstr("LFB phys ");
-	put_hex((unsigned int)vga_fb_lfb_addr());
+	put_hex(vga_fb_lfb_phys());
 	io_putchar('\n');
 	io_putstr("rendering: ");
 	io_putstr(vga_shadow_is_active() ? "shadow buffer" : "direct");
@@ -113,7 +113,7 @@ void program_display_main(void)
 		}
 	}
 	if (gpu_vendor() == GPU_VENDOR_NVIDIA) {
-		unsigned int lfb = (unsigned int)vga_fb_lfb_addr();
+		unsigned int lfb = vga_fb_lfb_phys();
 		unsigned int bar3 = nvidia_bar3();
 
 		io_putstr("NVIDIA ");

+ 6 - 6
kernel/e1000e.c

@@ -7,10 +7,9 @@
  * MDIC.  Everything is polled: interrupts are masked and the ring status
  * bits are checked directly, like the other nebos NIC drivers.
  *
- * Mapped BAR0 (128 KB on the 82577LM) is above the 128 MB identity map, so
- * process_map_mmio_all() installs it in every process page directory: kernel
- * code runs with the current process's CR3 and the network syscalls follow
- * the user process.
+ * BAR0 (128 KB on the 82577LM) is mapped with ioremap(); the window is
+ * shared by every process directory, because kernel code runs with the
+ * current process's CR3 and the network syscalls follow the user process.
  */
 #include "e1000e.h"
 #include "bootcon.h"
@@ -1601,8 +1600,9 @@ static int e1000e_init(void)
 		return -1;
 	}
 
-	process_map_mmio_all(bar_base, bar_size);
-	mmio = (unsigned char *)(unsigned long)bar_base;
+	mmio = ioremap(bar_base, bar_size, IOREMAP_MMIO);
+	if (!mmio)
+		return -1;
 	if (r32(REG_STATUS) == 0xFFFFFFFFu) {
 		dbg("MMIO read failed");
 		mmio = 0;

+ 8 - 4
kernel/ehci.c

@@ -165,7 +165,8 @@ struct ehci_int {
 };
 
 struct ehci_ctrl {
-	unsigned int base;
+	unsigned int base;      /* kernel mapping of the register BAR */
+	unsigned int phys;      /* the BAR itself (reported by lsusb) */
 	unsigned int cap_len;
 	unsigned int bus, dev, func;
 	unsigned int n_ports;
@@ -1197,12 +1198,15 @@ static int ehci_setup_one(unsigned int bus, unsigned int dev,
 		return -1;
 	c = &ctrls[n_ctrls];
 	zero_ctrl(c);
-	c->base = bar;
+	c->phys = bar;
 	c->bus = bus;
 	c->dev = dev;
 	c->func = func;
 
-	paging_map_mmio(bar, 0x1000);
+	bar = (unsigned int)ioremap(bar, 0x1000, IOREMAP_MMIO);
+	if (!bar)
+		return -1;
+	c->base = bar;
 
 	if (rd32(bar) == 0xFFFFFFFFu)
 		return -1;
@@ -1324,7 +1328,7 @@ static void ehci_info(unsigned int ctrl, unsigned int *bus, unsigned int *dev,
 	*bus = c->bus;
 	*dev = c->dev;
 	*func = c->func;
-	*base = c->base;
+	*base = c->phys;
 	*mmio = 1;
 }
 

+ 6 - 4
kernel/intel.c

@@ -62,7 +62,7 @@
 #define GTT_PTE_VALID    0x1u
 #define GTT_PTE_UNCACHED 0x0u
 
-static unsigned int intel_mmio;
+static unsigned int intel_mmio;   /* kernel mapping of BAR0 */
 static volatile unsigned int *intel_gtt;
 static unsigned int intel_gtt_entries;
 static unsigned int intel_gtt_limit;
@@ -272,8 +272,10 @@ void intel_init(unsigned int bus, unsigned int dev, unsigned int device_id)
 	if (bar0_size > 0x1000000)
 		bar0_size = 0x1000000;
 
-	intel_mmio = intel_bar_base(bar0);
-	paging_map_mmio(intel_mmio, bar0_size);
+	intel_mmio = (unsigned int)ioremap(intel_bar_base(bar0), bar0_size,
+					   IOREMAP_MMIO);
+	if (!intel_mmio)
+		return;
 
 	pgetbl = mmio_read(INTEL_PGETBL_CTL);
 	if (pgetbl == 0xFFFFFFFFu) {
@@ -295,7 +297,7 @@ void intel_init(unsigned int bus, unsigned int dev, unsigned int device_id)
 	intel_gtt_limit = intel_gtt_entries;
 	if (bar2_size >= 0x1000000 && bar2_size / 4096 < intel_gtt_limit)
 		intel_gtt_limit = bar2_size / 4096;
-	lfb = vga_fb_lfb_addr();
+	lfb = vga_fb_lfb_phys();
 	if (bar2_base && lfb >= bar2_base)
 		lfb_gtt = (lfb - bar2_base) & 0xFFFFF000u;
 

+ 3 - 2
kernel/ioapic.c

@@ -49,8 +49,9 @@ void ioapic_mask_all(void)
 
 		if (!addr)
 			continue;
-		paging_map_mmio(addr, 0x1000);
-		base = (volatile unsigned int *)addr;
+		base = ioremap(addr, 0x1000, IOREMAP_MMIO);
+		if (!base)
+			continue;
 
 		ver = ioapic_read(base, IOAPIC_VER);
 		if (ver == 0xFFFFFFFFu)

+ 5 - 4
kernel/iwl_trans.c

@@ -505,10 +505,11 @@ int iwl_probe(void)
 						(cmd & 0xffffu) | 0x0006u | 0x0400u);
 				/* PCI_CFG_RETRY_TIMEOUT (0x41) = 0, like Linux. */
 				pci_cfg_write8(bus, dev, func, 0x41, 0);
-				/* Map the 8 KB CSR window in every address space
-				 * (probe may run after processes exist). */
-				process_map_mmio_all(bar, 0x2000);
-				S->mmio = (volatile unsigned char *)bar;
+				/* The 8 KB CSR window (the ioremap window is
+				 * shared by every address space). */
+				S->mmio = ioremap(bar, 0x2000, IOREMAP_MMIO);
+				if (!S->mmio)
+					return -1;
 				iwl_wr(CSR_INT_MASK, 0);
 				S->hw_rev = iwl_rd(CSR_HW_REV);
 				serial_puts(SERIAL_COM1, "iwl: ");

+ 2 - 1
kernel/lapic.c

@@ -1,6 +1,7 @@
 /* Local APIC driver.  The MMIO window must already be mapped by smp.c. */
 
 #include "lapic.h"
+#include "paging.h"
 #include "timer.h"
 
 #define LAPIC_ID        0x020
@@ -49,7 +50,7 @@ static inline void wrmsr(unsigned int msr, unsigned long long value)
 void lapic_set_base(unsigned int base)
 {
 	lapic_phys = base;
-	lapic_mmio = (volatile unsigned int *)base;
+	lapic_mmio = ioremap(base, 0x1000, IOREMAP_MMIO);
 }
 
 unsigned int lapic_base(void)

+ 1 - 1
kernel/nv50.c

@@ -82,7 +82,7 @@ void nv50_probe(void)
 
 	if (!nvidia_available())
 		return;
-	nv_mmio = nvidia_bar0();
+	nv_mmio = nvidia_regs();
 
 	log_hex("nv50: pdobj ", mmio_rd(NV_PDISP_OBJECTS));
 	log_hex(" cursor2 ", mmio_rd(NV_PDISP_CURSOR_CTRL2(0)));

+ 23 - 13
kernel/nvidia.c

@@ -32,6 +32,7 @@ static unsigned int nv_present;      /* BAR0 mapped + PMC_BOOT_0 sane */
 static unsigned int nv_boot0;
 static unsigned int nv_chipset;
 static unsigned int nv_bar0;
+static unsigned int nv_regs;   /* kernel mapping of BAR0 */
 static unsigned int nv_bar1;
 static unsigned int nv_bar1_size;
 static unsigned int nv_bar3;
@@ -115,12 +116,10 @@ void nvidia_init(unsigned int bus, unsigned int dev, unsigned int device_id)
 	bar0 = pci_cfg_read32(bus, dev, 0, 0x10);
 	if (!(bar0 & 1) && (bar0 & 0xFFFFFFF0)) {
 		nv_bar0 = bar0 & 0xFFFFFFF0;
-		/* Map into every process page directory: gpu_init() runs
-		 * before the first process exists, but drivers may run in a
-		 * terminal context later (same reason r8168 does this). */
-		process_map_mmio_all(nv_bar0, NV_BAR0_SIZE);
+		nv_regs = (unsigned int)ioremap(nv_bar0, NV_BAR0_SIZE,
+						IOREMAP_MMIO);
 
-		boot0 = *(volatile unsigned int *)nv_bar0;
+		boot0 = nv_regs ? *(volatile unsigned int *)nv_regs : 0;
 		if (boot0 != 0 && boot0 != 0xFFFFFFFF) {
 			nv_boot0 = boot0;
 			nv_chipset = (boot0 >> 20) & 0xFF;
@@ -139,9 +138,9 @@ void nvidia_init(unsigned int bus, unsigned int dev, unsigned int device_id)
 			nv_bar1 = b1 & 0xFFFFFFF0;
 		if (nv_bar1) {
 			nv_bar1_size = nv_bar_size(bus, dev, 0x14, b1 & 4);
-			if (nv_bar1_size > 0x10000000u)   /* cap the mapping */
+			if (nv_bar1_size > 0x10000000u)   /* cap the size */
 				nv_bar1_size = 0x10000000u;
-			process_map_mmio_all(nv_bar1, nv_bar1_size);
+			/* Not mapped: nothing dereferences BAR1. */
 		}
 
 		b3 = pci_cfg_read32(bus, dev, 0, 0x1C);
@@ -152,7 +151,8 @@ void nvidia_init(unsigned int bus, unsigned int dev, unsigned int device_id)
 			nv_bar3_size = nv_bar_size(bus, dev, 0x1C, b3 & 4);
 			if (nv_bar3_size > 0x04000000u)
 				nv_bar3_size = 0x04000000u;
-			process_map_mmio_all(nv_bar3, nv_bar3_size);
+			/* Not mapped: the VBE LFB inside it has its own
+			 * mapping (paging_lfb_va()). */
 		}
 	}
 
@@ -162,8 +162,8 @@ void nvidia_init(unsigned int bus, unsigned int dev, unsigned int device_id)
 	 * the display engine's VRAM address for the LFB. */
 	if (nv_present && (nv_chipset == 0xaa || nv_chipset == 0xac ||
 			   nv_chipset == 0xaf)) {
-		unsigned int vb = *(volatile unsigned int *)(nv_bar0 + 0x100E10);
-		unsigned int vs = *(volatile unsigned int *)(nv_bar0 + 0x100E14);
+		unsigned int vb = *(volatile unsigned int *)(nv_regs + 0x100E10);
+		unsigned int vs = *(volatile unsigned int *)(nv_regs + 0x100E14);
 
 		nv_vram_base = vb << 12;
 		nv_vram_size = vs << 12;
@@ -225,6 +225,11 @@ unsigned int nvidia_bar0(void)
 	return nv_bar0;
 }
 
+unsigned int nvidia_regs(void)
+{
+	return nv_regs;
+}
+
 unsigned int nvidia_bar1(void)
 {
 	return nv_bar1;
@@ -266,17 +271,22 @@ unsigned int nvidia_lfb_in_vram(void)
 	static const unsigned int mark[4] = { 0xA5C3F00Du, 0x5A3C0FF0u,
 					      0xDEADBEEFu, 0x12345678u };
 	unsigned int vram = nv_vram_base, vsize = nv_vram_size;
-	unsigned int lfb = (unsigned int)vga_fb_lfb_addr();
+	unsigned int lfb_phys = vga_fb_lfb_phys();
+	unsigned int lfb = (unsigned int)vga_fb_lfb_addr();   /* mapped */
 	unsigned int off, i, hit = 0;
 
 	if (!vram || vsize < 0x01000000u || !nv_bar3 || !nv_bar3_size)
 		return 0;
-	if (lfb < nv_bar3 || lfb > nv_bar3 + nv_bar3_size - 0x1000u)
+	if (lfb_phys < nv_bar3 || lfb_phys > nv_bar3 + nv_bar3_size - 0x1000u)
 		return 0;          /* the VBE LFB is not in the VRAM aperture */
 	if (nv_bar3_size < 0x00801000u)
 		return 0;          /* no room for a marker past the window */
 
-	paging_map_ram(vram, vsize);
+	/* The stolen VRAM is DRAM: map it write-back (the WC MTRR the caller
+	 * adds over it makes the effective type WC). */
+	vram = (unsigned int)ioremap(vram, vsize, IOREMAP_RAM);
+	if (!vram)
+		return 0;
 
 	/* Marker at LFB + 8 MB: past any 1920x1080x32 frame. */
 	{

+ 4 - 1
kernel/nvidia.h

@@ -25,6 +25,8 @@ unsigned int nvidia_chipset(void);
 /* BAR0 register aperture / BAR1 VRAM window / BAR3 VBE window bases
  * (0 = absent) and their probed sizes. */
 unsigned int nvidia_bar0(void);
+/* Kernel mapping of BAR0 (the register aperture), 0 when absent. */
+unsigned int nvidia_regs(void);
 unsigned int nvidia_bar1(void);
 unsigned int nvidia_bar1_size(void);
 unsigned int nvidia_bar3(void);
@@ -40,7 +42,8 @@ unsigned int nvidia_vram_size(void);
  * therefore much faster to write than the PCIe aperture.  Locates the
  * framebuffer in the stolen VRAM by writing a marker through the LFB and
  * scanning the (now cacheably mapped) VRAM at page granularity.  Returns the
- * framebuffer's address in the stolen VRAM, or 0 when not applicable. */
+ * kernel (ioremap) address of the framebuffer in the stolen VRAM, or 0 when
+ * not applicable. */
 unsigned int nvidia_lfb_in_vram(void);
 
 #endif

+ 3 - 3
kernel/ohci.c

@@ -297,14 +297,14 @@ static int ohci_setup_one(unsigned int bus, unsigned int dev, unsigned int func)
 	pci_cfg_write16(bus, dev, func, 0xC0, 0);
 
 	c = &dma->ctrls[n_ctrls];
-	c->regs = (volatile unsigned int *)bar;
+	c->regs = ioremap(bar, 0x1000, IOREMAP_MMIO);
+	if (!c->regs)
+		return -1;
 	c->bus = bus;
 	c->dev = dev;
 	c->func = func;
 	c->bar = bar;
 
-	paging_map_mmio(bar, 0x1000);
-
 	if (rd(c, R_REVISION) == 0xFFFFFFFFu || rd(c, R_REVISION) == 0) {
 		serial_puts(SERIAL_COM1, "ohci: no register response (rev ");
 		serial_puthex(SERIAL_COM1, rd(c, R_REVISION));

+ 218 - 45
kernel/paging.c

@@ -38,15 +38,26 @@ _Static_assert(MM_KMAP_VA >= USER_STACK_TOP &&
 static unsigned int *kmap_pt;
 static unsigned int kmap_depth[MAX_CPUS];
 
-/* Every PDE writer below leaves the kmap slot alone: a BIOS-placed ACPI
- * table, MMIO BAR or LFB in that 4 MB range cannot be identity-mapped. */
-static int kmap_pde_guard(unsigned int i)
-{
-	if (i != KMAP_PDE)
-		return 0;
-	serial_puts(SERIAL_COM1, "paging: refusing to map over the kmap window\n");
-	return 1;
-}
+/* ioremap window: [IOMAP_BASE, 4 GB), kernel-only, 4 KB pages.  Its page
+ * tables are allocated on first use and their PDEs copied into every live
+ * process directory (process_sync_kernel_pde), so nothing is mapped at its
+ * physical address above lowmem and ring-3 space is free up to IOMAP_BASE. */
+_Static_assert(IOMAP_BASE >= USER_HEAP_BASE + USER_HEAP_MAX &&
+	       IOMAP_BASE >= WM_CONTENT_VA_BASE +
+			     WM_CONTENT_SLOT * WM_MAX_WINDOWS &&
+	       (IOMAP_BASE & 0x3FFFFFu) == 0,
+	       "ioremap window overlaps ring-3 space");
+#define IOMAP_MAX   96
+struct iomap {
+	unsigned int phys;   /* page-aligned */
+	unsigned int size;   /* page multiple */
+	unsigned int va;
+	int type;
+};
+static struct iomap iomaps[IOMAP_MAX];
+static unsigned int n_iomaps;
+static unsigned int iomap_next = IOMAP_BASE;  /* bump pointer; 0 = full */
+static void *lfb_va;
 
 /* CR4 bits that must not survive a BIOS protected-mode call.  A BIOS that
  * returns with CR4.PAE or EFER.LME set makes the first CR0.PG enable either
@@ -138,8 +149,9 @@ void paging_prepare_cpu(void)
 static unsigned int lfb_wc_base, lfb_wc_size;
 static int lfb_wc_ok;
 
-/* Write the LFB PDEs: WC-covered pages cacheable (effective type WC), the
- * rest uncacheable. */
+/* Write the LFB PDEs for a framebuffer in lowmem (stolen system RAM below
+ * 1 GB): WC-covered pages cacheable (effective type WC), the rest
+ * uncacheable.  An LFB above lowmem goes through ioremap() instead. */
 static void lfb_apply_pages(unsigned int *pd)
 {
 	const volatile struct vga_boot_fb_info *fb = VGA_BOOT_FB;
@@ -161,8 +173,6 @@ static void lfb_apply_pages(unsigned int *pd)
 		unsigned int page = i << 22;
 		unsigned int flags = PDE_PS | PDE_WRITABLE | PDE_PRESENT;
 
-		if (kmap_pde_guard(i))
-			continue;
 		if (!lfb_wc_ok || page < lfb_wc_base ||
 		    page + 0x400000 > lfb_wc_base + lfb_wc_size)
 			flags |= PDE_PCD;
@@ -170,34 +180,15 @@ static void lfb_apply_pages(unsigned int *pd)
 	}
 }
 
-/* Re-apply the framebuffer mapping; call after any MMIO mapping that may have
- * covered the same addresses (the NVIDIA BAR3 aperture includes the VBE LFB). */
-void paging_lfb_refresh(void)
-{
-	if (!lfb_wc_ok || !page_dir)
-		return;
-	lfb_apply_pages(page_dir);
-	serial_puts(SERIAL_COM1, "paging: LFB write-combining re-applied\n");
-}
-
 /* 1 when the LFB is mapped for write-combining (WC MTRR + cacheable pages). */
 int paging_lfb_wc(void)
 {
 	return lfb_wc_ok;
 }
 
-void paging_map_ram(unsigned int base, unsigned int size)
+void *paging_lfb_va(void)
 {
-	unsigned int p0 = base >> 22;
-	unsigned int p1 = (base + size + 0x3FFFFFu) >> 22;
-	unsigned int i;
-
-	if (!page_dir)
-		return;
-	for (i = p0; i < p1 && i < 1024; i++)
-		if (!kmap_pde_guard(i))
-			page_dir[i] = (i << 22) | PDE_PS | PDE_WRITABLE |
-				      PDE_PRESENT;
+	return lfb_va;
 }
 
 void paging_init(void)
@@ -269,7 +260,18 @@ void paging_init(void)
 		lfb_wc_ok = mtrr_set_wc(fb->lfb_base, want,
 					&lfb_wc_base, &lfb_wc_size);
 		bootcon_puts("paging: map\n");
-		lfb_apply_pages(page_dir);
+		if ((unsigned long long)fb->lfb_base + LFB_MAP_SIZE <= MM_MAX_PHYS) {
+			lfb_apply_pages(page_dir);
+			lfb_va = (void *)fb->lfb_base;
+		} else {
+			unsigned long long end = (unsigned long long)fb->lfb_base +
+						 LFB_MAP_SIZE;
+			unsigned int sz = end > 0x100000000ull
+				? (unsigned int)(0x100000000ull - fb->lfb_base)
+				: LFB_MAP_SIZE;
+
+			lfb_va = ioremap(fb->lfb_base, sz, IOREMAP_MMIO);
+		}
 		/* Tell the truth about VRAM writes: with cacheable pages the WC
 		 * MTRR applies (fast bursts); otherwise every dword is a separate
 		 * uncached write. */
@@ -295,6 +297,8 @@ void paging_init(void)
 		: "r"((unsigned int)page_dir)
 		: "eax", "memory");
 
+	/* The boot console drew through the physical LFB until now. */
+	bootcon_set_lfb(lfb_va);
 	bootcon_puts("paging: on\n");
 	serial_puts(SERIAL_COM1, "paging: enabled (identity map)\n");
 }
@@ -453,22 +457,159 @@ void paging_release_user_range(unsigned int cr3, unsigned int va,
 
 /* Identity-map an MMIO range uncacheable into one page directory (4 MB pages;
  * the range is rounded up to a whole number of them). */
-void paging_map_mmio_cr3(unsigned int cr3, unsigned int phys_base, unsigned int size)
+/* MMIO inside lowmem (a BAR the firmware put below 1 GB): make the 4 MB
+ * identity pages uncacheable, as the identity mapper always did, in the
+ * kernel directory and every process copy. */
+static void lowmem_uncached(unsigned int phys, unsigned int size)
 {
-	unsigned int *pd = (unsigned int *)cr3;
-	unsigned int p0 = phys_base >> 22;
-	unsigned int p1 = (phys_base + size + 0x3FFFFFu) >> 22;
+	unsigned int p0 = phys >> 22;
+	unsigned int p1 = (unsigned int)(((unsigned long long)phys + size +
+					  0x3FFFFFu) >> 22);
 	unsigned int i;
 
-	for (i = p0; i < p1 && i < 1024; i++)
-		if (!kmap_pde_guard(i))
-			pd[i] = (i << 22) | PDE_PS | PDE_PCD |
-			PDE_WRITABLE | PDE_PRESENT;
+	for (i = p0; i < p1 && i < MM_MAX_PHYS / 0x400000u; i++) {
+		if (!(page_dir[i] & PDE_PS))
+			continue;   /* split PDE 0 keeps its own page types */
+		page_dir[i] = (i << 22) | PDE_PS | PDE_PCD | PDE_WRITABLE |
+			      PDE_PRESENT;
+		process_sync_kernel_pde(i);
+	}
 }
 
-void paging_map_mmio(unsigned int phys_base, unsigned int size)
+/* Page type for one 4 KB window page: RAM is write-back; MMIO is uncached,
+ * except pages the LFB's WC MTRR covers, which are cacheable (effective
+ * type WC) in every alias, so a GPU BAR that contains the framebuffer and
+ * the framebuffer mapping itself never disagree. */
+static unsigned int iomap_pte_flags(unsigned int phys, int type)
 {
-	paging_map_mmio_cr3((unsigned int)page_dir, phys_base, size);
+	unsigned int f = PTE_PRESENT | PTE_WRITABLE;
+
+	if (type == IOREMAP_RAM)
+		return f;
+	if (lfb_wc_ok && phys >= lfb_wc_base &&
+	    (unsigned long long)phys + PAGE_SIZE <=
+	    (unsigned long long)lfb_wc_base + lfb_wc_size)
+		return f;
+	return f | PDE_PCD;
+}
+
+static int iomap_set(unsigned int va, unsigned int phys, unsigned int flags)
+{
+	unsigned int pde = va >> 22;
+	unsigned int *pt;
+
+	if (!(page_dir[pde] & PDE_PRESENT)) {
+		unsigned int t = mm_alloc_page();
+		unsigned int i;
+
+		if (!t)
+			return 0;
+		pt = (unsigned int *)t;
+		for (i = 0; i < 1024; i++)
+			pt[i] = 0;
+		page_dir[pde] = t | PDE_WRITABLE | PDE_PRESENT;
+		process_sync_kernel_pde(pde);
+	} else {
+		pt = (unsigned int *)(page_dir[pde] & 0xFFFFF000u);
+	}
+	pt[(va >> 12) & 0x3FFu] = (phys & 0xFFFFF000u) | flags;
+	return 1;
+}
+
+void *ioremap(unsigned int phys, unsigned int size, int type)
+{
+	unsigned long long end;
+	unsigned int p0, bytes, va, i;
+
+	if (size == 0)
+		size = 1;
+	end = (unsigned long long)phys + size;
+	if (end > 0x100000000ull || !page_dir)
+		return 0;
+
+	/* Lowmem is identity-mapped already. */
+	if (end <= MM_MAX_PHYS) {
+		if (type == IOREMAP_MMIO)
+			lowmem_uncached(phys, size);
+		return (void *)phys;
+	}
+
+	/* Reuse a mapping of the same type that already covers the range. */
+	for (i = 0; i < n_iomaps; i++) {
+		const struct iomap *m = &iomaps[i];
+
+		if (m->type == type && phys >= m->phys &&
+		    end <= (unsigned long long)m->phys + m->size)
+			return (void *)(m->va + (phys - m->phys));
+	}
+
+	/* Grow the newest mapping in place when the request starts inside it
+	 * and only runs past its end (a table header mapped first, then the
+	 * whole table): its pages are the last ones in the window. */
+	if (n_iomaps && iomap_next) {
+		struct iomap *m = &iomaps[n_iomaps - 1];
+		unsigned long long mend = (unsigned long long)m->phys + m->size;
+		unsigned long long need = (end + PAGE_SIZE - 1) &
+					  ~(unsigned long long)(PAGE_SIZE - 1);
+
+		if (m->type == type && phys >= m->phys && phys < mend &&
+		    m->va + m->size == iomap_next &&
+		    (unsigned long long)iomap_next + (need - mend) <=
+		    0x100000000ull) {
+			unsigned int extra = (unsigned int)(need - mend);
+
+			for (i = 0; i < extra; i += PAGE_SIZE)
+				if (!iomap_set(iomap_next + i,
+					       (unsigned int)mend + i,
+					       iomap_pte_flags((unsigned int)mend + i,
+							       type)))
+					return 0;
+			m->size += extra;
+			iomap_next = ((unsigned long long)iomap_next + extra >=
+				      0x100000000ull) ? 0 : iomap_next + extra;
+			return (void *)(m->va + (phys - m->phys));
+		}
+	}
+
+	p0 = phys & 0xFFFFF000u;
+	bytes = (unsigned int)(((end + PAGE_SIZE - 1) &
+				~(unsigned long long)(PAGE_SIZE - 1)) - p0);
+	if (n_iomaps >= IOMAP_MAX || !iomap_next ||
+	    (unsigned long long)iomap_next + bytes > 0x100000000ull) {
+		serial_puts(SERIAL_COM1, "paging: ioremap window full for ");
+		serial_puthex(SERIAL_COM1, phys);
+		serial_puts(SERIAL_COM1, "\n");
+		return 0;
+	}
+	va = iomap_next;
+	for (i = 0; i < bytes; i += PAGE_SIZE)
+		if (!iomap_set(va + i, p0 + i, iomap_pte_flags(p0 + i, type))) {
+			serial_puts(SERIAL_COM1, "paging: ioremap out of memory\n");
+			return 0;
+		}
+	iomap_next = ((unsigned long long)va + bytes >= 0x100000000ull)
+		     ? 0 : va + bytes;
+	iomaps[n_iomaps].phys = p0;
+	iomaps[n_iomaps].size = bytes;
+	iomaps[n_iomaps].va = va;
+	iomaps[n_iomaps].type = type;
+	n_iomaps++;
+	return (void *)(va + (phys - p0));
+}
+
+/* The window address of `phys` if some ioremap() covers it, else 0. */
+void *ioremap_lookup(unsigned int phys)
+{
+	unsigned int i;
+
+	for (i = 0; i < n_iomaps; i++) {
+		const struct iomap *m = &iomaps[i];
+
+		if (phys >= m->phys &&
+		    (unsigned long long)phys < (unsigned long long)m->phys + m->size)
+			return (void *)(m->va + (phys - m->phys));
+	}
+	return 0;
 }
 
 /* Split the 4 MB page containing `va` in one directory and set PCD on that
@@ -546,3 +687,35 @@ void kunmap(void *addr)
 		kmap_depth[c->id]--;
 	preempt_enable();
 }
+
+static int pa_is_ram(unsigned int pa)
+{
+	unsigned int i, n = mm_e820_count();
+
+	for (i = 0; i < n; i++) {
+		const struct e820_entry *e = mm_e820_entry(i);
+
+		if (e->type == E820_RAM && pa >= e->base &&
+		    (unsigned long long)pa < e->base + e->length)
+			return 1;
+	}
+	return 0;
+}
+
+volatile unsigned char *paging_debug_map(unsigned int pa, void **km)
+{
+	unsigned char *v;
+
+	*km = 0;
+	if (pa < MM_MAX_PHYS)
+		return (volatile unsigned char *)pa;
+	if (pa_is_ram(pa)) {
+		*km = kmap(pa & 0xFFFFF000u);
+		return (volatile unsigned char *)*km + (pa & 0xFFFu);
+	}
+	v = ioremap_lookup(pa);          /* includes the offset */
+	if (v)
+		return v;
+	v = ioremap(pa & 0xFFFFF000u, 0x1000, IOREMAP_MMIO);
+	return v ? v + (pa & 0xFFFu) : 0;
+}

+ 25 - 15
kernel/paging.h

@@ -8,15 +8,34 @@ void paging_init(void);
  * own page directory.  Called by paging_init(); the AP trampoline applies
  * the same policy inline. */
 void paging_prepare_cpu(void);
-/* Re-apply the framebuffer page types (after GPU BAR mappings that cover it). */
-void paging_lfb_refresh(void);
 /* 1 when the LFB is mapped for write-combining. */
 int paging_lfb_wc(void);
 
-/* Map a physical window identity-mapped with caching enabled (4 MB pages).
- * Used for the stolen-VRAM alias, which is ordinary DRAM; the effective type
- * is whatever the MTRRs say (write-combining when we added an entry). */
-void paging_map_ram(unsigned int base, unsigned int size);
+/* Kernel address of the boot framebuffer (the identity address when it is
+ * in lowmem, else its ioremap() window address); 0 without an LFB. */
+void *paging_lfb_va(void);
+
+/* Map physical [phys, phys+size) for kernel access and return the address
+ * of `phys`.  Lowmem (below MM_MAX_PHYS) is identity-mapped and returned
+ * as-is (an IOREMAP_MMIO range there has its 4 MB pages made uncacheable);
+ * anything else gets 4 KB pages in the kernel-only window at IOMAP_BASE,
+ * shared by every process directory.  IOREMAP_MMIO pages are uncached
+ * except where the LFB's WC MTRR covers them; IOREMAP_RAM pages (firmware
+ * tables, the stolen-VRAM alias) are write-back.  A range already covered
+ * by a mapping of the same type reuses it.  Mappings are never torn down.
+ * Returns 0 when the window is full or out of page tables. */
+#define IOREMAP_MMIO 0
+#define IOREMAP_RAM  1
+void *ioremap(unsigned int phys, unsigned int size, int type);
+
+/* Window address of `phys` if an ioremap() covers it, else 0 (peek/poke). */
+void *ioremap_lookup(unsigned int phys);
+
+/* Debug access (peek/poke/dbg) to physical byte `pa`: lowmem directly,
+ * highmem RAM through a kmap (*km is set; kunmap(*km) when done), MMIO
+ * through its ioremap() mapping, created on demand (4 KB, uncached).
+ * 0 when it cannot be mapped. */
+volatile unsigned char *paging_debug_map(unsigned int pa, void **km);
 
 
 
@@ -71,14 +90,5 @@ void paging_release_user_range(unsigned int cr3, unsigned int va,
 int paging_user_range_ok(unsigned int cr3, unsigned int va,
 			 unsigned int size, int write);
 
-/* Identity-map an MMIO range (e.g. a GPU register aperture) uncacheable into
- * the current page directory.  Uses 4 MB pages; the range is rounded up to a
- * whole number of 4 MB pages. */
-void paging_map_mmio(unsigned int phys_base, unsigned int size);
-
-/* Same, but into an explicit page directory (used to update per-process
- * directories; see process_map_mmio_all()). */
-void paging_map_mmio_cr3(unsigned int cr3, unsigned int phys_base,
-			 unsigned int size);
 
 #endif

+ 31 - 9
kernel/peek.c

@@ -1,14 +1,17 @@
-/* peek command: hex dump of memory (identity-mapped, so this also covers
- * MMIO registers, e.g. a driver's BAR once mapped).
+/* peek command: hex dump of physical memory, including MMIO registers.
  *
  *   peek <addr> [len]      default len 64, max 256
  *   poke <addr> <val>      write one dword (e.g. an MMIO register), print
  *                          the read-back
  *
- * Reading an unmapped address faults; the exception handler shows EIP/CR2.
+ * Addresses are physical.  Lowmem is read through the identity map, RAM
+ * above it (highmem) through a temporary kmap, and anything else (MMIO)
+ * through its ioremap() window mapping, created on demand (4 KB,
+ * uncached) when no driver mapped it yet.
  */
 
 #include "io.h"
+#include "paging.h"
 
 static void put_hex2(unsigned int v)
 {
@@ -80,12 +83,20 @@ void program_peek_main(const char *arg)
 		return;
 	}
 
-	mem = (const volatile unsigned char *)addr;
 	for (i = 0; i < len; i += 16) {
 		put_hex8(addr + i);
 		io_putstr(": ");
 		for (j = 0; j < 16 && i + j < len; j++) {
-			put_hex2(mem[i + j]);
+			void *km;
+
+			mem = paging_debug_map(addr + i + j, &km);
+			if (!mem) {
+				io_putstr("?? ");
+				continue;
+			}
+			put_hex2(*mem);
+			if (km)
+				kunmap(km);
 			io_putchar(' ');
 		}
 		io_putchar('\n');
@@ -99,7 +110,7 @@ void program_poke_main(const char *arg)
 {
 	const char *p = arg;
 	volatile unsigned int *mem;
-	unsigned int addr, val;
+	unsigned int addr, val, back;
 
 	if (parse_hex(p, &addr, &p) != 0) {
 		io_putstr("poke: usage poke <hexaddr> <hexval>\n");
@@ -118,14 +129,25 @@ void program_poke_main(const char *arg)
 		return;
 	}
 
-	mem = (volatile unsigned int *)(addr & ~3u);
-	*mem = val;
+	{
+		void *km;
+
+		mem = (volatile unsigned int *)paging_debug_map(addr & ~3u, &km);
+		if (!mem) {
+			io_putstr("poke: cannot map that address\n");
+			return;
+		}
+		*mem = val;
+		back = *mem;
+		if (km)
+			kunmap(km);
+	}
 
 	io_putstr("poke ");
 	put_hex8(addr & ~3u);
 	io_putstr(" <- ");
 	put_hex8(val);
 	io_putstr(" reads ");
-	put_hex8(*mem);
+	put_hex8(back);
 	io_putchar('\n');
 }

+ 6 - 9
kernel/process.c

@@ -560,23 +560,20 @@ void process_set_render_target(struct surface *s)
 		p->target = s;
 }
 
-void process_map_mmio_all(unsigned int phys_base, unsigned int size)
+void process_sync_kernel_pde(unsigned int index)
 {
+	const unsigned int *kpd = (const unsigned int *)paging_kernel_cr3();
 	unsigned int i, flags;
 
-	/* Map the kernel directory (the one new process directories clone) and
-	 * every live process copy.  paging_current_cr3() is the *current* process
-	 * directory, which may be one of these; mapping it twice is harmless.
-	 * The reaper frees PCBs, so hold the process lock while iterating. */
+	/* The reaper frees PCBs, so hold the process lock while iterating. */
 	flags = sched_lock_acquire();
-	paging_map_mmio(phys_base, size);
 	for (i = 0; i < PROC_MAX; i++) {
 		struct process *p = procs[i];
 
-		if (!p || p->state == PROC_DEAD)
+		if (!p || p->state == PROC_DEAD || !p->cr3 ||
+		    p->cr3 == (unsigned int)kpd)
 			continue;
-		if (p->cr3)
-			paging_map_mmio_cr3(p->cr3, phys_base, size);
+		((unsigned int *)p->cr3)[index] = kpd[index];
 	}
 	sched_lock_release(flags);
 }

+ 4 - 6
kernel/process.h

@@ -117,12 +117,10 @@ int process_create_user_tty(const char *name, const void *image,
 struct surface *process_render_target(void);
 void process_set_render_target(struct surface *s);
 
-/* Map an MMIO range into the kernel's page directory and every live process
- * copy (and thus every process created later, which clones the kernel map).
- * Drivers that map their register window at command time must use this: a
- * process runs with its own page-directory copy, so paging_map_mmio() alone
- * would only be visible to the kernel bootstrap context. */
-void process_map_mmio_all(unsigned int phys_base, unsigned int size);
+/* Copy kernel page-directory entry `index` into every live process
+ * directory.  paging.c calls it when ioremap() adds a window page table or
+ * makes a lowmem PDE uncacheable; processes created later clone it. */
+void process_sync_kernel_pde(unsigned int index);
 void process_exit(void);
 /* Called by the kworker: free switched-out dead processes. */
 void process_reap_dead(void);

+ 3 - 2
kernel/r8168.c

@@ -640,8 +640,9 @@ int r8168_init(void)
 	buf[i] = '\0';
 	dbg(buf);
 
-	process_map_mmio_all(bar_base, bar_size);
-	mmio = (unsigned char *)bar_base;
+	mmio = ioremap(bar_base, bar_size, IOREMAP_MMIO);
+	if (!mmio)
+		return -1;
 
 	/* Master abort / unmapped window reads back all-ones. */
 	if (r32(REG_TxConfig) == 0xFFFFFFFFu) {

+ 5 - 5
kernel/radeon.c

@@ -28,7 +28,7 @@
 #include "serial.h"
 #include "vga.h"
 
-static unsigned int radeon_mmio;   /* BAR2 register aperture (1:1 mapped) */
+static unsigned int radeon_mmio;   /* kernel mapping of the BAR2 registers */
 static int radeon_present;
 
 #define RADEON_MM_INDEX              0x0000
@@ -98,9 +98,9 @@ void radeon_init(unsigned int bus, unsigned int dev, unsigned int device_id)
 	if (bar2_size < 0x10000)
 		return;
 
-	paging_map_mmio(bar2_base, bar2_size);
-
-	radeon_mmio = bar2_base;
+	radeon_mmio = (unsigned int)ioremap(bar2_base, bar2_size, IOREMAP_MMIO);
+	if (!radeon_mmio)
+		return;
 
 	/* Validate the aperture: a master abort reads back all-ones. */
 	mmio_write(RADEON_MM_INDEX, RADEON_CRTC_GEN_CNTL);
@@ -111,7 +111,7 @@ void radeon_init(unsigned int bus, unsigned int dev, unsigned int device_id)
 
 	radeon_present = 1;
 	serial_puts(SERIAL_COM1, "radeon: cursor MMIO ");
-	serial_puthex(SERIAL_COM1, radeon_mmio);
+	serial_puthex(SERIAL_COM1, bar2_base);
 	serial_puts(SERIAL_COM1, "\n");
 }
 

+ 3 - 2
kernel/sdhci.c

@@ -398,10 +398,11 @@ static int sdhci_init(void)
 	serial_puthex(SERIAL_COM1, bar);
 	serial_puts(SERIAL_COM1, "\n");
 
-	process_map_mmio_all(bar, 0x1000);
 	cmd = pci_cfg_read32(bus, dev, func, 0x04);
 	pci_cfg_write32(bus, dev, func, 0x04, cmd | 0x0006u);
-	mmc.bar = bar;
+	mmc.bar = (unsigned int)ioremap(bar, 0x1000, IOREMAP_MMIO);
+	if (!mmc.bar)
+		return -1;
 	mmc.state = 0;
 
 	caps = rd32(SDHCI_CAPABILITIES);

+ 0 - 4
kernel/smp.c

@@ -104,10 +104,6 @@ void smp_init(void)
 		return;
 
 	lapic_set_base(acpi_lapic_base());
-	paging_map_mmio(acpi_lapic_base(), 0x1000);
-	/* The 4 MB MMIO page could share a PDE with the framebuffer; restore
-	 * the LFB's WC/PCD choice if so. */
-	paging_lfb_refresh();
 	if (!lapic_init_bsp()) {
 		serial_puts(SERIAL_COM1, "smp: BSP LAPIC init failed\n");
 		return;

+ 5 - 2
kernel/touchpad.c

@@ -352,9 +352,12 @@ int touchpad_init(void)
 	for (r = 0; r < 2; r++) {
 		for (k = 0; k < TP_CTRL_MAX; k++) {
 			unsigned int i = (k + 7) % TP_CTRL_MAX;
-			unsigned int base = ranges[r] + i * TP_CTRL_STEP;
+			unsigned int base = (unsigned int)ioremap(
+				ranges[r] + i * TP_CTRL_STEP, 0x1000,
+				IOREMAP_MMIO);
 
-			paging_map_mmio(base, 0x1000);
+			if (!base)
+				continue;
 			/* A block left in functional reset keeps its
 			 * register file but never clocks the I2C FSM. */
 			i2c_dw_reset_release(base);

+ 4 - 18
kernel/tpec.c

@@ -89,22 +89,6 @@ static void copy_trim(char *dst, unsigned int size, const unsigned char *src)
 	dst[i] = '\0';
 }
 
-/* Map every 4 MB page touched by [addr, addr+len); SMBIOS tables can sit
- * above the 128 MB identity map on larger machines (same approach as
- * acpi.c). */
-static void map_range(unsigned int addr, unsigned int len)
-{
-	unsigned int p = addr & 0xFFC00000u;
-	unsigned int last = (addr + len - 1u) & 0xFFC00000u;
-
-	for (;;) {
-		paging_map_ram(p, 0x400000u);
-		if (p >= last || p > 0xFFC00000u - 0x400000u)
-			break;
-		p += 0x400000u;
-	}
-}
-
 /* The n-th (1-based) string that follows a formatted SMBIOS structure. */
 static const unsigned char *smbios_string(const unsigned char *p,
 					  const unsigned char *end,
@@ -163,8 +147,10 @@ static int smbios_type1(char vendor[16], char product[32], char version[32])
 	    table_addr + table_len < table_addr)
 		return -1;
 
-	map_range(table_addr, table_len);
-	table = (const unsigned char *)table_addr;
+	/* SMBIOS tables can sit above lowmem on larger machines. */
+	table = ioremap(table_addr, table_len, IOREMAP_RAM);
+	if (!table)
+		return -1;
 	end = table + table_len;
 
 	for (p = table; p + 4 <= end; ) {

+ 15 - 7
kernel/vga.c

@@ -16,7 +16,8 @@
 /* Cached boot framebuffer info (set once when vga_fb_* first used). */
 static unsigned int vga_fb_w, vga_fb_h, vga_fb_pitch_val, vga_fb_bpp_val;
 static unsigned int vga_fb_vram_kb_val;
-static volatile unsigned char *vga_fb_lfb;
+static volatile unsigned char *vga_fb_lfb;   /* kernel address */
+static unsigned int vga_fb_lfb_phys_val;      /* physical LFB */
 static int vga_fb_cached;
 /* UEFI GOP can hand us RGBX instead of the VBE/BGRX byte order.  The shadow
  * buffer is always BGRX, so only the shadow->LFB flush has to swizzle. */
@@ -251,6 +252,7 @@ static void vga_cache_boot_fb(void)
 	if (!info->valid) {
 		vga_fb_w = vga_fb_h = vga_fb_pitch_val = vga_fb_bpp_val = 0;
 		vga_fb_lfb = 0;
+		vga_fb_lfb_phys_val = 0;
 		gpu_init(0);
 		return;
 	}
@@ -270,7 +272,12 @@ static void vga_cache_boot_fb(void)
 		if (vga_fb_pitch_val < min_pitch)
 			vga_fb_pitch_val = min_pitch;
 	}
-	vga_fb_lfb = (volatile unsigned char *)(unsigned long)info->lfb_base;
+	/* The LFB is reached through its paging_init() mapping (the ioremap
+	 * window when it is above lowmem), not its physical address. */
+	vga_fb_lfb_phys_val = info->lfb_base;
+	vga_fb_lfb = (volatile unsigned char *)paging_lfb_va();
+	if (!vga_fb_lfb)
+		vga_fb_lfb = (volatile unsigned char *)(unsigned long)info->lfb_base;
 	vga_fb_vram_kb_val = (unsigned int)info->vram_64k * 64;
 	/* Clamp height to what fits in VRAM (prevents wrap on small VRAM cards) */
 	if (vga_fb_vram_kb_val > 0 && vga_fb_pitch_val > 0) {
@@ -339,6 +346,12 @@ unsigned long vga_fb_lfb_addr(void)
 	return (unsigned long)vga_fb_lfb;
 }
 
+unsigned int vga_fb_lfb_phys(void)
+{
+	vga_cache_boot_fb();
+	return vga_fb_lfb_phys_val;
+}
+
 int vga_has_vesa(void)
 {
 	vga_cache_boot_fb();
@@ -627,11 +640,6 @@ void vga_shadow_init(void)
 	if (!vga_fb_lfb || vga_fb_w == 0 || vga_fb_h == 0)
 		return;
 
-	/* The GPU probe (inside vga_cache_boot_fb) maps its BARs -- including the
-	 * NVIDIA VRAM aperture, which covers the VBE LFB -- uncacheable, so the
-	 * framebuffer mapping is re-applied once everything is up. */
-	paging_lfb_refresh();
-
 	/* On ION-class hardware the framebuffer is stolen DRAM and the VBE LFB
 	 * is just a BAR3 window into it.  The BIOS leaves a 512 MB uncacheable
 	 * MTRR over the PCIe apertures (and the lowest-numbered matching entry

+ 4 - 1
kernel/vga.h

@@ -31,8 +31,11 @@ unsigned int vga_fb_bpp(void);
 unsigned int vga_fb_vram_kb(void);
 unsigned int vga_fb_pci_id(void);
 void *vga_fb_ptr(void);
-/* Physical address of the linear framebuffer (for off-screen VRAM access). */
+/* Kernel address of the linear framebuffer (for off-screen VRAM access);
+ * the LFB's ioremap() window address when it is above lowmem. */
 unsigned long vga_fb_lfb_addr(void);
+/* Physical address of the linear framebuffer (for BAR comparisons). */
+unsigned int vga_fb_lfb_phys(void);
 int vga_has_vesa(void);
 
 /* A drawing surface: the screen's shadow buffer, a window layer or a canvas.

+ 9 - 6
kernel/xhci.c

@@ -186,7 +186,8 @@ struct xhci_bulk {
 	int error;
 };
 
-static unsigned int mmio_base;
+static unsigned int mmio_base;   /* kernel mapping of BAR0 */
+static unsigned int mmio_phys;   /* BAR0 itself (logged, reported) */
 static unsigned int cap_len;
 static unsigned int rt_off;
 static unsigned int db_off;
@@ -1583,8 +1584,10 @@ found_xhci:
 	cmd = pci_cfg_read32(pci_bus, pci_dev, pci_func, 0x04);
 	pci_cfg_write32(pci_bus, pci_dev, pci_func, 0x04, cmd | 0x0006);
 
-	mmio_base = bar;
-	paging_map_mmio(mmio_base, 0x10000);
+	mmio_phys = bar;
+	mmio_base = (unsigned int)ioremap(bar, 0x10000, IOREMAP_MMIO);
+	if (!mmio_base)
+		return -1;
 
 	pci_id = pci_cfg_read32(pci_bus, pci_dev, pci_func, 0x00);
 	cap_len = mmio_rd(0) & 0xFF;
@@ -1608,7 +1611,7 @@ found_xhci:
 	ctx_size = (hcc_params & 0x0004u) ? 64 : 32;
 
 	dbg("xhci: found\n");
-	dbg_hex("xhci: mmio ", mmio_base);
+	dbg_hex("xhci: mmio ", mmio_phys);
 	dbg_dec("xhci: slots ", max_slots);
 	dbg_dec("xhci: ports ", max_ports);
 	dbg_dec("xhci: cap len ", cap_len);
@@ -1761,7 +1764,7 @@ found_xhci:
 	scan_protocols();
 	xlog_hex("xhci: ", pci_id);
 	xlog_hex(" hci ", hci_version);
-	xlog_hex(" mmio ", mmio_base);
+	xlog_hex(" mmio ", mmio_phys);
 	xlog_hex(" ports ", port_mask);
 	xlog_hex(" csz ", ctx_size);
 	xlog_hex(" scratch ", n);
@@ -1797,7 +1800,7 @@ static void xhci_info(unsigned int ctrl, unsigned int *bus,
 	*bus = pci_bus;
 	*dev = pci_dev;
 	*func = pci_func;
-	*base = mmio_base;
+	*base = mmio_phys;
 	*mmio = 1;
 }