Procházet zdrojové kódy

Speed up the software blitter and stop losing flushes

- cpu.c/h (new): SSE2 detection done safely for i586 (EFLAGS ID-bit probe
  before CPUID) and the CR4.OSFXSR/OSXMMEXCPT enable, since the kernel is
  built -mno-sse.  Cached after the first call.
- blit.c/vga_copy.S: the software fill uses an SSE2 row store, and streaming
  (movntdq, one sfence per fill) for rows that do not fit the cache: a
  full-screen fill of the 5 MB shadow paid read-for-ownership for every line
  (416 MB/s on the ION).  `gfxbench` measured 2777 MB/s after the change.
- vga.c: vga_present_shadow() refuses to re-enter while another process is
  mid-flush.  The flush walks and trims a shared damage list and a full-screen
  copy is long enough to be preempted, so two flushes could interleave and
  corrupt it (found as a #PF writing past the LFB).  A skipped flush leaves
  its damage queued for the next one.
- fbtext.c: the full-screen console scroll copies rows instead of bytes.
- vga.c: the software cursor's save/restore copies dwords on 32 bpp.
- gfxbench (new command): times fill screen/content, window copy, cell grid,
  canvas 565->888 and the flush against a RAM layer; reports the bytes the
  presenter really copied (`vga_present_copied`).
- paging.c/mtrr.c: the WC MTRR was defeated by the PCD bit paging set on the
  LFB pages (UC regardless of the MTRRs, ~33 MB/s of separate PCI writes).
  mtrr_set_wc() now reports whether and what it covered; paging maps only the
  covered pages cacheable and keeps the rest UC.  paging_lfb_refresh()
  re-applies the mapping because the GPU probe maps its BARs (the NVIDIA BAR3
  aperture contains the VBE LFB) uncacheable afterwards.
- serial.c: fix serial_puthex's broken table (digits 6 and 7 were missing),
  which garbled the MTRR log lines.
Johannes Zellner před 2 týdny
rodič
revize
f4306304cf
15 změnil soubory, kde provedl 550 přidání a 60 odebrání
  1. 16 3
      AGENTS.md
  2. 9 3
      Makefile
  3. 39 10
      blit.c
  4. 47 0
      cpu.c
  5. 12 0
      cpu.h
  6. 7 9
      fbtext.c
  7. 179 0
      gfxbench.c
  8. 25 11
      mtrr.c
  9. 7 3
      mtrr.h
  10. 64 17
      paging.c
  11. 4 0
      paging.h
  12. 2 0
      terminal.c
  13. 64 4
      vga.c
  14. 9 0
      vga.h
  15. 66 0
      vga_copy.S

+ 16 - 3
AGENTS.md

@@ -108,7 +108,16 @@ below the run area), so adding code no longer moves any address.
   through the `blit` vtable (`blit.c/h`: fill/copy/blend/canvas/present/flip,
   software only today) so an accelerator can replace single ops without
   touching the WM.  `vga_target_damage()` is a no-op while the target is not
-  the screen.  The boot log prints `blit backend: <name>`.  Drawing marks
+  the screen.  `cpu.c` detects SSE2 (CPUID, with the EFLAGS ID-bit probe and
+  the CR4.OSFXSR/OSXMMEXCPT enable, since the kernel is -mno-sse) and the
+  software fill uses an SSE2 row store when it is there -- streaming
+  (`movntdq`, one sfence per fill) for rows that do not fit the cache, which
+  avoids the read-for-ownership traffic that makes big fills DRAM-bound on the
+  Atom.  `gfxbench` times the fill/copy/cell/canvas/flush paths and reports
+  the bytes the presenter really copied (`vga_present_copied`).  The flush trims a shared
+  damage list, so `vga_present_shadow()` refuses to re-enter while another
+  process is mid-flush (a full-screen copy is long enough to be preempted).
+  The boot log prints `blit backend: <name>`.  Drawing marks
   damage rectangles (a small
   list that merges rects when they overlap or touch, so a keystroke's cells
   collapse while unrelated damage stays apart); the main loop copies them to
@@ -116,7 +125,11 @@ below the run area), so adding code no longer moves any address.
   fixed frame rate.  A rect that fits the budget is copied whole, so a window
   repaint never tears; only a bigger one (a full-screen redraw) is chunked, and
   its remaining rows continue after the newer damage.  ATI Mach64 CRTC
-  fixup; NVIDIA is VBE-only (never touch the legacy VGA CRTC). `nvidia.c`
+  fixup; NVIDIA is VBE-only (never touch the legacy VGA CRTC).  The LFB is
+  mapped UC unless the WC MTRR covered the range (`mtrr_set_wc()` returns the
+  covered range and `paging.c` makes only those pages cacheable -- a PCD=1
+  page is UC no matter what the MTRRs say, which held VRAM writes at ~33 MB/s
+  on the ION until it was fixed). `nvidia.c`
   identifies the card, maps BAR0/BAR1/BAR3 and reads the MCP7x stolen-VRAM
   geometry; `nv50.c` probes the nv50-family display engine at boot (read-only,
   serial/netmon). The ION/AT3IONT-I hardware cursor is deliberately not driven:
@@ -272,7 +285,7 @@ below the run area), so adding code no longer moves any address.
 `cd [path]`, `ls [path]`, `cat <path>`, `save <path> <text>`, `rm <path>`,
 `mkdir <path>`, `rmdir <path>`, `hello`, `reboot`, `poweroff`, `run <name>|<file>`, `space`,
 `prog`,
-`display`, `cpuinfo`, `mouseinfo`, `lsusb`, `lspci [id]`, `peek <addr> [len]`, `poke <addr> <val>`, `storage`, `meminfo`, `top`, `format <device>`,
+`display`, `cpuinfo`, `mouseinfo`, `gfxbench`, `lsusb`, `lspci [id]`, `peek <addr> [len]`, `poke <addr> <val>`, `storage`, `meminfo`, `top`, `format <device>`,
 `ps`, `kill <pid>`, `help`, `clear`, `serial test|recv`, `kexec`, `dbg`.
 
 Network: `netinfo`, `netrecv`, `netsend`, `netip [a.b.c.d]`, `dhcp` (re-run the

+ 9 - 3
Makefile

@@ -80,7 +80,13 @@ serial_recv.o: serial_recv.c serial.h bin_recv.h io.h
 vga.o: vga.c vga.h blit.h pci.h gpu.h ati.h process.h
 	$(CC) $(CFLAGS) -o $@ $<
 
-blit.o: blit.c blit.h vga.h gfx.h
+blit.o: blit.c blit.h vga.h gfx.h cpu.h
+	$(CC) $(CFLAGS) -o $@ $<
+
+cpu.o: cpu.c cpu.h
+	$(CC) $(CFLAGS) -o $@ $<
+
+gfxbench.o: gfxbench.c blit.h cpu.h fbtext.h gfx.h io.h mm.h timer.h vga.h
 	$(CC) $(CFLAGS) -o $@ $<
 
 vga_copy.o: vga_copy.S
@@ -251,7 +257,7 @@ sys_net.o: sys_net.c sys_net.h syscall.h dhcp.h dns.h net.h netdev.h netlog.h
 syscall_stub.o: syscall_stub.S irqret.inc
 	$(AS) $(ASFLAGS) -o $@ $<
 
-kernel.elf: vesa_stage.o entry.o kernel.o timer.o switch.o process.o mm.o kmalloc.o paging.o elf.o tss.o syscall.o syscall_stub.o screen.o io.o rtc.o terminal.o wm.o reboot.o poweroff.o disk.o fat.o run.o serial.o serial_ring_buf.o serial_loop.o serial_recv.o bin_recv.o vga.o blit.o vga_copy.o gpu.o ati.o cirrus.o nvidia.o nv50.o radeon.o mtrr.o font_8x8.o fbtext.o gfx.o sys_gfx.o display.o mouse.o uhci.o ohci.o usb.o usbhid.o lsusb.o kdbg.o blkdev.o ata.o ahci.o fatdetect.o fatformat.o format.o vfs.o sys_fs.o sys_info.o sys_net.o idle.o kexec.o kexec_tramp.o dbg.o lspci.o peek.o ne2000.o r8168.o netdev.o net.o netkexec.o netlog.o dhcp.o dns.o tcp.o prog_embed.o space_embed.o cmds_pack.o ramfs.o payload.o
+kernel.elf: vesa_stage.o entry.o kernel.o timer.o switch.o process.o mm.o kmalloc.o paging.o elf.o tss.o syscall.o syscall_stub.o screen.o io.o rtc.o terminal.o wm.o reboot.o poweroff.o disk.o fat.o run.o serial.o serial_ring_buf.o serial_loop.o serial_recv.o bin_recv.o vga.o blit.o cpu.o vga_copy.o gfxbench.o gpu.o ati.o cirrus.o nvidia.o nv50.o radeon.o mtrr.o font_8x8.o fbtext.o gfx.o sys_gfx.o display.o mouse.o uhci.o ohci.o usb.o usbhid.o lsusb.o kdbg.o blkdev.o ata.o ahci.o fatdetect.o fatformat.o format.o vfs.o sys_fs.o sys_info.o sys_net.o idle.o kexec.o kexec_tramp.o dbg.o lspci.o peek.o ne2000.o r8168.o netdev.o net.o netkexec.o netlog.o dhcp.o dns.o tcp.o prog_embed.o space_embed.o cmds_pack.o ramfs.o payload.o
 	$(LD) $(KERNEL_LDFLAGS) -o $@ $^
 
 MBR_LDFLAGS = -T mbr.ld -nostdlib -m elf_i386
@@ -586,4 +592,4 @@ net-dev: kernel.bin netkexec_send
 	./netkexec_send $(BOARD_IP) $(NETKEXEC_PORT) 0x$$entry kernel.bin
 
 clean:
-	rm -f boot.o vesa_stage.o entry.o kernel.o timer.o switch.o process.o mm.o kmalloc.o paging.o elf.o tss.o syscall.o syscall_stub.o screen.o io.o rtc.o terminal.o wm.o hello.o reboot.o poweroff.o fat.o run.o disk.o serial.o serial_ring_buf.o serial_loop.o serial_recv.o bin_recv.o vga.o blit.o vga_copy.o gpu.o ati.o cirrus.o nvidia.o nv50.o radeon.o mtrr.o font_8x8.o fbtext.o gfx.o sys_gfx.o display.o mouse.o uhci.o ohci.o usb.o usbhid.o lsusb.o kdbg.o blkdev.o ata.o ahci.o fatdetect.o fatformat.o format.o vfs.o sys_fs.o sys_info.o idle.o kexec.o kexec_tramp.o dbg.o lspci.o peek.o ne2000.o r8168.o netdev.o net.o netkexec.o netlog.o dhcp.o dns.o tcp.o mbr.o boot.elf kernel.elf mbr.elf boot.bin mbr.bin kernel.bin boot.bin.tmp boot.bin.patched image.bin floppy.img usb.img final.bin root.img prog.o prog.elf prog.bin space.o space.elf space.bin hello.elf hello_embed.o prog_embed.o space_embed.o cmds.pack cmds_pack.o ramfs.o payload.o mkpack $(addsuffix .o,$(UCMDS)) $(addsuffix .elf,$(UCMDS)) $(TLSLIB_OBJS) tls_test bootstrap.o bootstrap.elf bootstrap.bin send_serial kexec_send netkexec_send netmonitor
+	rm -f boot.o vesa_stage.o entry.o kernel.o timer.o switch.o process.o mm.o kmalloc.o paging.o elf.o tss.o syscall.o syscall_stub.o screen.o io.o rtc.o terminal.o wm.o hello.o reboot.o poweroff.o fat.o run.o disk.o serial.o serial_ring_buf.o serial_loop.o serial_recv.o bin_recv.o vga.o blit.o cpu.o vga_copy.o gfxbench.o gpu.o ati.o cirrus.o nvidia.o nv50.o radeon.o mtrr.o font_8x8.o fbtext.o gfx.o sys_gfx.o display.o mouse.o uhci.o ohci.o usb.o usbhid.o lsusb.o kdbg.o blkdev.o ata.o ahci.o fatdetect.o fatformat.o format.o vfs.o sys_fs.o sys_info.o idle.o kexec.o kexec_tramp.o dbg.o lspci.o peek.o ne2000.o r8168.o netdev.o net.o netkexec.o netlog.o dhcp.o dns.o tcp.o mbr.o boot.elf kernel.elf mbr.elf boot.bin mbr.bin kernel.bin boot.bin.tmp boot.bin.patched image.bin floppy.img usb.img final.bin root.img prog.o prog.elf prog.bin space.o space.elf space.bin hello.elf hello_embed.o prog_embed.o space_embed.o cmds.pack cmds_pack.o ramfs.o payload.o mkpack $(addsuffix .o,$(UCMDS)) $(addsuffix .elf,$(UCMDS)) $(TLSLIB_OBJS) tls_test bootstrap.o bootstrap.elf bootstrap.bin send_serial kexec_send netkexec_send netmonitor

+ 39 - 10
blit.c

@@ -6,6 +6,7 @@
  * for now; a flip/VRAM backend would replace them with GPU operations). */
 
 #include "blit.h"
+#include "cpu.h"
 #include "gfx.h"
 #include "vga.h"
 
@@ -18,6 +19,21 @@ static unsigned int pixel_bytes(unsigned int bpp)
 	return 4;
 }
 
+/* Fill `dwords` with `value`; SSE2 for anything worth the call. */
+static void fill_dwords(unsigned int *p, unsigned int value, unsigned int dwords)
+{
+	unsigned int i;
+
+	if (dwords == 0)
+		return;
+	if (dwords >= 8 && cpu_has_sse2()) {
+		vga_fb_fill_row_sse2(p, value, dwords);
+		return;
+	}
+	for (i = 0; i < dwords; i++)
+		p[i] = value;
+}
+
 /* Solid fill of the target. */
 static void software_fill(unsigned int x, unsigned int y, unsigned int w,
 			  unsigned int h, unsigned int rgb)
@@ -42,12 +58,14 @@ static void software_fill(unsigned int x, unsigned int y, unsigned int w,
 	if (t->bpp == 16) {
 		unsigned short val = (unsigned short)(((r >> 3) << 11) |
 					((g >> 2) << 5) | (b >> 3));
+		unsigned int val32 = (unsigned int)val | ((unsigned int)val << 16);
 		for (row = y; row < y1; row++) {
-			unsigned short *p = (unsigned short *)(t->px +
+			unsigned int *p = (unsigned int *)(t->px +
 					(unsigned long)row * t->pitch + x * 2);
-			unsigned int xx;
-			for (xx = x; xx < x1; xx++)
-				*p++ = val;
+			unsigned int px = x1 - x;
+			fill_dwords(p, val32, px / 2);
+			if (px & 1)
+				((unsigned short *)p)[px - 1] = val;
 		}
 	} else if (t->bpp == 24) {
 		for (row = y; row < y1; row++) {
@@ -63,12 +81,23 @@ static void software_fill(unsigned int x, unsigned int y, unsigned int w,
 	} else {
 		unsigned int val = ((unsigned int)t->alpha << 24) |
 				   (r << 16) | (g << 8) | b;
-		for (row = y; row < y1; row++) {
-			unsigned int *p = (unsigned int *)(t->px +
-					(unsigned long)row * t->pitch + x * 4);
-			unsigned int xx;
-			for (xx = x; xx < x1; xx++)
-				*p++ = val;
+		unsigned int dwords = x1 - x;
+
+		/* Long rows are bigger than the cache: normal stores would
+		 * read every line in (RFO) before overwriting it.  Stream
+		 * them instead and order once at the end. */
+		if (dwords >= 32 && cpu_has_sse2()) {
+			for (row = y; row < y1; row++)
+				vga_fb_fill_row_nt((unsigned int *)(t->px +
+					(unsigned long)row * t->pitch + x * 4),
+					val, dwords);
+			__asm__ volatile ("sfence" ::: "memory");
+		} else {
+			for (row = y; row < y1; row++) {
+				unsigned int *p = (unsigned int *)(t->px +
+						(unsigned long)row * t->pitch + x * 4);
+				fill_dwords(p, val, dwords);
+			}
 		}
 	}
 	vga_target_damage(x, y, x1 - x, y1 - y);

+ 47 - 0
cpu.c

@@ -0,0 +1,47 @@
+/* CPU feature detection (see cpu.h). */
+
+#include "cpu.h"
+
+static int sse2 = -1;      /* -1 = not probed yet, 0 = no, 1 = yes */
+
+/* CPUID exists only if the ID bit in EFLAGS can be toggled (i586+). */
+static int cpuid_supported(void)
+{
+	unsigned int a, b;
+
+	__asm__ volatile ("pushfl; popl %0" : "=r"(a));
+	b = a ^ (1u << 21);
+	__asm__ volatile ("pushl %0; popfl" :: "r"(b));
+	__asm__ volatile ("pushfl; popl %0" : "=r"(b));
+	return ((a ^ b) & (1u << 21)) != 0;
+}
+
+int cpu_has_sse2(void)
+{
+	unsigned int eax, ebx, ecx, edx;
+
+	if (sse2 >= 0)
+		return sse2;
+	sse2 = 0;
+	if (cpuid_supported()) {
+		eax = 1;
+		__asm__ volatile ("cpuid"
+				  : "+a"(eax), "=b"(ebx), "=c"(ecx), "=d"(edx));
+		if (edx & (1u << 26)) {
+			/* SSE instructions #UD unless CR4 enables the state
+			 * (OSFXSR).  The kernel is built -mno-sse, so the
+			 * registers are only used inside a single helper and
+			 * need no per-process save. */
+			if (edx & (1u << 24)) {          /* FXSR */
+				unsigned int cr4;
+
+				__asm__ volatile ("mov %%cr4, %0" : "=r"(cr4));
+				cr4 |= (1u << 9) | (1u << 10);  /* OSFXSR | OSXMMEXCPT */
+				__asm__ volatile ("mov %0, %%cr4" :: "r"(cr4)
+						  : "memory");
+			}
+			sse2 = 1;
+		}
+	}
+	return sse2;
+}

+ 12 - 0
cpu.h

@@ -0,0 +1,12 @@
+/* CPU feature detection for the optional fast paths.
+ *
+ * The kernel targets i586, so SSE/SSE2 must never be assumed: CPUID itself
+ * may not exist (early Pentium and below), which is why the ID bit in EFLAGS
+ * is toggled first.  Detected features are cached after the first call. */
+#ifndef CPU_H
+#define CPU_H
+
+/* 1 when the CPU supports SSE2 (the ION does, the Pentium targets do not). */
+int cpu_has_sse2(void);
+
+#endif

+ 7 - 9
fbtext.c

@@ -251,16 +251,14 @@ void fbtext_scroll_up_one_line(void)
 	if (height <= CONTENT_TOP + ROW_HEIGHT)
 		return;
 	/* Scroll only the content area (below rainbow); header and rainbow stay fixed. */
-	for (y = CONTENT_TOP; y < height - ROW_HEIGHT; y++) {
-		unsigned int src_off = (y + ROW_HEIGHT) * pitch;
-		unsigned int dst_off = y * pitch;
-		for (i = 0; i < pitch; i++)
-			fb[dst_off + i] = fb[src_off + i];
-	}
+	for (y = CONTENT_TOP; y < height - ROW_HEIGHT; y++)
+		vga_fb_copy_row(fb + (unsigned long)y * pitch,
+				fb + (unsigned long)(y + ROW_HEIGHT) * pitch,
+				pitch / 4);
 	for (y = height - ROW_HEIGHT; y < height; y++) {
-		unsigned int off = y * pitch;
-		for (i = 0; i < pitch; i++)
-			fb[off + i] = 0;
+		unsigned int *p = (unsigned int *)(fb + (unsigned long)y * pitch);
+		for (i = 0; i < pitch / 4; i++)
+			p[i] = 0;
 	}
 	/* Mark scrolled rows dirty so the flush copies them. */
 	vga_fb_mark_dirty_rect(CONTENT_TOP, height - CONTENT_TOP);

+ 179 - 0
gfxbench.c

@@ -0,0 +1,179 @@
+/* gfxbench: time the software drawing paths, so each rendering step can be
+ * measured on target instead of guessed.
+ *
+ * The fill/copy/cell/canvas cases run against a screen-sized RAM layer, which
+ * is what the window manager draws into (the shadow buffer), and the flush
+ * case times the real shadow -> VRAM copy.  The screen itself is never
+ * disturbed: no drawing lands in the shadow, so the flush copies the WM's
+ * unchanged content. */
+
+#include "blit.h"
+#include "cpu.h"
+#include "fbtext.h"
+#include "gfx.h"
+#include "io.h"
+#include "mm.h"
+#include "paging.h"
+#include "timer.h"
+#include "vga.h"
+
+#define LAYER_W 1280
+#define LAYER_H 1024
+#define CONTENT_W 514
+#define CONTENT_H 320
+#define WIN_W 514
+#define WIN_H 338
+#define CELLS_COLS 64
+#define CELLS_ROWS 32
+
+static void put_dec(unsigned int n)
+{
+	char buf[11];
+	unsigned int i = sizeof(buf);
+
+	if (n == 0) {
+		io_putchar('0');
+		return;
+	}
+	while (n && i) {
+		buf[--i] = (char)('0' + n % 10);
+		n /= 10;
+	}
+	while (i < sizeof(buf))
+		io_putchar(buf[i++]);
+}
+
+static void bench(const char *name, unsigned int n, unsigned int bytes,
+		  unsigned int ms)
+{
+	io_putstr(name);
+	io_putstr("  ");
+	put_dec(n);
+	io_putstr("x ");
+	put_dec(ms);
+	io_putstr(" ms");
+	if (ms) {
+		/* 32-bit friendly MB/s: bytes/1024 is small enough to scale. */
+		unsigned int kbs = (bytes / 1024) * n;
+		unsigned int mbs = (kbs * 1000 / ms) / 1024;
+
+		io_putstr("  ");
+		put_dec(mbs);
+		io_putstr(" MB/s");
+	}
+	io_putstr("\n");
+}
+
+static unsigned char *alloc_pages(unsigned int bytes)
+{
+	unsigned int pages = (bytes + 4095) / 4096;
+
+	return (unsigned char *)mm_alloc_pages(pages);
+}
+
+static void free_pages(unsigned char *p, unsigned int bytes)
+{
+	unsigned int pages = (bytes + 4095) / 4096;
+	unsigned int i;
+
+	for (i = 0; i < pages; i++)
+		mm_free_page((unsigned int)p + i * 4096);
+}
+
+void program_gfxbench_main(void)
+{
+	struct surface layer;
+	struct gfx_canvas canvas;
+	unsigned short *cells;
+	unsigned char *layer_px, *cell_px;
+	unsigned int t0, t1, i;
+	unsigned int layer_bytes = LAYER_W * LAYER_H * 4;
+
+	io_putstr("gfxbench (sse2: ");
+	io_putstr(cpu_has_sse2() ? "yes" : "no");
+	io_putstr(", lfb wc: ");
+	io_putstr(paging_lfb_wc() ? "yes" : "no");
+	io_putstr(")\n");
+
+	layer_px = alloc_pages(layer_bytes);
+	if (!layer_px) {
+		io_putstr("gfxbench: out of memory\n");
+		return;
+	}
+	layer.px = layer_px;
+	layer.w = LAYER_W;
+	layer.h = LAYER_H;
+	layer.pitch = LAYER_W * 4;
+	layer.bpp = 32;
+	layer.alpha = 255;
+
+	vga_set_target(&layer);
+
+	/* Full-screen fill: what a full redraw costs. */
+	t0 = get_ticks_ms();
+	for (i = 0; i < 10; i++)
+		blit->fill(0, 0, LAYER_W, LAYER_H, 0x203040);
+	t1 = get_ticks_ms();
+	bench("fill screen ", 10, layer_bytes, t1 - t0);
+
+	/* Window content fill (the per-cell background). */
+	t0 = get_ticks_ms();
+	for (i = 0; i < 100; i++)
+		blit->fill(0, 0, CONTENT_W, CONTENT_H, 0x000000);
+	t1 = get_ticks_ms();
+	bench("fill content", 100, CONTENT_W * CONTENT_H * 4, t1 - t0);
+
+	/* Window move: an overlapping rect copy, as a drag step does. */
+	t0 = get_ticks_ms();
+	for (i = 0; i < 100; i++)
+		blit->copy(1, 1, 0, 0, WIN_W, WIN_H);
+	t1 = get_ticks_ms();
+	bench("copy window ", 100, WIN_W * WIN_H * 4, t1 - t0);
+
+	/* Cell grid: a whole terminal content repaint (glyph blitting). */
+	cell_px = alloc_pages(CELLS_COLS * CELLS_ROWS * 2);
+	if (cell_px) {
+		cells = (unsigned short *)cell_px;
+		for (i = 0; i < CELLS_COLS * CELLS_ROWS; i++)
+			cells[i] = (unsigned short)(0x0A00 | ('A' + (i % 26)));
+		t0 = get_ticks_ms();
+		for (i = 0; i < 50; i++)
+			fbtext_draw_cells_clip(cells, CELLS_COLS, CELLS_ROWS,
+				0, 0, 0, 0, CELLS_COLS * 8, CELLS_ROWS * 10);
+		t1 = get_ticks_ms();
+		bench("cells 64x32 ", 50, CELLS_COLS * CELLS_ROWS * 80 * 4, t1 - t0);
+		free_pages(cell_px, CELLS_COLS * CELLS_ROWS * 2);
+	}
+
+	/* Canvas presentation: RGB565 -> 32 bpp expansion. */
+	if (gfx_alloc(&canvas, CONTENT_W, CONTENT_H) == 0) {
+		for (i = 0; i < CONTENT_W * CONTENT_H; i++)
+			canvas.px[i] = (unsigned short)(i * 7);
+		t0 = get_ticks_ms();
+		for (i = 0; i < 20; i++)
+			blit->canvas(&canvas, 0, 0, 0, 0, CONTENT_W, CONTENT_H);
+		t1 = get_ticks_ms();
+		bench("canvas 565  ", 20, CONTENT_W * CONTENT_H * 4, t1 - t0);
+		gfx_free(&canvas);
+	}
+
+	vga_set_target(0);
+
+	/* Flush: shadow -> VRAM (bus bound, not CPU bound).  Report the bytes
+	 * the presenter really copied: a long flush can be preempted, and a
+	 * flush started while another process is inside one is skipped. */
+	{
+		unsigned int before = vga_present_copied;
+
+		t0 = get_ticks_ms();
+		for (i = 0; i < 2; i++) {
+			vga_fb_mark_dirty_rect(0, vga_fb_height());
+			vga_fb_flush();
+		}
+		t1 = get_ticks_ms();
+		/* one "run" whose size is the bytes actually copied */
+		bench("flush screen", 1, vga_present_copied - before, t1 - t0);
+	}
+
+	free_pages(layer_px, layer_bytes);
+}

+ 25 - 11
mtrr.c

@@ -1,11 +1,14 @@
 /*
  * MTRR write-combining.
  *
- * The VESA LFB is mapped uncacheable (UC) by paging.c, so every dword the
- * shadow flush writes to VRAM is a separate uncached PCI transaction.
+ * VRAM writes (the shadow flush) used to be separate uncached PCI
+ * transactions: ~33 MB/s on the AT3IONT-I, so a full screen took 150 ms.
  * Marking the LFB range write-combining (WC) via an MTRR lets the CPU batch
- * those writes into burst transfers, which is the single biggest rendering
- * win on real hardware.
+ * those writes into bursts -- but only if the *pages* allow caching too:
+ * paging.c used to map the LFB with PCD=1, which is UC regardless of the
+ * MTRR, so the WC entry alone did nothing.  paging.c now installs the MTRR
+ * first (through mtrr_set_wc(), which reports whether it covered the range)
+ * and maps only the covered pages cacheable, leaving the rest UC.
  *
  * Two hard rules (the first key press froze an ASUS AT3IONT-I; these were
  * the only places where the kernel touched cacheability behind the BIOS's
@@ -157,7 +160,8 @@ static void dump_mtrrs(unsigned int vcnt)
 	}
 }
 
-void mtrr_set_wc(unsigned int base, unsigned int size)
+int mtrr_set_wc(unsigned int base, unsigned int size,
+		unsigned int *wc_base, unsigned int *wc_size)
 {
 	unsigned int lo, hi, mlo;
 	unsigned int def_type;
@@ -169,14 +173,14 @@ void mtrr_set_wc(unsigned int base, unsigned int size)
 	char *p;
 
 	if (!cpu_has_mtrr() || size == 0)
-		return;
+		return 0;
 
 	rdmsr(MSR_MTRRCAP, &lo, &hi);
 	vcnt = lo & 0xFF;
 	if (vcnt > 16)
 		vcnt = 16;   /* sanity cap; CPUs ship 8..10 variable ranges */
 	if (vcnt == 0)
-		return;
+		return 0;
 
 	/* Dump the first 8 at most (kdbg info rows are a shared pool); the
 	 * search below still covers every variable range. */
@@ -187,7 +191,7 @@ void mtrr_set_wc(unsigned int base, unsigned int size)
 	while (region_size < size) {
 		region_size <<= 1;
 		if (region_size == 0)
-			return;   /* overflow (size > 2 GB) -- give up */
+			return 0;   /* overflow (size > 2 GB) -- give up */
 	}
 
 	/* A variable-range MTRR must be aligned to its own size.  The largest
@@ -199,7 +203,7 @@ void mtrr_set_wc(unsigned int base, unsigned int size)
 	if (region_size > align)
 		region_size = align;
 	if (region_size < 0x1000)
-		return;
+		return 0;
 	base &= ~(region_size - 1);
 
 	/* A kernel reload does not reset the MTRRs: reuse the WC entry an
@@ -217,7 +221,11 @@ void mtrr_set_wc(unsigned int base, unsigned int size)
 		if (esize < region_size)
 			continue;
 		serial_puts(SERIAL_COM1, "mtrr: LFB already write-combining\n");
-		return;
+		if (wc_base)
+			*wc_base = base;
+		if (wc_size)
+			*wc_size = esize;
+		return 1;
 	}
 
 	/* Only ever use a free variable entry.  BIOS-populated entries may
@@ -238,7 +246,7 @@ void mtrr_set_wc(unsigned int base, unsigned int size)
 		if (row >= 0)
 			kdbg_info((unsigned int)row, buf);
 		serial_puts(SERIAL_COM1, "mtrr: no free variable entry; skipping WC\n");
-		return;
+		return 0;
 	}
 
 	/* PhysMask bits [35:12] are compare bits: 1 = the address bit must
@@ -273,4 +281,10 @@ void mtrr_set_wc(unsigned int base, unsigned int size)
 	serial_puts(SERIAL_COM1, " +");
 	serial_puthex(SERIAL_COM1, region_size);
 	serial_puts(SERIAL_COM1, "]\n");
+
+	if (wc_base)
+		*wc_base = base;
+	if (wc_size)
+		*wc_size = region_size;
+	return 1;
 }

+ 7 - 3
mtrr.h

@@ -2,9 +2,13 @@
 #ifndef MTRR_H
 #define MTRR_H
 
-/* Program one variable-range MTRR as write-combining over [base, base+size).
- * No-op if the CPU lacks MTRR, has no free variable ranges, or the range
+/* Program one variable-range MTRR as write-combining over [base, base+size),
+ * or reuse an entry an earlier boot already left there.  Returns 1 when a WC
+ * range covering the request is in place and fills wc_base and wc_size with the
+ * range that entry actually covers (it is size-aligned and rounded up);
+ * returns 0 if the CPU lacks MTRR, has no free variable ranges, or the range
  * cannot be covered by a single size-aligned region. */
-void mtrr_set_wc(unsigned int base, unsigned int size);
+int mtrr_set_wc(unsigned int base, unsigned int size,
+		unsigned int *wc_base, unsigned int *wc_size);
 
 #endif

+ 64 - 17
paging.c

@@ -27,6 +27,50 @@
 #define LFB_MAP_SIZE 0x1000000
 
 static unsigned int *page_dir;
+/* LFB page-type decision, kept so it can be re-applied after other MMIO
+ * mappings (the GPU BARs cover the framebuffer address) have set PCD. */
+static unsigned int lfb_wc_base, lfb_wc_size;
+static int lfb_wc_ok;
+
+/* Write the LFB PDEs: WC-covered pages cacheable (effective type WC), the
+ * rest uncacheable. */
+static void lfb_apply_pages(unsigned int *pd)
+{
+	const volatile struct vga_boot_fb_info *fb = VGA_BOOT_FB;
+	unsigned int base, p0, p1, i;
+
+	__asm__ volatile ("" : "+r"(fb));
+	if (!fb->valid || !fb->lfb_base)
+		return;
+	base = fb->lfb_base;
+	p0 = base >> 22;
+	p1 = (base + LFB_MAP_SIZE) >> 22;
+	for (i = p0; i < p1 && i < 1024; i++) {
+		unsigned int page = i << 22;
+		unsigned int flags = PDE_PS | PDE_WRITABLE | PDE_PRESENT;
+
+		if (!lfb_wc_ok || page < lfb_wc_base ||
+		    page + 0x400000 > lfb_wc_base + lfb_wc_size)
+			flags |= PDE_PCD;
+		pd[i] = page | flags;
+	}
+}
+
+/* Re-apply the framebuffer mapping; call after any MMIO mapping that may have
+ * covered the same addresses (the NVIDIA BAR3 aperture includes the VBE LFB). */
+void paging_lfb_refresh(void)
+{
+	if (!lfb_wc_ok || !page_dir)
+		return;
+	lfb_apply_pages(page_dir);
+	serial_puts(SERIAL_COM1, "paging: LFB write-combining re-applied\n");
+}
+
+/* 1 when the LFB is mapped for write-combining (WC MTRR + cacheable pages). */
+int paging_lfb_wc(void)
+{
+	return lfb_wc_ok;
+}
 
 void paging_init(void)
 {
@@ -53,24 +97,27 @@ void paging_init(void)
 	for (i = 0; i < MM_MAX_PHYS / 0x400000; i++)
 		page_dir[i] = (i << 22) | PDE_PS | PDE_WRITABLE | PDE_PRESENT;
 
-	/* Identity-map the framebuffer (MMIO, uncacheable) so the screen works. */
+	/* Identity-map the framebuffer and make VRAM writes batch.  Use the
+	 * actual framebuffer size (not the whole mapping) so the WC range
+	 * cannot swallow unrelated MMIO, and install the MTRR *before* choosing
+	 * the page type: a PCD=1 page is uncacheable no matter what the MTRRs
+	 * say, which made every VRAM dword a separate PCI write (~30 MB/s).
+	 * Only the pages fully covered by the WC entry become cacheable (their
+	 * effective type is then WC); everything else stays UC. */
 	if (fb->valid && fb->lfb_base) {
-		unsigned int base = fb->lfb_base;
-		unsigned int p0 = base >> 22;
-		unsigned int p1 = (base + LFB_MAP_SIZE) >> 22;
-		for (i = p0; i < p1 && i < 1024; i++)
-			page_dir[i] = (i << 22) | PDE_PS | PDE_PCD |
-				      PDE_WRITABLE | PDE_PRESENT;
-		/* Mark the framebuffer write-combining so VRAM writes batch.
-		 * Use the actual framebuffer size (not the whole mapping) so
-		 * the WC range cannot swallow unrelated MMIO. */
-		{
-			unsigned int wc_size = (unsigned int)fb->pitch *
-					       fb->height;
-			if (wc_size == 0 || wc_size > LFB_MAP_SIZE)
-				wc_size = LFB_MAP_SIZE;
-			mtrr_set_wc(base, wc_size);
-		}
+		unsigned int want = (unsigned int)fb->pitch * fb->height;
+
+		if (want == 0 || want > LFB_MAP_SIZE)
+			want = LFB_MAP_SIZE;
+		lfb_wc_ok = mtrr_set_wc(fb->lfb_base, want,
+					&lfb_wc_base, &lfb_wc_size);
+		lfb_apply_pages(page_dir);
+		/* Tell the truth about VRAM writes: with cacheable pages the WC
+		 * MTRR applies (fast bursts); otherwise every dword is a separate
+		 * uncached write. */
+		serial_puts(SERIAL_COM1, lfb_wc_ok
+			    ? "paging: LFB write-combining\n"
+			    : "paging: LFB uncached (no WC MTRR)\n");
 	}
 
 	/* Load CR3, enable PSE, then PG + WP.  The jump flushes the prefetch

+ 4 - 0
paging.h

@@ -3,6 +3,10 @@
 #define PAGING_H
 
 void paging_init(void);
+/* Re-apply the framebuffer page types (after GPU BAR mappings that cover it). */
+void paging_lfb_refresh(void);
+/* 1 when the LFB is mapped for write-combining. */
+int paging_lfb_wc(void);
 
 /* Physical address of the bootstrap (current) page directory. */
 unsigned int paging_current_cr3(void);

+ 2 - 0
terminal.c

@@ -13,6 +13,7 @@ extern void program_reboot_main(void);
 extern void program_poweroff_main(void);
 extern void program_run_main(const char *filename);
 extern void program_display_main(void);
+extern void program_gfxbench_main(void);
 extern void program_serial_loop_main(const char *arg);
 extern void program_serial_recv_main(const char *arg);
 extern void program_run_serial_main(const char *arg);
@@ -95,6 +96,7 @@ static const struct kcmd cmds[] = {
 	{ "reboot",     "reboot",                        CK_CUSTOM, 0, 0, 0, 0, cmd_reboot },
 	{ "poweroff",   "poweroff",                      CK_CUSTOM, 0, 0, 0, 0, cmd_poweroff },
 	{ "display",    "display",                       CK_SPAWN, 0, program_display_main, 0, 0, 0 },
+	{ "gfxbench",   "gfxbench",                      CK_SPAWN, 0, program_gfxbench_main, 0, 0, 0 },
 	{ "lsusb",      "lsusb",                         CK_SPAWN, 0, program_lsusb_main, 0, 0, 0 },
 	{ "format",     "format <device>",               CK_CUSTOM, 0, 0, 0, 0, cmd_format },
 	{ "lspci",      "lspci [id]",                    CK_SPAWN_ARG, 0, 0, program_lspci_main, 0, 0 },

+ 64 - 4
vga.c

@@ -5,6 +5,7 @@
 #include "pci.h"
 #include "gpu.h"
 #include "ati.h"
+#include "paging.h"
 #include "process.h"
 #include "serial.h"
 
@@ -405,6 +406,17 @@ void vga_fb_read_rect(unsigned int x, unsigned int y, unsigned int w, unsigned i
 	pitch = vga_shadow_pitch;
 	pb = vga_pixel_bytes();
 	fb = vga_shadow_buf;
+	if (pb == 4) {
+		for (row = 0; row < h; row++) {
+			const unsigned int *src = (const unsigned int *)
+				(fb + (unsigned long)(y + row) * pitch + x * 4);
+			unsigned int *dst = (unsigned int *)
+				(buf + (unsigned long)row * w * 4);
+			for (col = 0; col < w; col++)
+				dst[col] = src[col];
+		}
+		return;
+	}
 	for (row = 0; row < h; row++) {
 		src = fb + (unsigned long)(y + row) * pitch + x * pb;
 		for (col = 0; col < w * pb; col++)
@@ -423,11 +435,23 @@ void vga_fb_write_rect(unsigned int x, unsigned int y, unsigned int w, unsigned
 		return;
 	pitch = vga_shadow_pitch;
 	pb = vga_pixel_bytes();
+	if (pb == 4) {
+		for (row = 0; row < h; row++) {
+			unsigned int *dst = (unsigned int *)
+				(vga_shadow_buf + (unsigned long)(y + row) * pitch + x * 4);
+			const unsigned int *src = (const unsigned int *)
+				(buf + (unsigned long)row * w * 4);
+			for (col = 0; col < w; col++)
+				dst[col] = src[col];
+		}
+		goto mark;
+	}
 	for (row = 0; row < h; row++) {
 		dst = vga_shadow_buf + (unsigned long)(y + row) * pitch + x * pb;
 		for (col = 0; col < w * pb; col++)
 			dst[col] = buf[row * w * pb + col];
 	}
+mark:
 	{
 		unsigned int xe = x + w;
 		if (xe > vga_fb_w)
@@ -449,15 +473,17 @@ void vga_clear_rgb(unsigned int r, unsigned int g, unsigned int b)
 
 /* ---- Shadow buffer init and flush ---- */
 
-/* Assembly row copy: rep movsl optimized for P5 USWC write-combining to UC memory. */
-extern void vga_fb_copy_row(void *dst, void *src, unsigned int dwords);
-
 void vga_shadow_init(void)
 {
 	vga_cache_boot_fb();
 	if (!vga_fb_lfb || vga_fb_w == 0 || vga_fb_h == 0)
 		return;
 
+	/* The GPU probe (inside vga_cache_boot_fb) maps its BARs -- including the
+	 * NVIDIA VRAM aperture, which covers the VBE LFB -- uncacheable, so the
+	 * framebuffer mapping is re-applied once everything is up. */
+	paging_lfb_refresh();
+
 	vga_shadow_buf = (unsigned char *)VGA_SHADOW_BASE;
 	vga_shadow_pitch = vga_fb_pitch_val;
 	vga_shadow_size = vga_fb_h * vga_shadow_pitch;
@@ -536,18 +562,52 @@ static unsigned int shadow_copy_damage(unsigned int max_bytes)
 	return bytes;
 }
 
+/* Set while a process is inside shadow_copy_damage().  The flush walks and
+ * trims a shared damage list, so it must not be entered twice: a process can
+ * be preempted mid-flush (a full-screen copy takes several milliseconds) and
+ * the scheduler may well run another process that flushes too. */
+static int present_busy;
+
+/* Bytes copied by the presenter so far (see vga.h). */
+unsigned int vga_present_copied;
+
+static inline unsigned int irq_save_disable(void)
+{
+	unsigned int flags;
+
+	__asm__ volatile ("pushfl; popl %0; cli" : "=r"(flags));
+	return flags;
+}
+
+static inline void irq_restore(unsigned int flags)
+{
+	__asm__ volatile ("pushl %0; popfl" :: "r"(flags));
+}
+
 /* Copy up to `max_bytes` of the pending damage to VRAM.  Called by the main
  * loop on every iteration: small updates (a cursor move is ~1 KB) reach the
  * screen within one loop instead of waiting for a fixed 30 Hz tick, while a
  * full-screen repaint is chunked so it cannot stall input. */
 void vga_present_shadow(unsigned int max_bytes)
 {
+	unsigned int flags;
+
 	vga_cache_boot_fb();
 	if (!vga_shadow_buf || !vga_fb_lfb)
 		return;
 	if (!damage_any())
 		return;
-	shadow_copy_damage(max_bytes);
+	flags = irq_save_disable();
+	if (present_busy) {
+		/* Another process is mid-flush; its remaining damage stays
+		 * queued and is copied by whoever flushes next. */
+		irq_restore(flags);
+		return;
+	}
+	present_busy = 1;
+	irq_restore(flags);
+	vga_present_copied += shadow_copy_damage(max_bytes);
+	present_busy = 0;
 }
 
 /* Make drawn content visible: the backend's present() (the software one copies

+ 9 - 0
vga.h

@@ -68,6 +68,10 @@ void vga_target_damage(unsigned int x, unsigned int y, unsigned int w,
  * the pending damage rects from the shadow buffer to VRAM. */
 void vga_present_shadow(unsigned int max_bytes);
 
+/* Bytes the presenter has copied in total (lets a benchmark measure the real
+ * work even when the flush is skipped because another process is in one). */
+extern unsigned int vga_present_copied;
+
 void vga_put_pixel_rgb(unsigned int x, unsigned int y, unsigned int r, unsigned int g, unsigned int b);
 void vga_fill_rect(unsigned int x, unsigned int y, unsigned int w, unsigned int h,
 		unsigned int r, unsigned int g, unsigned int b);
@@ -91,6 +95,11 @@ void vga_fb_flush(void);
  * within one loop, large ones are chunked so they cannot stall input. */
 void vga_fb_flush_budget(unsigned int max_bytes);
 
+/* Low-level row helpers from vga_copy.S (the blit backend uses the fill). */
+void vga_fb_copy_row(void *dst, const void *src, unsigned int dwords);
+void vga_fb_fill_row_sse2(void *dst, unsigned int value, unsigned int dwords);
+void vga_fb_fill_row_nt(void *dst, unsigned int value, unsigned int dwords);
+
 /* Per-iteration flush budget: large enough that a window-sized repaint
  * (~0.7 MB) is flushed as one frame, small enough that a full-screen repaint
  * is chunked instead of stalling input behind it. */

+ 66 - 0
vga_copy.S

@@ -21,3 +21,69 @@ vga_fb_copy_row:
 	popl %edi
 	popl %esi
 	ret
+
+/* void vga_fb_fill_row_sse2(void *dst, unsigned int value, unsigned int dwords);
+ *
+ * 16 bytes of the (broadcast) dword per iteration, then the remaining dwords.
+ * Only called when cpuid reported SSE2; unaligned-safe (movdqu). */
+.globl vga_fb_fill_row_sse2
+vga_fb_fill_row_sse2:
+	pushl %edi
+	movl  8(%esp), %edi	/* dst */
+	movl  12(%esp), %eax	/* value */
+	movl  16(%esp), %ecx	/* dword count */
+	movd  %eax, %xmm0
+	pshufd $0, %xmm0, %xmm0
+	movl  %ecx, %edx
+	shrl  $2, %edx
+	jz    2f
+1:	movdqu %xmm0, (%edi)
+	addl  $16, %edi
+	decl  %edx
+	jnz   1b
+2:	andl  $3, %ecx
+	jz    4f
+3:	movl  %eax, (%edi)
+	addl  $4, %edi
+	decl  %ecx
+	jnz   3b
+4:	popl  %edi
+	ret
+
+/* void vga_fb_fill_row_nt(void *dst, unsigned int value, unsigned int dwords);
+ *
+ * Streaming (non-temporal) store version for long rows: avoids the
+ * read-for-ownership traffic of a cached fill, which is what makes large
+ * fills DRAM-bound on the Atom.  Aligns to 16 bytes with normal stores, then
+ * movntdq.  The caller must sfence before the data is read back. */
+.globl vga_fb_fill_row_nt
+vga_fb_fill_row_nt:
+	pushl %edi
+	movl  8(%esp), %edi	/* dst */
+	movl  12(%esp), %eax	/* value */
+	movl  16(%esp), %ecx	/* dword count */
+	movd  %eax, %xmm0
+	pshufd $0, %xmm0, %xmm0
+1:	testl $15, %edi
+	jz    2f
+	testl %ecx, %ecx
+	jz    9f
+	movl  %eax, (%edi)
+	addl  $4, %edi
+	decl  %ecx
+	jmp   1b
+2:	movl  %ecx, %edx
+	shrl  $2, %edx
+	jz    4f
+3:	movntdq %xmm0, (%edi)
+	addl  $16, %edi
+	decl  %edx
+	jnz   3b
+4:	andl  $3, %ecx
+	jz    9f
+5:	movl  %eax, (%edi)
+	addl  $4, %edi
+	decl  %ecx
+	jnz   5b
+9:	popl  %edi
+	ret