pfSense Plus Recovery After Power Outage: Netgate 8200

01. Overview

A production Netgate 8200 firewall running pfSense Plus became completely unbootable following an unexpected power outage. The device entered a kernel panic loop on every boot attempt, dropping into the FreeBSD kernel debugger (db>) before the OS could load. This case study documents every diagnostic step taken, the failed recovery attempts, and the final successful recovery path — including a creative network workaround that unblocked the installation process.

02. Root Cause

Why ZFS Corrupts on Power Loss

ZFS is a copy-on-write filesystem that writes data in transactions. If power is cut mid-transaction, the pool is left in an inconsistent state — some pointers reference data that does not yet exist. pfSense Plus uses ZFS as its primary filesystem, making it vulnerable to this exact scenario.

panic: zfs(panic): zfs: rt={spa=pfSense vdev_guid=5984199624752192702 ms_id=99
ms_allocatable}: adding segment (offset=18eac56000 size=7000) overlapping
with existing one (offset=18eac58000 size=3000)

This indicates the ZFS pool had an inconsistent segment allocation map — a direct result of an interrupted write transaction during the power outage.

Why pfSense Was Especially Vulnerable

• The Netgate 8200 uses NVMe/SSD storage with write caches that do not flush instantly on power loss
• pfSense is a live firewall — constantly writing logs, state tables, and DHCP leases even when idle
• The combination makes mid-write interruptions statistically likely during any unexpected power event

03. Symptoms & Remote Access

Symptoms

• Device powered on but immediately entered kernel panic
• Console showed KDB: enter: panic followed by db> prompt
• Every attempt to continue (cont) triggered another panic
• Boot loader could not see the ZFS pool — lszfs zroot/ROOT returned device not configured
• Only the EFI partition was accessible, not the main ZFS filesystem

Remote Access Setup

Serial console access was used throughout the recovery via:

set autoboot_delay=30

04. Diagnostic Steps

1. Kernel Debugger Assessment (db>)

• Confirmed panic type: ZFS segment range tree corruption
• Retrieved pfSense version via sysctl kern.version → pfSense Plus 23.05.1, FreeBSD 14.0
• Attempted cont to continue boot — resulted in repeated panic at same address

2. Boot Loader Diagnostics (OK prompt)

• Accessed boot loader by catching the pfSense boot menu and pressing Esc
• lszfs zroot/ROOT → returned device not configured — pool completely invisible
• ls / showed only EFI partition contents — ZFS filesystem inaccessible

3. EFI Boot Manager Discovery

Identified an early EFI Boot Manager menu appearing before the pfSense menu with options for r (clear boot order), p (PXE boot), and F2 (Boot Manager Menu). This menu later became the path used to boot the USB installer.

05. Recovery Attempts

set vfs.zfs.recover=1
set vfs.zfs.unsafe_reads=1
boot -s

set vfs.zfs.recover=1
set vfs.zfs.debug=1
boot

06. Config File Recovery

Once booted into the pfSense installer recovery shell, the priority was retrieving the existing configuration before reinstallation wiped the drive.

Step 1 — Mount the Corrupted Pool Read-Only

zpool export pfSense
zpool import -f -o readonly=on -R /tmp pfSense

The -f flag forces import despite corruption. readonly=on prevents writes that could worsen the state. The pool imported successfully.

Step 2 — Locate and Mount the Config Dataset

The ZFS datasets are not automatically mounted into the path hierarchy — they need to be mounted manually. The config lives in pfSense/ROOT/default/cf.

mkdir /tmp/cf
zfs mount pfSense/ROOT/default/cf
ls -lah /tmp/cf/conf    # config.xml (192KB), backup/, rules.debug.old

Step 3 — Copy Config to USB

mkdir /media
mount -t msdosfs /dev/da1s1 /media
cp /tmp/cf/conf/config.xml /media/
cp /tmp/cf/conf/config.xml /media/config_backup.xml
grep "<lastchange>" /tmp/cf/conf/config.xml    # verify not empty

07. OS Reinstallation

The Network Validation Problem

The pfSense Plus installer attempts to contact Netgate's activation servers during installation, requiring internet on the WAN interface. Connecting WAN to a standard home router at 192.168.1.1 caused a subnet conflict: pfSense's default LAN is also 192.168.1.1/24, creating a routing loop. The installer looped back to the network configuration screen indefinitely.

Attempts to resolve this by modifying network settings inside the recovery console failed — exiting the recovery shell triggered a full restart of the installation, clearing all changes.

Solution — Android USB Tethering

Installation Steps

1. Boot from USB via EFI Boot Manager (F2)
2. Accept copyright notice → Enter
3. Select: Install pfSense Plus
4. Partitioning: Auto ZFS
5. ZFS configuration: Stripe (single disk)
6. Select drive: nda0 (internal NVMe)
7. Confirm: destroy existing data → YES
8. Wait ~5–10 minutes
9. Remove USB when prompted → Reboot

08. Config Restore & Verification

• Connected a laptop to the LAN port, browsed to https://192.168.1.1
• Logged in with default credentials: admin / pfsense
• Navigated to Diagnostics → Backup & Restore
• Uploaded config.xml from USB — pfSense applied config and rebooted
• Verified all interfaces, firewall rules, NAT, VPN, and DHCP settings intact

Recovery Summary

09. Key Lessons & Prevention

Key Lessons

Prevention

• UPS on all network infrastructure — even a basic unit (€50–100) prevents this scenario entirely
• Remote config backups — pfSense Auto Config Backup stores up to 100 encrypted snapshots automatically on every config change
• Pre-made installer USB — keep a current pfSense Plus installer USB stored on-site, ready to use
• Document Netgate serial numbers and account credentials before an incident — the installer requires a valid Netgate account

10. References & Resources

Official documentation and tools referenced during this recovery.

• Netgate Installer DownloadOfficial pfSense Plus installer download portal. Requires a valid Netgate account with the device serial number claimed. Download the AMD64 ISO matching your installed version.
• Netgate Installer DocumentationOfficial guide to preparing and booting the pfSense Plus installer USB, including EFI boot menu navigation and installation options.
• Install Config Recovery — pfSense DocsDocumentation covering config recovery options available during the installation process, including accessing the recovery shell to retrieve existing configuration files.
• Configuration Restore — pfSense DocsStep-by-step guide for restoring a saved config.xml backup via Diagnostics → Backup & Restore after a clean OS installation.
• Balena EtcherCross-platform USB flashing tool used to write the pfSense Plus ISO to a USB drive. Reliable alternative to Rufus on non-Windows systems.