An Introduction, a Prelude If You Will#

The Palo Alto Networks PA-220 is well known for being extremely slow, prone to crashing, and generally being an all-around piece of shit. "Why is that though?", you ask? Well, put simply, it is because it contains the name "Palo Alto Networks".

We had a large surplus of PA-220s waiting to be scrapped after we migrated them out of our network for something newer, so I took a few home with me to do some lab work. This was more than three years ago now, and while I was running a pair of them for a little while, the lack of licensing - and the fact that they were long-since out of support - soon relegated them to a stack of paperweights.

Fast-forward to today and that is where they had remained, sitting either on-top, or underneath, other random junk. After working on OpenWrt support for the Ubiquiti Edgerouter (ERPoE-5, which I also have a number of from work), I wanted another bit of Octeon hardware to play with. There was one obvious candidate.

The Hardware#

At the time, I hadn't realized that the PA-220 actually had a pretty decent hardware specification. It runs:

Four cores? Eight gigabytes of memory? Are you kidding me? This is a firewall that takes 15-30 minutes to boot, by the way. This is a firewall known for crashing the management-plane on commit, by the way. But is the hardware really to blame here?

No, of course it isn't. PAN-OS 8.x did not have this problem. Back in Palo Alto Network's glory days, they actually cared about the quality of software they wrote, which led to a tight and efficient firewall system which brought them prestige of which they are still riding the coat-tails today.

"Well ackshually, PAN-OS splits the hardware into a management plane and a data plane, so the management plane is starved" Okay, and? This is bad engineering plain and simple. You should be writing a proper scheduler, not partitioning a CPU because you can't trust yourselves to write performant code. Is there a place for this? Absolutely, a large-chassis firewall pushing 100Gbps with dedicated FPGAs/ASICs, go for it, physical plane separation makes sense here! On a four-core box rated for 150Mbps of IPSec throughput? It's lazy resource management.

Don't take my word for it, take Palo Alto's words themselves!

"Upgrading a PA-220 firewall takes up to an hour or more"
"PA-220 firewalls are experiencing slower web interface and CLI performance times"
"The PA-220 firewall reaches the maximum disk usage capacity multiple [times] a day that requires a disk cleanup"

Let me just reiterate, this is a 4-core, 8 GiB memory firewall. This is a four core, eight gigabyte memory firewall. I can only shake my head in disappointment.

Getting into the nitty-gritty of the hardware, the 88E1680 provides the physical interfaces for the eight ports on the front of the unit. The management port has a separate path through the SoC.

Getting OpenWrt to Boot#

OpenWrt already supports other Octeon devices, which gave me a great starting-point. The PA-220 still needed a device tree describing the memory, ethernet interfaces, PHYs, storage, LEDs, temp sensors, and RTC. It also needed some changes to the Octeon code itself.

The first useful milestone here is getting a working OpenWrt image booting from RAM. This would let me inspect the hardware properly and test any changes before attempting an install to eMMC.

A Four-Core Firewall Should Have Four Cores#

If you'll remember from about 60 seconds ago when you read it, the PA-220 has a four-core CPU. In typical fashion, though, Palo's U-Boot reports bootinfo version 1.4 but leaves garbage in ext_core_mask (0x3000000000000000) while the legacy 32-bit core_mask is correct (0xf), so Linux brought up only one CPU. This required a kernel patch to use the legacy core_mask when the two disagree. This is one of those issues that really just confuses me more than anything else, PAN-OS boots fine so I guess they just completely ignore ext_core_mask and therefore do not care about the fact it contains garbage data. Thank you Palo Alto Networks!

What's a Firewall Without Ports?#

A lot of code needed to be written here. Thankfully in the age of AI, I do not need to bother with menial tasks like describing ethernet ports in a device tree. The 88E1680 PHY needed its reset released through the board's CPLD, the copper ports powered up, and the LED matrix configured. The factory MAC Addresses also needed assigning.

The MAC layout is the EEPROM base address + offsets 0-7 for the front ports 1-8, followed by base + 8 for MGT. OpenWrt describes these front interfaces eth1-8, and the management interface mgt. On the topic of the management... The separate route through the SoC I alluded to earlier meant getting the CN70XX AGL management port working required additional work in the Octeon code. Fun.

Storage and a Watchdog With PTSD#

It was to be expected that things wouldn't work out of the box. In DDR52 mode, the eMMC could stall during sustained reads. Fixing the device tree to disable DDR for this board let it use high-speed SDR mode properly, which did not have this issue.

Then there was the watchdog. The driver implemented a longer heartbeat using shorter hardware periods, but reported the initial 5-sec period to userspace. When procd initialized it again, the heartbeat could be reduced to five seconds, which just so happened to match the default keepalive interval... The result of this was a watchdog handover that could reset the firewall during installation or upgrades. Reporting the configured heartbeat correctly fixed this.

There was also a smaller platform change to instantiate devices below a simple-mfd node, so that the CPLD-backed status LEDs appeared through standard kernel mechanisms.

The Bootloader#

A lot of the work here had actually been done here only a couple of months ago. Lucky me! They had done a lot of work on the kernel patches however it contained a fair bit that could not be upstreamed to OpenWrt so they had kept it in their own repository.

One thing they'd done was implemented a custom U-Boot implementation to reduce boot time. That had me thinking though, why the hell is the Palo Alto Networks PA-220's boot time so slow? Well, the answer is obvious as I've said before, because it contains "Palo Alto Networks"!

The original U-Boot implementation is perfectly capable of loading and booting the OpenWrt image. It expected the kernel image in a specific place (/boot/vmlinux.oct3-mp) on the boot partition. The problem with it though is that it takes SEVENTY NINE SECONDS TO READ A 13.5 MEGABYTE KERNEL. Hello? Palo Alto Networks? It's me, 1995 calling, we want our disk read speeds back!

This is, by the way, also an issue loading the stock PAN-OS kernel. This is why the firewall takes so horribly long to boot - pure SWE incompetence. The controller continually shuts down between reads, and the shutdown and re-enable paths included fixed delays. This means that every time the controller shuts down and starts back up again, it burns time just waiting.

By temporarily patching the copy of U-Boot to bypass these repeated shutdowns, I had the same kernel loading in ~300 milliseconds. I then made this persistent using an environment record that the stock bootloader reads from eMMC. The environment commands apply the workaround to U-Boot's RAM copy while the kernel loads, and then restores the original function and normal MMC shutdown before handing control over to Linux.

pa220_mmc_crc=crc32 0xc009e2d4 0x19c 0x23000000; mw.l 0x23000004 0xea2eba96; itest.b 0 == 0
pa220_mmc_check_hi=itest.w *0x23000000 == *0x23000004
pa220_mmc_check_lo=itest.w *0x23000002 == *0x23000006
pa220_mmc_patch=mw.l 0xc009e2d4 0x03e00008; mw.l 0xc009e2d8 0; flush_dcache; inv_icache; echo PA-220 fast MMC enabled; itest.b 0 == 0
pa220_fast_mmc=run pa220_mmc_crc pa220_mmc_check_hi pa220_mmc_check_lo pa220_mmc_patch; itest.b 0 == 0
pa220_mmc_restore_crc=crc32 0xc009e2d4 0x19c 0x23000000; mw.l 0x23000004 0x255a8387; itest.b 0 == 0
pa220_mmc_restore=mw.l 0xc009e2d4 0x27bdffb0; mw.l 0xc009e2d8 0xffb10008; flush_dcache; inv_icache; mmc read 0x23000000 0x70 1; echo PA-220 MMC handoff restored; itest.b 0 == 0
pa220_mmc_finish=run pa220_mmc_restore_crc pa220_mmc_check_hi pa220_mmc_check_lo pa220_mmc_restore; itest.b 0 == 0
bootcmd=run pa220_fast_mmc; run kld; run pa220_mmc_finish; run boot

What is all this doing, you ask? The first three variables calculate and check a CRC of the original MMC function. If those checks pass, pa220_mmc_patch writes the two instructions that make the function return immediately. flush_dcache and inv_icache make those changes visible to the CPU. After the stock kld command loads the kernel, pa220_mmc_finish checks the patched function, restores the original instructions and performs an MMC read to trigger the original shutdown before boot hands over to Linux.

The reason this is filled with stupid looking halfword comparisons and final itest.b commands are due to quirks in Palo's U-Boot comparison and command-return behaviour / implementation. Do not blindly copy and paste this into another PA-220. The addresses, instruction words, and checksums involved are specific to the U-Boot build on my PA-220 (11.0.0.0-48) and could cause havoc on a different build, an incorrect environment could stop the device from booting. This is not part of the OpenWrt support patch, I was just utterly perplexed at how the firewall was taking so long to boot.

For the total boot time, a warm reboot went from 174.4 seconds-to-SSH to 95.7 seconds-to-SSH with this workaround. A physical power-cycle took 84.2 seconds-to-SSH. That's not wicked quick, and there are actually better implementations out there like the custom U-Boot implementation I mentioned earlier, which claims to drop boot times down to just 28 seconds. This isn't something I could reproduce (I measured 48.6s), but interesting nonetheless. Going even further than this and using IronDemiGod's U-Boot as a base, with a zero-second boot delay and modifications to skip reset waits for absent PHYs - with the front-port PHY held in reset - I reduced U-Boot-to-Linux handoff from 6.1 to 1.9 seconds. Then, using a 115200-baud quiet kernel commandline, I was able to bring it down to just 35.2 seconds-to-SSH. So to reiterate, we've gone from a documented fifteen to thirty minutes PAN-OS boot-time, to a 35 second boot-time. Insane.

Route, I am Your Father#

Can it actually route? Well, yeah, obviously.
By this point I had four working cores, 8 GiB of memory, and nine ethernet interfaces. Link was coming up at 1Gb/s, all looked good. On the software side of things, Octeon supports multiple receive groups, so I enabled four on the PA-220 and distributed their interrupted across it's four cores. The shared TX cleanup interrupt went to core 3.

Testing brought a couple of problems in the older driver code to the surface though, and they needed fixing.
Luckily, the first issue I identified had actually already been fixed upstream. The cleanup sweep was supposed to run once every 1024 packets. Instead what I was seeing was that it was running for 1023/1024 packets!!!!

/* Before */
if (total_to_clean & 0x3ff) {

/* After */
if ((total_to_clean & 0x3ff) == 0) {

That is a lot of cleanups. Backporting this fix improved performance significantly.

The second issue I ran into was PKO (Packet Output), queue initialization. CN70XX has 256 output queues, but the existing setup treated it as if it had 128 and omitted the appropriate queue-memory mode setup. The calculation also needed to find the highest queue across all active interfaces, the last interface in the list did not necessarily use the highest-numbered resource.

Under sustained bidirectional load, the result could be an output queue stalling or a panic in TX cleanup. Correcting the queue setup resolved those failures.

For testing, I connected two PC Ethernet interfaces in separate network namespaces to the PA-220's front ports. The traffic crossed the router's IPv4 NAT path. The main run used a 1500-byte MTU, four TCP streams in each direction, and a testing duration of 5 minutes. Software and hardware flow offloading were disabled.

Wicked. That's basically line-rate. Testing with NAPI GRO increased this throughput to around 924Mbps, which was not included in my upstream PR. Now obviously, yes, yes, these are just IPv4 TCP NAT results. I didn't benchmark VPN or traffic-inspection, so on and so on, yes I know it's "not performing the functions of the PA-220" yada yada, as if the PA-220 could actually do decrypt in the first place, and it was rated for 150Mbps IPSec throughput!

I also tested all nine ports at gigabit with full-MTU traffic, VLAN trunk/access forwarding and rejection cases, RSTP blocking and failover, classic STP interoperability, and more that I won't bother going into. Long and short of it is that it routes well.

Goodbye, PAN-OS#

Give image me give flash image me flash image give me flash image give me you
- Koko, the Network Engineer.

The changes are in OpenWrt PR #26312, as of writing it's not been merged, or even reviewed by a human. Claude Code had some good comments though.
Obvious warning that this will erase the eMMC. It's probably easy enough to repartition and get PAN-OS running again, but let's be honest, who wants to be running PAN-OS on this thing anymore after they butchered the software so badly?

Build it using the normal OpenWrt process, select "Cavium Network Octeon", the "Generic" subtarget, and the "Palo Alto Networks PA-220" profile. You're looking for these outputs:

openwrt-octeon-generic-paloaltonetworks_pa-220-initramfs-kernel.bin
openwrt-octeon-generic-paloaltonetworks_pa-220-ext4-sysupgrade.tar

Connect a console cable at 9600 baud, 8N1, with no flow control. Connect MGT to an isolated interface, I'm going to be using 192.168.1.10/24 to serve the initramfs image over TFTP.

During autoboot, enter maint to (as the name suggests) break into maintenance mode. Once in maintenance mode, select Disk Image -> Advanced Options, using the password MA1NT. Use Status to identify the active sysroot. Select the installed PAN-OS version, that sysroot and the maint component, then select Bootstrap and reboot.

We're expecting the kernel load to fail here. This allows us to drop into a U-Boot prompt, from which we can then load and boot the RAM image:

setenv ipaddr 192.168.1.1
setenv serverip 192.168.1.10
setenv ethact octrgmii0
tftpboot 0x21000000 openwrt-octeon-generic-paloaltonetworks_pa-220-initramfs-kernel.bin
bootoctlinux 0x21000000 numcores=4 endbootargs

Once OpenWrt has booted and brought up its network, copy the sysupgrade archive from the PC.

scp -O openwrt-octeon-generic-paloaltonetworks_pa-220-ext4-sysupgrade.tar [email protected]:/tmp/

At the OpenWrt console, create the partition layout, format the boot partition, and install.

sfdisk --wipe always --wipe-partitions always /dev/mmcblk0 <<'EOF'
label: dos
unit: sectors
start=2048, size=524288, type=83
start=526336, type=83
EOF
sleep 2
mke2fs -F -t ext3 -b 4096 -I 128 /dev/mmcblk0p1
sysupgrade -n /tmp/openwrt-octeon-generic-paloaltonetworks_pa-220-ext4-sysupgrade.tar

The PA-220 will reboot automatically and boot from eMMC.

So, What Have We Learned?#

Unfortunately, not a whole lot. I could say "we've learned the invaluable lesson that Palo Alto Networks don't know how to utilize the hardware in front of them", but I think that's probably well known at this point.

I suppose the most surprising thing out of all of this is that the PA-220 has 8GiB of memory. Who would have thought!