BGP PIC vs ECMP
on a Full-Table Edge
The hidden encap limit behind add-path, BGP PIC and ECMP on a 6PE edge
Customer Story
BGP PIC and ECMP both start with the same step, a second BGP path on every edge router. At Fiberway Networks, a French network and data center operator we work with, that step also exposed a hardware limit that no route table showed.
We rolled out add-path on the route reflectors so the edges could install BGP PIC backup paths and, where the paths allowed it, ECMP. The same afternoon the second paths arrived, some prefixes stopped programming on the Jericho-based linecards, while every route table was green. The cause was a single egress encap bank that had been full since July 2025.
Engineering
Client: Fiberway Networks
Industry: Service Provider
Services: Professional Network Services, 24/7 NOC
Technologies: BGP, Add-Path, BGP PIC, ECMP, 6PE, MPLS, Cisco IOS-XR, Juniper vRR
Why add-path and BGP PIC
In an iBGP network built on route reflectors, each edge router normally learns one path per prefix, the one the route reflector picked as best. When that exit fails, the edge has nothing to fall back on until BGP withdraws the route and the route reflector sends a new best path. On a full table, BGP convergence takes time, and traffic drops while it happens.
With add-path (RFC 7911), the route reflectors advertise more than one path per prefix, so every edge already holds an alternative before anything breaks. What the edge does with that second path is a design choice:
- BGP PIC (Prefix Independent Convergence) keeps one path forwarding and pre-installs the second in the forwarding table as a backup. When the active next hop fails, the linecard switches locally in milliseconds, and the switchover time does not depend on how many prefixes use that next hop.
- ECMP forwards on every equal-cost path at once and hashes flows across them, so the edge uses the capacity of all its exits.
Architecture overview
The design has three parts:
- Route reflectors on Juniper vRR advertise up to two paths per prefix with add-path send.
- Cisco IOS-XR edge routers negotiate add-path receive with the route reflectors and keep every path they receive.
- On the edges,
maximum-pathsenables ECMP where the paths tie, and an additional-paths selection policy installs a BGP PIC backup where they do not.
The network carries the full IPv4 and IPv6 tables. IPv6 runs as labeled unicast over the IPv4 MPLS core (6PE, RFC 4798). That detail caused trouble twice, first on the route reflectors and later on the linecards.
Getting the second path to the edges
The first surprise was on the route reflectors. Each one received the same IPv6 prefix from four edge routers, yet advertised only one path, even with add-path set to send two. Three of the four paths were marked Unusable.
Add-path only advertises extra paths that are usable, and these were not. 6PE next hops are IPv4-mapped IPv6 addresses, and Junos resolves them in inet.3, the MPLS table. The route reflectors sit outside the forwarding path, run no MPLS, and had an empty inet.3. The active path still went out, because a route reflector always reflects its best path, but nothing qualified as a second one.
Pointing the inet6.0 resolution at inet.0 does not work, because Junos only accepts resolution tables of the same address family. The fix was to make the PE loopbacks resolvable in inet.3 by leaking the IGP routes into it. Once the next hops resolved, the paths became usable and the route reflectors started sending a second path.
On the edges, the matching settings belong under address-family ipv6 labeled-unicast. We had them under ipv6 unicast at first, where they do nothing for labeled routes. After the move, the edges negotiated add-path and started receiving the extra paths.
Why the edge got BGP PIC instead of ECMP
We expected ECMP toward two exits. BGP installed one path with a BGP PIC backup instead. For BGP multipath, the candidate paths must tie on local preference, MED, AS-path length and the IGP metric to the next hop. Everything tied except the last one. From that edge, one next hop sat at IGP metric 11 and the other at 21.
Equalizing the IGP metric would have produced ECMP, and it would also have changed how every other prefix routes through that part of the network. Where one exit really is closer, a BGP PIC backup is the better fit. The closer exit carries the traffic, and the backup is already programmed if it fails.
The two options use the same links very differently, and they behave differently when one of those links fails.
The linecards that would not program
The same afternoon the route reflectors began sending second paths, the edge linecards started logging hardware programming failures.
%PLATFORM-DPA-3-ERROR : IPv4 route <prefix> update to hardware failed for
NPU# 0, 'DPA' detected the 'warning' condition 'SDK - Entry not found'
%PLATFORM-PLAT_FIB-3-HW_PROG_ERROR : HW Programming failed for table,iproute,
... failure-reason,EnNotFoundThe RIB was healthy. The route tables in hardware were green, the busiest at 53 percent with just over a million IPv4 routes in the external TCAM. The problem was one egress table.
Encap bank 0 sat at 100 percent and in OOR Red on every NPU, and its timestamps showed it had been there since July 2025, long before anything needed new space. The IPv6 table runs as 6PE, so about 245,000 IPv6 prefixes impose a label on egress, and label imposition uses encap entries. Traffic kept flowing because the entries already in place still worked. Once the edges had BGP PIC backups and second paths to program, some prefixes needed new entries, and the bank had no room left.
One detail we still can’t fully explain. Bank 7 has the same phase as bank 0 on the same NPU and held 8 of its 4,096 entries, and the failures happened anyway.
Results
- The edges receive a second path for every prefix through add-path from the route reflectors.
- BGP PIC backup paths are installed where the paths do not tie, so failover no longer waits for BGP to reconverge.
- The programming failures were traced to a single egress encap bank in OOR Red, with every route table green.
Lessons from production
- Watch the egress tables as closely as the route tables. Next-hop groups, FEC entries and encap rewrites sit in hardware tables far smaller than LEM, LPM or the external TCAM, and every BGP PIC backup and every ECMP member uses them. Check
show controllers npu resourcesbefore any change that adds paths, and alert on out-of-resource state changes, yellow as well as red. - Check the IGP metric before you promise ECMP. Local preference, MED and AS-path length often tie. The IGP metric to the next hop is usually what decides it.
- Make next hops resolvable on the route reflectors. A route reflector outside the forwarding path still has to resolve next hops before add-path will advertise extra paths. For 6PE that means
inet.3. - Put the configuration under the address family the routes use. Settings under
ipv6 unicastdo nothing foripv6 labeled-unicast. - Install backups where they pay off. On merchant silicon carrying full tables, a BGP PIC backup for every prefix is expensive. Protect the prefixes that need fast failover.
Where this fits
Add-path with BGP PIC suits operators that want failover in milliseconds without waiting on BGP. ECMP suits networks whose exits really are equal. On merchant silicon with full tables, check the egress tables before you turn either one on network-wide, because that is where the extra paths use hardware.
If you want a second pair of eyes on your route reflector design or your add-path, BGP PIC or ECMP rollout, or you want it designed and operated for you, talk to us.
FAQ
What is the difference between BGP PIC and ECMP?
BGP PIC installs one active path plus an idle, pre-installed backup, so the linecard can switch in milliseconds when the active next hop fails. ECMP installs several equal-cost paths that all forward at once and share the load. Many networks run both, ECMP where the paths are equal and a BGP PIC backup where they are not.
Why would routes fail to program when the routing table is not full?
The routing table and the forwarding hardware are separate resources. Backup paths and ECMP members consume egress tables such as next-hop groups, FEC entries and encapsulation rewrites, which are much smaller than the route tables. On Cisco IOS-XR with Jericho-based linecards this showed up in our case as PLATFORM-PLAT_FIB-3-HW_PROG_ERROR with failure-reason EnNotFound, with an encap bank at 100 percent in show controllers npu resources.
Why does a route reflector mark add-path routes as Unusable?
Add-path only advertises extra paths whose next hops resolve. With 6PE the next hops are IPv4-mapped IPv6 addresses, and Junos resolves them in inet.3. A route reflector that runs no MPLS has an empty inet.3, so only the best path goes out. Making the PE loopbacks resolvable in inet.3, for example by leaking IGP routes into it, fixes it.
be our next success story!
Drop us a line! We care for your Business and are here to answer your questions 24/7