OAM & Troubleshooting¶

Effective OAM (Operations, Administration, and Maintenance) is critical for operating SRv6 networks. This page covers how traditional OAM tools work in SRv6, plus SRv6-native OAM mechanisms.

SRv6 Ping¶

SRv6 ping uses ICMPv6 Echo Request/Reply with the SRH to verify reachability along a specific SRv6 path.

How It Works¶

1. Source sends ICMPv6 Echo Request with an SRH containing the desired path
2. Packet follows the segment list (same as data traffic)
3. The target SID node generates an ICMPv6 Echo Reply
4. Reply returns via normal IPv6 routing (not the reverse SRv6 path)

sequenceDiagram
    participant S as Source
    participant A as Node A (End)
    participant B as Node B (End)
    participant T as Target SID

    S->>A: ICMPv6 Echo Request + SRH[T, B, A]
    A->>B: Forward (SL decremented)
    B->>T: Forward (SL decremented)
    T->>S: ICMPv6 Echo Reply (normal IPv6 routing)

Verification Commands¶

!! Ping a specific SRv6 SID
ping ipv6 fc00:0:2::100 source fc00:0:1::1

!! Ping through an SRv6 path (with segment list)
ping sr-mpls nil-fec labels fc00:0:2::1,fc00:0:3::100

# Ping an SRv6 SID directly
ping6 fc00:0:2::100

# Ping with SRv6 encapsulation through specific path
# (requires custom route or scapy)
ip route add fc00:0:2::100/128 encap seg6 mode encap \
  segs fc00:0:2::1,fc00:0:3::100 dev eth0
ping6 fc00:0:2::100

SRv6 Traceroute¶

SRv6 traceroute discovers each hop along an SRv6 path by incrementing the IPv6 Hop Limit:

1. Send packet with Hop Limit = 1 → first hop returns ICMPv6 Time Exceeded
2. Send packet with Hop Limit = 2 → second hop returns Time Exceeded
3. Continue until the target SID responds

Key Difference from IP Traceroute¶

SRv6 traceroute reveals both SRv6-aware and transit nodes:

• Transit nodes (non-SRv6) return Time Exceeded based on the IPv6 DA
• SRv6 endpoint nodes process the SID and update the DA before forwarding

This means traceroute shows the actual forwarding path, including underlay hops between SRv6 nodes.

IOAM Deep Dive (RFC 9326)¶

In-situ OAM (IOAM) embeds telemetry data directly into live data packets as they traverse the network. Unlike traditional OAM that sends separate probe packets, IOAM instruments the actual traffic, providing accurate measurements that reflect real forwarding behavior including load-balanced paths.

IOAM Option Types¶

RFC 9326 defines four IOAM option types, each designed for different telemetry use cases:

Type 0: Pre-allocated Trace¶

The ingress node allocates a fixed-size trace buffer with empty slots for each expected hop. As the packet transits the network, each IOAM-capable node fills in its designated slot.

 Ingress allocates N slots         Each transit node fills its slot
 ┌──────┬──────┬──────┬──────┐     ┌──────┬──────┬──────┬──────┐
 │ Empty│ Empty│ Empty│ Empty│ --> │Node 4│Node 3│Node 2│Node 1│
 └──────┴──────┴──────┴──────┘     └──────┴──────┴──────┴──────┘
  Slot 0  Slot 1 Slot 2 Slot 3     Filled right-to-left (LIFO)

Type 1: Incremental Trace¶

Each IOAM-capable node appends its telemetry data to the header. The packet grows at each hop.

 Ingress (minimal header)      After Node 1          After Node 2
 ┌─────────┐                ┌─────────┬──────┐   ┌─────────┬──────┬──────┐
 │ IOAM Hdr│       -->      │ IOAM Hdr│Node 1│-->│ IOAM Hdr│Node 1│Node 2│
 └─────────┘                └─────────┴──────┘   └─────────┴──────┴──────┘

Type 2: Proof of Transit (POT)¶

Provides a cryptographic proof that a packet has traversed a specific set of nodes. Each node updates a cumulative value using a shared secret and a polynomial-based scheme (Shamir's Secret Sharing).

 Ingress sets            Node 1 updates        Node 2 updates       Egress verifies
 initial CML value       CML with its share     CML with its share   CML == expected
 ┌─────┐                ┌─────┐               ┌─────┐              ┌─────┐
 │CML=0│      -->       │CML=a│     -->       │CML=b│    -->       │CML=S│ ✓
 └─────┘                └─────┘               └─────┘              └─────┘

Type 3: Edge-to-Edge (E2E)¶

Carries data that is written by the ingress node and read by the egress node. No intermediate node modifies E2E data.

IOAM Option Types Summary¶

IOAM Data Fields¶

Each trace node data entry (used by Type 0 and Type 1) can include any combination of the following fields, selected via a 24-bit IOAM-Trace-Type bitmask:

IOAM Namespaces¶

IOAM uses namespaces (16-bit Namespace-ID) to isolate IOAM domains and allow multiple independent IOAM deployments within the same network.

 ┌─────────────────────────────────────────────────────┐
 │                  Physical Network                    │
 │                                                      │
 │  ┌──────────────────────┐  ┌──────────────────────┐ │
 │  │  Namespace 100       │  │  Namespace 200        │ │
 │  │  (Production SLA)    │  │  (Lab/Testing)        │ │
 │  │                      │  │                       │ │
 │  │  IOAM fields:        │  │  IOAM fields:         │ │
 │  │  - node_id           │  │  - node_id            │ │
 │  │  - timestamp          │  │  - queue_depth        │ │
 │  │  - transit_delay      │  │  - buffer_occupancy   │ │
 │  └──────────────────────┘  └──────────────────────┘ │
 └─────────────────────────────────────────────────────┘

Key namespace behaviors:

• A node that does not recognize the Namespace-ID in an IOAM header must not modify the IOAM data fields -- it forwards the packet unchanged
• Different namespaces can request different sets of data fields (e.g., namespace 100 requests timestamps, namespace 200 requests queue depth)
• Namespace 0x0000 is the default namespace -- all IOAM-capable nodes must support it
• Namespaces enable multi-tenant IOAM: each VPN or service can have its own telemetry domain

IOAM in SRv6 (draft-ietf-ippm-ioam-srv6-options)¶

How IOAM Is Encapsulated in SRv6¶

There are two encapsulation options:

Option A: IPv6 Hop-by-Hop Options Header (most common for trace types)

Used for IOAM Pre-allocated Trace, Incremental Trace, and Proof of Transit, because these require processing at every hop.

Option B: IPv6 Destination Options Header

Used for IOAM Edge-to-Edge data, because E2E data is only read at the destination (egress) node.

Packet Structure¶

 ┌─────────────────────────────────────────────────────────────┐
 │                     IPv6 Header                              │
 │  Next Header = Hop-by-Hop (0)                               │
 │  SA = Encapsulating node     DA = Active SID                │
 ├─────────────────────────────────────────────────────────────┤
 │              Hop-by-Hop Options Header                       │
 │  Next Header = Routing (43)                                 │
 │  ┌─────────────────────────────────────────────────────┐    │
 │  │  IOAM Trace Option (Type 0 or 1)                    │    │
 │  │  ┌─────────────────────────────────────────────┐    │    │
 │  │  │ Namespace-ID │ NodeLen │ Flags │ IOAM-Trace │    │    │
 │  │  │ RemainingLen │ Type    │       │ Data [...] │    │    │
 │  │  └─────────────────────────────────────────────┘    │    │
 │  └─────────────────────────────────────────────────────┘    │
 ├─────────────────────────────────────────────────────────────┤
 │              Segment Routing Header (SRH)                    │
 │  Segments Left = N                                          │
 │  Segment List [SID_0, SID_1, ..., SID_N]                   │
 ├─────────────────────────────────────────────────────────────┤
 │                     Payload                                  │
 └─────────────────────────────────────────────────────────────┘

IOAM vs SRH Processing at Each Node¶

The following table clarifies what happens at each type of node:

flowchart LR
    A["Ingress<br/>(IOAM encap +<br/>SRH encap)"] --> B["Transit Node<br/>(IOAM: write trace<br/>SRH: ignore)"]
    B --> C["SRv6 Endpoint<br/>(IOAM: write trace<br/>SRH: update DA)"]
    C --> D["Transit Node<br/>(IOAM: write trace<br/>SRH: ignore)"]
    D --> E["Egress<br/>(IOAM: extract & export<br/>SRH: decap)"]

Configuration Example¶

!! Enable IOAM on an SRv6 policy
segment-routing
 traffic-eng
  policy POLICY1
   ioam
    trace pre-allocated
     namespace 100
     node-data timestamp ingress-if egress-if transit-delay
    !
   !
  !
!

!! Verify IOAM state
show ioam trace profile
show ioam namespace

# Enable IOAM pre-allocated trace on an interface
# (requires kernel 5.15+ with IOAM support)
ip ioam namespace add 100
ip ioam schema add 1 "Production"

# Attach IOAM to an SRv6 route
ip -6 route add fc00:0:3::/48 encap seg6 mode encap \
  segs fc00:0:2::1,fc00:0:3::1 \
  encap ioam6 trace prealloc type 0x920000 ns 100 size 96 \
  dev eth0

# View IOAM data in captured packets
ip ioam namespace show

What IOAM Reveals¶

• Per-hop latency -- timestamp at each node shows exact per-segment delay
• Queue depth -- congestion visibility at every hop
• Path verification -- confirm the packet actually traversed the expected nodes
• Packet loss localization -- identify exactly where packets are dropped
• Transit delay -- time spent inside each node (processing + queuing)
• Buffer occupancy -- memory pressure per node for capacity planning

SRv6 Path Tracing Deep Dive¶

Path Tracing (draft-filsfils-spring-path-tracing) is an SRv6-native mechanism that provides a complete path trace in a single round-trip, unlike traditional traceroute which requires N round-trips for N hops, and unlike IOAM which adds significant per-hop overhead.

How Path Tracing Differs from IOAM¶

Path Tracing was designed as a lightweight alternative to IOAM, optimized for production networks where header overhead is a concern:

MidPoint Compressed (MPC) Format¶

Path Tracing uses the MidPoint Compressed (MPC) format to pack per-hop information into just 2 bytes (16 bits) per node:

    16 bits per hop
 ┌────────────────────┐
 │ IF_ID  │ IF_LOAD   │
 │ (12b)  │ (4b)      │
 └────────────────────┘

The 12-bit interface ID is a locally significant compressed identifier mapped from the platform's actual interface index. The 4-bit load/timestamp field encodes either the interface utilization level or a compressed transit timestamp, depending on the deployment configuration.

Single-Probe Advantage¶

Traditional ICMP traceroute has a fundamental flaw in networks with ECMP: each probe (with a different hop limit) may take a different path through the network, because the hash inputs change. The result is a "stitched" path that no real packet ever followed.

Path Tracing solves this by using a single probe packet that follows the exact same path as data traffic:

 Traditional Traceroute (3 probes, 3 different ECMP paths):
 Probe 1 (TTL=1): S ──▶ A ──✗ (Time Exceeded)
 Probe 2 (TTL=2): S ──▶ B ──▶ C ──✗         ← different ECMP path!
 Probe 3 (TTL=3): S ──▶ A ──▶ D ──▶ T       ← yet another path!
 Reported path: S → A → C → T               ← WRONG! This path never existed.

 Path Tracing (1 probe, real path):
 Probe 1: S ──▶ A ──▶ D ──▶ T
          │     │      │     │
          └─────┴──────┴─────┘
          MPC: [A:IF3:Load2] [D:IF1:Load0] [T:IF2:Load1]
 Reported path: S → A → D → T               ← Correct actual path.

Implementation: SRH TLV¶

Path Tracing data is carried in a TLV within the SRH (not in a Hop-by-Hop Options header like IOAM). Each SRv6-aware node along the path writes its 2-byte MPC entry into the TLV:

 ┌──────────────────────────────────────────────────┐
 │                  IPv6 Header                      │
 ├──────────────────────────────────────────────────┤
 │           Segment Routing Header (SRH)            │
 │  Segments Left = N                                │
 │  Segment List [SID_0, SID_1, ..., SID_N]         │
 │                                                   │
 │  TLV: Path Tracing                               │
 │  ┌──────────┬──────────┬──────────┬──────────┐   │
 │  │ MPC Hop 1│ MPC Hop 2│ MPC Hop 3│  (empty) │   │
 │  │  2 bytes │  2 bytes │  2 bytes │  2 bytes │   │
 │  └──────────┴──────────┴──────────┴──────────┘   │
 ├──────────────────────────────────────────────────┤
 │                    Payload                        │
 └──────────────────────────────────────────────────┘

Comprehensive Comparison: Traceroute vs IOAM vs Path Tracing¶

In-situ Performance Measurement (IPM)¶

IPM measures delay and loss for SRv6 paths using alternate color marking:

How It Works¶

1. Packets in an SRv6 flow are alternately marked with two "colors" (bit flag in the header)
2. Ingress and egress nodes count packets of each color
3. By comparing counters, you can calculate:
4. Packet loss — difference between ingress and egress counts
5. One-way delay — timestamp comparison between ingress and egress

Advantages¶

• No probe traffic — measures actual data plane performance
• Per-SRv6-policy — measure performance per SR Policy or per service chain
• Continuous — always-on measurement, not periodic sampling

Common Troubleshooting Scenarios¶

Packet Not Following SRv6 Path¶

SRv6 Packets Being Dropped¶

Packet Capture with SRH¶

# Capture SRv6 packets (IPv6 Routing Header)
tcpdump -i eth0 -vv 'ip6 proto 43'

# Capture with full SRH decode
tcpdump -i eth0 -vvv -X 'ip6 proto 43'

# Decode SRH fields
tshark -i eth0 -O ipv6 -Y 'ipv6.routing.type == 4'

# Show segment list
tshark -i eth0 -T fields \
  -e ipv6.src -e ipv6.dst \
  -e ipv6.routing.srh.addr \
  -Y 'ipv6.routing.type == 4'

Display filter: ipv6.routing.type == 4
Wireshark fully decodes SRH including:
- Segments Left
- Segment List (all SIDs)
- TLVs (HMAC, IOAM)

Troubleshooting Playbook¶

The following scenarios cover the most common SRv6 operational issues with step-by-step diagnostic procedures.

Scenario 1: SRv6 VPN Traffic Not Reaching Destination¶

Symptoms: Customer VPN traffic (L3VPN or EVPN over SRv6) enters the ingress PE but never arrives at the egress PE. No ICMP errors are returned to the source.

Diagnostic Flow:

flowchart TD
    A["VPN traffic not reaching destination"] --> B{"Is the SRv6 SID<br/>installed on egress PE?"}
    B -->|No| C["Fix: Check BGP SRv6 locator<br/>and SID allocation"]
    B -->|Yes| D{"Is the SID reachable<br/>via the underlay?"}
    D -->|No| E["Fix: Check IS-IS/OSPF<br/>locator advertisement"]
    D -->|Yes| F{"Is the SRH being<br/>applied at ingress?"}
    F -->|No| G["Fix: Check VRF SRv6<br/>encapsulation config"]
    F -->|Yes| H{"Are intermediate nodes<br/>forwarding correctly?"}
    H -->|No| I["Fix: Check MTU,<br/>ACLs, SRH filtering"]
    H -->|Yes| J["Fix: Check egress PE<br/>SID table and VRF import"]

!! Step 1: Verify the SRv6 SID exists on the egress PE
show segment-routing srv6 sid

!! Step 2: Check that the locator is advertised in IGP
show isis segment-routing srv6 locators
show route ipv6 fc00:0:2::/48

!! Step 3: Verify the VRF is configured for SRv6 encapsulation
show vrf CUSTOMER-A detail
show bgp vpnv4 unicast vrf CUSTOMER-A

!! Step 4: Check CEF for the SRv6 path
show cef ipv6 fc00:0:2::100 detail

!! Step 5: Verify the packet is being encapsulated with SRH
show policy-map interface GigabitEthernet0/0/0/0 input
debug segment-routing srv6 forwarding

!! Step 6: Check for drops on intermediate nodes
show interfaces counter drops
show cef drops

# Step 1: Verify the SRv6 SID is in the local SID table
ip -6 route show table localsid

# Step 2: Check underlay reachability to the egress PE locator
ip -6 route get fc00:0:2::100
ping6 -c 3 fc00:0:2::1

# Step 3: Verify the SRv6 encapsulation route exists
ip -6 route show encap seg6
ip -6 rule show

# Step 4: Capture traffic to confirm SRH is present
tcpdump -i eth0 -vvv 'ip6 proto 43' -c 5

# Step 5: Check for drops
ip -s link show eth0
cat /proc/net/snmp6 | grep -i drop
nstat -z | grep -i drop

Root Cause Checklist:

Scenario 2: Asymmetric Path / Return Traffic Takes Different Path¶

Symptoms: Forward traffic follows the SRv6 policy path as expected, but return traffic takes a different (often suboptimal) path. One-way delay measurements show asymmetry. In some cases, RPF (Reverse Path Forwarding) checks cause drops.

!! Step 1: Verify the forward SR Policy path
show segment-routing traffic-eng policy name POLICY-FWD detail

!! Step 2: Check what path return traffic is using
!! (from the remote PE perspective)
show cef ipv6 fc00:0:1::/48
show segment-routing traffic-eng policy color 100 endpoint fc00:0:1::1

!! Step 3: Check for RPF drops
show cef rpf drops
show interface accounting

!! Step 4: Use IOAM or Path Tracing to see the actual return path
show ioam trace cache

!! Step 5: Verify symmetric SR Policies exist
show segment-routing traffic-eng policy summary

# Step 1: Verify forward path encapsulation
ip -6 route show encap seg6

# Step 2: Check reverse path from remote node
# (run on remote node)
ip -6 route get fc00:0:1::1

# Step 3: Check RPF settings
sysctl net.ipv6.conf.all.rp_filter
cat /proc/net/snmp6 | grep Ip6InAddrErrors

# Step 4: Trace both directions
traceroute6 fc00:0:2::1
# (and from remote): traceroute6 fc00:0:1::1

Root Cause & Fix:

• Missing reverse SR Policy: The return path uses default IGP routing instead of an SR Policy. Fix: configure a symmetric SR Policy on the remote PE, or use a bidirectional SR Policy.
• RPF check failure: If strict uRPF is enabled and return traffic arrives on a different interface than the one used to reach the source, packets are dropped. Fix: use loose-mode RPF (rpf-mode loose) or configure appropriate RPF allow lists.
• Different ECMP hash on return: Even without SR Policies, ECMP may select different paths in each direction. Fix: use SR-TE policies to enforce symmetric paths for latency-sensitive traffic.

Scenario 3: MTU Issues Causing Silent Drops¶

Symptoms: Large packets (jumbo frames, or packets near the MTU limit) are silently dropped when SRv6 encapsulation is applied. Small packets (e.g., pings) work fine. TCP sessions stall after the initial handshake (because SYN/ACK is small, but data segments are large).

!! Step 1: Check interface MTU values
show interfaces GigabitEthernet0/0/0/0 | include MTU

!! Step 2: Look for MTU-exceeded drops
show interfaces counter errors
show cef drops | include MTU

!! Step 3: Check PMTUD (Path MTU Discovery) status
show tcp statistics | include MTU
show icmpv6 statistics | include "too big"

!! Step 4: Verify ICMPv6 Packet Too Big is NOT being filtered
show access-list | include icmpv6

!! Step 5: Test with specific packet sizes
ping ipv6 fc00:0:2::100 size 1400 count 5
ping ipv6 fc00:0:2::100 size 1500 count 5
ping ipv6 fc00:0:2::100 size 8900 count 5

!! Fix: Increase MTU on all transit links
interface GigabitEthernet0/0/0/0
 mtu 9216
!

# Step 1: Check interface MTU
ip link show eth0 | grep mtu

# Step 2: Look for fragmentation-needed / Packet Too Big
ip -s -6 route get fc00:0:2::100
cat /proc/net/snmp6 | grep -E "Frag|TooBig"

# Step 3: Test with different sizes using ping
ping6 -s 1400 -c 3 -M do fc00:0:2::100  # do = don't fragment
ping6 -s 1500 -c 3 -M do fc00:0:2::100
ping6 -s 8000 -c 3 -M do fc00:0:2::100

# Step 4: Check if ICMPv6 PTB messages are reaching the source
tcpdump -i eth0 'icmp6 and ip6[40] == 2' -c 5

# Fix: Increase MTU on transit interfaces
ip link set dev eth0 mtu 9216

# Fix: Or adjust TCP MSS clamping
ip6tables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
  -j TCPMSS --set-mss 1300

MTU Planning Guide:

Scenario 4: SRv6 SID Not Being Programmed in Hardware¶

Symptoms: SRv6 SID appears in the software routing table but is not installed in the hardware forwarding table (FIB/TCAM). Traffic forwarded in software experiences high CPU utilization and significantly increased latency. In some cases, the router drops packets when the CPU punt queue overflows.

!! Step 1: Check if the SID is in the software table
show segment-routing srv6 sid
show cef ipv6 fc00:0:1::100 detail

!! Step 2: Check if the SID is in the hardware FIB
show cef ipv6 fc00:0:1::100 hardware egress detail
show controllers npu resources all

!! Step 3: Look for hardware programming errors
show cef ipv6 unresolved
show health resource-utilization

!! Step 4: Check if the platform supports the SID behavior
show segment-routing srv6 manager
show platform | include SRv6

!! Step 5: Check for TCAM exhaustion
show controllers npu resources tcam
show controllers npu resources lem

!! Fix: If TCAM is full, reduce prefix scale or use uSID
!! uSID uses fewer TCAM entries than classic SRv6 SIDs
segment-routing
 srv6
  locators
   locator MAIN
    micro-segment behavior unode psp-usd
   !
  !
!

# Step 1: Check if the SID is in the kernel routing table
ip -6 route show table localsid

# Step 2: Check if hardware offload is active (switchdev / ASIC)
ip -6 route show table localsid | grep offload
# "offload" flag indicates hardware programming

# Step 3: Monitor CPU utilization from software forwarding
top -bn1 | head -20
cat /proc/net/softnet_stat

# Step 4: Check for hardware resource errors (platform-specific)
dmesg | grep -i -E "tcam|fib|overflow|resource"
ethtool -S eth0 | grep -i drop

# Step 5: Check offload statistics
ip -s -6 route show table localsid
tc -s filter show dev eth0 ingress

Root Cause & Fix:

Further Reading¶

• SRH Mechanics & Packet Walk - Detailed SRH processing
• Security - HMAC and SRH authentication
• Telemetry & Monitoring - IPFIX, YANG models
• RFC 9259 - OAM in SRv6

References¶

1. RFC 9259 - OAM in SRv6 - Defines SRv6-specific OAM mechanisms including ping and traceroute
2. RFC 9326 - In-situ OAM (IOAM) Data Fields - Defines IOAM option types (Pre-allocated Trace, Incremental Trace, Proof of Transit, Edge-to-Edge), data fields, and namespace semantics
3. RFC 8200 - IPv6 Specification - IPv6 base specification including extension header ordering (relevant for IOAM Hop-by-Hop placement)
4. draft-ietf-ippm-ioam-srv6-options - Defines how IOAM data is carried in SRv6 using IPv6 Hop-by-Hop and Destination Options headers (NOT SRH TLVs)
5. draft-filsfils-spring-path-tracing - SRv6 Path Tracing with MidPoint Compressed (MPC) format for lightweight single-probe path discovery