文章目录

• 功能流表
• • 加 Pod
  • 加子网
  • 跨子网 pod 互访
  • 访问外部网络
  • DHCP
  • 静态路由
  • 策略路由
  • 安全组
  • DNAT
  • LB
  • FullNAT LB
  • Service 扩展
  • • Src-ip LB

功能流表

基于此模型配置功能，来查看流表变化，探讨功能实现方式。

加 Pod

流表添加：

Datapath: "ovn-cluster" (5ec1c73b-e8f2-4b11-8779-28e02a091b87)  Pipeline: ingress从 ovn-cluster logical_router 到 ovn-default logical_switch 且目的 IP46 地址是 pod8 的, 目的 mac 地址换成 00:00:00:C6:A4:F9。table=15(lr_in_arp_resolve  ), priority=100  , match=(outport == "ovn-cluster-ovn-default" && reg0 == 10.16.4.138), action=(eth.dst = 00:00:00:c6:a4:f9; next;)table=15(lr_in_arp_resolve  ), priority=100  , match=(outport == "ovn-cluster-ovn-default" && xxreg0 == dd:10:16::48a), action=(eth.dst = 00:00:00:c6:a4:f9; next;)Datapath: "ovn-default" (b6a669bb-984d-4c04-9622-64dba046cd55)  Pipeline: ingress从 pod8 进来的流量，nexttable=0 (ls_in_port_sec_l2  ), priority=50   , match=(inport == "pod8.default"), action=(next;)目的 IP46 地址是 pod8 的table=14(ls_in_after_lb     ), priority=50   , match=(ip4.dst == 10.16.4.138), action=(eth.dst = 00:00:00:c6:a4:f9; next;)table=14(ls_in_after_lb     ), priority=50   , match=(ip6.dst == dd:10:16::48a), action=(eth.dst = 00:00:00:c6:a4:f9; next;)从 pod8 发出的 arp 请求 且请求的地址是本身 IP，next；GARPtable=19(ls_in_arp_rsp      ), priority=100  , match=(arp.tpa == 10.16.4.138 && arp.op == 1 && inport == "pod8.default"), action=(next;)table=19(ls_in_arp_rsp      ), priority=100  , match=(nd_ns && ip6.dst == {dd:10:16::48a, ff02::1:ff00:48a} && nd.target == dd:10:16::48a && inport == "pod8.default"), action=(next;)请求 pod8 IP 的 arp，直接返回 pod8 的 MAC 地址。本段详细介绍 action，后续带过。// arp.op = 2 ARP 回复// arp.tha = arp.sha; arp.sha = 00:00:00:c6:a4:f9 ARP 回复报文的 mac 设置// arp.tpa = arp.spa; arp.spa = 10.16.4.138  ARP 回复报文的 IP 设置// outport = inport; flags.loopback = 1; output; 从来接口返回table=19(ls_in_arp_rsp      ), priority=50   , match=(arp.tpa == 10.16.4.138 && arp.op == 1), action=(eth.dst = eth.src; eth.src = 00:00:00:c6:a4:f9; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = 00:00:00:c6:a4:f9; arp.tpa = arp.spa; arp.spa = 10.16.4.138; outport = inport; flags.loopback = 1; output;)table=19(ls_in_arp_rsp      ), priority=50   , match=(nd_ns && ip6.dst == {dd:10:16::48a, ff02::1:ff00:48a} && nd.target == dd:10:16::48a), action=(nd_na { eth.src = 00:00:00:c6:a4:f9; ip6.src = dd:10:16::48a; nd.target = dd:10:16::48a; nd.tll = 00:00:00:c6:a4:f9; outport = inport; flags.loopback = 1; output; };)目的 MAC 是 pod8 的 MAC 地址，从 pod8.default interface 发出table=25(ls_in_l2_lkup      ), priority=50   , match=(eth.dst == 00:00:00:c6:a4:f9), action=(outport = "pod8.default"; output;)Datapath: "ovn-default" (b6a669bb-984d-4c04-9622-64dba046cd55)  Pipeline: egresstable=9 (ls_out_port_sec_l2 ), priority=50   , match=(outport == "pod8.default"), action=(output;)

加子网

会加 subnet 网络 logical_switch 的 ingress，egress 表。除此之外，看 ovn-cluster 上的流表变化
流表添加：

Datapath: "ovn-cluster" (5ec1c73b-e8f2-4b11-8779-28e02a091b87)  Pipeline: ingress从 subnet 子网来的流量，arp 请求，learn mac_bingindtable=1 (lr_in_lookup_neighbor), priority=100  , match=(inport == "ovn-cluster-subnet" && arp.spa == 192.168.2.0/24 && arp.op == 1), action=(reg9[2] = lookup_arp(inport, arp.spa, arp.sha); next;)LR 收到报文的 ttl 为 1 或者 0，任务到达不了目的地，原路返回 icmp 包，告知 TTL equals 0 during transittable=3 (lr_in_ip_input     ), priority=100  , match=(inport == "ovn-cluster-subnet" && ip4 && ip.ttl == {0, 1} && !ip.later_frag), action=(icmp4 {eth.dst <-> eth.src; icmp4.type = 11; /* Time exceeded */ icmp4.code = 0; /* TTL exceeded in transit */ ip4.dst = ip4.src; ip4.src = 192.168.2.1 ; ip.ttl = 254; outport = "ovn-cluster-subnet"; flags.loopback = 1; output; };)table=3 (lr_in_ip_input     ), priority=100  , match=(inport == "ovn-cluster-subnet" && ip6 && ip6.src == bb00:192:168::/64 && ip.ttl == {0, 1} && !ip.later_frag), action=(icmp6 {eth.dst <-> eth.src; ip6.dst = ip6.src; ip6.src = bb00:192:168::1 ; ip.ttl = 254; icmp6.type = 3; /* Time exceeded */ icmp6.code = 0; /* TTL exceeded in transit */ outport = "ovn-cluster-subnet"; flags.loopback = 1; output; };)drop 源 ip 为网关或广播地址的报文table=3 (lr_in_ip_input     ), priority=100  , match=(ip4.src == {192.168.2.1, 192.168.2.255} && reg9[0] == 0), action=(drop;)ipv6 目的地址为 subnet 网关，dhcp 报文，执行 dhcpv6_reply 的操作table=3 (lr_in_ip_input     ), priority=100  , match=(ip6.dst == bb00:192:168::1 && udp.src == 547 && udp.dst == 546), action=(reg0 = 0; handle_dhcpv6_reply;)table=3 (lr_in_ip_input     ), priority=100  , match=(ip6.dst == fe80::200:ff:fe40:8eb1 && udp.src == 547 && udp.dst == 546), action=(reg0 = 0; handle_dhcpv6_reply;)subnet 中 arp 或 nd 网关的报文回复table=3 (lr_in_ip_input     ), priority=90   , match=(inport == "ovn-cluster-subnet" && arp.op == 1 && arp.tpa == 192.168.2.1 && arp.spa == 192.168.2.0/24), action=(eth.dst = eth.src; eth.src = xreg0[0..47]; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = xreg0[0..47]; arp.tpa <-> arp.spa; outport = inport; flags.loopback = 1; output;)table=3 (lr_in_ip_input     ), priority=90   , match=(inport == "ovn-cluster-subnet" && ip6.dst == {bb00:192:168::1, ff02::1:ff00:1} && nd_ns && nd.target == bb00:192:168::1), action=(nd_na_router { eth.src = xreg0[0..47]; ip6.src = nd.target; nd.tll = xreg0[0..47]; outport = inport; flags.loopback = 1; output; };)table=3 (lr_in_ip_input     ), priority=90   , match=(inport == "ovn-cluster-subnet" && ip6.dst == {fe80::200:ff:fe40:8eb1, ff02::1:ff40:8eb1} && nd_ns && nd.target == fe80::200:ff:fe40:8eb1), action=(nd_na_router { eth.src = xreg0[0..47]; ip6.src = nd.target; nd.tll = xreg0[0..47]; outport = inport; flags.loopback = 1; output; };)subnet 中 ping 网关的报文回复table=3 (lr_in_ip_input     ), priority=90   , match=(ip4.dst == 192.168.2.1 && icmp4.type == 8 && icmp4.code == 0), action=(ip4.dst <-> ip4.src; ip.ttl = 255; icmp4.type = 0; flags.loopback = 1; next; )table=3 (lr_in_ip_input     ), priority=90   , match=(ip6.dst == {bb00:192:168::1, fe80::200:ff:fe40:8eb1} && icmp6.type == 128 && icmp6.code == 0), action=(ip6.dst <-> ip6.src; ip.ttl = 255; icmp6.type = 129; flags.loopback = 1; next; )其他到网关的报文丢弃table=3 (lr_in_ip_input     ), priority=60   , match=(ip4.dst == {192.168.2.1}), action=(drop;)table=3 (lr_in_ip_input     ), priority=60   , match=(ip6.dst == {bb00:192:168::1, fe80::200:ff:fe40:8eb1}), action=(drop;)下面两条IPv6 路由请求，返回的配置和回复报文操作table=8 (lr_in_nd_ra_options), priority=50   , match=(inport == "ovn-cluster-subnet" && ip6.dst == ff02::2 && nd_rs), action=(reg0[5] = put_nd_ra_opts(addr_mode = "dhcpv6_stateful", slla = 00:00:00:40:8e:b1, prefix = bb00:192:168::/64); next;)table=9 (lr_in_nd_ra_response), priority=50   , match=(inport == "ovn-cluster-subnet" && ip6.dst == ff02::2 && nd_ra && reg0[5]), action=(eth.dst = eth.src; eth.src = 00:00:00:40:8e:b1; ip6.dst = ip6.src; ip6.src = fe80::200:ff:fe40:8eb1; outport = inport; flags.loopback = 1; output;)子网访问 fe80::，直接源地址换成网关地址返回table=11(lr_in_ip_routing   ), priority=194  , match=(inport == "ovn-cluster-subnet" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[0..15] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe40:8eb1; eth.src = 00:00:00:40:8e:b1; outport = "ovn-cluster-subnet"; flags.loopback = 1; next;)其他网络访问该子网，源 mac 该为网关 mac，并确定 outport 为子网网关 port，静态路由 table=11(lr_in_ip_routing   ), priority=194  , match=(ip6.dst == bb00:192:168::/64), action=(ip.ttl--; reg8[0..15] = 0; xxreg0 = ip6.dst; xxreg1 = bb00:192:168::1; eth.src = 00:00:00:40:8e:b1; outport = "ovn-cluster-subnet"; flags.loopback = 1; next;)table=11(lr_in_ip_routing   ), priority=74   , match=(ip4.dst == 192.168.2.0/24), action=(ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 192.168.2.1; eth.src = 00:00:00:40:8e:b1; outport = "ovn-cluster-subnet"; flags.loopback = 1; next;)策略路由，从子网且在 master 节点的报文发送端，master 节点的 ovn0 作为下一跳，源 mac 改为 join 网络的网关。并更新 outport。table=13(lr_in_policy       ), priority=31000, match=(ip4.dst == 192.168.2.0/24), action=(reg8[0..15] = 0; next;)table=13(lr_in_policy       ), priority=31000, match=(ip6.dst == bb00:192:168::/64), action=(reg8[0..15] = 0; next;)table=13(lr_in_policy       ), priority=29000, match=(ip4.src == $subnet.master_ip4), action=(reg0 = 100.64.0.2; reg1 = 100.64.0.1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; reg8[0..15] = 0; next;)table=13(lr_in_policy       ), priority=29000, match=(ip4.src == $subnet.worker2_ip4), action=(reg0 = 100.64.0.4; reg1 = 100.64.0.1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; reg8[0..15] = 0; next;)table=13(lr_in_policy       ), priority=29000, match=(ip4.src == $subnet.worker_ip4), action=(reg0 = 100.64.0.3; reg1 = 100.64.0.1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; reg8[0..15] = 0; next;)table=13(lr_in_policy       ), priority=29000, match=(ip6.src == $subnet.master_ip6), action=(xxreg0 = dd:100:64::2; xxreg1 = dd:100:64::1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; reg8[0..15] = 0; next;)table=13(lr_in_policy       ), priority=29000, match=(ip6.src == $subnet.worker2_ip6), action=(xxreg0 = dd:100:64::4; xxreg1 = dd:100:64::1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; reg8[0..15] = 0; next;)table=13(lr_in_policy       ), priority=29000, match=(ip6.src == $subnet.worker_ip6), action=(xxreg0 = dd:100:64::3; xxreg1 = dd:100:64::1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; reg8[0..15] = 0; next;)Datapath: "ovn-cluster" (5ec1c73b-e8f2-4b11-8779-28e02a091b87)  Pipeline: egresstable=6 (lr_out_delivery    ), priority=100  , match=(outport == "ovn-cluster-subnet"), action=(output;)

table3 中 两条都是，第一条是 subnet 设置的 ipv6 网关，第二条是网关 port mac地址生成的 ipv6；

生成方法为前面固定 fe80::； 后面 mac 地址第一组第七位取反，第一组 00，第七位取反后变成 02，即 mac 变成 02:00:00:40:8E:B1；中间再插入 ff:fe -> 0200:00ff:fe40:8eb1。最后得到 ipv6 fe80::0200:00ff:fe40:8eb1

跨子网 pod 互访

下面在 subnet 创建一个 pod9，trace pod9 到 pod8 的访问。

$ kubectl ko trace default/pod8  192.168.2.11 icmpingress(dp="ovn-default", inport="pod8.default")
------------------------------------------------解决准入问题，pod8 发来的报文允许0. ls_in_port_sec_l2 (northd.c:5607): inport == "pod8.default", priority 50, uuid db8107b9next;标记 acl 提示，reg0[8]：可能命中允许，但不需要连接 conntrack; reg0[9]：可能命中 drop。8. ls_in_acl_hint (northd.c:6141): !ct.trk, priority 5, uuid ea0a4f48reg0[8] = 1;reg0[9] = 1;next;查看是否目的地址是同网段的，不是的话到下一跳流表，是的话，更新目的 mac，因为子网内已知目的 ip 的 mac
14. ls_in_after_lb (northd.c:7095): ip4.dst != {10.16.0.0/16}, priority 60, uuid 822e2991next;需要从网关 port 发出，目的 mac 改为网关 mac 地址，执行 output action
25. ls_in_l2_lkup (northd.c:8745): eth.dst == 00:00:00:63:b3:f9, priority 50, uuid aaecf203outport = "ovn-default-ovn-cluster";output;egress(dp="ovn-default", inport="pod8.default", outport="ovn-default-ovn-cluster")
----------------------------------------------------------------------------------0. ls_out_pre_lb (northd.c:5757): ip && outport == "ovn-default-ovn-cluster", priority 110, uuid 35b4eb79next;同样检查 egress acl 提示3. ls_out_acl_hint (northd.c:6141): !ct.trk, priority 5, uuid 80cbad44reg0[8] = 1;reg0[9] = 1;next;9. ls_out_port_sec_l2 (northd.c:5704): outport == "ovn-default-ovn-cluster", priority 50, uuid 9a68d043output;/* output to "ovn-default-ovn-cluster", type "l3gateway" */ingress(dp="ovn-cluster", inport="ovn-cluster-ovn-default")
-----------------------------------------------------------准入检查，从子网过来的，目的 mac 是该网关的允许0. lr_in_admission (northd.c:11022): eth.dst == 00:00:00:63:b3:f9 && inport == "ovn-cluster-ovn-default", priority 50, uuid 75733dd4xreg0[0..47] = 00:00:00:63:b3:f9;next;1. lr_in_lookup_neighbor (northd.c:11166): 1, priority 0, uuid 498b58fareg9[2] = 1;next;2. lr_in_learn_neighbor (northd.c:11175): reg9[2] == 1, priority 100, uuid c709ddc3next;
10. lr_in_ip_routing_pre (northd.c:11398): 1, priority 0, uuid f62bcaafreg7 = 0;next;目的地址是 192.168.2.0/24, ttl--，reg0 为目的地址，reg1 为目的地址的网关，源 mac 为目的地址的网关 mac，outport 是目的子网的网关
11. lr_in_ip_routing (northd.c:9909): ip4.dst == 192.168.2.0/24, priority 74, uuid 31ceb650ip.ttl--;reg8[0..15] = 0;reg0 = ip4.dst;reg1 = 192.168.2.1;eth.src = 00:00:00:40:8e:b1;outport = "ovn-cluster-subnet";flags.loopback = 1;next;
12. lr_in_ip_routing_ecmp (northd.c:11474): reg8[0..15] == 0, priority 150, uuid d30dd2a0next;策略路由，未更改 outport
13. lr_in_policy (northd.c:9125): ip4.dst == 192.168.2.0/24, priority 31000, uuid 614460dereg8[0..15] = 0;next;
14. lr_in_policy_ecmp (northd.c:11610): reg8[0..15] == 0, priority 150, uuid e0a09b2enext;目的地址为 192.168.2.11, 设置 目的 mac 为 00:00:00:0b:d8:da
15. lr_in_arp_resolve (northd.c:11811): outport == "ovn-cluster-subnet" && reg0 == 192.168.2.11, priority 100, uuid 508186d4eth.dst = 00:00:00:0b:d8:da;next;目的 mac 已设置，不发送 arp
19. lr_in_arp_request (northd.c:12308): 1, priority 0, uuid 3c422c04output;egress(dp="ovn-cluster", inport="ovn-cluster-ovn-default", outport="ovn-cluster-subnet")
----------------------------------------------------------------------------------------0. lr_out_chk_dnat_local (northd.c:13530): 1, priority 0, uuid b0688753reg9[4] = 0;next;6. lr_out_delivery (northd.c:12355): outport == "ovn-cluster-subnet", priority 100, uuid 559ce0d5output;/* output to "ovn-cluster-subnet", type "l3gateway" */ingress(dp="subnet", inport="subnet-ovn-cluster")
-------------------------------------------------0. ls_in_port_sec_l2 (northd.c:5607): inport == "subnet-ovn-cluster", priority 50, uuid d6f2499bnext;6. ls_in_pre_lb (northd.c:5754): ip && inport == "subnet-ovn-cluster", priority 110, uuid b1914c0fnext;8. ls_in_acl_hint (northd.c:6141): !ct.trk, priority 5, uuid ea0a4f48reg0[8] = 1;reg0[9] = 1;next;目的地址为 192.168.2.11, 设置 目的 mac 为 00:00:00:0b:d8:da
14. ls_in_after_lb (northd.c:7165): ip4.dst == 192.168.2.11, priority 50, uuid ad5447a8eth.dst = 00:00:00:0b:d8:da;next;目的 mac 为 00:00:00:0b:d8:da，outport 为 pod9.default
25. ls_in_l2_lkup (northd.c:8674): eth.dst == 00:00:00:0b:d8:da, priority 50, uuid eb131381outport = "pod9.default";output;egress(dp="subnet", inport="subnet-ovn-cluster", outport="pod9.default")
------------------------------------------------------------------------LS 上配置了 LB，需要检查0. ls_out_pre_lb (northd.c:6012): ip, priority 100, uuid 16fc6effreg0[2] = 1;next;2. ls_out_pre_stateful (northd.c:6062): reg0[2] == 1, priority 110, uuid 953b71bbct_lb_mark;ct_lb_mark
----------3. ls_out_acl_hint (northd.c:6119): ct.new && !ct.est, priority 7, uuid 26402d99reg0[7] = 1;reg0[9] = 1;next;4. ls_out_acl (northd.c:6716): ip && (!ct.est || (ct.est && ct_mark.blocked == 1)), priority 1, uuid fdd00aa8reg0[1] = 1;next;7. ls_out_stateful (northd.c:7231): reg0[1] == 1 && reg0[13] == 0, priority 100, uuid fc7cf441ct_commit { ct_mark.blocked = 0; };next;9. ls_out_port_sec_l2 (northd.c:5704): outport == "pod9.default", priority 50, uuid 5412055eoutput;/* output to "pod9.default", type "" */
+ set +x
--------

访问外部网络

pod9 访问 114.114.114.114

$ kubectl ko  trace default/pod9 114.114.114.114 udp 53
ingress(dp="subnet", inport="pod9.default")
-------------------------------------------egress(dp="subnet", inport="pod9.default", outport="subnet-ovn-cluster")
------------------------------------------------------------------------ingress(dp="ovn-cluster", inport="ovn-cluster-subnet")
------------------------------------------------------
...策略路由生效，ip4.src == $ovn.default.worker_ip4         reroute                100.64.0.3 outport 是 join 网络，下一跳设置为 worker 节点的 ovn0 ip。
13. lr_in_policy (northd.c:9125): ip4.src == $subnet.worker_ip4, priority 29000, uuid 2e224492reg0 = 100.64.0.3;reg1 = 100.64.0.1;eth.src = 00:00:00:e9:15:ff;outport = "ovn-cluster-join";flags.loopback = 1;reg8[0..15] = 0;next;设置 目的 mac 为 ovn0 的 mac
15. lr_in_arp_resolve (northd.c:11811): outport == "ovn-cluster-join" && reg0 == 100.64.0.3, priority 100, uuid 36bca89ceth.dst = 00:00:00:32:7e:12;next;
19. lr_in_arp_request (northd.c:12308): 1, priority 0, uuid 3c422c04output;egress(dp="ovn-cluster", inport="ovn-cluster-subnet", outport="ovn-cluster-join")
---------------------------------------------------------------------------------ingress(dp="join", inport="join-ovn-cluster")
---------------------------------------------目的 mac 是 worker 节点上 ovn0 的 mac，outport 是 node-worker
25. ls_in_l2_lkup (northd.c:8674): eth.dst == 00:00:00:32:7e:12, priority 50, uuid bd93d956outport = "node-worker";output;egress(dp="join", inport="join-ovn-cluster", outport="node-worker")
-------------------------------------------------------------------9. ls_out_port_sec_l2 (northd.c:5704): outport == "node-worker", priority 50, uuid 16e2ea51output;/* output to "node-worker", type "" */

DHCP

比较子网是否带 DHCP 配置

在 logical_router 上配置
Datapath: "ovn-cluster" (5ec1c73b-e8f2-4b11-8779-28e02a091b87)  Pipeline: ingress从此子网来的 ipv6 nd 报文，配置前缀，用网关 mac 回复。table=8 (lr_in_nd_ra_options), priority=50   , match=(inport == "ovn-cluster-subnet" && ip6.dst == ff02::2 && nd_rs), action=(reg0[5] = put_nd_ra_opts(addr_mode = "dhcpv6_stateful", slla = 00:00:00:40:8e:b1, prefix = bb00:192:168::/64); next;)
181a183table=9 (lr_in_nd_ra_response), priority=50   , match=(inport == "ovn-cluster-subnet" && ip6.dst == ff02::2 && nd_ra && reg0[5]), action=(eth.dst = eth.src; eth.src = 00:00:00:40:8e:b1; ip6.dst = ip6.src; ip6.src = fe80::200:ff:fe40:8eb1; outport = inport; flags.loopback = 1; output;)

那具体的 IP 如何呢？
可以看为每个子网添加 pod 时，该子网的流表变化.

Datapath: "subnet" (bc26eedb-61fd-42f7-8841-52442ed3598e)  Pipeline: ingress匹配到 pod9 lsp 来的流量如果是 dhcp 请求，给流的 dhcp optionstable=20(ls_in_dhcp_options ), priority=100  , match=(inport == "pod9.default" && eth.src == 00:00:00:a1:01:c6 && ip4.src == 0.0.0.0 && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst == 67), action=(reg0[3] = put_dhcp_opts(offerip = 192.168.2.11, lease_time = 3600, netmask = 255.255.255.0, router = 192.168.2.1, server_id = 169.254.0.254); next;)table=20(ls_in_dhcp_options ), priority=100  , match=(inport == "pod9.default" && eth.src == 00:00:00:a1:01:c6 && ip4.src == 192.168.2.11 && ip4.dst == {169.254.0.254, 255.255.255.255} && udp.src == 68 && udp.dst == 67), action=(reg0[3] = put_dhcp_opts(offerip = 192.168.2.11, lease_time = 3600, netmask = 255.255.255.0, router = 192.168.2.1, server_id = 169.254.0.254); next;)ipv6table=20(ls_in_dhcp_options ), priority=100  , match=(inport == "pod9.default" && eth.src == 00:00:00:a1:01:c6 && ip6.dst == ff02::1:2 && udp.src == 546 && udp.dst == 547), action=(reg0[3] = put_dhcpv6_opts(ia_addr = bb00:192:168::2, server_id = 00:00:00:5E:04:65); next;)dhcp 回复操作table=21(ls_in_dhcp_response), priority=100  , match=(inport == "pod9.default" && eth.src == 00:00:00:a1:01:c6 && ip4 && udp.src == 68 && udp.dst == 67 && reg0[3]), action=(eth.dst = eth.src; eth.src = 00:00:02:2E:2F:B8; ip4.src = 169.254.0.254; udp.src = 67; udp.dst = 68; outport = inport; flags.loopback = 1; output;)table=21(ls_in_dhcp_response), priority=100  , match=(inport == "pod9.default" && eth.src == 00:00:00:a1:01:c6 && ip6.dst == ff02::1:2 && udp.src == 546 && udp.dst == 547 && reg0[3]), action=(eth.dst = eth.src; eth.src = 00:00:00:5E:04:65; ip6.dst = ip6.src; ip6.src = fe80::200:ff:fe5e:465; udp.src = 547; udp.dst = 546; outport = inport; flags.loopback = 1; output;)

静态路由

查看默认状态下 ovn-cluster 的路由表

$ kubectl ko nbctl lr-route-list ovn-cluster
IPv4 Routes
Route Table 
:0.0.0.0/0                100.64.0.1 dst-ipIPv6 Routes
Route Table 
:::/0              dd:100:64::1 dst-ip

流表如下，包括逻辑路由里的 Logical_Switch，静态路由。

  table=11(lr_in_ip_routing   ), priority=194  , match=(ip6.dst == bb00:192:168::/64), action=(ip.ttl--; reg8[0..15] = 0; xxreg0 = ip6.dst; xxreg1 = bb00:192:168::1; eth.src = 00:00:00:c8:1c:8a; outport = "ovn-cluster-subnet"; flags.loopback = 1; next;)table=11(lr_in_ip_routing   ), priority=74   , match=(ip4.dst == 192.168.2.0/24), action=(ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 192.168.2.1; eth.src = 00:00:00:c8:1c:8a; outport = "ovn-cluster-subnet"; flags.loopback = 1; next;)table=11(lr_in_ip_routing   ), priority=50   , match=(ip4.dst == 10.16.0.0/16), action=(ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 10.16.0.1; eth.src = 00:00:00:63:b3:f9; outport = "ovn-cluster-ovn-default"; flags.loopback = 1; next;)table=11(lr_in_ip_routing   ), priority=50   , match=(ip4.dst == 100.64.0.0/16), action=(ip.ttl--; reg8[0..15] = 0; reg0 = ip4.dst; reg1 = 100.64.0.1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; next;)table=11(lr_in_ip_routing   ), priority=1    , match=(reg7 == 0 && ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.1; reg1 = 100.64.0.1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; next;)table=11(lr_in_ip_routing   ), priority=1    , match=(reg7 == 0 && ip6.dst == ::/0), action=(ip.ttl--; reg8[0..15] = 0; xxreg0 = dd:100:64::1; xxreg1 = dd:100:64::1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; next;)

静态路由不仅可以设置 目的 IP 路由，也可以设置源 IP 路由。
具体应用联合 Nat 一起一个例子。

$ kubectl ko nbctl --policy=src-ip   lr-route-add  ovn-cluster  192.168.2.11/32  100.64.0.1

流表中

table=11(lr_in_ip_routing   ), priority=96   , match=(ip4.src == 192.168.2.11/32), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 100.64.0.1; reg1 = 100.64.0.1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; next;)

策略路由

查看默认状态下 ovn-cluster 的策略路由表

     30000                               ip4.dst == 1.2.3.215         reroute                100.64.0.229000                 ip4.src == $ovn.default.master_ip4         reroute                100.64.0.229000                      ip6.src == $subnet.master_ip6         reroute              dd:100:64::2

流表如下：依次是到主机 internel_ip ，master 节点上 default 子网 ipv4 的 port，master 节点上 subnet 子网 ipv6 的 port，吓一跳到 ovn0 的路由。

  table=13(lr_in_policy       ), priority=30000, match=(ip4.dst == 1.2.3.215), action=(reg0 = 100.64.0.2; reg1 = 100.64.0.1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; reg8[0..15] = 0; next;)table=13(lr_in_policy       ), priority=29000, match=(ip4.src == $ovn.default.master_ip4), action=(reg0 = 100.64.0.2; reg1 = 100.64.0.1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; reg8[0..15] = 0; next;table=13(lr_in_policy       ), priority=29000, match=(ip6.src == $subnet.master_ip6), action=(xxreg0 = dd:100:64::2; xxreg1 = dd:100:64::1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; reg8[0..15] = 0; next;)

添加 到 localdns 的策略路由可以像下面一样添加。

kubectl ko nbctl lr-policy-add ovn-cluster 29100 "ip4.src == \$ovn.default.master_ip4 && ip4.dst == ip4.dst == 169.254.25.10" reroute 100.64.0.2

流表中

table=13(lr_in_policy       ), priority=29100, match=(ip4.src == $ovn.default.master_ip4 && ip4.dst == ip4.dst == 169.254.25.10), action=(reg0 = 100.64.0.2; reg1 = 100.64.0.1; eth.src = 00:00:00:e9:15:ff; outport = "ovn-cluster-join"; flags.loopback = 1; reg8[0..15] = 0; next;)

安全组

sg1 为 安全组名称 （kubeOVN CRD）
egress：allow all
ingress：22 allow，icmp drop

apiVersion: kubeovn.io/v1 
kind: SecurityGroup 
metadata: name: sg1 
spec: egressRules: - ipVersion: ipv4 policy: allow priority: 1 protocol: all remoteAddress: 0.0.0.0/0 remoteType: address ingressRules: - ipVersion: ipv4 policy: drop priority: 10 protocol: icmp remoteAddress: 0.0.0.0/0 remoteType: address - ipVersion: ipv4 policy: allow priority: 10 protocol: tcp portRangeMin: 22 portRangeMax: 22 remoteAddress: 0.0.0.0/0 remoteType: address

流表变化：

logical_switch 的 ingress 对应 lsp 的 egress
Datapath: "ovn-default" (b6a669bb-984d-4c04-9622-64dba046cd55)  Pipeline: ingress允许 ipv4 的 dhcptable=1 (ls_in_port_sec_ip  ), priority=90   , match=(inport == "pod8.default" && eth.src == 00:00:00:c6:a4:f9 && ip4.src == 0.0.0.0 && ip4.dst == 255.255.255.255 && udp.src == 68 && udp.dst == 67), action=(next;)允许 ip 10.16.4.138，mac 00:00:00:c6:a4:f9，lsp pod8.default 流量table=1 (ls_in_port_sec_ip  ), priority=90   , match=(inport == "pod8.default" && eth.src == 00:00:00:c6:a4:f9 && ip4.src == {10.16.4.138}), action=(next;)允许 到 ff02::/16 的 RAtable=1 (ls_in_port_sec_ip  ), priority=90   , match=(inport == "pod8.default" && eth.src == 00:00:00:c6:a4:f9 && ip6.src == :: && ip6.dst == ff02::/16 && icmp6.type == {131, 135, 143}), action=(next;)允许 ip6 地址 dd:10:16::48a, fe80::200:ff:fec6:a4f9, mac 00:00:00:c6:a4:f9，的流量table=1 (ls_in_port_sec_ip  ), priority=90   , match=(inport == "pod8.default" && eth.src == 00:00:00:c6:a4:f9 && ip6.src == {fe80::200:ff:fec6:a4f9, dd:10:16::48a}), action=(next;)其他流量 drop，上面第二条和第四条已经 next了，所以如果 ip 和 mac 正确，匹配不到本条流表。table=1 (ls_in_port_sec_ip  ), priority=80   , match=(inport == "pod8.default" && eth.src == 00:00:00:c6:a4:f9 && ip), action=(drop;)garp 的配置table=2 (ls_in_port_sec_nd  ), priority=90   , match=(inport == "pod8.default" && eth.src == 00:00:00:c6:a4:f9 && arp.sha == 00:00:00:c6:a4:f9 && arp.spa == {10.16.4.138}), action=(next;)table=2 (ls_in_port_sec_nd  ), priority=90   , match=(inport == "pod8.default" && eth.src == 00:00:00:c6:a4:f9 && ip6 && nd && ((nd.sll == 00:00:00:00:00:00 || nd.sll == 00:00:00:c6:a4:f9) || ((nd.tll == 00:00:00:00:00:00 || nd.tll == 00:00:00:c6:a4:f9) && (nd.target == fe80::200:ff:fec6:a4f9 || nd.target == dd:10:16::48a)))), action=(next;)table=2 (ls_in_port_sec_nd  ), priority=80   , match=(inport == "pod8.default" && (arp || nd)), action=(drop;)源 port 是安全组 sg1 port_group 里的（pod 绑定 sg，后会将 lsp 放到 port_group 里）table=9 (ls_in_acl          ), priority=3299 , match=(reg0[7] == 1 && (inport==@ovn.sg.sg1 && ip4 && ip4.dst==0.0.0.0/0)), action=(reg0[1] = 1; next;)table=9 (ls_in_acl          ), priority=3299 , match=(reg0[8] == 1 && (inport==@ovn.sg.sg1 && ip4 && ip4.dst==0.0.0.0/0)), action=(next;)Datapath: "ovn-default" (b6a669bb-984d-4c04-9622-64dba046cd55)  Pipeline: egress22 端口 table=4 (ls_out_acl         ), priority=3290 , match=(reg0[8] == 1 && (outport==@ovn.sg.sg1 && ip4 && ip4.src==0.0.0.0/0 && 22<=tcp.dst<=22)), action=(next;)icmp droptable=4 (ls_out_acl         ), priority=3290 , match=(reg0[9] == 1 && (outport==@ovn.sg.sg1 && ip4 && ip4.src==0.0.0.0/0 && icmp4)), action=(/* drop */)table=4 (ls_out_acl         ), priority=3003 , match=(reg0[9] == 1 && (outport==@ovn.sg.kubeovn_deny_all && ip)), action=(/* drop */)

Trace

$ kubectl ko trace default/pod9  10.16.4.138  icmp......4. ls_out_acl (northd.c:6472): reg0[9] == 1 && (outport==@ovn.sg.sg1 && ip4 && ip4.src==0.0.0.0/0 && icmp4), priority 3290, uuid 8d3eb6cfdrop;

DNAT

创建 ovn-external-gw-config
pod10 配置 eip 为 1.1.1.11/24，如上配置完成后。

apiVersion: v1 
kind: ConfigMap 
metadata: name: ovn-external-gw-config namespace: kube-system 
data: enable-external-gw: "true" external-gw-nodes: "worker" external-gw-nic: "eth1" nic-ip: "1.1.1.254/24" nic-mac: "00:00:3e:e7:00:eb" external-gw-addr: "1.1.1.1/24"

$ kubectl ko nbctl showswitch af2cc54a-ff49-4d5a-8037-ea07369de09d (ovn-external)port ln-ovn-externaltype: localnetaddresses: ["unknown"]port ovn-external-ovn-clustertype: routerrouter-port: ovn-cluster-ovn-external
router 5471f042-ce6f-470f-8f0d-55e75972206a (ovn-cluster)port ovn-cluster-subnetmac: "00:00:00:C8:1C:8A"networks: ["192.168.2.1/24", "bb00:192:168::1/64"]port ovn-cluster-ovn-defaultmac: "00:00:00:63:B3:F9"networks: ["10.16.0.1/16", "dd:10:16::1/64"]port ovn-cluster-ovn-externalmac: "00:00:3e:e7:00:eb"networks: ["1.1.1.254/24"]gateway chassis: [fcbaedd2-3ab3-41c5-bc75-e04436ed9eac]port ovn-cluster-joinmac: "00:00:00:E9:15:FF"networks: ["100.64.0.1/16", "dd:100:64::1/64"]nat b07e765d-7568-4389-ba75-3ebd8e8e421dexternal ip: "1.1.1.11"logical ip: "192.168.2.12"type: "dnat_and_snat"$ kubectl ko nbctl lr-route-list ovn-cluster
IPv4 Routes
Route Table 
:192.168.2.13                   1.1.1.1 src-ip

关键流表变化

Datapath: "ovn-cluster" (5ec1c73b-e8f2-4b11-8779-28e02a091b87)  Pipeline: ingress到 1.1.1.11 eip 的 arp 直接回复table=3 (lr_in_ip_input     ), priority=90   , match=(arp.op == 1 && arp.tpa == 1.1.1.11), action=(eth.dst = eth.src; eth.src = xreg0[0..47]; arp.op = 2; /* ARP reply */ arp.tha = arp.sha; arp.sha = xreg0[0..47]; arp.tpa <-> arp.spa; outport = inport; flags.loopback = 1; output;)table=4 (lr_in_unsnat       ), priority=90   , match=(ip && ip4.dst == 1.1.1.11), action=(ct_snat;)到 1.1.1.11 ct_dnat pod10 iptable=6 (lr_in_dnat         ), priority=100  , match=(ip && ip4.dst == 1.1.1.11), action=(flags.loopback = 1; ct_dnat(192.168.2.13);)pod 到外部的路由，下一跳到 1.1.1.1，源 mac 为 1.1.1.254 的 mactable=11(lr_in_ip_routing   ), priority=96   , match=(ip4.src == 192.168.2.13/32), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 1.1.1.1; reg1 = 1.1.1.254; eth.src = 00:00:3e:e7:00:eb; outport = "ovn-cluster-ovn-external"; flags.loopback = 1; next;)Datapath: "ovn-cluster" (5ec1c73b-e8f2-4b11-8779-28e02a091b87)  Pipeline: egresspod 到外部时的流量 snat 成 1.1.1.11table=3 (lr_out_snat        ), priority=33   , match=(ip && ip4.src == 192.168.2.13 && (!ct.trk || !ct.rpl)), action=(ct_snat(1.1.1.11);)

LB

LB 分为 LogicalRouter LB 和 LogicalSwitch LB，对于 OVN 来说，LB 是一个资源，然后将该资源关联到 LR 或 LS 上。

当前 kubeOVN 在 enable-lb 时，会关联到 LR 和 LS 上，以后会针对 子网 VPC 进行可选配置。

示例 10.96.99.99:8080 192.168.2.16:80 192.168.2.18:80

$ kubectl ko nbctl  lb-add lb0 10.96.99.99:8080 192.168.2.16:80,192.168.2.18:80 tcp
$ kubectl ko nbctl lr-lb-add ovn-cluster lb0

流表变化：

Datapath: "ovn-cluster" (5ec1c73b-e8f2-4b11-8779-28e02a091b87)  Pipeline: ingress访问 vip，reg0 存 vip，reg9[16..31] 存目的端口，ct_dnat 处理table=5 (lr_in_defrag       ), priority=110  , match=(ip && ip4.dst == 10.96.99.99 && tcp), action=(reg0 = 10.96.99.99; reg9[16..31] = tcp.dst; ct_dnat;)未建立连接的执行执行对应的 backendstable=6 (lr_in_dnat         ), priority=120  , match=(ct.est && ip4 && reg0 == 10.96.99.99 && tcp && reg9[16..31] == 8080 && ct_mark.natted == 1), action=(next;)table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.96.99.99 && tcp && reg9[16..31] == 8080), action=(ct_lb_mark(backends=192.168.2.16:80,192.168.2.18:80);)

FullNAT LB

FullNAT LB 即 在 LB 时，即做 DNAT 又 做 SNAT。

lb_force_snat_ip=router_ip：设置 对应接口的 ip 为 snat 的 ip，该模型下为子网的网关。即192.168.2.1

$ kubectl ko nbctl  set logical_router ovn-cluster options:lb_force_snat_ip=router_ip$ kubectl ko nbctl --wait=hv  set logical_router ovn-cluster options:chassis=fcbaedd2-3ab3-41c5-bc75-e04436ed9eac

流表变化：

Datapath: "ovn-cluster" (5ec1c73b-e8f2-4b11-8779-28e02a091b87)  Pipeline: ingress和普通 LB，action 加 flag force_snat_for_lb = 1table=6 (lr_in_dnat         ), priority=120  , match=(ct.est && ip4 && reg0 == 10.96.99.99 && tcp && reg9[16..31] == 8080 && ct_mark.natted == 1), action=(flags.force_snat_for_lb = 1; next;)table=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.96.99.99 && tcp && reg9[16..31] == 8080), action=(flags.force_snat_for_lb = 1; ct_lb_mark(backends=192.168.2.16:80,192.168.2.18:80);)egress 时，做 snat 到指定子网的网关
Datapath: "ovn-cluster" (5ec1c73b-e8f2-4b11-8779-28e02a091b87)  Pipeline: egresstable=3 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "ovn-cluster-join"), action=(ct_snat(100.64.0.1);)table=3 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "ovn-cluster-ovn-default"), action=(ct_snat(10.16.0.1);)table=3 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "ovn-cluster-ovn-external"), action=(ct_snat(1.1.1.254);)table=3 (lr_out_snat        ), priority=110  , match=(flags.force_snat_for_lb == 1 && ip4 && outport == "ovn-cluster-subnet"), action=(ct_snat(192.168.2.1);)

Service 扩展

Src-ip LB

Src-ip 是 kubernetes 对 service 能力的要求，根据 src-ip 访问不同的 endpoint。

$ kubectl ko nbctl  set Load_Balancer lb0 selection_fields=ip_src

流表变化：

Datapath: "ovn-cluster" (5ec1c73b-e8f2-4b11-8779-28e02a091b87)  Pipeline: ingresstable=6 (lr_in_dnat         ), priority=120  , match=(ct.new && ip4 && reg0 == 10.96.99.99 && tcp && reg9[16..31] == 8080), action=(ct_lb_mark(backends=192.168.2.16:80,192.168.2.18:80);)
=>更改计算方式match=(ct.new && ip4 && reg0 == 10.96.99.99 && tcp && reg9[16..31] == 8080), action=(ct_lb_mark(backends=192.168.2.16:80,192.168.2.18:80; hash_fields="ip_src");)