LACP 802.3ad の実験
今回の実験は、スイッチにおいて etherchannel 、サーバにおいて bonding を使って、Link Aggregation を実現できるか、というところです。
今回のトポロジは以下となります。
さて、設定していきます。
スイッチから。
IOU1#debug etherchannel event
PAgP/LACP Shim Events debugging is on
IOU1#conf t
IOU1(config)#int range e0/0 - 1
IOU1(config-if-range)#chann
IOU1(config-if-range)#channel-pro
IOU1(config-if-range)#channel-protocol ?
lacp Prepare interface for LACP protocol
pagp Prepare interface for PAgP protocol
IOU1(config-if-range)#channel-protocol lacp
IOU1(config-if-range)#channel-group 1 mode active
IOU1#sho ip int bri
Interface IP-Address OK? Method Status Protocol
Ethernet0/0 unassigned YES unset up up
Ethernet0/1 unassigned YES unset up up
Ethernet0/2 unassigned YES unset up up
Ethernet0/3 unassigned YES unset up up
Ethernet1/0 unassigned YES unset up up
Ethernet1/1 unassigned YES unset up up
Ethernet1/2 unassigned YES unset up up
Ethernet1/3 unassigned YES unset up up
Ethernet2/0 unassigned YES unset up up
Ethernet2/1 unassigned YES unset up up
Ethernet2/2 unassigned YES unset up up
Ethernet2/3 unassigned YES unset up up
Ethernet3/0 unassigned YES unset up up
Ethernet3/1 unassigned YES unset up up
Ethernet3/2 unassigned YES unset up up
Ethernet3/3 unassigned YES unset up up
Port-channel1 unassigned YES unset up up
Vlan1 unassigned YES unset administratively down down
うまくいくと、こうなる。
IOU1#show etherchannel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregatorM - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port
Number of channel-groups in use: 1
Number of aggregators: 1Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
1 Po1(SU) LACP Et0/0(P) Et0/1(P)
IOU1#
*Mar 25 12:02:34.740: FEC: lacp_fec_unbundle_internal: Et0/0
*Mar 25 12:02:34.740: FEC: lacp_switch_delete_port_from_agport_internal: removing Et0/0 from Po1
*Mar 25 12:02:34.740: FEC: pagp_switch_delete_port_from_agport_list: afb->nports-- = 1 [Et0/0]
*Mar 25 12:02:35.027: FEC: lacp_fec_unbundle_internal: Et0/1
*Mar 25 12:02:35.027: FEC: lacp_switch_delete_port_from_agport_internal: removing Et0/1 from Po1
*Mar 25 12:02:35.027: FEC: pagp_switch_delete_port_from_agport_list: afb->nports-- = 0 [Et0/1]
*Mar 25 12:02:35.744: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to down
IOU1#
*Mar 25 12:02:36.031: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to down
IOU1#
*Mar 25 12:02:37.031: %LINK-3-UPDOWN: Interface Port-channel1, changed state to down
*Mar 25 12:02:38.031: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to down
IOU1#
*Mar 25 12:02:39.878: FEC: lacp_fec_dontbundle_internal: Et0/0
IOU1#
*Mar 25 12:02:39.878: %EC-5-L3DONTBNDL2: Et0/0 suspended: LACP currently not enabled on the remote port.
*Mar 25 12:02:40.270: FEC: lacp_fec_dontbundle_internal: Et0/1
IOU1#
*Mar 25 12:02:40.270: %EC-5-L3DONTBNDL2: Et0/1 suspended: LACP currently not enabled on the remote port.
IOU1#
*Mar 25 12:02:50.219: FEC: lacp_fec_unbundle_internal: Et0/1
*Mar 25 12:02:50.219: FEC: Unbundling req for port: Et0/1 neitherstand-alone Agport nor in a bundle
*Mar 25 12:02:50.219: FEC: lacp_switch_add_port_to_associated_list_internal: Et0/1 added to list for Po1
*Mar 25 12:02:50.219: FEC: lacp_fec_unbundle_internal: Et0/0
*Mar 25 12:02:50.219: FEC: Unbundling req for port: Et0/0 neitherstand-alone Agport nor in a bundle
*Mar 25 12:02:50.220: FEC: lacp_switch_add_port_to_associated_list_internal: Et0/0 added to list for Po1
*Mar 25 12:02:51.223: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to up
*Mar 25 12:02:51.223: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up
IOU1#
*Mar 25 12:02:53.810: FEC: pagp_switch_add_port_to_agport_list: afb->nports++ = 1 [Et0/0]
*Mar 25 12:02:53.810: FEC: lacp_switch_add_port_to_agport_internal: Et0/0 added to aggregator Po1 list
*Mar 25 12:02:53.889: FEC: pagp_switch_add_port_to_agport_list: afb->nports++ = 2 [Et0/1]
*Mar 25 12:02:53.889: FEC: lacp_switch_add_port_to_agport_internal: Et0/1 added to aggregator Po1 list
IOU1#
*Mar 25 12:02:55.814: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
*Mar 25 12:02:56.819: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up
次のコマンドの結果
IOU1#show etherchannel port-channel
Channel-group listing:
----------------------Group: 1
----------
Port-channels in the group:
---------------------------Port-channel: Po1 (Primary Aggregator)
------------
Age of the Port-channel = 0d:01h:33m:19s
Logical slot/port = 16/0 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Port security = DisabledPorts in the Port-channel:
Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 00 Et0/0 Active 0
0 00 Et0/1 Active 0
Time since last port bundled: 0d:00h:28m:03s Et0/1
Time since last port Un-bundled: 0d:00h:29m:30s Et0/1
次のコマンドの結果
IOU1#show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ipEtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
次のコマンドの結果
IOU1#show etherchannel detail
Channel-group listing:
----------------------Group: 1
----------
Group state = L2
Ports: 2 Maxports = 4
Port-channels: 1 Max Port-channels = 4
Protocol: LACP
Minimum Links: 0
Ports in the group:
-------------------
Port: Et0/0
------------Port state = Up Mstr Assoc In-Bndl
Channel group = 1 Mode = Active Gcchange = -
Port-channel = Po1 GC = - Pseudo port-channel = Po1
Port index = 0 Load = 0x00 Protocol = LACPFlags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Et0/0 SA bndl 32768 0x1 0x1 0x1 0x3D
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Et0/0 FA 255 0800.27a9.4bf0 16s 0x0 0x9 0x1 0xF
Age of the port in the current state: 0d:00h:22m:22s
Port: Et0/1
------------
Port state = Up Mstr Assoc In-Bndl
Channel group = 1 Mode = Active Gcchange = -
Port-channel = Po1 GC = - Pseudo port-channel = Po1
Port index = 0 Load = 0x00 Protocol = LACP
Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.
Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Et0/1 SA bndl 32768 0x1 0x1 0x2 0x3D
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Et0/1 FA 255 0800.27a9.4bf0 16s 0x0 0x9 0x2 0xF
Age of the port in the current state: 0d:00h:22m:21s
Port-channels in the group:
---------------------------
Port-channel: Po1 (Primary Aggregator)
------------
Age of the Port-channel = 0d:01h:27m:37s
Logical slot/port = 16/0 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Port security = Disabled
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 00 Et0/0 Active 0
0 00 Et0/1 Active 0
Time since last port bundled: 0d:00h:22m:21s Et0/1
Time since last port Un-bundled: 0d:00h:23m:48s Et0/1
サーバの bonding の設定は、よくあるやつでOK(802.3ad なので、mode = 4)
うまくいくと、こうなる。
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 08:00:27:a9:4b:f0
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 9
Partner Key: 1
Partner Mac Address: aa:bb:cc:00:01:00Slave Interface: enp0s3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:a9:4b:f0
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 1
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: aa:bb:cc:00:01:00
oper key: 1
port priority: 32768
port number: 1
port state: 61Slave Interface: enp0s8
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:e0:d6:ae
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 2
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: aa:bb:cc:00:01:00
oper key: 1
port priority: 32768
port number: 2
port state: 61
ここで、実験。
# ifdown bond0
# ifup bond0
したり、
スイッチを停止したり、再起動したりして確認する。
をやってみる。
サーバで、# ifdown bond0 としたときの、スイッチのデバッグログ(一例)
IOU1#
*Mar 25 12:09:08.045: FEC: lacp_fec_unbundle_internal: Et0/0
*Mar 25 12:09:08.045: FEC: lacp_switch_delete_port_from_agport_internal: removing Et0/0 from Po1
*Mar 25 12:09:08.045: FEC: pagp_switch_delete_port_from_agport_list: afb->nports-- = 1 [Et0/0]
*Mar 25 12:09:08.329: FEC: lacp_fec_unbundle_internal: Et0/1
*Mar 25 12:09:08.329: FEC: lacp_switch_delete_port_from_agport_internal: removing Et0/1 from Po1
*Mar 25 12:09:08.329: FEC: pagp_switch_delete_port_from_agport_list: afb->nports-- = 0 [Et0/1]
*Mar 25 12:09:09.052: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to down
IOU1#
*Mar 25 12:09:09.330: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to down
*Mar 25 12:09:10.330: %LINK-3-UPDOWN: Interface Port-channel1, changed state to down
IOU1#
*Mar 25 12:09:11.331: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to down
IOU1#
*Mar 25 12:09:13.590: FEC: lacp_fec_dontbundle_internal: Et0/1
IOU1#
*Mar 25 12:09:13.590: %EC-5-L3DONTBNDL2: Et0/1 suspended: LACP currently not enabled on the remote port.
*Mar 25 12:09:13.917: FEC: lacp_fec_dontbundle_internal: Et0/0
IOU1#
*Mar 25 12:09:13.917: %EC-5-L3DONTBNDL2: Et0/0 suspended: LACP currently not enabled on the remote port.
スイッチを停止した時の、/proc/net/bonding/bond0 の結果(Link Aggregation できていない)(一例)
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0802.3ad info
LACP rate: fast
Min links: 2
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 08:00:27:a9:4b:f0
Active Aggregator Info:
Aggregator ID: 7
Number of ports: 1
Actor Key: 9
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00Slave Interface: enp0s3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:a9:4b:f0
Slave queue ID: 0
Aggregator ID: 7
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 1
port state: 79
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1Slave Interface: enp0s8
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:e0:d6:ae
Slave queue ID: 0
Aggregator ID: 8
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 2
port state: 71
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1
サーバで、# ifup bnod0 したときの、スイッチのデバッグログ(一例)
IOU1#
*Mar 25 12:11:37.006: FEC: lacp_fec_unbundle_internal: Et0/1
*Mar 25 12:11:37.006: FEC: Unbundling req for port: Et0/1 neitherstand-alone Agport nor in a bundle
*Mar 25 12:11:37.006: FEC: lacp_switch_add_port_to_associated_list_internal: Et0/1 added to list for Po1
*Mar 25 12:11:37.006: FEC: lacp_fec_unbundle_internal: Et0/0
*Mar 25 12:11:37.006: FEC: Unbundling req for port: Et0/0 neitherstand-alone Agport nor in a bundle
*Mar 25 12:11:37.006: FEC: lacp_switch_add_port_to_associated_list_internal: Et0/0 added to list for Po1
*Mar 25 12:11:38.006: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to up
*Mar 25 12:11:38.006: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up
IOU1#
*Mar 25 12:11:40.447: FEC: pagp_switch_add_port_to_agport_list: afb->nports++ = 1 [Et0/0]
*Mar 25 12:11:40.447: FEC: lacp_switch_add_port_to_agport_internal: Et0/0 added to aggregator Po1 list
*Mar 25 12:11:40.448: %IDBMAN-3-INVALIDAGGPORTBANDWIDTH: Port-channel1(16 / 0) has an invalid bandwidth value of 0
*Mar 25 12:11:40.639: FEC: pagp_switch_add_port_to_agport_list: afb->nports++ = 2 [Et0/1]
*Mar 25 12:11:40.639: FEC: lacp_switch_add_port_to_agport_internal: Et0/1 added to aggregator Po1 list
IOU1#
*Mar 25 12:11:40.639: %IDBMAN-3-INVALIDAGGPORTBANDWIDTH: Port-channel1(16 / 0) has an invalid bandwidth value of 0
IOU1#
*Mar 25 12:11:42.450: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
*Mar 25 12:11:43.455: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up
スイッチを再起動した時の、/proc/net/bonding/bond0 の結果(一例)
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0802.3ad info
LACP rate: fast
Min links: 2
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 08:00:27:a9:4b:f0
Active Aggregator Info:
Aggregator ID: 7
Number of ports: 2
Actor Key: 9
Partner Key: 1
Partner Mac Address: aa:bb:cc:00:01:00Slave Interface: enp0s3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:a9:4b:f0
Slave queue ID: 0
Aggregator ID: 7
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 1
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: aa:bb:cc:00:01:00
oper key: 1
port priority: 32768
port number: 1
port state: 13Slave Interface: enp0s8
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:e0:d6:ae
Slave queue ID: 0
Aggregator ID: 7
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 1
Partner Churned Count: 2
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 2
port state: 15
details partner lacp pdu:
system priority: 32768
system mac address: aa:bb:cc:00:01:00
oper key: 1
port priority: 32768
port number: 2
port state: 5
再起動しても、異常なく繋がった。
(参考)
https://www.kernel.org/doc/Documentation/networking/bonding.txt