computerの日記

Cisco,SHELL,C,Qt,C++,Linux,ネットワーク,Windows Scriptなどの発言です

LACP 802.3ad の実験

今回の実験は、スイッチにおいて etherchannel 、サーバにおいて bonding を使って、Link Aggregation を実現できるか、というところです。

今回のトポロジは以下となります。

f:id:intrajp:20180325190140p:plain

さて、設定していきます。

スイッチから。

IOU1#debug etherchannel event
PAgP/LACP Shim Events debugging is on

IOU1#conf t

IOU1(config)#int range e0/0 - 1
IOU1(config-if-range)#chann
IOU1(config-if-range)#channel-pro
IOU1(config-if-range)#channel-protocol ?
lacp Prepare interface for LACP protocol
pagp Prepare interface for PAgP protocol

IOU1(config-if-range)#channel-protocol lacp

IOU1(config-if-range)#channel-group 1 mode active

IOU1#sho ip int bri
Interface IP-Address OK? Method Status Protocol
Ethernet0/0 unassigned YES unset up up
Ethernet0/1 unassigned YES unset up up
Ethernet0/2 unassigned YES unset up up
Ethernet0/3 unassigned YES unset up up
Ethernet1/0 unassigned YES unset up up
Ethernet1/1 unassigned YES unset up up
Ethernet1/2 unassigned YES unset up up
Ethernet1/3 unassigned YES unset up up
Ethernet2/0 unassigned YES unset up up
Ethernet2/1 unassigned YES unset up up
Ethernet2/2 unassigned YES unset up up
Ethernet2/3 unassigned YES unset up up
Ethernet3/0 unassigned YES unset up up
Ethernet3/1 unassigned YES unset up up
Ethernet3/2 unassigned YES unset up up
Ethernet3/3 unassigned YES unset up up
Port-channel1 unassigned YES unset up up
Vlan1 unassigned YES unset administratively down down

うまくいくと、こうなる。

IOU1#show etherchannel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator

M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port


Number of channel-groups in use: 1
Number of aggregators: 1

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
1 Po1(SU) LACP Et0/0(P) Et0/1(P)

 

 

IOU1#
*Mar 25 12:02:34.740: FEC: lacp_fec_unbundle_internal: Et0/0
*Mar 25 12:02:34.740: FEC: lacp_switch_delete_port_from_agport_internal: removing Et0/0 from Po1
*Mar 25 12:02:34.740: FEC: pagp_switch_delete_port_from_agport_list: afb->nports-- = 1 [Et0/0]
*Mar 25 12:02:35.027: FEC: lacp_fec_unbundle_internal: Et0/1
*Mar 25 12:02:35.027: FEC: lacp_switch_delete_port_from_agport_internal: removing Et0/1 from Po1
*Mar 25 12:02:35.027: FEC: pagp_switch_delete_port_from_agport_list: afb->nports-- = 0 [Et0/1]
*Mar 25 12:02:35.744: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to down
IOU1#
*Mar 25 12:02:36.031: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to down
IOU1#
*Mar 25 12:02:37.031: %LINK-3-UPDOWN: Interface Port-channel1, changed state to down
*Mar 25 12:02:38.031: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to down
IOU1#
*Mar 25 12:02:39.878: FEC: lacp_fec_dontbundle_internal: Et0/0
IOU1#
*Mar 25 12:02:39.878: %EC-5-L3DONTBNDL2: Et0/0 suspended: LACP currently not enabled on the remote port.
*Mar 25 12:02:40.270: FEC: lacp_fec_dontbundle_internal: Et0/1
IOU1#
*Mar 25 12:02:40.270: %EC-5-L3DONTBNDL2: Et0/1 suspended: LACP currently not enabled on the remote port.
IOU1#
*Mar 25 12:02:50.219: FEC: lacp_fec_unbundle_internal: Et0/1
*Mar 25 12:02:50.219: FEC: Unbundling req for port: Et0/1 neitherstand-alone Agport nor in a bundle
*Mar 25 12:02:50.219: FEC: lacp_switch_add_port_to_associated_list_internal: Et0/1 added to list for Po1
*Mar 25 12:02:50.219: FEC: lacp_fec_unbundle_internal: Et0/0
*Mar 25 12:02:50.219: FEC: Unbundling req for port: Et0/0 neitherstand-alone Agport nor in a bundle
*Mar 25 12:02:50.220: FEC: lacp_switch_add_port_to_associated_list_internal: Et0/0 added to list for Po1
*Mar 25 12:02:51.223: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to up
*Mar 25 12:02:51.223: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up
IOU1#
*Mar 25 12:02:53.810: FEC: pagp_switch_add_port_to_agport_list: afb->nports++ = 1 [Et0/0]
*Mar 25 12:02:53.810: FEC: lacp_switch_add_port_to_agport_internal: Et0/0 added to aggregator Po1 list
*Mar 25 12:02:53.889: FEC: pagp_switch_add_port_to_agport_list: afb->nports++ = 2 [Et0/1]
*Mar 25 12:02:53.889: FEC: lacp_switch_add_port_to_agport_internal: Et0/1 added to aggregator Po1 list
IOU1#
*Mar 25 12:02:55.814: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
*Mar 25 12:02:56.819: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up

 

次のコマンドの結果

IOU1#show etherchannel port-channel
Channel-group listing:
----------------------

Group: 1
----------
Port-channels in the group:
---------------------------

Port-channel: Po1 (Primary Aggregator)

------------

Age of the Port-channel = 0d:01h:33m:19s
Logical slot/port = 16/0 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Port security = Disabled

Ports in the Port-channel:

Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 00 Et0/0 Active 0
0 00 Et0/1 Active 0

Time since last port bundled: 0d:00h:28m:03s Et0/1
Time since last port Un-bundled: 0d:00h:29m:30s Et0/1

 

次のコマンドの結果

IOU1#show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address

 

次のコマンドの結果

IOU1#show etherchannel detail
Channel-group listing:
----------------------

Group: 1
----------
Group state = L2
Ports: 2 Maxports = 4
Port-channels: 1 Max Port-channels = 4
Protocol: LACP
Minimum Links: 0


Ports in the group:
-------------------
Port: Et0/0
------------

Port state = Up Mstr Assoc In-Bndl
Channel group = 1 Mode = Active Gcchange = -
Port-channel = Po1 GC = - Pseudo port-channel = Po1
Port index = 0 Load = 0x00 Protocol = LACP

Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.

Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Et0/0 SA bndl 32768 0x1 0x1 0x1 0x3D

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Et0/0 FA 255 0800.27a9.4bf0 16s 0x0 0x9 0x1 0xF

Age of the port in the current state: 0d:00h:22m:22s

Port: Et0/1
------------

Port state = Up Mstr Assoc In-Bndl
Channel group = 1 Mode = Active Gcchange = -
Port-channel = Po1 GC = - Pseudo port-channel = Po1
Port index = 0 Load = 0x00 Protocol = LACP

Flags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.

Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Et0/1 SA bndl 32768 0x1 0x1 0x2 0x3D

Partner's information:

LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Et0/1 FA 255 0800.27a9.4bf0 16s 0x0 0x9 0x2 0xF

Age of the port in the current state: 0d:00h:22m:21s

Port-channels in the group:
---------------------------

Port-channel: Po1 (Primary Aggregator)

------------

Age of the Port-channel = 0d:01h:27m:37s
Logical slot/port = 16/0 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Port security = Disabled

Ports in the Port-channel:

Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 00 Et0/0 Active 0
0 00 Et0/1 Active 0

Time since last port bundled: 0d:00h:22m:21s Et0/1
Time since last port Un-bundled: 0d:00h:23m:48s Et0/1

 

サーバの bonding の設定は、よくあるやつでOK(802.3ad なので、mode = 4)

うまくいくと、こうなる。

# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 08:00:27:a9:4b:f0
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 9
Partner Key: 1
Partner Mac Address: aa:bb:cc:00:01:00

Slave Interface: enp0s3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:a9:4b:f0
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 1
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: aa:bb:cc:00:01:00
oper key: 1
port priority: 32768
port number: 1
port state: 61

Slave Interface: enp0s8
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:e0:d6:ae
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 2
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: aa:bb:cc:00:01:00
oper key: 1
port priority: 32768
port number: 2
port state: 61

  

ここで、実験。

# ifdown bond0

# ifup bond0

したり、

スイッチを停止したり、再起動したりして確認する。

をやってみる。

サーバで、# ifdown bond0 としたときの、スイッチのデバッグログ(一例)

IOU1#
*Mar 25 12:09:08.045: FEC: lacp_fec_unbundle_internal: Et0/0
*Mar 25 12:09:08.045: FEC: lacp_switch_delete_port_from_agport_internal: removing Et0/0 from Po1
*Mar 25 12:09:08.045: FEC: pagp_switch_delete_port_from_agport_list: afb->nports-- = 1 [Et0/0]
*Mar 25 12:09:08.329: FEC: lacp_fec_unbundle_internal: Et0/1
*Mar 25 12:09:08.329: FEC: lacp_switch_delete_port_from_agport_internal: removing Et0/1 from Po1
*Mar 25 12:09:08.329: FEC: pagp_switch_delete_port_from_agport_list: afb->nports-- = 0 [Et0/1]
*Mar 25 12:09:09.052: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to down
IOU1#
*Mar 25 12:09:09.330: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to down
*Mar 25 12:09:10.330: %LINK-3-UPDOWN: Interface Port-channel1, changed state to down
IOU1#
*Mar 25 12:09:11.331: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to down
IOU1#
*Mar 25 12:09:13.590: FEC: lacp_fec_dontbundle_internal: Et0/1
IOU1#
*Mar 25 12:09:13.590: %EC-5-L3DONTBNDL2: Et0/1 suspended: LACP currently not enabled on the remote port.
*Mar 25 12:09:13.917: FEC: lacp_fec_dontbundle_internal: Et0/0
IOU1#
*Mar 25 12:09:13.917: %EC-5-L3DONTBNDL2: Et0/0 suspended: LACP currently not enabled on the remote port.

 

 

スイッチを停止した時の、/proc/net/bonding/bond0 の結果(Link Aggregation できていない)(一例)

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 2
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 08:00:27:a9:4b:f0
Active Aggregator Info:
Aggregator ID: 7
Number of ports: 1
Actor Key: 9
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00

Slave Interface: enp0s3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:a9:4b:f0
Slave queue ID: 0
Aggregator ID: 7
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 1
port state: 79
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1

Slave Interface: enp0s8
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:e0:d6:ae
Slave queue ID: 0
Aggregator ID: 8
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 2
port state: 71
details partner lacp pdu:
system priority: 65535
system mac address: 00:00:00:00:00:00
oper key: 1
port priority: 255
port number: 1
port state: 1

 

サーバで、# ifup bnod0 したときの、スイッチのデバッグログ(一例)

IOU1#
*Mar 25 12:11:37.006: FEC: lacp_fec_unbundle_internal: Et0/1
*Mar 25 12:11:37.006: FEC: Unbundling req for port: Et0/1 neitherstand-alone Agport nor in a bundle
*Mar 25 12:11:37.006: FEC: lacp_switch_add_port_to_associated_list_internal: Et0/1 added to list for Po1
*Mar 25 12:11:37.006: FEC: lacp_fec_unbundle_internal: Et0/0
*Mar 25 12:11:37.006: FEC: Unbundling req for port: Et0/0 neitherstand-alone Agport nor in a bundle
*Mar 25 12:11:37.006: FEC: lacp_switch_add_port_to_associated_list_internal: Et0/0 added to list for Po1
*Mar 25 12:11:38.006: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/1, changed state to up
*Mar 25 12:11:38.006: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up
IOU1#
*Mar 25 12:11:40.447: FEC: pagp_switch_add_port_to_agport_list: afb->nports++ = 1 [Et0/0]
*Mar 25 12:11:40.447: FEC: lacp_switch_add_port_to_agport_internal: Et0/0 added to aggregator Po1 list
*Mar 25 12:11:40.448: %IDBMAN-3-INVALIDAGGPORTBANDWIDTH: Port-channel1(16 / 0) has an invalid bandwidth value of 0
*Mar 25 12:11:40.639: FEC: pagp_switch_add_port_to_agport_list: afb->nports++ = 2 [Et0/1]
*Mar 25 12:11:40.639: FEC: lacp_switch_add_port_to_agport_internal: Et0/1 added to aggregator Po1 list
IOU1#
*Mar 25 12:11:40.639: %IDBMAN-3-INVALIDAGGPORTBANDWIDTH: Port-channel1(16 / 0) has an invalid bandwidth value of 0
IOU1#
*Mar 25 12:11:42.450: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
*Mar 25 12:11:43.455: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up

 

スイッチを再起動した時の、/proc/net/bonding/bond0 の結果(一例)

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 2
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 08:00:27:a9:4b:f0
Active Aggregator Info:
Aggregator ID: 7
Number of ports: 2
Actor Key: 9
Partner Key: 1
Partner Mac Address: aa:bb:cc:00:01:00

Slave Interface: enp0s3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:a9:4b:f0
Slave queue ID: 0
Aggregator ID: 7
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 1
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: aa:bb:cc:00:01:00
oper key: 1
port priority: 32768
port number: 1
port state: 13

Slave Interface: enp0s8
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:e0:d6:ae
Slave queue ID: 0
Aggregator ID: 7
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 1
Partner Churned Count: 2
details actor lacp pdu:
system priority: 65535
system mac address: 08:00:27:a9:4b:f0
port key: 9
port priority: 255
port number: 2
port state: 15
details partner lacp pdu:
system priority: 32768
system mac address: aa:bb:cc:00:01:00
oper key: 1
port priority: 32768
port number: 2
port state: 5

 

 

再起動しても、異常なく繋がった。

 (参考)

https://www.kernel.org/doc/Documentation/networking/bonding.txt