Article Number: 000020056
Isilon OneFS: Understanding Isilon Source Based Routing (SBR)
Summary: This article describes how standard routing and Source Based Routing (SBR) works within an Isilon node.
Article Content
Instructions
What is routing?
Routing is the process of determining how to get IP packets from a source to a destination.
Source-Based Routing (SBR) with Isilon OneFS is discussed in the Isilon OneFS 8.1.0 External Network Connectivity Guide. This section clarifies how SBR functions. The naming convention suggests that SBR is routing packets based on a source IP address. However, SBR is a mechanism to dynamically create per-subnet default routes. The router used as this gateway is derived from the subnet configuration. Gateways must be defined for each subnet. For example, consider a cluster with subnets A, B, and C, as illustrated in the following figure:
In the example above, each gateway has a defined priority. If SBR is not configured, the highest priority gateway, for example a gateway with the lowest reachable value, is used as the default route. Once SBR is enabled, when traffic arrives from a subnet that is not reachable through the default gateway, firewall rules are added. As OneFS is FreeBSD-based, these are added through ipfw (IP firewall). In the example above, the following ipfw rules are provisioned:
If src-ip is in subnetA and dst-ip is not in (subnetA,B,C) set next-hop to gatewayA
If src-ip is in subnetB and dst-ip is not in (subnetA,B,C) set next-hop to gatewayB
If src-ip is in subnetC and dst-ip is not in (subnetA,B,C) set next-hop to gatewayC
The process of adding ipfw rules is stateless and essentially translates to per-subnet default routes. SBR depends entirely on the source IP address that sends traffic to the cluster. If a session is initiated from the source subnet, the ipfw rule is created. The session must be initiated from the source subnet, otherwise the ipfw rule is not created. If the cluster has not received traffic that originated from a subnet not reachable through the default gateway, OneFS transmits traffic it originates through the default gateway. Given how SBR creates per-subnet default routes, consider the following:
If the source IP address and the destination IP address are in subnet A (in the same broadcast domain), the packet goes directly to the client IP address. If the source IP address is in subnet A and the destination IP address is NOT in subnet A, the packet is sent to the next-hop gateway.
Shown here, the client must send a packet to the Isilon cluster at IP address 10.3.1.90. First, the client determines that the destination IP address is not on its local network, and does not have a static route defined for that address. The client sends the packet to its default gateway (Router C) for more processing. Next, Router C receives the packet from the client and examines the packet destination IP address and determines that it has a route to the destination through router A at 10.3.1.1. Then, router A receives the packet on its external interface and determines that it has a direct connection to the destination IP address 10.3.1.90. Router A sends the packet directly to its destination using its interface on the 10GbE network.
Next, the Isilon must send a response packet to the client.
SBR Disabled:
The node determines that the destination IP address 10.2.1.50 is not local (not on the same LAN) and does not have a static route defined for that address. OneFS determines which gateway it must send the response packet based on the routing table. Gateways with higher priority (lower integer value) have precedence over those with lower priority (higher integer value). For example, 1 is a higher priority than 5, 5 is a higher priority than 11, and so forth. The Isilon node has one default gateway, which is the highest priority subnet the node is configured in. Since there is no static route configured on the node, OneFS chooses the default gateway 10.1.1.1 (router B) through the 1GbE interface.
The reply packet sent by the 1GbE interface has a source IP header of 10.3.1.90. Some networks have security measures that block this type of behavior called spoofing. This condition may also cause performance issues as the connection may be rate limited based on the lowest network link (1 GbE). Where a customer may expect 10 GbE speeds but getting 1 GbE throughput.
SBR Enabled:
The node does not decide the route based off the destination IP address of the client. Instead, OneFS reads the egress packet source IP header, then sends the packet out the physical port and gateway associated with the source IP subnet. SBR does not override statically configured routes. If SBR is enabled and a static route is created, a new rule is added. SBR only works on reply traffic, it does NOT apply to traffic initiated by the node. For example, when a node reaches out to Domain Name Service (DNS), or to Active Directory, traditional routing rules (as though SBR is disabled) apply. SBR is a cluster-wide configuration option.
The nodes reply packet has a source IP address of 10.3.1.90. SBR routing rules say that the gateway of this IP is 10.3.1.1. The path to this gateway is through 10 GbE. When the response has reached Router A, it travels back to Router C through the core network, and, finally, returns to the client.
Remember! Though SBR was developed to be enabled or disabled as easily as possible, when enabling, packets leaving the cluster may be routed differently. How this affects a customer depends on their network setup. Consider enabling source-based routing when implementing a cluster in a large network with a complex topology. For example, if the network is a multi-tenant environment with several gateways, traffic is more efficiently distributed with source-based routing.
How to check SBR in the cluster
To view if SBR is enabled on a cluster, run # isi network external view. In the output, if SBR is not enabled on the cluster, Source Based Routing is False. If SBR is enabled, Source Based Routing is True.
# isi network external view
Client TCP Ports: 2049, 445, 20, 21, 80, 53, 1234
Default Groupnet: groupnet0
SC Rebalance Delay: 0
Source Based Routing: True
SBR can only be enabled from the CLI on OneFS 7.x. However, the ability to enable/disable SBR on 8.x was added to the WebUI. Thus, 8.x can be enabled/disabled through the WebUI and CLI.
SBR can be enabled or disabled by running the following command:
# isi network external modify --sbr=[false|true]
It can also be enabled/disabled through the WebUI at the following location:
Cluster Management (main tab) > Network Configuration > Settings (sub tab) > Enable source based routing (check box)
Routing is the process of determining how to get IP packets from a source to a destination.
Source-Based Routing (SBR) with Isilon OneFS is discussed in the Isilon OneFS 8.1.0 External Network Connectivity Guide. This section clarifies how SBR functions. The naming convention suggests that SBR is routing packets based on a source IP address. However, SBR is a mechanism to dynamically create per-subnet default routes. The router used as this gateway is derived from the subnet configuration. Gateways must be defined for each subnet. For example, consider a cluster with subnets A, B, and C, as illustrated in the following figure:
In the example above, each gateway has a defined priority. If SBR is not configured, the highest priority gateway, for example a gateway with the lowest reachable value, is used as the default route. Once SBR is enabled, when traffic arrives from a subnet that is not reachable through the default gateway, firewall rules are added. As OneFS is FreeBSD-based, these are added through ipfw (IP firewall). In the example above, the following ipfw rules are provisioned:
If src-ip is in subnetA and dst-ip is not in (subnetA,B,C) set next-hop to gatewayA
If src-ip is in subnetB and dst-ip is not in (subnetA,B,C) set next-hop to gatewayB
If src-ip is in subnetC and dst-ip is not in (subnetA,B,C) set next-hop to gatewayC
The process of adding ipfw rules is stateless and essentially translates to per-subnet default routes. SBR depends entirely on the source IP address that sends traffic to the cluster. If a session is initiated from the source subnet, the ipfw rule is created. The session must be initiated from the source subnet, otherwise the ipfw rule is not created. If the cluster has not received traffic that originated from a subnet not reachable through the default gateway, OneFS transmits traffic it originates through the default gateway. Given how SBR creates per-subnet default routes, consider the following:
- A subnet setting of 0.0.0.0 is not supported and is severely problematic, as OneFS does not support RIP, RARP, or CDP.
- The default gateway is the path for all traffic intended for clients not on the local subnet and not covered by a routing table entry. Utilizing SBR does not negate the requirement for a default gateway, as SBR in effect overrides the default gateway, but not static routes.
- Static routes are an option when the cluster originates the traffic, and the route is not accessible through the default gateway. As mentioned above, static routes are prioritized over source-based routing rules.
If the source IP address and the destination IP address are in subnet A (in the same broadcast domain), the packet goes directly to the client IP address. If the source IP address is in subnet A and the destination IP address is NOT in subnet A, the packet is sent to the next-hop gateway.
Shown here, the client must send a packet to the Isilon cluster at IP address 10.3.1.90. First, the client determines that the destination IP address is not on its local network, and does not have a static route defined for that address. The client sends the packet to its default gateway (Router C) for more processing. Next, Router C receives the packet from the client and examines the packet destination IP address and determines that it has a route to the destination through router A at 10.3.1.1. Then, router A receives the packet on its external interface and determines that it has a direct connection to the destination IP address 10.3.1.90. Router A sends the packet directly to its destination using its interface on the 10GbE network.
Next, the Isilon must send a response packet to the client.
SBR Disabled:
The node determines that the destination IP address 10.2.1.50 is not local (not on the same LAN) and does not have a static route defined for that address. OneFS determines which gateway it must send the response packet based on the routing table. Gateways with higher priority (lower integer value) have precedence over those with lower priority (higher integer value). For example, 1 is a higher priority than 5, 5 is a higher priority than 11, and so forth. The Isilon node has one default gateway, which is the highest priority subnet the node is configured in. Since there is no static route configured on the node, OneFS chooses the default gateway 10.1.1.1 (router B) through the 1GbE interface.
The reply packet sent by the 1GbE interface has a source IP header of 10.3.1.90. Some networks have security measures that block this type of behavior called spoofing. This condition may also cause performance issues as the connection may be rate limited based on the lowest network link (1 GbE). Where a customer may expect 10 GbE speeds but getting 1 GbE throughput.
SBR Enabled:
The node does not decide the route based off the destination IP address of the client. Instead, OneFS reads the egress packet source IP header, then sends the packet out the physical port and gateway associated with the source IP subnet. SBR does not override statically configured routes. If SBR is enabled and a static route is created, a new rule is added. SBR only works on reply traffic, it does NOT apply to traffic initiated by the node. For example, when a node reaches out to Domain Name Service (DNS), or to Active Directory, traditional routing rules (as though SBR is disabled) apply. SBR is a cluster-wide configuration option.
The nodes reply packet has a source IP address of 10.3.1.90. SBR routing rules say that the gateway of this IP is 10.3.1.1. The path to this gateway is through 10 GbE. When the response has reached Router A, it travels back to Router C through the core network, and, finally, returns to the client.
Remember! Though SBR was developed to be enabled or disabled as easily as possible, when enabling, packets leaving the cluster may be routed differently. How this affects a customer depends on their network setup. Consider enabling source-based routing when implementing a cluster in a large network with a complex topology. For example, if the network is a multi-tenant environment with several gateways, traffic is more efficiently distributed with source-based routing.
How to check SBR in the cluster
To view if SBR is enabled on a cluster, run # isi network external view. In the output, if SBR is not enabled on the cluster, Source Based Routing is False. If SBR is enabled, Source Based Routing is True.
# isi network external view
Client TCP Ports: 2049, 445, 20, 21, 80, 53, 1234
Default Groupnet: groupnet0
SC Rebalance Delay: 0
Source Based Routing: True
SBR can only be enabled from the CLI on OneFS 7.x. However, the ability to enable/disable SBR on 8.x was added to the WebUI. Thus, 8.x can be enabled/disabled through the WebUI and CLI.
SBR can be enabled or disabled by running the following command:
# isi network external modify --sbr=[false|true]
It can also be enabled/disabled through the WebUI at the following location:
Cluster Management (main tab) > Network Configuration > Settings (sub tab) > Enable source based routing (check box)
Additional Information
Source for this material: Dell PowerScale: Network Design Considerations
Here are some recommended resources related to this topic that might be of interest:
Here are some recommended resources related to this topic that might be of interest:
- Isilon: (Gen6): Performance degradation can occur when a subnet containing a bge interface is configured for the highest gateway priority and Source-Based Routing (SBR) is enabled.
- Isilon: How to view network routing rules with Source Based Routing enabled on an Isilon cluster
- PowerScale: OneFS Upgrades Info Hub
- PowerScale: How to Run the IOCA Cluster Analysis Tool
Article Properties
Affected Product
Isilon, PowerScale OneFS, Isilon X210
Last Published Date
28 Feb 2024
Version
5
Article Type
How To