Interconnectivity with Xsan is usually a pretty straight forward beast. Make sure you can communicate in an unfettered manner on a house network, on a metadata network and on a fibre channel network and you're pretty much good to go. One thing that seems to confuse a lot of people when they're first starting out is how to configure the two ethernets. We're going to go ahead and do two things at once, explain how to configure the interface and show how to automate said configuration from the command line so you can quickly deploy and then subsequently troubleshoot issues that you encounter from the perspective of the Ethernet networks.

Note: Strictly speaking you don't have to have two networks with Xsan. But you really should. Otherwise if you encounter volume corruption everyone else is just going to laugh at you in an Adam Sandler kinda way. Moving on...

Before getting started, it is important to note that there is a very important distinction in the nomenclature used in Mac OS X in regard to network interfaces vs. network services. An interface is a physical network adapter. These are indicated by traditional Unix names for them, en0, en1, fw0, etc. You can isolate which is which in a variety of ways, such as using ifconfig or Network Utility from /Applications/Utilities. There are also services. Each service will have a physical adapter, and a physical adapter can have multiple services. This is, for example, how you would go about assigning two IP addresses to a single physical adapter. You can further discombobulate things by bonding interfaces, whereas you are virtualizing the service to spread across multiple interfaces, but this is beyond the scope of this article.

I have no clue where it started, but we often call the network that you already use for Internet, directory services, DNS and all of that fun stuff the house network. The house network can be configured pretty much any old way you want it to and should be listed as the first of your services in your search order. For the purposes of this article we're going to simply call it HouseNetwork but you can feel free to call it whatever you like. In this case we're going to assume that HouseNetwork is sitting on en1. To get a list of the network services running on your machine you can use the following command:

networksetup -listallnetworkservices

Which, for example, might return with the following:

Ethernet
Ethernet1
FireWire

Now we want to configure one of these to say HouseNetwork. How do you know which one? Well, it's whichever one you have patched into your primary network. For the purposes of this example, we're going to patch Ethernet into the network and go ahead and rename it to HouseNetwork. This is done using the networksetup command again, with the -renamenetworkservice option as follows:

networksetup -renamenetworkservice Ethernet HouseNetwork

While it's not required to rename your network services I'd recommend it. As you can see, it's quick and easy and can save you a bunch of time in the future in terms of troubleshooting and remote support. Renaming is very specific, it looks for a pattern in the name and it replaces it with a new pattern. So Built-in Ethernet would become "Built-in Ethernet", etc. Now let's go ahead and rename the other services to MDNetwork (short for Metadata Network) using the following command:

networksetup -renamenetworkservice Ethernet1 MDNetwork

Next, we want to make sure that the HouseNetwork is listed above MDNetwork. This will help to keep DNS, directory services, HTTP management traffic and other unnecessary traffic off the MDNetwork so it can focus on what it's there for, metadata. To start, let's look at what order the services are listed in we're going to use networksetup yet again, this time with a -listnetworkserviceorder option as follows:

networksetup -listnetworkserviceorder

Which should provide a listing similar to the following (although potentially in a different order):

(1) MDNetwork
(Hardware Port: Ethernet, Device: en1)

(2) HouseNetwork
(Hardware Port: Ethernet, Device: en0)

(3) FireWire
(Hardware Port: FireWire, Device: fw0)

Above we see that MDNetwork is listed as the first item in the network service order. Because we actually want the HouseNetwork there we're going to go ahead and reorder our services real quick using the networksetup command with the -ordernetworkservices option. Using this option, you simply list each service, in order following the option as you can see below:

networksetup -ordernetworkservices HouseNetwork MDNetwork FireWire

You might notice that in the above command we included Firewire. This is because you have to include all of your network services for the command to execute successfully. Now we are actually going to go ahead and disable the FireWire network service (when we do the interface itself will still function) using the -setnetworkserviceenabled option of the networksetup command. Because the FireWire service is automatically named FireWire we will simply tell networksetup to setnetworkserviceenabled to off as follows:

networksetup -setnetworkserviceenabled FireWire off

Because most environments do no support IPv6 yet, we're going to go ahead and disable this for the HouseNetwork and MDNetwork using the -setv6off option as follows:

networksetup -setv6off HouseNetwork
networksetup -setv6off MDNetwork

Once IPv6 has been disabled we're going to move into configuring the IPv4 settings for our two network interfaces. For example, the HouseNetwork might be setup to use DHCP. In that case there's not much configuration that needs to occur. While DHCP should be the default setting used with the controller it would still be wise to go ahead and specify that again anyway (just in case) using the following command, where -setdhcp is the option that enables DHCP for the HouseNetwork service.

networksetup -setdhcp HouseNetwork

While the HouseNetwork could be DHCP, in this case we're going to set it as a static IP address of 10.0.1.11. The subnet mask will be 255.255.0.0 and the gateway will be 10.0.0.1. This is all sent to the service in one command, sing the -setmanual option with networksetup. When you are using this option you use the -setmanual option followed by the name of the service to configure, then the IP address that will be given to the service, then the subnet and finally the router (default gateway). So our command would be:

networksetup -setmanual HouseNetwork 10.0.1.11 255.255.0.0 10.0.0.1 

The metadata network is a bit more persnickety. As is typical, we will not need to (or want to for that matter) use a routable interface for the metadata network and will not be specifying a Default Gateway. DHCP is problematic on metadata networks and so we're also not going to use DHCP anywhere on that physical segment of the network. The IP scheme for our example metadata network is 192.168.1.x with a subnet mask of 255.255.255.0. So the command to set this up following the above example is:

networksetup -setmanual MDNetwork 192.168.1.11 255.255.255.0

Note: If you notice, in the above command there is no router defined. While it is syntactically possible to leave a router undefined, a subnet mask is required.

Now that we have the services configured, we need to assign name servers. In order to setup DNS we will use the -setdnsservers option with networksetup. In this case our DNS servers are 10.0.0.2 and 10.0.0.3. When using the -setdnsservers option you simply list the primary name server followed by the secondary name server and any tertiary name servers. DNS is used on the HouseNetwork service as we want to keep all traffic on the metadata network that is non-Xsan essential at a minimum. So to configure the DNS servers for the HouseNetwork service we would use the following command:

networksetup -setdnsservers HouseNetwork 10.0.0.2 10.0.0.3

At this point you're probably thinking to yourself that you could have done all of this in the Network System Preference pane in about 2 minutes. Well, you would be right. But think about how long it will take you to do on 50 or 60 Xsan clients (not to mention waiting for people to let you onto their computer and save/quit their apps). For this reason, you might consider using a shell script. So now we're going to take all of the commands that we were using through this example and put them into a shell script, replacing the actual IP addresses with positional parameters for the HouseNetwork and MDNetwork IP addresses, so that we can send the script along with the IP that it will receive to each workstation. The script would look something like this:

networksetup -renamenetworkservice Ethernet HouseNetwork
networksetup -renamenetworkservice Ethernet2 MDNetwork
networksetup -ordernetworkservices HouseNetwork MDNetwork FireWire
networksetup -setnetworkserviceenabled FireWire off
networksetup -setv6off HouseNetwork
networksetup -setv6off MDNetwork
networksetup -setmanual HouseNetwork $1 255.255.0.0 10.0.0.1
networksetup -setdnsservers HouseNetwork 10.0.0.2 10.0.0.3
networksetup -setmanual MDNetwork $2 255.255.255.0

Now the script can be copied to each workstation. For this example we're going to call the script setnetworkservices.sh. In order to send an IP address for the HouseNetwork of 10.0.1.12 and an IP for the MDNetwork of 192.168.1.12 you would simply send the following command (including the path of course):

setnetworkservices.sh 10.0.1.12 192.168.1.12

Then, to setup the host the next host using the same convention you would use:

setnetworkservices.sh 10.0.1.13 192.168.1.13

Now you have a script that can be run on your systems during your initial setup. The downtime for each new Xsan user will be greatly reduced, as you have front loaded that time into the script. New clients will be able to quickly be setup and when you go to reload systems you will have less work to do.

If you want to get a bit more complicated with the above script you could add some logic. For example, you might query for en0 and convert a service name to be used with en0 based on the Interface, to keep the script from failing due to someone having renamed the service in the past. Because a common issue during setup is to patch the wrong interfaces into the networks you could also use the ping command to test each network to verify it is live and if not (else) go ahead and swap the IP settings and names. You might also go ahead and turn every single setting into a variable to make it much more portable. However, this is all a bit more programming.

Finally, as you are updating this information, you are actually augmenting the /Library/Preferences/SystemConfiguration/com.apple.network.identification.plist file. While there are a variety of ways to edit this file directly I wouldn't really suggest it. Reason being that most adapters are referenced by MAC and have generated ServiceIDs (for example F8166C7E-CCFC-438C-98C6-CB05C7FA13E7). It is far easier to simply use the networksetup tool than it is to actually use a file drop of the plist or augment this file directly.