Now that Xsan is headed toward its third birthday, and from most accounts (save the occasional bout of admin insanity) is more stable than ever, I turn my attention to the 800 lb. gorilla that eventually faces all folks involved with this file system: volume expansion.

What should clearly be stated up front is that the promise of “storage pool expansion,” greatly touted by Apple in its marketing materials with the promise of increasing both capacity and bandwidth, rarely works in the real world.

Storage Pool Expansion is the process of adding like-sized and like-RAID-level LUNs to the end of an already existing storage pool, in the name of increasing both capacity and bandwidth. This can be done directly within the Xsan Admin GUI. But as many of us have discovered through many attempts, if said Storage Pool has any substantial amount of data on it before the expansion, the attempt to expand it ultimately fails.

The result we receive in the /var/run/xsan_cvupdatefs_volumename.log of the MDC is always the same: the resulting storage pool would have been too fragmented to provide any semblance of performance. The process aborts, and our volume returns to the state that it was in before the attempt.

What we need, therefore, is a way to expand the volume and guarantee that we can expand bandwidth in the process, without having to rely on the expansion of existing storage pools. Welcome to the concept of Affinity Groups.

Xsan, being a cousin of the StorNext filesystem, can take advantage of affinity groups even though it’s not a documented feature. The fact is, it’s well documented on the StorNext side. An Affinity Group is simply an affinity that stands for more than one storage pool.

Obviously, the implementation of Affinity Groups takes place best when planning a new volume, or when expanding an existing volume with a substantial amount of new storage.

For those who use Xsan in the post-production and broadcast industries, Affinity Groups will also benefit from being made of Storage Pools that have the same size and, if possible, are created out of a binary-divisible amount of LUNs (2, 4, 8, 16, etc.).

Lastly, creating Affinity Groups is not for the faint of heart; since the feature is undocumented, it must be accomplished using command line tools, and AppleCare might not support a volume created out of this idea. Rest assured however, that many integrators and admins are using the process, and it’s clear that Apple Professional Services have been using the technique as well.

Here are steps for a sample new volume. Similar steps could be followed to expand an existing volume.

1. Create a volume using the Xsan Admin GUI, making sure that “Balance” is used for the allocation strategy.

[img:20070531070144972 align:center]

Using Balance ensures that data for the Affinity Group is distributed evenly across the Storage Pools within it. As you’ll recall, the Balance strategy allocates data across multiple storage pools by sending an individual file to the pool that has the most free space, and when all Storage Pools are about as full as each other, will allocate additional files in a Round Robin-ish fashion. I say “ish” because a particularly large file written to one pool would then dictate that a few smaller files be written to the other pools to balance out. Using balance also ensures that the total free space of the volume is useful to the end user: they won’t ever have to wonder why they can’t write to a folder even though there is plenty of free space on the volume, as happens sometimes with regular affinities.

Later, when additional Storage Pools are created and made part of the Affinity Group, data will be allocated to the new Storage Pool(s) until they are as full as the original ones.

2. Create as many Storage Pools as you want to be in the group, and name them all the same, except for a trailing numeral or letter, such as DATA_1, DATA_2 or HDVIDEO1, HDVIDEO2, etc.

You actually can name them anything you want, but for your sanity as an admin, it helps to keep part of their name similar.

The number of Storage Pools you create for the group is based on the number of LUNs you’ll have overall in the volume (or expansion of the volume). It will also be based on the number of LUNs per Storage Pool you’ll want to use to ensure that each Storage Pool will be able to receive one or two (or more!) ingest station’s worth of write bandwidth.

[img:2007053107014990 align:center]

In the example above, after making a Storage Pool for metadata and journaling, there are eight 7-drive RAID5 LUNs available to make the volume (4 Xserve RAIDs). Two Storage Pools named DATA_1 and DATA_2 are created, and will each house 4 LUNs. This is satisfactory for this facility because the four LUNs within each pool would be able to each take on the 4 or so workstations that might write or read Standard Definition uncompressed video files to it. The facility has 8 workstations, and the volume has two such pools. Ideally, at any given moment, four of the workstations would be reading and writing to four LUNs within each pool. Notice also that we’ll create these pools out of a binary-divisble number of LUNs (4), and this will help as well for expansion later on.

3. Add your LUNs to the Storage Pools within the group.

Once again, we will want each Storage Pool to contain an equal amount of LUNs and, if possible, a binary-divisible number.

4. Save the volume, but don’t start it quite yet.

We’ll need to take a trip into the Terminal to finish the job.

5. In the Terminal, shell into the primary MDC and use your favorite text editor to open up the volumename.cfg file that resides in /Library/Filesystems/Xsan/config/.

Near the end of the file, as it mentions each Stripe Group (StorNext speak for Storage Pool) find the line that states “Affinity DATA_1”. All you’ll want to do is to get rid of the trailing numeral or other identifying character, so that all Affinity names are the same for the Storage Pools you wish to be in this group. In this case, we’ll truncate the Affinity name to DATA.

6. Save the file. IMPORTANT: do these steps again for the .cfg file on any additional MDC(s).

7. Back in the Xsan Admin GUI, start the volume.

That’s it! You’ve just created an Affinity Group. Now it’s time to assign that affinity to a folder or two. The GUI will not help you here, since it’s hard coded to think that a Storage Pool name is the Affinity name and nothing else.

Two command lines can be used to assign affinities. It’s best to run them as root:

cvmkdir –k affinityname /Volumes/volumename/path/to/folder

or

cvaffinity –s affinityname /Volumes/volumename/path/to/folder

The advantage of the cvmkdir command is that has the ability to create a folder and assign an affinity in one step.

When data is written to a folder assigned this affinity, the MDC will now use a balance allocation strategy amongst all of the pools within this group. Watch with glee as files are allocated to each of the Storage Pools elegantly, making sure each one has about the same amount of data in it.

Down the road, it will be easy to expand the volume and its bandwidth capability by adding another Storage Pool with an equal amount of LUNs within the Xsan Admin GUI, saving the volume, and then modifying the volumename.cfg file on each MDC using the steps above.

Many facilities are choosing to create their Affinity Groups out of Storage Pools that contain 4 or 8 LUNs. This way, as it comes time to expand volumes, facilities budget for the purchase of 2 or 4 Xserve RAIDs, respectively.

Keep us posted if you choose to use this method for you next volume creation, and how it goes.