Relocating meta- and journal data to a new LUN

ravi's picture

You may have a situation where a customer needs to move an existing metadata LUN to a new Raid LUN. The procedure below explains how this can be done. Note that in this case, the new LUN needs to be equal to the same size as the old one or bigger. More discussion relating to LUN size follows later.

Please note that you should be very careful with techniques like these. Test everything thoroughly in a lab simulation before dealing with live systems.

Step I.

  1. Unmount the Xsan volume from all the clients
  2. Stop the volume
  3. Use cvfsck -nv to make sure the file system is clean. Modify it with cvfsck -w if necessary.

It is a very good idea at this point to save a copy of your meta- and journal data using snmetadump.

Step II.

  1. Attach the new LUN to the fibre network
  2. From a terminal, do
  3. 
    sudo cvlabel -c > cvlabel_beforemigration
    
  4. Edit the file cvlabel_beforemigration and keep only the entries corresponding to the current metadata LUN and the new LUN. Here is an example:
  5. 
    CvfsDisk_UNKNOWN /dev/rdisk1    
    
    1. host 0 lun 0 sectors 1818652 sector_size 512 inquiry [SanDisk Cruzer Mini 0.1 ] serial unknown
    mdata /dev/rdisk3 362602 VTOC
    1. host 0 lun 0 sectors 362602 sector_size 512 inquiry [AVB Flash 2.0 P1.0] serial unknown

In the above example, the new LUN corresponds to the entry CvfsDisk_UNKNOWN and is associated with raw device /dev/rdisk1 and the mdata entry corresponds to the current metadata LUN and is associated with raw device /dev/rdisk3.

Step III.

  1. From a terminal, do
    
    sudo dd if=/dev/rdisk3 of=/dev/rdisk1 bs=131072
    

    The block size 131072 above is used instead of the default 512 value to speed up the dd transfer process.

  2. Now there are two LUNs with the mdata label.
    Use cvlabel twice to unlabel the mdata label.
    
    sudo cvlabel -u mdata
    sudo cvlabel -u mdata
    

    Note: With Xsan2, the option -U can be used with cvlabel, with the raw device as an argument, to unlabel a LUN.

Step IV.

  1. Copy the file cvlabel_beforemigration to cvlabel_aftermigration.
  2. Edit the entries in cvlabel_aftermigration to give a different name to the current LUN and with the correct sector size and comments entries for the new LUN.
  3. In the above example, after the edit, the entries look like this
    
    mdata /dev/rdisk1 1818652 VTOC   
    
    1. host 0 lun 0 sectors 1818652 sector_size 512 inquiry [SanDisk Cruzer Mini 0.1 ] serial unknown
    mdata.old /dev/rdisk3 362602 VTOC
    1. host 0 lun 0 sectors 362602 sector_size 512 inquiry [AVB Flash 2.0 P1.0] serial unknown

    Note that, in the above example, even though the new LUN has 1818652 sectors, only 362602 sectors can be used. The Xsan volume config file has the following information and this has been saved in the metadata originally when the SAN was created.

    
    [DiskType mdataType]
    Sectors 362602
    SectorSize 512
    

Step V.

  1. From the terminal, do
    
    sudo cvlabel cvlabel_aftermigration
    

Step VI.

  1. Do a cvfsck to make sure everything is ok, start the volume and mount it.

Procedure with Xsan2

The above procedure works with both Xsan 1.4.x and Xsan2. However, Xsan2 has a command called sndiskmove which makes this whole procedure very simple:

  1. Follow steps I and II as above.
    Using the example above, the cvlabel_beforemigration has these entries:
    
    CvfsDisk_UNKNOWN /dev/rdisk1    
    
    1. host 0 lun 0 sectors 1818652 sector_size 512 inquiry [SanDisk Cruzer Mini 0.1 ] serial unknown
    mdata /dev/rdisk3 362602 VTOC
    1. host 0 lun 0 sectors 362602 sector_size 512 inquiry [AVB Flash 2.0 P1.0] serial unknown

    Notice the VTOC entry for the existing metadata LUN above.

Step III.

  1. Label the new LUN as follows:
    
    mdata_new  /dev/rdisk1 1818652 VTOC 
    
    1. host 0 lun 0 sectors 1818652 sector_size 512 inquiry [SanDisk Cruzer Mini 0.1 ] serial unknown
  2. Remove the old mdata entry, so that the resulting file cvlabel_beforemigration will have only the above line.
  3. - From a terminal, do
    
    sudo cvlabel cvlabel_beforemigration
    

    It is better to label the new LUN this way instead of using Xsan Admin since Xsan2 appears to write the EFI label for all LUNs. Therefore if your old LUN has the VTOC label, sndiskmove command will quit with a mismatched label error.

Step IV.

  1. From a terminal, do
    
    sudo sndiskmove mdata mdata_new
    

    This will move the contents of the current LUN to the new LUN, label it with the name mdata, and relabel the current LUN as mdata.old.

  2. Do a cvfsck to make sure everything is ok, start the volume, and mount it.

Note: sndiskmove will quit with an error if the new LUN is smaller in size than the older LUN.

While it is possible to move the meta- and journal data from a bigger LUN to a smaller LUN using dd with the count argument that covers only sectors up to the maximum sectors in the smaller LUN, the problem will be the entry in the Xsan volume config file for metadata Type where the maximum sectors value from the bigger LUN has been recorded during the creation of the SAN. This information is also there in the metadata LUN. The file system was initialized with this information and it is not possible to "shrink" the sectors. If you move the bigger LUN to a smaller LUN this way, the volume will start, but you will not be able to mount the volume since fsm will panic and crash due to the “shrunken” size.

Comments

4
pradeep.pg's picture

wowwwwww!!!!!!!!!!
hats off........u r a genius man....
Cool trick bt I am scared of doing this @ my customer place, if something goes
wrong I'd get crucified.

ravi's picture

Hi Pradeep

I understand, that is why it is very important for "non-standard" techniques like
these be thoroughly tested in a lab simulation. I was recently in situations at
customer sites where peculiar circumstances demanded that I had to have the
meta- and journal data directed to the local system drive in the metadata
controller via the undocumented PseudoDevice feature in Xsan (at least till
1.4.2, I haven't checked Xsan 2) before we could migrate to a proper RAID 1 LUN
for metadata.

bs3375's picture

instead of using dd or sndiskmove, could you restore the metadata from a
backup to the new LUN using snmetadump?

ravi's picture

I don't use and haven't tested snmetadump or snmetatar, so can't say.