Turbonomic Appliance SPLIT | CLONING vs MIGRATION

Document created by carlos.castellanos Expert on Dec 13, 2018Last modified by carlos.castellanos Expert on Feb 18, 2019
Version 2Show Document
  • View in full screen mode

VALIDATION

Yes Turbonomic internal documentation reference

Yes Field tested

Yes Peer revision

 

APPLIES TO TURBONOMIC VERSION(S):

v5.9.x

v6.0.x

v6.1.x

v6.2.x

 

BACKGROUND

While we have a standard operating procedure (SOP) to migrate a Turbonomic appliance we want to focus this article on a particular use case: that of a SPLIT. This is instrumental when either for procedural reasons or high load and performance detriment we need to distribute the number of service entities (VMS or Workloads in a broader sense).

 

REQUIREMENTS & SCOPE:

  • Presented and discussed with the customer.
  • A clear identified need to SPLIT a Turbonomic appliance.
  • Document written and field tested focusing on vCenter (VC) Targets.

 

DETAILS

Authoritative Source of Information: (gDoc) <Customer>_SOP_SPLIT.vsin22p5981_Ticket #103676 (internal use only)

Split by Cloning (Recommended)

Clone Migration Process

  1. Take snapshot of the current Turbonomic instance VM.
  2. If the intent of the migration is to execute a VC split, then ensure that a list of target VCs has been prepared for each Turbonomic (current and clone) instance.
  3. Disable automation on Turbonomic instance that will be cloned.
  4. [RECOMMENDATION] Before executing Cloning, delete any old reports to reduce cloning time.
    1. Delete old reports:
    2. From ssh session in:
      • cd /srv/reports/pdf_files
      • rm -rf &lt;dates you no longer need&gt;
        • Example: this will delete reports from 2017
        • rm -rf 2017*
  5. Clone Turbonomic instance
  6. Once Clone is deployed, set up the network configuration in the hypervisor
    1. Verify MAC address is different than between original and clone
    2. Static IP
    3. Configure TB appliance FQDN (verify internal DNS is responsive, nslookup)
    4. Configure DNS, Gateway, etc
    5. Verify NTP (Time Sync
  7. Establish a console or putty session to the Turbonomic Clone instance
  8. Login as root user, verify network setting via ifconfig

VC Split Process

  1. Following the prepared VC target lists for each Turbonomic instance, delete VC targets that are not on list for current and new Turbonomic instances.
  2. Stop tomcat:
    • service tomcat stop
  3. Modify disc.config.topology and remove targets

[OPTIONALLY: this process can be done in the UI and modify disc.config.topology if needed. Since automation is halted and the appliance has a clean new MAC and new IP, it could potentially be accessed via web interface to remove targets there instead. If going this way tomcat must be started]  

  1. cd /srv/tomcat/data/config
  2. cp disc.config.topology disc.config.topology.backup
  3. vi disc.config.topology
  4. remove targets like below
    1. &lt;targets xsi:type=&quot;vim:VimTarget&quot; uuid=&quot;_DJZkkIojEeao_dcdKBI_XA&quot;

      name=&quot;VMTTarget_<FQDN>&quot;

                     displayName=&quot;<FQDN>&quot;

      nameOrAddress=&quot;<FQDN>&quot;

      timeout=&quot;30000&quot; template=&quot;_xM1eNeNsEd6prOnjD-gNVw&quot; version=&quot;5.5.0&quot;

      lastValidationStatus=&quot;Validated&quot; lastValidationDescription=&quot;Validated&quot;

      lastValidationTime=&quot;2018-08-06T12:09:45.581-0400&quot;

      vcInstanceUuid=&quot;5F553D63-BEA9-4734-B32F-C31E8A35D8C0&quot; uniqueId=&quot;4&quot;&gt;

      &lt;credentials xsi:type=&quot;Mediation:UsernamePassword&quot; uuid=&quot;_DJaysIojEeao_dcdKBI_XA&quot;

      derived=&quot;false&quot; username=&quot;NAEAST\N571010&quot;

      password=&quot;AAAAAQAAACCr2M4NlECMqBP0mojRgTbg0o/+LMeGnA/DQBBzFFpkvwAAABC8VLH

      FC1oXx28J46prrhYSAAAAELbxO3cELxiSXKtVvKr4AD8=&quot;

      encrypted=&quot;true&quot;/&gt;

      &lt;/targets&gt;

  5. Start tomcat
    • service tomcat start
  6. Verify that VC targets are correct.
  7. Verify that Host, VMs, Tags have populated in inventory and polling is occurring.
  8. Turn on automation.

Turbonomic Migration Process

v011720192200

Adaptation of our Turbonomic 6.0 Installation Guide | p22  

I | Preparation:

  1. Take snapshot of the current Turbonomic instance VM.
  2. If the intent of the migration is to execute a VC split, then ensure that a list of target VCs has been prepared for each Turbonomic (current and new) instance.
  3. Verify version your current Turbonomic instance. (Current and new Turbonomic instances must be the same version).
  4. Proceed to deploy a new turbonomic instance.
    1. Ensure the current and new Turbonomic instances are the same version.
      • <cat /srv/tomcat/data/version/version_name.txt>
    2. Once deployed, set up the network configuration in the hypervisor
      • Static IP
      • Configure TB appliance FQDN (verify internal DNS is responsive, nslookup)
      • Configure DNS, Gateway, etc
      • Verify NTP (Time Sync
  5. Establish a console or putty session to the Turbonomic instance
  6. Login using root (default password is 'vmturbo'), then run 'ipsetup' from the command line and follow the inline instructions

II | Migration Data (from OLD to NEW):

Note: If you are here, at this update path, this means you have verified the current and new Turbonomic instances are the same version. But did not complete the steps to migrate the data and configuration onto a new Turbonomic server running the latest operating system and other components.

  1. Ensure a snapshot of current Turbonomic instance VM has been executed This will provide a reliable restore point if an issue occurs during the upgrade.
  2. After taking a snapshot (or cloning the current Turbonomic) power it back on.
  3. Before executing backup
    1. Delete any old reports to reduce backup time.
      • From ssh session in:
      • vmturbo:~ #  cd /srv/reports/pdf_files
      • vmturbo:~ #  rm -rf <dates you no longer need>
      • example (this will delete reports from 2017)  --►   rm -rf 2017*
    2. Cleanup /tmp folder
      • Delete old DIAG files
      • Delete any other temporary or manually created file.
  4. Disable ALL Turbonomic Automation Actions before executing backup.
    1. Backup to policies.config.topology files
      • From ssh session in:
      • vmturbo:~ #  cd /srv/tomcat/data/config
      • vmturbo:~ #  cp policies.config.topology policies.config.topology.BACKUP
    2. UI | SETTINGS | POLICIES --► Search for “Service Entity defaults”
      • ENABLE “Disable All Actions” under the “- ACTION AUTOMATION” section
      • IMPACT ANALYSIS: the downtime of automation depends on how long the backup process takes to complete. This usually depends on size of the environment, the DB and the performance of the underlying infrastructure (re. storage).
        • For instance w/TBN on ScaleIO typical datastore we measured an average of 434MB/s that translates to an effective (/15) 25MB/s approximately(greg.aligiannis@turbonomic.com)
        • With a large 115GB DB, this could take some 76 min of downtime to complete. (Empirical data from a run showed ~2hrs)
        • The best way to know if running a dry test over the platform before w/o disabling automation.
  5. Run the following command to create the back up
    1. By default the backup file will be created under /tmp folder. As a guideline you should have at least [TBN DB Size/50] of free space. (Based on estimated compression ratio of 50x. Empirical data from a run showed a ~115GB DB down to 1.47GB or 76x compression)
    2. vmturbo:~ #  nohup /srv/tomcat/script/appliance/vmtbackup.sh -o full &
    3. Notes:  
      1. This may take hours to complete.
      2. The resulting backup file is /tmp/vmtbackup.zip
      3. The "&" symbol at the end of the command instructs bash to run nohup mycommand in the background.
      4. Prefixing a command with nohup prevents the command from being aborted automatically when you log out or exit the shell.
    4. You can track progress running a “watch” command
      • watch "mysql -uroot -pvmturbo vmtdb -e 'show processlist'"
  6. Retrieve the vmtbackup.zip either downloading or by direct copy to the DESTination appliance.
    1. If Copy the vmtbackup.zip file to your new Turbonomic use the command below
      1. Check the DEST TBN instance /tmp folder space, make sure it has enough space for the backup file..
      2. vmturbo:~ # scp /tmp/vmtbackup.zip root@[IP_addr of NEW instance]:/tmp
      3. This command will ask you for the root password (default is 'vmturbo') and send the backup file to the new instance you've deployed
  7. IMPORTANT:  Typically before restoring your backup to the a new Turbonomic instance (steps below), you would have to shut down the old Turbonomic instance.  This is to ensure that there is only one Turbonomic appliance operating in your environment with the same configuration. Nonetheless, given that we enforced DISABLE AUTOMATION in step II.4.b above you do not need  to do this.
  8. If the MIGRATION procedure is to be driven by low impact on downtime, you should resume automation on the SOURCE TBN at this point (revert step II.4.b above).
    1. We have a safety net against conflict when the restore is done on any other TBN given that the backup was taken and will carry Automation Disabled when restored in next steps.
    2. IMPACT ANALYSIS: take into account that from the moment you resume automation on the SOURCE to the moment you complete restore (see below) into the new TBN appliances, there will be a DELTA of data that will be “left-behind”. In other words, assuming the restore takes 10hrs, then there will be this time of data left on SOURCE that the new TBN will NOT BE AWARE of.
      1. The risk when you finally turn on the new TBN is similar to the one of a 10 hrs shutdown or disconnect. The new TBN Market (ESE) will resume where “it left” and recalculate matters based on a new FULL discover. Similar to turning the TBN back after these 10hrs blackout. No significant risk, the system will know where entities are, any inventory change and new demand data will drive decisions and recommendations for action, business as usual.

III | Restore the Backup file (on NEW instance):

  1. Ensure a snapshot of DESTination Turbonomic instance VM has been executed.This will provide a reliable restore point if an issue occurs during the RESTORE.
  2. Login to the new instance using root / vmturbo
  3. Verify the health of your partitions
    1. /var/log/tomcat | at least 50% free
    2. /root | at least 30% free
    3. /var/lib/mysql | at least 1.5x the size of the DB
  4. Notes:
    1. The restore process assumes the backup file is in the /tmp folder
    2. This may take hours to complete (See IMPACT ANALYSIS below).
  5. Run these commands to restore your backup file
    1. vmturbo:~ # cd /
    2. vmturbo:~ #  nohup /srv/tomcat/script/appliance/vmtrestore.sh -o full &
  6. CHECK PROGRESS (**)
    1. # while true

      > do

      > echo "*******Checking sql process status on `hostname`"

      > ps -ealf | grep -i mysql

      > sleep 60

      > done

  7. IMPACT ANALYSIS: Similar to the previous backup (dump) procedure the restore will depend on size of the restore and performance of underlying infrastructure. Nevertheless, restore is a more intensive task
    1. Empirical data suggests that a 30GB original DB can take up to 3hrs to restore.
    2. For a large 115GB DB, this could take NOT LESS than 9 hrs. (greg.aligiannis@turbonomic.com)
    3. The appliance is not in PROD yet, but if more accurate data is needed before actually committing to this process PROD, you can use the actual results of this restore to document time of completion.
      1. This is important if knowing with accuracy the aforementioned DELTA is part of the requirements.

IV | VC Split Process:

  1. Following VC target list previously prepared reflecting which VC target is kept on each Turbonomic instance, delete|halt the other VC targets that are not meant to stay. Do this for both, NEW and SOURCE (*OBS: you might want to consider the recommendations below) Turbonomic instances.
    1. As an extra buffer, rather than deleting the VC Targets on SOURCE TBN right away you could “halt” discovery by changing login name  credentials (*NOT the password as it might lock your account).
    2. You should still delete the corresponding targets marked for deletion on each NEW Turbonomic appliance.
    3. We recommend (re. v6.0.x and below) to remove/process targets one at a time per Turbonomic appliance.
    4. IMPACT ANALYSIS: the removal of targets can also be a lengthy process depending on the # of entities handled by each VC target being deleted.
    5. [ALTERNATE RECOMMENDATION | Multiple & Large Targets] Manually Modify disc.config.topology and remove targets
      1.  cd /srv/tomcat/data/config
      2. cp disc.config.topology disc.config.topology.backup
      3. disc.config.topology
      4. remove targets like below
        1. &lt;targets xsi:type=&quot;vim:VimTarget&quot; uuid=&quot;_DJZkkIojEeao_dcdKBI_XA&quot;

          name=&quot;VMTTarget_<FQDN>&quot;

                         displayName=&quot;<FQDN>&quot;

          nameOrAddress=&quot;<FQDN>&quot;

          timeout=&quot;30000&quot; template=&quot;_xM1eNeNsEd6prOnjD-gNVw&quot; version=&quot;5.5.0&quot;

          lastValidationStatus=&quot;Validated&quot; lastValidationDescription=&quot;Validated&quot;

          lastValidationTime=&quot;2018-08-06T12:09:45.581-0400&quot;

          vcInstanceUuid=&quot;5F553D63-BEA9-4734-B32F-C31E8A35D8C0&quot; uniqueId=&quot;4&quot;&gt;

          &lt;credentials xsi:type=&quot;Mediation:UsernamePassword&quot; uuid=&quot;_DJaysIojEeao_dcdKBI_XA&quot;

          derived=&quot;false&quot; username=&quot;NAEAST\N571010&quot;

          password=&quot;AAAAAQAAACCr2M4NlECMqBP0mojRgTbg0o/+LMeGnA/DQBBzFFpkvwAAABC8VLH

          FC1oXx28J46prrhYSAAAAELbxO3cELxiSXKtVvKr4AD8=&quot;

          encrypted=&quot;true&quot;/&gt;

          &lt;/targets&gt;

      5. At this point all NEW TBN instances have only one VC target and the SOURCE TBN have only one with correct credentials and all others either deleted or halted via credential (login name) change
      6. Execute service tomcat restart on each NEW TBN instance following VC target deletion.
      7. After restart verify that VC targets are correct on each NEW TBN instance.
      8. Verify that Host, VMs, Tags have populated in inventory and polling is occurring.
      9. Enable ALL Turbonomic Automation Actions.
        1. UI | SETTINGS | POLICIES --► Search for “Service Entity defaults”
        2. NOTE: If you turn it on on SOURCE earlier in the process,  right after backup (dump) focusing on “lesser downtime” you do NOT need to do this on SOURCE. You do need to do it on the new DESTination TBN appliances where you restored the backup.
      10. Ensure Recommendations/Actions are occurring as expected.
      11. At this point if the SOURCE TBN has halted targets (rather than having been deleted) please proceed to delete them. We recommend one at a time or the Manual edit of disc.config.topology file when there are multiple and large targets.
      12. Once the SOURCE TBN targets have been deleted, proceed to Execute service tomcat restart.
      13. After the restart of the SOURCE TBN verify its health.

       

      Other resources

      Originating ZenDesk Ticket: #103676  

       

      Credits

      Prepared by carlos.castellanos & robert.drouin

      (*) Thanks to robert.fagnoni & greg.aligiannis@turbonomic.com

      (**) Thank you gopalakrishna.satuluri@jpmorgan.com

      Attachments

        Outcomes