> Source: [sk174202](https://support.checkpoint.com/results/sk/sk174202) # sk174202 - How to replace a Quantum Maestro Orchestrator | Property | Value | |----------|-------| | Solution ID | sk174202 | | Date Created | 2021-06-24 | | Last Modified | 2026-06-11 | | Technical Level | General | | Products | Scalable Platforms | | Versions | R82, R81.20, R81.10 (EOS) | | OS | Gaia | | Platform | Maestro Orchestrator | ## Solution **Table of Contents:** * Important Notes * Procedure for Orchestrators R82 - Failed Orchestrator is still accessible * Procedure for Orchestrators R82 - Failed Orchestrator is not accessible * Procedure for Orchestrators R81.20, R81.20 Jumbo Hotfix (all Takes), or R81.10 Jumbo Hotfix Take 61 (and higher) * Procedure for Orchestrators R81.10, or R81.10 Jumbo Hotfix Take 55 (and lower) * Procedure for Orchestrators R80.20SP - Failed Orchestrator is still accessible * Procedure for Orchestrators R80.20SP Jumbo Hotfix Take 309 (and lower) - Failed Orchestrator is not accessible * Revision History Follow the steps below to replace (RMA) a failed Quantum Maestro Orchestrator. ### Important Notes {#TOC01} * We recommend to schedule a maintenance window. * In a Dual Site deployment, do the RMA procedure on the Site that is the Standby Site at the time of replacement. If the Orchestrator fails on Site # 1 and this Site was active, you must failover to Site # 2 and make it Active, with Site # 1 as the Standby. Then continue with the RMA procedure. The procedure below uses these Orchestrator IDs: * ID of the Orchestrator that continues to work - 1_1 * ID of the Orchestrator that failed - 1_2 ### Procedure for Orchestrators that run the R82 version - when the failed Orchestrator is still accessible {#TOC02} Show / Hide this section |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Important Note - It is not supported to import a backup file from another working Orchestrator. This procedure requires a backup file from the failed Orchestrator.** | 1. On the failed Orchestrator, stop the Orchestrator service: 1. Connect to the command line on the failed Orchestrator. 2. Log in to the Expert mode. 3. Stop the service: |--------------| | `orchd stop` | 2. If previously you configured the authentication between Orchestrators, then on each working Orchestrator you must revoke the authentication with the failed Orchestrator. See the [R82 Scalable Platforms Administration Guide](https://sc1.checkpoint.com/documents/R82/WebAdminGuides/EN/CP_R82_ScalablePlatforms_AdminGuide/Default.htm#cshid=ID002) \> Chapter "**Working with Quantum Maestro** " \> Section "**Authentication between Maestro Orchestrators**". You can revoke the authentication in Gaia Portal (recommended) or in CLI (Gaia Clish, or Expert mode). 3. On the failed Orchestrator, create the Gaia backup (it contains the Maestro configuration): **Critical Note** - On the replacement Orchestrator, you must import this Gaia backup file that contains the most recent Security Group configuration. You can create this Gaia backup in Gaia Portal (recommended) or in Gaia Clish. * To create this Gaia backup in Gaia Portal: 1. With a web browser, connect to the failed Orchestrator: |-----------------------------------------| | `https://` | Log in with the username "`admin`" and the corresponding password. 2. Go to **Maintenance** \> **System Backup**. 3. Click **Backup** \> select **This appliance**. 4. Click **Export** to download this Gaia backup file to your computer. * To create this Gaia backup in Gaia Clish: 1. Connect to the command line on the failed Orchestrator. 2. Log in to Gaia Clish. 3. Create a local Gaia backup: |--------------------| | `add backup local` | 4. Copy the Gaia backup file from the failed Orchestrator: 1. Connect with WinSCP to the failed Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to this path: |------------------------------| | `/var/log/CPbackup/backups/` | 3. Download the Gaia backup file to your computer. 4. On the failed Orchestrator, write down the port numbers, to which the **Uplink** cables, the **Downlink** cables, and the **Sync** cables are connected. 5. On the failed Orchestrator, disconnect all network cables and the power cables. 6. Remove the failed Orchestrator from the rack. 7. Install the replacement Orchestrator **of the same model** in the rack. |-----------------------------------------------------------------------------------------------------| | **Important Note** - Do **not** connect the power or network cables yet. You do it gradually later. | 8. On the replacement Orchestrator, connect only these cables: See the [Quantum Maestro Getting Started Guide](https://sc1.checkpoint.com/documents/Appliances/GSG_Maestro/EN/Default.htm). 1. Connect a network cable from your computer to the Orchestrator's "**MGMT**" port. 2. Connect the console cable from your computer to the Orchestrator's **Console** port. In your console client application, configure these settings: * Baud Rate - 9600 * Data bits - 8 * Stop bits - 1 * Parity - None * Flow Control - None 3. Connect the power cables to the Orchestrator's Power Supply Units. 9. On the replacement Orchestrator, in the **console** session, log in to Gaia Clish. |----------------------------------------------------------------------------------| | **Important Note** - Do **not** perform the First Time Configuration Wizard yet. | 10. On the replacement Orchestrator, configure the Orchestrator's "**MGMT**" port: 1. Configure the IPv4 address and Mask Length: |-------------------------------------------------------------------------------------| | `set interface Mgmt1 ipv4-address mask-length ` | 2. Enable the "MGMT" port: |--------------------------------| | `set interface Mgmt1 state on` | 3. Configure the default gateway: |-----------------------------------------------------------------------------------------| | `set static-route default nexthop gateway address on` | 4. Save the changes: |---------------| | `save config` | For information about these Gaia Clish commands, see the [R82 Gaia Administration Guide](https://sc1.checkpoint.com/documents/R82/WebAdminGuides/EN/CP_R82_Gaia_AdminGuide/Default.htm). 11. On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the failed Orchestrator: * The **Downlink** cables * The **Sync** cables |-------------------------------------------------------------------------------| | **Important Note** - Make sure you connected the cables to the correct ports. | 12. On the replacement Orchestrator, run the First Time Configuration Wizard: |------------------------------------------------------------------------------------------------------------| | **Important Note** - Do **not** import the backup file from the failed Orchestrator. You must do it later. | 1. With a web browser, connect to the replacement Orchestrator: |-----------------------------------------| | `https://` | Log in with the username "`admin`" and the default password "`admin`". 2. On the second page **Deployment Options** , select **Join an existing Maestro environment**. 3. Follow the rest of the wizard pages. 4. Wait for the Orchestrator to reboot. 13. On the replacement Orchestrator, install the same Take of the **R82 Jumbo Hotfix Accumulator** as installed on **working** Orchestrators (in our example, "1_1"). 1. Copy the Jumbo Hotfix Accumulator package to the replacement Orchestrator: 1. Connect with WinSCP to the IPv4 address of the "MGMT" port on the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to some path. For example: |-------------| | `/var/log/` | 3. Upload the Jumbo Hotfix Accumulator package from your computer. 2. Connect with an SSH client to the IPv4 address of the "MGMT" port. 3. Log in to Gaia Clish. 4. Obtain the lock over the Gaia configuration database: |--------------------------| | `lock database override` | 5. Import the CPUSE package from the hard disk: Note: When the import completes, this package is deleted from the original location. |----------------------------------------------------------| | `installer import local /` | 6. Show the imported packages: |------------------------------------| | `show installer packages imported` | 7. Verify the package - see whether this package can be installed without conflicts: |----------------------------------------------------------------------------------------------| | `installer verify`*\[press Space key\]\[press Tab key\]* `installer verify ` | 8. Install the package: |--------------------------------------| | `installer install ` | 9. Wait for the Orchestrator to reboot. 14. On the replacement Orchestrator, import the Gaia backup file from the failed Orchestrator. You can import this Gaia backup in Gaia Portal (recommended) or in Gaia Clish. * To import this Gaia backup in Gaia Portal: 1. With a web browser, connect to the replacement Orchestrator: |-----------------------------------------| | `https://` | Log in with the username "`admin`" and the new password you configured in the First Time Configuration Wizard. 2. Go to **Maintenance** \> **System Backup**. 3. Click **Import** \> select the Gaia backup file on your computer. 4. In Gaia Portal, click the imported Gaia backup file. 5. Click **Restore**. 6. Wait for the Orchestrator to reboot. * To import this Gaia backup in Gaia Clish: 1. Copy the Gaia backup file from your computer to the replacement Orchestrator: 1. Connect with WinSCP to the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to this path: |------------------------------| | `/var/log/CPbackup/backups/` | 3. Upload the Gaia backup file from your computer. 2. Connect to the command line on the replacement Orchestrator. 3. Log in to Gaia Clish. 4. Restore the local Gaia backup: |---------------------------------------------------------------------| | `set backup restore ` | 5. Wait for the Orchestrator to reboot. 15. If previously you configured the authentication between Orchestrators, you must configure it again (between all working Orchestrators) to trust the replacement Orchestrator. See the [R82 Scalable Platforms Administration Guide](https://sc1.checkpoint.com/documents/R82/WebAdminGuides/EN/CP_R82_ScalablePlatforms_AdminGuide/Default.htm#cshid=ID002) \> Chapter "**Working with Quantum Maestro** " \> Section "**Authentication between Maestro Orchestrators**". You can configure the authentication in Gaia Portal (recommended) or in CLI (Gaia Clish, or Expert mode). 16. On all Orchestrators, make sure the traffic can pass between Orchestrators: * In a Single Site deployment: On a working Orchestrator (in our example, "1_1"), send several pings to the replacement Orchestrator (in our example, "1_2"): |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: On the replacement Orchestrator (in our example, "1_2"), send several pings to **each** working Orchestrator: |-----------------| | `ping -c 5 1_1` | | `ping -c 5 2_1` | | `ping -c 5 2_2` | 17. Make sure the Security Group Members can pass traffic to each other: 1. Connect to the command line on the Security Group. 2. Log in to the Expert mode. 3. Identify the Security Group Member that runs as SMO: |---------------------| | `asg stat -i tasks` | 4. If the SMO task runs on another Security Group Member than the Security Group Member to which you are connected, then go to the context of that SMO Security Group Member: |-----------------------------------------------------| | `member _` | Example: `member 1_2` 5. Examine the cluster state of the Security Group Members. |------------------| | `cphaprob state` | The output must show the state of **each** Security Group Member as "`ACTIVE`". 6. Send several pings between Security Group Members: 1. Connect to one of the Security Group Members (in our example, we connect to the first one - "1_1"): |--------------| | `member 1_1` | 2. On this Security Group Member, send several pings to any other Security Group Member (in our example, we send several pings to the second one - "1_2" / "2_2"): * In a Single Site deployment: |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: |-----------------| | `ping -c 5 1_2` | | `ping -c 5 2_2` | 18. On the replacement Orchestrator, connect the **Uplink** cables to the ports with the same numbers as they were connected on the failed Orchestrator. 19. On each Security Group, make sure all the required links have the status "`UP`": 1. Connect to the command line on the Security Group. 2. Log in. 3. If your default shell is the Expert mode, then go to Gaia gClish: |----------| | `gclish` | 4. Examine the state of links: |--------------------------------| | `show cluster info interfaces` | ### Procedure for Orchestrators that run the R82 version - when the failed Orchestrator is not accessible {#TOC03} Show / Hide this section 1. On a working Orchestrator, export the configuration files for the failed Orchestrator: 1. Connect to the command line on the working Orchestrator - the peer Orchestrator on the same Site. 2. Log in to Gaia Clish. 3. Export the configuration files for the failed Orchestrator: |--------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `set maestro export remote orchestrator id configuration archive-name path ` | Example: |--------------------------------------------------------------------------------------------------------------------------------| | `set maestro export remote orchestrator id 1_2 configuration archive-name Export_Of_Config_for_Failed_Orch_1_2 path /var/log/` | The output file is: `/var/log/Export_Of_Config_for_Failed_Orch_1_2.tgz` 2. Copy the archive with the exported configuration files from the working Orchestrator: 1. Connect with WinSCP to this working Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to the path you used during the export: |---------------------------------------| | `` | In our example, go to: `/var/log/` 3. Download the archive file to your computer. In our example, download this file: `Export_Of_Config_for_Failed_Orch_1_2.tgz` 3. On the failed Orchestrator, stop the Orchestrator service: 1. Connect to the command line on the failed Orchestrator. 2. Log in to the Expert mode. 3. Stop the service: |--------------| | `orchd stop` | 4. If previously you configured the authentication between Orchestrators, then on each working Orchestrator you must revoke the authentication with the failed Orchestrator. See the [R82 Scalable Platforms Administration Guide](https://sc1.checkpoint.com/documents/R82/WebAdminGuides/EN/CP_R82_ScalablePlatforms_AdminGuide/Default.htm#cshid=ID002) \> Chapter "**Working with Quantum Maestro** " \> Section "**Authentication between Maestro Orchestrators**". You can revoke the authentication in Gaia Portal (recommended) or in CLI (Gaia Clish, or Expert mode). 5. On the failed Orchestrator, write down the port numbers, to which the **Uplink** cables, the **Downlink** cables, and the **Sync** cables are connected. 6. On the failed Orchestrator, disconnect all network cables and the power cables. 7. Remove the failed Orchestrator from the rack. 8. Install the replacement Orchestrator **of the same model** in the rack. |-----------------------------------------------------------------------------------------------------| | **Important Note** - Do **not** connect the power or network cables yet. You do it gradually later. | 9. On the replacement Orchestrator, connect only these cables: See the [Quantum Maestro Getting Started Guide](https://sc1.checkpoint.com/documents/Appliances/GSG_Maestro/EN/Default.htm). 1. Connect a network cable from your computer to the Orchestrator's "**MGMT**" port. 2. Connect the console cable from your computer to the Orchestrator's **Console** port. In your console client application, configure these settings: * Baud Rate - 9600 * Data bits - 8 * Stop bits - 1 * Parity - None * Flow Control - None 3. Connect the power cables to the Orchestrator's Power Supply Units. 10. On the replacement Orchestrator, in the **console** session, log in to Gaia Clish. |----------------------------------------------------------------------------------| | **Important Note** - Do **not** perform the First Time Configuration Wizard yet. | 11. On the replacement Orchestrator, configure the Orchestrator's "**MGMT**" port: 1. Configure the IPv4 address and Mask Length: |-------------------------------------------------------------------------------------| | `set interface Mgmt1 ipv4-address mask-length ` | 2. Enable the "MGMT" port: |--------------------------------| | `set interface Mgmt1 state on` | 3. Configure the default gateway: |-----------------------------------------------------------------------------------------| | `set static-route default nexthop gateway address on` | 4. Save the changes: |---------------| | `save config` | For information about these Gaia Clish commands, see the [R82 Gaia Administration Guide](https://sc1.checkpoint.com/documents/R82/WebAdminGuides/EN/CP_R82_Gaia_AdminGuide/Default.htm). 12. On the replacement Orchestrator, run the First Time Configuration Wizard: |------------------------------------------------------------------------------------------------------------| | **Important Note** - Do **not** import the backup file from the failed Orchestrator. You must do it later. | 1. With a web browser, connect to the replacement Orchestrator: |-----------------------------------------| | `https://` | Log in with the username "`admin`" and the default password "`admin`". 2. On the second page **Deployment Options** , select **Join an existing Maestro environment**. 3. Follow the rest of the wizard pages. 4. Wait for the Orchestrator to reboot. 13. On the replacement Orchestrator, install the same Take of the **R82 Jumbo Hotfix Accumulator** as installed on **working** Orchestrators (in our example, "1_1"). 1. Copy the Jumbo Hotfix Accumulator package to the replacement Orchestrator: 1. Connect with WinSCP to the IPv4 address of the "MGMT" port on the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to some path. For example: |-------------| | `/var/log/` | 3. Upload the Jumbo Hotfix Accumulator package from your computer. 2. Connect with an SSH client to the IPv4 address of the "MGMT" port. 3. Log in to Gaia Clish. 4. Obtain the lock over the Gaia configuration database: |--------------------------| | `lock database override` | 5. Import the CPUSE package from the hard disk: Note: When import completes, this package is deleted from the original location. |----------------------------------------------------------| | `installer import local /` | 6. Show the imported packages: |------------------------------------| | `show installer packages imported` | 7. Verify the package - see whether this package can be installed without conflicts: |----------------------------------------------------------------------------------------------| | `installer verify`*\[press Space key\]\[press Tab key\]* `installer verify ` | 8. Install the package: |--------------------------------------| | `installer install ` | 9. Wait for the Orchestrator to reboot. 14. On the replacement Orchestrator, import the configuration files you exported earlier from a working Orchestrator: 1. Copy the archive file from your computer to the replacement Orchestrator: 1. Connect with WinSCP to the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to some path. For example: |-------------| | `/var/log/` | 3. Upload the archive file with the configuration files from your computer. In our example, upload this file: `Export_Of_Config_for_Failed_Orch_1_2.tgz` 2. Connect to the command line on the replacement Orchestrator. 3. Log in to Gaia Clish. 4. Import the configuration files: |-------------------------------------------------------------------------------------------------------------| | `set maestro import configuration archive-name path ` | In our example, run: |---------------------------------------------------------------------------------------------------------| | `set maestro import configuration archive-name Export_Of_Config_for_Failed_Orch_1_2.tgz path /var/log/` | 15. On all Orchestrators, make sure the configuration files are the same: 1. Connect to the command line on **each** Orchestrator. 2. Log in to the Expert mode. 3. Calculate the MD5 checksum for the Orchestrator configuration: |------------------------------------------------------------------------------------------------------| | `jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum` | The MD5 checksums must be the same on all Orchestrators. If the MD5 checksums are different: 1. Copy the `/etc/sgdb.json` file from the working Orchestrator (in our example, "1_1") to the replacement Orchestrator (in our example, "1_2") to the `/etc/` directory. Available copy options: * Copy the file directly between Orchestrators. In the Expert mode on the working Orchestrator, run: |----------------------------------------------------------------------------------------| | `scp /etc/sgdb.json admin@:/etc/` | Example: `scp /etc/sgdb.json admin@192.168.3.57:/etc/` * Use WinSCP and your computer as an intermediate place. 2. Calculate the MD5 checksum for the Orchestrator configuration again: |------------------------------------------------------------------------------------------------------| | `jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum` | 16. On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the failed Orchestrator: * The **Downlink** cables * The **Sync** cables |-------------------------------------------------------------------------------| | **Important Note** - Make sure you connected the cables to the correct ports. | 17. On the replacement Orchestrator, start the Orchestrator service: 1. Log in to the Expert mode. 2. Check the Orchestrator service status: |----------------| | `orchd status` | 3. If the Orchestrator is **not** active, then start the Orchestrator service: |---------------| | `orchd start` | 18. If previously you configured the authentication between Orchestrators, you must configure it again (between all working Orchestrators) to trust the replacement Orchestrator. See the [R82 Scalable Platforms Administration Guide](https://sc1.checkpoint.com/documents/R82/WebAdminGuides/EN/CP_R82_ScalablePlatforms_AdminGuide/Default.htm#cshid=ID002) \> Chapter "**Working with Quantum Maestro** " \> Section "**Authentication between Maestro Orchestrators**". You can configure the authentication in Gaia Portal (recommended) or in CLI (Gaia Clish, or Expert mode). 19. On all Orchestrators, make sure the traffic can pass between Orchestrators: * In a Single Site deployment: On a working Orchestrator (in our example, "1_1"), send several pings to the replacement Orchestrator (in our example, "1_2"): |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: On the replacement Orchestrator (in our example, "1_2"), send several pings to **each** working Orchestrator: |-----------------| | `ping -c 5 1_1` | | `ping -c 5 2_1` | | `ping -c 5 2_2` | 20. Make sure the Security Group Members can pass traffic to each other: 1. Connect to the command line on the Security Group. 2. Log in to the Expert mode. 3. Identify the Security Group Member that runs as SMO: |---------------------| | `asg stat -i tasks` | 4. If the SMO task runs on another Security Group Member than the Security Group Member to which you are connected, then go to the context of that SMO Security Group Member: |-----------------------------------------------------| | `member _` | Example: `member 1_2` 5. Examine the cluster state of the Security Group Members. |------------------| | `cphaprob state` | The output must show the state of **each** Security Group Member as "`ACTIVE`". 6. Send several pings between Security Group Members: 1. Connect to one of the Security Group Members (in our example, we connect to the first one - "1_1"): |--------------| | `member 1_1` | 2. On this Security Group Member, send several pings to any other Security Group Member (in our example, we send several pings to the second one - "1_2" / "2_2"): * In a Single Site deployment: |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: |-----------------| | `ping -c 5 1_2` | | `ping -c 5 2_2` | 21. On the replacement Orchestrator, connect the **Uplink** cables to the ports with the same numbers as they were connected on the failed Orchestrator. 22. On each Security Group, make sure all the required links have the status "`UP`": 1. Connect to the command line on the Security Group. 2. Log in. 3. If your default shell is the Expert mode, then go to Gaia gClish: |----------| | `gclish` | 4. Examine the state of links: |--------------------------------| | `show cluster info interfaces` | ### Procedure for Orchestrators that run the version R81.20, R81.20 Jumbo Hotfix (all Takes), or R81.10 Jumbo Hotfix Take 61 (and higher) {#TOC04} Show / Hide this section This procedure applies only to Orchestrators that run one of these versions: * R81.20 without the R81.20 Jumbo Hotfix Accumulator * R81.20 with the R81.20 Jumbo Hotfix Accumulator, any Take * R81.10 with the R81.10 Jumbo Hotfix Accumulator, Take 61 and higher Procedure: 1. On a working Orchestrator, export the configuration files for the failed Orchestrator: 1. Connect to the command line on the working Orchestrator - the peer Orchestrator on the same Site. 2. Log in to Gaia Clish. 3. Export the configuration files for the failed Orchestrator: |--------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `set maestro export remote orchestrator id configuration archive-name path ` | Example: |--------------------------------------------------------------------------------------------------------------------------------| | `set maestro export remote orchestrator id 1_2 configuration archive-name Export_Of_Config_for_Failed_Orch_1_2 path /var/log/` | The output file is: `/var/log/Export_Of_Config_for_Failed_Orch_1_2.tgz` 2. On the failed Orchestrator, stop the Orchestrator service: 1. Connect to the command line on the failed Orchestrator. 2. Log in to the Expert mode. 3. Stop the service: |--------------| | `orchd stop` | 3. On the failed Orchestrator, write down the port numbers, to which the Uplink cable, the Downlink cables, and the Sync cables are connected. 4. On the failed Orchestrator, disconnect all cables. 5. Remove the failed Orchestrator from the rack. 6. Install the replacement Orchestrator **of the same model** in the rack. Do **not** connect the power or network cables yet. You do it gradually later. 7. On the replacement Orchestrator, connect only these cables: See the [Quantum Maestro Getting Started Guide](https://sc1.checkpoint.com/documents/Appliances/GSG_Maestro/EN/Default.htm). 1. Connect a network cable from your computer to the Orchestrator's "**MGMT**" port. 2. Connect the console cable from your computer to the Orchestrator's **Console** port. In your console client, configure these settings: * Baud Rate - 9600 * Data bits - 8 * Stop bits - 1 * Parity - None * Flow Control - None 3. Connect the power cables to the Orchestrator's Power Supply Units. 8. On the replacement Orchestrator, in the **console** session, log in to Gaia Clish. |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Choose **not** to activate the Orchestrator. If you activated the Orchestrator, then: 1. Go to the Expert mode: `expert` 2. Stop the Orchestrator service: `orchd stop` 3. Exit the Expert mode: `exit` | 9. On the replacement Orchestrator, configure the Orchestrator's "MGMT" port: 1. Configure the IPv4 address and Mask Length: |-------------------------------------------------------------------------------------| | `set interface Mgmt1 ipv4-address mask-length ` | 2. Enable the "MGMT" port: |--------------------------------| | `set interface Mgmt1 state on` | 3. Configure the default gateway: |-----------------------------------------------------------------------------------------| | `set static-route default nexthop gateway address on` | 4. Save the changes: |---------------| | `save config` | For information about these Gaia Clish commands, see the [R81.20 Gaia Administration Guide](https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Gaia_AdminGuide/Default.htm). 10. On the replacement Orchestrator, configure the required Gaia settings: 1. Connect with an SSH client to the IPv4 address of the "MGMT" port. 2. Log in to Gaia Clish. 3. Configure the same date and time settings as configured on all other working Orchestrators (in our example, "1_1"). **Note** - The date and time settings must be the same on all Orchestrators. |----------------------------------| | `set timezone / ` | | `set date ` | | `set time ` | 4. Configure the hostname: |---------------------------| | `set hostname ` | 5. Change the default password for the '`admin`' user: |---------------------------| | `set user admin password` | 6. Save the changes: |---------------| | `save config` | For information about these Gaia Clish commands, see the [R81.20 Gaia Administration Guide](https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Gaia_AdminGuide/Default.htm). 11. On the replacement Orchestrator, install the same Take of the [R81.20 Jumbo Hotfix Accumulator](https://sc1.checkpoint.com/documents/Jumbo_HFA/R81.20/Default.htm) / [R81.10 Jumbo Hotfix Accumulator](https://sc1.checkpoint.com/documents/Jumbo_HFA/R81.10/Default.htm) as installed on the working Orchestrator (in our example, "1_1"). 1. Copy the Jumbo Hotfix Accumulator package to the replacement Orchestrator: 1. Connect with WinSCP to the IPv4 address of the "MGMT" port on the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to some path. For example: |-------------| | `/var/log/` | 3. Upload the Jumbo Hotfix Accumulator package from your computer. 2. Connect with an SSH client to the IPv4 address of the "MGMT" port. 3. Log in to Gaia Clish. 4. Obtain the lock over the Gaia configuration database: |--------------------------| | `lock database override` | 5. Import the CPUSE package from the hard disk: Note: When the import completes, this package is deleted from the original location. |----------------------------------------------------------| | `installer import local /` | 6. Show the imported packages: |------------------------------------| | `show installer packages imported` | 7. Verify the package - see whether this package can be installed without conflicts: |----------------------------------------------------------------------------------------------| | `installer verify`*\[press Space key\]\[press Tab key\]* `installer verify ` | 8. Install the package: |--------------------------------------| | `installer install ` | 9. Wait for the Orchestrator to reboot. 12. On the replacement Orchestrator, import the configuration files you exported earlier from a working Orchestrator: 1. Copy the archive file from your computer to the replacement Orchestrator: 1. Connect with WinSCP to the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to some path. For example: |-------------| | `/var/log/` | 3. Upload the archive file with the configuration files from your computer. In our example, upload this file: `Export_Of_Config_for_Failed_Orch_1_2.tgz` 2. Connect to the command line on the replacement Orchestrator. 3. Log in to Gaia Clish. 4. Import the configuration files: |-------------------------------------------------------------------------------------------------------------| | `set maestro import configuration archive-name path ` | In our example, run: |---------------------------------------------------------------------------------------------------------| | `set maestro import configuration archive-name Export_Of_Config_for_Failed_Orch_1_2.tgz path /var/log/` | 13. On all Orchestrators, make sure the configuration files are the same: 1. Connect to the command line on **each** Orchestrator. 2. Log in to the Expert mode. 3. Calculate the MD5 checksum for the Orchestrator configuration: |------------------------------------------------------------------------------------------------------| | `jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum` | The MD5 checksums must be the same on all Orchestrators. If the MD5 checksums are different: 1. Copy the `/etc/sgdb.json` file from the working Orchestrator (in our example, "1_1") to the replacement Orchestrator (in our example, "1_2") to the `/etc/` directory. Available copy options: * Copy the file directly between Orchestrators. In the Expert mode on the working Orchestrator, run: |----------------------------------------------------------------------------------------| | `scp /etc/sgdb.json admin@:/etc/` | Example: `scp /etc/sgdb.json admin@192.168.3.57:/etc/` * Use WinSCP and your computer as an intermediate place. 2. Calculate the MD5 checksum for the Orchestrator configuration again: |------------------------------------------------------------------------------------------------------| | `jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum` | 14. On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the failed Orchestrator: * The **Downlink** cables * The **Sync** cables |-------------------------------------------------------------------------------| | **Important Note** - Make sure you connected the cables to the correct ports. | 15. On the replacement Orchestrator, start the Orchestrator service: 1. Log in to the Expert mode. 2. Disable the Link State Propagation (LSP): |------------------------------------------------------| | `jsont -f /etc/smodb.json -s /orch_lsp_state -v off` | 3. Check the Orchestrator service status: |----------------| | `orchd status` | 4. If the Orchestrator is not active, then activate it (it will start the Orchestrator service): |------------------| | `orchd activate` | (**Otherwise** , start the Orchestrator service: `orchd start`) 5. Enable the Link State Propagation (LSP): |-----------------------------------------------------| | `jsont -f /etc/smodb.json -s /orch_lsp_state -v on` | 6. Restart the Orchestrator port monitoring daemon with these commands: |----------------------------| | `tellpm process:ssm_lsp` | | `tellpm process:ssm_lsp t` | **Note** - It may take a few seconds for the Orchestrator to discover the connected Security Appliances for the first time. In the Expert mode, run the "`watch -d -n 1 orch_stat -l`" command and wait until the "`Status`" columns shows "`UP`" for all "`LAGs(Bonds)`". 16. On all Orchestrators, make sure the traffic can pass between Orchestrators: * In a Single Site deployment: On the working Orchestrator (in our example, "1_1"), send several pings to the replacement Orchestrator (in our example, "1_2"): |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: On the replacement Orchestrator (in our example, "1_2"), send several pings to each working Orchestrator: |-----------------| | `ping -c 5 1_1` | | `ping -c 5 2_1` | | `ping -c 5 2_2` | 17. Make sure the Security Group Members can pass traffic to each other: 1. Connect to the command line on the Security Group. 2. Log in to the Expert mode. 3. Identify the Security Group Member that runs as SMO: |---------------------| | `asg stat -i tasks` | 4. Examine the cluster state of the Security Group Members. On the SMO Security Group Member, run: |------------------| | `cphaprob state` | The output must show that the state of all Security Group Members is "`ACTIVE`". 5. Send several pings between Security Group Members: 1. Connect to one of the Security Group Members (in our example, we connect to the first one - "1_1"): |--------------| | `member 1_1` | 2. On this Security Group Member, send several pings to any other Security Group Member (in our example, we send several pings to the second one - "1_2" / "2_2"): * In a Single Site deployment: |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: |-----------------| | `ping -c 5 1_2` | | `ping -c 5 2_2` | 18. On the replacement Orchestrator, connect the **Uplink** cables to ports with the same numbers as they were connected on the failed Orchestrator. 19. On each Security Group, make sure all links are up on the Security Group: 1. Connect to the command line on the Security Group. 2. Examine the state of links: |----------| | `asg_if` | ### Procedure for Orchestrators that run the version R81.10, or R81.10 Jumbo Hotfix Take 55 (and lower) {#TOC05} Show / Hide this section This procedure applies only to Orchestrators that run one of these versions: * R81.10 without the R81.10 Jumbo Hotfix Accumulator * R81.10 with the R81.10 Jumbo Hotfix Accumulator, Take 55 and lower Procedure: 1. On the working Orchestrator, export the configuration files for the failed Orchestrator: 1. Connect to the command line on the Orchestrator that **continues to work**. 2. Log in to Gaia Clish. 3. Export the configuration files for the Orchestrator that **failed**: |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `set maestro export remote orchestrator id configuration archive-name path ` | Example: |-----------------------------------------------------------------------------------------------------------------------| | `set maestro export remote orchestrator id 1_2 configuration archive-name Export_from_failed_Orch_1_2 path /var/log/` | 2. On the working Orchestrator, copy the archive with the exported configuration files to your computer. Connect with WinSCP to this working Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 3. On the failed Orchestrator, stop the Orchestrator service: 1. Connect to the command line on the failed Orchestrator. 2. Log in to the Expert mode. 3. Stop the service: |--------------| | `orchd stop` | 4. On the failed Orchestrator, write down the port numbers, to which the Uplink cable, the Downlink cables, and the Sync cables are connected. 5. On the failed Orchestrator, disconnect all cables. 6. Remove the failed Orchestrator from the rack. 7. Install the replacement Orchestrator **of the same model** in the rack. Do **not** connect the power or network cables yet. You do it gradually later. 8. On the replacement Orchestrator, connect only these cables: See the [Quantum Maestro Getting Started Guide](https://sc1.checkpoint.com/documents/Appliances/GSG_Maestro/EN/Default.htm). 1. Connect a network cable from your computer to the Orchestrator's "**MGMT**" port. 2. Connect the console cable from your computer to the Orchestrator's **Console** port. In your console client, configure these settings: * Baud Rate - 9600 * Data bits - 8 * Stop bits - 1 * Parity - None * Flow Control - None 3. Connect the power cables to the Orchestrator's Power Supply Units. 9. On the replacement Orchestrator, in the **console** session, log in to Gaia Clish. |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Choose **not** to activate the Orchestrator. If you activated the Orchestrator, then: 1. Go to the Expert mode: `expert` 2. Stop the Orchestrator service: `orchd stop` 3. Exit the Expert mode: `exit` | 10. On the replacement Orchestrator, configure the Orchestrator's "MGMT" port: 1. Configure the IPv4 address and Mask Length: |-------------------------------------------------------------------------------------| | `set interface Mgmt1 ipv4-address mask-length ` | 2. Enable the "MGMT" port: |--------------------------------| | `set interface Mgmt1 state on` | 3. Configure the default gateway: |-----------------------------------------------------------------------------------------| | `set static-route default nexthop gateway address on` | 4. Save the changes: |---------------| | `save config` | For information about these Gaia Clish commands, see the [R81.10 Gaia Administration Guide](https://sc1.checkpoint.com/documents/R81.10/WebAdminGuides/EN/CP_R81.10_Gaia_AdminGuide/Default.htm). 11. On the replacement Orchestrator, configure the required Gaia settings: 1. Connect with an SSH client to the IPv4 address of the "MGMT" port. 2. Log in to Gaia Clish. 3. Configure the same date and time settings as configured on all other working Orchestrators (in our example, "1_1"). **Note** - The date and time settings must be the same on all Orchestrators. |----------------------------------| | `set timezone / ` | | `set date ` | | `set time ` | 4. Configure the hostname: |---------------------------| | `set hostname ` | 5. Change the default password for the '`admin`' user: |---------------------------| | `set user admin password` | 6. Save the changes: |---------------| | `save config` | For information about these Gaia Clish commands, see the [R81.10 Gaia Administration Guide](https://sc1.checkpoint.com/documents/R81.10/WebAdminGuides/EN/CP_R81.10_Gaia_AdminGuide/Default.htm). 12. On the replacement Orchestrator, install the same Take of the [R81.10 Jumbo Hotfix Accumulator](https://sc1.checkpoint.com/documents/Jumbo_HFA/R81.10/Default.htm) as installed on the working Orchestrator (in our example, "1_1"). 1. Copy the Jumbo Hotfix Accumulator package to the replacement Orchestrator: 1. Connect with WinSCP to the IPv4 address of the "MGMT" port on the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to some path. For example: |-------------| | `/var/log/` | 3. Upload the Jumbo Hotfix Accumulator package from your computer. 2. Connect with an SSH client to the IPv4 address of the "MGMT" port. 3. Log in to Gaia Clish. 4. Obtain the lock over the Gaia configuration database: |--------------------------| | `lock database override` | 5. Import the CPUSE package from the hard disk: Note: When the import completes, this package is deleted from the original location. |----------------------------------------------------------| | `installer import local /` | 6. Show the imported packages: |------------------------------------| | `show installer packages imported` | 7. Verify the package - see whether this package can be installed without conflicts: |----------------------------------------------------------------------------------------------| | `installer verify`*\[press Space key\]\[press Tab key\]* `installer verify ` | 8. Install the package: |--------------------------------------| | `installer install ` | 9. Wait for the Orchestrator to reboot. 13. On the replacement Orchestrator, copy the archive with the exported configuration files from your computer to the replacement Orchestrator to some directory (for example, `/var/log/`). Connect with WinSCP to the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 14. On the replacement Orchestrator, import the configuration files you collected earlier: 1. Connect to the command line on the replacement Orchestrator. 2. Log in to Gaia Clish. 3. Import the configuration files: |------------------------------------------------------------------------------------------------------| | `set maestro import configuration archive-name path ` | Example: |------------------------------------------------------------------------------------------------| | `set maestro import configuration archive-name Export_from_failed_Orch_1_2.tgz path /var/log/` | 15. On all Orchestrators, make sure the configuration files are the same: 1. Connect to the command line on **each** Orchestrator. 2. Log in to the Expert mode. 3. Calculate the MD5 checksum for the Orchestrator configuration: |------------------------------------------------------------------------------------------------------| | `jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum` | The MD5 checksums must be the same on all Orchestrators. If the MD5 checksums are different: 1. Copy the `/etc/sgdb.json` file from the working Orchestrator (in our example, "1_1") to the replacement Orchestrator (in our example, "1_2") to the `/etc/` directory. Available copy options: * Copy the file directly between Orchestrators. In the Expert mode on the working Orchestrator, run: |----------------------------------------------------------------------------------------| | `scp /etc/sgdb.json admin@:/etc/` | Example: `scp /etc/sgdb.json admin@192.168.3.57:/etc/` * Use WinSCP and your computer as an intermediate place. 2. Calculate the MD5 checksum for the Orchestrator configuration again: |------------------------------------------------------------------------------------------------------| | `jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum` | 16. On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the failed Orchestrator: * The **Downlink** cables * The **Sync** cables |-------------------------------------------------------------------------------| | **Important Note** - Make sure you connected the cables to the correct ports. | 17. On the replacement Orchestrator, start the Orchestrator service: 1. Log in to the Expert mode. 2. Disable the Link State Propagation (LSP): |------------------------------------------------------| | `jsont -f /etc/smodb.json -s /orch_lsp_state -v off` | 3. Check the Orchestrator service status: |----------------| | `orchd status` | 4. If the Orchestrator is not active, then activate it (it will start the Orchestrator service): |------------------| | `orchd activate` | (**Otherwise** , start the Orchestrator service: `orchd start`) 5. Enable the Link State Propagation (LSP): |-----------------------------------------------------| | `jsont -f /etc/smodb.json -s /orch_lsp_state -v on` | 6. Restart the Orchestrator port monitoring daemon with these commands: |----------------------------| | `tellpm process:ssm_pmd` | | `tellpm process:ssm_pmd t` | **Note** - It may take a few seconds for the Orchestrator to discover the connected Security Appliances for the first time. In the Expert mode, run the "`watch -d -n 1 orch_stat -l`" command and wait until the "`Status`" columns shows "`UP`" for all "`LAGs(Bonds)`". 18. On all Orchestrators, make sure the traffic can pass between Orchestrators: * In a Single Site deployment: On the working Orchestrator (in our example, "1_1"), send several pings to the replacement Orchestrator (in our example, "1_2"): |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: On the replacement Orchestrator (in our example, "1_2"), send several pings to each working Orchestrator: |-----------------| | `ping -c 5 1_1` | | `ping -c 5 2_1` | | `ping -c 5 2_2` | 19. Make sure the Security Group Members can pass traffic to each other: 1. Connect to the command line on the Security Group. 2. Log in to the Expert mode. 3. Identify the Security Group Member that runs as SMO: |---------------------| | `asg stat -i tasks` | 4. Examine the cluster state of the Security Group Members. On the SMO Security Group Member, run: |------------------| | `cphaprob state` | The output must show that the state of all Security Group Members is "`ACTIVE`". 5. Send several pings between Security Group Members: 1. Connect to one of the Security Group Members (in our example, we connect to the first one - "1_1"): |--------------| | `member 1_1` | 2. On this Security Group Member, send several pings to any other Security Group Member (in our example, we send several pings to the second one - "1_2" / "2_2"): * In a Single Site deployment: |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: |-----------------| | `ping -c 5 1_2` | | `ping -c 5 2_2` | 20. On the replacement Orchestrator, connect the **Uplink** cables to ports with the same numbers as they were connected on the failed Orchestrator. 21. On each Security Group, make sure all links are up on the Security Group: 1. Connect to the command line on the Security Group. 2. Examine the state of links: |----------| | `asg_if` | ### Procedure for Orchestrators that run the R80.20SP version - when the failed Orchestrator is still accessible {#TOC06} Show / Hide this section 1. Export the configuration files of the failed Orchestrator: * If the Orchestrators run R80.20SP with the Jumbo Hotfix Accumulator Take 310 and higher: **Note** - Starting from R80.20SP Jumbo Hotfix Accumulator Take 310, each Orchestrator keeps the configuration of the peer Orchestrator as a local backup. 1. Connect to the command line on the Orchestrator that **continues to work**. 2. Log in to Gaia Clish. 3. Export the configuration files of the Orchestrator that **failed**: |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `set maestro export remote orchestrator id configuration archive-name path ` | Example: |-----------------------------------------------------------------------------------------------------------------------| | `set maestro export remote orchestrator id 1_2 configuration archive-name Export_from_failed_Orch_1_2 path /var/log/` | * If the Orchestrators run R80.20SP with the Jumbo Hotfix Accumulator Take 309 and lower: 1. Connect to the command line on the Orchestrator that **failed**. 2. Log in to the Expert mode. 3. Export the configuration files of the Orchestrator that **failed**: |---------------------------------------------------------------------------------------------------------------------------------------------| | `tar -czf /.tgz -C /etc maestro.json sgdb.json smodb.json maestro_full.json` | Example: |-----------------------------------------------------------------------------------------------------------------| | `tar -czf /var/log/Export_from_failed_Orch_1_2.tgz -C /etc maestro.json sgdb.json smodb.json maestro_full.json` | 2. On the failed Orchestrator, copy the archive with the exported configuration files to your computer. Connect with WinSCP to the failed Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 3. On the failed Orchestrator, stop the Orchestrator service: 1. Connect to the command line on the failed Orchestrator. 2. Log in to the Expert mode. 3. Stop the service: |--------------| | `orchd stop` | 4. On the failed Orchestrator, write down the port numbers, to which the Uplink cable, the Downlink cables, and the Sync cables are connected. 5. On the failed Orchestrator, disconnect all cables. 6. Remove the failed Orchestrator from the rack. 7. Install the replacement Orchestrator **of the same model** in the rack. Do **not** connect the power or network cables yet. You do it gradually later. 8. On the replacement Orchestrator, connect only these cables: See the [Quantum Maestro Getting Started Guide](https://sc1.checkpoint.com/documents/Appliances/GSG_Maestro/EN/Default.htm). 1. Connect a network cable from your computer to the Orchestrator's "**MGMT**" port. 2. Connect the console cable from your computer to the Orchestrator's **Console** port. In your console client, configure these settings: * Baud Rate - 9600 * Data bits - 8 * Stop bits - 1 * Parity - None * Flow Control - None 3. Connect the power cables to the Orchestrator's Power Supply Units. 9. On the replacement Orchestrator, in the **console** session, log in to Gaia Clish. 10. On the replacement Orchestrator, configure the Orchestrator's "MGMT" port: 1. Configure the IPv4 address and Mask Length: |-------------------------------------------------------------------------------------| | `set interface Mgmt1 ipv4-address mask-length ` | 2. Enable the "MGMT" port: |--------------------------------| | `set interface Mgmt1 state on` | 3. Configure the default gateway: |-----------------------------------------------------------------------------------------| | `set static-route default nexthop gateway address on` | 4. Save the changes: |---------------| | `save config` | For information about these Gaia Clish commands, see the [R80.20SP Maestro Gaia Administration Guide](https://sc1.checkpoint.com/documents/R80.20SP/WebAdminGuides/EN/CP_R80.20SP_Maestro_Gaia_AdminGuide/html_frameset.htm). 11. On the replacement Orchestrator, configure the required Gaia settings: 1. Connect with an SSH client to the IPv4 address of the "MGMT" port. 2. Log in to Gaia Clish. 3. Configure the same date and time settings as configured on all other working Orchestrators (in our example, "1_1"). **Note** - The date and time settings must be the same on all Orchestrators. |----------------------------------| | `set timezone / ` | | `set date ` | | `set time ` | 4. Configure the hostname: |---------------------------| | `set hostname ` | 5. Change the default password for the '`admin`' user: |---------------------------| | `set user admin password` | 6. Save the changes: |---------------| | `save config` | For information about these Gaia Clish commands, see the [R80.20SP Maestro Gaia Administration Guide](https://sc1.checkpoint.com/documents/R80.20SP/WebAdminGuides/EN/CP_R80.20SP_Maestro_Gaia_AdminGuide/html_frameset.htm). 12. On the replacement Orchestrator, install the same Take of the [R80.20SP Jumbo Hotfix Accumulator](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk155832) as installed on the working Orchestrator (in our example, "1_1"). 1. Copy the Jumbo Hotfix Accumulator package to the replacement Orchestrator: 1. Connect with WinSCP to the IPv4 address of the "MGMT" port on the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to some path. For example: |-------------| | `/var/log/` | 3. Upload the Jumbo Hotfix Accumulator package from your computer. 2. Connect with an SSH client to the IPv4 address of the "MGMT" port. 3. Log in to Gaia Clish. 4. Obtain the lock over the Gaia configuration database: |--------------------------| | `lock database override` | 5. Import the CPUSE package from the hard disk: Note: When the import completes, this package is deleted from the original location. |----------------------------------------------------------| | `installer import local /` | 6. Show the imported packages: |------------------------------------| | `show installer packages imported` | 7. Verify the package - see whether this package can be installed without conflicts: |----------------------------------------------------------------------------------------------| | `installer verify`*\[press Space key\]\[press Tab key\]* `installer verify ` | 8. Install the package: |--------------------------------------| | `installer install ` | 9. Wait for the Orchestrator to reboot. 13. On the replacement Orchestrator, stop the Orchestrator service: 1. Connect to the command line the replacement Orchestrator through the "MGMT" port (over SSH). 2. Log in to the Expert mode. 3. Stop the Orchestrator service: |--------------| | `orchd stop` | 4. Stop the LLDP service: |------------------------| | `tellpm process:lldpd` | 14. On the replacement Orchestrator - copy the archive with the exported configuration files from your computer to the replacement Orchestrator to some directory (for example, `/var/log/`). Connect with WinSCP to the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 15. On the replacement Orchestrator, import the configuration files you collected earlier: * If the Orchestrator runs R80.20SP with the Jumbo Hotfix Accumulator Take 310 and higher: 1. Connect to the command line on the replacement Orchestrator. 2. Log in to Gaia Clish. 3. Import the configuration files: |------------------------------------------------------------------------------------------------------| | `set maestro import configuration archive-name path ` | Example: |------------------------------------------------------------------------------------------------| | `set maestro import configuration archive-name Export_from_failed_Orch_1_2.tgz path /var/log/` | * If the Orchestrator runs R80.20SP with the Jumbo Hotfix Accumulator Take 309 and lower: 1. Connect to the command line on the replacement Orchestrator. 2. Log in to the Expert mode. 3. Unpack the configuration files to the `/etc/` directory: |------------------------------------------------------------------------| | `tar -xzf /.tgz -C /etc` | Example: |-------------------------------------------------------------| | `tar -xzf /var/log/Export_from_failed_Orch_1_2.tgz -C /etc` | 16. On all Orchestrators, make sure the configuration files are the same: 1. Connect to the command line on **each** Orchestrator. 2. Log in to the Expert mode. 3. Calculate the MD5 checksum for the Orchestrator configuration: |------------------------------------------------------------------------------------------------------| | `jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum` | The MD5 checksums must be the same on all Orchestrators. If the MD5 checksums are different: 1. Copy the `/etc/sgdb.json` file from the working Orchestrator (in our example, "1_1") to the replacement Orchestrator (in our example, "1_2") to the `/etc/` directory. Available copy options: * Copy the file directly between Orchestrators. In the Expert mode on the working Orchestrator, run: |----------------------------------------------------------------------------------------| | `scp /etc/sgdb.json admin@:/etc/` | Example: `scp /etc/sgdb.json admin@192.168.3.57:/etc/` * Use WinSCP and your computer as an intermediate place. 2. Calculate the MD5 checksum for the Orchestrator configuration again: |------------------------------------------------------------------------------------------------------| | `jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum` | 17. On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the failed Orchestrator: * The **Downlink** cables * The **Sync** cables |-------------------------------------------------------------------------------| | **Important Note** - Make sure you connected the cables to the correct ports. | 18. On the replacement Orchestrator, start the Orchestrator service: 1. Connect to the command line on each Orchestrator. 2. Log in to the Expert mode. 3. Disable the Link State Propagation (LSP): |------------------------------------------------------| | `jsont -f /etc/smodb.json -s /orch_lsp_state -v off` | 4. Start the Orchestrator service: |---------------| | `orchd start` | 5. Enable the Link State Propagation (LSP): |-----------------------------------------------------| | `jsont -f /etc/smodb.json -s /orch_lsp_state -v on` | 6. Restart the Orchestrator service (this also starts the LLDP service): |-----------------| | `orchd restart` | 19. On all Orchestrators, make sure the traffic can pass traffic between Orchestrators: * In a Single Site deployment: On the working Orchestrator (in our example, "1_1"), send several pings to the replacement Orchestrator (in our example, "1_2"): |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: On the replacement Orchestrator (in our example, "1_2"), send several pings to each working Orchestrator: |-----------------| | `ping -c 5 1_1` | | `ping -c 5 2_1` | | `ping -c 5 2_2` | 20. Make sure the Security Group Members can pass traffic to each other: 1. Connect to the command line on the Security Group. 2. Log in to the Expert mode. 3. Identify the Security Group Member that runs as SMO: |---------------------| | `asg stat -i tasks` | 4. Examine the cluster state of the Security Group Members. On the SMO Security Group Member, run: |------------------| | `cphaprob state` | The output must show that the state of all Security Group Members is "`ACTIVE`". 5. Send several pings between Security Group Members: 1. Connect to one of the Security Group Members (in our example, we connect to the first one - "1_1"): |--------------| | `member 1_1` | 2. On this Security Group Member, send several pings to any other Security Group Member (in our example, we send several pings to the second one - "1_2" / "2_2"): * In a Single Site deployment: |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: |-----------------| | `ping -c 5 1_2` | | `ping -c 5 2_2` | 21. On the replacement Orchestrator, connect the **Uplink** cables to ports with the same numbers as they were connected on the failed Orchestrator. 22. On each Security Group, make sure all links are up on the Security Group: 1. Connect to the command line on the Security Group. 2. Examine the state of links: |----------| | `asg_if` | ### Procedure for Orchestrators that run the R80.20SP version with the Jumbo Hotfix Accumulator Take 309 and lower - when the failed Orchestrator is not accessible anymore, or after restoring an Orchestrator to factory defaults {#TOC07} Show / Hide this section **Important** - Follow the procedure below if the failed Orchestrator is not accessible, or if its configuration is empty (for example, after restoring an Orchestrator to factory defaults). 1. On the working Orchestrator, export the required configuration files: 1. Connect with WinSCP to this working Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Copy these files from the working Orchestrator to your computer: * `/etc/maestro_remote.json` * `/etc/maestro_full.json` * `/etc/sgdb.json` * `/etc/smodb.json` 2. On the failed Orchestrator, write down the port numbers, to which the Uplink cable, the Downlink cables, and the Sync cables are connected. 3. Remove the failed Orchestrator from the rack. 4. Install the replacement Orchestrator **of the same model** in the rack. Do **not** connect the power or network cables yet. You do it gradually later. 5. On the replacement Orchestrator, connect only these cables: See the [Quantum Maestro Getting Started Guide](https://sc1.checkpoint.com/documents/Appliances/GSG_Maestro/EN/Default.htm). 1. Connect a network cable from your computer to the Orchestrator's "**MGMT**" port. 2. Connect the console cable from your computer to the Orchestrator's **Console** port. In your console client, configure these settings: * Baud Rate - 9600 * Data bits - 8 * Stop bits - 1 * Parity - None * Flow Control - None 3. Connect the power cables to the Orchestrator's Power Supply Units. 6. On the replacement Orchestrator, in the **console** session, log in to Gaia Clish. 7. On the replacement Orchestrator, configure the Orchestrator's "MGMT" port: 1. Configure the IPv4 address and Mask Length: |-------------------------------------------------------------------------------------| | `set interface Mgmt1 ipv4-address mask-length ` | 2. Enable the "MGMT" port: |--------------------------------| | `set interface Mgmt1 state on` | 3. Configure the default gateway: |-----------------------------------------------------------------------------------------| | `set static-route default nexthop gateway address on` | 4. Save the changes: |---------------| | `save config` | For information about these Gaia Clish commands, see the [R80.20SP Maestro Gaia Administration Guide](https://sc1.checkpoint.com/documents/R80.20SP/WebAdminGuides/EN/CP_R80.20SP_Maestro_Gaia_AdminGuide/html_frameset.htm). 8. On the replacement Orchestrator, configure the required Gaia settings: 1. Connect with an SSH client to the IPv4 address of the "MGMT" port. 2. Log in to Gaia Clish. 3. Configure the same date and time settings as configured on all other working Orchestrators (in our example, "1_1"). **Note** - The date and time settings must be the same on all Orchestrators. |----------------------------------| | `set timezone / ` | | `set date ` | | `set time ` | 4. Configure the hostname: |---------------------------| | `set hostname ` | 5. Change the default password for the '`admin`' user: |---------------------------| | `set user admin password` | 6. Save the changes: |---------------| | `save config` | For information about these Gaia Clish commands, see the [R80.20SP Maestro Gaia Administration Guide](https://sc1.checkpoint.com/documents/R80.20SP/WebAdminGuides/EN/CP_R80.20SP_Maestro_Gaia_AdminGuide/html_frameset.htm). 9. On the replacement Orchestrator, install the same Take of the [R80.20SP Jumbo Hotfix Accumulator](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk155832) as installed on the working Orchestrator (in our example, "1_1"). 1. Copy the Jumbo Hotfix Accumulator package to the replacement Orchestrator: 1. Connect with WinSCP to the IPv4 address of the "MGMT" port on the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Go to some path. For example: |-------------| | `/var/log/` | 3. Upload the Jumbo Hotfix Accumulator package from your computer. 2. Connect with an SSH client to the IPv4 address of the "MGMT" port. 3. Log in to Gaia Clish. 4. Obtain the lock over the Gaia configuration database: |--------------------------| | `lock database override` | 5. Import the CPUSE package from the hard disk: Note: When the import completes, this package is deleted from the original location. |----------------------------------------------------------| | `installer import local /` | 6. Show the imported packages: |------------------------------------| | `show installer packages imported` | 7. Verify the package - see whether this package can be installed without conflicts: |----------------------------------------------------------------------------------------------| | `installer verify`*\[press Space key\]\[press Tab key\]* `installer verify ` | 8. Install the package: |--------------------------------------| | `installer install ` | 9. Wait for the Orchestrator to reboot. 10. On the replacement Orchestrator, stop the Orchestrator service: 1. Connect with an SSH client to the IPv4 address of the "MGMT" port. 2. Log in to the Expert mode. 3. Stop the Orchestrator service: |--------------| | `orchd stop` | 4. Stop the LLDP service: |------------------------| | `tellpm process:lldpd` | 11. On the replacement Orchestrator, copy the required configuration files from your computer: 1. Connect with WinSCP to the replacement Orchestrator. |------------------------------------------------------------------------------------------------------------------------------------| | **Important Note** - Refer to [sk42178](https://support.checkpoint.com/results/sk/sk42178) to make sure this SCP connection works. | 2. Copy these files from your computer to the replacement Orchestrator to the **/etc/** directory: * `maestro_remote.json` * `maestro_full.json` * `sgdb.json` * `smodb.json` 12. On the replacement Orchestrator, edit the configuration files (in the Expert mode): 1. Rename the `/etc/maestro_remote.json` file: |----------------------------------------------------| | `mv -v /etc/maestro_remote.json /etc/maestro.json` | 2. In the file `/etc/smodb.json`, configure the ID of the failed Orchestrator in the "`member_id`" parameter: 1. Examine the current value of the "`member_id`" parameter: |----------------------------------| | `grep member_id /etc/smodb.json` | 2. Edit the file: |----------------------| | `vi /etc/smodb.json` | 3. Change the value of the "`member_id`" parameter to the ID of the failed Orchestrator. In our example, the required value is "2": |--------------------| | `"member_id" : 2,` | 4. Save the changes in the file and exit Vi editor. 13. On all Orchestrators, make sure the configuration files are the same: 1. Connect to the command line on **each** Orchestrator. 2. Log in to the Expert mode. 3. Calculate the MD5 checksum for the Orchestrator configuration: |------------------------------------------------------------------------------------------------------| | `jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum` | The MD5 checksums must be the same on all Orchestrators. If the MD5 checksums are different: 1. Copy the `/etc/sgdb.json` file from the working Orchestrator (in our example, "1_1") to the replacement Orchestrator (in our example, "1_2") to the `/etc/` directory. Available copy options: * Copy the file directly between Orchestrators. In the Expert mode on the working Orchestrator, run: |----------------------------------------------------------------------------------------| | `scp /etc/sgdb.json admin@:/etc/` | Example: `scp /etc/sgdb.json admin@192.168.3.57:/etc/` * Use WinSCP and your computer as an intermediate place. 2. Calculate the MD5 checksum for the Orchestrator configuration again: |------------------------------------------------------------------------------------------------------| | `jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum` | 14. On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the failed Orchestrator: * The **Downlink** cables * The **Sync** cables |-------------------------------------------------------------------------------| | **Important Note** - Make sure you connected the cables to the correct ports. | 15. On the replacement Orchestrator, start the Orchestrator service: 1. Connect to the command line. 2. Log in to the Expert mode. 3. Disable the Link State Propagation (LSP): |------------------------------------------------------| | `jsont -f /etc/smodb.json -s /orch_lsp_state -v off` | 4. Start the Orchestrator service: |---------------| | `orchd start` | 5. Enable the Link State Propagation (LSP): |-----------------------------------------------------| | `jsont -f /etc/smodb.json -s /orch_lsp_state -v on` | 6. Restart the Orchestrator service (this also starts the LLDP service): |-----------------| | `orchd restart` | 16. On all Orchestrators, make sure the traffic can pass between Orchestrators: * In a Single Site deployment: On the working Orchestrator (in our example, "1_1"), send several pings to the replacement Orchestrator (in our example, "1_2"): |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: On the replacement Orchestrator (in our example, "1_2"), send several pings to each working Orchestrator: |-----------------| | `ping -c 5 1_1` | | `ping -c 5 2_1` | | `ping -c 5 2_2` | 17. Make sure the Security Group Members can pass traffic to each other: 1. Connect to the command line on the Security Group. 2. Log in to the Expert mode. 3. Identify the Security Group Member that runs as SMO: |---------------------| | `asg stat -i tasks` | 4. Examine the cluster state of the Security Group Members. On the SMO Security Group Member, run: |------------------| | `cphaprob state` | The output must show that the state of all Security Group Members is "`ACTIVE`". 5. Send several pings between Security Group Members: 1. Connect to one of the Security Group Members (in our example, we connect to the first one - "1_1"): |--------------| | `member 1_1` | 2. On this Security Group Member, send several pings to any other Security Group Member (in our example, we send several pings to the second one - "1_2" / "2_2"): * In a Single Site deployment: |-----------------| | `ping -c 5 1_2` | * In a Dual Site deployment: |-----------------| | `ping -c 5 1_2` | | `ping -c 5 2_2` | 18. On the replacement Orchestrator, connect the **Uplink** cables to ports with the same numbers as they were connected on the failed Orchestrator. 19. On each Security Group, make sure all links are up on the Security Group: 1. Connect to the command line on the Security Group. 2. Examine the state of links: |----------| | `asg_if` | ### Revision History {#TOC08} Show / Hide this section |---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Date | Change | | 04 Nov 2024 | * Improved the procedure for Orchestrators that run the version R81.20, R81.20 Jumbo Hotfix (all Takes), or R81.10 Jumbo Hotfix Take 61 (and higher) * Improved formatting | | 03 Nov 2024 | * Added the procedure for Orchestrators R82 - Failed Orchestrator is still accessible * Added the procedure for Orchestrators R82 - Failed Orchestrator is not accessible * Improved formatting | | 01 Nov 2024 | Added the procedure for Orchestrators that run the version R81.20, R81.20 Jumbo Hotfix (all Takes), or R81.10 Jumbo Hotfix Take 61 (and higher) | | 15 May 2022 | Corrected the Baud rate values - removed "115200" | | 21 March 2022 | * Improved the procedures * Added the procedure for Orchestrators that run the R80.20SP version - when a failed Orchestrator is not accessible anymore, or after restoring an Orchestrator to factory defaults | | 14 March 2022 | Improved the terms used in procedures | | 10 Aug 2021 | Added the procedure for Orchestrators that run the R81.10 version | | 28 June 2021 | Created this article for Orchestrators that run the R80.20SP version | --- # Agent Instructions This content is from the Check Point Support Center (https://support.checkpoint.com), the official knowledge base for Check Point cybersecurity products. ## Navigating This Knowledge Base - **Complete index**: [llms.txt](https://support.checkpoint.com/llms.txt) - **All SK articles**: [SecureKnowledge Sitemap](https://support.checkpoint.com/sitemaps/secureknowledge-sitemap-index.xml) - **SK article URL pattern**: `https://support.checkpoint.com/results/sk/{skId}` - **Markdown responses**: AI bot User-Agents automatically receive `text/markdown` content