> Source: [sk162357](https://support.checkpoint.com/results/sk/sk162357) # sk162357 - Maestro Known Issues and Solutions | Property | Value | |----------|-------| | Solution ID | sk162357 | | Date Created | 2019-09-11 | | Last Modified | 2025-01-22 | | Technical Level | General | | Products | Scalable Platforms | | Versions | R82.10, R82, R81.20, R81.10 (EOS) | | OS | Gaia | ## Solution This article describes Known Issues in Maestro appliances. The Table of Contents lists all Known Issues, each of which has its own cause and solution. This article will be updated as we discover more known Issues. **This article is for INTERNAL use only.** ### **Table of Contents:** 1. Traffic outage after Jumbo Hotfix installation 2. "*Fail to load Maestro security groups* " error after initial deployment 3. NAT port exhaustion 4. LACP Bonds are suspended after rebooting Maestro Orchestrator 5. Maestro Gateway fails to join an existing the Security Group 6. Security Groups topology apply fails with "*SG X has no hostname* " error 7. sx_sdk core dump during installation of R80.20SP JHF 8. Traffic does not reach Maestro Gateways because "fwha_ssm_type=16" was removed from the Gateway's *fwkern.conf* file 9. "*Failed to run {\['tor_util',} 'attach_sgm_to_uplink* '" error when adding an interface to a Security Group through the Orchestrator WebUI (in a Maestro Dual Site environment) 10. New Security Group is inaccessible through the Management IP after you configure the Management IP address in subnet 192.0.1.0/24 11. "*No connectivity to Orchestrators on site 2* " message appears on WebUI after uninstalling Jumbo Hotfix 12. Link establishment issue with specific Cisco switches 13. No connectivity between Orchestrators on different sites in Dual-site environment 14. Failed to load Security Groups topology on the Orchestrator WebUI / Failed to apply Security Groups topology on remote Orchestrator / REST-API between Orchestrators and Security Appliances fails / Security Appliances uplinks link-state are down 15. `sx_api_ports_dump.py` command fails with error 16. `orch_info` command fails with "*...KeyError: 'HOSTNAME'...* " 17. Traffic outage of up to 5 minutes when upgrading Jumbo Hotfix on MHO 18. Traffic outage because of Bond configuration when working with Active/Backup bond mode 19. In Dual Site, if an administrator changes the state of a port on one of the Orchestrators, the corresponding port on the peer Orchestrator on the other site changes its state in the same way 20. The "`set maestro port X/Y/Z admin-state {down | up}`" command on the Orchestrator does not survive reboot Click Here to Show the Entire Article ### (1) Traffic outage after Jumbo Hotfix installation {#01} Show / Hide this section **Symptoms:** After the user installs a Jumbo Hotfix on the Orchestrators, the output of the "*cpinfo -y all*" command shows that the SMO package is missing: ``` [Expert@orch1:0]# cpinfo -y all ... ... [SMO] No hotfixes ... ... ``` **Cause:** The CPUSE Agent was upgraded incorrectly on R80.20SP Maestro/chassis products. **Solution:** Follow the instructions in [sk159272: CPUSE Agent was wrongly upgraded on R80.20SP on Maestro/chassis products](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk159272). ### (2) "Fail to load Maestro security groups" error after initial deployment {#02} Show / Hide this section **Symptoms:** One of these error appears: * `Failed to load Security Groups: Orchestrator id is missing` * `Orchestrator was not able to determine its ID` **Solution:** 1. Verify that the Orchestrator date is configured properly. Run in the Expert mode: *# date*. If the configured date is older than the Epoch time (1 January 1970), configure the current correct date. 2. Follow the instructions in [sk161593: Fail to load Maestro security groups during initial configuration](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk161593). ### (3) NAT port exhaustion {#03} Show / Hide this section **Symptoms:** * Kernel debug for drop shows one of these lines: * `static_allocate_port_ex: failed to allocate port` * `allocate_port_impl: could not find a free port for hiding src , dst , port 59593, during rebuild? false;` **Solution:** Follow the instructions in [sk159572: Port Exhaustion issue when using NAT with Maestro](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk159572). ### (4) LACP Bonds are suspended after rebooting Maestro Orchestrator {#04} Show / Hide this section **Symptoms:** LACP Bonds are suspended after rebooting Maestro Orchestrator. **Cause:** This is a known issue: refer to [sk162335: LACP bonds are suspended after rebooting Maestro Orchestrator](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk162335). **Solution:** 1. Install the following Hotfix: \\\\galaxy\\ckp\\pkg\\smo\\R80_20SP_JHF_T121_LSP_main\\B992000002_1\\linux50\\release.dynamic.x32.ping\\smo_HOTFIX_R80_20SP_JHF_T121_LSP_MAIN_GA_FULL.tgz The Hotfix should be installed on top of JHF T121, only on the Orchestrators, and on all of them. Note that the Hotfix installation will initiate an Orchestrator reboot which will impact traffic. 2. Set the value of the kernel parameter **fwha_ch_if_preempt_time** to 1 (one) on all Security Group Members. Run this command on the Security Group in the Global Clish: `update_conf_file $FWDIR/boot/modules/fwkern.conf fwha_ch_if_preempt_time=1` 3. Reboot the Security Group: `reboot -b all` 4. Check the value of the kernel parameter. Run this command on the Security Group in the Expert mode: `g_fw ctl get int fwha_ch_if_preempt_time` ### (5) Maestro Gateway fails to join an existing Security Group {#05} Show / Hide this section **Symptoms:** * When trying to add a Maestro Gateway to an existing Security Group, the Gateway fails to join. * A ping from SMO to the newly added Gateway fails. * Output of the tcpdump on the SMO shows that the Sync traffic (192.0.2.X) is hidden behind NAT. **Cause:** Refer to [sk163093: Maestro Gateway fails to join an existing Security Group](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk163093). **Solution:** Refer to [sk163093: Maestro Gateway fails to join an existing Security Group](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk163093). ### (6) Security Groups topology apply fails - SG X has no hostname {#06} Show / Hide this section **Symptoms:** "`SG has no hostname`" error appears when applying the Security Groups topology. **Cause:** The hostname of the Security Group \ is missing from in */etc/sgdb.json* file. **Solution:** **This problem was fixed. The fix is included in [Jumbo Hotfix Accumulator for R80.20SP (Take 178)](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk155832).** The following **workaround** is also available: Add the missing hostname to the `/etc/sgdb.json` file. Run this command in the Expert mode on the Orchestrator: `jsont -f /etc/sgdb.json -s /security_groups//hostname -v ` Example for the Security Group 1: `jsont -f /etc/sgdb.json -s /security_groups/1/hostname -v sg1_hostname` ### (7) sx_sdk core dump during installation of R80.20SP JHF {#07} Show / Hide this section **Symptom:** Core dump files are created for the `sx_sdk` process during installation of JHF. **Solution:** This is a known issue that occurs during JHF installation. It has no impact and will be resolved in future JHFs (see [MBS-6650](https://jira-prd.checkpoint.com/browse/MBS-6650)). ### (8) Traffic does not reach Maestro Gateways since "fwha_ssm_type=16" was removed from the Gateways fwkern.conf {#08} Show / Hide this section **Symptom:** * Traffic does not reach Security Group Members because the line "`fwha_ssm_type=16`" was removed from the *$FWDIR/boot/modules/fwkern.conf* file on Security Group Members (see [MBS-8359](https://jira-prd.checkpoint.com/browse/MBS-8359)). * */var/log/messages* contains these lines: `sgm_pmd: Aligning eth1-Mgmt1 cluster link state to up (current cluster state is ` `pending)` `sgm_pmd: Setting interface eth1-Mgmt1 link state to up` **Solution:** Add the line "`fwha_ssm_type=16`" to the *$FWDIR/boot/modules/fwkern.conf* file on Security Group Members. Run this command in the Global Clish on the Security Group: `update_conf_file $FWDIR/boot/modules/fwkern.conf fwha_ssm_type=16` ### (9) "Failed to run {\['tor_util',} 'attach_sgm_to_uplink'" error when adding an interface to a Security Group via the Orchestrator WebUI (in a Maestro Dual Site environment {#09} Show / Hide this section **Symptom:** In a Maestro Dual Site environment, when the user adds an interface to a Security Group via the Orchestrator WebUI, the following error appears: "`Failed to run {['tor_util',} 'attach_sgm_to_uplink'`" This error may occur if the QSFP mode of the same port is different on parallel Orchestrators. For example, port 5 is split on Orchestrator 1_1, but is not split on Orchestrator 2_1. **Solution:** Uplinks split must be identical on parallel Orchestrators (for example, 1_1 and 2_1). Change the port QSFP mode via the Orchestrator Clish to be identical on parallel Orchestrators. For example: `set maestro port 1/5/1 qsfp-mode 4x10G` ### (10) New Security Group is inaccessible via the Management IP after configured the Management IP address in subnet 192.0.1.0/24 {#10} Show / Hide this section **Symptoms:** * New Security Group is inaccessible via the Management IP address when using the subnet 192.0.1.0/24. * The output of the Gaia Clish command "`show interface eth-Mgmt`" shows that IP address is not configured. **Solution:** See [sk164372](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk164372). ### (11)"No connectivity to Orchestrators on site 2" message appears in WebUI after uninstalling Jumbo Hotfix {#11} Show / Hide this section **Symptoms:** * When uninstalling a Jumbo Hotfix and then installing a new Jumbo Hotfix, and there is a Security Group with Gateways only on site 2, the Orchestrators on site 2 might not respond to Rest requests. * When the Orchestrators on site 2 are in GA state, they add site 1 to the Security Group with only site 2 (even though site 1 does not exist). **Workaround:** 1. Connect to the command line on the Orchestrators on Site 2. 2. Log in to the Expert mode. 3. Delete site 1 on these Security Groups. **Note:** Run this command only for Security Groups with no Gateways on site 1. You can see this in the Gaia Portal. `jsont -d /security_groups//chassis/1 -f /etc/sgdb.json` 4. Restart the REST utility: `rest_api_stop` ### (12) Link establishment with specific Cisco switch models in specific Cisco ports (depends on the Cisco model) that are plugged into the Orchestrator fails or takes a long time {#12} Show / Hide this section **Symptoms:** Link establishment with specific Cisco switch models in specific Cisco ports (depends on the Cisco model) that are plugged into the Orchestrator fails or takes a long time. **Cause:** Cisco auto negotiation issues. For an example, see **Solution:** Disable auto negotiation and use forced speed on the Cisco switch. ### (13) No connectivity between GWs on different sites in Dual-site environment {#13} Show / Hide this section **Symptoms:** 1. No ping from GW 1_1 to GW 2_1: * `ping 192.0.2.15` * `ping 192.0.2.32` 2. Output of the Gaia Clish command "`show maestro configuration orchestrator-site-vlan`" on the Orchestrator shows different VLAN IDs on different Orchestrators **Solution:** 1. Connect to the command line on each Maestro Orchestrator on each site. 2. Log in to Gaia Clish. 3. Configure the same site-sync VLAN 3600: `set maestro configuration orchestrator-site-vlan 3600` 4. Log in to the Expert mode. 5. Restart the *orchd* process: `orchd restart` ### (14) Failed to load Security Groups topology on the Orchestrator WebUI / Failed to apply Security Groups topology on remote Orchestrator / REST-API between Orchestrators and Security Appliances fails / Security Appliances uplinks link-state are down {#14} Show / Hide this section **WARNING: Changing the Orchestrators hosts access can cause Security Appliances and Orchestrators to fail to communicate. This may cause Security Appliances uplinks link-state to go down.** **Possible Symptoms:** * This issue (number # 14) is **only**relevant to R80.20SP MHO (and not, for example, to R81.10 or higher). * Failed to apply Security Groups topology on a remote Orchestrator, or REST-API between Orchestrators fails. During topology application in the Orchestrator's Gaia Portal, this error may appear: ``` Warning: The other Orchestrators <...> are unreachable. If you apply the configuration now, it cannot be synchronized to those Orchestrators. ``` * When you open the Orchestrator's Gaia Portal \> Topology page, this error may appear: "`Failed to load Security Groups: Failed to get remote site appliances. Verify connectivity between sites.`" * These errors appear in the */var/log/messages* file: ``` http_res_code = 403 403 Forbidden You don't have the permission to access the requested resource. It is either read-protected or not readable by the server. ``` **Cause:** The file */etc/hosts.allow* was modified manually on the Orchestrator (Gaia Portal \> Allowed hosts). The current settings in this file do not allow "All hosts to connect" (the file no longer contains '`ALL: ALL`'), and specifically does not allow remote Orchestrators and Gateways to connect. **Solution:** Add all remote Orchestrators' internal IP addresses (of the Sync-ext / Sync-int ports) and all remote Gateways' CIN IP addresses (of the eth1-CIN / eth2-CIN) to the list of allowed hosts in the Orchestrator's Gaia Portal or Gaia Clish. Run these commands in the Orchestrator's Gaia Clish: `add allowed-client host ipv4-address 127.0.0.1 ` `add allowed-client network ipv4-address 192.0.2.0 mask-length 24 ` `add allowed-client network ipv4-address 203.0.113.0 mask-length 24` and for each Security Group ID \ created: On Orchestrator 1: `add allowed-client network ipv4-address 198.51.100+.0 mask-length 25` On Orchestrator 2: `add allowed-client network ipv4-address 198.51.100+.128 mask-length 25` Example for Security Group 1: On Orchestrator 1: `add allowed-client network ipv4-address 198.51.101.0 mask-length 25` On Orchestrator 2: `add allowed-client network ipv4-address 198.51.101.128 mask-length 25` Run these commands in the Security Group ID \'s Gaia Clish: `add allowed-client host ipv4-address 127.0.0.1` `add allowed-client network ipv4-address 192.0.2.0 mask-length 24` `add allowed-client network ipv4-address 198.51.100+.0 mask-length 24` Example for the Security Group 1: `add allowed-client network ipv4-address 198.51.101.0 mask-length 24` ### (15) sx_api_ports_dump.py command fails with error {#15} Show / Hide this section **Symptoms:** The "`sx_api_ports_dump.py and other sx_api_*`" commands fail with a Python error: ``` [Expert@Orechstrator:0]# sx_api_ports_dump.py Traceback (most recent call last): File "/bin/sx_api_ports_dump.py", line 31, in from test_infra_common import * File "/usr/local/bin/test_infra_common.py", line 17, in from python_sdk_api.sx_api import * ImportError: No module named python_sdk_api.sx_api ``` **Cause:** Python environment variables were removed from the `~/.bashrc` file **Solution:** 1. Add these lines to the `~/.bashrc` file: ``` export PYTHONHOME=/usr/bin/Python64/ export PYTHONPATH=:/usr/lib/python2.7/site-packages/ ``` 2. Open a **new** shell. ### (16) orch_info command fails with "`...KeyError: 'HOSTNAME'...`" {#16} Show / Hide this section **Symptoms:** The "`orch_info`" command fails with "`...KeyError: 'HOSTNAME'...`" **Root Cause:** The BASH shell does not export the HOSTNAME variable due to manual configuration changes. **Solution:** The issue was fixed in [R80.20SP JHF Take 240](https://supportcenter.us.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk155832) (see MBS-9394). As a workaround, you can also run this command in the Expert mode: ``HOSTNAME=`hostname` orch_info`` ### (17) Traffic outage up to 5 minutes when upgrading JHF on MHO {#17} Show / Hide this section **Symptoms:** Traffic outage up to 5 minutes (usually 2-3 minutes) is experienced when upgrading the JHF on MHO. **Root Cause:** When updating a JHF, the first step taken by CPUSE is to uninstall the current JHF. This leads to an undesirable scenario where CPUSE runs the "`service orchd stop`" command while there is no JHF installed. **Solution:** The issue will be fixed in an upcoming JHF (see MBS-10473). As a workaround, in the Expert mode run the "`orchd stop`" command on each MHO before beginning a JHF update: 1. On MHO 1, run: `orchd stop` 2. On MHO 1, update JHF 3. MHO 1 reboots 4. Wait several minutes 5. On MHO 2, run: `orchd stop` 6. On MHO 2, update JHF 7. MHO 2 reboots ### (18) Traffic outage due to Bond configuration when working with Active/Backup bond mode {#18} Show / Hide this section **Symptoms:** 1. A Bond is configured as Active / Backup. This Bond includes two uplink slave interfaces (for example, eth1-05 and eth2-05). 2. There are traffic outages on networks connected to this Bond. **Root Cause:** If a primary slave is not configured in this Bond, different Security Group Members may use different slaves of the same bond as Active. **Solution:** **THIS PROCEDURE MAY CAUSE DOWNTIME. MAINTENANCE WINDOW IS REQUIRED!!!** Configure a primary slave with this command in Global Clish on the Security Group: `set bonding group 1 primary ` ### (19) In Dual Site, if an administrator changes the state of a port on one of the Orchestrators, the corresponding port on the peer Orchestrator on the other site changes its state in the same way {#19} Show / Hide this section **Symptoms:** In a Dual Site environment, when running the "`set maestro port X/Y/Z admin-state {down | up}`" command on one Orchestrator (for example, MHO 1_1), the corresponding port on the peer Orchestrator on the other site (in our example, MHO 2_1) also goes down/up. **Root Cause:** By design, if port state is changed on one Orchestrator, the peer Orchestrator on the other site automatically changes the state of the peer port. Note - In the R81.10SP release, this behavior changed - the Orchestrators do not mirror each other in port status when bringing down / up a port on one of the Orchestrators. **Solution:** For more information, see [sk175549](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk175549). ### (20) The "`set maestro port X/Y/Z admin-state {down | up}`" command on the Orchestrator does not survive reboot {#20} Show / Hide this section **Symptoms:** If an administrator changes the state of a port the Orchestrator with the "`set maestro port X/Y/Z admin-state {down | up}`" command, then this change does not survive reboot of the Orchestrator. **Root Cause:** Known issue - see MBS-11339 in [sk148074](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk148074) [MBS-11339](https://jira-prd.checkpoint.com/browse/MBS-11339) **Solution:** The fix will be integrated (time of this writing is 15 Oct 2021): * R80.20SP Jumbo Hotfix Accumulator (see MBS-11339) * R81.10 Jumbo Hotfix Accumulator (see MBS-11339 + PRJ-32097) --- # Agent Instructions This content is from the Check Point Support Center (https://support.checkpoint.com), the official knowledge base for Check Point cybersecurity products. ## Navigating This Knowledge Base - **Complete index**: [llms.txt](https://support.checkpoint.com/llms.txt) - **All SK articles**: [SecureKnowledge Sitemap](https://support.checkpoint.com/sitemaps/secureknowledge-sitemap-index.xml) - **SK article URL pattern**: `https://support.checkpoint.com/results/sk/{skId}` - **Markdown responses**: AI bot User-Agents automatically receive `text/markdown` content