> Source: [sk98348](https://support.checkpoint.com/results/sk/sk98348) # sk98348 - Best Practices - Security Gateway Performance | Property | Value | |----------|-------| | Solution ID | sk98348 | | Date Created | 2014-01-30 | | Last Modified | 2026-01-11 | | Technical Level | General | | Products | Security Gateway | | Versions | R82.10, R82, R81.20, R81.10 (EOS) | ## Solution This article provides best practices for Security Gateway performance. [Click here](https://supportcenter.checkpoint.com//supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk98348) to view the article. **Click Here to Show the Entire Article** **Table of Contents (click on section titles to see sub-sections):** * (1) Background * (2) Introduction and Limitations * (2-1) SecureXL * (2-2) CoreXL * (2-3) SMT (HyperThreading) * (2-4) Multi-Queue * (3) Best practices * (3-1) Network interface cards * (3-2) Throughput * (3-3) SecureXL * (3-4) CoreXL * (3-5) SecureXL with CoreXL * (3-6) SMT (HyperThreading) * (3-7) Multi-Queue * (3-8) Rule Base optimization * (3-9) IPS optimization * (3-10) Application Control \& URL Filtering optimization * (3-11) Anti-Virus \& Anti-Bot optimization * (4) Initial diagnostics * (4-1) CPU * (4-2) Memory * (4-3) Network interface cards * (4-4) SecureXL * (4-5) CoreXL * (5) Advanced diagnostics * (5-1) CPU * (5-2) Memory * (5-3) Network interface cards * (5-4) SecureXL * (5-5) CoreXL * (6) Command Line syntax * (6-1) SecureXL * (6-1-A) 'fwaccel' command * (6-1-B) 'sim' command * (6-2) CoreXL * (6-2-A) Gateway mode * (6-2-B) VSX mode * (6-3) Multi-Queue * (7) Examples * (8) Related documentation * (9) Related solutions * (10) Revision history Organization of this article: * Chapter 1 "*Background*" - provides a short background on the performance of Security Gateway. * Chapter 2 "*Introduction*" - lists the relevant definitions, supported configurations, limitations, and commands specific to a product. * Chapter 3 "*Best practices*" - provides the recommendations and guidelines for achieving the optimal performance. * Chapter 4 "*Initial diagnostics*" - lists the basic commands and guidelines for checking the current utilization of machine's resources. * Chapter 5 "*Advanced diagnostics*" - lists the advanced commands and guidelines for checking the current utilization of machine's resources. * Chapter 6 "*Command Line syntax*" - provides the complete list of commands and their options specific to a product. * Chapter 7 "*Examples*" - provides real-life examples. * Chapter 8 "*Related documentation*" - lists the relevant documents. * Chapter 9 "*Related solutions*" - lists the relevant solutions. *** ** * ** *** (1) Background {#Background} ---------------------------- The performance of Security Gateway depends on: * CPU - utilization / saturation / errors * Memory - utilization / saturation / errors * Network Interfaces - utilization / saturation / errors * Storage device I/O, capacity, controller - utilization / saturation / errors * Throughput (packet rate \* packet size) Check Point solutions for improving the performance of Security Gateway: * SecureXL - refer to [sk153832 - ATRG: SecureXL for R80.20 and higher](https://support.checkpoint.com/results/sk/sk153832) * CoreXL - refer to [sk98737 - ATRG: CoreXL](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk98737) * SMT (HyperThreading) - refer to [sk93000 - SMT (HyperThreading) Feature Guide](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk93000) * [Performance Tuning Administration Guide](https://support.checkpoint.com/product/530) for your version - Chapter "Multi-Queue" * [ClusterXL Administration Guide](https://support.checkpoint.com/product/531) for your version * VPN - refer to [sk105119 - Best Practices - VPN Performance](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk105119) and to [sk104760 - ATRG: VPN Core](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk104760) *** ** * ** *** (2) Introduction and Limitations {#Introduction} ------------------------------------------------ ### (2-1) Introduction and Limitations - SecureXL {#Introduction and Limitations - SecureXL} Show / Hide introduction information > **Definitions:** > > * *Performance Pack* is a software acceleration product installed on Security Gateways. The Performance Pack uses *SecureXL* technology and other innovative network acceleration techniques to deliver wire-speed performance for Security Gateways. SecureXL is implemented either in software, or in hardware (SAM cards on Check Point 21000 appliances; ADP cards on IP Series appliances). > > * *Affinity* - Association of a particular network interface with a CPU core (either 'Automatic' (default), or 'Static' / 'Manual'). Interfaces are bound to CPU cores via SMP IRQ affinity settings (refer to [sk61962 - SMP IRQ Affinity on Check Point Security Gateway)](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk61962). > > Notes: > * The default SIM Affinity setting for all interfaces is 'Automatic' - the affinity for each interface is automatically reset every 60 seconds, and balanced between all available CPU cores based on the current CPU load. > * "sim affinity" has been deprecated in R80.40. To change the affinity, use either: "fw ctl affinity" (see (6-2) Command Line syntax - CoreXL), or mq_mng tools (refer to [Performance Tuning Administration Guide](https://support.checkpoint.com/product/530) for your version) > * *FWACCEL* - FireWall Accelerator (acceleration feature). > > * *SIM* - SecureXL Implementation Module (acceleration device). > > * *Connection offload* - Firewall kernel passes the relevant information about the connection from Firewall Connections Table to SecureXL Connections Table. > Note: In ClusterXL High Availability, the connections are ***not*** offloaded to SecureXL on *Standby* member. > > * *Connection notification* - SecureXL passes the relevant information about the accelerated connection from SecureXL Connections Table to Firewall Connections Table. > > * *Partial connection* - Connection that exists in the Firewall Connections Table, but not in the SecureXL Connections Table (versions R70 and above). > > * In Cluster HA - partial connections are offloaded when a cluster member becomes Active > * In Cluster LS - partial connections are offloaded upon post-sync (only for NAT / VPN connections) > > Such connections must be offloaded to SecureXL, since packets in these connections must not be dropped. > If a packet matched a partial connection in the outbound, then it should be dropped. > * *Delayed connection* - Connection created from SecureXL Connection Templates without notifying the Firewall for a predefined period of time. > > * *Anticipated connection* - Connection that is anticipated by SecureXL based on policy rules to avoid dropping it by Drop Template. > > In the following firewall policy: > > |-----|--------|-------------|---------------|---------|----------| > | NO. | SOURCE | DESTINATION | VPN | SERVICE | ACTION | > | 1 | `Any` | `Any` | `Any Traffic` | `ftp` | `Accept` | > | 2 | `Any` | `Any` | `Any Traffic` | `Any` | `Drop` | > > 1. Any FTP connection that will be opened will be accepted. > 2. A SYN packet of a Telnet connection will be dropped, and a Drop Template will be offloaded to SecureXL. > 3. The FTP Data connection FTP might match this Drop Template and will be dropped. > > Therefore, to prevent the drop of a legitimate connection: > 1. SecureXL will offload such connections (in this example, FTP Data) and will mark them as anticipated. > 2. SecureXL will try to match an anticipated connection to an existing connection or an existing Accept Template. > 3. If such a match was found, the packet will be forwarded to the firewall and will not be matched to a Drop Template. > * *Accept Template* - Feature that accelerates the speed, at which a connection is established by matching a new connection to a set of attributes. When a new connection matches the Accept Template, subsequent connections are established without performing a rule match and therefore are accelerated. Accept Templates are generated from active connections according to policy rules. Currently, Accept Template acceleration is performed only on connections with the **same** destination port (using wildcards for source ports). > Note: Size of Templates table (*cphwd_tmpl* , id 8111) is limited to 1/4 of the size of Firewall Connections Table (*connections*, id 8158). > > * *Drop Template* - Feature that accelerates the speed, at which a connection is dropped by matching a new connection to a set of attributes. When a new connection matches the Drop Template, subsequent connections are dropped without performing a rule match and therefore are accelerated. Currently, Drop Template acceleration is performed only on connections with the **same** destination port (does not use wildcards for source ports). Drop Templates are generated from policy rules by a special algorithm: > > * Analyze the Rule Base > * Produce mutually exclusive ranges > * Offload the ranges to SecureXL > * Once a packet is dropped, offload a Drop Template > * All subsequent packets matching that range will be dropped by SecureXL > * *Accelerated path* - Packet flow when the packet is completely handled by the SecureXL device. It is processed and forwarded to the network. > > * *Medium path* (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL Firewall instances to perform the processing (even when CoreXL is disabled, the CoreXL infrastructure is used by the SecureXL device to send the packet to the single CoreXL Firewall instance that still functions). > > * *Firewall path* / *Slow path* (F2F) - Packet flow when the SecureXL device is unable to process the packet (refer to [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578)). The packet is passed on to the CoreXL layer and then to one of the CoreXL Firewall instances for full processing. This path also processes all packets when SecureXL is disabled. > > * *Active Streaming* (CPAS) - Technology that sends streams of data to be inspected in the kernel, since more than a single packet at a time is needed to understand the application that is running (such as HTTP data). Active Streaming is Read-enabled and Write-enabled, and works as a transparent proxy. Connections that pass through Active Streaming cannot be accelerated by SecureXL. > > * *Passive Streaming* - Technology that sends streams of data to be inspected in the kernel, since more than a single packet at a time is needed to understand the application that is running (such as HTTP data). Passive Streaming is Read-only, and it cannot hold packets, but the connections are accelerated by SecureXL. > > * *Passive Streaming Library* (PSL) - IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to [sk95193 - ATRG: IPS](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk95193). > > * *PXL* - Technology name for the combination of SecureXL and PSL. > > * *QXL* - Technology name for the combination of SecureXL and QoS (R77.10 and above). > > * *F2F* / *F2Fed* - Packets that cannot be accelerated by SecureXL (refer to [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578)) are Forwarded to the Firewall. > > * *F2P* - Forward to PSL/Applications. Feature that allows to perform the PSL processing on the CPU cores, which are dedicated to the Firewall. > > * *IRQ Swizzling* - Traditionally, in a PCIe bus, all PCIe ports are mapped to one interrupt. Swizzling allows the PCIe slots to be balanced across four interrupts instead of one (enabling IRQ Swizzling requires a BIOS update). > > **Supported operating systems:** > > * Gaia OS > * Gaia Embedded OS > > **Traffic flow:** > > [![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_1.png)](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_1.png "Click the image to see it in full size in a new tab/window") > > > > [![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_2.png)](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_2.png "Click the image to see it in full size in a new tab/window") > > **Packet flow:** > > * Flow logic: > > > > [![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_3.png)](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_3.png "Click the image to see it in full size in a new tab/window") > > * Accelerated packet: > > > > [![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_4.png)](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_4.png "Click the image to see it in full size in a new tab/window") > > * Packet matched to a template: > > > > [![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_5.png)](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_5.png "Click the image to see it in full size in a new tab/window") > > * Non-accelerated packet: > > > > [![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_6.png)](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/SecureXL_Flow_6.png "Click the image to see it in full size in a new tab/window") > > **Limitations:** > > * For limitations of traffic acceleration and templating, refer to [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578). > * SIM Affinity is deprecated in R80.40 and higher versions, refer to [sk170012](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk170012). > * SIM Affinity is applicable only to **physical** interfaces (VLAN / Bond interfaces are influenced by the "physical" decision). > Note: Some systems may require BIOS update to enable IRQ Swizzling and EIRQ technologies. To determine whether a specific system supports the required technology, contact your hardware vendor. > * On VSX Gateway, WARP interfaces can not be assigned to a CPU core. > * The SecureXL NAT Templates are supported ([sk71200](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk71200)). > * The SecureXL Drop Templates are supported ([sk66402](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk66402)). > * The SecureXL Optimized Drops feature (used for DoS/DDoS protection) is supported ([sk90861](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk90861)). > * The SecureXL Penalty box feature (used for DoS/DDoS protection) is supported (on VSX Gateway, the penalty box would only be enforced on Virtual System 0) ([sk74520](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk74520)). > * SecureXL does ***not*** support Point-to-Point interfaces (PPP, PPTP, PPPoE). > * In R80.20, SAM is supported only for non-accelerated usage. Traffic connected to the Acceleration-ready 10G Interface Card (CPAC-ACCL-4-10F-21000) will be handled by the host. 10G Ports on the CPAC-ACCL-4-10F-21000 cannot be assigned as SAM ports in R80.20. > Note: In case a PPP-interface is detected, SecureXL disables itself on that interface ([sk79880](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk79880)). > * ClusterXL Sticky Decision Function (SDF) **disables** SecureXL. > * It is possible that a connection will exist in the Firewall Connections Table, but not in the SecureXL Connections Table (partial connection). > This situation can occur: > * After policy installation > * After cluster failover > * If a user turned off ('`fwaccel off`' command) and turned on ('`fwaccel on`' command) acceleration > > **Documentation:** > > * [Performance Tuning Administration Guide](https://support.checkpoint.com/product/530) for your version - Chapter "SecureXL" > > **Command Line syntax:** > > FWACCEL (controls acceleration feature) > > ``` > [Expert@HostName]# fwaccel [-h] > [Expert@HostName]# fwaccel6 [-h] > ``` ### (2-2) Introduction and Limitations - CoreXL {#Introduction and Limitations - CoreXL} Show / Hide introduction information > **Definitions:** > > * *CoreXL* - A performance-enhancing technology for Security Gateways on multicore processing platforms. CoreXL enhances Security Gateway performance by enabling the CPU processing cores to concurrently perform multiple tasks. > > * *Secure Network Distributor* (*SND*) - Traffic entering network interface cards (NICs) is directed to a processing CPU core running the SND, which is responsible for: > > * Processing incoming traffic from the network interfaces > * Securely accelerating authorized packets (if SecureXL is enabled) > * Distributing non-accelerated packets among Firewall kernel instances (SND maintains global dispatching table - which connection was assigned to which instance) > * *Firewall Instance* / *FW Instance* - On a Security Gateway with CoreXL enabled, the Firewall kernel is replicated multiple times. Each replicated copy, or Firewall Instance, runs on one CPU processing core. These Firewall instances handle traffic concurrently, and each Firewall instance is a complete and independent Firewall inspection kernel. When CoreXL is enabled, all the Firewall kernel instances on the Security Gateway process traffic through the same interfaces and apply the same security policy. > > * *Affinity* - Association of a particular network interface / Firewall kernel instance / daemon with a CPU core (either 'Automatic' (default), or 'Manual'). > Note: The default CoreXL interface affinity setting for all interfaces is 'Automatic' when SecureXL is installed and enabled. > > * *Accelerated path* - Packet flow when the packet is completely handled by the SecureXL device. It is processed and forwarded to the network. > > * *Medium path* (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL Firewall instances to perform the processing. This path is available only when CoreXL is enabled. > > * *Firewall path* / *Slow path* (F2F) - Packet flow when the SecureXL device is unable to process the packet (refer to [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578)). The packet is passed on to the CoreXL layer and then to one of the CoreXL Firewall instances for full processing. This path also processes all packets when SecureXL is disabled. > > * *Passive Streaming Library* (PSL) - IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to [sk95193 - ATRG: IPS](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk95193). > > * *PXL* - Technology name for the combination of SecureXL and PSL. > > **Supported operating systems:** > > * Gaia OS > > **Architecture:** > > *Example of default configuration for machine with 8 CPU cores*: > > [![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/8_CPUs_2_SNDs_6_FWs.png)](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/8_CPUs_2_SNDs_6_FWs.png "Click the image to see it in full size in a new tab/window") > > **Default number of CoreXL IPv4 FW instances:** > > Note: The real number of CoreXL FW instances depends on the current CoreXL license. > > > > |---------------------|--------------------------------------------|------------------------------------------------------| > > | Number of CPU cores | Default number of CoreXL IPv4 FW instances | Default number of Secure Network Distributors (SNDs) | > > | 1 | 1 Note: CoreXL is disabled | 0 Note: CoreXL is disabled | > > | 2 | 2 | 2 | > > | 4 | 3 | 1 | > > | 6 - 20 | `[Number of CPU cores] - 2` | 2 | > > | 21-59^(1)^ | `[Number of CPU cores] - 4` | 4 | > > | More than 59 | `[Number of CPU cores] - 10` | 10 | > > > >
> > > > **Notes:** > > > > * For Maximal number of CoreXL IPv4 FW instances, check [sk98737 - ATRG: CoreXL](https://support.checkpoint.com/results/sk/sk98737) > > > **Relation between CoreXL IPv4 FW instances and CoreXL IPv6 FW instances:** > (run the '`fw ctl multik stat`' command and '`fw6 ctl multik stat`' command on Security Gateway) > > * The number of **IPv4** Firewall instances - from a minimum of 2 to a number equal to the maximal number of CPU cores on the Security Gateway. > * The number of **IPv6** Firewall instances - from a minimum of 2 to a number equal to the number of IPv4 Firewall instances. > * The number of **IPv6** Firewall instances cannot exceed the number of **IPv4** Firewall instances. > * The total number of **IPv4** Firewall instances and **IPv6** Firewall instances together cannot exceed **32**. > > **CoreXL and ClusterXL:** > > * The number of CoreXL Firewall instances must be identical on all members of the cluster because the state synchronization between members is performed for each CoreXL Firewall instance (e.g., CoreXL Firewall Instance #2 on Member_A can synchronize only with CoreXL Firewall Instance #2 on Member_B). > Note: A Cluster Member with a higher number of CoreXL Firewall instances enters the '`Ready`' state. Refer to [sk42096 - Cluster member is stuck in 'Ready' state](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk42096). > > **Documentation:** > > * [Performance Tuning Administration Guide](https://support.checkpoint.com/product/530) for your version - Chapter "CoreXL" > > **Command Line syntax:** > > ``` > [Expert@HostName]# fw ctl multik > [Expert@HostName]# fw ctl affinity [-flag] > ``` ### (2-3) Introduction and Limitations - SMT (HyperThreading) {#Introduction and Limitations - SMT (HyperThreading)} Show / Hide introduction information > **Definitions:** > > * *SMT* (Simultaneous Multi-Threading - Intel� HyperThreading, or Intel� HT) is a feature that is supported on Check Point appliances running Gaia OS. When enabled, SMT doubles the number of logical CPUs on the Security Gateway, which enhances physical processor utilization. When SMT is disabled, the number of logical CPUs equals the number of physical cores. > SMT improves performance up to 30% in NGFW software blades such as **IPS** , **Application \& URL Filtering** and **Threat Prevention** by increasing the number of CoreXL Firewall instances based on the number of logical CPUs. > > **Supported configurations:** > > * SMT is supported ***only*** on Security Gateways running **Gaia OS** with the **64-bit** kernel. > * SMT is supported on Check Point appliances. > * SMT is supported on Open Servers starting from R80.40 Jumbo Hotfix Take #45. > > **Limitations:** > > * SMT is relevant ***only*** if CoreXL is enabled (this is the default). > > **Notes:** > > * When SMT (HyperThreading) is enabled, the number of available CPU cores is **not** linear. > > *Example*: > > ``` > CPU core 0-5 - socket 1 > CPU core 6-11 - socket 2 > CPU core 12-15 - socket 1 HyperThreading > CPU core 16-23 - socket 2 HyperThreading > ``` > > **Documentation:** > > * [sk93000 - SMT (HyperThreading) Feature Guide](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk93000) ### (2-4) Introduction and Limitations - Multi-Queue {#Introduction and Limitations - Multi-Queue} Show / Hide introduction information > **Background:** > > Today, each network interface card has one traffic queue that is handled by one CPU at a time. Since the Secure Network Distributor (SND) - SecureXL and CoreXL Distributor is running on the CPU cores that handle the traffic queues, user cannot use more CPU cores for acceleration than the number of network interface cards passing the traffic. > > **Definitions:** > > * *Multi-Queue* is an acceleration feature that lets the user configure more than one traffic queue for each network interface card, which allows using more CPU cores for acceleration. > > * *RX queue* - Receive packet queue. > > * *TX queue* - Transmit packet queue. > > * *Secure Network Distributor (SND)* - Traffic entering network interface cards (NICs) is directed to a processing CPU core running the SND, which is responsible for: > > * Processing incoming traffic from the network interfaces > * Securely accelerating authorized packets (if SecureXL is enabled) > * Distributing non-accelerated packets among Firewall kernel instances > * *IRQ affinity* - Process of binding a network interface card's IRQ to one or more CPU cores. > > * *Accelerated path* - Packet flow when the packet is completely handled by the SecureXL device. It is processed and forwarded to the network. > > * *Medium path* (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled. > > * *Firewall path* / *Slow path* - Packet flow when the SecureXL device is unable to process the packet (refer to [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578)). The packet is passed on to the CoreXL layer and then to one of the CoreXL FW instances for full processing. This path also processes all packets when SecureXL is disabled. > > * *Passive Streaming Library* (PSL) - IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to [sk95193 - ATRG: IPS](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk95193). > > * *PXL* - Technology name for combination of SecureXL and PSL. > > **Supported configurations:** > > * Multi-Queue is supported on Check Point Appliances (including IP Series Appliances) and on Open Servers. > * Multi-Queue is supported ***only*** for network interface cards that use **igb** (1 GbE) and **ixgbe** (10 GbE) drivers. > > **Limitations:** > > * Multi-Queue is relevant ***only*** if SecureXL is enabled (this is the default). > * Multi-Queue is relevant ***only*** if CoreXL is enabled (this is the default). > * Multi-queue is ***not*** supported on Appliances / Open Servers with a single CPU core. > * In Gaia with kernel 2.6.18: Multi-Queue allows to configure a **maximum** of 5 interfaces (due to IRQ limitations). > * The number of traffic queues is **limited** by the number of CPU cores and the type of network interface card driver: > * **igb** driver: > * Gaia with kernel 2.6.18: up to 4 RX queues > * Gaia with kernel 3.10: 2-16 (depends on the interface) > * **ixgbe** driver - up to 16 RX queues > > **Documentation:** > > * [Performance Tuning Administration Guide](https://support.checkpoint.com/product/530) for your version - Chapter "Multi-Queue" > > **Command Line syntax:** > > ``` > [Expert@HostName]# mq_mng -h > ``` *** ** * ** *** (3) Best practices {#Best practices} ------------------------------------ ### (3-1) Best practices - Network interface cards {#Best practices - NICs} Show / Hide best practices information > * Refer to the list of [Certified Network Interfaces](https://www.checkpoint.com/support-services/hcl/). > * You should use the [PCI Express](http://en.wikipedia.org/wiki/PCI-Express) (PCIe) cards, because they have better performance than [PCI-X](http://en.wikipedia.org/wiki/PCI-X) cards. > * If you are using a motherboard with multiple PCI or PCI-X buses, make sure that each Network Interface Card is installed in a slot connected to a **different** bus. > * If you are using more than two Network Interface Cards in a system with only two 64-bit/66Mhz PCI buses, make sure that the least-used cards are installed in slots connected to the same bus. > * If the traffic rate is very high and causes high amount of SoftIRQ (and traffic drops on interfaces), consider increasing the sizes of receive/transmit buffer on network interface cards as described in [sk42181 - How to increase sizes of buffer on SecurePlatform/Gaia for Intel NIC and Broadcom NIC](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk42181). ### (3-2) Best practices - Throughput {#Best practices - Throughput} Show / Hide best practices information > **Setting the maximal number of concurrent connections:** > > 1. SmartConsole - open Security Gateway object. > 2. Go to '`Optimizations`' pane. > 3. The '`Calculate the maximum limit for concurrent connections`' should be set to '`Automatically`'. > Note: Manual limit should be set only for security reasons. > 4. Enter the desired value. > 5. The '`Calculate connections hash table size and memory pool`' should be set to '`Automatically`'. > 6. Click on '`OK`'. > 7. Install the policy. > > Notes: > > * You should ensure that the total number of concurrent connections is appropriate to the TCP end timeout. > Too many concurrent connections will adversely affect the Security Gateway's performance. > * You can calculate the maximum number of concurrent connections by multiplying the *session establishment rate* by the *TCP session timeout* : > `[MAXIMAL number of concurrent connections] = [MAXIMAL session establishment rate] x [TCP entire session timeout]` > Notes: > * This formula is used to understand what number of concurrent connections a session rate test will generate. > * It is very hard to predict the maximal connections capacity of a Security Gateway because of multiple varying factors. > * Administrator should monitor the memory utilization on Security Gateway after changing these settings. > * Only the maximal (possible) session rate should be considered. > * This formula will *not* predict connections capacity, which is stated in Check Point datasheet documents. > * Average session rate can be obtained using the [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878) - from '`Traffic`' tab. > * Additional information can be obtained based on [sk67560 - How to export History Report from SmartView Monitor](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk67560) - from '`Traffic`' view. > * TCP timeout varies highly between applications and protocols (e.g., SSH is usually long, HTTP is usually short). > * By reducing the following timeouts, you increase the capacity of actual TCP and UDP connections (SmartConsole - '`Policy`' menu - '`Global Properties`' - '`Stateful Inspection`'): > * *TCP end timeout* - determines the amount of time a TCP connection will stay in the FireWall Connections Table (id 8158) after a TCP session has ended. > * *UDP virtual session timeout* - determines the amount of time a UDP connection will stay in the FireWall Connections Table (id 8158) after the last UDP packet was seen by the Security Gateway. > > **Optimal connection/sec rate:** > > 1. SmartConsole - go to '`Policy`' menu - click on '`Global Properties`'. > 2. Go to '`FireWall`' pane. > 3. Uncheck the following boxes: > * Accept RIP > * Accept Domain Name over UDP (Queries) > * Accept Domain Name over TCP (Zone Transfer) > * Accept ICMP requests > 4. Click on '`OK`'. > 5. Install the policy. > > **ARP cache table:** > > Increase the number of entries in the ARP cache table: > > * If you are testing large subnets that are directly connected to the Security Gateway without a router. > * If '`kernel: neighbour table overflow`' appears repeatedly in `/var/log/messages` files and in the output of the '`dmesg`' command. > Related solutions: > * [sk43772 - 'kernel: neighbour table overflow' appears repeatedly in /var/log/messages files](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk43772) > * [sk92372 - How to change the size of IPv6 Neighbors cache table](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk92372) > > **NAT session rate:** > > Enable SecureXL NAT Templates per [sk71200 - SecureXL NAT Templates](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk71200).
### (3-3) Best practices - SecureXL {#Best practices - SecureXL} Show / Hide best practices information > **Note:** This section does ***not*** take the CoreXL into consideration. > > **BIOS settings:** > > * If your BIOS supports CPU clock setting, then make sure that the BIOS is set to the **actual** CPU speed (no over-clocking). > * If your BIOS supports Hyper-Threading, then refer to "SMT (HyperThreading)" section. > > **Network Interface Cards:** > > * Refer to the list of [Certified Network Interfaces](https://www.checkpoint.com/support-services/hcl/). > * You should use the [PCI Express](http://en.wikipedia.org/wiki/PCI-Express) (PCIe) cards, because they have better performance than [PCI-X](http://en.wikipedia.org/wiki/PCI-X) cards. > * If you are using a motherboard with multiple PCI or PCI-X buses, make sure that each Network Interface Card is installed in a slot connected to a **different** bus. > * If you are using more than two Network Interface Cards in a system with only two 64-bit/66Mhz PCI buses, make sure that the least-used cards are installed in slots connected to the same bus. > > **Network interface affinity:** > > * Each NIC should be bound (affined) to a separate CPU core using the 'Static' affinity mode (run the '`sim affinity -s`' command). > * **Note** : The '`sim affinity`' command has been deprecated in R80.40 (refer to [sk170012](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk170012)). To configure affinity, refer to the section "(3-4) CoreXL". > * Pairs of interfaces carrying significant data flows (based on network topology) should be assigned to pairs of CPU cores on the same physical CPU processor. > * Pairs of interfaces that serve the same connections (based on network topology) should be assigned to pairs of CPU cores on the same physical CPU core. > * On Open Servers with 4 CPU cores and Dual Port NICs, the IRQ Swizzling technology should be enabled in the BIOS to properly distribute IRQs among 4 CPU cores. > Note: Applies only to Dual Port NICs. IRQ Swizzling is not required with Quad Port NICs. > * For systems with 8 CPU cores and Quad Port NICs, the EIRQ technology should be enabled to properly distribute IRQs among all 8 CPU cores. > Note: At the time of this writing, there is no certified platform with this ability. > > **NAT session rate:** > > Enable SecureXL NAT Templates per [sk71200 - SecureXL NAT Templates](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk71200). > > **Accelerated Drop Rules:** > > * Accelerated Drop Rules protect the Security Gateway and site from Denial of Service attacks by dropping packets at the acceleration layer (SecureXL). > * The drop rules are configured in a file on the Security Gateway (using the '`sim dropcfg `' command), which is then offloaded to the SecureXL device for enforcement. > Note: There is no relation between the rules being configured in the file and the rules configured in SmartConsole. > >
> > **Optimized Drops:** > > * Allows SecureXL to accelerate dropped traffic. > * Once the feature is enabled, dropped traffic is accelerated depending on traffic drop rate per second. Once drop rate matches the thresholds, feature is dynamically activated/deactivated. > * Refer to [sk90861 - Optimized Drops feature in R76 and above.](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk90861) > > **Delayed Synchronization in cluster:** > > * To decrease the load on CPU in cluster environment: > > * Either disable the synchronization of non-critical connections (e.g., UDP DNS, ICMP). > * Or (if connection must be synchronized) start synchronizing the connection only some time after its initiation (right-click on the service - '`Edit...`' - '`Advanced...`' - check the box '`Start synchronizing [X] seconds after connection initiation`' - install policy). > > Notes: > * Applies only to TCP services whose '`Protocol Type`' is set to '`HTTP`' or '`None`'. > * Delayed Synchronization is disabled if the '`Track`' option in the rule is set to '`Log`' or '`Account`'. > * Delayed Synchronization is performed only for connections matching a SecureXL Connection Template. ### (3-4) Best practices - CoreXL {#Best practices - CoreXL} Show / Hide best practices information > **Note:** This section does ***not*** take the SecureXL into consideration. > > **Notes:** > > * CoreXL improves performance with almost linear scalability in the following scenarios: > > * Much of the traffic can not be accelerated by SecureXL > * Many IPS features enabled, which disable SecureXL functionality > * Large rulebase > * NAT rules > * CoreXL will ***not*** improve performance in the following scenarios: > > * SecureXL accelerates much of the traffic > * Traffic mostly consists of VPN > * Traffic mostly consists of VoIP > * It is very important to verify that the CPU cores are equally utilized (run the '`top`' command). If this is not the case, you should consider changing the distribution of the Secure Network Distributors (SNDs) and CoreXL FW instances. > > * Under normal circumstances, it is ***not*** recommended for the SND and a CoreXL FW instance to share the same CPU core. > However, it is necessary in the following cases: > > * When using a machine with exactly two cores. It is better for both SNDs and CoreXL FW instances to share CPU cores, instead of allocating only one CPU core to each. > * When you know that almost all of the packets are being processed in the *Accelerated path*, and you want to assign all CPU cores to this path. If the CoreXL FW instances do not process significant amount of traffic, then it is appropriate to share the CPU cores. > * The interfaces affinity should be configured only to to CPU cores that are running as SNDs (CPU cores that are ***not*** running CoreXL FW instances), with these exceptions: > > * On machines with exactly **two** CPU cores (both SNDs and CoreXL FW instances use the same CPU cores). > * For tests, in which traffic is accelerated by SecureXL (if it is enabled). > * If you replaced the CPU on the machine to a CPU with more/less CPU cores than the previous CPU, then you must reconfigure the number of CoreXL FW instances in the '`cpconfig`' menu. > > * In a cluster environment, changing the number of CoreXL Firewall instances should be treated as a version upgrade - a cluster member member with a higher number of CoreXL Firewall instances enters the '`Ready`' state. Refer to [sk42096 - Cluster member is stuck in 'Ready' state](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk42096). > To change the number of CoreXL Firewall instances, schedule a maintenance window and follow either a *Minimal Effort Upgrade* procedure, or a *Zero Downtime Upgrade* procedure from the [*Installation and Upgrade Guide*](https://support.checkpoint.com/product/531) for your version. > > **Changing the distribution of the SND, CoreXL FW instances, and daemons among the CPU cores:** > > Note: Run the '`fw ctl affinity -l -r -a -v`' command to see the current distribution. > > * To change the distribution of the SND, CoreXL FW instances, and daemons, change the current affinities of interfaces and/or of daemons. > > Notes: > * To make the affinity settings persistent, edit the '`$FWDIR/conf/fwaffinity.conf`' file on Security Gateway (see the contents of the file for the correct syntax). > * To apply the configuration from the '`$FWDIR/conf/fwaffinity.conf`' file *on-the-fly* (without reboot), execute the '`$FWDIR/scripts/fwaffinity_apply`' shell script on Security Gateway (see the contents of the script for available flags). > * To ensure CoreXL's efficiency, all traffic must be directed to CPU cores that are running as SNDs (CPU cores that are ***not*** running CoreXL FW instances). > Therefore, if you change affinities of interfaces and/or daemons, you will need to accordingly set the number of CoreXL FW instances and ensure that the CoreXL FW instances run on other CPU cores. > > * It is recommended to allocate an additional CPU core to the SND ***only*** if all of the following conditions are met: > > * Your platform has at least 8 CPU cores. > * The '`idle`' value (run '`top`' command and press 1 to display all CPU cores) for the CPU core currently running the SND is in the 0%-5% range. > * The sum of the '`idle`' values (run the '`top`' command and press 1 to display all CPU cores) for the CPU cores running CoreXL FW instances is significantly higher than 100%. > > If any of the above conditions are ***not*** met, the default configuration of one processing core allocated to the SND is sufficient, and no further configuration is necessary. > * It is recommended to allocate an additional CPU core to the FWD daemon if the Security Gateway is performing heavy logging. > > **Notes:** > * A Security Gateway with 20 cores or more automatically reassign one worker core for exclusive use by the FWD daemon for logging. > * Avoiding the CPU cores that are running the SND is important ***only*** if these CPU cores are explicitly defined as affinities of interfaces. If interface affinities are set to 'Automatic', any CPU core that is ***not*** running a CoreXL FW instance can be used for the FWD daemon, and interface traffic will be automatically diverted to other CPU cores. > > > *Example of configuration for machine with 8 CPU cores*: > > [![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/8_CPUs_2_SNDs_5_FWs_1_Daemon.png)](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/8_CPUs_2_SNDs_5_FWs_1_Daemon.png "Click the image to see it in full size in a new tab/window") ### (3-5) Best practices - SecureXL with CoreXL {#Best practices - SecureXL with CoreXL} Show / Hide best practices information > **Background:** > > When most of the traffic is accelerated by SecureXL (run '`fwaccel stats -s`' command), the load on CPU cores that run as Secure Network Distributor (SND) can be very high, while the load on CPU cores that run CoreXL FW instances can be very low. This is an inefficient utilization of CPU capacity. > > **Notes:** > > * Traffic is processed by the CoreXL FW instances only when the traffic is ***not*** accelerated by SecureXL (if SecureXL is installed and enabled). > > * With CoreXL, there are cases when performance **without** SecureXL is better than with it, even when SecureXL does manage to accelerate part of the traffic. > > **Packet flow:** > > When SecureXL is enabled, a packet enters the Security Gateway and first reaches the SecureXL device. The packet will be handled in one of three ways: > > 1. *Accelerated path* - The packet is completely handled by the SecureXL device. It is processed and forwarded to the network. > > 2. *Medium path* (PXL) - Packet flow when the packet is handled by the SecureXL device, except for IPS (some protections) / VPN (in some configurations) / Application Control / Content Awareness / Anti-Virus / Anti-Bot / HTTPS Inspection / Proxy mode / Mobile Access / VoIP / Web Portals. The CoreXL layer passes the packet to one of the CoreXL FW instances to perform the processing. > This path is available only when CoreXL is enabled. > > 3. *Firewall path* / *Slow path* - The SecureXL device is unable to process the packet (refer to [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578)). The packet is passed on to the CoreXL layer and then to one of the CoreXL FW instances for full processing. > This path also processes all packets when SecureXL is disabled. > > > [![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/Packet_Flow.png)](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/Packet_Flow.png "Click the image to see it in full size in a new tab/window") > > **Default affinity settings for interfaces:** > > * If SecureXL is *enabled* - the default affinities of all interfaces are 'Automatic' - the affinity for each interface is automatically reset every 60 seconds, and balanced between available CPU cores based on the current load. > > * If SecureXL is *disabled* - the default affinities of all interfaces are with available CPU cores - those CPU cores that are ***not*** running a CoreXL FW instance or ***not*** defined as the affinity for a daemon. > > **Setting interface affinities:** > > * If SecureXL is *enabled* - the affinities of all interfaces are handled by the '`sim affinity`' command. Interface affinities are automatically distributed among CPU cores that are ***not*** running CoreXL FW instances and that are ***not*** set as the affinity for any daemon. > > **Note** : To configure affinity, refer to the section "(3-4) CoreXL". > > * If SecureXL is *disabled* - interface affinities are loaded at boot based on the `$FWDIR/conf/fwaffinity.conf` configuration file (if SecureXL is *enabled*, then lines beginning with "i" in this file are ignored). > > * To balance the load on the CPU cores that are running as SNDs (CPU cores that are ***not*** running CoreXL FW instances): > > 1. Check the amount of interrupts from each interface - run: > > *`[Expert@HostName]# cat /proc/interrupts`* > 2. Check, which CPU cores (that run as SNDs) are most utilized - run: > > *`[Expert@HostName]# top`* > > Note: Press digit 1 (above the letter Q) to display all CPU cores and press Shift+W to save this configuration. > 3. Assign the interfaces accordingly across the CPU cores that run as SNDs - run: > > *`[Expert@HostName]# sim affinity -s`* ### (3-6) Best practices - SMT (HyperThreading) {#Best practices - SMT (HyperThreading)} Show / Hide best practices information > **Notes:** > > * SMT is ***not*** recommended if only FireWall / IPSec VPN blades are used, because performance improvement by SMT is achieved on NGFW software blades. > > * SMT is ***not*** recommended for environments that have high memory utilization. > > > Reason: Firewall consumes memory for traffic inspection. If the memory utilization is already very high before enabling the SMT, the performance will decrease noticeably because SMT adds more CoreXL FW instances. > > > > To check the extent of memory utilization on the Security Gateway, refer to: > > * Initial diagnostics - Memory > > * Advanced diagnostics - Memory > * SMT is ***not*** recommended if these blades/features are enabled: > > * Data Loss Prevention blade > * Anti-Virus in Traditional Mode > * Using Services with Resources in Firewall policy > > > Reason: Each of these blades might have high memory consumption. These blades run Security Servers that are executed per CoreXL FW instance. Since SMT adds more CoreXL FW instances, overall memory consumption on Security Gateway might increase considerably. > > > > To check the extent of memory utilization on the Security Gateway, refer to: > > * Initial diagnostics - Memory > > * Advanced diagnostics - Memory > * If you have two ports on 12600/12700/12800, 13500/13800 and 21000 appliances to handle most of the traffic, it is also recommended to enable **Multi-Queue feature** to increase SMT performance (refer to [Performance Tuning Administration Guide](https://support.checkpoint.com/product/530) for your version - Chapter "Multi-Queue"). > > **Documentation:** > > * [sk93000 - SMT (HyperThreading) Feature Guide](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk93000) ### (3-7) Best practices - Multi-Queue {#Best practices - Multi-Queue} Show / Hide best practices information > **When Multi-Queue will not help:** > > * When most of the processing is done in CoreXL - either in the *Medium path* , or in the *Firewall path* (*Slow path*). > * All current CoreXL FW instances are highly loaded, so there are no CPU cores that can be reassigned to SecureXL. > * When IPS, or other deep inspection Software Blades are heavily used. > * When all network interface cards are processing the same amount of traffic. > * When all CPU cores that are currently used by SecureXL are congested. > * When trying to increase traffic session rate. > * When there is not enough diversity of traffic flows. In the extreme case of a single flow, for example, traffic will be handled only by a single CPU core. (Clarification: The more traffic is passing to/from different ports/IP addresses, the more you benefit from Multi-Queue. If there is a single traffic flow from a single Client to a single Server, then Multi-Queue will not help.) > > **Guidelines for configuring Multi-Queue:** > > * Network interface cards must support Multi-Queue. > * Multi-Queue is relevant ***only*** if SecureXL is enabled. > * Multi-Queue is relevant ***only*** if CoreXL is enabled. > * Examine the CPU affinity (for interfaces (SND) and for CoreXL FW instances). > * Examine the CPU utilization. > * Decide if more CPU cores can be allocated to the Secure Network Distributor (SND). > > **Multi-Queue is recommended when all these conditions occur:** > > * Load on CPU cores that run as SND is high (idle \< 20%). > * Load on CPU cores that run CoreXL FW instances is low (idle \> 50%). > * There are no CPU cores left to be assigned to the SND by changing interface affinity. > > **Number of active RX queues:** > > * By default on a Security Gateway, the number of active RX queues is calculated by: > `Number of Active RX queues = [Total Number of CPU cores] - [Number of CoreXL FW instances]` > > * By default on a VSX gateway, the number of active RX queues is calculated by: > `Number of Active RX queues = The lowest CPU ID, to which the `*fwk*` process on VSX is assigned` > > **Notes:** > > * For Multi-Queue enhancements in Gaia 3.10 refer to: [sk153373 -](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk153373)[Multi-Queue Management for Check Point R80.30 with Gaia 3.10 kernel](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk153373) > * For best performance, it is ***not*** recommended to assign both SND and a CoreXL FW instance to the same CPU core. > * Do ***not*** change the IRQ affinity of queues manually. Changing the IRQ affinity of the queues manually can adversely affect performance. > > **Documentation:** > > * [Performance Tuning Administration Guide](https://support.checkpoint.com/product/530) for your version - Chapter "Multi-Queue" ### (3-8) Best practices - Rule Base optimization {#Best practices - Rulebase optimization} Show / Hide best practices information > 1. Refer to [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578) to allow more connections to be accelerated by SecureXL. > > 2. In addition, you can provide a full Connections Table from your Security Gateway to [Check Point Support](http://www.checkpoint.com/services/contact/index.html) for thorough analysis (run the command '`fw tab -t connections -u > /var/log/Connections_Table.txt`'). > > In versions R82 and higher versions you can use the *connview* tool to collect the connections information. For more information refer to [R82 CLI Reference Guide](https://sc1.checkpoint.com/documents/R82/WebAdminGuides/EN/CP_R82_CLI_ReferenceGuide/Default.htm) \> Security Gateway Commands. ### (3-9) Best practices - IPS optimization {#Best practices - IPS optimization} Show / Hide best practices information > 1. There are different IPS protections with different confidence levels and performance impacts. > > 2. The default Optimize Profile for IPS contains the best setup in terms of performance and reliability of the detection rate. > > 3. Deviation from the Optimize Profile can cause a performance impact on the environment and false positive detection. > > 4. Avoid setting protections to run in "Detect" mode - it might increase CPU consumption (without increasing the security). > > 5. Identify the protections that consume most of the CPU resources - follow [sk43733 - How to measure CPU time consumed by IPS protections](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk43733). > > 6. Disable protections that are not needed in your environment. To check, which protections are needed, you may use vulnerability tools (such as Nessus). > > 7. Using null TP profiles and blade-based global exceptions to exclude certain traffic from IPS inspection completely can improve the Security Gateway performance. > **Important:** You must be aware of the security implications of this configuration. ### (3-10) Best practices - Application Control \& URL Filtering optimization {#Best practices - APPI URLF optimization} Show / Hide best practices information > **Application Control \& URL Filtering policy:** > > 1. Create as specific rules as possible, to prevent the unwanted traffic from hitting the wrong rule. > > Limit the scope of the rule by selecting only the relevant services (instead of the default '`Any`') - right-click on the column titles - click on the '`Service`' column: > [](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/APPI_URLF_Service_Column.png "Click the image to see it in full size in a new tab/window") > ![Application Control & URL Filtering](https://sc1.checkpoint.com/sc/SolutionsStatics/sk98348/sk98348_Application_Control_URL_Filtering_policy202501081005451.png) > 2. Avoid using "*Any* " in "*Source* " and "*Destination*" columns. > > 3. In the "*Destination* " column, use the default "*Internet*", unless a specific destination has to be explicitly selected (e.g., in case of internal servers). > > **Advanced - Engine Settings:** > > 1. Go to "*Check Point Online Web Service*" section. > > 2. In the "*Website categorization mode* " section, select "*Background*" (to prevent latency due to packet holding until categorization is completed). > > **Related solutions:** > > * [sk174045 - Wrong rule match on the first access to a URL/Website](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk174045) ### (3-11) Best practices - Anti-Virus \& Anti-Bot optimization {#Best practices - AV AB optimization} Show / Hide best practices information > **Anti-Virus \& Anti-Bot policy:** > > 1. Create as specific rules as possible, to prevent the unwanted traffic from hitting the wrong rule. > > 2. Avoid using "*Any* " in "*Source* " and "*Destination*" columns. > > 3. There is **no** need for "Clean Up" rule at the bottom (because the behavior of Anti-Virus \& Anti-Bot Blades is different from that of the Firewall Blade). > Such rule should only be used for short period of time and only for logging purposes. > > **Advanced - Engine Settings:** > > 1. Go to "*Check Point Online Web Service*" section. > > 2. In the "*Website categorization mode* " section, select "*Background*" (to prevent latency due to packet holding until categorization is completed). > > **Decreasing performance impact:** > > 1. Edit the Anti-Virus \& Anti-Bot profile: > > 1. Go to '`Profiles`' pane. > > 2. Select the relevant profile - click on '`Edit...`' button. > > 3. Go to '`General Properties`' pane. > > 4. In the '`Performance Impact`' field, select '`Low`'. > > 5. Click on '`OK`'. > > 6. Save the changes: go to '`File`' menu - click on '`Save`'. > > 7. Install the policy onto the relevant Security Gateway / Cluster object. > > 2. Exclude networks: > > Consider excluding networks, whose traffic does not have to be inspected - follow [sk92515 - How to configure Anti-Virus Exceptions](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk92515). > > Note: Standard exceptions are still being inspected (i.e., CPU is consumed). However, the traffic will be allowed. The exceptions per [sk92515](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk92515) are completely excluded from the Anti-Virus \& Anti-Bot engine inspection (i.e., decrease the load on CPU). > 3. Decrease the CPU consumption by RAD process and DLPU process: > > Consider disabling MD5 check and GZIP inspection - follow [sk94056 - When Anti-Virus and Anti-Bot blades are enabled, RAD process and DLPU process consume CPU at high level](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk94056). > 4. Disable scanning of archive file: > > Consider disabling '`Archive Scanning`' because it requires high amount of CPU resources: > 1. Go to '`Profiles`' pane. > > 2. Select the relevant profile - click on '`Edit...`' button. > > 3. Go to '`Anti-Virus Settings`' pane. > > 4. In the '`Archives`' section, uncheck the box '`Enable Archive scanning (impacts performance).`'. > > 5. Click on '`OK`'. > > 6. Save the changes: go to '`File`' menu - click on '`Save`'. > > 7. Install the policy onto the relevant Security Gateway / Cluster object. > > 5. Using a global exception. > > A blade-based global exception exclude traffic completely from Anti-Virus/Anti-Bot inspection and improve performance. Example for global exception: > > |-----|-----------------|--------|-------------|----------------------------|---------|----------|-------| > | No. | Protected Scope | Source | Destination | Protection/Site/File/Blade | Service | Action | Track | > | 1 | Any | Any | Any | Anti-Virus Anti-Bot | Any | Inactive | Log | *** ** * ** *** (4) Initial diagnostics {#Initial diagnostics} ---------------------------------------------- ### (4-1) Initial diagnostics - CPU {#Initial diagnostics - CPU} Show / Hide initial diagnostics information > 1. **`cpview`** > > Background: > * Displays the CPU utilization (and many other counters). Refer to [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878). > > Diagnostics: > * Monitor the CPU utilization during the problem > > Analysis: > * On the '`Overview`' tab, refer to '`CPU:`' section - look at counter '`Used`' > * On the '`I/S`' tab, go to '`CPU`' menu - go to '`Overview`' menu - refer to section '`CPU`' > 2. **`cpstat -f multi_cpu os`** > > Background: > * Displays internal statistics for OS about all CPU cores as collected by Check Point > > Diagnostics: > * Collect this output continuously during the problem > > Analysis: > * Look at all the columns > > *Example from machine with 8 CPU cores*: > > > > ``` > > Processors load > > --------------------------------------------------------------------------------- > > |CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec| > > --------------------------------------------------------------------------------- > > | 1| 1| 2| 96| 4| ?| 0| > > | 2| 2| 4| 94| 6| ?| 0| > > | 3| 1| 1| 98| 2| ?| 0| > > | 4| 2| 3| 94| 6| ?| 0| > > --------------------------------------------------------------------------------- > > ``` > > 3. **`top`** > > Background: > * Manual page - > * Displays dynamic real-time view of a running system on Linux > > Usage: > * When running the command for the first time: > > 1. Press '1' to display all CPU cores > 2. Press 'd' to set update interval - type 2 and press Enter > 3. Press 'n' to set maximum tasks displayed - type 15 and press Enter > 4. Press Shift+W to save the top's configuration > * When running the command for diagnostics: > > * Press Shift+P to sort the output by CPU utilization > * Press Shift+M to sort the output by Memory utilization > > Diagnostics: > * Collect this output continuously during the problem > > Analysis: > * Look at the load in "User Space" - counter *us* > High CPU consumption in "User Space" can be caused by processes that perform heavy tasks (e.g., too much logging by *fwd*, reloading the configuration during policy installation, etc.) > * Look at the load in "System (kernel) Space" - counter *sy* > High CPU consumption in "System (kernel) Space" can be caused by heavy tasks (e.g., deep inspection of packets, enabling of all blades, enabling of all IPS protections in *Prevent* mode, etc.) > * Look at the amount of "Idle" - counter *id* > The more CPU is idle, the better the machine's performance is > * Look at the amount of "I/O waiting" - counter *wa* > High amount of "I/O waiting" is caused by heavy reading from/writing to hard disk (e.g., during policy installation, heavy logging, insufficient RAM, etc.) > * Look at the amount of "SoftIRQ" - counter *si* > High amount of "SoftIRQ" is usually caused by high amount of traffic > * Look at the amount of "Hardware IRQs" - counter *hi* > * Look at the CPU consumption by the processes - column *%CPU* > Constant high CPU consumption by a process can be caused by numerous factors - function stack should be collected from the process using a [special Check Point shell script](https://supportcenter.checkpoint.com/supportcenter/portal/role/supportcenterUser/page/default.psml/media-type/html?action=portlets.DCFileAction&eventSubmit_doGetdcdetails=&fileid=32051) ('`pstack`') - refer to "(5-1) Advanced diagnostics - CPU" section > **Note** - You must be logged in to access the `pstack `script. > Related solution: [sk116740 - "top" command on VSX Gateway shows that FWK process consumes CPU at more than 100%](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk116740). > > *Example from machine with 2 CPU cores*: > > > > ``` > > top - 15:18:47 up 14:06, 1 user, load average: 0.55, 0.37, 0.35 > > Tasks: 213 total, 1 running, 212 sleeping, 0 stopped, 0 zombie > > Cpu0 : 0.7%us, 0.7%sy, 0.0%ni, 97.3%id, 0.0%wa, 0.0%hi, 1.4%si, 0.0%st > > Cpu1 : 0.7%us, 0.7%sy, 0.0%ni, 97.3%id, 0.0%wa, 0.0%hi, 1.4%si, 0.0%st > > Cpu2 : 0.0%us, 0.7%sy, 0.0%ni, 98.6%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st > > Cpu3 : 2.0%us, 3.4%sy, 0.0%ni, 93.3%id, 0.0%wa, 0.0%hi, 1.3%si, 0.0%st > > Mem: 2005908k total, 1907412k used, 98496k free, 103952k buffers > > Swap: 4225084k total, 323268k used, 3901816k free, 198288k cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 16948 admin 24 0 71692 10m 4744 S 2 0.6 15:14.00 DAService > > 4103 admin 15 0 0 0 0 S 1 0.0 8:38.37 fw_worker_0 > > 4105 admin 15 0 0 0 0 S 1 0.0 10:46.19 fw_worker_2 > > 4104 admin 15 0 0 0 0 S 1 0.0 4:05.49 fw_worker_1 > > 1 admin 15 0 2040 720 624 S 0 0.0 0:01.44 init > > 2 admin RT -5 0 0 0 S 0 0.0 0:01.82 migration/0 > > 3 admin 15 0 0 0 0 S 0 0.0 0:00.02 ksoftirqd/0 > > ... ... ... > > ``` > > 4. **`ps auxwwwf`** > > Background: > * Manual page - > * Displays information about the current processes (daemons) > > Usage: > * Refer to the manual page > * By default, output is sorted by *PID* column > * Simplest sorting can be done by pipelining the output of *`ps`* command to *`sort`* command - for example, to sort the CPU usage, run: > *`[Expert@HostName]# ps auxww | sort -nrk 3`* > > Diagnostics: > * Collect this output continuously during the problem > > Analysis: > * Look at the amount of "*CPU* ", "*MEM* ", "*VSZ* ", "*RSS* ", "*TIME*" consumed by the daemons > * Constant high CPU consumption can be caused by numerous factors - function stack should be collected from the process using a [special Check Point shell script](https://supportcenter.checkpoint.com/supportcenter/portal/role/supportcenterUser/page/default.psml/media-type/html?action=portlets.DCFileAction&eventSubmit_doGetdcdetails=&fileid=32051) ('`pstack`') - refer to "(5-1) Advanced diagnostics - CPU" section > > *Example* (excerpt): > > > > ``` > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > admin 1 0.0 0.0 2040 720 ? Ss 01:12 0:01 init [3] > > admin 2 0.0 0.0 0 0 ? S< 01:12 0:01 [migration/0] > > admin 3 0.0 0.0 0 0 ? S 01:12 0:00 [ksoftirqd/0] > > admin 4 0.0 0.0 0 0 ? S< 01:12 0:00 [watchdog/0] > > .................. > > admin 4103 1.0 0.0 0 0 ? S 01:13 8:41 [fw_worker_0] > > admin 4104 0.4 0.0 0 0 ? S 01:13 4:07 [fw_worker_1] > > admin 4105 1.2 0.0 0 0 ? S 01:13 10:50 [fw_worker_2] > > admin 4115 0.0 0.2 13976 4184 ? Ss 01:13 0:02 /opt/CPshrd-R77/bin/cpwd > > admin 5098 0.2 2.4 342988 49860 ? Ssl 01:13 2:07 \_ cpd > > admin 5136 0.0 0.3 153612 6216 ? Ss 01:13 0:00 \_ mpdaemon /opt/CPshrd-R77/log/ > > .................. > > admin 5374 0.1 5.6 610672 113844 ? Ssl 01:13 1:32 \_ fwd > > admin 5449 0.0 0.4 27856 9232 ? S 01:13 0:00 | \_ cpca > > admin 5720 0.0 2.0 207256 41100 ? S 01:13 0:01 | \_ syslog 514 all > > admin 6030 0.0 2.7 295536 54336 ? Sl 01:14 0:06 | \_ vpnd 0 > > admin 6033 0.0 2.5 239748 52012 ? Sl 01:14 0:15 | \_ pdpd 0 -t > > admin 6034 0.0 2.5 211336 50924 ? S 01:14 0:11 | \_ pepd 0 -t > > admin 6036 0.0 2.1 233164 43804 ? Sl 01:14 0:02 | \_ fwpushd 0 > > admin 6038 0.0 0.6 35016 13232 ? S 01:14 0:00 | \_ wstlsd 0 0 > > .................. > > admin 5382 0.5 7.5 453272 151396 ? Ssl 01:13 4:18 \_ fwm > > admin 5390 0.0 1.0 46240 21504 ? Ss 01:13 0:00 \_ status_proxy > > admin 5750 0.0 0.8 35228 17264 ? Ss 01:13 0:01 \_ rad > > admin 5827 0.0 1.1 42500 22748 ? Ss 01:13 0:00 \_ cpstat_monitor > > admin 5996 0.0 0.5 57176 11516 ? Ssl 01:14 0:02 \_ fwucd > > admin 6022 0.0 1.0 132424 21652 ? Ssl 01:14 0:03 \_ dlpu -i 0 0 > > .................. > > admin 4636 0.0 0.3 25020 7784 ? Ss 01:13 0:00 /bin/pm > > admin 4653 0.1 0.5 31620 11908 ? Ss 01:13 1:02 \_ /bin/confd > > admin 4654 0.1 0.7 38040 14312 ? SNsl 01:13 0:57 \_ /bin/searchd -niceboost 10 > > admin 4656 0.0 0.4 99740 9880 ? Ssl 01:13 0:01 \_ /bin/rconfd /etc/actions_mapping.xml > > admin 4657 0.0 0.1 6472 3824 ? Ss 01:13 0:18 \_ /bin/monitord > > admin 4683 0.0 0.3 25252 7168 ? Ss 01:13 0:00 \_ /bin/cloningd > > admin 4685 0.0 0.1 10048 2348 pts/0 Ss+ 01:13 0:00 \_ /bin/clishd default server > > admin 4687 0.0 0.1 19032 2796 ? Ssl 01:13 0:00 \_ /bin/clish -p > > admin 4688 0.0 0.0 2948 868 ? Ss 01:13 0:00 \_ /usr/bin/tclsh /usr/libexec/netflowd > > admin 4689 0.0 0.3 30360 7368 ? Ss 01:13 0:07 \_ /usr/sbin/snmpd -f -c /etc/snmp/userDefinedSettings.conf > > admin 5041 0.0 0.3 32768 6912 ? Ss 01:13 0:00 \_ /bin/routed -N > > admin 5047 0.0 0.3 33128 7820 ? S 01:13 0:00 | \_ /bin/routed -i default -f /etc/routed0.conf -h 0 > > .................. > > admin 9438 0.0 0.0 1652 496 tty1 Ss+ 01:14 0:00 /sbin/agetty 9600 tty1 > > admin 9439 0.0 0.0 1648 500 tty2 Ss+ 01:14 0:00 /sbin/agetty 9600 tty2 > > admin 9440 0.0 0.0 1652 504 tty3 Ss+ 01:14 0:00 /sbin/agetty 9600 tty3 > > admin 9441 0.0 0.0 2416 1052 ? Ss 01:14 0:00 /bin/bash /bin/console_agetty > > admin 9463 0.0 0.0 1652 508 tty4 Ss+ 01:14 0:00 \_ /sbin/agetty 9600 tty4 vt100 > > admin 16948 2.1 0.5 71692 11072 ? Sl 03:15 15:21 /opt/CPda/bin/DAService > > ``` > > 5. **`cat /proc/interrupts`** > > Background: > * Manual page - > * Displays the number of interrupts on each CPU core from each IRQ > > Usage: > * Refer to the manual page > * By default, output is sorted by IRQ number > * The only relevant devices are network interfaces > * To shorten the output, use the '`grep`' command - for example, run: > *`[Expert@HostName]# cat /proc/interrupts | grep -E "CPU|eth"`* > * To collect this output continuously, use the '`watch`' command - for example, run: > *`[Expert@HostName]# watch -d -n 1 'cat /proc/interrupts | grep -E "CPU|eth"'`* > > Diagnostics: > * Collect this output continuously during the problem > > Analysis: > * Look at the general trend - which CPU receives more interrupts and from which interfaces > * If some CPU cores receive more interrupts than others, then affinity of interfaces to CPU cores should be optimized - interface should be redistributed better > > *Example from machine with 4 CPU cores and 4 interfaces*: > > > > ``` > > CPU0 CPU1 CPU2 CPU3 > > 0: 51254849 0 0 0 IO-APIC-edge timer > > 1: 4 0 0 5 IO-APIC-edge i8042 > > 6: 2 3 0 0 IO-APIC-edge floppy > > 7: 0 0 0 0 IO-APIC-edge parport0 > > 8: 3 0 0 0 IO-APIC-edge rtc > > 9: 0 0 0 0 IO-APIC-level acpi > > 12: 0 0 0 115 IO-APIC-edge i8042 > > 15: 302 1043 168 1127 IO-APIC-edge ide1 > > 59: 320145 6732 5944 6189 IO-APIC-level ioc0, eth3 > > 67: 15765753 0 0 0 IO-APIC-level eth0 > > 75: 24271 1285 36 0 IO-APIC-level eth1 > > 83: 24272 0 1322 0 IO-APIC-level eth2 > > NMI: 0 0 0 0 > > LOC: 50950084 50945459 50949418 50945128 > > ERR: 0 > > MIS: 0 > > ``` > > 6. **`cat /proc/cpuinfo`** > > Background: > * Manual page - > * Displays a collection of CPU and system architecture dependent items about CPU (on multi-CPU (SMP) machines will show information for each CPU) > > Diagnostics: > * Collect this output to see the information about CPU (architecture, vendor, number) > > Analysis: > * Look at the processor number > * Look at the processor model > * Look at the CPU clock frequency > * Look at the supported flags (e.g., *`pae`*) > > *Example* (excerpt): > > > > ``` > > processor : 0 > > vendor_id : GenuineIntel > > cpu family : 6 > > model : 44 > > model name : Intel(R) Xeon(R) CPU E5645 @ 2.40GHz > > stepping : 2 > > cpu MHz : 2399.483 > > cache size : 12288 KB > > physical id : 0 > > siblings : 2 > > core id : 0 > > cpu cores : 2 > > fdiv_bug : no > > hlt_bug : no > > f00f_bug : no > > coma_bug : no > > fpu : yes > > fpu_exception : yes > > cpuid level : 11 > > wp : yes > > flags : fpu vme de pse tsc msr pae mce ... > > bogomips : 4806.12 > > ``` > > 7. **`dmesg`** > > Background: > * Manual page - > * Displays the Linux kernel ring buffer - bootup messages from various kernel modules and hardware components > > Diagnostics: > * Collect this output to search for errors from various kernel modules and hardware components > > Analysis: > * Check each line for kernel boot parameters (e.g., *vmalloc*), errors, failures, components/features being disabled ### (4-2) Initial diagnostics - Memory {#Initial diagnostics - Memory} Show / Hide initial diagnostics information > Related solution: [sk115072 - How to determine amount of installed RAM on Check Point Appliance](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk115072). > > 1. **`cpview`** > > Background: > * Displays the Memory utilization (and many other counters). Refer to [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878). > > Diagnostics: > * Monitor the Memory utilization during the problem > > Analysis: > * On the '`Overview`' tab, refer to '`Memory:`' section - look at counters '`Free MB`' > * On the '`I/S`' tab, go to '`Memory`' menu - go to '`FW-Kernel`' menu - refer to section '`Firewall kernel memory usage summary:`' - look at counter '`%Usage`' > 2. **`fw ctl pstat`** > > Background: > * Displays FireWall internal statistics about memory and traffic > > Diagnostics: > * Collect the output *before* and *after* the suspected problem > * Use different flags to get more data: > * for HMEM: *`fw ctl pstat -h`* > * for SMEM: *`fw ctl pstat -s`* > * for KMEM: *`fw ctl pstat -k`* > * for Handles (kbufs): *`fw ctl pstat -l`* > * Counters are reset when Check Point Services are stopped (with '`cpstop`' command) > * Under memory, '`allocations`' counter always grows, may wrap around > > Analysis: > * No single field that indicates a problem - need to interpret all counters together > * HMEM > * failures under HMEM - no real memory problem, just mean HMEM is full ; HMEM should have been configured larger > * "*`failed allocations`*" under HMEM (only) do not indicate any problem > * SMEM > * failures under SMEM - reached Check Point memory limit , exhausted OS memory, large non-sleep allocation , indicate some shortage > * "*`failed allocations`*" under SMEM may not mean that a user's allocation failed, maybe HMEM extension failed > * "*`failed free`*" under SMEM means an overrun or freeing an invalid pointer - indicates a bug > * KMEM > * failures under KMEM - application asked for memory and could not get it , usually, it is a memory problem > * "*`failed allocations`*" under KMEM means that the application didn't get memory > > *Example*: > > > > ``` > > System Capacity Summary: > > Memory used: 12% (183 MB out of 1515 MB) - below watermark > > Concurrent Connections: 79 (Unlimited) > > Aggressive Aging is not active > > > > Hash kernel memory (hmem) statistics: > > Total memory allocated: 155189248 bytes in 37888 (4096 bytes) blocks using 37 pools > > Total memory bytes used: 33563132 unused: 121626116 (78.37%) peak: 53675664 > > Total memory blocks used: 9082 unused: 28806 (76%) peak: 13407 > > Allocations: 96170689 alloc, 0 failed alloc, 95799050 free > > > > System kernel memory (smem) statistics: > > Total memory bytes used: 280891432 peak: 297288060 > > Total memory bytes wasted: 12156438 > > Blocking memory bytes used: 902804 peak: 933368 > > Non-Blocking memory bytes used: 279988628 peak: 296354692 > > Allocations: 10526 alloc, 1 failed alloc, 8437 free, 0 failed free > > vmalloc bytes used: 7340032 expensive: yes > > > > Kernel memory (kmem) statistics: > > Total memory bytes used: 159039668 peak: 185833572 > > Allocations: 96179738 alloc, 1 failed alloc > > 95807459 free, 0 failed free > > External Allocations: 0 for packets, 63652 for SXL > > > > Cookies: > > 17293941 total, 0 alloc, 0 free, > > 19087 dup, 17386724 get, 117090 put, > > 32037742 len, 0 cached len, 0 chain alloc, > > 0 chain free > > > > Connections: > > 82980 total, 31766 TCP, 50795 UDP, 7 ICMP, > > 412 other, 0 anticipated, 0 recovered, 79 concurrent, > > 568 peak concurrent > > > > Fragments: > > 0 fragments, 0 packets, 0 expired, 0 short, > > 0 large, 0 duplicates, 0 failures > > > > NAT: > > 0/0 forw, 0/0 bckw, 0 tcpudp, > > 0 icmp, 0-0 alloc > > > > Sync: off > > ``` > > 3. **`cpstat -f memory os`** > > Background: > * Displays internal statistics for OS about memory as collected by Check Point > > Diagnostics: > * Collect this output continuously during the problem (usually, over some time period) > > Analysis: > * High memory consumption can be caused by high amount of traffic > * Constant increase in memory consumption might suggest some memory leak - follow [sk35496 - How to detect a memory leak on Security Gateway with SecurePlatform OS / Gaia OS](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk35496) > > *Example*: > > > > ``` > > Total Virtual Memory (Bytes): 6380535808 > > Active Virtual Memory (Bytes): 1882390528 > > Total Real Memory (Bytes): 2054049792 > > Active Real Memory (Bytes): 1643986944 > > Free Real Memory (Bytes): 410062848 > > Memory Swaps/Sec: - > > Memory To Disk Transfers/Sec: - > > ``` > > 4. **`cat /proc/meminfo`** > > Background: > * Manual page - > * Displays the amount of free and used memory (both physical and swap) on the system as well as the shared memory and buffers used by the kernel > > Diagnostics: > * Collect this output continuously during the problem (usually, over some time period) > > Analysis: > * Sum up these counters: *`MemFree`* + *`Buffers`* + *`Cached`* , and compare the result with counter *`MemTotal`* - follow [sk42717 - How to read the output of 'cat /proc/meminfo' on Linux-based system](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk42717) > * Constant increase in memory consumption might suggest some memory leak - follow [sk35496 - How to detect a memory leak on Security Gateway with SecurePlatform OS / Gaia OS](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk35496) > > *Example*: > > > > ``` > > MemTotal: 2005908 kB > > RawMemTotal: 2097152 kB > > MemFree: 95092 kB > > Buffers: 107136 kB > > Cached: 198520 kB > > SwapCached: 90452 kB > > Active: 1335364 kB > > Inactive: 121220 kB > > HighTotal: 262016 kB > > HighFree: 808 kB > > LowTotal: 1743892 kB > > LowFree: 94284 kB > > SwapTotal: 4225084 kB > > SwapFree: 3901816 kB > > Dirty: 864 kB > > Writeback: 0 kB > > AnonPages: 1137312 kB > > Mapped: 108660 kB > > Slab: 82164 kB > > PageTables: 19132 kB > > NFS_Unstable: 0 kB > > Bounce: 0 kB > > CommitLimit: 5228036 kB > > Committed_AS: 4025512 kB > > VmallocTotal: 247800 kB > > VmallocUsed: 114732 kB > > VmallocChunk: 117952 kB > > HugePages_Total: 0 > > HugePages_Free: 0 > > HugePages_Rsvd: 0 > > Hugepagesize: 2048 kB > > ``` > > 5. **`dmesg`** > > Background: > * Manual page - > * Displays the Linux kernel ring buffer - bootup messages from various kernel modules and hardware components > > Diagnostics: > * Collect this output to search for errors from various kernel modules and hardware components > > Analysis: > * Check each line for kernel boot parameters (e.g., *vmalloc*), errors, failures, components/features being disabled ### (4-3) Initial diagnostics - Network interface cards {#Initial diagnostics - NICs} Show / Hide initial diagnostics information > 1. **`cpview`** > > Background: > * Displays the traffic and interfaces statistics (and many other counters). Refer to [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878). > > Diagnostics: > * Monitor the traffic and interfaces during the problem > > Analysis: > * On the '`Overview`' tab, refer to '`Traffic counters:`' section > * On the '`SysInfo`' tab, refer to '`Hardware Information:`' section > * On the '`Traffic`' tab, go to '`Overview`' menu - refer to '`Traffic Rate:`' section > * On the '`Traffic`' tab, go to '`Overview`' menu - refer to '`Concurrent Connections:`' section > * On the '`Traffic`' tab, go to '`Overview`' menu - refer to '`Interface drops:`' section > * On the '`Traffic`' tab, go to '`Interfaces`' menu - refer to '`Errors and Drops:`' section > 2. **`netstat -ni`** > > Background: > * Manual page - > * Displays a table of all network interfaces > > Diagnostics: > * Collect this output continuously during the problem (usually, over some time period) > > Analysis: > * *RX-OK* - number of packets that have been received error-free > * *RX-ERR* - number of packets that have been received damaged (refer to [sk61922](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk61922)) > * *RX-DRP* - number of received packets that have been dropped (refer to [sk61922](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk61922)) > * *RX-OVR* - number of received packets that have been lost because of an overrun (number of times the receiver hardware was unable to hand received data to a hardware buffer - the internal FIFO buffer of the chip is full, but is still tries to handle incoming traffic ; most likely, the input rate of traffic exceeded the ability of the receiver to handle the data) > * *TX-OK* - number of packets that have been transmitted error-free > * *TX-ERR* - number of packets that have been transmitted damaged > * *TX-DRP* - number of transmitted packets that have been dropped > * *TX-OVR* - number of transmitted packets that have been lost because of an overrun > * *Flg* - shows the flags that have been set for this interface - these characters are one-character versions of the long flag names that are displayed in the output of '*ifconfig* ' command: > * *A* = this interface will receive all Multicast addresses > * *B* = a Broadcast address has been set > * *D* = debugging is turned on > * *L* = this interface is a loopback device > * *M* = all packets are received (promiscuous mode) > * *m* = master > * *N* = trailers are avoided > * *O* = ARP is turned off for this interface > * *P* = this is a Point-to-Point connection > * *R* = interface is running > * *s* = slave > * *U* = interface is up > > *Example*: > > > > ``` > > Kernel Interface table > > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg > > br0 1500 0 4384062 0 0 0 28 0 0 0 BMRU > > eth0 1500 0 12505484 0 0 0 20359171 0 0 0 BMRU > > eth1 1500 0 11882710 0 62 0 449677832 0 0 0 BMRU > > eth2 1500 0 2592575900 362803 3626586 0 1158425513 0 0 0 BMRU > > eth3 1500 0 1189952933 30 9382997 0 2575575506 0 0 0 BMRU > > lo 16436 0 174524 0 0 0 174524 0 0 0 LRU > > ``` > > 3. **`ifconfig `*name_of_interface*** > > Background: > * Manual page - > * Displays the status and traffic statistics for the currently active interfaces > > Diagnostics: > * Collect this output continuously during the problem (usually, over some time period) > > Analysis: > * *UP* - indicates that the kernel modules related to the Ethernet interface has been loaded > * *BROADCAST* - indicates that interface supports broadcasting - a necessary characteristic to obtain IP address via DHCP > * *NOTRAILERS* - indicates that trailer encapsulation is disabled (Linux usually ignore trailer encapsulation so this value has no effect at all) > * *RUNNING* - indicates that interface is ready to accept data > * *MULTICAST* - indicates that the interface supports multicasting > * *MTU* - Maximum Transmission Unit is the size of each packet received by the interface (default value is 1500; setting MTU to a higher value could hazard packet fragmentation or buffer overflows) > * *Metric* - used to compute the cost of a route - it tells the OS, to which interface a packet should be forwarded, when multiple interfaces could be used to reach the packet's destinationTakes values of 0,1,2,3,... The lower the value, the more leverage it has. This parameter has significance only while routing packets. For example, if you have two Ethernet cards and you want to forcibly make your machine use one card over the other in sending the data. Then you can set the Metric value of the Ethernet card which you favor lower than that of the other Ethernet card. > * *RX packets* - total number of packets received via the interface > * *RX errors* - number of damaged packets received (refer to [sk61922](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk61922)) > * *RX dropped* - number of dropped packets due to reception errors (refer to [sk61922](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk61922)) > * *RX overruns* - number of received packets that experienced data overruns (number of times the receiver hardware was unable to hand received data to a hardware buffer) > * *RX frame* - number received packets that experienced frame errors > * *TX packets* - total number of packets transmitted via the interface > * *TX errors* - number of packets that experienced transmission error > * *TX dropped* - number of dropped transmitted packets due to transmission errors > * *TX overruns* - number of transmitted packets that experienced data overruns (number of times the transmitter hardware was unable to hand received data to a hardware buffer) > * *TX carriers* - number received packets that experienced loss of carriers > * *TX collisions*- number of transmitted packets that experienced Ethernet collisions (a nonzero value of this field indicates possibility of network congestion) > * *TX txqueuelen* - configured length of transmission queue > * *RX bytes* - total bytes received over this interface > * *TX bytes* - total total bytes transmitted over this interface > > *Example*: > > > > ``` > > eth0 Link encap:Ethernet HWaddr 00:50:56:AA:69:33 > > inet addr:172.30.41.90 Bcast:172.30.255.255 Mask:255.255.0.0 > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:21363169 errors:0 dropped:0 overruns:0 frame:0 > > TX packets:128400 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:1608073037 (1.4 GiB) TX bytes:33703584 (32.1 MiB) > > ``` > > 4. **`netstat -anp`** > > Background: > * Manual page - > * Displays both listening and non-listening (for TCP this means established connections) sockets > > Diagnostics: > * Collect this output continuously during the problem (usually, over some time period) > > Analysis: > * Under "*`Active Internet connections`* " look at "*`Recv-Q`* " and at "*`Send-Q`*" > * *Recv-Q* - data (in bytes), which has not yet been pulled from the socket buffer by the application (value should be as close to 0 as possible) > * *Send-Q* - data (in bytes), which the sending application has given to the transport, but has yet to be ACKnowledged by the receiving TCP (value should be as close to 0 as possible - a large number may indicate a network bottleneck) > > *Example*: > > > > ``` > > [Expert@FW]# netstat -anp > > Active Internet connections (servers and established) > > Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name > > tcp 0 2368 172.30.20.69:22 172.30.20.228:3676 ESTABLISHED 16179/0 > > udp 256956 0 0.0.0.0:9282 0.0.0.0:* > > [Expert@FW]# > > ``` > > 5. **`ethtool `*name_of_interface*** > > Background: > * Manual page - > * Displays Ethernet card settings > > Diagnostics: > * Collect this output to check the current status > > Analysis: > * Check every line of the output > * Configuration on both ends of the cable should be identical (speed /duplex / auto-negotiation) > > *Example*: > > > > ``` > > Settings for eth0: > > Supported ports: [ TP ] > > Supported link modes: 10baseT/Half 10baseT/Full > > 100baseT/Half 100baseT/Full > > 1000baseT/Full > > Supports auto-negotiation: Yes > > Advertised link modes: 10baseT/Half 10baseT/Full > > 100baseT/Half 100baseT/Full > > 1000baseT/Full > > Advertised auto-negotiation: Yes > > Speed: 1000Mb/s > > Duplex: Full > > Port: Twisted Pair > > PHYAD: 1 > > Transceiver: internal > > Auto-negotiation: on > > Supports Wake-on: d > > Wake-on: d > > Current message level: 0x00000007 (7) > > Link detected: yes > > ``` > > 6. **`ethtool -i `*name_of-interface*** > > Background: > * Manual page - > * Displays information about associated driver > > Diagnostics: > * Collect this output to check the current status > > Analysis: > * Use driver with [NAPI](http://en.wikipedia.org/wiki/NAPI) > * Use the latest version of the driver > > *Example*: > > > > ``` > > driver: e1000 > > version: 7.6.15.5-NAPI > > firmware-version: N/A > > bus-info: 0000:02:00.0 > > ``` > > 7. **`ethtool -g `*name_of-interface*** > > Background: > * Manual page - > * Displays information about RX/TX ring parameters > > Diagnostics: > * Collect this output to check the current status > > Analysis: > * Check the current size of the buffers versus the maximum allowed - follow [sk42181 - How to increase sizes of buffer on SecurePlatform/Gaia for Intel NIC and Broadcom NIC](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk42181) > > *Example*: > > > > ``` > > Ring parameters for eth0: > > Pre-set maximums: > > RX: 4096 > > RX Mini: 0 > > RX Jumbo: 0 > > TX: 4096 > > Current hardware settings: > > RX: 256 > > RX Mini: 0 > > RX Jumbo: 0 > > TX: 1024 > > ``` > > 8. **`arp -an | wc -l`** > > Background: > * Manual page - > * Displays the system's ARP cache table > > Diagnostics: > * Collect this output to check the current status > > Analysis: > * Check the number of entries in the ARP table > * Refer to [sk43772 - 'kernel: neighbour table overflow' appears repeatedly in /var/log/messages files](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk43772) > * Refer to [sk92372 - How to change the size of IPv6 Neighbors cache table](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk92372) > > *Example*: > > > > ``` > > [Expert@HostName]# arp -an | wc -l > > 1 > > [Expert@HostName]# arp -an > > ? (172.30.1.1) at 00:1C:EB:1C:B4:43 [ether] on eth0 > > [Expert@HostName]# arp -en > > Address HWtype HWaddress Flags Mask Iface > > 172.30.1.1 ether 00:1C:EB:1C:B4:43 C eth0 > > [Expert@HostName]# > > ``` > > 9. **`dmesg`** > > Background: > * Manual page - > * Displays the Linux kernel ring buffer - bootup messages from various kernel modules and hardware components > > Diagnostics: > * Collect this output to search for errors from various kernel modules and hardware components > > Analysis: > * Check each line for kernel boot parameters (e.g., *vmalloc*), errors, failures, components/features being disabled ### (4-4) Initial diagnostics - SecureXL {#Initial diagnostics - SecureXL} Show / Hide initial diagnostics information > 1. **`cpview`** > > Background: > * Displays the SecureXL status and statistics (and many other counters). Refer to [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878). > > Diagnostics: > * Monitor the SecureXL performance during the problem > > Analysis: > * On the '`SysInfo`' tab, refer to '`Configuration Information:`' section - look at '`PPack Status`' > * On the '`Traffic`' tab, go to '`Overview`' menu - refer to section '`Templates:`' > * On the '`I/S`' tab, go to '`SXL`' menu - go to '`Overview`' menu > 2. **`fwaccel stat`** > > Background: > * Displays the status of SecureXL and SecureXL Templates > > Diagnostics: > * Collect this output to check the current status > > Analysis: > * "*Accelerator Status* " should be "*on*" > * "*Accept Templates* " should be "*enabled*" > * If "*Accept Templates* " are "*disabled* " from a specific rule, then optimize the rulebase - follow [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578) > * [](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk66402)Status of "*NAT Templates* " is determined by [sk71200 - SecureXL NAT Templates](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk71200) > * "*Cryptography Features*" section appears only if SecureXL device supports cryptography, or if a VPN Accelerator card is installed on the machine. > > *Example*: > > > > ``` > > Accelerator Status : on > > Accept Templates : disabled by Firewall > > disabled from rule #31 > > Drop Templates : disabled > > NAT Templates : disabled by user > > > > Accelerator Features : Accounting, NAT, Cryptography, Routing, > > HasClock, Templates, Synchronous, IdleDetection, > > Sequencing, TcpStateDetect, AutoExpire, > > DelayedNotif, TcpStateDetectV2, CPLS, WireMode, > > DropTemplates, NatTemplates, Streaming, > > MultiFW, AntiSpoofing, ViolationStats, > > Nac, AsychronicNotif, ERDOS > > Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL, > > 3DES, DES, CAST, CAST-40, AES-128, AES-256, > > ESP, LinkSelection, DynamicVPN, NatTraversal, > > EncRouting, AES-XCBC, SHA256 > > ``` > > 3. **`fwaccel stats -s`** > > Background: > * Displays summary of SecureXL acceleration statistics > > Diagnostics: > * Collect this output continuously during the problem > > Analysis: > * If the ratio of "*Accelerated/Total*" is less than 80%, it means that acceleration is poor, and the optimization is required > * The higher the ratio of "*F2Fed pkts/Total pkts* " is, the more traffic is Forwarded to Firewall for inspection (the poorer the performance is) - see the *`fwaccel conns`* command below > * The higher the ratio of "*PXL pkts/Total pkts* " is, the more packets passed in *Medium path*. Meaning, packets were handled by the SecureXL device, except for IPS / Application Control / Anti-Virus / Anti-Bot processing. The CoreXL layer passed the packets to one of the CoreXL FW instances to perform the processing. This path is available only when CoreXL is enabled. > * The higher the ratio of "*QXL pkts/Total pkts*" is, the more packets were processed by QoS. > > *Example*: > > > > ``` > > Accelerated conns/Total conns : 364/13215 (2%) > > Delayed conns/(Accelerated conns + PXL conns) : 48/12023 (0%) > > Accelerated pkts/Total pkts : 18252/564927 (3%) > > F2Fed pkts/Total pkts : 36776/564927 (6%) > > PXL pkts/Total pkts : 509899/564927 (90%) > > QXL pkts/Total pkts : 0/564927 (0%) > > ``` > > 4. **`fwaccel conns`** > > Background: > * Displays entries from SecureXL connections table > > Diagnostics: > * Collect this output continuously during the problem (**Important Note:** This command consumes high amount of memory) > > Analysis: > * Look at the *Flags* column: > * *U* = Unidirectional - the connection can pass data on either C2S or S2C - data packets from the opposite direction will be F2F'ed > * *N* = NAT is being performed on the connection by the device > * *A* = Accounting is performed on the connection (the connection is viewed by either rulebase accounting or SmartView Monitor) > * *C* = Encryption is done on the connection by the device > * *W* = the connection is in wire mode > * *P* = Partial (versions R70 and higher) > * *S* = Streaming - PXL (versions R70 and higher) > * *F* : > * In R81 and higher = Fast accelerated connection. > > *Example* (all IP addresses were replaced by "X"): > > > > ``` > > Source SPort Destination DPort PR Flags C2S i/f S2C i/f > > ------------- ----- -------------- ----- -- ------- --------- --------- > > X.X.X.X 61242 X.X.X.X 80 6 ..N.... eth5/eth0 eth0/eth5 > > X.X.X.X 6000 X.X.X.X 3842 6 ....... eth0/eth4 eth4/eth0 > > X.X.X.X 1620 X.X.X.X 88 17 ....... eth4/eth2 eth2/eth4 > > X.X.X.X 50829 X.X.X.X 80 6 ..N.... eth5/eth0 eth0/eth5 > > X.X.X.X 4285 X.X.X.X 80 6 F.N.... eth5/eth0 eth0/eth5 > > X.X.X.X 80 X.X.X.X 49312 6 F.N.... eth5/eth0 eth0/eth5 > > X.X.X.X 80 X.X.X.X 11450 6 ..N.... eth5/eth0 eth0/eth5 > > X.X.X.X 58562 X.X.X.X 80 6 F.N.... eth5/eth0 eth0/eth5 > > X.X.X.X 161 X.X.X.X 5002 17 F...... eth2/eth2 -/- > > X.X.X.X 21891 X.X.X.X 0 1 F...... eth2/eth4 eth4/eth2 > > X.X.X.X 34303 X.X.X.X 389 6 F.N.... eth2/eth2 eth2/- > > ``` > > * **`fwaccel templates`** > > Background: > * Displays SecureXL Connection Templates > Note: Size of Templates table (*cphwd_tmpl* , id 8111) is limited to 1/4 of the size of Firewall Connections Table (*connections*, id 8158). > > Diagnostics: > * Collect this output continuously during the problem (**Important Note:** This command consumes high amount of memory) > > Analysis: > * Check whether templates are created: > *`[Expert@HostName]# fwaccel templates -s`* > * Check whether templates were created for relevant connections (search for relevant IP addresses using the *`grep`* command) > * Look at the *Flags* column: > * You should not see connections with *F* , *N* , or *C* flags > * *F* = Forward to Firewall - the connection is not accelerated > * *U* = Unidirectional - the connection can pass data on either C2S or S2C - data packets from the opposite direction will be F2F'ed > * *N* = NAT is being performed on the connection by the device > * *A* = Accounting is performed on the connection (the connection is viewed by either rulebase accounting or SmartView Monitor) > * *C* = Encryption is done on the connection by the device > * *W* = the connection is in wire mode > * *P* = Partial (versions R70 and higher) > * *S* = Streaming - PXL (versions R70 and higher) > * *D* = Drop Template > * *L* = Log drop action > * The *LCT* column - shows how many seconds ago a connection was created by the template > * The *DLY* column - shows Delayed Synchronization value for the template > > *Example* (all IP addresses were replaced by "X"): > > > > ``` > > Source SPort Destination DPort PR Flags LCT DLY C2S i/f S2C i/f Inst Identity > > ------------ ----- ------------ ----- -- --------- ---- --- ------- ------- ---- -------- > > X.X.X.X * X.X.X.X 161 17 ...A...S. 65 0 36/21 21/36 0 0 > > X.X.X.X * X.X.X.X 161 17 ...A...S. 5 0 36/8 8/36 1 0 > > X.X.X.X * X.X.X.X 1437 17 ...A...S. 27 0 36/8 8/36 1 0 > > X.X.X.X * X.X.X.X 161 17 ...A...S. 23 0 36/21 21/36 2 0 > > X.X.X.X * X.X.X.X 88 17 ...A...S. 11 0 8/36 36/8 4 0 > > ``` > > * **`sim if`** > > Background: > * Displays the list of interfaces used and seen by the SecureXL > > Diagnostics: > * Collect this output to see the current status > > Analysis: > * Look at the *Name* column and *Address* column > * Look at the *F* column (refer to "(6-1-B) 'sim' command" section - 'sim if' command) > > *Example*: > > > > ``` > > Name | Address | MTU | F | SIM F | IRQ | Dev | Output > > -------------------------------------------------------------------------------- > > eth0 | 172.30.168.38 | 1500 | 039 | 00000 | 67 | 0x85b1b000 | 0xffffffff > > eth1 | 10.20.30.38 | 1500 | 029 | 00008 | 75 | 0xbc5d3000 | 0xffffffff > > eth2 | 10.5.0.1 | 1500 | 029 | 00000 | 83 | 0xbcbbb000 | 0xffffffff > > ``` > > * **`sim affinity -l`** > > Background: > * Displays the current affinity of network interfaces to CPU cores > > Diagnostics: > * Collect this output to see the current status > > Check the SIM Affinity mode: > * If SIM Affinity was configured in *Static* mode, then the following configuration file should exist and should ***not*** be empty: > > *\[Expert@HostName\]# cat `$PPKDIR/boot/modules/sim_aff.conf`* > * If SIM Affinity was configured in *Automatic* mode, then the following Task should exist in CPD Scheduler: > > *\[Expert@HostName\]# cpd_sched_config print \| grep -A 5 "Sim_Affinity"* > > ``` > Task: "Sim_Affinity" > Command: sim > Arguments: affinity -c > Interval: 60 > Active: true > RunAtStart: false > ``` > > *Example*: > > > > ``` > > eth0 : 0 > > eth1 : 0 > > eth2 : 1 > > eth3 : 1 > > ``` ### (4-5) Initial diagnostics - CoreXL {#Initial diagnostics - CoreXL} Show / Hide initial diagnostics information > 1. **`cpview`** > > Background: > * Displays the CoreXL status and statistics (and many other counters). Refer to [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878). > > Diagnostics: > * Monitor the CoreXL performance during the problem > > Analysis: > * On the '`SysInfo`' tab, refer to '`Configuration Information:`' section - look at '`CoreXL Status`' > * On the '`SysInfo`' tab, refer to '`Configuration Information:`' section - look at '`CoreXL instances`' > * On the '`I/S`' tab, go to '`CoreXL`' menu - go to '`General`' menu - refer to '`Queues:`' section - look at counter '`Enqueue fail`' > * On the '`I/S`' tab, go to '`CoreXL`' menu - go to '`Instances`' menu - activate the statistics (!may affect performance) - go to each CoreXL FW instance ('`FW-Instance`'): > * refer to '`FW Stats`' section > * refer to '`Top FW-Lock consumers:`' section > 2. **`fw ctl multik stat`** > > Background: > * Displays status of CoreXL instances and summary for traffic that passes through each CoreXL FW instance (current number and peak number of concurrent connections) > > Diagnostics: > * Collect this output to see the current status > > Analysis: > * Check the number and the allocation of FW instances to CPU cores (in cluster, the output must be identical on all members) > * Check the peak number connections on FW instances (instances should be loaded as equally as possible) > > *Example from machine with 4 CPU cores (1 SND + 3 CoreXL FW instances)*: > > > > ``` > > ID | Active | CPU | Connections | Peak > > ---------------------------------------------- > > 0 | Yes | 3 | 26 | 66 > > 1 | Yes | 2 | 36 | 64 > > 2 | Yes | 1 | 39 | 53 > > ``` > > 3. **`fw ctl affinity -l -r -v -a`** > > Background: > * Displays affinity of interfaces, processes and CoreXL FW instances to CPU cores > > Diagnostics: > * Collect this output to see the current status > > Analysis: > * Check the affinity settings (of interfaces, processes and FW instances) to CPU cores (in cluster, the output must be identical on all members) > * The interfaces affinity should be configured only to to CPU cores that are running as SNDs (CPU cores that are ***not*** running CoreXL FW instances) > * Under normal circumstances, it is ***not*** recommended for the SND and a CoreXL FW instance to share the same CPU core > > *Example from machine with 4 CPU cores and 4 interfaces (1 SND + 3 CoreXL FW instances)*: > > > > ``` > > CPU 0: eth0 (irq 67) eth3 (irq 59) > > CPU 1: eth1 (irq 75) > > fw_2 > > CPU 2: eth2 (irq 83) > > fw_1 > > CPU 3: fw_0 > > All: mpdaemon fwucd cpca vpnd fwm ... cpd cprid > > ``` > > 4. **`cat /proc/interrupts`** > > Background: > * Manual page - > * Displays the number of interrupts on each CPU core from each IRQ > > Usage: > * Refer to the manual page > * By default, output is sorted by IRQ number > * The only relevant devices are network interfaces > * To shorten the output, use the '`grep`' command - for example, run: > *`[Expert@HostName]# cat /proc/interrupts | grep -E "CPU|eth"`* > * To collect this output continuously, use the '`watch`' command - for example, run: > *`[Expert@HostName]# watch -d -n 1 'cat /proc/interrupts | grep -E "CPU|eth"'`* > > Diagnostics: > * Collect this output continuously during the problem > > Analysis: > * Look at the general trend - which CPU receives more interrupts and from which interfaces > * If some CPU cores receive more interrupts than others, then affinity of interfaces to CPU cores should be optimized - interface should be redistributed better > > *Example from machine with 8 CPU cores and 5 interfaces*: > > > > ``` > > CPU0 CPU1 CPU2 CPU3 > > 0: 51254849 0 0 0 IO-APIC-edge timer > > 1: 4 0 0 5 IO-APIC-edge i8042 > > 6: 2 3 0 0 IO-APIC-edge floppy > > 7: 0 0 0 0 IO-APIC-edge parport0 > > 8: 3 0 0 0 IO-APIC-edge rtc > > 9: 0 0 0 0 IO-APIC-level acpi > > 12: 0 0 0 115 IO-APIC-edge i8042 > > 15: 302 1043 168 1127 IO-APIC-edge ide1 > > 59: 320145 6732 5944 6189 IO-APIC-level ioc0, eth3 > > 67: 15765753 0 0 0 IO-APIC-level eth0 > > 75: 24271 1285 36 0 IO-APIC-level eth1 > > 83: 24272 0 1322 0 IO-APIC-level eth2 > > NMI: 0 0 0 0 > > LOC: 50950084 50945459 50949418 50945128 > > ERR: 0 > > MIS: 0 > > ``` *** ** * ** *** (5) Advanced diagnostics {#Advanced diagnostics} ------------------------------------------------ ### (5-1) Advanced diagnostics - CPU {#Advanced diagnostics - CPU} Show / Hide advanced diagnostics information > 1. **`cpview`** > > Background: > * Displays the CPU utilization (and many other counters). Refer to [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878). > > Diagnostics: > * Monitor the CPU utilization during the problem > > Analysis: > * On the '`I/S`' tab, go to '`CPU`' menu - go to '`Contexts`' menu - refer to section '`Contexts`' > 2. **`vmstat [-n] [`*delay_in_sec*` [`*number_of_samples*`]]`** > > Background: > * Manual page - > * Displays information about processes, memory, paging, block IO, and CPU activity > > Usage: > * Refer to the manual page > * By default, the output's header (with names of columns) is displayed every 20 samples. If output is redirected to a file, then it will be very difficult to analyze it using such commands as '`grep`' / '`awk`'. Therefore, if output is redirected to a file, use the '`-n`' flag to display the header only once (at the very top) - run: > *`[Expert@HostName]# vmstat -n [`*delay_in_sec*` [`*number_of_samples*`]]`* > * By default, only 1 sample is printed (with the average values since boot). Therefore, to collect this output continuously, specify the delay between the samples - run: > *`[Expert@HostName]# vmstat [-n] `*delay_in_sec*` [`*number_of_samples*`]`* > * If you specified the delay between the samples, then the output will be collected indefinitely (until the command is stopped or killed). If you want to limit the output, then specify the total number of samples - run: > *`[Expert@HostName]# vmstat [-n] `*delay_in_sec*` `*number_of_samples** > > Diagnostics: > * Collect this output continuously during the problem > > Analysis: > * Look at the "*`procs`* " section - number of processes waiting for CPU (counter *`r`*) > * Look at the "*`swap`* " section - reading from swap file (*`si`* ) and writing to swap file (*`so`*) > * Look at the "*`io`* " section - reading from hard disk (*`bi`* ) and writing to hard disk (*`bo`*) > * Look at the "*`system`* " section - number of Context Switches (*`cs`*) > * Look at the "*`cpu`* " section - at all counters: > * "User Space" - counter *`us`* > * "System (kernel) Space" - counter *`sy`* > * "Idle" - counter *`id`* > * "I/O waiting" - counter *`wa`* > > *Example*: > > > > ``` > > procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ > > r b swpd free buff cache si so bi bo in cs us sy id wa st > > 2 0 323280 100128 103192 198132 1 2 8 35 117 130 2 3 96 0 0 > > 2 0 323280 100400 103196 198132 0 0 0 260 1297 6904 1 4 95 0 0 > > 1 0 323280 100404 103196 198132 0 0 0 0 1289 7100 1 4 95 0 0 > > 7 0 323280 100252 103200 198132 0 0 0 32 1308 6971 0 2 97 0 0 > > 1 0 323280 100428 103200 198132 0 0 0 0 1265 7111 1 3 97 0 0 > > ``` > > 3. **`sar [-u] [-P { | ALL }] [`*interval_in_sec*` [`*number_of_samples*`]]`** > > Background: > * Manual page - > * Displays information about CPU activity, network devices, memory, paging, block IO, etc. > > Usage: > * Refer to [sk112734 - How to collect System Activity Report using the "sar" command](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk112734) > > Diagnostics: > * Collect this output during the problem > > Analysis: > * Look at the load in "User Space" - counter *user* > High CPU consumption in "User Space" can be caused by processes that perform heavy tasks (e.g., too much logging by *fwd*, reloading the configuration during policy installation, etc.) > * Look at the load in "System (kernel) Space" - counter *system* > High CPU consumption in "System (kernel) Space" can be caused by heavy tasks (e.g., deep inspection of packets, enabling of all blades, enabling of all IPS protections in *Prevent* mode, etc.) > * Look at the amount of "Idle" - counter *idle* > The more CPU is idle, the better the machine's performance is > * Look at the amount of "I/O waiting" - counter *iowait* > High amount of "I/O waiting" is caused by heavy reading from/writing to hard disk (e.g., during policy installation, heavy logging, insufficient RAM, etc.) > * Look at the counter *steal* - Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor. > > *Example*: > > > > ``` > > [Expert@R77-GW:0]# sar -u > > Linux 2.6.18-92cp (R77-GW) 08/04/16 > > > > 16:10:01 CPU %user %nice %system %iowait %steal %idle > > 16:20:01 all 0.50 0.00 1.49 0.43 0.00 97.58 > > 16:30:01 all 0.53 0.00 1.78 0.36 0.00 97.32 > > 16:40:01 all 0.53 0.00 1.55 0.24 0.00 97.68 > > 16:50:01 all 0.51 0.00 1.76 0.26 0.00 97.46 > > 17:00:01 all 0.51 0.21 1.71 0.33 0.00 97.25 > > 17:10:01 all 0.48 0.00 1.57 0.35 0.00 97.59 > > 17:20:01 all 0.49 0.00 1.64 0.31 0.00 97.57 > > 17:30:01 all 0.48 0.00 1.60 0.48 0.00 97.44 > > 17:40:01 all 0.48 0.00 1.61 0.31 0.00 97.60 > > 17:50:01 all 0.50 0.00 1.68 0.32 0.00 97.51 > > 18:00:01 all 0.49 0.18 1.44 0.29 0.00 97.60 > > 18:10:01 all 0.48 0.00 1.76 0.45 0.00 97.31 > > 18:20:01 all 0.49 0.00 1.71 0.32 0.00 97.48 > > 18:30:01 all 0.49 0.00 1.53 0.41 0.00 97.56 > > 18:40:01 all 0.53 0.00 1.88 0.32 0.00 97.26 > > 18:50:01 all 0.49 0.00 1.56 0.29 0.00 97.66 > > Average: all 0.50 0.02 1.64 0.34 0.00 97.49 > > [Expert@R77-GW:0]# > > > > [Expert@R77-GW:0]# sar -P 1 > > Linux 2.6.18-92cp (R77-GW) 08/04/16 > > > > 16:10:01 CPU %user %nice %system %iowait %steal %idle > > 16:20:01 1 0.51 0.00 1.28 0.20 0.00 98.01 > > 16:30:01 1 0.52 0.00 1.41 0.03 0.00 98.04 > > 16:40:01 1 0.51 0.00 1.35 0.04 0.00 98.10 > > 16:50:01 1 0.47 0.00 1.38 0.01 0.00 98.14 > > 17:00:01 1 0.52 0.20 1.37 0.05 0.00 97.86 > > 17:10:01 1 0.47 0.00 1.39 0.08 0.00 98.06 > > 17:20:01 1 0.51 0.00 1.38 0.11 0.00 98.01 > > 17:30:01 1 0.50 0.00 1.36 0.20 0.00 97.94 > > 17:40:01 1 0.49 0.00 1.30 0.08 0.00 98.14 > > 17:50:01 1 0.49 0.00 1.37 0.11 0.00 98.03 > > Average: 1 0.50 0.02 1.36 0.09 0.00 98.03 > > [Expert@R77-GW:0]# > > ``` > > 4. **`sh pstack_sym.sh `**** > > Background: > * Manual page - > * Displays a stack trace of a running process > * Check Point R\&D have created a special shell script to collect the necessary information based on the 'pstack' utility > > Usage: > 1. [Download the package](https://supportcenter.checkpoint.com/supportcenter/portal/role/supportcenterUser/page/default.psml/media-type/html?action=portlets.DCFileAction&eventSubmit_doGetdcdetails=&fileid=32051) that contains the 'pstack' utility and special Check Point shell script that collects the necessary information. > > 2. Transfer the package to the problematic machine (into some directory, e.g., `/some_path_to_pstack/`). > > 3. Navigate to the directory with the package: > > **`[Expert@HostName]# cd /some_path_to_pstack/`** > 4. Unpack the package: > > **`[Expert@HostName]# tar -xvf pstack_sym.tar`** > 5. Install the 'pstack' utility: > > **`[Expert@HostName]# rpm -ihv --force --nodeps pstack-1.1-7.i386.rpm`** > Note: there is no need to restart any service or to reboot the machine. > 6. Determine the PID of the problematic daemon (that consumes the CPU at high level) - run the **`top`** command and look at the left-most column '`PID`'. > > Alternatively, if you already know the name of the daemon, run this command: > **`[Expert@HostName]# ps auxw | grep -v grep | grep -i NAME_OF_PROBLEMATIC_DAEMON_THAT_CONSUMES_CPU | awk '{print $2}'`** > 7. During the problem, run the special shell script at least 5-7 times: > > **`[Expert@HostName]# sh pstack_sym.sh `**** > > **Important Note:** It might take 10-15 minutes for the script to complete the analysis - please wait patiently. > 1. run the script > 2. wait until the script finishes > 3. follow the instructions on the screen > 4. run the script > 5. wait until the script finishes > 6. follow the instructions on the screen > 7. etc., etc., etc. > 8. Send the following to [Check Point Support](https://www.checkpoint.com/support-services/contact-support/)for analysis: > > * Outputs collected by the script (location is shown on the screen) > * Log files from the problematic daemon (should be located either in `$CPDIR/log/`, or in `$FWDIR/log/`) > * CPinfo file from the problematic machine (just to see the versions and the configuration) > > Analysis: > * All the outputs will be analyzed by Check Point Support and R\&D > > *Example*: > > > > ``` > > [Expert@FW]# sh pstack_sym.sh 30477 > > > > ************************************************** > > Starting the analysis - please wait patiently > > ************************************************** > > > > > > 30477: cpd > > (No symbols found) > > 0x77751db7 : select + 57 (libc-2.3.2.so) (112, 7f9b7de0, 7f9b85e0, 7f9b8de0, 779c3244, 0) + 1850 > > 0x779acaac : T_event_mainloop_iter_select + 1ec (libComUtils.so) (87f8b90, 779c3244, 96, 779a6259, 779a4560, 779c3738) + 20 > > 0x779ad23f : T_event_mainloop_iter + 16f (libComUtils.so) (87f8b90, 77f95218, 0, 885dad8, 7f9b9678, 77fb5250) + 10 > > 0x779ad298 : T_event_mainloop_e + 28 (libComUtils.so) (87f8b90, 805359c) > > 0x77f12402 : opsec_mainloop + 22 (libopsec.so) (885dc88, 0, 80516f2, 770d, 0, 0) + 9a0 > > 0x0804cfd8 : /opt/CPshrd-R71/bin/cpd + 4fd8 (1, 7f9ba0c4, 7f9ba0cc, 0, 777b1898, 77fbe020) + 40 > > 0x776917fd : __libc_start_main + ed (libc-2.3.2.so) (804cb10, 1, 7f9ba0c4, 80509e0, 8050a28, 77fb5be0) + 80645f48 > > > > ************************************************** > > This script has completed the analysis > > ************************************************** > > > > > > Going to TAR the output files : > > ---------------------------------- > > tar: Removing leading `/' from member names > > tmp/pstack_30477.out > > tmp/pstack_30477_FINAL_OUTPUT.txt > > tmp/pstack_30477_maps.out > > tmp/pstack_30477_objdump.out > > tmp/pstack_30477_proc-stat.txt > > tmp/pstack_30477_ps_auxxxwwf.txt > > tmp/pstack_30477_top.txt > > opt/CPshrd-R71/log/cpwd.elg > > opt/CPshrd-R71/log/cpwd.elg.1 > > opt/CPshrd-R71/log/cpwd.elg.2 > > var/log/messages > > var/log/messages.1 > > var/log/messages.2 > > var/log/messages.3 > > var/log/messages.4 > > > > > > Please send the following file to Check Point Support... > > ---------------------------------- > > /tmp/pstack_30477.tar > > ---------------------------------- > > > > [Expert@FW]# > > ``` > > 5. `perf top` > > Description: > * The `perf top` command is used for real time system profiling and functions similarly to the `top` utility. However, where the `top` utility generally shows you how much CPU time a given process or thread is using, `perf top` shows you how much CPU time each specific function uses. In its default state, `perf top` tells you about functions being used across all CPUs in both the user-space and the kernel-space. To use perf top you need root access. > > Usage: > * `perf top [-e | --event=EVENT] []` > > For a list of the available options, refer to the [perf-top - Linux manual page](https://man7.org/linux/man-pages/man1/perf-top.1.html). > > *Example output*: > > > > ``` > > Samples: 8K of event 'cycles', 2000 Hz, Event count (approx.): 4579432780 lost: 0/0 drop: 0/0 > > Overhead Shared Object Symbol > > 2.20% [kernel] [k] do_syscall_64 > > 2.17% [kernel] [k] module_get_kallsym > > 1.49% [kernel] [k] copy_user_enhanced_fast_string > > 1.37% libpthread-2.29.so [.] pthread_mutex_lock 1.31% [unknown] [.] 0000000000000000 1.07% [kernel] [k] psi_task_change 1.04% [kernel] [k] switch_mm_irqs_off 0.94% [kernel] [k] fget > > 0.74% [kernel] [k] entry_SYSCALL_64 > > 0.69% [kernel] [k] syscall_return_via_sysret > > 0.69% libxul.so [.] 0x000000000113f9b0 > > 0.67% [kernel] [k] kallsyms_expand_symbol.constprop.0 > > 0.65% firefox [.] moz_xmalloc > > 0.65% libpthread-2.29.so [.] __pthread_mutex_unlock_usercnt > > 0.60% firefox [.] free > > 0.60% libxul.so [.] 0x000000000241d1cd > > 0.60% [kernel] [k] do_sys_poll > > 0.58% [kernel] [k] menu_select > > 0.56% [kernel] [k] _raw_spin_lock_irqsave > > 0.55% perf [.] 0x00000000002ae0f3 > > > > ``` ### (5-2) Advanced diagnostics - Memory {#Advanced diagnostics - Memory} Show / Hide advanced diagnostics information > 1. **`cpview`** > > Background: > * Displays the Memory utilization (and many other counters). Refer to [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878). > > Diagnostics: > * Monitor the Memory utilization during the problem > > Analysis: > * On the '`I/S`' tab, go to '`Memory`' menu - go to '`FW-Kernel`' menu - refer to section '`KMEM:`' - look at counter '`Failed`' > * On the '`I/S`' tab, go to '`Memory`' menu - go to '`FW-Kernel`' menu - refer to section '`HMEM:`' - look at counter '`Failed`' > * On the '`I/S`' tab, go to '`Memory`' menu - go to '`FW-Kernel`' menu - refer to section '`SMEM:`' - look at counter '`Failed`' > * On the '`I/S`' tab, go to '`Memory`' menu - go to '`SMEM-Failures`' menu > * On the '`I/S`' tab, go to '`Memory`' menu - go to '`Contexts`' menu > 2. **`vmstat [-n] [`*delay_in_sec*` [`*number_of_samples*`]]`** > > Background: > * Manual page - > * Displays information about processes, memory, paging, block IO, and CPU activity > > Usage: > * Refer to the manual page > * By default, the output's header (with names of columns) is displayed every 20 samples. If output is redirected to a file, then it will be very difficult to analyze it using such commands as '`grep`' / '`awk`'. Therefore, if output is redirected to a file, use the '`-n`' flag to display the header only once (at the very top) - run: > *`[Expert@HostName]# vmstat -n [`*delay_in_sec*` [`*number_of_samples*`]]`* > * By default, only 1 sample is printed (with the average values since boot). Therefore, to collect this output continuously, specify the delay between the samples - run: > *`[Expert@HostName]# vmstat [-n] `*delay_in_sec*` [`*number_of_samples*`]`* > * If you specified the delay between the samples, then the output will be collected indefinitely (until the command is stopped or killed). If you want to limit the output, then specify the total number of samples - run: > *`[Expert@HostName]# vmstat [-n] `*delay_in_sec*` `*number_of_samples** > > Diagnostics: > * Collect this output continuously during the problem > > Analysis: > * Look at the "*`procs`* " section - number of processes waiting for CPU (*`r`*) > * Look at the "*`memory`*" section - sum of these counters should be compared to the total amount of RAM > * Look at the "*`swap`* " section - reading from swap file (*`si`* ) and writing to swap file (*`so`*) > > *Example*: > > > > ``` > > procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ > > r b swpd free buff cache si so bi bo in cs us sy id wa st > > 2 0 323280 100128 103192 198132 1 2 8 35 117 130 2 3 96 0 0 > > 2 0 323280 100400 103196 198132 0 0 0 260 1297 6904 1 4 95 0 0 > > 1 0 323280 100404 103196 198132 0 0 0 0 1289 7100 1 4 95 0 0 > > 7 0 323280 100252 103200 198132 0 0 0 32 1308 6971 0 2 97 0 0 > > 1 0 323280 100428 103200 198132 0 0 0 0 1265 7111 1 3 97 0 0 > > ``` > > 3. **`sar [-q][-r][-B][-W] [`*interval_in_sec*` [`*number_of_samples*`]]`** > > Background: > * Manual page - > * Displays information about CPU activity, network devices, memory, paging, block IO, etc. > > Usage: > * Refer to [sk112734 - How to collect System Activity Report using the "sar" command](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk112734) > > Diagnostics: > * Collect this output during the problem > > Analysis: > * Look at the number of tasks waiting for run time - counter *runq-sz* > * Look at the system load average - counters *ldavg-1* , *ldavg-5* , *ldavg-15* > * Look at the percentage of used memory - counter *memused* > * Look at the percentage of used swap memory - counter *swpused* > * Look at the total number of kB the system paged in from disk per second - counter *pgpgin/s* > * Look at the total number of kB the system paged out to disk per second - counter *pgpgout/s* > * Look at the number of page faults (major + minor) made by the system per second - counter *fault/s* > * Look at the number of major faults the system has made per second - counter *majflt/s* > * Look at the total number of swap pages - counter *pswpin/s* and counter *pswpout/s* > > *Example*: > > > > ``` > > [Expert@R77-GW:0]# sar -q > > Linux 2.6.18-92cp (R77-GW) 08/04/16 > > > > 16:10:01 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 > > 16:20:01 1 114 1.11 1.04 1.06 > > 16:30:01 0 115 1.07 1.13 1.09 > > 16:40:01 0 115 1.15 1.11 1.09 > > 16:50:01 1 114 1.03 1.04 1.06 > > 17:00:01 0 114 1.10 1.09 1.08 > > 17:10:01 1 114 1.08 1.08 1.08 > > 17:20:01 1 112 1.01 1.10 1.09 > > 17:30:01 0 112 1.23 1.11 1.09 > > 17:40:01 1 112 1.15 1.12 1.09 > > 17:50:01 0 114 1.14 1.10 1.09 > > Average: 0 114 1.11 1.09 1.08 > > [Expert@R77-GW:0]# > > > > [Expert@R77-GW:0]# sar -r > > Linux 2.6.18-92cp (R77-GW) 08/04/16 > > > > 16:10:01 kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad > > 16:20:01 1221768 784220 39.09 43328 381364 2128604 0 0.00 0 > > 16:30:01 1220728 785260 39.15 44648 381408 2128604 0 0.00 0 > > 16:40:01 1219280 786708 39.22 45880 381432 2128604 0 0.00 0 > > 16:50:01 1218016 787972 39.28 47300 381456 2128604 0 0.00 0 > > 17:00:01 1216792 789196 39.34 48628 381420 2128604 0 0.00 0 > > 17:10:01 1215504 790484 39.41 50076 381444 2128604 0 0.00 0 > > 17:20:01 1214964 791024 39.43 51272 381472 2128604 0 0.00 0 > > 17:30:01 1213444 792544 39.51 52524 381492 2128604 0 0.00 0 > > 17:40:01 1212624 793364 39.55 53568 381436 2128604 0 0.00 0 > > 17:50:01 1210840 795148 39.64 54940 381488 2128604 0 0.00 0 > > Average: 1216396 789592 39.36 49216 381441 2128604 0 0.00 0 > > [Expert@R77-GW:0]# > > > > [Expert@R77-GW:0]# sar -B > > Linux 2.6.18-92cp (R77-GW) 08/04/16 > > > > 16:10:01 pgpgin/s pgpgout/s fault/s majflt/s > > 16:20:01 0.00 34.64 4417.09 0.00 > > 16:30:01 0.03 37.03 4506.64 0.00 > > 16:40:01 0.01 35.28 4402.80 0.00 > > 16:50:01 0.00 36.98 4462.80 0.00 > > 17:00:01 0.00 36.63 4519.38 0.00 > > 17:10:01 0.00 37.15 4489.80 0.00 > > 17:20:01 0.00 35.27 4440.20 0.00 > > 17:30:01 0.01 35.86 4487.54 0.00 > > 17:40:01 0.01 35.87 4460.36 0.00 > > Average: 0.01 36.08 4464.99 0.00 > > [Expert@R77-GW:0]# > > > > [Expert@R77-GW:0]# sar -W > > Linux 2.6.18-92cp (R77-GW) 08/04/16 > > > > 16:10:01 pswpin/s pswpout/s > > 16:20:01 0.00 0.00 > > 16:30:01 0.00 0.00 > > 16:40:01 0.00 0.00 > > 16:50:01 0.00 0.00 > > 17:00:01 0.00 0.00 > > 17:10:01 0.00 0.00 > > 17:20:01 0.00 0.00 > > 17:30:01 0.00 0.00 > > 17:40:01 0.00 0.00 > > Average: 0.00 0.00 > > [Expert@R77-GW:0]# > > ``` > > 4. **`slabtop [-d `*refresh_in_sec* `] [-s`*Sort_Criteria*`]`** > > Background: > * Manual pages: > * > * > * > * and > * Displays kernel slab cache information in real time (caches of frequently used objects in the Linux kernel - buffer heads, inodes, dentries, etc.) > * Instead of manually parsing the highly verbose */proc/slabinfo* file manually, it is recommended to use the */usr/bin/slabtop* program. > > Usage: > * Refer to the manual page > * By default, output is refreshed every 3 seconds, unless you use the "`-d `*refresh_in_sec*" option > * By default, output is sorted by the number of objects (column "*OBJS* ", which is equal to using the "*-so*" option). > * Recommended output sort is by the cache size of the slab (column "*CACHE SIZE* ", use the "*-sc*" option). > > Diagnostics: > * Collect this output continuously during the problem > > Analysis: > * Provide the output(s) to Check Point Support (involvement of R\&D is required) > * The most interesting information is the amount of resources a certain kernel module is using (if this amount seems too high, then there may be something wrong with this module) > * Some of the more commonly used statistics from */proc/slabinfo* file that are included in the output of */usr/bin/slabtop*: > > * OBJS - The total number of objects (memory blocks), including those in use (allocated), and some spares not in use > * ACTIVE - The number of objects (memory blocks) that are in use (allocated) > * USE - Percentage of total objects that are active \[`(ACTIVE/OBJS)*100%`\] > * OBJ SIZE - The size of the objects > * SLABS - The total number of slabs > * OBJ/SLAB - The number of objects that fit into a slab > * CACHE SIZE - The cache size of the slab > * NAME - The name of the slab > > *Example of 'slabtop' output* (excerpt): > > > > ``` > > Active / Total Objects (% used) : 160565 / 174730 (91.9%) > > Active / Total Slabs (% used) : 5005 / 5005 (100.0%) > > Active / Total Caches (% used) : 94 / 141 (66.7%) > > Active / Total Size (% used) : 19571.55K / 20754.12K (94.3%) > > Minimum / Average / Maximum Object : 0.01K / 0.12K / 128.00K > > > > OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME > > 85536 82456 96% 0.05K 1188 72 4752K buffer_head > > 12876 10385 80% 0.13K 444 29 1776K dentry_cache > > 9516 9498 99% 0.05K 122 78 488K selinux_inode_security > > 7571 6711 88% 0.03K 67 113 268K size-32 > > 7524 6869 91% 0.09K 171 44 684K vm_area_struct > > 7056 7056 100% 0.48K 882 8 3528K ext3_inode_cache > > 5936 5703 96% 0.27K 424 14 1696K radix_tree_node > > 4524 4453 98% 0.05K 58 78 232K sysfs_dir_cache > > ``` > > *Example of '`cat /proc/slabinfo`' output* (excerpt): > > > > ``` > > # name : tunables ... > > bridge_fdb_cache 0 0 64 59 1 : tunables 120 60 8 : slabdata 0 0 0 > > jbd_1k 0 0 1024 4 1 : tunables 54 27 8 : slabdata 0 0 0 > > jbd_4k 1 1 4096 1 1 : tunables 24 12 8 : slabdata 1 1 0 > > ...... > > scsi_io_context 0 0 104 37 1 : tunables 120 60 8 : slabdata 0 0 0 > > ext3_inode_cache 4596 4888 492 8 1 : tunables 54 27 8 : slabdata 611 611 0 > > ext3_xattr 0 0 48 78 1 : tunables 120 60 8 : slabdata 0 0 0 > > journal_handle 89 169 20 169 1 : tunables 120 60 8 : slabdata 1 1 0 > > journal_head 205 432 52 72 1 : tunables 120 60 8 : slabdata 6 6 0 > > ...... > > size-32(DMA) 0 0 32 113 1 : tunables 120 60 8 : slabdata 0 0 0 > > size-64 3724 4071 64 59 1 : tunables 120 60 8 : slabdata 69 69 0 > > size-32 17449 18306 32 113 1 : tunables 120 60 8 : slabdata 162 162 60 > > kmem_cache 141 165 256 15 1 : tunables 120 60 8 : slabdata 11 11 0 > > ``` > > 5. **`fw ctl failmem `*PERCENTAGE_of_ALLOCATIONS_to_FAIL*** > > Background: > * Administrator can test the behaviour of the Security Gateway during a memory shortage by failing a certain percentage of all memory allocations > > Usage: > * Refer to [sk100766 - Simulation of low memory resources on Security Gateway - random failing of memory allocations](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk100766) > > Diagnostics: > * Run this test to check how the Security Gateway is going to behave during a memory shortage > > Analysis: > * Monitor Security Gateway's resources utilization during this test - Memory, CPU, Interfaces, how traffic is passing, etc. ### (5-3) Advanced diagnostics - Network interface cards {#Advanced diagnostics - NICs} Show / Hide advanced diagnostics information > 1. **`netstat -s`** > > Background: > * Manual page - > * Displays summary statistics for each protocol > > Diagnostics: > * Collect this output continuously during the problem (usually, over some time period) > > Analysis: > * In the "*Ip* " section, look at "*incoming packets discarded*" > * In the "*Icmp* " section, look at "*ICMP messages failed*" > * In the "*Tcp* " section, look at "*bad segments received*" > * In the "*Udp* " section, look at "*packet receive errors*" > * Search for lines with "*error* ", "*fail* ", "*timeout* ", "*loss* ", "*lost*" > > *Example*: > > > > ``` > > Ip: > > 17862464 total packets received > > 0 forwarded > > 0 incoming packets discarded > > 948002 incoming packets delivered > > 788408 requests sent out > > 287 outgoing packets dropped > > 1 dropped because of missing route > > Icmp: > > 297 ICMP messages received > > 0 input ICMP message failed. > > ...... > > IcmpMsg: > > InType0: 7 > > InType3: 290 > > OutType3: 290 > > OutType8: 11 > > Tcp: > > 66034 active connections openings > > 11704 passive connection openings > > 23572 failed connection attempts > > ...... > > Udp: > > 15488 packets received > > 287 packets to unknown port received. > > ...... > > TcpExt: > > 1 packets pruned from receive queue because of socket buffer overrun > > 1142 TCP sockets finished time wait in fast timer > > ...... > > IpExt: > > InMcastPkts: 420 > > InBcastPkts: 277064 > > ``` > > 2. **`ethtool -S `*name_of_interface*** > > Background: > * Manual page - > * Displays NIC-specific and driver-specific statistics > > Diagnostics: > * Collect this output continuously during the problem (usually, over some time period) > > Analysis: > * Check every line that contains "*error* ", "*drop* ", "*buffer* ", "*fail*" > * [sk61922 - Excessive RX Errors / RX Drops found on interface](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk61922) > > *Example*: > > > > ``` > > NIC statistics: > > rx_packets: 21610536 > > tx_packets: 130262 > > rx_bytes: 1712923726 > > tx_bytes: 34001412 > > ...... > > rx_errors: 0 > > tx_errors: 0 > > tx_dropped: 0 > > ...... > > rx_length_errors: 0 > > rx_over_errors: 0 > > rx_crc_errors: 0 > > rx_frame_errors: 0 > > rx_no_buffer_count: 0 > > rx_missed_errors: 0 > > tx_aborted_errors: 0 > > tx_carrier_errors: 0 > > tx_fifo_errors: 0 > > ...... > > ``` > > 3. **`sar [-n { DEV | EDEV }] [`*interval_in_sec*` [`*number_of_samples*`]]`** > > Background: > * Manual page - > * Displays information about CPU activity, network devices, memory, paging, block IO, etc. > > Usage: > * Refer to [sk112734 - How to collect System Activity Report using the "sar" command](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk112734) > > Diagnostics: > * Collect this output during the problem > > Analysis: > * Look at all the counters > > *Example*: > > > > ``` > > [Expert@R77-GW:0]# sar -n DEV 1 2 > > Linux 2.6.18-92cp (R77-GW) 08/04/16 > > > > 19:12:02 IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s > > 19:12:03 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > 19:12:03 eth0 1.01 2.02 107.07 193.94 0.00 0.00 0.00 > > 19:12:03 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > > > 19:12:03 IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s > > 19:12:04 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > 19:12:04 eth0 6.06 9.09 595.96 1076.77 0.00 0.00 0.00 > > 19:12:04 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > > > Average: IFACE rxpck/s txpck/s rxbyt/s txbyt/s rxcmp/s txcmp/s rxmcst/s > > Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > Average: eth0 3.54 5.56 351.52 635.35 0.00 0.00 0.00 > > Average: eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > [Expert@R77-GW:0]# > > > > [Expert@R77-GW:0]# sar -n EDEV 1 2 > > Linux 2.6.18-92cp (R77-GW) 08/04/16 > > > > 19:12:36 IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s > > 19:12:37 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > 19:12:37 eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > 19:12:37 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > > > 19:12:37 IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s > > 19:12:38 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > 19:12:38 eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > 19:12:38 eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > > > Average: IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s > > Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > Average: eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > Average: eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > [Expert@R77-GW:0]# > > ``` ### (5-4) Advanced diagnostics - SecureXL {#Advanced diagnostics - SecureXL} Show / Hide advanced diagnostics information > Refer to [sk153832 (ATRG: SecureXL for R80.20 and higher)](https://support.checkpoint.com/results/sk/sk153832). > > 1. **`cpview`** > > Background: > * Displays the SecureXL status and statistics (and many other counters). Refer to [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878). > > Diagnostics: > * Monitor the SecureXL performance during the problem > > Analysis: > * On the '`Traffic`' tab, go to '`Overview`' menu - refer to section '`Templates:`' > * On the '`Traffic`' tab, go to '`Overview`' menu - refer to section '`Logged drops:`' - look at counter '`SXL`' > * On the '`I/S`' tab, go to '`SXL`' menu - go to '`F2F-Reasons`' menu > * On the '`I/S`' tab, go to '`SXL`' menu - go to '`Drop-Reasons`' menu > 2. **Forwarded to Firewall (F2F) traffic** > > Background: > * Some connections cannot be accelerated by design - refer to [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578) > > Diagnostics: > * Collect this information during the problem (**Important Note:** These commands might increase CPU and memory consumption) > > Analysis: > * To check the total number of F2F connections: > *`[Expert@HostName]# fw tab -t connections -u -z`* > * To see why specific connections are Forwarded to Firewall, run the following kernel debug (debug will apply only to new connections): > > ``` > [Expert@HostName]# fw ctl debug 0 > [Expert@HostName]# fwaccel dbg resetall > [Expert@HostName]# sim dbg resetall > [Expert@HostName]# fw ctl debug -buf 32000 > [Expert@HostName]# fwaccel dbg + offload > [Expert@HostName]# sim dbg + f2f > [Expert@HostName]# fwaccel dbg list > [Expert@HostName]# sim dbg list > [Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt > ``` > > 3. **`fwaccel stats`** > > Background: > * Displays SecureXL acceleration statistics > > Diagnostics: > * Collect this output continuously during the problem > > Analysis: > * [Performance Tuning Administration Guide](https://support.checkpoint.com/product/530) for your version - Chapter "SecureXL". > > *Example*: > > > > ``` > > Name Value Name Value > > -------------------- --------------- -------------------- --------------- > > > > Accelerated Path > > ------------------------------------------------------------------------------ > > accel packets 0 accel bytes 0 > > conns created 60665 conns deleted 60606 > > ... ... > > > > Accelerated VPN Path > > ------------------------------------------------------------------------------ > > C crypt conns 0 enc bytes 0 > > dec bytes 0 ESP enc pkts 0 > > ... ... > > > > Medium Path > > ------------------------------------------------------------------------------ > > PXL packets 0 PXL async packets 0 > > PXL bytes 0 C PXL conns 0 > > C PXL templates 0 > > > > Accelerated QoS Path > > ------------------------------------------------------------------------------ > > QXL packets 0 QXL async packets 0 > > QXL bytes 0 C QXL conns 0 > > C QXL templates 0 > > > > Firewall Path > > ------------------------------------------------------------------------------ > > F2F packets 14870123 F2F bytes 966432478 > > C F2F conns 59 TCP violations 0 > > C partial conns 0 C anticipated conns 0 > > port alloc f2f 0 > > > > General > > ------------------------------------------------------------------------------ > > memory used 0 free memory 0 > > C used templates 0 pxl tmpl conns 0 > > ... ... > > > > (*) Statistics marked with C refer to current value, others refer to total value > > ``` > > 4. **`cat /proc/ppk/stats`** > > Background: > * Displays total number of packets that passed through interfaces > > Diagnostics: > * Collect this output to see the current status > > *Example*: > > > > ``` > > IRQ | Stats | Interface > > ---------------------------------------------- > > 67 793651087 eth0 > > 75 0 eth1 > > 83 0 eth2 > > 59 0 eth3 > > > > Ifn1 | Ifn2 | Stats > > ----------------------------- > > ``` > > 5. **`cat /proc/ppk/conf`** > > Background: > * Displays SecureXL configuration and basic statistics > > Diagnostics: > * Collect this output to see the current status > > Analysis: > * For *Flags* line, refer to "(6-1-B) 'sim' command" section - 'sim if' command > * For *TmplQuota* lines, refer to "(6-1-B) 'sim' command" section - 'sim tmplquota' command > * For *Debug flags* section, refer to "(6-1-B) 'sim' command" section - 'sim dbg' command > (Note: the numbers in the left column correspond to order of debug modules in the output of '`sim dbg list`' command) > > *Example*: > > > > ``` > > Flags : 0x00009a16 > > Accounting Update Interval : 60 > > Conn Refresh Interval : 512 > > SA Sync Notification Interval : 0 > > UDP Encapsulation Port : 0 > > Min TCP MSS : 0 > > TCP Auto-Expire Timeout : 20 > > Connection Limit : 18446744073709551615 > > TmplQuota Enabled : 0 > > TmplQuota Quota (rate) : 512 > > TmplQuota Drop Dduration : 300 > > TmplQuota Monitor only : 0 > > TmplQuota Dropped pkts : 0 > > > > Total Number of conns : 54 > > Number of F2F conns : 54 > > Number of Crypt conns : 0 > > Number of TCP conns : 18 > > Number of Non-TCP conns : 36 > > Number of Delayed TCP conns : 0 > > Number of Delayed Non-TCP conns: 0 > > > > Debug flags : > > 0 : 0x0 > > 1 : 0x0 > > 2 : 0x0 > > 3 : 0x0 > > 4 : 0x0 > > 5 : 0x0 > > 6 : 0x0 > > 7 : 0x0 > > 8 : 0x0 > > 9 : 0x0 > > 10 : 0x0 > > 11 : 0x0 > > 12 : 0x0 > > 13 : 0x0 > > 14 : 0x0 > > 15 : 0x0 > > ``` > > 6. **`cat /proc/ppk/ifs`** > > Background: > * Displays the list of interfaces used and seen by the SecureXL > > Diagnostics: > * Collect this output to see the current status > > Analysis: > * Look at the *Name* column and *Address* column > * Look at the *F* column (refer to "(6-1-B) 'sim' command" section - 'sim if' command) > > *Example*: > > > > ``` > > No | Interface | Address | IRQ | F | SIM F | Dev | Output Func | Features > > ------------------------------------------------------------------------------------------------------------- > > 2 | eth0 | 172.30.41.90 | 67 | 39 | 0 | 0xeffc2000 | 0xf0a862e0 | 0x000013a0 > > 3 | eth1 | 10.10.10.90 | 75 | 29 | 0 | 0xefc33000 | 0xf0a862e0 | 0x000013a0 > > 4 | eth2 | 20.20.20.90 | 83 | 29 | 0 | 0xef7fd000 | 0xf0a862e0 | 0x000013a0 > > 5 | eth3 | 30.30.30.90 | 59 | 29 | 0 | 0xef496000 | 0xf0a862e0 | 0x000013a0 > > ``` > > 7. **`cat /proc/ppk/affinity`** > > Background: > * Displays the status and the thresholds for New Affinity (feature will be activated only if there is no massive VPN traffic and the packets-per-second rate (cut-through) is high enough to benefit from the new affinity; feature will be activated only if CPU strength is greater than 3 GHz) > > Diagnostics: > * Collect this output to see the current status > > *Example*: > > > > ``` > > New affinity enabled : no > > Min CPU speed to act (MHz) : 3000 > > Min accelerated PPS to act : 250000 > > Max enc. bytes rate to act : 10000 > > Current accelerated PPS : 0 > > Current enc. bytes rate : 0 > > Currently new sim affinity set : no > > ``` > > 8. **`cat /proc/ppk/cpls`** > > Background: > * Displays the SecureXL configuration for ClusterXL Load Sharing support > > Diagnostics: > * Collect this output to see the current status > > Analysis: > * The most important data in this output is the value of *fwha_conf_flags* - it should include the value of 0x1 (i.e., *ACTIVE*). > > *Example*: > > > > ``` > > fwha_conf_flags: 0 > > fwha_df_type: 0 > > fwha_member_id: 255 > > fwha_port: 0 > > fwha_src_mac: 00 00 00 00 00 00 > > fwha_src_mac_mask: 00 00 00 00 00 00 > > udp_enc_port: 0 > > selection table size: 0 > > selection table: > > 00 00 00 00 00 00 00 00 00 00 > > ``` > > 9. **`cat /proc/ppk/statistics`** > > Background: > * Displays SecureXL statistics - the same output as from '`fwaccel stats -l`' command > > Diagnostics: > * Collect this output to see the current status > > *Example*: > > > > ``` > > Name Value Name Value > > -------------------- --------------- -------------------- --------------- > > conns created 67518 conns deleted 67475 > > temporary conns 0 templates 0 > > nat conns 0 accel packets 0 > > accel bytes 0 F2F packets 16599213 > > ESP enc pkts 0 ESP enc err 0 > > ESP dec pkts 0 ESP dec err 0 > > ESP other err 0 espudp enc pkts 0 > > espudp enc err 0 espudp dec pkts 0 > > espudp dec err 0 espudp other err 0 > > AH enc pkts 0 AH enc err 0 > > AH dec pkts 0 AH dec err 0 > > AH other err 0 memory used 0 > > free memory 0 acct update interval 60 > > current total conns 43 TCP violations 0 > > conns from templates 0 TCP conns 15 > > delayed TCP conns 0 non TCP conns 28 > > delayed nonTCP conns 0 F2F conns 43 > > F2F bytes 1076600709 crypt conns 0 > > enc bytes 0 dec bytes 0 > > partial conns 0 anticipated conns 0 > > dropped packets 0 dropped bytes 0 > > nat templates 0 port alloc templates 0 > > conns from nat templ 0 port alloc conns (tm 0 > > port alloc f2f 0 PXL templates 0 > > PXL conns 0 PXL packets 0 > > PXL bytes 0 PXL async packets 0 > > conns auto expired 0 C used templates 0 > > pxl tmpl conns 0 C conns from tmpl 0 > > C non TCP F2F conns 28 C tcp handshake conn 14 > > C tcp established co 1 C tcp closed conns 0 > > C tcp f2f handshake 14 C tcp f2f establishe 1 > > C tcp f2f closed con 0 C tcp pxl handshake 0 > > C tcp pxl establishe 0 C tcp pxl closed con 0 > > QXL templates 0 QXL conns 0 > > QXL packets 0 QXL bytes 0 > > QXL async packets 0 outbound packets 0 > > outbound pxl packets 0 outbound f2f packets 124862 > > outbound bytes 0 outbound pxl bytes 0 > > outbound f2f bytes 33552839 trimmed pkts 0 > > ``` > > 10. **`fwaccel stats -d`** > > Background: > * Displays SecureXL drop statistics > > Diagnostics: > * Collect this output to see the current status > > *Example*: > > > > ``` > > Reason Value Reason Value > > -------------------- --------------- -------------------- --------------- > > General 0 CPASXL Decision 0 > > PSLXL Decision 0 Clear Packet on VPN 12 > > Encryption Failed 0 Drop Template 0 > > Decryption Failed 0 Interface Down 0 > > Cluster Error 0 XMT Error 0 > > Anti-Spoofing 23 Local Spoofing 0 > > Sanity Error 0 Monitored Spoofed 0 > > QoS Decision 0 QoS LLQ Decision 0 > > QoS Packet Rejected 0 C2S Violation 0 > > S2C Violation 0 Loop Prevention 0 > > DOS Fragments 0 DOS IP Options 0 > > DOS Deny Lists 0 DOS Penalty Box 0 > > DOS Rate Limiting 0 Syn Attack 0 > > Reorder 0 Virt Defrag Timeout 0 > > Invalid Interface 0 Null Routing info 0 > > Unable to get out ifn 0 Resource exhausted 0 > > Conn not found 0 Failed to del corr 0 > > Corr instead of conn 0 Del zombie conn fail 0 > > FW UUID no match 0 Offload mismatch 0 > > SIM init failed 0 Null stream init info 0 > > Unable to get CGNAT 0 Null stream app info 0 > > Failed get init info 0 Collid conn not found 0 > > Del collid conn fail 0 Add conn after collid 0 > > SEQ valid 0 Enqueue QoS failed 0 > > AUX CI null 0 Link dead 0 > > VPN packet too big 0 NAT64 failed 0 > > NAT46 failed 0 Packet > MTU 0 > > NAC validation 0 TCP state violation 0 > > Enforce packet 0 GTP check packet 0 > > Bridge route error 0 Route ifn changed 0 > > IP forwarding 0 Copy MACS failed 0 > > Fragments Drops 0 Send Notification 0 > > Conn not found RST 0 Forward to PPAK fail 0 > > Cluster forward fail 0 F2F before encrypt 0 > > Forward dst encrypt 0 Correction I/S fail 0 > > Do inbound F2F 0 Packet UDP failed 0 > > F2F not allowed 0 Do routing 0 > > Fanout won't F2F 0 SCTP validation fail 0 > > SCTP not data 0 Invalid TCP option 4 > > Invalid MSS option 0 Invalid MSS value 0 > > Invalid window scale 0 Invalid IP option 0 > > Resume from Host err 0 Invalid ethertype 0 > > ``` > > 11. **`cat /proc/ppk/viol_statistics`** > > Background: > * Displays violations statistics > > Diagnostics: > * Collect this output to see the current status > > *Example*: > > > > ``` > > Violation Packets Violation Packets > > -------------------- --------------- -------------------- --------------- > > pkt is a fragment 0 pkt has IP options 406 > > ICMP miss conn 932 TCP-SYN miss conn 169 > > TCP-other miss conn 17 UDP miss conn 10305933 > > other miss conn 0 VPN returned F2F 0 > > ICMP conn is F2Fed 7 TCP conn is F2Fed 23665 > > UDP conn is F2Fed 6275273 other conn is F2Fed 0 > > uni-directional viol 0 possible spoof viol 0 > > TCP state viol 0 out if not def/accl 0 > > bridge, src=dst 0 routing decision err 0 > > sanity checks failed 0 temp conn expired 0 > > fwd to non-pivot 0 broadcast/multicast 0 > > cluster message 0 partial conn 0 > > PXL returned F2F 0 cluster forward 0 > > chain forwarding 0 general reason 0 > > ``` > > 12. **`cat /proc/ppk/mcast_statistics`** > > Background: > * Displays SecureXL multicast statistics > > Diagnostics: > * Collect this output to see the current status > > *Example*: > > > > ``` > > Name Value Name Value > > -------------------- --------------- -------------------- --------------- > > in packets 406 out packets 0 > > if restricted 0 conns with down if 0 > > f2f packets 0 f2f bytes 0 > > dropped packets 0 dropped bytes 0 > > accel packets 0 accel bytes 0 > > mcast conns 1 > > ``` ### (5-5) Advanced diagnostics - CoreXL {#Advanced diagnostics - CoreXL} Show / Hide advanced diagnostics information > Refer to [sk98737 - ATRG: CoreXL](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk98737). > > 1. **`fw -i `*FW_INSTANCE_ID*` tab -t connections [`*flags*`]`** > > Background: > * Displays the Connections Table for specified CoreXL FW instance > > Diagnostics: > * Collect this output to see the current status > > *Example*: > > ` localhost:` > > ` -------- connections --------` > > ` dynamic, id 8158, attributes: keep, sync, aggressive aging, kbufs 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34, expires 25, refresh, , hashsize 1048576, unlimited` > > ` <00000001, ac1e295a, 0000b47d, 54279820, 00000050, 00000006; 00020001, 00046000, 49000000, 00000164, 00000000, 531e03e8, 00000000, 5a291eac, c0000000, ffffffff, ffffffff, ffffffff, 00000001, 00000000, 00000000, 00008000, 49196800, 00000000, 00000000, 00000000, 00000000, 00000000, fb13c000, 00000000, 0f13d800, 00000000, 00000000, 00000000, df175800, 00000000, 00000000; 25/25>` > > ` <00000000, ac1e7560, 0000008a, ac1effff, 0000008a, 00000011; 00010001, 00004000, 00000001, 0000018b, 00000106, 531e03da, 00000000, 5a291eac, c0000000, 00000001, ffffffff, ffffffff, ffffffff, 02000000, 00000000, 00000000, 99163800, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 7a179800, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000; 23/40> ` > 2. **`cat $FWDIR/conf/fwaffinity.conf`** > > Background: > * Displays the CoreXL affinity configuration > > Diagnostics: > * Collect this output to see the current status > > Analysis: > * Check the affinity settings > > *Example*: > > > > ``` > > i eth0 1 > > n fwd 3 > > ``` > > 3. **`cat /etc/fw.boot/boot.conf`** > > Background: > * Displays the CoreXL FW instances configuration > > Diagnostics: > * Collect this output to see the current status if CoreXL configuration is not saved correctly / does not survive reboot > > Analysis: > * Check each line under the '`DEFAULT_FILTER_PATH`' > > *Example*: > > > > ``` > > CTL_IPFORWARDING 1 > > DEFAULT_FILTER_PATH /etc/fw.boot/default.bin > > KERN_INSTANCE_NUM 3 > > COREXL_INSTALLED 1 > > KERN6_INSTANCE_NUM 2 > > IPV6_INSTALLED 0 > > ``` > > 4. **`cpview`** > > Background: > * Displays the CoreXL status and statistics (and many other counters). Refer to [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878). > > Diagnostics: > * Monitor the CoreXL performance during the problem > > Analysis: > * On the '`I/S`' tab, go to '`CoreXL`' menu - go to '`General`' menu > * On the '`I/S`' tab, go to '`CoreXL`' menu - go to '`Global`' menu > * On the '`I/S`' tab, go to '`CoreXL`' menu - go to '`Instances`' menu - activate the statistics (!may affect performance) *** ** * ** *** (6) Command Line syntax {#Command Line syntax} ---------------------------------------------- ### (6-1) Command Line syntax - SecureXL {#Command Line syntax - SecureXL} * (6-1-A) FWACCEL (controls acceleration feature) Show / Hide SecureXL '`fwaccel`' syntax > **`[Expert@HostName]# fwaccel [-h]`** > > **`[Expert@HostName]# fwaccel6 [-h]`** > > |--------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| > | Command | Description | > | `fwaccel help` | Prints the general help message with available parameters | > | `fwaccel ehelp` | Prints the extended help message with available parameters | > | `fwaccel <`*parameter*`> -h` | Prints the specific help message for given \<*parameter*\> | > | `fwaccel on [-a] [-q]` | Starts acceleration on-the-fly (if Performance Pack is installed) * "`-q`" flag suppresses the output * "`-a`" flag means to start acceleration on all Virtual Systems Returned strings: * `SecureXL device is enabled.` * `Failed to start SecureXL.` * `No license for SecureXL.` * `SecureXL is disabled by the firewall. Please try again later.` * `The installed SecureXL device is not compatible with the installed firewall (version mismatch).` * `The SecureXL device is in the process of being stopped. Please try again later.` * `SecureXL cannot be started while "flows" are active.` * `SecureXL is already started.` * `SecureXL will be started after a policy is loaded.` * `fwaccel: Failed to check FloodGate-1 status. Acceleration will not be started.` * `FW-1: SecureXL acceleration cannot be started while QoS is running in express mode.` `Please disable FloodGate-1 express mode or SecureXL` * `FW-1: SecureXL acceleration cannot be started while QoS is running with citrix printing rule.` `Please remove the citrix printing rule to enable SecureXL.` * `FW-1: SecureXL acceleration cannot be started while QoS is running with UAS rule.` `Please remove the UAS rule to enable SecureXL.` * `FW-1: SecureXL acceleration cannot be started while QoS is running.` `Please remove the QoS blade to enable SecureXL.` * `Failed to enable SecureXL device` * `fwaccel_on: failed to set process context `** | > | `fwaccel off [-a] [-q]` | Stops acceleration on-the-fly * "`-q`" flag suppresses the output * "`-a`" flag means to stop acceleration on all Virtual Systems Returned strings: * `SecureXL device disabled` * `SecureXL device is not active` * `Failed to disable SecureXL device` * `fwaccel_on: failed to set process context `** | > | `fwaccel ver` | Shows SecureXL FWAccel and FireWall version *Example*: ``` Firewall version: R77.10 - Build 243 Acceleration Device: Performance Pack Accelerator Version 2.1 Firewall API version: 2.90NG (13/6/2013) Accelerator API version: 2.90NG (13/6/2013) ``` | > | `fwaccel stat [-a]` | Shows SecureXL status * "`-a`" flag means to show SecureXL status for all Virtual Systems *Example*: ``` Accelerator Status : on Accept Templates : enabled Drop Templates : disabled NAT Templates : disabled by user ``` Returned strings: * Accelerator Status: * `Accelerator Status : on` * `Accelerator Status : off` * `Accelerator Status : disabled by Firewall` * `Accelerator Status : waiting for policy load` * `Accelerator Status : no license for SecureXL` * Accept Templates: * `Accept Templates : enabled` * `Accept Templates : not supported by the SecureXL device` * `Accept Templates : disabled by Firewall` * `Accept Templates : disabled by user` * Drop Templates: * `Drop Templates : enabled` * `Drop Templates : disabled` * `Drop Templates : not supported by the SecureXL device` * `Drop Templates : disabled by Firewall` * `Drop Templates : disabled by user` * NAT Templates (refer to [sk71200](https://support.checkpoint.com/results/sk/sk71200)): * `NAT Templates : enabled` * `NAT Templates : not supported by the SecureXL device` * `NAT Templates : disabled by Firewall` * `NAT Templates : disabled by user` | > | `fwaccel stats` | Prints all acceleration statistics (output is divided into relevant sections) | > | `fwaccel stats -h` | Prints the help message with available flags for '`stats`' parameter | > | `fwaccel stats -s` | Prints the summary of all acceleration statistics | > | `fwaccel stats -d` | Prints the acceleration statistics for dropped packets | > | `fwaccel stats -n` | Prints the acceleration statistics for Network Access Control (NAC) Note: Refer to the [Identity Awareness Administration Guide](https://support.checkpoint.com/product/436) for your version | > | `fwaccel stats -p` | Prints the acceleration statistics for SecureXL violations (F2F packets) | > | `fwaccel stats -l` | Prints all acceleration statistics in Legacy mode (output is *not* divided into sections) | > | `fwaccel stats -m` | Prints the acceleration statistics for multicast traffic | > | `fwaccel stats -r` | Resets all acceleration statistics | > | `fwaccel conns` | Prints the SecureXL Connections Table ('`cphwd_db`') Note: Depending on the number of concurrent connections, might consume memory at very high level Returned strings: * `The SecureXL connections table is empty` * `Failed to read the SecureXL connections table` * `There are `*VALUE*` connections in SecureXL connections table` `...` `Total number of connections: `*VALUE* | > | `fwaccel conns -h` | Prints the help message with available flags for '`conns`' parameter | > | `fwaccel conns -s` | Prints the summary of SecureXL Connections Table (number of connections) Note: Depending on the number of concurrent connections, might consume memory at very high level | > | `fwaccel conns -m `*max_entries* | Prints the SecureXL Connections Table limited to the number of *max_entries* | > | `fwaccel conns -f <`*filter*`>` | Prints the SecureXL Connections Table entries based on \ (run '`fwaccel conns -h`'); Filtering flag is a single letter (either capital, or small) | > | `fwaccel templates` | Prints the SecureXL Connections Templates ('`cphwd_tmpl`') Note: Depending on the number of current templates, might consume memory at very high level Returned strings: * `The SecureXL templates table is empty` * `Failed to read the SecureXL templates table` * `There are `*VALUE*` templates in SecureXL templates table` `...` `Total number of templates: `*VALUE* | > | `fwaccel templates -h` | Prints the help message with available flags for '`templates`' parameter | > | `fwaccel templates -s` | Prints the summary of SecureXL Connections Templates (number of templates) Note: Depending on the number of current templates, might consume memory at very high level | > | `fwaccel templates -d` | Prints the summary of SecureXL Connections Templates for dropped packets | > | `fwaccel templates -m `*max_entries* | Prints the SecureXL Connections Templates limited to the number of *max_entries* | > | `fwaccel identities` | Prints the SecureXL Identities Table ('`cphwd_dev_identity_table`') Returned strings: * `The SecureXL identity table is empty` * `Failed to read the SecureXL identity table - no memory` * `There are `*VALUE*` identities in SecureXL identities table` `...` `Total number of identities: `*VALUE* Notes: * Refer to the [Identity Awareness Administration Guide](https://support.checkpoint.com/product/436) for your version * Command is not available for IPv6 | > | `fwaccel identities -h` | Prints the help message with available flags for '`identities`' parameter | > | `fwaccel identities -s` | Prints the summary of SecureXL Identities Table (number of entries) | > | `fwaccel identities -m `*max_entries* | Prints the SecureXL Identities Table limited to the number of *max_entries* | > | `fwaccel revoked_ips` | Prints the SecureXL Revoked IPs Table ('`cphwd_dev_revoked_ips_table`') Returned strings: * `The SecureXL templates' revoked IPs table is empty` * `Failed to read the SecureXL templates' revoked IPs table - no memory` * `There are `*VALUE*` revoked IPs in SecureXL templates' revoked IPs table` `...` `Total number of templates' revoked IPs:: `*VALUE* Notes: * Refer to the [Identity Awareness Administration Guide](https://support.checkpoint.com/product/436) for your version * Command is not available for IPv6 | > | `fwaccel revoked_ips -h` | Prints the help message with available flags for '`revoked_ips`' parameter | > | `fwaccel revoked_ips -s` | Prints the summary of SecureXL Revoked IPs Table (number of entries) | > | `fwaccel revoked_ips -m `*max_entries* | Prints the SecureXL Revoked IPs Table limited to the number of *max_entries* | > | `fwaccel dbg` | Controls SecureXL debugging (run '`fwaccel dbg -h`') By default, debug messages will be printed to `/var/log/messages` file, therefore you must set the usual kernel debugging options with: ` [Expert@HostName]# fw ctl debug 0` ` [Expert@HostName]# fwaccel dbg resetall` ` [Expert@HostName]# sim dbg resetall` ` [Expert@HostName]# fw ctl debug -buf 32000` ` [Expert@HostName]# fwaccel dbg -m `*MODULE*` + `*FLAG1 FLAG2 ... FLAGn* ` [Expert@HostName]# fwaccel dbg list` ` [Expert@HostName]# sim dbg list` ` [Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt ` | > | `fwaccel dbg -h` | Prints the help message with available options, list of debug modules and their flags | > | `fwaccel dbg list` | Prints all currently enabled debug flags for all modules | > | `fwaccel dbg resetall` | Resets all debug flags for all modules to their default (none) | > | `fwaccel dbg -m `*MODULE*` reset` | Resets all debug flags for specified module to their default (none) | > | `fwaccel dbg -m `*MODULE*` all` | Enables all supported debug flags for specified module | > | `fwaccel dbg -m `*MODULE*` + `*FLAG1 FLAG2 ... FLAGn* | Enables specified debug flags for specified module | > | `fwaccel dbg -m `*MODULE*` - `*FLAG1 FLAG2 ... FLAGn* | Disables specified debug flags for specified module | > | `fwaccel dbg -f `*Source_IP* `,`*Source_Port* `,`*Dest_IP* `,`*Dest_Port* `,`*Proto* | Sets debugging filter - only the specified connection will be printed in the debug output Notes: * Only 1 filter can be set at one time * You can use the asterisk "\*" as a wildcard for IP/port/proto *Example for SSH connection*: `[Expert@HostName]# fwaccel dbg -f 172.30.1.1,*,172.30.41.90,22,6` | > | `fwaccel dbg -f reset` | Resets the debugging filter | > >
> > New 'fwaccel' commands added in R80.20: > > |-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| > | Command | Description | > | `fwaccel dos` | Interface to DOS mitigation techniques in SecureXL. Commands: * `blacklist` - Interface to the IP blacklist in SecureXL. * `config` - Interface to DOS mitigation configuration in SecureXL. * `pbox` - Interface to the penalty box policy in SecureXL. * `rate` - Interface to the rate limiting policy in SecureXL. * `stats` - Interface to DOS real-time statistics. * `tab` - Manage DOS tables. * `whitelist` - Interface to the IP whitelist in SecureXL. | > | `fwaccel dos blacklist` | Interface to the IP blacklist in SecureXL.. Commands: * `add` - Add IPs to the blacklist. * `clear` - Remove all IPs from the blacklist. * `get` - Print the blacklist. * `remove` - Remove IPs from the blacklist. | > | `fwaccel dos config` | Interface to DOS mitigation configuration in SecureXL. Commands: * `get` - View configuration parameters. * `set` - Modify configuration parameters. | > | `fwaccel dos pbox` | Interface to the penalty box policy in SecureXL. Commands: * `whitelist` -Interface to the penalty box source IP whitelist in SecureXL. | > | `fwaccel dos rate` | Interface to the rate limiting policy in SecureXL. Commands: * `get` - View rule information in the rate limiting policy. * `install` - Manually install a rate limiting policy via stdin. E.g.: `fw samp get -l -k req_type -t in -v quota | fwaccel dos rate install`

Note: Installing a new rate limiting policy with more than one rule will automatically enable the rate limiting feature. To manually disable the feature after this command, run: `fwaccel dos config set --disable-rate-limit` In order to delete the current policy, simply install a new policy with zero rules. | > | `fwaccel dos stats` | Interface to DOS real-time statistics. Commands: * `clear` - Clear real-time statistics. * `get` - View real-time statistics. | > | `fwaccel dos tab` | Manage DOS tables. Commands: * `get` - Print contents of the given DOS table. | > | `fwaccel dos whitelist` | Interface to the IP whitelist in SecureXL. Flags: * `-a addr/mask` - Add entry to whitelist (/mask is optional) * `-d addr/mask` - Delete entry from whitelist (/mask is optional) * `-l filename `- Load whitelist from filename * `-L` - Load whitelist from $FWDIR/conf/dos-whitelist-v4.conf * `-s` - Show current whitelist entries * `-F` - Flush all whitelist entries * This whitelist overrides entries in the blacklist. Before using 3rd-party or automatic blacklists, add trusted networks and hosts to the whitelist to avoid outages. * This whitelist also unblocks ip options and fragments from trusted sources when the --enable-drop-opts or --enable-drop-frags features are active. * For whitelisting of rate limiting policy, refer to the bypass action for the fw samp_policy command. I.e.: "`fw samp -a b ...`" * To replace the current whitelist with the contents of a new file, use both the -F and -l (or -L) options on the same command line. * The whitelist file should contain one entry per line in the format of the -a option. Lines beginning with '#' and blank lines are ignored. * `fwaccel dos whitelist -L` is automatically run at boot time. * The file that the `-L` option loads, does not exist by default. * See also: `fwaccel dos pbox whitelist` ` fwaccel synatk whitelist` ` fw samp -a b` | {#Command Line syntax - SecureXL - fwaccel} {#Command Line syntax - SecureXL - fwaccel} * (6-1-B) SIM (controls acceleration device) Show / Hide SecureXL '`sim`' syntax > **`[Expert@HostName]# sim [-h]`** > > **`[Expert@HostName]# sim6 [-h]`** > > |----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| > | Command | Description | > | `sim help` | Prints the general help message with available parameters | > | `sim ver [-k]` | Shows SecureXL SIM version | > | `sim if` | Prints the list of interfaces used and seen by the SecureXL implementation (Performance Pack) Configuration flags (sum of the following values in the "*F*" column): * 0x0001 - If set, the packet should be dropped at the end of the inbound processing, if the packet is "cut-through" packet. In outbound, all the packets should ne forwarded to the network. * 0x0002 - If set, and the SIM "*tcp*" feature is enabled (on), then an appropriate notification should be sent whenever a TCP state change occurs (connection established/teared down). * 0x0004 - If set, then when encapsulating an encrypted packet (UDP encapsulation), the UDP header's checksum field should be set correctly. If flag is not set, then the UDP header's checksum field should be set to zero. It is safe to ignore this flag if it is set to 0 (i.e., still calculate the checksum). * 0x0008 - If set, then when the Connections Table has reached the specified limit, new connections that match a template should not be created and the packet matching the template should be dropped. If flag is not set, the packet should be forwarded to the firewall. * 0x0010 - If set, then fragments should be forwarded to the firewall. * 0x0020 - If set, then connection creation from templates should be disabled. Connection can still be offloaded to the device. This flag disables only the creation of *TCP* templates. * 0x0040 - Accelerated connections should periodically be refreshed in the firewall through the notification. The refreshing should be done only if this global flag is set. * 0x0080 - If set, then connection creation from templates should be disabled. Connection can still be offloaded to the device. This flag disables only the creation of *non-TCP* templates. * 0x0100 - If set, then sequence verification violations should be allowed for connections that did not complete the 3-way handshake process (instead of F2F-ing violating packets). * 00x200 - If set, then sequence verification violations should be allowed for connections that completed the 3-way handshake process (instead of F2F-ing violating packets). * 0x0400 - If set, then RST TCP packets should be forwarded to firewall. * 0x0800 - If set, then Path MTU Discovery should not be enforced for IP multicast packets. * 0x1000 - If set, then SIM "drop_templates" feature should be disabled. * 0x2000 - Indicates whether Link Selection Load Sharing feature was enabled by the user. * 0x4000 - If set, then SecureXL asynchronic notification feature should be disabled. * 0x8000 - Indicates that Firewall Connections Table capacity is unlimited. *Examples*: `F=039` means the sum of the following flags: * 0x0001 * 0x0008 * 0x0010 * 0x0020 `F=0x00009a16` means the sum of the following flags: * 0x0002 * 0x0004 * 0x0010 * 00x200 * 0x0800 * 0x1000 * 0x8000 | > | `sim vpn ` | Enables/Disables acceleration of VPN traffic: * *sim vpn on* = enable acceleration of VPN traffic * *sim vpn off* = disable acceleration of VPN traffic | > | `sim ranges` | Prints the loaded ranges | > | `sim ranges -l` | Prints the list of loaded ranges | > | `sim ranges -a` | Prints all loaded ranges | > | `sim ranges `*range_id* | Prints specified loaded range | > | `sim ranges -s `*range_id* | Prints summary for specified loaded range | > | `sim affinity` | Controls network interfaces' affinity settings Note: Command is available only on Linux-based OS | > | `sim affinity -h` | Prints the help message with available options for '`affinity`' parameter | > | `sim affinity -l` | Prints the current network interfaces' affinity | > | `sim affinity -a` | Sets the network interfaces' affinity in 'Automatic' mode | > | `sim affinity -s` | Sets the network interfaces' affinity in 'Static' ('Manual') mode | > | `sim tmplquota` | Controls SIM module's template quota Note: Command is available only on Linux-based OS | > | `sim tmplquota -h` | Prints the help message with available options for '`tmplquota`' parameter | > | `sim tmplquota -e <1 | 0>` | Enables/Disables template quota feature: * *sim tmplquota -e 1* = enable the template quota feature * *sim tmplquota -e 0* = disable the template quota feature | > | `sim tmplquota -q <`*quota*`>` | Sets maximum of connections per second per template (quota is allowed) | > | `sim tmplquota -d <`*drop_duration*`>` | Sets drop duration (in seconds) for drop state | > | `sim tmplquota -m <0 | 1>` | Controls monitor only mode: * *sim tmplquota -m 0* = disable monitor only mode (default) * *sim tmplquota -m 1* = enable monitor only mode | > | `sim tmplquota -r` | Resets SIM module template quota values to their defaults | > | `sim tmplquota -d <`*file_name*`>` | Loads exclusion list of Source IP addresses / Source Subnets from the file | > | `sim tab -d templates` | Prints only templates in drop state (output is printed into `/var/log/messages` files and into Linux kernel ring buffer (output of '`dmesg`' command)) | > | `sim affinityload` | Applies SIM Affinity in 'Automatic' mode Note: Command is available only on Linux-based OS | > | `sim dropcfg` | Configures drop parameters (run '`sim dropcfg`') Notes: * Since R80.20 "sim dropcfg" is deprecated, replaced by "fw samp" (see [sk112454](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk112454)) * Command is available only on Linux-based OS. * Feature is available in R75.40 and above. * Refer to [sk67861 - Accelerated Drop Rules Feature in R75.40 - R80.10](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk67861). | > | `sim dropcfg -h` | Prints the help message with available options for '`dropcfg`' parameter | > | `sim dropcfg -l` | Prints current drop configuration | > | `sim dropcfg -f <`*/path_to/file_name*`>` | Sets drop configuration file Note: The drop rules configuration does not survive the reboot. Therefore, in order to apply the configured drop rules after the reboot, use a startup script (e.g., `/etc/rc.d/rc.local`) to run the '`sim dropcfg -f `' command automatically during each boot) | > | `sim dropcfg -e` | Enforces drop configuration on the external interface only | > | `sim dropcfg -y` | Avoids confirmation | > | `sim dropcfg -r` | Resets drop rules | > | `sim dbg` | **Note:** From R80.20 "sim dbg" is deprecated, replaced by "fwaccel dbg". Controls SecureXL SIM debugging (run '`sim dbg -h`') By default, debug messages will be printed to `/var/log/messages` file, therefore you must set the usual kernel debugging options with: ` [Expert@HostName]# fw ctl debug 0` ` [Expert@HostName]# sim dbg resetall` ` [Expert@HostName]# fwaccel dbg resetall` ` [Expert@HostName]# fw ctl debug -buf 32000` ` [Expert@HostName]# sim dbg -m `*MODULE*` + `*FLAG1 FLAG2 ... FLAGn* ` [Expert@HostName]# sim dbg list` ` [Expert@HostName]# fwaccel dbg list` ` [Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt ` | > | `sim dbg -h` | Prints the help message with available options, list of debug modules and their flags | > | `sim dbg list` | Prints all currently enabled debug flags for all modules | > | `sim dbg resetall` | Resets all debug flags for all modules to their default (none) | > | `sim dbg -m `*MODULE*` reset` | Resets all debug flags for specified module to their default (none) | > | `sim dbg -m `*MODULE*` all` | Enables all supported debug flags for specified module | > | `sim dbg -m `*MODULE*` + `*FLAG1 FLAG2 ... FLAGn* | Enables specified debug flags for specified module | > | `sim dbg -m `*MODULE*` - `*FLAG1 FLAG2 ... FLAGn* | Disables specified debug flags for specified module | > | `sim dbg -f `*Source_IP* `,`*Source_Port* `,`*Dest_IP* `,`*Dest_Port* `,`*Proto* | Sets debugging filter - only the specified connection will be printed in the debug output Notes: * Only 1 filter can be set at one time * You can use the asterisk "\*" as a wildcard for IP/port/proto *Example for SSH connection*: `[Expert@HostName]# sim dbg -f 172.30.1.1,*,172.30.41.90,22,6` | > | `sim dbg -f reset` | Resets the debugging filter | > | `sim feature` | Controls SIM module features | > | `sim feature -h` | Prints the help message with available options for '`feature`' parameter | > | `sim feature <`*feature_name*`> ` | Enables/Disables the specified SIM module feature Available features: * `anti_spoofing` * `delayed` * `drop_templates` * `dynamic_vpn` * `linksel` * `linksel_ls` * `mcast_route` * `mcast_route_v2` * `nac` * `qos` * `routing` * `streaming` * `tcp` * `vpn` * `wire` | {#Command Line syntax - SecureXL - sim} {#Command Line syntax - SecureXL - sim} ### (6-2) Command Line syntax - CoreXL {#Command Line syntax - CoreXL} (6-2-A) Show / Hide CoreXL syntax - in Gateway mode > **`[Expert@HostName]# fw ctl multik `** > > **`[Expert@HostName]# fw ctl affinity [-flag1] [-flag2] ...`** > > **`[Expert@HostName]# fw6 ctl multik `** > > **`[Expert@HostName]# fw6 ctl affinity [-flag1] [-flag2] ...`** > > |------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| > | Command | Description | > | `fw ctl multik` | Controls CoreXL FW instances | > | `fw ctl multik` | Prints the general help message with available parameters | > | `fw ctl multik stat` | Prints the summary table for CPU cores and CoreXL FW instances | > | `fw ctl multik start` | Starts CoreXL | > | `fw -i `*Instance_ID*` ctl multik start` | Starts specific CoreXL FW instance | > | `fw ctl multik stop` | Stops CoreXL | > | `fw -i `*Instance_ID*` ctl multik stop` | Stops specific CoreXL FW instance | > | `fw ctl affinity <`*options*`>` | Controls CoreXL affinities of interfaces/processes/CoreXL FW instances to CPU cores | > | `fw ctl affinity` | Prints the help message with available options | > | `fw -d ctl affinity -corelicnum` | Prints the number of system CPU cores allowed by CoreXL license | > | `fw ctl affinity -l` | Prints the current CoreXL affinities - output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores | > | `fw ctl affinity -l -r` | Prints the current CoreXL affinities in reverse order - output shows CPU cores and which interface/process/CoreXL FW instance is affined to each CPU core | > | `fw ctl affinity -l -a` | Prints all current CoreXL affinities - output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores, and also shows targets without specific affinity | > | `fw ctl affinity -l -v` | Prints the current CoreXL affinities - verbose output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores (targets are shown as 'Interface' (with IRQ), 'Kernel', 'Process' | > | `fw ctl affinity -l -q` | Prints the current CoreXL affinities - output shows affinities of interfaces/processes/CoreXL FW instances to CPU cores, and suppresses errors | > | `fw ctl affinity -l -r -a -v` | Prints the current CoreXL affinities - verbose output that combines all possible outputs (shows all targets in reverse order) | > | `fw ctl affinity -l -p `*PID*` [-r] [-a] [-v]` | Prints the current CoreXL affinity of the specified process (by PID) to CPU cores | > | `fw ctl affinity -l -n `*Daemon_Name*` [-r] [-a] [-v]` | Prints the current CoreXL affinity of the specified process (by name \[maximal length = 255 characters\]) to CPU cores | > | `fw ctl affinity -l -k `*Instance_ID*` [-r] [-a] [-v]` | Prints the current CoreXL affinity of the specified CoreXL FW instance to CPU cores | > | `fw ctl affinity -l -i `*Interface_Name*` [-r] [-a] [-v]` | Prints the current CoreXL affinity of the specified interface to CPU cores | > | `fw ctl affinity -s <`*target* `> { `*CPU_ID*` [ `*CPU_ID*` ... ] | all }` | Sets CoreXL Affinity | > | `fw ctl affinity -s -p `*PID*` { `*CPU_ID*` [ `*CPU_ID*` ... ] | all }` | Sets CoreXL affinity of the specified process (by PID) to CPU cores | > | `fw ctl affinity -s -n `*Daemon_Name*` { `*CPU_ID*` [ `*CPU_ID*` ... ] | all }` | Sets CoreXL affinity of the specified process (by name \[maximal length = 255 characters\]) to CPU cores | > | `fw ctl affinity -s -k `*Instance_ID*` { `*CPU_ID*` [ `*CPU_ID*` ... ] | all }` | Sets CoreXL affinity of the specified CoreXL FW instance to CPU cores | > | `fw ctl affinity -s -i `*Interface_Name*` { `*CPU_ID*` [ `*CPU_ID*` ... ] | all }` | Sets CoreXL affinity of the specified interface to CPU cores | (6-2-B) Show / Hide CoreXL syntax - in VSX mode > **`[Expert@HostName]# fw ctl multik `** > > **`[Expert@HostName]# fw ctl affinity [-flag1] [-flag2] ...`** > > **`[Expert@HostName]# fw6 ctl multik `** > > **`[Expert@HostName]# fw6 ctl affinity [-flag1] [-flag2] ...`** > > |-------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| > | Command | Description | > | `fw ctl multik` | Controls CoreXL FW instances | > | `fw ctl multik stat` | Prints the summary table for CPU cores and CoreXL FW instances | > | `fw ctl multik stat` | Prints the general help message with available parameters | > | `fw ctl multik start` | Starts CoreXL | > | `fw -i `*Instance_ID*` ctl multik start` | Starts specific CoreXL FW instance | > | `fw ctl multik stop` | Stops CoreXL | > | `fw -i `*Instance_ID*` ctl multik stop` | Stops specific CoreXL FW instance | > | `fw ctl affinity <`*options*`>` | Controls CoreXL affinities of Virtual Devices/interfaces/processes/CoreXL FW instances to CPU cores | > | `fw ctl affinity` | Prints the help message with available options | > | `fw -d ctl affinity -corelicnum` | Prints the number of system CPU cores allowed by CoreXL license | > | `fw ctl affinity -l` | Prints the current CoreXL affinities | > | `fw ctl affinity -l -r` | Prints the current CoreXL affinities in reverse order | > | `fw ctl affinity -l -a` | Prints all current CoreXL affinities | > | `fw ctl affinity -l -v` | Prints the current CoreXL affinities - verbose output | > | `fw ctl affinity -l -q` | Prints the current CoreXL affinities and suppresses errors | > | `fw ctl affinity -l -r -a -v` | Prints the current CoreXL affinities - verbose output that combines all possible outputs (shows all targets in reverse order) | > | `fw ctl affinity -l -x [-vsid `*VSID_ranges* `] [-cpu `*CPU_ID_ranges*`] [-flags e|k|t|n|h] [-r] [-a] [-v]` | Prints the current CoreXL affinities - extended output Notes: * If "`-vsid`" flag is omitted, the current context will be used (in which the command was issued) * The "`-vsid`" flag accepts either a single VSID (e.g., '`-vsid 7`'), or a range of VSID numbers (e.g., '`-vsid 0-5`'), or a combination (e.g., '`-vsid 0-2 4`') * The "`-cpu`" flag accepts either a single CPU ID (e.g., '`-cpu 7`'), or a range of CPU ID numbers (e.g., '`-cpu 0-5`'), or a combination (e.g., '`-cpu 0-2 4`') * The "`-flags`" requires at least one of the following arguments (multiple arguments must be specified together): * `e` - do not print exception processes * `k` - do not print kernel threads * `t` - print all process threads * `n` - print process name instead of `/proc/<`*PID*`>/cmdline` * `h` - print CPU mask in Hex format * `o` - print output into the '`/tmp/affinity_list_output`' file (VSX R77.10 and above) | > | `fw ctl affinity -s` | Sets CoreXL Affinity | > | `fw ctl affinity -s -i `*Interface_Name*` `*CPU_IDs*` | all` | Sets affinities of the specified interface to CPU cores | > | `fw ctl affinity -s -p `*PID*` `*CPU_IDs*` | all` | Sets affinities of the specified process (by PID) to CPU cores | > | `fw ctl affinity -s -d -pname `*Daemon_Name*` [-vsid `*VSID_ranges* `] -cpu `*CPU_ID_ranges* | Sets affinity of the specified process (by name) to CPU cores Notes: * If "`-vsid`" flag is omitted, the current context will be used (in which the command was issued) * The "`-vsid`" flag accepts either a single VSID (e.g., '`-vsid 7`'), or a range of VSID numbers (e.g., '`-vsid 0-5`'), or a combination (e.g., '`-vsid 0-2 4`') * The "`-cpu`" flag accepts either a single CPU ID (e.g., '`-cpu 7`'), or a range of CPU ID numbers (e.g., '`-cpu 0-5`'), or a combination (e.g., '`-cpu 0-2 4`') | > | `fw ctl affinity -s -d [-vsid `*VSID_ranges* `] -cpu `*CPU_ID_ranges* | Sets affinities of Virtual Devices (VS, VR, VSW) to CPU cores Notes: * If "`-vsid`" flag is omitted, the current context will be used (in which the command was issued) * The "`-vsid`" flag accepts either a single VSID (e.g., '`-vsid 7`'), or a range of VSID numbers (e.g., '`-vsid 0-5`'), or a combination (e.g., '`-vsid 0-2 4`') * The "`-cpu`" flag accepts either a single CPU ID (e.g., '`-cpu 7`'), or a range of CPU ID numbers (e.g., '`-cpu 0-5`'), or a combination (e.g., '`-cpu 0-2 4`') | > | `fw ctl affinity -s -d -inst `*Instances_ranges*` -cpu `*CPU_ID_ranges* | Sets affinities of the specified FWK instances to CPU cores Notes: * The "`-inst`" flag accepts either a single FWK_ID (e.g., '`-inst 7`'), or a list of FWK_ID numbers (e.g., '`-inst 0 2 4`') * The "`-cpu`" flag accepts either a single CPU ID (e.g., '`-cpu 7`'), or a range of CPU ID numbers (e.g., '`-cpu 0-5`'), or a combination (e.g., '`-cpu 0-2 4`') | > | `fw ctl affinity -s -d -fwkall `*Number_of_CPUs* | Sets affinities of all FWK instances to CPU cores (where *Number_of_CPUs* is an integer number) | > | `fw ctl affinity -vsx_factory_defaults` | Resets all VSX affinity settings (prompts the user) (VSX R77 and above) | > | `fw ctl affinity -vsx_factory_defaults_no_prompt` | Resets all VSX affinity settings (does not prompt the user) (VSX R77 and above) | ### (6-3) Command Line syntax - Multi-Queue {#Command Line syntax - Multi-Queue} Show / Hide Multi-Queue '`cpmq`' syntax > **`[Expert@HostName]# cpmq `** > > |-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| > | Command | Description | > | `cpmq get` | Shows Multi-Queue status of active supported interfaces | > | `cpmq get -a` | Shows Multi-Queue status of all supported interfaces (those with enabled Multi-Queue and those with disabled Multi-Queue) | > | `cpmq get -v` | Shows Multi-Queue status of active supported interfaces with IRQ affinity information | > | `cpmq get rx_num igb` | Shows the number of active RX queues for interfaces that use *igb* driver | > | `cpmq get rx_num ixgbe` | Shows the number of active RX queues for interfaces that use *ixgbe* driver | > | `cpmq set [-f]` | Enables/Disables Multi-Queue per interface (the "-f" flag forces the operation) | > | `cpmq set rx_num all default [-f]` | Sets the number of active RX queues to the number of CPU cores that run as CoreXL SND (CPU cores that are not used by CoreXL FW instances) - for all interfaces (those that use *igb* driver and those that use *ixgbe* driver) Note: This is recommended configuration | > | `cpmq set rx_num igb default [-f]` | Sets the number of active RX queues to the number of CPU cores, which are not used by CoreXL FW instances (recommended) - for interfaces that use *igb* driver | > | `cpmq set rx_num ixgbe default [-f]` | Sets the number of active RX queues to the number of CPU cores, which are not used by CoreXL FW instances (recommended) - for interfaces that use *ixgbe* driver | > | `cpmq set rx_num all <`*number*`> [-f]` | Sets the number of active RX queues to the number between 2 and the number of CPU cores - for all interfaces (those that use *igb* driver and those that use *ixgbe* driver) | > | `cpmq set rx_num igb <`*number*`> [-f]` | Sets the number of active RX queues to the number between 2 and the number of CPU cores - for interfaces that use *igb* driver | > | `cpmq set rx_num ixgbe <`*number*`> [-f]` | Sets the number of active RX queues to the number between 2 and the number of CPU cores - for interfaces that use *ixgbe* driver | > | `cpmq set affinity` | Sets the IRQ affinity for Multi-Queue interfaces after the following occurs (in the given order): 1. Multi-Queue is enabled on an interface 2. The interface status is changed to 'down' 3. The machine is rebooted 4. The interface status is changed back to 'up' Run this command after the interface status is changed back to 'up' to reset the IRQ affinity for this interface. | > | `cpmq reconfigure [-q]` | Reconfigures the Multi-Queue (the "-q" flag suppresses the output): * After changing the number of CoreXL FW instances * After changing the physical interfaces on the machine * After changing the number of CPU cores, to which the *fwk* processes on VSX are assigned | *** ** * ** *** (7) Examples {#Examples} ------------------------ Show / Hide example #1 1. Output of the '`top`' command shows that '`fw_worker_X`' processes constantly consume the CPU at 100%: ``` Cpu0 : 3.3%us, 9.6%sy, 0.0%ni, 0.6%id, 0.0%wa, 0.0%hi, 86.5%si, 0.0%st Cpu1 : 7.6%us, 16.6%sy, 0.0%ni, 0.5%id, 0.0%wa, 1.0%hi, 74.2%si, 0.0%st Cpu2 : 3.0%us, 14.6%sy, 0.0%ni, 0.6%id, 0.0%wa, 0.0%hi, 81.8%si, 0.0%st Cpu3 : 3.0%us, 11.3%sy, 0.0%ni, 0.5%id, 0.0%wa, 0.0%hi, 85.2%si, 0.0%st Cpu4 : 1.0%us, 7.6%sy, 0.0%ni, 0.7%id, 0.0%wa, 0.0%hi, 90.7%si, 0.0%st PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4083 admin 15 0 0 0 0 S 2 100 101:01.47 fw_worker_1 4019 admin 15 0 0 0 0 S 2 100 101:01.47 fw_worker_2 4023 admin 15 0 0 0 0 S 2 100 101:01.47 fw_worker_5 ``` 2. The output of the '`fwaccel stats -s`' command shows poor acceleration ratios and a high percentage of packets that are passing through *Medium path* (PXL): ``` Accelerated conns/Total conns : 364/13215 (2%) Delayed conns/(Accelerated conns + PXL conns) : 48/12023 (0%) Accelerated pkts/Total pkts : 18252/564927 (3%) F2Fed pkts/Total pkts : 36776/564927 (6%) PXL pkts/Total pkts : 509899/564927 (90%) QXL pkts/Total pkts : 0/564927 (0%) ``` 3. The output of the '`fwaccel stat`' command shows that '`Accept Templates`' are disabled after specific rule: ``` Accept Templates : disabled by Firewall disabled from rule #251 ``` 4. Thorough analysis of Firewall's Connections Table (using '`connstat`' utility from [sk85780](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk85780)) during the issue (\~99800 concurrent connections) showed that most matched rules are lower than Rule #251 (after which SecureXL Accept Templates are disabled): ``` Top 10 Rules: ============= Rule: 342 Hits: 71463 Rule: 343 Hits: 16732 Rule: 356 Hits: 4488 Rule: 354 Hits: 3126 Rule: 216 Hits: 2249 Rule: 361 Hits: 2003 Rule: 357 Hits: 1536 Rule: 51 Hits: 722 Rule: 25 Hits: 666 Rule: 362 Hits: 426 ``` 5. The rule base was optimized per [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578) 6. Performance has improved greatly after this change. 7. The output of '`fwaccel stats -s`' command showed that '`Accept Templates`' are now disabled after much lower rule: ``` Accept Templates : disabled by Firewall disabled from rule #396 ``` 8. The following steps were taken to decrease the percentage of packets that are passing through *Medium path* (PXL): 1. Current IPS profile: * 2400 protections in '`Detect`' mode * 870 protections in '`Prevent`' mode 2. Created a new IPS profile according to the environment: * 15 protections in '`Detect`' mode * 570 protections in '`Prevent`' mode * All other protections were set to '`Inactive`' 3. In the Application Control \& URL Filtering policy, disabled the "Any - Any - Allow" rule. 4. In the Anti-Virus \& Anti-Bot configuration, excluded networks, whose traffic does not have to be inspected. 9. Performance has improved greatly after the overall optimization: * Before the optimization: ``` Accelerated pkts/Total pkts : 18252/564927 (3%) PXL pkts/Total pkts : 509899/564927 (90%) ``` * After the optimization: ``` Accelerated pkts/Total pkts : 2454970439/2821805103 (87%) PXL pkts/Total pkts : 282180510/2821805103 (10%) ```

Show / Hide example #2 Relevant outputs were collected during a period of time from Security Gateway with the help of the shell script that runs relevant commands every 1 second. All outputs were analyzed and correlated with each other. 1. The output of the '`vmstat`' command showed that CPU time is consumed by: * IOWait : min 0% , average 36.6% , **max 82%** * Kernel Space : min 11% , average 16.2% , **max 45%** * User Space : min 0% , average 1.8% , **max 36%** This leaves IDLE at **min 0%** , average 45.3% , max 89% *Example*: ``` procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st ...... 1 7 334816 708020 820 37388 2152 0 3280 4 18012 17602 0 13 7 80 0 ...... 0 9 334348 652800 988 29420 1664 0 3944 4 21575 19725 1 13 5 80 0 ...... 1 12 291604 723400 884 29696 196 0 4272 12 20799 18273 5 18 0 77 0 ``` 2. The output of the '`cat /proc/stat`' command showed that CPU time is consumed by: * User Space : min 0% , average 1% , max 7% * Kernel Space : min 0% , average 1% , max 3% * Idle : **min 16%** , average 34% , max 54% * IOWait : min 32% , average 48% , **max 69%** * IRQ : min 0% , average 1% , max 1% * SoftIRQ : min 11% , average 14% , max 19 3. The output of the '`top`' command showed: * fw_worker_0 / fw_worker_1 / fw_worker_2 - constantly consume CPU at \~15-20% * fw_worker_X spike to 20-25% * **'fw' spikes up to 50, 60, 80%** * fwssd , in.asessiond , in.aufpd , in.ufclnt , mdq , igwd , sds , stormd , dtps *Example*: ``` PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6283 root 19 0 54688 16m 12m D 87 0.8 0:00.56 fw 2283 root 15 0 0 0 0 R 30 0.0 4844:09 fw_worker_2 2209 root 15 0 0 0 0 R 16 0.0 5254:57 fw_worker_1 2144 root 15 0 0 0 0 S 14 0.0 5227:59 fw_worker_0 ``` 4. The output of the '`ps auxwf`' command correlated to the output of '`top`' command showed that during the spikes of CPU load by '`fw`', the CPD daemon calls the '`sim affinity -c`' command, which calls a series of Check Point shell scripts that calculate the current load by running various '`fw ctl`' commands. These commands process high amount of data - and this causes the spike in CPU load. The CPD daemon calling the '`sim affinity -c`' command shows that SIM Affinity is configured in Automatic mode: ``` root 15829 0.1 1.2 231704 25748 ? Ssl Jul23 49:45 \_ cpd root 5153 0.0 0.0 2832 1088 ? S 12:19 0:00 | \_ sim affinity -c root 5212 0.0 0.0 1236 564 ? S 12:19 0:00 | \_ sh -c $FWDIR/scripts/fwaffinity_used_cpus > /tmp/sim_fw_cpus.tmp root 5213 0.0 0.0 1236 564 ? S 12:19 0:00 | \_ /bin/sh /opt/CPsuite-R75.20/fw1/scripts/fwaffinity_used_cpus root 5214 0.0 0.0 1236 424 ? S 12:19 0:00 | \_ /bin/sh /opt/CPsuite-R75.20/fw1/scripts/fwaffinity_used_cpus root 5215 0.0 0.0 1236 372 ? S 12:19 0:00 | \_ /bin/sh /opt/CPsuite-R75.20/fw1/scripts/fwaffinity_used_cpus root 5216 0.0 0.8 55636 18080 ? S 12:19 0:00 | \_ fw ctl affinity -l -v root 5226 0.0 0.0 1236 552 ? S 12:19 0:00 | | \_ sh -c /bin/netstat -n 2>&1 root 5227 0.0 0.0 1668 560 ? R 12:19 0:00 | | \_ /bin/netstat -n root 5217 0.0 0.0 1644 476 ? S 12:19 0:00 | \_ grep -E -v ^Interface.*: root 5218 0.0 0.0 2028 644 ? S 12:19 0:00 | \_ awk -F: {print $2} root 5219 0.0 0.0 1628 464 ? S 12:19 0:00 | \_ sed -e s/CPU// root 5220 0.0 0.0 1604 372 ? S 12:19 0:00 | \_ tr [:space:] \n root 5221 0.0 0.0 1644 468 ? S 12:19 0:00 | \_ grep -v ^[:space:]*$ root 5222 0.0 0.0 35424 508 ? S 12:19 0:00 | \_ sort root 5223 0.0 0.0 1592 384 ? S 12:19 0:00 | \_ uniq ``` 5. The SIM Affinity was configured from Auto Mode to Static Mode: **`[Expert@HostName]# sim affinity -s`** *** ** * ** *** (8) Related documentation {#Related documentation} -------------------------------------------------- Show / Hide related documentation * SecureXL (Performance Pack): * [Performance Pack Administration Guide](https://support.checkpoint.com/product/530) - Chapter 'SecureXL' * [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578) * CoreXL: * [Performance Tuning Administration Guide](https://support.checkpoint.com/product/530) - Chapter 'CoreXL'. * [sk61701 - CoreXL Known Limitations in R80.30 and lower](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk61701) * Multi-Queue: * [Performance Tuning Administration Guide](https://support.checkpoint.com/product/530) - Chapter 'Multi-queue'. * Gaia OS: * [Gaia Administration Guide](https://support.checkpoint.com/product/73). * [Command Line Interface Reference Guide](https://support.checkpoint.com/product/73). * [sk93000 - SMT (HyperThreading) Feature Guide](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk93000) * [Hardware Compatibility List](https://www.checkpoint.com/support-services/hcl/). *** ** * ** *** (9) Related solutions {#Related solutions} ------------------------------------------ Show / Hide related solutions * **Performance:** * [sk105119 - Best Practices - VPN Performance](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk105119) * [sk101878 - CPView Utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878) * [sk103212 - Traffic analysis using 'CPMonitor' tool](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk103212) * [sk85780 - How to use the 'connstat' utility](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk85780) * [sk106410 - How to check Top Rule Hits on Security Gateway](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk106410) * [](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk33781)[sk43733 - How to measure CPU time consumed by IPS protections](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk43733) * [sk67560 - How to export History Report from SmartView Monitor](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk67560) * [sk100766 - Simulation of low memory resources on Security Gateway - random failing of memory allocations](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk100766) * [sk36846 - Interpreting high SoftIRQ values](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk36846) * [sk101221 - TCP state logging](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101221) * [sk116740 - "top" command on VSX Gateway shows that FWK process consumes CPU at more than 100%](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk116740) * [sk43772 - 'kernel: neighbour table overflow' appears repeatedly in /var/log/messages files](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk43772) * [sk92372 - How to change the size of IPv6 Neighbor cache table](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk92372) * **SecureXL:** * [sk153832 - ATRG: SecureXL for R80.20 and higher](https://support.checkpoint.com/results/sk/sk153832) * [sk32578 - SecureXL Mechanism](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk32578) * [sk71200 - SecureXL NAT Templates](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk71200) * [sk61962 - SMP IRQ Affinity on Check Point Security Gateway](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk61962) * [sk104468 - How to exclude traffic from SecureXL](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk104468) * **CoreXL:** * [sk98737 - ATRG: CoreXL](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk98737) * [sk61962 - SMP IRQ Affinity on Check Point Security Gateway](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk61962) * [sk35990 - How Connections Table limit capacity behaves in CoreXL](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk35990) * [](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk43443)[sk42096 - Cluster member is stuck in 'Ready' state](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk42096) * [](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk63330)[sk65463 - 'Peak' number of connections - discrepancy between the output of 'fw tab -t connections -s' command and the output of 'fw ctl pstat' command when CoreXL is enabled](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk65463) * **SMT (HyperThreading):** * [sk93000 - SMT (HyperThreading) Feature Guide](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk93000) * **Check Point Infrastructure:** * [sk97638 - Check Point Processes and Daemons](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk97638) * [sk52421 - Ports used by Check Point software](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk52421) * [sk97443 - Support Debug Tools](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk97443) * [sk65133 - Connections Table Format](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk65133) *** ** * ** *** (10) Revision history {#Revision history} ----------------------------------------- Show / Hide revision history > |-----------------|-----------------------------------------------------------------------------------------------| > | Date | Description | > | 11 January 2026 | Removed EOS versions and OS other than Gaia. | > | 26 Nov 2024 | Updated section (3-6) SMT (HyperThreading) | > | 26 Apr 2023 | Improved formatting Removed links to outdated SK articles Removed links to outdated documents | > | 02 Oct 2017 | Added additional related solution (sk120131) | > | 19 June 2017 | Updated definition of "Medium path (PXL)" | > | 18 June 2017 | Added additional related solution (sk115072) | > | 20 Apr 2017 | Added additional related solution (sk110351) | > | 15 Apr 2017 | Added additional related solution | > | 08 Feb 2017 | Added additional related solutions | > | 21 Nov 2016 | Improved HTML design | > | 04 Aug 2016 | Added instructions for the `sar` command | > | 11 Jan 2016 | Corrected the syntax for using the `pstack` script | > | 11 Oct 2015 | Updated definition of "Medium path (PXL)" | > | 04 July 2015 | Added additional related solution | > | 11 June 2015 | Added additional related solution | > | 13 May 2015 | Added additional related solution | > | 08 Apr 2015 | Added additional related solution | > | 02 Mar 2015 | Updated the best practices for Application Control \& URL Filtering optimization | > | 10 Feb 2015 | Added additional related solutions | > | 14 Jan 2015 | Corrected the the formula for calculating the maximum number of concurrent connections | > | 20 Nov 2014 | Corrected the best practices for Application Control \& URL Filtering optimization | > | 05 Nov 2014 | Added additional related solutions | > | 08 Sep 2014 | Added CPView Utility | > | 02 July 2014 | Added additional related solutions | > | 19 May 2014 | Added additional related solutions | > | 13 May 2014 | Added additional related solutions | > | 12 Apr 2014 | Added additional related solutions | > | 16 Mar 2014 | Added additional related solutions Added SecureXL Penalty box feature to SecureXL Limitations | > | 12 Mar 2014 | Added instructions for 'pstack' shell script | > | 11 Mar 2014 | First release of this document | --- # Agent Instructions This content is from the Check Point Support Center (https://support.checkpoint.com), the official knowledge base for Check Point cybersecurity products. ## Navigating This Knowledge Base - **Complete index**: [llms.txt](https://support.checkpoint.com/llms.txt) - **All SK articles**: [SecureKnowledge Sitemap](https://support.checkpoint.com/sitemaps/secureknowledge-sitemap-index.xml) - **SK article URL pattern**: `https://support.checkpoint.com/results/sk/{skId}` - **Markdown responses**: AI bot User-Agents automatically receive `text/markdown` content