> Source: [sk163260](https://support.checkpoint.com/results/sk/sk163260) # sk163260 - "Log server is disconnected" message in SmartConsole | Property | Value | |----------|-------| | Solution ID | sk163260 | | Date Created | 2019-10-24 | | Last Modified | 2022-12-19 | | Technical Level | General | | Products | Security Management Server, SmartConsole, Multi-Domain Security Management Server | | Versions | R81.20, R81.10 (EOS), R81 (EOS), R81.20, R81.10 (EOS), R81 (EOS), R81 (EOS), R81.10 (EOS), R81.20 | | OS | Gaia | ## Solution This article describes different situations in which you might see the "`Log Server is disconnected (IP: )`" message in SmartConsole \> "Logs \& Monitor" view \> "Logs" tab. These are: 1. **"Log Server is disconnected" appears on the "Logs" tab when SmartConsole is connected to a dedicated SmartEvent / Log Server** 2. **"Log Server is disconnected" appears on the "Logs" tab when opening logs from a Secondary Management Server / Log Server** 3. **"Log Server is disconnected" appears on the "Logs" tab after the Security Management Server upgrade to R80 or higher** 4. **"Log Server is disconnected" appears on the "Logs" tab after a migration of a Multi-Domain Server** 5. **"Log Server is disconnected" appears on the "Logs" tab for one of the Domain Log Servers** 6. **"There are no log servers available, check your log server configuration" message in SmartConsole connected to a Domain Management Server** 7. **"There are no log servers available, check your log server configuration" message in SmartLog GUI after moving log index files to an external storage** 8. **"There are no log servers available, check your log server configuration" message in SmartLog GUI when Log Server is behind NAT** 9. **"There are no log servers available, check your log server configuration" message in SmartConsole because of an empty "empty core.properties" file** Click Here to Show the Entire Article (1) "Log Server is disconnected" appears on the "Logs" tab when SmartConsole is connected to a dedicated SmartEvent / Log Server {#1} ------------------------------------------------------------------------------------------------------------------------------------- > **Symptoms:** > > * "*Log Server is disconnected (IP: \)*" error appears on the SmartConsole "Logs" tab when SmartConsole is connected to a dedicated SmartEvent / Log Server. > > * SmartConsole "Logs" tab does not show logs when connected to the Security Management Server / Domain Management Server. > > * "*Query Failed*" error appears in SmartView portal on the Security Management Server / Domain Management Server. > > Show / Hide this section > **Cause:** > > Log Indexing is disabled on the Security Management Server / Domain Management Server. > > **Solution:** > > Enable Log Indexing on the Security Management Server / Domain Management Server: > > **Important** - Enabling Log Indexing on the Security Management Server / Domain Management Server increases the resource use. > > 1. Connect with SmartConsole to the Security Management Server / Domain Management Server. > > 2. Open the Management Server object. > > 3. In the left tree, click **Logs**. > > 4. Select **Enable Log Indexing**. > > ![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk119335/l11709100102.jpg) > 5. Click **OK**. > > 6. In the top left corner of SmartConsole, click the **Menu** button \> click **Installing database**. > > 7. Select the Security Management / Domain Management Server object \> click **Install** \> click **Close**. (2) "Log Server is disconnected" appears on the "Logs" tab when opening logs from a Secondary Management Server / Log Server {#2} --------------------------------------------------------------------------------------------------------------------------------- > **Symptoms:** > > * In SmartConsole \> **Logs \& Monitor** view \> **Options** \> **File** \> **Open Log File**, the Secondary Security Management Server / Domain Management Server / Domain Log Server does not appear > > * "*Problems have occurred during search. Click here for details.*" error appears when trying to open logs from the problematic Security Management Server / Domain Management Server / Domain Log Server. > > * "*Log Server is disconnected (IP: \)* " message appears after clicking "*Click here for details*". > > * A Security Gateway between the Management Server and Log Server drops the traffic on the TCP port 8211 by a cleanup rule or by some other drop rule in the policy. > > Show / Hide this section > **Cause:** > > Starting in R80, the Management Server connects to its Log Server over the TCP port 8211 to view logs from that Log Server. > > In most environments, it is enough to select **Global Properties** \> **FireWall** \> **Accept control connections** for the Security Gateway to allow this traffic. > > However, in more complex environments (in which NAT is used and in which the Multi-Domain Security Management Server and Multi-Domain Log Server are in different data centers), explicit rules are required to allow this traffic. > > **Solution:** > > 1. Add an explicit Access Control rule to allow traffic on the TCP port 8211 between the Management Server and Log Servers. > > 2. Install the Access Control Policy. (3) "Log Server is disconnected" appears on the "Logs" tab after the Security Management Server upgrade to R80 or higher {#3} ----------------------------------------------------------------------------------------------------------------------------- > **Symptoms:** > > * "*Log server is disconnected (IP: \)* " message appears in SmartConsole \> **Logs \& Monitoring** view \> **Logs** tab after an upgrade of the Security Management Server to R80 or higher. > > * The **Logs \& Monitoring** view shows the old IP address of the Log Server. > > * There is no related object with this IP address in SmartConsole, but this object name and the IP address exist in the management database. > > Show / Hide this section > **Cause:** > > A *CpmiMds* object with the cpmitable ''*mdss*'' was created during the upgrade, although this is a single domain (Security Management) installation and not a Multi-Domain Management Server environment. > > **Solution:** > > [Contact Check Point Support](https://www.checkpoint.com/support-services/contact-support/) to get a Hotfix for this issue (ID 02407060). > > A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix. > For faster resolution and verification, collect [CPinfo](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk92739) file from the Management Server involved in the case. > > **Hotfix installation instructions:** > > Refer to [sk168597 - How to install a Hotfix](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk168597). (4) "Log Server is disconnected" appears on the "Logs" tab after a migration of a Multi-Domain Server {#4} ---------------------------------------------------------------------------------------------------------- > **Symptoms:** > > * "*Log server is disconnected (IP: \)* " message appears in SmartConsole \> **Logs \& Monitoring** view \> **Logs** tab when viewing logs from a dedicated Log Server / SmartEvent Server. > > * Object of the dedicated Log Server / SmartEvent Server appears with the Yellow/Caution symbol in SmartConsole. > > * The *$RTDIR/log/RFL.log* file in the Domain Management Server context contains this line: > > `javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown.` > > Show / Hide this section > **Cause:** > > During a migration of a Multi-Domain Server, the Domain Management Server had the incorrect IP address in the CRL distribution point. > > **Solution:** > > Edit the CRL using on the Primary Domain Management Server: > > **Important** - This procedure requires the restart of all the processes on the Multi-Domain Server / Multi-Domain Log Server. Take a backup or snapshot of the server as described in [sk108902](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk108902). > > **Part 1** > > 1. Connect to the command line on the Multi-Domain Server. > > 2. Log in to the Expert mode. > > 3. If the Primary Domain Management Server is not Active, change its state to Active: > > 1. Go to the context of the Primary Domain Management Server: > > `mdsenv ` > 2. Get the Domain UID and assign it to a variable: > > ``DOMAIN_UID=`cpprod_util CPPROD_GetValue FW1 DomainUID 1``` > 3. Examine the Domain UID: > > `echo $DOMAIN_UID` > 4. Get the Domain IP Address and assign it to a variable: > > ``CUSTOMER_IP=`$MDSVERUTIL CMAIp -n ``` > 5. Examine the Domain IP Address: > > `echo $CUSTOMER_IP` > 6. Configure the Certificate Authority's (CA) Fully Qualified Domain Name (FQDN): > > `cp_conf ca fqdn $CUSTOMER_IP $DOMAIN_UID 1` > 4. Go to the MDS context: > > `mdsnev` > 5. Restart all Check Point services: > > `mdsstop;mdsstart` > > **Part 2** > > In SmartConsole, reset the SIC between the Domain Management Server and the dedicated Log Server / SmartEvent Server. > > **Part 3** > > On all Standby Domain Management Servers and Domain Log Servers in the same domain, renew SIC: > > 1. Go to the context of the Primary Domain Management Server: > > `mdsenv ` > 2. Back up the existing SIC certificate: > > `cp -v $CPDIR/conf/sic_cert.p12{,_BKP}` > 3. Create a new SIC certificate: > > `sicRenew -d` > 4. Restart all Check Point services: > > `mdsstop;mdsstart` (5) "Log Server is disconnected" appears on the "Logs" tab for one of the Domain Log Server {#5} ------------------------------------------------------------------------------------------------ > **Symptoms:** > > * "*Log server is disconnected (IP: \)* " message appears in SmartConsole \> **Logs \& Monitoring** view \> **Logs** tab for one of the Domain Log Server. > > * "*File: \[\\] does not exist, file is required to support Log Server connectivity*" message in the Pre-Upgrade Verifier (PUV) report during an upgrade of a Dedicated Log Server. > > Show / Hide this section > **Cause:** > > The required SIC trust files must reside in the *$CPDIR/conf/* directory on the Dedicated Log Server: > > * *SIC_DB* > > * *crl_distribution_point.map* > > * *sic_ca_bundle.pem* > > **Solution:** > > 1. Connect to the command line on the Multi-Domain Management Server which hosts the active Domain Management Server. > > 2. Log in to the Expert mode. > > 3. Create the required SIC trust files: > > `$MDS_FWDIR/scripts/cpm.sh -tm -op reset -d all -sd` > 4. Verify that all the required trust files are located on the Dedicated Log Server. (6) "There are no log servers available, check your log server configuration" message in SmartConsole connected to a Domain Management Server {#6} -------------------------------------------------------------------------------------------------------------------------------------------------- > **Symptoms:** > > * "*There are no log servers available, check your log server configuration* " error in SmartConsole \> **Logs \& Monitoring** view \> **Logs** tab when SmartConsole is connected to a Domain Management Server. > > ![](https://sc1.checkpoint.com/sc/SolutionsStatics/sk121054/sk121054-11710260318.png) > * "*Log server is disconnected (IP: \)* " message appears in SmartConsole \> **Logs \& Monitoring** view \> **Logs** tab . > > Show / Hide this section > **Cause:** > > The RFL component does not run a ping request to the SOLR database because the previous ping task is still running (because the SOLR database did not return a response or the request got stuck). > > **Solution:** > > 1. To verify that the solution below is relevant, run the following from Expert mode: > > 1. Run: > > `SmartEventSetDebugLevel rfl debug` > 2. Wait for approximately 2-3 minutes. > > 3. Run: > > `SmartEventSetDebugLevel rfl info` > 4. Run: > > `grep "the previous one is still running" $RTDIR/log/RFL.log*` > 5. Search for an output similar to this: > > `DEBUG [...] com.checkpoint.rfl.solr.monitoring.ServerConnectivityTask.call:32 - the connectivity task for [ObjID: [...]` > > `IP Address: [...], Port: [...], Local IP: [...], Connecting IP Address: [...], Enable SSL: [...],` > > `Enable Remote SSL: [...], SmartEvent: [...], Primary Management: [....]] is not started because `**the previous one is still running** > 2. If you see a similar output, restart the RFL component: > > 1. `$RTDIR/scripts/stopRfl.sh` > > 2. `$RTDIR/scripts/startRfl.sh` > > 3. If the issue persists, restart all Check Point services on the Security Management Server: > > `cpstop ; cpstart` (7) "There are no log servers available, check your log server configuration" message in SmartLog GUI after moving log index files to an external storage {#7} -------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Symptoms:** > > * "*There are no log servers available, check your log server configuration* " message in SmartLog GUI after applying [sk66003](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk66003) (Section (4) - Step 3) to move log index files to an external storage. > > * The *$RTDIR/log/RFL.log* file contains these lines: > > `Log Server is disconnected` > > `Error code: [NO_SME_AVAILABLE]` > > `Error message: [There are no SmartEvent Servers available]` > * The *$RTDIR/log_indexer/log/log_indexer.elg* file contains these lines: > > `[log_indexer XXX XXX]@HostName[Date Time] JavaBinCodec::unmarshall Invalid version (expected , but <) or the data in not in 'javabin' format` > > `[log_indexer XXX XXX]@HostName[Date Time] SolrClient::Implementation::ParseJavaBinReply Bad Java bin format` > > Show / Hide this section > **Cause:** > > The '*template*' index is the main index of the SOLR database and is the baseline for all other indexes. The absence of this template index causes failures in queries, indexing, and pings for this local SOLR. > > When you are connected with SmartConsole to the Multi-Domain Log Server (or Domain Log Server) and run a log query, SmartConsole GUI sends the query to the RFL component that checks if the SOLR database is responsive, and then sends the query to the SOLR database. > > In this case, the SOLR database is not responsive. > > **Solution:** > > 1. Search for the '*template*' directory on the Management Server. > > 2. If the '*template* ' directory is not located inside the *$RTDIR/log_indexes/* directory, then copy the '*template* ' directory to the *$RTDIR/log_indexes/* directory. > > 3. Search for the '*core.properties*' file on the Management Server. > > 4. If the '*core.properties* ' file is not located in the *$RTDIR/log_indexes/template/* directory, then copy the '*core.properties* ' file to the *$RTDIR/log_indexes/template/* directory. > > 5. Restart the Log Indexer: > > `evstop ; evstart` (8) "There are no log servers available, check your log server configuration" message in SmartLog GUI when Log Server is behind NAT {#8} ---------------------------------------------------------------------------------------------------------------------------------------- > **Symptoms:** > > * "*Problems have occurred during search*" error in SmartLog when querying logs when connected to a dedicated Log Server. > > * "*Log server is disconnected (IP: \)*" appears in the details for the above error. > > * The Log Server is located behind a Security Gateway that applies NAT. > > A "dummy" Check Point Host object was created in SmartConsole to represent the NATed IP of the Log Server. > > This "dummy" Check Point Host object has the "Logging and Status" Software Blade enabled. > * The */var/log/dump/usermode/* directory on the dedicated Log Server contains core dump files for SOLR. These core dump files were created during the time the Log Server became inaccessible in SmartLog GUI. > > * Analysis of the SOLR core dump files shows the crash was due to an out-of-memory (OOM) error. > > Show / Hide this section > **Cause:** > > SOLR connects to all Check Point Host objects with the "Logging and Status" Software Blade enabled. > > Eventually, SOLR runs out of memory from the failed connections to the objects and experience a crash. > > As a result, the Log Server becomes unavailable in SmartLog until the SOLR process recovers. > > **Solution:** > > Follow one of these possible procedures: > > * Disable the "Logging and Status" Software Blade in the "dummy" Check Point Host object. > > * Remove the "dummy" Check Point Host object that represents the Log Server's NATed IP address from the SOLR configuration file: *$RTDIR/conf/solrConnectionConfig.xml* > > 1. Connect to the command line on the Management Server. > > 2. Log in to the Expert mode. > > 3. On the Multi-Domain Security Management Server, go to the context of the applicable Domain Management Server: > > `mdsenv ` > 4. Back up the current *$RTDIR/conf/solrConnectionConfig.xml* file: > > `cp -v $RTDIR/conf/solrConnectionConfig.xml{,_BKP}` > 5. Edit the current *$RTDIR/conf/solrConnectionConfig.xml* file: > > `vi $RTDIR/conf/solrConnectionConfig.xml` > 6. Remove this section in the file: > > ``` > > NATed_IP_Address > false > true > 8211 > 8210 > 1c49f484-e1b2-9440-915c-efc71a54856e > false > false > > ``` > > 7. Save the changes in the file and exit Vi editor. > > 8. Restart the SmartLog services: > > `evstop ; evstart` > 9. In SmartConsole, delete the dummy object. > > 10. Configure NAT on the Log Server as specified in [sk66381](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk66381). (9) "There are no log servers available, check your log server configuration" message in SmartConsole because of an empty "empty core.properties" file {#9} ----------------------------------------------------------------------------------------------------------------------------------------------------------- > **Symptoms:** > > * "*There are no log servers available, check your log server configuration*" message in SmartConsole. > > * The *$RTDIR/log/solr.log* file on the Management Server / Log Server contains these lines: > > `ERROR [qtp853527933-1197] org.apache.solr.servlet.SolrDispatchFilter.doFilter:358 - Error processing the request. CoreContainer is either not initialized or shutting down.` > > `WARN [qtp853527933-1197] org.eclipse.jetty.server.HttpChannel.handleException:590 - /solr/template/update javax.servlet.ServletException: javax.servlet.UnavailableException: Error processing the request. CoreContainer is either not initialized or shutting down.` > > `at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]` > > `at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114]` > * These two articles were applied, but the issue persists: > > * [sk163260 - "Log Server is disconnected" message](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk163260) > > * [sk177343 - "There are no log servers available, check your log server configuration" on all Domain Management Servers and Multi-Domain Servers](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk177343) > > Show / Hide this section > **Cause:** > > Within the directory *$RTDIR/log_indexes/* , there is at least one sub-directory that contains an empty *core.properties* file. > > **Solution:** > > **Procedure:** > > 1. Connect to the command line on the Management Server / Log Server. > > 2. Log in to the Expert mode. > > 3. Stop the log indexing: > > `$RTDIR/scripts/stopSolr.sh` > 4. Find all empty `core.properties` files (with zero size). > > Run this long command (in the "`find`" command, make sure to enter the correct version "R8..."): > > `for FILE in $(find /var/log/opt/CPrt-R81.10/log_indexes/ -name core.properties -type f) ; do SIZE=$(stat ${FILE} | grep 'Size:' | awk '{print $2}') ; if [[ $SIZE -eq 0 ]] ; then echo "ZERO size - ${FILE}" ; fi ; done` > 5. Delete all empty `core.properties` files: > > 1. Make sure the file is empty: > > `cat ` > 2. Delete the empty file: > > `rm -i ` > 6. Start the log indexing: > > `$RTDIR/scripts/startSolr.sh` > > **Example from an R81.10 Management Server:** > > `[Expert@MyMgmt:0]# $RTDIR/scripts/stopSolr.sh` > > > > `... ...` > > > > `[Expert@MyMgmt:0]#` > > > > `[Expert@MyMgmt:0]# for FILE in $(find /var/log/opt/CPrt-R81.10/log_indexes/ -name core.properties -type f) ; do SIZE=$(stat ${FILE} | grep 'Size:' | awk '{print $2}') ; if [[ $SIZE -eq 211 ]] ; then echo "ZERO size - ${FILE}" ; fi ; done` > > > > `... ...` > > > > `ZERO size - /var/log/opt/CPrt-R81.10/log_indexes/other_2022-09-24T00-00-00/core.properties` > > > > `... ...` > > > > `[Expert@MyMgmt:0]#` > > > > `[Expert@MyMgmt:0]# cat /var/log/opt/CPrt-R81.10/log_indexes/other_2022-09-24T00-00-00/core.properties` > > > > `[Expert@MyMgmt:0]# ` > > > > `[Expert@MyMgmt:0]# rm -i /var/log/opt/CPrt-R81.10/log_indexes/other_2022-09-24T00-00-00/core.properties` > > > > `rm: remove regular file '/var/log/opt/CPrt-R81.10/log_indexes/other_2022-09-24T00-00-00/core.properties'? y` > > > > `[Expert@MyMgmt:0]#` > > > > `[Expert@MyMgmt:0]# $RTDIR/scripts/startSolr.sh` > > > > `... ...` > > > > `[Expert@MyMgmt:0]#` --- # Agent Instructions This content is from the Check Point Support Center (https://support.checkpoint.com), the official knowledge base for Check Point cybersecurity products. ## Navigating This Knowledge Base - **Complete index**: [llms.txt](https://support.checkpoint.com/llms.txt) - **All SK articles**: [SecureKnowledge Sitemap](https://support.checkpoint.com/sitemaps/secureknowledge-sitemap-index.xml) - **SK article URL pattern**: `https://support.checkpoint.com/results/sk/{skId}` - **Markdown responses**: AI bot User-Agents automatically receive `text/markdown` content