HCI Logging Configuration

Note

HCI 2.0 replaces Solr 6 with Solr 8 for new installs, and adds Solr 8 for HCIs installed with a version < 2.0.

In case HCI was upgraded from v1.x to v2.x, there will be the Solr 6 and Solr 8 services running, at least until all indexes have been migrated to Solr 8.

Until all Indexes have been migrated to Solr 8, the logging configuration described in this chapter must be applied to both Solr services. Out of the box, the Index service (Solr 6) uses port 8983, while the Solr 8 Index services uses port 28983 in the mentioned upgrade scenario. In an HCI cluster that was initially installed with version 2.0 or higher, the Solr 8 Index services uses port 8983, as usual.

You can find the used ports for both Solr 6 and Solr 8 using the Admin App > Services > Index (or Solr 8 Index) > Network. Look out for the value of PRIMARY_PORT.

Temporary Solution

Warning

This seems to be a bit unstable, at least in HCI v2.2.0.8. If you don’t get all the log records expected, use the permanent solution!

Solr logging can be changed temporarily (till next SOLR restart) using the Solr dashboard or through a REST command. Different from the permanent setting described later, it is enough to run this change against a single instance, as it will be propagated by Solr itself.

Either:

  • Use a Browser, navigate to http://<HCI-instance-IP>:<PRIMARY_PORT>/solr/#/~logging/level and change ‘’org.apache.solr.core.SolrCore.Request’’ to INFO.

  • Or use curl to make a REST call:

    curl -s http://<instance>:<PRIMARY_PORT>/solr/admin/info/logging --data-binary \
"set=org.apache.solr.core.SolrCore.Request:INFO&wt=json"

    Tip

    You can check the actual setting with these requests:

    Solr 6:

    # curl -s http://<instance>:<PRIMARY_PORT>/solr/admin/info/logging | xmllint --format - | grep -e 'SolrCore<' -e 'SolrCore.Request<' -A 2
      <str name="name">org.apache.solr.core.SolrCore</str>
      <str name="level">INFO</str>
      <bool name="set">true</bool>

    Solr 8: (pls. note the difference!)

    # curl -s http://<instance>:<PRIMARY_PORT>/solr/admin/info/logging | grep -e 'SolrCore"' -e 'SolrCore.Request"' -A 2
      "name":"org.apache.solr.core.SolrCore",
      "level":"WARN",
    {
      "name":"org.apache.solr.core.SolrCore.Request",
      "level":"INFO",
    {

Permanent Solution

Warning

Permanent means permanent till next HCI upgrade. That said, each time HCI gets upgraded, check the setting and re-apply if needed!

Warning

The permanent configuration does not work for Solr 8 in HCI 2.1.0.11, fixed in HCI 2.2.08.

Note

The procedure described in this section needs to be applied to each HCI instance!

The permanent solution changes the logging configuration’s template file, which is copied to the service’s private log4j.properties in case the private copy doesn’t exist when the service starts.

This is how to apply the change:

  • Shutdown HCI on all instances:

    sudo systemctl stop HCI.service

  • Remove the log configuration file from the containers data folder (again, on all instances):

    rm /opt/hci/data/com.hds.ensemble.plugins.service.solr/<UUID>/log4j.properties

  • Append the needed settings to the logging configuration template:

    sudo echo 'log4j.logger.org.apache.solr.core.SolrCore.Request=INFO' \
>> <install_path>/services/com.hds.ensemble.plugins.service.solr/<HCI-version>/package/log4j.properties.src

  • Start HCI on all instances:

    sudo systemctl start HCI.service

Alternative solution

Warning

This seems to be a bit unstable, at least in HCI v2.2.0.8. If you don’t get all the log records expected, use the permanent solution!

In case the permanent solution can’t be applied, the temporary method may be used, triggered to run every so many minutes. This might leave some blind spots, where the settings were just not changed, as the systemd timer might trigger a while after HCI was started.

/etc/systemd/system/hcisolrlogging.service

[Unit]
Description=HCI Solr logging settings refresh
Wants=hcisolrlogging.timer

[Service]
Type=simple
WorkingDirectory=/tmp
# Un-comment the next line if Solr 6 Index is still in use, and remove it once the Solr 6 Index
# service has been decommissioned from use in HCI!
# ExecStartPre=/usr/bin/curl -s http://<HCI-instance-IP>:<solr6_PRIMARY_PORT>/solr/admin/info/logging -o /dev/null --data-binary "set=org.apache.solr.core.SolrCore.Request:INFO&wt=json"
ExecStart=/usr/bin/curl -s http://<HCI-instance-IP>:<solr8_PRIMARY_PORT>/solr/admin/info/logging -o /dev/null --data-binary "set=org.apache.solr.core.SolrCore.Request:INFO&wt=json"

[Install]
WantedBy=multi-user.target

/etc/systemd/system/hcisolrlogging.timer

[Unit]
Description=Runs HCI Solr logging refresh service
Requires=hcisolrlogging.service

[Timer]
Unit=hcisolrlogging.service
# start the service hourly
OnCalendar=hourly

[Install]
WantedBy=timers.target

Once the unit files are in place, activate the timer by running:

# systemctl enable --now hcisolrlogging.timer

Check when it will run the next time:

# systemctl list-timers