Log Records

Log record types

There are two relevant log record types:

  1. a fairly simple one created by the HCI Search App, as well as the HCI Workflow Designer App - recording when an user initiated a search:

    • Search App:

      Feb 14 11:04:55 192.168.0.221 - admin [14/Feb/2022:10:03:00 +0000] "POST /api/search/query/federated HTTP/1.1" 200 13544

    • Workflow Designer App:

      Feb 14 11:07:32 192.168.0.221 - admin [14/Feb/2022:10:07:07 +0000] "POST /api/workflow/indexes/5bd7d677-504d-4410-b250-254555e5fd91/rawQuery HTTP/1.1" 200 3795

  2. a more complex one created by the HCI Index engine (Solr) - recording search requests directed at the Solr indexes by the Search or Workflow Designer App on behalf of an user:

    Feb 14 11:04:55 INFO  - 2022-02-14 10:03:00.015; [c:AAA_IDX s:shard1 r:core_node3 x:AAA_IDX_shard1_replica1] org.apache.solr.core.SolrCore; [AAA_IDX_shard1_replica1]  webapp=/solr path=/select params={facet.field=Content_Type&facet.field=HCI_dataSourceName&facet.field=aaa_Resultat_avlast&df=HCI_text&distrib=false&_stateVer_=AAA_IDX:84&fl=HCI_id&fl=score&shards.purpose=20&start=0&fsv=true&f.aaa_Resultat_avlast.facet.limit=160&f.Content_Type.facet.limit=160&shard.url=http://192.168.0.207:31009/solr/AAA_IDX_shard1_replica1/&rows=10&f.HCI_dataSourceName.facet.limit=160&version=2&q=Wankelmotor&f.Content_Type.facet.mincount=0&f.HCI_dataSourceName.facet.mincount=0&f.aaa_Resultat_avlast.facet.mincount=0&NOW=1644832980010&isShard=true&facet=true&wt=javabin} hits=0 status=0 QTime=1

Those two record types are locally stored into two different database tables:

  • the searchapp table and

  • the solrquery table

Storing in a local database is done for two reasons:

  • elimination of duplicate records

  • keeping them for a while (31 days per default) in case sending to syslog went wrong, to allow for a retry

The database tables structure can be found in the database description.

Log record relationship

Note

The only obvious relationship between the two record types is their timestamps.

To build an approximate relation between user activity and search queries against the HCI Index, one needs to start with a record of the first type and take its timestamp as a search criteria for records of the second type, using some fuzziness.

_images/searchapprecord.png

Sample searchapp record

Highlighted are the username and the point in time when the search was initiated.

That said, one would search for records of the second type with a timestamp in the range of the first records timestamp to maybe up to 1000 milliseconds later.

This will result in a number of records of the second type for each record from the first type - ideally one per Solr shard configured for the index.

_images/solrqueryrecord.png

Sample solrquery record

Here the timestamp is highlighted, closely matching the timestamp seen in the first sample record, as well as the search query (q=Wunderbar) and a filter that was applied to limit the search (fq=tom.evans@goodweather.com). The name of the index being queried is highlighted in green, btw.