Log Records =========== Log record types ---------------- There are two relevant log record types: 1. a fairly simple one created by the *HCI Search App*, as well as the *HCI Workflow Designer App* - recording **when** an user initiated a search: * *Search App*: .. code-block:: Feb 14 11:04:55 192.168.0.221 - admin [14/Feb/2022:10:03:00 +0000] "POST /api/search/query/federated HTTP/1.1" 200 13544 * *Workflow Designer App*: .. code-block:: Feb 14 11:07:32 192.168.0.221 - admin [14/Feb/2022:10:07:07 +0000] "POST /api/workflow/indexes/5bd7d677-504d-4410-b250-254555e5fd91/rawQuery HTTP/1.1" 200 3795 2. a more complex one created by the *HCI Index* engine (Solr) - recording search requests directed at the Solr indexes by the *Search* or *Workflow Designer App* on behalf of an user: .. code-block:: Feb 14 11:04:55 INFO - 2022-02-14 10:03:00.015; [c:AAA_IDX s:shard1 r:core_node3 x:AAA_IDX_shard1_replica1] org.apache.solr.core.SolrCore; [AAA_IDX_shard1_replica1] webapp=/solr path=/select params={facet.field=Content_Type&facet.field=HCI_dataSourceName&facet.field=aaa_Resultat_avlast&df=HCI_text&distrib=false&_stateVer_=AAA_IDX:84&fl=HCI_id&fl=score&shards.purpose=20&start=0&fsv=true&f.aaa_Resultat_avlast.facet.limit=160&f.Content_Type.facet.limit=160&shard.url=http://192.168.0.207:31009/solr/AAA_IDX_shard1_replica1/&rows=10&f.HCI_dataSourceName.facet.limit=160&version=2&q=Wankelmotor&f.Content_Type.facet.mincount=0&f.HCI_dataSourceName.facet.mincount=0&f.aaa_Resultat_avlast.facet.mincount=0&NOW=1644832980010&isShard=true&facet=true&wt=javabin} hits=0 status=0 QTime=1 Those two record types are locally stored into two different database tables: * the ``searchapp`` table and * the ``solrquery`` table Storing in a local database is done for two reasons: * elimination of duplicate records * keeping them for a while (31 days per default) in case sending to syslog went wrong, to allow for a retry The database tables structure can be found in the :ref:`database description `. Log record relationship ----------------------- .. Note:: **The only obvious relationship between the two record types is their timestamps.** To build an approximate relation between user activity and search queries against the *HCI Index*, one needs to start with a record of the first type and take its timestamp as a search criteria for records of the second type, using some fuzziness. .. figure:: _static/searchapprecord.png **Sample** ``searchapp`` **record** Highlighted are the username and the point in time when the search was initiated. That said, one would search for records of the second type with a timestamp in the range of the first records timestamp to maybe up to 1000 milliseconds later. This will result in a number of records of the second type for each record from the first type - ideally one per Solr shard configured for the index. .. figure:: _static/solrqueryrecord.png **Sample** ``solrquery`` **record** Here the timestamp is highlighted, closely matching the timestamp seen in the first sample record, as well as the search query (``q=Wunderbar``) and a filter that was applied to limit the search (``fq=tom.evans@goodweather.com``). The name of the index being queried is highlighted in green, btw.