Typically, most Siren dashboards have a search bar at the top for drilling down to the dashboard records that match the search.
But in this guide we will show how to create a dashboard that can act as an over-all search engine, show records that come from any of the Siren accessible indexes, get a preview, and ultimately reach the most appropriate dashboards.
With keyword highlighting, smart field preview selection, a detailed full document panel, and best in class ranking, a Siren search engine dashboard is a must have for most deployments.
As with Siren 10.0, the method described here work only for data which is stored in the Siren Elasticsearch nodes. Currently, data that is stored outside these nodes, for example accessed directly using Virtual Indexes, is not shown in this search and must instead be searched independently in their own dashboards.
Prerequisite: Have Data in the Siren Elasticsearch Nodes
The method works for any data available in Siren Elasticsearch nodes. No mandatory method or structure is required. However, consider that if you have control over the way data is loaded, then it is always convenient to have a somewhat similar schema across indexes.
For example, if we know in advance that most records will have a primary time stamp (or a strong concept, for example a "user ID" which is repeated across many indexes) then it is sensible to use the same field name consistently across indexes. This has advantages both in the Search Engine dashboards and across Siren.
In Siren Investigate, this provides consistency for users in the general dashboards when using those attributes, for example creating filters that can be pinned and will work when moved from one dashboard to another. It is also useful in creating an over-all search engine page because it enables you to use widgets (for example an over-all "time stamp" date histogram) that work well across the different indexes.
For the search engine page, you can use a "common denominator" column (in the result table) or drill-down widget to refine the search.
If no fields are purposely the same (have the same type and meaning) across indexes, the search engine will still work and enables you to restrict by content and at a minimum by the name of the originating index.
Consider collision in field names; fields will still be shown but ordering and aggregations on these fields may cause problems. (review technically also, not sure)
Configure a * Index Pattern with Exclusions
The next step is to configure a "*" index pattern. When doing this, it is important to add exclusions so that the index pattern will not take indexes which are private to Siren or that might belong to other users of the nodes outside Siren Investigate.
The exclusion pattern can be set in the advanced settings.
At this point, ensure you have a saved search associated with this index pattern. In Siren 10, a saved search is created automatically after the index pattern is created. You may want to rename it to "*" or "search over all"
Creating the Search Over All Dashboard
The "search over all" dashboard is a normal dashboard that has the "search over all" saved search as its core dashboard (this concept needs to be possibly better defined, linked to the other docs). Here is an example configuration:
If you have more shared fields across indexes, for example a time stamp, then more common widgets such as a timeline histogram for the number documents (possibly with a bar split on the "type" of document") could be an interesting addition.
Using the Dashboard and User Training
Typically, the dashboard works as the user expects, with some caveats:
- Search uses the same rules as the other dashboards. Default Siren configurations have an OR logic which might be unfamiliar to some users. However, the OR logic is useful because it enables a more expressive query (for example, you can copy and paste a paragraph of text and find documents that contain just a few words from it). Typically, the ranking returns the most significant results first, providing a good overall user experience.
- To select one or more documents and go to the specific dashboard, you must:
- The top right time filter has no effect unless a time filter variable is selected on the * index pattern. You should only do so if all the documents have a time stamp field that has the same name or the documents will not be shown. (all this to be verified).
Search Over All and the Graph Browser
The "search over all" dashboard works well in conjunction with the graph browser. All the results can be added at once using the standard graph browser "add document" mechanism.
Search Over All and Relational Links
Currently, you should not use "search over all" in conjunction with the relational buttons (that pivoting directly from the search interface to the dashboards). This is because with Siren 10, the * pattern is seen as a different entity class altogether in the ontology editor. As such, relations must be recreated and mappings will be very confusing with all the fields mixed together. Also, to go from a search result to its correspondent on the dashboard using a relational button, you would need a field that can work as primary key (outside the _uid field which cannot be used [to be verified]).
While the default search quality is good, it can be dramatically improved by taking into consideration the requirements of the application. The following resources are a great place to start improving the quality of your search:
- Indexing human language in Elasticsearch
- Use field popularity settings - read more about boosting ranking via field popularity
For even more advanced search (semantic, ontology-thesaurus based, multilingual, AI - learn to rank), contact your Siren technical account manager.