Sitecore: Solr spell check

During one of our projects, we faced a problem that Sitecore 7 does not support out of the box some cool features of Solr like spell check and similar results, which were in great demand on one of our projects. After some time of investigation with Reflector, we found out that this kind of customization would not be easy, as a lot of classes in Sitecore provider to Solr wasn’t designed to support third party customization. There are a lot of internal classes and important properties hidden in private properties.

Implementation

Enable spell check on Solr side

First of all you should configure spellcheck component in solrconfig.xml (for our solution it was enough to change spellchecker.field).

1
2
3
4
5
<lst name="spellchecker">
  ...
  <str name="field">field_name</str>
  ...
</lst>

Then you should enable this component for select handler in order to be able to get the component results in response to query generated from Sitecore.

1
2
3
4
5
6
7
<requestHandler name="/select" class="solr.SearchHandler">
  ...
  <arr name="last-components"> 
    ...
    <str>spellcheck</str>
  </arr>
</requestHandler>

After that you could check if your configuration is working by querying link similar to that:

1
http://<core url>/select?spellcheck=true&spellcheck.q=wroong+word&spellcheck.collate=true

Solr is ready and we could move forward to Sitecore.

Solr provider in Sitecore

As the usual process of Sitecore extension starts with digging into sources through ILSpy or similar tools, this time was not an exception. When we working with ContentSearch we are getting results by calling GetResults or GetFasets functions, which are implemented as an extension to the IQueriable interface, and add a required node to Linq calls chain. However, if we go further we will see that ContentSearch Linq parser and mapper understand only a limited set of functions (defined by an enum) and have no values reserved for the future, so the only possible way is getting closer to SolrNet queries.

Extending Solr Provider

In order to do this we need to create an extension for IQueryable and pass search context and spell check query in it. Within the extension we create QueryOptions object from SolrNet and configure spell check.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
public static string CheckSpelling(this IQueryable query, IProviderSearchContext context, string text = null) {
    var extendedQuery = (SolrCompositeQuery)((IHasNativeQuery) query).Query;
    extendedQuery.Methods.Add(new GetResultsMethod(GetResultsOptions.Default)); 
    var parameters = new SpellCheckingParameters { Collate = true };

    if (!string.IsNullOrEmpty(text))
    {
        parameters.Query = text;
    }

    var newQuery = new ExtendedCompositeQuery(
    extendedQuery.Query,
    extendedQuery.Filter,
    extendedQuery.Methods,
    extendedQuery.VirtualFieldProcessors,
    extendedQuery.FacetQueries,
    new QueryOptions
    {
        SpellCheck = parameters,
        Rows = 0
    }
    );

    var linqToSolr = new CustomLinqToSolrIndex((SolrSearchContext)context, null);
    var response = linqToSolr.Execute(newQuery);

    return GetSpellCheckedString(response.SpellCheckedResponse);
}

Than QueryOptions should be added to as a property to a type inherited from SolrCompositeQuery.

1
2
3
4
5
6
7
8
public class ExtendedCompositeQuery : SolrCompositeQuery
{
    public QueryOptions QueryOptions { get; set; }
    public ExtendedCompositeQuery(AbstractSolrQuery query, AbstractSolrQuery filterQuery, IEnumerable methods, IEnumerable virtualFieldProcessors, IEnumerable facetQueries, QueryOptions options)  : base(query, filterQuery, methods, virtualFieldProcessors, facetQueries)
    {
        this.QueryOptions = options;
    }
}

The composite query could be executed with help of LinqToSolrIndex class, but a standard realization of this class will not recognize our QueryOptions, so it should be used as a base class for CustomLinqToSolrIndex.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
public class CustomLinqToSolrIndex < TItem > : LinqToSolrIndex < TItem > {
    private readonly SolrSearchContext context;

    private readonly string cultureCode;

    /// <summary>
    /// Initializes a new instance of the <see cref="CustomLinqToSolrIndex{TItem}" /> class.
    /// </summary>
    /// <param name="context">The context.</param>
    /// <param name="executionContext">The execution context.</param>
    public CustomLinqToSolrIndex(SolrSearchContext context, IExecutionContext executionContext): base(context, executionContext) {
        Assert.ArgumentNotNull(context, "context");
        this.context = context;
        var executionContext1 = this.Parameters.ExecutionContext as CultureExecutionContext;
        var culture = executionContext1 == null ? CultureInfo.GetCultureInfo(Settings.DefaultLanguage) : executionContext1.Culture;
        this.cultureCode = culture.TwoLetterISOLanguageName;
        ((SolrFieldNameTranslator) this.Parameters.FieldNameTranslator).AddCultureContext(culture);
    }

    /// <summary>
    /// Executes the specified composite query.
    /// </summary>
    /// <typeparam name="TResult">The type of the result.</typeparam>
    /// <param name="compositeQuery">The composite query.</param>
    /// <returns></returns>
    public TResult Execute < TResult > (ExtendedCompositeQuery compositeQuery) {
        if (!typeof(TResult).IsGenericType || typeof(TResult).GetGenericTypeDefinition() != typeof(ExtendedSearchResults < > )) {
            return base.Execute < TResult > (compositeQuery);
        }

        var resultType = typeof(TResult).GetGenericArguments()[0];
        var solrQueryResults = this.Execute(compositeQuery, resultType);
        var type = typeof(SolrSearchResults < > ).MakeGenericType(
        new[] {
            resultType
        });
        var methodInfo = this.GetType().GetMethod("GetExtendedResults", BindingFlags.Instance | BindingFlags.NonPublic).MakeGenericMethod(typeof(TResult), resultType);
        var selectMethod = this.GetSelectMethod(compositeQuery);
        var instance = Activator.CreateInstance(
        type,
        new object[] {
            this.context,
            solrQueryResults,
            selectMethod,
            compositeQuery.VirtualFieldProcessors
        });
        return (TResult) methodInfo.Invoke(this, new[] {
            compositeQuery, instance, solrQueryResults
        });
    }

    /// <summary>
    /// Executes the specified composite query.
    /// </summary>
    /// <param name="compositeQuery">The composite query.</param>
    /// <param name="resultType">Type of the result.</param>
    /// <returns></returns>
    internal SolrQueryResults < Dictionary < string,
    object >> Execute(ExtendedCompositeQuery compositeQuery, Type resultType) {
        var options = compositeQuery.QueryOptions;
        if (compositeQuery.Methods != null) {
            var list1 = (compositeQuery.Methods).Where(m = > m.MethodType == QueryMethodType.Select).Select(m = > (SelectMethod) m).ToList();
            if ((list1).Any()) {
                foreach(var str in list1.SelectMany(selectMethod = > (IEnumerable < string > ) selectMethod.FieldNames)) {
                    options.Fields.Add(str.ToLowerInvariant());
                }
                if (!this.context.SecurityOptions.HasFlag(SearchSecurityOptions.DisableSecurityCheck)) {
                    options.Fields.Add("_uniqueid");
                    options.Fields.Add("_datasource");
                }
            }

            var list2 = compositeQuery.Methods.Where(m = > m.MethodType == QueryMethodType.GetResults).Select(m = > (GetResultsMethod) m).ToList();
            if (list2.Any()) {
                if (options.Fields.Count > 0) {
                    options.Fields.Add("score");
                } else {
                    options.Fields.Add("*");
                    options.Fields.Add("score");
                }
            }

            var list3 = compositeQuery.Methods.Where(m = > m.MethodType == QueryMethodType.OrderBy).Select(m = > (OrderByMethod) m).ToList();
            if (list3.Any()) {
                foreach(var orderByMethod in list3) {
                    var field = orderByMethod.Field;
                    options.AddOrder(
                    new[] {
                        new SortOrder(field, orderByMethod.SortDirection == SortDirection.Ascending ? Order.ASC : Order.DESC)
                    });
                }
            }

            var list4 = compositeQuery.Methods.Where(m = > m.MethodType == QueryMethodType.Skip).Select(m = > (SkipMethod) m).ToList();
            if (list4.Any()) {
                var num = list4.Sum(skipMethod = > skipMethod.Count);
                options.Start = num;
            }

            var list5 = compositeQuery.Methods.Where(m = > m.MethodType == QueryMethodType.Take).Select(m = > (TakeMethod) m).ToList();
            if (list5.Any()) {
                var num = list5.Sum(takeMethod = > takeMethod.Count);
                options.Rows = num;
            }

            var list6 = compositeQuery.Methods.Where(m = > m.MethodType == QueryMethodType.Count).Select(m = > (CountMethod) m).ToList();
            if (compositeQuery.Methods.Count == 1 && list6.Any()) {
                options.Rows = 0;
            }

            var list7 = compositeQuery.Methods.Where(m = > m.MethodType == QueryMethodType.Any).Select(m = > (AnyMethod) m).ToList();
            if (compositeQuery.Methods.Count == 1 && list7.Any()) {
                options.Rows = 0;
            }

            var list8 = compositeQuery.Methods.Where(m = > m.MethodType == QueryMethodType.GetFacets).Select(m = > (GetFacetsMethod) m).ToList();
            if (compositeQuery.FacetQueries.Count > 0 && (list8.Any() || list2.Any())) {
                foreach(
                var facetQuery in GetFacetsPipeline.Run(
                new GetFacetsArgs(
                null,
                compositeQuery.FacetQueries,
                this.context.Index.Configuration.VirtualFieldProcessors,
                this.context.Index.FieldNameTranslator)).FacetQueries.ToHashSet()) {
                    if (facetQuery.FieldNames.Any()) {
                        var minimumResultCount = facetQuery.MinimumResultCount;
                        if (facetQuery.FieldNames.Count() == 1) {
                            var fieldNameTranslator = this.FieldNameTranslator as SolrFieldNameTranslator;
                            var str = facetQuery.FieldNames.First();
                            if (fieldNameTranslator != null && str == fieldNameTranslator.StripKnownExtensions(str) && this.context.Index.Configuration.FieldMap.GetFieldConfiguration(str) == null) {
                                str = fieldNameTranslator.GetIndexFieldName(str.Replace("__", "!").Replace("_", " ").Replace("!", "__"), true);
                            }
                            var queryOptions = options;
                            var solrFacetQueryArray1 = new ISolrFacetQuery[1];
                            solrFacetQueryArray1[0] = new SolrFacetFieldQuery(str) {
                                MinCount = minimumResultCount
                            };
                            var solrFacetQueryArray2 = solrFacetQueryArray1;
                            queryOptions.AddFacets(solrFacetQueryArray2);
                        }
                        if (facetQuery.FieldNames.Count() > 1) {
                            var queryOptions = options;
                            var solrFacetQueryArray1 = new ISolrFacetQuery[1];
                            solrFacetQueryArray1[0] = new SolrFacetPivotQuery {
                                Fields = new[] {
                                    string.Join(",", facetQuery.FieldNames)
                                },
                                MinCount = minimumResultCount
                            };
                            var solrFacetQueryArray2 = solrFacetQueryArray1;
                            queryOptions.AddFacets(solrFacetQueryArray2);
                        }
                    }
                }
                if (!list2.Any()) {
                    options.Rows = 0;
                }
                //var list9 =
                //    compositeQuery.Methods.Where(m => m.MethodType == QueryMethodType.Cast).Select(m => (GetSpellCheck)m).ToList();
                //if (list9.Any())
                //{
                //    options.Rows = 0;
                //    options.SpellCheck = new SpellCheckingParameters { Collate = true };
                //}
            }
        }

        if (compositeQuery.Filter != null) {
            options.AddFilterQueries(
            new ISolrQuery[] {
                compositeQuery.Filter
            });
        }

        options.AddFilterQueries(
        new ISolrQuery[] {
            new SolrQueryByField("_indexname", this.context.Index.Name)
        });

        if (!Settings.DefaultLanguage.StartsWith(this.cultureCode)) {
            var queryOptions = options;
            var solrQueryArray1 = new ISolrQuery[1];
            solrQueryArray1[0] = new SolrQueryByField("_language", this.cultureCode + "*") {
                Quoted = false
            };
            var solrQueryArray2 = solrQueryArray1;
            queryOptions.AddFilterQueries(solrQueryArray2);
        }

        var loggingSerializer = new SolrLoggingSerializer();
        var q = loggingSerializer.SerializeQuery(compositeQuery.Query);

        try {
            if (!options.Rows.HasValue) {
                options.Rows = ContentSearchConfigurationSettings.SearchMaxResults;
            }
            SearchLog.Log.Info("Query - " + q);
            SearchLog.Log.Info("Serialized Query - ?q=" + q + "&" + string.Join("&", loggingSerializer.GetAllParameters(options).Select(p = > string.Format("{0}={1}", p.Key, p.Value)).ToArray()));

            return this.SolrOperations.Query(q, options);
        } catch (Exception ex) {
            if (!(ex is SolrConnectionException) && !(ex is SolrNetException)) {
                throw;
            }
            var message = ex.Message;
            if (ex.Message.StartsWith("<?xml")) {
                var xmlDocument = new XmlDocument();
                xmlDocument.LoadXml(ex.Message);
                var xmlNode1 = xmlDocument.SelectSingleNode("/response/lst[@name='error'][1]/str[@name='msg'][1]");
                var xmlNode2 = xmlDocument.SelectSingleNode("/response/lst[@name='responseHeader'][1]/lst[@name='params'][1]/str[@name='q'][1]");
                if (xmlNode1 != null && xmlNode2 != null) {
                    SearchLog.Log.Error(string.Format("Solr Error : [\"{0}\"] - Query attempted: [{1}]", xmlNode1.InnerText, xmlNode2.InnerText));
                    return new SolrQueryResults < Dictionary < string, object >> ();
                }
            }
            Log.Error(message, this);
            return new SolrQueryResults < Dictionary < string, object >> ();
        }
    }

    /// <summary>
    /// Gets the extended results.
    /// </summary>
    /// <typeparam name="TResult">The type of the result.</typeparam>
    /// <typeparam name="TDocument">The type of the document.</typeparam>
    /// <param name="compositeQuery">The composite query.</param>
    /// <param name="processedResults">The processed results.</param>
    /// <param name="results">The results.</param>
    /// <returns></returns>
    internal TResult GetExtendedResults < TResult,
    TDocument > (ExtendedCompositeQuery compositeQuery, SolrSearchResults < TDocument > processedResults, SolrQueryResults < Dictionary < string, object >> results) {
        var type = typeof(TResult);

        var hits = processedResults.GetSearchHits();
        var facetResults = this.FormatFacetResults(processedResults.GetFacets(), compositeQuery.FacetQueries);

        var obj = Activator.CreateInstance(type, hits, processedResults.NumberFound, facetResults);

        if (type.HasProperty("SpellCheckedResponse")) {
            var spellCheckPropetry = type.GetProperty("SpellCheckedResponse");
            if (spellCheckPropetry != null && spellCheckPropetry.CanWrite) {
                spellCheckPropetry.SetValue(obj, results.SpellChecking.Collation);
            }
        }

        if (type.HasProperty("SimilarResults")) {
            var similarResultsPropetry = type.GetProperty("SimilarResults");
            if (similarResultsPropetry != null && similarResultsPropetry.CanWrite) {
                similarResultsPropetry.SetValue(obj, results.SimilarResults);
            }
        }

        return (TResult) Convert.ChangeType(obj, typeof(TResult));
    }

    private SelectMethod GetSelectMethod(SolrCompositeQuery compositeQuery) {
        var type = this.GetType().BaseType;
        var method = type.GetMethod("GetSelectMethod", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.Static);
        try {
            return (SelectMethod) method.Invoke(this, new object[] {
                compositeQuery
            });
        } catch (Exception ex) {
            Log.Error("Signture of internal LinqToSolrIndex<TItem>.GetSelectMethod has changed or method not found", ex, this);
            return null;
        }
    }

    private FacetResults FormatFacetResults(Dictionary < string, ICollection < KeyValuePair < string, int >>> facetResults, List < FacetQuery > facetQueries) {
        var type = this.GetType().BaseType;
        var method = type.GetMethod("FormatFacetResults", BindingFlags.NonPublic | BindingFlags.Instance);
        try {
            return (FacetResults) method.Invoke(this, new object[] {
                facetResults, facetQueries
            });
        } catch (Exception ex) {
            Log.Error("Signture of internal LinqToSolrIndex<TItem>.FormatFacetResults has changed or method not found", ex, this);
            return new FacetResults();
        }
    }

    private ISolrOperations < Dictionary < string,
    object >> SolrOperations {
        get {
            var solrSearchIndex = this.context.Index as SolrSearchIndex;

            if (solrSearchIndex != null) {
                return typeof(SolrSearchIndex)
                    .GetProperty("SolrOperations", BindingFlags.NonPublic | BindingFlags.Instance)
                    .GetValue(solrSearchIndex) as ISolrOperations < Dictionary < string, object >> ;
            }
            return null;
        }
    }
}

In the class above we need to create Execute method that will take our ExtendedCompositeQuery as param and call overwritten logic (internal method SolrQueryResults) when we returning extended SearchResults (with SpellCheckedResponse field) and base in all other cases. We also need to make some tweaks with reflection implementing new query execute logic like:

getting ISolrOperations, SelectMethod, and FacetResults from the private field of the base class
duplicate source code for SearchResults and SolrSearchResults classes, as original is sealed or internal 😊

*** These points are definitely not the best options ever but as it was already mentioned ContentSearch.Solr.Provider was not designed for easy extensibility.

When all changes above are implemented we could call our extension and process results.

1
2
var query = context.GetQueryable(.Filter(  );
var checked = query.CheckSpelling(context, "  ");

This call will make a separate call to Solr and will not return any mutch results and QueryOptions define the amount of results rows = 0. To get search results or facets you could call the required method after calling CheckSpelling.

Follow me on twitter @true_shoorik. Share if the post was useful for you.

Implementation#

Enable spell check on Solr side#

Solr provider in Sitecore#

Extending Solr Provider#

Implementation

Enable spell check on Solr side

Solr provider in Sitecore

Extending Solr Provider