我有一个很大的名字数据库,主要来自苏格兰。我们目前正在生产一个原型,以替换执行搜索的现有软件。这仍在生产中,我们的目标是使我们的结果尽可能接近同一搜索的当前结果。
我希望有人可以帮助我,我正在对Elastic Search进行搜索,查询是“ Michael Heaney”,我得到了一些疯狂的结果。当前搜索返回两个主要的姓,分别是“ Heaney”和“ Heavey”,都以“ Michael”为姓,我可以在Elastic Search中获得“ Heaney”结果,但是我也无法获得“ Heavey”和ES返回没有姓“ Michael”的人,但是我知道这是由于它是模糊查询的一部分。我知道这是一个狭窄的用例,因为它只是一个搜索,但是获得此结果并知道如何获得它会有所帮助。
谢谢。
制图
{ "jr": { "_all": { "enabled": true, "index_analyzer": "index_analyzer", "search_analyzer": "search_analyzer" }, "properties": { "pty_forename": { "type": "string", "index": "analyzed", "boost": 2, "index_analyzer": "index_analyzer", "search_analyzer": "search_analyzer", "store": "yes" }, "pty_full_name": { "type": "string", "index": "analyzed", "boost": 4, "index_analyzer": "index_analyzer", "search_analyzer": "search_analyzer", "store": "yes" }, "pty_surname": { "type": "string", "index": "analyzed", "boost": 4, "index_analyzer": "index_analyzer", "search_analyzer": "search_analyzer", "store": "yes" } } } }'
索引设定
{ "settings": { "number_of_shards": 2, "number_of_replicas": 0, "analysis": { "analyzer": { "index_analyzer": { "tokenizer": "standard", "filter": [ "standard", "my_delimiter", "lowercase", "stop", "asciifolding", "porter_stem", "my_metaphone" ] }, "search_analyzer": { "tokenizer": "standard", "filter": [ "standard", "my_metaphone", "synonym", "lowercase", "stop", "asciifolding", "porter_stem" ] } }, "filter": { "synonym": { "type": "synonym", "synonyms_path": "synonyms/synonyms.txt" }, "my_delimiter": { "type": "word_delimiter", "generate_word_parts": true, "catenate_words": false, "catenate_numbers": false, "catenate_all": false, "split_on_case_change": false, "preserve_original": false, "split_on_numerics": false, "stem_english_possessive": false }, "my_metaphone": { "type": "phonetic", "encoder": "metaphone", "replace": false } } } } }'
模糊
{ "from":0, "size":100, "query": { "bool": { "should": [ { "fuzzy": { "pty_surname": { "min_similarity": 0.2, "value": "Heaney", "prefix_length": 0, "boost": 5 } } }, { "fuzzy": { "pty_forename": { "min_similarity": 1, "value": "Michael", "prefix_length": 0, "boost": 1 } } } ] } } }
首先,我在Play中重新创建了您当前的配置:https : //www.found.no/play/gist/867785a709b4869c5543
如果您去那里,请切换到“分析”选项卡以查看文本的转换方式:
请注意,例如,Heaney最终[hn, heanei]以search_analyzer和和[HN, heanei]一起被标记化index_analyzer。请注意变音素词的大小写差异。因此,那是不匹配的。
Heaney
[hn, heanei]
search_analyzer
[HN, heanei]
index_analyzer
该fuzzy-query没有做查询时间文本分析。因此,您最终将Heavey与进行比较heanei。其Damerau- Levenshtein距离比您的参数所允许的长。
fuzzy
Heavey
heanei
您真正想做的是使用的模糊功能match。Match 确实 会查询时间文本分析,并且具有 模糊 参数。
match
至于fuzziness,这在Lucene 4中有所改变。以前,通常将其指定为float。现在应将其指定为允许的距离。有一个出色的拉取请求来澄清:https : //github.com/elasticsearch/elasticsearch/pull/4332/files
fuzziness
之所以要招募没有名字的人,Michael是因为您正在做一个bool.should。这具有或语义。一场比赛就足够了,但是从得分上来说,比赛越多越好。
Michael
bool.should
最后,将所有过滤合并到同一术语中不一定是最好的方法。例如,您无法知道和增强准确的拼写。您应该考虑使用multi_field以多种方式处理该字段。
这是一个示例,您可以使用,使用curl命令在下面重新创建它。但是,我完全跳过使用“ porter”词干分析器。我保留它只是为了显示multi_field的工作方式。结合使用匹配,模糊性匹配和语音匹配可以使您走得更远。(请确保在进行语音匹配时不允许模糊不清- 否则您将获得无用的模糊匹配。:-)
#!/bin/bash export ELASTICSEARCH_ENDPOINT="http://localhost:9200" # Create indexes curl -XPUT "$ELASTICSEARCH_ENDPOINT/play" -d '{ "settings": { "analysis": { "text": [ "Michael", "Heaney", "Heavey" ], "analyzer": { "metaphone": { "type": "custom", "tokenizer": "standard", "filter": [ "my_metaphone" ] }, "porter": { "type": "custom", "tokenizer": "standard", "filter": [ "lowercase", "porter_stem" ] } }, "filter": { "my_metaphone": { "encoder": "metaphone", "replace": false, "type": "phonetic" } } } }, "mappings": { "jr": { "properties": { "pty_surename": { "type": "multi_field", "fields": { "pty_surename": { "type": "string", "analyzer": "simple" }, "metaphone": { "type": "string", "analyzer": "metaphone" }, "porter": { "type": "string", "analyzer": "porter" } } } } } } }' # Index documents curl -XPOST "$ELASTICSEARCH_ENDPOINT/_bulk?refresh=true" -d ' {"index":{"_index":"play","_type":"jr"}} {"pty_surname":"Heaney"} {"index":{"_index":"play","_type":"jr"}} {"pty_surname":"Heavey"} ' # Do searches curl -XPOST "$ELASTICSEARCH_ENDPOINT/_search?pretty" -d ' { "query": { "bool": { "should": [ { "bool": { "should": [ { "match": { "pty_surname": { "query": "heavey" } } }, { "match": { "pty_surname": { "query": "heavey", "fuzziness": 1 } } }, { "match": { "pty_surename.metaphone": { "query": "heavey" } } }, { "match": { "pty_surename.porter": { "query": "heavey" } } } ] } } ] } } } '