持续获取以下警告消息。不知道该怎么办。看到一些相关的帖子要求增加文件描述符的数量。
怎么做呢?
即使现在增加,在添加新索引时也会遇到同样的问题。(目前使用约400个索引,6个分片和1个副本)。指数的数量往往会增加。
[03:58:24,165][WARN ][cluster.action.shard ] [node1] received shard failed for [index9][2], node[node_hash3], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[index9][2] failed recovery]; nested: EngineCreationFailureException[[index9][2] failed to open reader on writer]; nested: FileNotFoundException[/data/elasticsearch/whatever/nodes/0/indices/index9/2/index/segments_1 (Too many open files)]; ]] [03:58:24,166][WARN ][cluster.action.shard ] [node1] received shard failed for [index15][0], node[node_hash2], [P], s[INITIALIZING], reason [Failed to create shard, message [IndexShardCreationException[[index15][0] failed to create shard]; nested: IOException[directory '/data/elasticsearch/whatever/nodes/0/indices/index15/0/index' exists and is a directory, but cannot be listed: list() returned null]; ]] [03:58:24,195][WARN ][cluster.action.shard ] [node1] received shard failed for [index16][3], node[node_hash3], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[index16][3] failed recovery]; nested: EngineCreationFailureException[[index16][3] failed to open reader on writer]; nested: FileNotFoundException[/data/elasticsearch/whatever/nodes/0/indices/index16/3/index/segments_1 (Too many open files)]; ]] [03:58:24,196][WARN ][cluster.action.shard ] [node1] received shard failed for [index17][0], node[node_hash3], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[index17][0] failed recovery]; nested: EngineCreationFailureException[[index17][0] failed to open reader on writer]; nested: FileNotFoundException[/data/elasticsearch/whatever/nodes/0/indices/index17/0/index/segments_1 (Too many open files)]; ]] [03:58:24,198][WARN ][cluster.action.shard ] [node1] received shard failed for [index21][4], node[node_hash3], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[index21][4] failed recovery]; nested: EngineCreationFailureException[[index21][4] failed to create engine]; nested: LockReleaseFailedException[Cannot forcefully unlock a NativeFSLock which is held by another indexer component: /data/elasticsearch/whatever/nodes/0/indices/index21/4/index/write.lock]; ]]
节点api的输出
curl -XGET 'http://localhost:9200/_nodes?os=true&process=true&pretty=true' { "ok" : true, "cluster_name" : "whatever", "nodes" : { "node_hash1" : { "name" : "node1", "transport_address" : "transportip1", "hostname" : "myhostip1", "version" : "0.20.4", "http_address" : "httpip1", "attributes" : { "data" : "false", "master" : "true" }, "os" : { "refresh_interval" : 1000, "available_processors" : 8, "cpu" : { "vendor" : "Intel", "model" : "Xeon", "mhz" : 2133, "total_cores" : 8, "total_sockets" : 8, "cores_per_socket" : 16, "cache_size" : "4kb", "cache_size_in_bytes" : 4096 }, "mem" : { "total" : "7gb", "total_in_bytes" : 7516336128 }, "swap" : { "total" : "30gb", "total_in_bytes" : 32218378240 } }, "process" : { "refresh_interval" : 1000, "id" : 26188, "max_file_descriptors" : 16384 } }, "node_hash2" : { "name" : "node2", "transport_address" : "transportip2", "hostname" : "myhostip2", "version" : "0.20.4", "attributes" : { "master" : "false" }, "os" : { "refresh_interval" : 1000, "available_processors" : 4, "cpu" : { "vendor" : "Intel", "model" : "Xeon", "mhz" : 2400, "total_cores" : 4, "total_sockets" : 4, "cores_per_socket" : 32, "cache_size" : "20kb", "cache_size_in_bytes" : 20480 }, "mem" : { "total" : "34.1gb", "total_in_bytes" : 36700303360 }, "swap" : { "total" : "0b", "total_in_bytes" : 0 } }, "process" : { "refresh_interval" : 1000, "id" : 24883, "max_file_descriptors" : 16384 } }, "node_hash3" : { "name" : "node3", "transport_address" : "transportip3", "hostname" : "myhostip3", "version" : "0.20.4", "attributes" : { "master" : "false" }, "os" : { "refresh_interval" : 1000, "available_processors" : 4, "cpu" : { "vendor" : "Intel", "model" : "Xeon", "mhz" : 2666, "total_cores" : 4, "total_sockets" : 4, "cores_per_socket" : 16, "cache_size" : "8kb", "cache_size_in_bytes" : 8192 }, "mem" : { "total" : "34.1gb", "total_in_bytes" : 36700303360 }, "swap" : { "total" : "0b", "total_in_bytes" : 0 } }, "process" : { "refresh_interval" : 1000, "id" : 25328, "max_file_descriptors" : 16384 } } }
如何增加允许打开的文件的最大数量取决于您的Linux发行版。以下是有关ubuntu和centos的一些说明:
http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/ http://pro.benjaminste.in/post/318453669/increase-the-number-of-file- 中心描述符
elasticsearch文档建议将最大文件限制设置为32k或64k。由于您的速度为16k,并且已经达到极限,因此我可能会将其设置为更高。大约128k 请参阅:http : //www.elasticsearch.org/guide/reference/setup/installation/
增加打开文件的数量并重新启动elasticsearch之后,您将需要通过重新运行您提到的curl命令来验证它是否有效:
curl -XGET 'http://localhost:9200/_nodes?os=true&process=true&pretty=true'
随着您添加更多索引(以及更多文档),您还将看到elasticsearch跟踪增加的文件数。如果您发现所有标记和文档的性能均下降,则可以尝试向集群添加新节点:http : //www.elasticsearch.org/guide/reference/setup/configuration/-由于您已经拥有分片,复制配置,这应该是一个相对轻松的过程。