我在SocketTimeoutException从Elastic检索数据/从Elastic插入数据时面对。周围有这种情况10-30 request/second。这些请求是get / put的组合。
SocketTimeoutException
10-30 request/second
这是我的弹性配置：
3 master nodes
4GB RAM
2 data nodes
8GM RAM
<dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> <version>7.2.0</version> </dependency> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>7.2.0</version> </dependency>
索引信息：
27.2 MB
Primaries: 12.2MB
{ "dev-index": { "mappings": { "properties": { "dataObj": { "type": "object", "enabled": false }, "generatedID": { "type": "keyword" }, "transNames": { //it's array of string "type": "keyword" } } } } }
以下是我的elastic Config文件。在这里，我有两个连接bean，一个用于读取，另一个用于写入弹性。
elastic Config
ElasticConfig.java：
@Configuration public class ElasticConfig { @Value("${elastic.host}") private String elasticHost; @Value("${elastic.port}") private int elasticPort; @Value("${elastic.user}") private String elasticUser; @Value("${elastic.pass}") private String elasticPass; @Value("${elastic-timeout:20}") private int timeout; @Bean(destroyMethod = "close") @Qualifier("readClient") public RestHighLevelClient readClient(){ final CredentialsProvider credentialsProvider = new BasicCredentialsProvider(); credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(elasticUser, elasticPass)); RestClientBuilder builder = RestClient .builder(new HttpHost(elasticHost, elasticPort)) .setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder .setDefaultCredentialsProvider(credentialsProvider) .setDefaultIOReactorConfig(IOReactorConfig.custom().setIoThreadCount(5).build()) ); builder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder .setConnectTimeout(10000) .setSocketTimeout(60000) .setConnectionRequestTimeout(0) ); RestHighLevelClient restClient = new RestHighLevelClient(builder); return restClient; } @Bean(destroyMethod = "close") @Qualifier("writeClient") public RestHighLevelClient writeClient(){ final CredentialsProvider credentialsProvider = new BasicCredentialsProvider(); credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(elasticUser, elasticPass)); RestClientBuilder builder = RestClient .builder(new HttpHost(elasticHost, elasticPort)) .setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder .setDefaultCredentialsProvider(credentialsProvider) .setDefaultIOReactorConfig(IOReactorConfig.custom().setIoThreadCount(5).build()) ); builder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder .setConnectTimeout(10000) .setSocketTimeout(60000) .setConnectionRequestTimeout(0) ); RestHighLevelClient restClient = new RestHighLevelClient(builder); return restClient; } }
这是调用弹性函数的函数，如果弹性数据可用，它将接受它，否则将生成数据并放入弹性。
public Object getData(Request request) { DataObj elasticResult = elasticService.getData(request); if(elasticResult!=null){ return elasticResult; } else{ //code to generate data DataObj generatedData = getData();//some function which will generated data //put above data into elastic by Async call. elasticAsync.putData(generatedData); return generatedData; } }
ElasticService.java getData函数：
@Service public class ElasticService { @Value("${elastic.index}") private String elasticIndex; @Autowired @Qualifier("readClient") private RestHighLevelClient readClient; public DataObj getData(Request request){ String generatedId = request.getGeneratedID(); GetRequest getRequest = new GetRequest() .index(elasticIndex) //elastic index name .id(generatedId); //retrieving by index id from elastic _id field (as key-value) DataObj result = null; try { GetResponse response = readClient.get(getRequest, RequestOptions.DEFAULT); if(response.isExists()) { ObjectMapper objectMapper = new ObjectMapper(); result = objectMapper.readValue(response.getSourceAsString(), DataObj.class); } } catch (Exception e) { LOGGER.error("Exception occurred during fetch from elastic !!!! " + ,e); } return result; } }
ElasticAsync.java异步放置数据功能：
@Service public class ElasticAsync { private static final Logger LOGGER = Logger.getLogger(ElasticAsync.class.getName()); @Value("${elastic.index}") private String elasticIndex; @Autowired @Qualifier("writeClient") private RestHighLevelClient writeClient; @Async public void putData(DataObj generatedData){ ElasticVO updatedRequest = toElasticVO(generatedData);//ElasticVO matches to the structure of index given above. try { ObjectMapper objectMapper = new ObjectMapper(); String jsonString = objectMapper.writeValueAsString(updatedRequest); IndexRequest request = new IndexRequest(elasticIndex); request.id(generatedData.getGeneratedID()); request.source(jsonString, XContentType.JSON); request.setRefreshPolicy(WriteRequest.RefreshPolicy.NONE); request.timeout(TimeValue.timeValueSeconds(5)); IndexResponse indexResponse = writeClient.index(request, RequestOptions.DEFAULT); LOGGER.info("response id: " + indexResponse.getId()); } } catch (Exception e) { LOGGER.error("Exception occurred during saving into elastic !!!!",e); } } }
这是将数据保存到弹性模块时发生异常时堆栈跟踪的一部分：
2019-07-19 07:32:19.997 ERROR [data-retrieval,341e6ecc5b10f3be,1eeb0722983062b2,true] 1 --- [askExecutor-894] a.c.s.a.service.impl.ElasticAsync : Exception occurred during saving into elastic !!!! java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-34 [ACTIVE] at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:789) ~[elasticsearch-rest-client-7.2.0.jar!/:7.2.0] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:225) ~[elasticsearch-rest-client-7.2.0.jar!/:7.2.0] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212) ~[elasticsearch-rest-client-7.2.0.jar!/:7.2.0] at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1448) ~[elasticsearch-rest-high-level-client-7.2.0.jar!/:7.2.0] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1418) ~[elasticsearch-rest-high-level-client-7.2.0.jar!/:7.2.0] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1388) ~[elasticsearch-rest-high-level-client-7.2.0.jar!/:7.2.0] at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:836) ~[elasticsearch-rest-high-level-client-7.2.0.jar!/:7.2.0] Caused by: java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-34 [ACTIVE] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.3.jar!/:4.1.3] at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.3.jar!/:4.1.3] at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.11.jar!/:4.4.11] ... 1 common frames omitted
这是在将数据检索到弹性期间发生异常时堆栈跟踪的某些部分：
2019-07-19 07:22:37.844 ERROR [data-retrieval,104cf6b2ab5b3349,b302d3d3cd7ebc84,true] 1 --- [o-8080-exec-346] a.c.s.a.service.impl.ElasticService : Exception occurred during fetch from elastic !!!! java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-30 [ACTIVE] at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:789) ~[elasticsearch-rest-client-7.1.1.jar!/:7.1.1] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:225) ~[elasticsearch-rest-client-7.1.1.jar!/:7.1.1] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212) ~[elasticsearch-rest-client-7.1.1.jar!/:7.1.1] at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1433) ~[elasticsearch-rest-high-level-client-7.1.1.jar!/:7.1.1] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403) ~[elasticsearch-rest-high-level-client-7.1.1.jar!/:7.1.1] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373) ~[elasticsearch-rest-high-level-client-7.1.1.jar!/:7.1.1] at org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:699) ~[elasticsearch-rest-high-level-client-7.1.1.jar!/:7.1.1] Caused by: java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-30 [ACTIVE] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.3.jar!/:4.1.3] at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.3.jar!/:4.1.3] at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.11.jar!/:4.4.11] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.11.jar!/:4.4.11] ... 1 common frames omitted
我浏览了一些相关的博客，他们提到这个问题可能是由于RAMElastic的cluster配置所致。然后，因为只有两个数据节点，所以我将分片从5更改为2。还将Data节点的内存从4GB增加到8GB，因为我知道弹性只会使用50%totalRAM。异常的发生率有所降低，但问题仍然存在。
RAM
cluster
50%
解决这个问题的可能方法是什么？从Java/弹性配置的角度来看，我经常缺少这种东西SocketTimeoutException吗？如果您需要有关配置的更多详细信息，请告诉我。
我们遇到了同样的问题，经过一番挖掘后，我发现了根本原因：客户端和弹性服务器之间的防火墙配置不匹配，tcp的内核配置仍然有效。
防火墙在3600秒后删除空闲连接。问题是tcp keep alive的内核参数设置为7200秒（RedHat 6.x / 7.x中的默认设置）：
sysctl -n net.ipv4.tcp_keepalive_time 7200
因此，在发送保持活动探测之前，连接已断开。弹性http客户端中的asyncHttpClient似乎无法很好地处理掉线的连接，它只是等待套接字超时。
因此，请检查您的客户端和服务器之间是否有任何网络设备（负载平衡器，防火墙，代理服务器等）具有会话超时或类似超时，并增加该超时或降低tcp_keep_alive内核参数。
