Elasticsearch-Java RestHighLevelClient-如何使用滚动API获取所有文档

一尘不染

Elasticsearch-Java RestHighLevelClient-如何使用滚动API获取所有文档

elasticsearch

在Elasticsearch的索引中，我保存了约30000个实体。我想使用RestHighLevelClient获得它们的所有ID。我读过，最好的方法是使用滚动API。但是，当我这样做时，我只能接收大约10个实体，而不是30k。如何解决这个问题

final class ElasticRepo {
    private final RestHighLevelClient restHighLevelClient;

List<ListingsData> getAllListingsDataIds() {
        val request = new SearchRequest(ELASTICSEARCH_LISTINGS_INDEX);
        request.types(ELASTICSEARCH_TYPE);
        val searchSourceBuilder = new SearchSourceBuilder()
                .query(matchAllQuery())
                .fetchSource(new String[]{"listing_id"}, new String[]{"backoffice_data", "search_and_match_data"});
        request.source(searchSourceBuilder);
        request.scroll(TimeValue.timeValueMinutes(3));
        return executeQuery(request);
    }

 private List<ListingsData> executeQuery(final SearchRequest searchQuery) {
        try {
            val hits = restHighLevelClient.search(searchQuery, RequestOptions.DEFAULT).getHits().getHits();
            return Arrays.stream(hits).map(SearchHit::getSourceAsString).map(ElasticRepo::toListingsData).collect(Collectors.toList());
        } catch (IOException e) {
            e.printStackTrace();
            throw new RuntimeException("");
        }
    }

}

当我这样做时，executeQuery仅返回大约11个实体。如何解决，如何获取索引中的所有文件？

阅读 1771

2020-06-22

共1个答案

一尘不染

尝试按照以下示例操作，我正在使用此代码，它可以正常工作：

        String query = "your query here";

        QueryBuilder matchQueryBuilder = QueryBuilders.boolQuery().must(new QueryStringQueryBuilder(query));

        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

        searchSourceBuilder.query(matchQueryBuilder);

        searchSourceBuilder.size(5000); //max is 10000

        searchRequest.indices("your index here");

        searchRequest.source(searchSourceBuilder);

        final Scroll scroll = new Scroll(TimeValue.timeValueMinutes(10L));

        searchRequest.scroll(scroll);

        SearchResponse searchResponse = client.search(searchRequest);
            String scrollId = searchResponse.getScrollId();

        SearchHit[] allHits = new SearchHit[0];

        SearchHit[] searchHits = searchResponse.getHits().getHits();

        while (searchHits != null && searchHits.length > 0)
        {

            allHits = Helper.concatenate(allHits, searchResponse.getHits().getHits()); //create a function which concatenate two arrays

            SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);

            scrollRequest.scroll(scroll);

            searchResponse = client.searchScroll(scrollRequest);

            scrollId = searchResponse.getScrollId();

            searchHits = searchResponse.getHits().getHits();

        }

        ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
        clearScrollRequest.addScrollId(scrollId);
        ClearScrollResponse clearScrollResponse = client.clearScroll(clearScrollRequest);

2020-06-22