How do I change Elastic Search setting index.max_result_window?

Question

For a migration, I need to work with a large set of Elastic Search objects. Setting the limit to 100000, I get the following error:

ElasticsearchException: Elasticsearch search documents failed because of: {"root_cause":[{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [100000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],...

How can the index.max_result_window setting be changed. Or - alternatively, how do I access the scroll API?

Designer 7.2.1

TIA Peter

Accepted Answer

This is a limitation of the BusinessDataRepository API. A query like the following will never work with the default elasticsearch index settings:

Ivy.repo().search(Data.class).limit(10000, 1000).execute().getAll();

There are two possibilities:

Make some filter condition in your query for your migration now. Query only the first 10000 which are not already migrated. After that you can fire the same query and you will get the next 10000 business data objects.
Change the max_result_window size of the index. But the problem is, that the customization of the index is not supported by ivy and must be done manually. https://stackoverflow.com/questions/35206409/elasticsearch-2-1-result-window-is-too-large-index-max-result-window

The best solution would be that we provide the scroll api for elasticsearch. We'll see how we can make that available.

How do I change Elastic Search setting index.max_result_window?

Follow this question

Related questions