For a migration, I need to work with a large set of Elastic Search objects. Setting the limit to 100000, I get the following error:

ElasticsearchException: Elasticsearch search documents failed because of: {"root_cause":[{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [100000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],...

How can the index.max_result_window setting be changed. Or - alternatively, how do I access the scroll API?

Designer 7.2.1

TIA Peter

asked 25.02.2019 at 11:17

peters's gravatar image

peters
(suspended)
accept rate: 0%

Hello @peters

I think you wold like to migrate business data? Do you migrate directly the data in elasticsearch? This won't work, because this data is just an index. You have to migrate the data in the system database.

(26.02.2019 at 01:27) Alex Suter ♦♦ Alex%20Suter's gravatar image

Hello @Alex Suter

We migrate ~40.000 objects from a SQL database into the BusinessDataRepository. But we wanted to provide the possibility to delete the migrated objects (for example to re-migrate them). We are therefore loading the objects again and delete them one-by-one. This was the situation where we got this problem. (Of course we can do this differently if there is no way to change the limit or to access the scroll API.)

(26.02.2019 at 07:36) peters peters's gravatar image

Which api do you exactly use? Are you doing the migration as recommended? See class BusinessDataJsonFormatMigrator in https://developer.axonivy.com/doc/latest/PublicAPI

(27.02.2019 at 02:05) Alex Suter ♦♦ Alex%20Suter's gravatar image

This is a limitation of the BusinessDataRepository API. A query like the following will never work with the default elasticsearch index settings:

Ivy.repo().search(Data.class).limit(10000, 1000).execute().getAll();

There are two possibilities:

  1. Make some filter condition in your query for your migration now. Query only the first 10000 which are not already migrated. After that you can fire the same query and you will get the next 10000 business data objects.

  2. Change the max_result_window size of the index. But the problem is, that the customization of the index is not supported by ivy and must be done manually. https://stackoverflow.com/questions/35206409/elasticsearch-2-1-result-window-is-too-large-index-max-result-window

The best solution would be that we provide the scroll api for elasticsearch. We'll see how we can make that available.

link

answered 27.02.2019 at 08:53

Alex%20Suter's gravatar image

Alex Suter ♦♦
3.1k122247
accept rate: 84%

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×22

Asked: 25.02.2019 at 11:17

Seen: 9,914 times

Last updated: 27.02.2019 at 08:53