For a migration, I need to work with a large set of Elastic Search objects. Setting the limit to 100000, I get the following error:

ElasticsearchException: Elasticsearch search documents failed because of: {"root_cause":[{"type":"query_phase_execution_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [100000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],...

How can the index.max_result_window setting be changed. Or - alternatively, how do I access the scroll API?

Designer 7.2.1

TIA Peter

asked 25.02 at 11:17

peters's gravatar image

peters
21339
accept rate: 0%

Hello @peters

I think you wold like to migrate business data? Do you migrate directly the data in elasticsearch? This won't work, because this data is just an index. You have to migrate the data in the system database.

(26.02 at 01:27) Alex Suter ♦♦ Alex%20Suter's gravatar image

Hello @Alex Suter

We migrate ~40.000 objects from a SQL database into the BusinessDataRepository. But we wanted to provide the possibility to delete the migrated objects (for example to re-migrate them). We are therefore loading the objects again and delete them one-by-one. This was the situation where we got this problem. (Of course we can do this differently if there is no way to change the limit or to access the scroll API.)

(26.02 at 07:36) peters peters's gravatar image

Which api do you exactly use? Are you doing the migration as recommended? See class BusinessDataJsonFormatMigrator in https://developer.axonivy.com/doc/latest/PublicAPI

(27.02 at 02:05) Alex Suter ♦♦ Alex%20Suter's gravatar image

This is a limitation of the BusinessDataRepository API. A query like the following will never work with the default elasticsearch index settings:

Ivy.repo().search(Data.class).limit(10000, 1000).execute().getAll();

There are two possibilities:

  1. Make some filter condition in your query for your migration now. Query only the first 10000 which are not already migrated. After that you can fire the same query and you will get the next 10000 business data objects.

  2. Change the max_result_window size of the index. But the problem is, that the customization of the index is not supported by ivy and must be done manually. https://stackoverflow.com/questions/35206409/elasticsearch-2-1-result-window-is-too-large-index-max-result-window

The best solution would be that we provide the scroll api for elasticsearch. We'll see how we can make that available.

link

answered 27.02 at 08:53

Alex%20Suter's gravatar image

Alex Suter ♦♦
2.0k81936
accept rate: 83%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×11

Asked: 25.02 at 11:17

Seen: 50 times

Last updated: 27.02 at 08:53