Limit 1000 fields while persisting with elasticsearch

Question

Hi guys,

Recently we got into some problem while trying to save an object with more than 1000 fields. I saw that it was reported to ivy.team and the bug is in progress.

But may I ask if there is any work around for the moment? or will not take another migration?

Thank you very much

Caused by: ch.ivyteam.ivy.business.data.store.search.internal.elasticsearch.ElasticsearchException: Elasticsearch update index of document failed because of: {"root_cause":[{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [ivy.businessdata-ch.axonivy.finform.deskpost.individualdossier] has been exceeded"}],"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [ivy.businessdata-ch.axonivy.finform.deskpost.individualdossier] has been exceeded"}

Accepted Answer

0

Hi @ToanLC

You can manually create an index template which allows over 1000 fields. When elasticsearch creates the index, it takes all matching index templates and creates the index.

Afterwards you must recreate the index over the api. Here an index template example.

 "fix_totat_fields" : {
    "order" : 0,
    "version" : 1,
    "template" : "ivy.businessdata*",
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        }
      }
    },
    "mappings" : { },
    "aliases" : { } 
}

link

answered 17.04.2018 at 03:05

Alex Suter ♦♦
3.1k●12●22●47
accept rate: 84%

Hello @alex, I inserted the template onto my elasticsearch server but it didn't help. Not really sure if I did something wrong here.

https://codebeautify.org/jsonviewer/cbb1a15d

(17.04.2018 at 03:59) ToanLC

1

it looks good, but you need to recreate the index. the best way is to shutdown the engine, delete the index manually and start the engine again. i recommend to check the settings of the index. there you can see if your template settings are really applied.

(17.04.2018 at 04:04) Alex Suter ♦♦

I tried but then I cannot create the index again as it got into the exception while trying to persist data.

(17.04.2018 at 06:41) ToanLC

you must restart the engine. 1) Delete Index 2) Start engine

(17.04.2018 at 07:37) Alex Suter ♦♦

Hello @alex, I'm so sorry for the confusion as I actually worked on another instance of elasticsearch of other ivy engine(Mine was at 19201) and I changed the settings on 19200. By the way, there is also other way to do this as to change setting of the designated index curl -XPUT http://localhost:19200/ivy.businessdata-ch.axonivy.fintech.individualch.individualdossier/_settings -d '{"index.mapping.total_fields.limit": 20000}'

(17.04.2018 at 22:43) ToanLC

1

Yes you are right. But the problem is, if we (ivy itself) reindex, your indrx will be deleted and you will loos this setting. If you use an template, then newly created indicies will alwasy have this setting. We work on a solution for that!

(18.04.2018 at 02:02) Alex Suter ♦♦

oh, that's why. Thanks for the explanation, I will stick to the template solution then. :D BTW, Can I ask how can I set elasticsearch to start on a specific port? I changed the http.port inside elasticsearch.yml, but it doesn't work

(18.04.2018 at 02:41) ToanLC

showing 5 of 7 show 2 more comments

Limit 1000 fields while persisting with elasticsearch

Follow this question

Related questions