ElasticSearch fails to recreate index due to data Field type change

Question

We converted a field of a BusinessData object from long to String. Now we face index recreation problems after upgrading the ivy engine to a new version.

2019-05-06 15:36:36.038 ERROR [ch.ivyteam.ivy.business.data.store.search.internal.BusinessDataSearchIndecesRecreator] [ivy immediate job pool-thread-1] [executionContext=SYSTEM] 
  Recreation of business data search index failed
    [errorId=16A8D5C46C63FD9B, executionContext=SYSTEM]
    ch.ivyteam.ivy.business.data.store.search.internal.elasticsearch.ElasticsearchException: Elasticsearch add documents to index failed because of: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
    bulk error: id=f08fe3d90a454be4b95a0b59785686ff, index=ivy.businessdata-com.axonivy.hrwf.vea.arbeitsvertragsaenderungencasemodel, error={"type":"illegal_argument_exception","reason":"mapper [antrag.lohnUndOderArbeitszeitaenderung.baseSalaryPtNew] cannot be changed from type [float] to [long]"}
    bulk error: id=b15b99eb022b4e1f95cd1b34ee03ae2f, index=ivy.businessdata-com.axonivy.hrwf.vea.arbeitsvertragsaenderungencasemodel, error={"type":"mapper_parsing_exception","reason":"failed to parse [antrag.lohnUndOderArbeitszeitaenderung.expensesPtCurrent]","caused_by":{"type":"number_format_exception","reason":"For input string: \"CHF500\""}}

How can we get a proper data store and make the index recreation running?

Answer 1

The error occurs as ElasticSearch determines the type of a field once the first non empty value is defined. So if a BusinessData object with long expensesPtCurrent is once persistent, the type can be changed to String without facing compatiblity issues.

To takle the problem I'd write a conversion in Java to:

read old values (where the field was a long) as raw JSON
load and update the persistent businssData object and apply a compatible value in new format (String)
store the updated value in BusinessData repo

Sample:

public static void convertLegacy() throws IOException
  {
      ObjectMapper mapper = new ObjectMapper();
      BusinessDataRepository repository = BusinessDataRepository.get();
      List<BusinessDataInfo<Antrag>> infos = repository.search(Antrag.class).execute().getAllInfos();
      for(BusinessDataInfo<Antrag> info : infos)
      {
          String json = BusinessDataSearchIndex.toSearchIndexJson(info.getRawValue());
          JsonNode node = mapper.readTree(json); // JSON = simple API to work with JSON objects
          JsonNode ptCurr = node.get("lohnUndOderArbeitszeitaenderung").get("expensesPtCurrent");
          if (ptCurr.isLong())
          {
              // legacy value
              long oldVal = ptCurr.asLong();
              Antrag antrag = repository.find(info.getId(), Antrag.class);
              antrag.setExpensesCurrent(String.valueOf(oldVal)+ " CHF");
              repository.save(antrag);
          }
      }
  }

To avoid the problem at all:

Do not change the type of already persistent data. But add a new replacement field. Optionally with fallback support to read legacy values. See: https://answers.axonivy.com/questions/2606/can-not-persist-business-data-to-repo-after-changing-data-type

ElasticSearch fails to recreate index due to data Field type change

Follow this question

Related questions