1
1

Hi Axon.ivy Team,

Given the snippet of code:

Person p = new Person();
p.id = UUID.randomUUID().toString();
p.name = "John Doe";
p.birthdate = LocalDate.now();

ivy.repo.save(p);

Object result = ivy.repo.search(Person.class)
    .textField("id").containsPhrase(p.id)
    .execute().getFirst();

ivy.log.error("HHH - it is null " + #result);

Thread.sleep(TimeUnit.SECONDS.toMillis(1));

result = ivy.repo.search(Person.class)
    .textField("id").containsPhrase(p.id)
    .execute().getFirst();

ivy.log.error("HHH - it is null " + #result);

I found that there was a 1-second latency between the call to ivy.repo.save() and ivy.repo.search(). This is super annoying because we had to wait for 1-second every time until the saved object can be searched via the API. Blocking the execution for 1-second to wait until it is indexed to ES every time we save the object into the BusinessDataRepository is a terrible hack.

Is it by design? Can this be fixed so that the two calls work?

P.S: Please don't advise me to use the returned BusinessData id because it doesn't address my issue. I want to search back the data I save based on the object's field, not fetching it again.

P.P.S: Tested with Axon.ivy Designer 7.2.1 on my local workstation.

UPDATE

25.2.2019:

In our project, we were able to develop RESTful API for other parties. As we already have persisted our objects (for example Dossier), in BusinessDataRespository, we would want to preserve that behavior.

An excerpt from of our OpenAPI spec:

paths:
  /dossiers/{dossier_id}/persons/:
    post:
     operationId: addPersonToDossier
      description: add a single person into an existing Dossier
      requestbody:
        'application/json':
          schema:
            $ref: '#/components/schemas/Person'
      responses:
        '201':

    get:
      operationId: getAllPersonsInDossier
      description: get *all* existing persons of an existing Dossier
      responses:
        '200':
          content:
            'application/json':
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Person'

From our client, after they have already invoked several calls to addPersonToDossier, they may want to invoke getAllPersonsInDossier to fetch all Persons of a Dossier.

We persists Person separately from our Dossier to avoid ConcurrentModificationException, each Person holds a reference to its Dossier. Now in order to implement getAllPersonsInDossier, we have to use Ivy.repo().search(..).

Unfortunately, due to the 1-second latency, calling getAllPersonsInDossier too soon will probably return an empty result. This forces us to either:

  • Require our clients to WAIT on their side for 1-second until they can call getAllPersonsInDossier.
  • On the server side, we somehow have to implement a while loop to check until the Person could be found via search API, then we consider the addPersonToDossier finishes (or worse, a Thread.sleep(1000)).

Jack

asked 22.02 at 01:16

vagabond's gravatar image

vagabond
76269
accept rate: 50%

edited 25.02 at 00:52

it sound to me as if the Person is part of that dossier. Especially if you mention, that you had to move it for technical reasons (ConcurrentModificationEx). BusinessData uses a no-SQL document approach. One record should contain all the data belongs to that entity. So maybe we should get behind the problem of the ConcurrentModificationEx This exception should only occur if you are completely unaware how your data changes and what screen/api changes which part of the data. Otherwise the update / overwrite methods on the repo should give you many possibilities to store changes safely.

(25.02 at 09:35) Reguel Werme... ♦♦ Reguel%20Wermelinger's gravatar image

BusinessData is not for every use case. If you have clearly relational entities that have their own lifecycle they should not be used like classic relational db objects. When the BusinessData can clearly be connected with a root entity (e.g. Dossier) that holds most of the object related to it then you are fine. If the root is unclear or use-case driven (Person vs Dossier) then maybe BusinessData is not the best choice for your model.

(25.02 at 09:39) Reguel Werme... ♦♦ Reguel%20Wermelinger's gravatar image

Hi @Reguel Wermelinger, some of your points may be correct in my given example. However, even if we put everything in Dossier, our api at POST /dossiers and GET /dossiers still encounter the same issue.

(25.02 at 23:58) vagabond vagabond's gravatar image

Yeah for sure the delay to sync elasticSearch data will not disappear by changing the data structure - even though we can guess that the sync & search will be much faster if its all in one document. You really should avoid to split up internal data if you can somehow... The question stays, why you need instant millisecond perfect results. A normal user working with the API would not even recognize the delay. The results are ready before he can type a key. Are you building a javascript frontend with REST/BusinessData as backend?

(26.02 at 02:52) Reguel Werme... ♦♦ Reguel%20Wermelinger's gravatar image

A new idea: if you really just want to get ALL entities (without a real search scenario) we could easily provide a rock hard 100% sync API for that. The delay is just required for search - but not to store/load objects.

(26.02 at 02:54) Reguel Werme... ♦♦ Reguel%20Wermelinger's gravatar image

Are you building a javascript frontend with REST/BusinessData as backend? This is one of our use cases.

A normal user working with the API would not even recognize the delay. No, they will notice. A 1-second API call is not something can go unnoticeable.

(26.02 at 03:11) vagabond vagabond's gravatar image

we could easily provide a rock hard 100% sync API for that. Ok, so we'll have to wait until the next LTS release. And I guess you were referring to connect directly with the database and execute SQL query.

(26.02 at 03:11) vagabond vagabond's gravatar image

It is not like the user will not see all UI actions after a huge delay. Its just that if he modifies/adds new data, the just added record is may not 100% accurate. So why not just keep that just added reference on the UI?

(26.02 at 03:20) Reguel Werme... ♦♦ Reguel%20Wermelinger's gravatar image

BusinessData are always stored and loaded from a classical DB table. So we can easily provide a getAll() impl. Whether provided as hotfix or release must be discusses elsewhere. Anyway: this question is already far beyond the boundaries of a Q&A question where the community can benefit. I'd recommend that you setup a meeting with our support the walk trough pro/cons of your architecture decisions. There are many possible solutions for your current issue...

(26.02 at 03:24) Reguel Werme... ♦♦ Reguel%20Wermelinger's gravatar image
showing 5 of 9 show 4 more comments

It is by design that we rely on ElasticSearch to deliver search results. And yes we accept the fact that search results can be not 100% sync for some millis. On the other hand queries execute blazing fast (even one tremendous data sets) and allow very powerful google like search terms.

If you need a rock hard, 100% sync at anytime, persistency solution - BusinessData is maybe not the feature you are looking for. Just use JPA (ivy EntityClass) or any other persistency technology you like that relies on classical (often slow and inconvenient) relational database schemas.

And of course you already mentioned it by yourself. Our BusinessData set is always consistenly saved. And therefore you have access to them e.g. by Id. But search features could take some millies to deliver accurate results.

I think it is very unlikely that one would favor a 100% up to date search result over the benefits of a powerful query engine. Unless you are implementing a stock broker service. But maybe you can share your use-case....

link

answered 22.02 at 10:01

Reguel%20Wermelinger's gravatar image

Reguel Werme... ♦♦
8.4k21651
accept rate: 68%

edited 22.02 at 10:14

Hi @Reguel Wermelinger, I have updated the question to add our current use case. It is not as sophisticated as the stock broker service but simply we want to expose our existing system with an RESTful API.

(25.02 at 00:54) vagabond vagabond's gravatar image

Hi @Reguel Wermelinger, I'm looking for a technical explanation why the latency of 1-second is needed (at every case, even if the object structure is super simple). And I expect this latency and your recommendation is somewhere in the Designer Guide or the javadoc of BusinessDataRepository.

(26.02 at 03:13) vagabond vagabond's gravatar image

Yeah I agree that such an explanation is currently missing and should be addressed: in the meantime the short answer is.

the 'search' part of the BusinessDataRepository relies on a third-party called ElasticSearch (ES). The update of new documents from Axon.ivy Engine to ES takes some milliseconds so latest updated document contents may can not be addresses for some milliseconds. But we favor the tremendous search features way over an 100% up to date result. Search and reading of data is much more important than write/update scenarios.

(26.02 at 03:34) Reguel Werme... ♦♦ Reguel%20Wermelinger's gravatar image

Hi @Reguel Wermelinger, according to your explanation, the latency should be some milliseconds and this is acceptable if it is true. However, the actual latency which we have experience is >= 1000 milliseconds. I have found that the 1-second latency probably comes from the implementation of JestIndexSynchronizer#updateDocument.

(27.02 at 03:23) vagabond vagabond's gravatar image

The question is on what system is it 1 second. I'm not aware of suge a long delay in a productive environment. In dev environment the embedded ElasticSearch instance does run with limited power in order to spare especially RAM for the Designer. But in production the delay should not be in this area. However - if so you could measure how long a normal PUT to elasticSearch takes. And whether your index has any strange customization in it. If you can detect an additional delay through our communication implementation we are definitively willing to give a hand for help.

(27.02 at 08:16) Reguel Werme... ♦♦ Reguel%20Wermelinger's gravatar image
link

answered 27.02 at 08:21

Alex%20Suter's gravatar image

Alex Suter ♦♦
2.4k91937
accept rate: 84%

Thank @Alex Suter, I also came across this when I looked into ES settings. Setting this value to -1 is a no-go and setting it more than the default will increase the latency between the time the index is created and the time the index can be visible for search.

(28.02 at 00:24) vagabond vagabond's gravatar image

Maybe the platform should provide an api for this - refresh=wait_for: https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-refresh.html

(28.02 at 01:24) Alex Suter ♦♦ Alex%20Suter's gravatar image
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×19
×14
×7

Asked: 22.02 at 01:16

Seen: 689 times

Last updated: 28.02 at 01:24