[searchisko-dev] How to speed up full contributors resync

[Libor Krzyžanek]

Hi,
I’m investigating how to get faster full resync form jive to searchisko.

Jive performs not so bad - aproxx 25 users in 2 seconds which leads aprox 8.5 hours for 380.000 users.

But the full sync task looks like 10 days.

I checked production and looks like flush operation is very expensive - 5.7 hours (refresh 30 mins) so I focused where it is used.

See here: 
https://github.com/searchisko/searchisko/blob/master/api/src/main/java/org/searchisko/api/service/ContributorService.java#L588 <https://github.com/searchisko/searchisko/blob/master/api/src/main/java/org/searchisko/api/service/ContributorService.java#L588>

The searchClientService.performIndexFlushAndRefreshBlocking(SEARCH_INDEX_NAME); is called twice within same method.

Isn’t second call redundant?
(https://github.com/searchisko/searchisko/blob/master/api/src/main/java/org/searchisko/api/service/ContributorService.java#L656 <https://github.com/searchisko/searchisko/blob/master/api/src/main/java/org/searchisko/api/service/ContributorService.java#L656>)


Other question is where we’re with ES 1.4 with flush and refresh indices methods. Isn’t refresh enough? How it changed from ES 0.9 from time when the contributor service was implemented?

Thanks,

Libor Krzyžanek
jboss.org <http://jboss.org/> Development Team

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/searchisko-dev/attachments/20150717/729ba6f3/attachment.html