<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://hibernate.atlassian.net">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0" bgcolor="#f5f5f5"> <!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="https://secure.gravatar.com/avatar/2a8bdd4ffd282b7185c74b52ab452617?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FYR-6.png&size=48&s=48" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="yrodiere" style="color:#6c797f;; color: #3b73af; text-decoration: none" id="email_yrodiere" href="https://hibernate.atlassian.net/secure/ViewProfile.jspa?accountId=557058%3A58fa1ced-171a-4c00-97e8-5d70d442cc4b"> Yoann Rodière </a> <strong>commented</strong> on <a href="https://hibernate.atlassian.net/browse/HSEARCH-3900?atlOrigin=eyJpIjoiY2VjZjMzNzJiNDhmNDFiYjgwZDNjMTI4OGU2OTVlYTAiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-0ba35d9b-f95a-4ef6-ab04-78c2ff4432a7" height="16" width="16" border="0" align="absmiddle" alt="Bug"> HSEARCH-3900</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; border-spacing: 0; border-collapse: separate">
<tbody>
<tr> <!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td style="vertical-align: top;; padding: 0; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3900?atlOrigin=eyJpIjoiY2VjZjMzNzJiNDhmNDFiYjgwZDNjMTI4OGU2OTVlYTAiLCJwIjoiaiJ9" style="color: #3b73af; text-decoration: none">Re: drop-and-create strategy is not working at app. startup when "hibernate.search.autoregister_listeners" is set to false. And if true, startup index creation thread are not released, sending the app in out of memory.</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="text-paragraph-pattern-top" class="email-content-main mobile-expand comment-top-pattern" style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; border-bottom: none; padding-bottom: 0" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0; border-collapse: collapse; padding: 0 0 10px">
<blockquote style="margin: 10px 0 0; margin-top: 0; border-left: 1px solid #cccccc; color: #707070; margin-left: 19px; padding: 10px 20px">
<p style="margin: 10px 0 0; margin-top: 0">We have this set correctly.</p>
</blockquote> <p style="margin: 10px 0 0">Then you probably have something like that somewhere:</p>
<div class="code panel" style="border-width: 1px;; border: 1px solid #cccccc; background: #f5f5f5; font-size: 12px; line-height: 1.333; font-family: monospace; border: 1px solid #cccccc; -moz-border-radius: 3px; border-radius: 3px; margin: 9px 0">
<div class="codeContent panelContent" style="padding: 9px 12px">
<pre class="code-java" style="margin: 10px 0 0; margin-top: 0; max-height: 30em; overflow: auto; white-space: pre-wrap; word-wrap: normal">
hibernate.search.myIndex1.indexmanager = <something that is not <span class="code-quote" style="color: #009100">"elasticsearch"</span>>
</pre>
</div>
</div> <p style="margin: 10px 0 0">Specifically, I can tell you from your thread dump that "TABLEWC30" is using a Lucene index, not an Elasticsearch one.</p>
<blockquote style="margin: 10px 0 0; border-left: 1px solid #cccccc; color: #707070; margin-left: 19px; padding: 10px 20px">
<p style="margin: 10px 0 0; margin-top: 0">But on startup we are not indexing anything.</p>
</blockquote> <p style="margin: 10px 0 0">The transport threads are also used to send requests to Elasticsearch to drop/create indexes, or simply to execute search queries. My point is, if they do something, it's because you asked them to.</p>
<blockquote style="margin: 10px 0 0; border-left: 1px solid #cccccc; color: #707070; margin-left: 19px; padding: 10px 20px">
<p style="margin: 10px 0 0; margin-top: 0">> The threads will not stop as long as your application is running. That's on purpose.</p>
</blockquote>
<blockquote style="margin: 10px 0 0; border-left: 1px solid #cccccc; color: #707070; margin-left: 19px; padding: 10px 20px">
<p style="margin: 10px 0 0; margin-top: 0">Why is so?</p>
</blockquote> <p style="margin: 10px 0 0">Because spawning new threads every time we need to do something would be disastrous performance-wise.</p>
<blockquote style="margin: 10px 0 0; border-left: 1px solid #cccccc; color: #707070; margin-left: 19px; padding: 10px 20px">
<p style="margin: 10px 0 0; margin-top: 0">> Was your only problem that "Hibernate Search creates threads"?</p>
</blockquote>
<blockquote style="margin: 10px 0 0; border-left: 1px solid #cccccc; color: #707070; margin-left: 19px; padding: 10px 20px">
<p style="margin: 10px 0 0; margin-top: 0">No, issue is these threads are throwing my application in out of memory condition.</p>
</blockquote> <p style="margin: 10px 0 0">Let's say you have two threads T1 and T2, and 500MB of memory available. Let's say T1 allocates 499MB of memory, then T2 allocates 2MB of memory. The <tt>OutOfMemoryError</tt> will occur in T2, but obviously T1 is the biggest consumer of memory.</p> <p style="margin: 10px 0 0">Just because the <tt>OutOfMemoryError</tt> happpened in these threads, doesn't necessarily mean these threads are the source of your trouble. They could be, but that's far from certain.</p>
<blockquote style="margin: 10px 0 0; border-left: 1px solid #cccccc; color: #707070; margin-left: 19px; padding: 10px 20px">
<p style="margin: 10px 0 0; margin-top: 0">Also If I switch elastic search off in my app, then the application runs fine.</p>
</blockquote> <p style="margin: 10px 0 0">Certainly. If you remove Hibernate Search <b>and</b> Elasticsearch from your application, you will need less memory. That doesn't mean that Hibernate Search and Elasticsearch consume too much memory; maybe there is simply not enough memory available.</p> <p style="margin: 10px 0 0">In your case, the Elasticsearch transport threads are probably just serializing JSON to sent requests to drop/create indexes. Unless you have a very, very complicated schema, I wouldn't expect that to cause too many memory allocations. On the Elasticsearch side, however, the creation of an index can result in significant memory allocations. Maybe Elasticsearch is eating up the memory and your application no longer has enough RAM available?</p> <p style="margin: 10px 0 0">What is the amount of RAM available on your server, and what are the memory flags (<tt>-Xms</tt>/<tt>-Xmx</tt> on your JVMs (Tomcat <b>and</b> Elasticsearch)? What are the processes using the most memory when you encounter your <tt>OutOfMemoryError</tt>, and how much memory are they using?</p> <p style="margin: 10px 0 0">Also, if you actually have data that points toward Hibernate Search allocating too much memory (for example Java Flight Recorder recordings), please show me that data, so I can pinpoint which class allocates so much. You can find various tutorials on the web, <span class="error">[this one]</span>(<a href="https://stackoverflow.com/a/54143988/6692043" class="external-link" rel="nofollow noreferrer" style="color: #3b73af; text-decoration: none">https://stackoverflow.com/a/54143988/6692043</a>) looks like a good starting point.</p> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff"> <script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"description": "View Issue",
"potentialAction": {
"@type": "ViewAction",
"target": "https://hibernate.atlassian.net/browse/HSEARCH-3900?inbox=true&focusedCommentId=106578&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-106578",
"name": "View Comment"
},
"publisher": {
"@type": "Organization",
"name": "Atlassian",
"url": "https://www.atlassian.com"
}
}
</script>
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3900#add-comment?atlOrigin=eyJpIjoiY2VjZjMzNzJiNDhmNDFiYjgwZDNjMTI4OGU2OTVlYTAiLCJwIjoiaiJ9" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-bbb7bb10-3d9e-4b5d-9f9f-e67df0f15b57" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://hibernate.atlassian.net/browse/HSEARCH-3900#add-comment?atlOrigin=eyJpIjoiY2VjZjMzNzJiNDhmNDFiYjgwZDNjMTI4OGU2OTVlYTAiLCJwIjoiaiJ9" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr> <!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-mobile-separated-links" class="mobile-resize-text" width="100%" colspan="2" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> Get Jira notifications on your phone! Download the Jira Cloud app for <a href="https://play.google.com/store/apps/details?id=com.atlassian.android.jira.core&referrer=utm_source%3DNotificationLink%26utm_medium%3DEmail" style="color: #3b73af; text-decoration: none">Android</a> or <a href="https://itunes.apple.com/app/apple-store/id1006972087?pt=696495&ct=EmailNotificationLink&mt=8" style="color: #3b73af; text-decoration: none">iOS</a>
<hr> </td>
</tr>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian Jira <span id="footer-build-information">(v1001.0.0-SNAPSHOT#100125-<span title="29f7b8126e490ce758eae6051620be7946c857a3" data-commit-id="29f7b8126e490ce758eae6051620be7946c857a3}">sha1:29f7b81</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0; mso-table-rspace: 0">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0; border-collapse: collapse; padding-top: 3px; opacity: 0.150"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-c8d21196-aaf6-4c39-9dd0-bd846c9b48a7" alt="Atlassian logo" title="Atlassian logo" width="192" height="24" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
</body>
</html>