<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://issues.jboss.org">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt" bgcolor="#f5f5f5">
<!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0px; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="https://static.jboss.org/developer/gravatar/307bdfa3d580d78abd03814ba5b451a7?d=mm&s=48" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="aliok2" id="email_aliok2" href="https://issues.jboss.org/secure/ViewProfile.jspa?name=aliok2" style="color:#3b73af;; color: #3b73af; text-decoration: none">Ali Ok</a> <strong>edited a comment</strong> on <a href="https://issues.jboss.org/browse/AEROGEAR-7937" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-37c00ab8-ec8d-4a59-8b88-b279b0eecedc" height="16" width="16" border="0" align="absmiddle" alt="Task"> AEROGEAR-7937</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; border-collapse: separate">
<tbody>
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td style="vertical-align: top;; padding: 0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.jboss.org/browse/AEROGEAR-7937" style="color: #3b73af; text-decoration: none">Re: Spike: how to enable OpenShift centralized logging</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="text-paragraph-pattern-top" class="email-content-main mobile-expand comment-top-special-margin comment-top-pattern" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; border-bottom: none; padding-bottom: 0" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0; padding-top: 10px"> <span class="diffcontext">h2. Enabling centralized logging on OpenShift<br><br>h4. How<br><br>There's an Ansible playbook in openshift-ansible repository that takes enables logging.<br>https://github.com/openshift/openshift-ansible<br><br>h4. Minishift<br>It is not possible to enable it on a cluster setup using Minishift.<br><br>* Running Ansible locally and targeting Minishift cluster is very problematic (needs RHEL on local machine, mising tools, assumptions about the local machine don't work)<br>* Running Ansible within the Minishift VM is also not possible because that VM lacks lots of tools (Ansible, Pythong, Git etc.) and it is super hard to install them<br><br>h4. oc cluster up<br>Not possible either.<br><br>Targeting the local cluster with Ansible sounds good, but it ends up in similar problems to Minishift one.<br><br>.h4 Remote cluster that's created using {{cluster_create}} pipeline<br><br>I was able to enable logging on "apb-testing" cluster Pavel gave me. Access to this cluster requires Red Hat VPN.<br><br>Steps:<br><br>1. SSH into master node:<br>{code}<br>ssh hadmin@apb-testing.skunkhenry.com<br>{code}<br>Your key should be already in authorized keys for hadmin user.<br>OpenShift console for this cluster is at https://apb-testing.skunkhenry.com:8443/console<br><br>2. Clone openshift-ansible<br>{code}<br>git clone https://github.com/openshift/openshift-ansible.git<br>{code}<br><br>3. Checkout the relevant tag of that repo:<br>{code}<br># find the correct one from https://github.com/openshift/openshift-ansible/releases<br># that matches the OpenShift version you get using 'oc version' command<br>git checkout openshift-ansible-3.9.33-1<br>{code}<br><br>4. There was a bug in that version of the Ansible role. It is fixed in master. Make that update manually according to https://github.com/openshift/openshift-ansible/blob/1cb319e8030961f77d751f4be115fe5ddba89bda/roles/openshift_logging_elasticsearch/handlers/main.yml#L8<br><br>5. Login with system admin<br>{code}<br>oc login -u system:admin<br>{code}<br><br>6. Enable logging<br>{code}<br>ansible-playbook -i /home/hadmin/.config/openshift/hosts ./playbooks/openshift-logging/config.yml -e openshift_logging_install_logging=true -e openshift_logging_es_allow_external=True -e openshift_logging_es_hostname=elasticsearch.example.com<br>{code}<br><br>The cluster is created initially with cluster_create pipeline and that pipeline stores Ansible inventory in the master node at /home/hadmin/.config/openshift/hosts<br><br>7. Wait until Ansible playbook completes and all pods are up on OpenShift's {{logging}} project<br><br>8. Update the route in {{logging}} project for ElasticSearch. Change it using the UI from elasticsearch.example.com to something you like such as es.apb-testing.skunkhenry.com<br><br>h2. Disabling OpenShift Centralized logging<br>{code}<br>ansible-playbook -i /home/hadmin/.config/openshift/hosts ./playbooks/openshift-logging/config.yml -e openshift_logging_install_logging=false<br>{code}<br><br>h2. How it works?<br><br>* A Fluentd instance is created per node. Using a DaemonSet with a node selector.<br>* Fluentd collects logs from all pods and sends them to ElasticSearch.<br>* Logs are pushed to different indices:<br>** Operation logs: pushed using "operation.*" indices. These are Kubernetes infra logs like container creation, deployments, project creation etc.<br>** Project logs: pushed using "project.*" indices. These are logs from the user pods. Like, audit logs of sync service.<br><br>We are interested in project logs in our use cases.<br><br>Project logs are pushed to indices that have the project name in their name.<br>For example, "project.datasync.c360bfe5-bbfb-11e8-87ae-fa163e4c9c9e.2018.09.20". The format is "project.<project name>.<project uid>.<yyyy.mm.dd>".<br>This means, all logs from all pods within a single project would go to the same index.<br>We do have what pod it is, or what service it is though in the document itself.<br><br>Sample document:<br>{code:json}<br><br> "_index": "project.datasync.c360bfe5-bbfb-11e8-87ae-fa163e4c9c9e.2018.09.20",<br> "_type": "com.redhat.viaq.common",<br> "_id": "MmNhZTZlZTctZWU5ZS00YzFkLWJjNDQtNjQwYmVhZjc3OTFh",<br> "_score": null,<br> "_source": {<br> "level": "30",<br> "msg": "request completed",<br> "pid": 19,<br> "hostname": "172.16.72.62",<br> "req": {<br> "id": 8453,<br> "method": "GET",<br> "url": "/healthz",<br> "headers": {<br> "host": "10.128.1.223:8000",<br> "user-agent": "kube-probe/1.9",<br> "accept-encoding": "gzip",<br> "connection": "close"<br> },<br> "remoteAddress": "::ffff:10.128.0.1",<br> "remotePort": 52172<br> },<br> "res": {<br> "statusCode": 200,<br> "header": "HTTP/1.1 200 OK\r\nX-Powered-By: Express\r\nAccess-Control-Allow-Origin: *\r\nContent-Type: application/json; charset=utf-8\r\nContent-Length: 53\r\nETag: W/\"35-3EKymgknC0UZgUjhN7E3BXc98h8\"\r\nDate: Thu, 20 Sep 2018 10:30:32 GMT\r\nConnection: close\r\n\r\n"<br> },<br> "responseTime": 6,<br> "v": 1,<br> "docker": {<br> "container_id": "dffd0934be27113027208ddf4aed233b162fedb3bc758f6cd8956980aa90982f"<br> },<br> "kubernetes": {<br> "container_name": "data-sync-server",<br> "namespace_name": "datasync",<br> "pod_name": "data-sync-server-2-2nk7p",<br> "pod_id": "df4a7a42-bbfc-11e8-87ae-fa163e4c9c9e",<br> "labels": {<br> "app": "data-sync",<br> "deployment": "data-sync-server-2",<br> "deploymentconfig": "data-sync-server",<br> "service": "data-sync-server"<br> },<br> "host": "172.16.72.62",<br> "master_url": "https://kubernetes.default.svc.cluster.local",<br> "namespace_id": "c360bfe5-bbfb-11e8-87ae-fa163e4c9c9e"<br> },<br> "message": "{\"level\":30,\"time\":1537439432240,\"msg\":\"request completed\",\"pid\":19,\"hostname\":\"data-sync-server-2-2nk7p\",\"req\":{\"id\":8453,\"method\":\"GET\",\"url\":\"/healthz\",\"headers\":{\"host\":\"10.128.1.223:8000\",\"user-agent\":\"kube-probe/1.9\",\"accept-encoding\":\"gzip\",\"connection\":\"close\"},\"remoteAddress\":\"::ffff:10.128.0.1\",\"remotePort\":52172},\"res\":{\"statusCode\":200,\"header\":\"HTTP/1.1 200 OK\\r\\nX-Powered-By: Express\\r\\nAccess-Control-Allow-Origin: *\\r\\nContent-Type: application/json; charset=utf-8\\r\\nContent-Length: 53\\r\\nETag: W/\\\"35-3EKymgknC0UZgUjhN7E3BXc98h8\\\"\\r\\nDate: Thu, 20 Sep 2018 10:30:32 GMT\\r\\nConnection: close\\r\\n\\r\\n\"},\"responseTime\":6,\"v\":1}\n",<br> "pipeline_metadata": {<br> "collector": {<br> "ipaddr4": "10.128.0.7",<br> "ipaddr6": "fe80::e06d:e7ff:fec1:3d8c",<br> "inputname": "fluent-plugin-systemd",<br> "name": "fluentd",<br> "received_at": "2018-09-20T10:30:32.801089+00:00",<br> "version": "0.12.42 1.6.0"<br> }<br> },<br> "@timestamp": "2018-09-20T10:30:32.240454+00:00",<br> "viaq_msg_id": "MmNhZTZlZTctZWU5ZS00YzFkLWJjNDQtNjQwYmVhZjc3OTFh"<br> },<br> "fields": {<br> "pipeline_metadata.collector.received_at": [<br> 1537439432801<br> ],<br> "@timestamp": [<br> 1537439432240<br> ]<br> },<br> "sort": [<br> 1537439432240<br> ]<br>}<br>{code}<br><br>h2. Problem: logs from pods are treated as strings, even when they're JSON<br></span> <span class="diffaddedchars" style="background-color:#ddfade;"><br>By default, OpenShift logging is setup in a way that causes ElasticSearch</span> <span class="diffcontext">to</span> <span class="diffaddedchars" style="background-color:#ddfade;"> treat log messages as strings (https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/index_templates/com.redhat.viaq-openshift-project.template.json#L46).<br>But, in our case in sync audit logs we log messages as JSON and we want them being treated as objects. So that we can do search/aggregation on the properties.<br><br>There are 2 possible places to change this behavior.<br><br>h4. Fluentd<br><br>Well, not really possible here:<br><br>* There's a default index template here that matches every index: https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/index_templates/com.redhat.viaq-openshift-project.template.json#L46<br>* So, there needs to</span> <span class="diffcontext"> be</span> <span class="diffaddedchars" style="background-color:#ddfade;"> some change in ElasticSearch anyway.<br>* Plus, Fluentd config is set up in a complex way: https://github.com/openshift/openshift-ansible/blob/f1ae5deec6f9f5b6e6f63e88b2d5682ea40234c6/roles/openshift_logging_fluentd/templates/fluent.conf.j2#L42<br><br>h4. ElasticSearch<br><br>We can change the ElasticSearch default index template set up here: https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/index_templates/com.redhat.viaq-openshift-project.template.json<br>However we can't merge that in that repo. Thus, we can simply call the ElasticSearch API to override the default index.<br><br>ElasticSearch supports multiple index templates and the index template we create will only override the "message" field related mapping (https://www.elastic.co/guide/en/elasticsearch/reference/2.4/indices-templates.html#multiple-templates).<br><br>First, login:<br>{code}<br>oc login<br># with user "admin"<br>token=$(oc whoami -t)<br>{code}<br><br>Check if our named template exists already:<br>{code}<br>curl -k -H "Authorization: Bearer $token" -H "X-Proxy-Remote-User: $(oc whoami)" -H "X-Forwarded-For: 127.0.0.1" https://es.apb-testing.skunkhenry.com/_template/aerogear_data_sync_log_template<br>{code}<br><br><br>Create or update the index template:<br>{code}<br>curl -k -H "Authorization: Bearer $token" -H "X-Proxy-Remote-User: $(oc whoami)" -H "X-Forwarded-For: 127.0.0.1" -XPUT https://es.apb-testing.skunkhenry.com/_template/aerogear_data_sync_log_template -d '<br>{<br> "template" : "project.datasync.*",<br> "order" : 100,<br> "dynamic_templates": [<br> {<br> "message_field": {<br> "mapping": {<br> "type": "object"<br> },<br> "match": "message"<br> }<br> }<br> ],<br> "mappings": {<br> "message": {<br> "enabled": true,<br> "properties": {<br> "tag": {"type": "string", "index": "not_analyzed"},<br> "requestId": {"type": "integer"},<br> "operationType": {"type": "string", "index": "not_analyzed"},<br> "parentTypeName": {"type": "string", "index": "not_analyzed"},<br> "path": {"type": "string", "index": "analyzed"},<br> "success": {"type": "boolean"},<br> "dataSourceType": {"type": "string", "index": "not_analyzed"}<br> }<br> }<br> }<br>}<br>'<br>{code}<br><br>Here,<br>* {{"template" : "project.datasync.*"}} says this template will only match the indices for that pattern<br>* {{"properties": {...}}} defines mapping for the relevant fields in the message data. This is required. Fields that are not defined here won't be available as separate fields on the ElasticSearch document.<br>* {{"order" : 100}} tells ElasticSearch to use this after the template defined by OpenShift logging, which has an order of 10 (https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/index_templates/com.redhat.viaq-openshift-project.template.json#L936)<br><br><br>Delete the index template (if necessary):<br>{code}<br>curl -k -H "Authorization: Bearer $token" -H "X-Proxy-Remote-User: $(oc whoami)" -H "X-Forwarded-For: 127.0.0.1" -XDELETE https://es.apb-testing.skunkhenry.com/_template/aerogear_data_sync_log_template<br>{code}<br><br>h4. How index template works<br><br>After we create the index template above, when a document comes ElasticSearch will do the mappings for all fields except "message" according the other default template.<br>"message" field will be mapped using the mapping we overrode.<br><br>But, there's a problem: changing ElasticSearch index templates won't affect the mapping definitions of existing indices. Only the new indices will use our template.<br>So, even though a new document that is written to "project.datasync.c360bfe5-bbfb-11e8-87ae-fa163e4c9c9e.2018.09.20" index matches, it won't use our new mapping.<br>The solution is to wait until midnight. Fluentd changes the index name nightly and the new index will use the mapping we defined.<br><br>If we would like to update mappings for the existing indices ------ TB</span> <span class="diffcontext"> documented</span> <span class="diffaddedchars" style="background-color:#ddfade;">? reindex overwrite/copy<br><br>Problem:<br>- Stackoverflow problem</span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://issues.jboss.org/browse/AEROGEAR-7937#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-3d2e2fe2-6d08-416a-af30-7478ac83bcf2" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://issues.jboss.org/browse/AEROGEAR-7937#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0px; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian JIRA <span id="footer-build-information">(v7.5.0#75005-<span title="fd8c849d4e278dd8bbaccc61e707a716ad697024" data-commit-id="fd8c849d4e278dd8bbaccc61e707a716ad697024}">sha1:fd8c849</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-4c6c28b9-712a-4586-b2ba-f58db8e42e8d" alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
</body>
</html>