<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://issues.jboss.org">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt" bgcolor="#f5f5f5">
<!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0px; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="https://static.jboss.org/developer/gravatar/307bdfa3d580d78abd03814ba5b451a7?d=mm&s=48" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="aliok2" id="email_aliok2" href="https://issues.jboss.org/secure/ViewProfile.jspa?name=aliok2" style="color:#3b73af;; color: #3b73af; text-decoration: none">Ali Ok</a> <strong>edited a comment</strong> on <a href="https://issues.jboss.org/browse/AEROGEAR-7937" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-c77f1b39-2ddb-4c2a-bdd4-cdd653815cc6" height="16" width="16" border="0" align="absmiddle" alt="Task"> AEROGEAR-7937</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; border-collapse: separate">
<tbody>
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td style="vertical-align: top;; padding: 0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.jboss.org/browse/AEROGEAR-7937" style="color: #3b73af; text-decoration: none">Re: Spike: how to enable OpenShift centralized logging</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="text-paragraph-pattern-top" class="email-content-main mobile-expand comment-top-special-margin comment-top-pattern" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; border-bottom: none; padding-bottom: 0" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0; padding-top: 10px"> <span class="diffcontext">h2. Enabling centralized logging on OpenShift<br><br>h4. How<br><br>There's an Ansible playbook in openshift-ansible repository that takes enables logging.<br>https://github.com/openshift/openshift-ansible<br><br>h4. Minishift<br>It is not possible to enable it on a cluster setup using Minishift.<br><br>* Running Ansible locally and targeting Minishift cluster is very problematic (needs RHEL on local machine, mising tools, assumptions about the local machine don't work)<br>* Running Ansible within the Minishift VM is also not possible because that VM lacks lots of tools (Ansible, Pythong, Git etc.) and it is super hard to install them<br><br>h4. oc cluster up<br>Not possible either.<br><br>Targeting the local cluster with Ansible sounds good, but it ends up in similar problems to Minishift one.<br><br>.h4 Remote cluster that's created using {{cluster_create}} pipeline<br><br>I was able to enable logging on "apb-testing" cluster Pavel gave me. Access to this cluster requires Red Hat VPN.<br><br>Steps:<br><br>1. SSH into master node:<br>{code}<br>ssh hadmin@apb-testing.skunkhenry.com<br>{code}<br>Your key should be already in authorized keys for hadmin user.<br>OpenShift console for this cluster is at https://apb-testing.skunkhenry.com:8443/console<br><br>2. Clone openshift-ansible<br>{code}<br>git clone https://github.com/openshift/openshift-ansible.git<br>{code}<br><br>3. Checkout the relevant tag of that repo:<br>{code}<br># find the correct one from https://github.com/openshift/openshift-ansible/releases<br># that matches the OpenShift version you get using 'oc version' command<br>git checkout openshift-ansible-3.9.33-1<br>{code}<br><br>4. There was a bug in that version of the Ansible role. It is fixed in master. Make that update manually according to https://github.com/openshift/openshift-ansible/blob/1cb319e8030961f77d751f4be115fe5ddba89bda/roles/openshift_logging_elasticsearch/handlers/main.yml#L8<br><br>5. Login with system admin<br>{code}<br>oc login -u system:admin<br>{code}<br><br>6. Enable logging<br>{code}<br>ansible-playbook -i /home/hadmin/.config/openshift/hosts ./playbooks/openshift-logging/config.yml -e openshift_logging_install_logging=true -e openshift_logging_es_allow_external=True -e openshift_logging_es_hostname=elasticsearch.example.com<br>{code}<br><br>The cluster is created initially with cluster_create pipeline and that pipeline stores Ansible inventory in the master node at /home/hadmin/.config/openshift/hosts<br><br>7. Wait until Ansible playbook completes and all pods are up on OpenShift's {{logging}} project<br><br>8. Update the route in {{logging}} project for ElasticSearch. Change it using the UI from elasticsearch.example.com to something you like such as es.apb-testing.skunkhenry.com<br><br>h2. Disabling OpenShift Centralized logging<br>{code}<br>ansible-playbook -i /home/hadmin/.config/openshift/hosts ./playbooks/openshift-logging/config.yml -e openshift_logging_install_logging=false<br>{code}<br><br>h2. How it works?<br><br>* A Fluentd instance is created per node. Using a DaemonSet with a node selector.<br>* Fluentd collects logs from all pods and sends them to ElasticSearch.<br>* Logs are pushed to different indices:<br>** Operation logs: pushed using "operation.*" indices. These are Kubernetes infra logs like container creation, deployments, project creation etc.<br>** Project logs: pushed using "project.*" indices. These are logs from the user pods. Like, audit logs of sync service.<br><br>We are interested in project logs in our use cases.<br><br>Project logs are pushed to indices that have the project name in their name.<br>For example, "project.datasync.c360bfe5-bbfb-11e8-87ae-fa163e4c9c9e.2018.09.20". The format is "project.<project name>.<project uid>.<yyyy.mm.dd>".</span> <span class="diffaddedchars" style="background-color:#ddfade;"><br>This means, all logs from all pods within a single project would go to the same index.<br>We do have what pod it is, or what service it is though in the document itself.<br><br>Sample document:<br>{code:json}<br><br> "_index": "project.datasync.c360bfe5-bbfb-11e8-87ae-fa163e4c9c9e.2018.09.20",<br> "_type": "com.redhat.viaq.common",<br> "_id": "MmNhZTZlZTctZWU5ZS00YzFkLWJjNDQtNjQwYmVhZjc3OTFh",<br> "_score": null,<br> "_source": {<br> "level": "30",<br> "msg": "request completed",<br> "pid": 19,<br> "hostname": "172.16.72.62",<br> "req": {<br> "id": 8453,<br> "method": "GET",<br> "url": "/healthz",<br> "headers": {<br> "host": "10.128.1.223:8000",<br> "user-agent": "kube-probe/1.9",<br> "accept-encoding": "gzip",<br> "connection": "close"<br> },<br> "remoteAddress": "::ffff:10.128.0.1",<br> "remotePort": 52172<br> },<br> "res": {<br> "statusCode": 200,<br> "header": "HTTP/1.1 200 OK\r\nX-Powered-By: Express\r\nAccess-Control-Allow-Origin: *\r\nContent-Type: application/json; charset=utf-8\r\nContent-Length: 53\r\nETag: W/\"35-3EKymgknC0UZgUjhN7E3BXc98h8\"\r\nDate: Thu, 20 Sep 2018 10:30:32 GMT\r\nConnection: close\r\n\r\n"<br> },<br> "responseTime": 6,<br> "v": 1,<br> "docker": {<br> "container_id": "dffd0934be27113027208ddf4aed233b162fedb3bc758f6cd8956980aa90982f"<br> },<br> "kubernetes": {<br> "container_name": "data-sync-server",<br> "namespace_name": "datasync",<br> "pod_name": "data-sync-server-2-2nk7p",<br> "pod_id": "df4a7a42-bbfc-11e8-87ae-fa163e4c9c9e",<br> "labels": {<br> "app": "data-sync",<br> "deployment": "data-sync-server-2",<br> "deploymentconfig": "data-sync-server",<br> "service": "data-sync-server"<br> },<br> "host": "172.16.72.62",<br> "master_url": "https://kubernetes.default.svc.cluster.local",<br> "namespace_id": "c360bfe5-bbfb-11e8-87ae-fa163e4c9c9e"<br> },<br> "message": "{\"level\":30,\"time\":1537439432240,\"msg\":\"request completed\",\"pid\":19,\"hostname\":\"data-sync-server-2-2nk7p\",\"req\":{\"id\":8453,\"method\":\"GET\",\"url\":\"/healthz\",\"headers\":{\"host\":\"10.128.1.223:8000\",\"user-agent\":\"kube-probe/1.9\",\"accept-encoding\":\"gzip\",\"connection\":\"close\"},\"remoteAddress\":\"::ffff:10.128.0.1\",\"remotePort\":52172},\"res\":{\"statusCode\":200,\"header\":\"HTTP/1.1 200 OK\\r\\nX-Powered-By: Express\\r\\nAccess-Control-Allow-Origin: *\\r\\nContent-Type: application/json; charset=utf-8\\r\\nContent-Length: 53\\r\\nETag: W/\\\"35-3EKymgknC0UZgUjhN7E3BXc98h8\\\"\\r\\nDate: Thu, 20 Sep 2018 10:30:32 GMT\\r\\nConnection: close\\r\\n\\r\\n\"},\"responseTime\":6,\"v\":1}\n",<br> "pipeline_metadata": {<br> "collector": {<br> "ipaddr4": "10.128.0.7",<br> "ipaddr6": "fe80::e06d:e7ff:fec1:3d8c",<br> "inputname": "fluent-plugin-systemd",<br> "name": "fluentd",<br> "received_at": "2018-09-20T10:30:32.801089+00:00",<br> "version": "0.12.42 1.6.0"<br> }<br> },<br> "@timestamp": "2018-09-20T10:30:32.240454+00:00",<br> "viaq_msg_id": "MmNhZTZlZTctZWU5ZS00YzFkLWJjNDQtNjQwYmVhZjc3OTFh"<br> },<br> "fields": {<br> "pipeline_metadata.collector.received_at": [<br> 1537439432801<br> ],<br> "@timestamp": [<br> 1537439432240<br> ]<br> },<br> "sort": [<br> 1537439432240<br> ]<br>}<br>{code}</span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://issues.jboss.org/browse/AEROGEAR-7937#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-64b36c1f-0f3e-40a4-b2c1-83feb3b04e78" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://issues.jboss.org/browse/AEROGEAR-7937#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0px; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian JIRA <span id="footer-build-information">(v7.5.0#75005-<span title="fd8c849d4e278dd8bbaccc61e707a716ad697024" data-commit-id="fd8c849d4e278dd8bbaccc61e707a716ad697024}">sha1:fd8c849</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-b0364f93-0141-4bbe-ac26-cbc82db3b238" alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
</body>
</html>