Configuring the Prometheus JSON Exporter to Parse a JSON Array

 Background

The Prometheus JSON Exporter allows you to parse arbitrary JSON data into Prometheus metrics. You'll even find some examples at the link. The problem is that all of the examples show a single JSON object. What is the syntax supposed to be if you're dealing with JSON that is an array, like this data? This question came up on Reddit.

Solution

The solution is to specify the path as:

path: '{[*]}'

That's it. That will return the entire array as a list, which is what's needed to have the JSON Exporter loop through it. 

Here's a link to the github gist with more details about how you can use the above information.

I bet you don't fully understand the power of a CI/CD pipeline

If your team is delivering something digital, you MUST use a CI/CD pipeline. 

I'm sure you've heard of CI/CD (Continuous Integration/Continuous Development) pipelines, but I bet a lot of you don't truly understand how powerful they can be. Before the other day, I basically understood their power, but then I submitted my first Pull Request to a huge open source project and was simply blown away. A Pull Request (I hate the name because the words don't make sense to me, but that's the name) is a mechanism for a developer/contributor to notify team members that they have completed a feature or change. So it's really a request to merge a change into the code base. 

In my case, I was reading the Grafana Agent documentation and saw an error that bugged me. There was an incorrect statement in the technical description. The wrong label was specified. I've run across this type of error in numerous vendor documents, so I'm used to it, but ti still gets me every time I come across one. The difference here was that the error was close to the bottom of the page, and at the bottom was this group of links:

So I clicked on "Suggest an edit" and was taken to the Github repository storing the docs. I already had an account, so I made the small change I needed, and it automatically created a new branch for me with the change and prompted me to make a Pull Request. So I did that, and it let me know that the first issue was that I needed to sign the Contributor License Agreement, and it provided a link to that. I signed the agreement, and the pull request automatically got put into a "Needs Review" state and was assigned to one of the maintainers. So I figured "Well, I did something good. Maybe that update will show up on the website one day, eventually". A couple of hours later I got an email stating that my pull request was reviewed, approved, and merged into the main trunk. So I figured I would check the Grafana Agent page for grins, AND MY CHANGE WAS THERE, LIVE ON THE SITE!

Now for my "bigger picture" opinion on this:

In working with large software vendors, I have made similar change requests that took me hours to complete and that were NEVER implemented in the product documentation, so I was completely amazed. After going through this process, it is my strong opinion that any company that provides documentation for their products should have a publicly available repository that allows public contributions. I realize that the legalese for any particular Contributor License Agreement would need to be ironed out, along with many other details. Or there could be a restriction that updates are allowed only by Business Partners (who have already signed numberous documents). My point is that a huge number of extremely useful updates could be crowdsourced in this way.

Sending Kibana (free/open source) Alerts via Webhook Using Fluent-Bit (free)

Background

This is a case where we helped a customer save quite a bit of money by using software they already owned rather than paying an large upcharge for additional licenses that they didn't need.

For any number of good reasons, your use case only calls for the free version of Elastic in your environment. In your environment, you also want to integrate alerts with your ticketing system. The challenge is that the free version of Kibana does not include a webhook connector for alerts. Only the Server log connector is available with the free license, whereas the Webhook connector (and others) are only available with the paid licenses.

I have a customer in the above situation. An application they purchased is bundled in an appliance running a packaged Kubernetes distribution. The application also includes Fluent Bit for log collection into Elasticsearch. The initial challenge was to send alerts to their on-prem Netcool environment when certain log messages were written. We helped them meet this challenge using the webhook output of Fluent Bit to send the appropriate messages to the Netcool message bus probe, which would then create an incident in their ticketing system for each of these alerts.

Their next requirement was to only create incidents based on some aggregation of log messages. Specifically, they obtained several Elasticsearch queries from the vendor that should be used to generate incidents. This is really straightforward when using one of the paid Elastic licenses because you can simply write a rule with the Elasticsearch query as a condition and the built-in webhook connector to define an action that sends a message. With the free license of Kibana, that connector isn't available. 

My Solution

The trick to the solution in this case is to just use the Server Log connector in Kibana to write a specifically-formatted message to the log when the Elasticsearch query condition is met. The message can be similar to:

CREATE_INCIDENT Vendor Query X has breached the prescribed threshold. Take action Y to correct.

This message is written to the log file for the Kibana pod, which is already being tail'ed by Fluent Bit. So we just needed to create a FILTER in Fluent Bit to match this log message and route that to the message bus probe. 

Tunneling X11 over SSH as a different user

Background

X11 tunneling over SSH is pretty straightforward as long as you don't need to su to another user on the target system. When you have to do that, it gets a little tricky, and that's the reason for this post.

Solution

In my case, I'm usually starting the process from a Windows server with Putty, so that's the basis for this solution. I have tested this with both xming and Moba Xterm on Windows. So before connecting to a remove server, make sure that your Windows X server is running and Putty is configured to allow X11 forwarding:

Ensure X11 tunneling is configured for your session:

 

Open the session (connect to the remove system) and ensure that your xauth exists and your local display is set so you can get your MIT-MAGIC-COOKIE:

[franktate@linux1 ~]$ echo $DISPLAY

localhost:10.0

[franktate@linux1 ~]$ xauth list | grep :10

linux1.gulfsoft.com/unix:10  MIT-MAGIC-COOKIE-1  a229706ccb496af61501ea25a9548851

[franktate@linux1 ~]$

 

Note how your display number is used to identify the appropriate MIT-MAGIC-COOKIE

 

Ensure that an X application can connect to your Windows X server by running xterm or some other application.

Switch users and set the MIT-MAGIC-COOKIE:

[franktate@linux1 ~]$ su - db2inst1

Password:

-bash: TMOUT: readonly variable

[db2inst1@linux1 ~]$ xauth add linux1.gulfsoft.com/unix:10  MIT-MAGIC-COOKIE-1  a229706ccb496af61501ea25a9548851

[db2inst1@linux1 ~]$

 

Run xterm or some other X application to be sure X is tunneled correctly. Assuming that works, now connect from the first machine to another.

 

SSH to the next hop host  and get your MIT-MAGIC-COOKIE

 

[db2inst1@linux1 ~]$ ssh -Y frank2@linux2

frank2@linux2's password:

Last failed login: Sat Feb 23 16:17:29 EST 2019 on pts/0

[frank2@linux2 ~]$ echo $DISPLAY

localhost:10.0

[frank2@linux2 ~]$ xauth list | grep :10

linux2.gulfsoft.com/unix:10  MIT-MAGIC-COOKIE-1  2d31b43034bfc9da1c0d2848c1b718d8

[frank2@linux2 ~]$

 

Run xterm or some other X application to be sure X is tunneled correctly.

Switch users and set the MIT-MAGIC-COOKIE

 

[frank2@linux2 ~]$ su - db2inst1

Password:

[db2inst1@linux2 ~]$ xauth add linux2.gulfsoft.com/unix:10  MIT-MAGIC-COOKIE-1  2d31b43034bfc9da1c0d2848c1b718d8

 

Run an X application like xterm to validate that it's working.

 

Modify kibana.yml after deploying Kibana with Helm

If you deploy Kibana using the Elastic helm chart with default values, what you'll find is that you don't have any obvious way to modify the kibana.yml file. For example, if you log into the Kibana pod with

kubectl exec --stdin --tty kibana_podname -- /bin/bash

you'll find that there's no editor available (like vi or even ed). You can cat config/kibana.yml, but the comments state that it is auto-generated. So what are you supposed to do to add an a setting to the file? For example, you might need to add a value for xpack.encryptedSavedObjects.encryptionKey so you can configure alerting.

The solution I came up with is a multi-step process:

1. Get the default values.yaml file for the chart and store that in a file with the command:

helm show values elastic/kibana > /tmp/kibana.yaml

2. Edit that file to add a section for kibana.yml under kibanaConfig. Originally, kibanaConfig is empty (set to {}). You need to change it to be something like:

3. Now (unintuitively at least to me) uninstall the helm chart with:

helm uninstall kibana

3. Then install the helm chart again with:

helm install kibana elastic/kibana -f /tmp/kibana.yaml

And that's it. Your changes will be applied and you're good to go.

I'm pretty sure there's a way to create a configMap and reference it, which would then allow you to just delete the pod to have it re-read the configMap, but I haven't figured out those exact details. Maybe in another post.

Installing .pak Files on WebSphere Application Server 8.5.x

Background

In WAS 7.0 (and possibly earlier), the WebSphere Update Installer was used to install WAS fix packs, which would have a file extension of .pak. Additionally, some other software (IBM Security Identity Manager 6, for example) that runs on WAS decided to package their updates in the same way, with .pak files to be installed with the Update Installer. WAS 8.5 moved to using IBM Installation Manager for its installation and the installation of fix packs. The last version of the WebSphere Update Installer is 7.0.0.45.

Let's say after you installed ISIM 6 on WAS 7, and then later upgraded WAS to 8.5. How do you install an ISIM 6 fixpack onto WAS 8.5?

Solution

You use the WAS 7.0.0.45 Update Installer, of course! 

WebSphere Update Installer is actually a standalone product that isn't reliant on any particular version of WebSphere to be installed. Its version number does its best to throw you off, but it works just fine when run against WAS 8.5 (or even 8.5.5.23 in my latest test).

I couldn't find this spelled out anywhere, so I thought I would share.

Installing additional software on the Rancher docker container

 If you read one of my previous posts to install Rancher on a single docker container, you may have found that it doesn't include several commands like ping, netstat, ss, and even apt. And if you run 'uname -a', you might think that the image you're in is Ubuntu, but it's not. It's SUSE Linux (the same people who maintain Rancher), and they package manager there is accessed via the command 'zypper'. So to install several of the tools you know and love, run the following

zypper install net-tools iproute2 bind fping lsof

That's it. Now you have a few more tools for debugging.

Installing Rancher in a Single Docker Container on Ubuntu 20.04

This is MUCH easier than my last couple of posts because this just takes one step after you configure your OS. Rancher is a cloud native (runs on its own K8s/K3s cluster) K8s manager and container orchestration platform. It is a competitor to Red Hat OpenShift and VMWare Tanzu.

This solution is for a DEV/practice environment. 

I've uploaded the script to configure Ubuntu as a gist to Github. So all you need to do is start with a working install of Ubuntu 20.04 desktop (my test systems have been configured with 16 cores and 64GB RAM). Your user must have sudo access (you'll be prompted for the password as the scripts run) and you can run this script:

configureUbuntu.sh

Now run this command:

docker run -d --restart=unless-stopped \

  -p 80:80 -p 443:443 \

  --privileged \

  rancher/rancher:latest

Now open your browser to http://localhost and follow the directions. It will instruct you how to get the password, then prompt you to change the password, and you're good to go. You have a local Rancher K3s cluster running in a docker container. From the UI you can probe your cluster configuration, install new applications, etc. One application of interest is:

Monitoring - This is similar to (though not exactly) the kube-prometheus-stack, with Prometheus, Grafana, and several Grafana dashboards configured.

To access the cluser from the CLI, you first need to get the container-id of your rancher container with:

docker ps

Then run:

docker exec -it container-id /bin/bash

At this point you have a root shell with access to the kubectl command.

Another application that will probably interest you is Elasticsearch. Be prepared for a LOT of failure if you try to install this one. I simply could not get it to install, and I could not determine why it failed. I couldn't find any useful logs describing where it was getting hung up. If you can figure it out, please let me know. I will keep on trying.

Update 3/16/2023: I was able to get Elasticsearch installed, and I can verify via curl to port 9200 that it's running, but that's it. I can't get any logs sent to it because the Logging app won't let me configure anything. And while I can install Kibana, I cannot figure out how to access the UI once it's installed. I've tried quite a few different things, but it's not working.

To get Elasticsearch installed, you need to perform some additional steps:

Create a directory like /home/mypv inside the Rancher docker container.

Set the owner of that directory to the user "rancher"

create a PersistentVolume in the Rancher UI to be a HostPath that points to /home/mypv with a size of 30Gi (to match the defaults for the Elasticsearch install)

In the Elasticsearch yaml, change the values of these two keys as listed here:

replicas: 1

minimumMasterNodes: 1

But, like I said, you won't be able to actually do anything with it at this point.

Installing the ELK stack and Fluent-Bit on Minikube on Ubuntu 20.04

 Background

This should be easy, but it took me a couple of days to successfully get it running, so that showed me that I needed to create this post. The problems are:

1. There are a LOT of out-of-date articles out there that are now just wrong (this one was written on 3/14/2023 and will be obsolete at some point; I apologize in advance if you are reading this after that point of obsolescence). It's not the fault of the authors. Components in this space are simply changing very quickly. Event some of the latest HOWTO documentation in the different github repositories is wrong (invalid/deprecated flag used, etc.)

2. The various helm charts include some example yaml files (yay!) that don't work without modification (dammit!).

3. The Fluent Bit helm chart defaults simply do not work with a default Elasticsearch install. Specifically, Elasticsearch requires (and there is no way to disable this) TLS connections with authentication, while the Fluent Bit chart is only set up for an HTTP connection to Elasticsearch with NO authentication.

So those are some of the reasons for this article.

This solution is for a DEV/practice environment. I can't possibly list all of the reasons why. Those reasons start with "it's on minikube" and include "the Elastic password is in plaintext", among many, many others.

Solution

I've uploaded the scripts as gists to Github. So all you need to do is start with a working install of Ubuntu 20.04 desktop (my test systems have been configured with 16 cores and 64GB RAM). Your user must have sudo access (you'll be prompted for the password as the scripts run) and you can run these two scripts in order:

configureUbuntu.sh

install-elk-and-fluentbit.sh

Installing Minikube and Prometheus on Ubuntu 20.04 as of 3/11/2023

Background

You might think it's strange that I've included a specific date in the title of this post, which means that you haven't tried to perform this kind of installation at two points in time some number of months apart. See, EVERYTHING in this space is changing rapidly. The latest and greatest way to install Prometheus in Kubernetes (whether it's actual K8s or minikube or anything else) is to install kube-prometheus-stack (https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) via a helm chart. But the specific details can be changed at any time. None of the many links I found gave me a working installation without modifying the commands at least a little. So I'm hoping this post is useful to at least one person before one or more changes make it obsolete.

Solution

Here's the script that will get everything installed. You can Google any of the commands you want to see why they're in here if you're curious. But if you just need a stinkin' cluster with Prometheus installed, the exact script to do it is below. Some additional links I used to get to this point:

Installing kube-prometheus-stack:

https://github.com/LianDuanTrain/Helm3/blob/master/3%20Helm%20Deep%20Dive/3-11%20Use%20Helm%203%20to%20Install%20Prometheus%20%20on%20MiniKube%20in%2010%20Mins.md

Better instructions that still needed to be cleaned up a bit:

https://getbetterdevops.io/setup-prometheus-and-grafana-on-kubernetes/

Recent versions of the Netcool Message Bus Probe support Kafka

 We are working with a client who needed to send events from their cloud-native application to their legacy on-prem netcool Operations Insight implementation. After researching a bit, we found that their application was already writing the events of interest to a Kafka topic. The only issue was that they had an old version of the Message Bus Probe. So we installed version 21 of the probe and used the included Nokia NFMP files as a starting point to configure the probe to pull the events from this topic so that they could be processed by Netcool. 

Reach out to us if you're using Netcool/Watson AIOps and need some help working through some obstacles.

The Fluent Bit rewrite_tag filter doesn't fully work until version 1.8.12

 I'm working with a client who has a packaged Kubernetes distribution installed that includes Fluent Bit 1.8.3. I tried the config from my last blog post on their system, and it just does NOT work as expected. In their system, it creates a new message with the new tag, but then none of the subsequent filters are applied. I had been working in the latest version (2.0.9), and everything worked like a champ. So I downloaded 1.8.3 and found that the same configuration didn't work. It seemed to partially call the rewrite_tag filter (if I set KEEP to false, it would delete the message, but if I set KEEP to true, it did nothing). The test configuration they suggest, using an input of type Dummy actually works exactly as expected. But the problem seems to be when you have an Input of type tail. And there is no workaround other than upgrading to a newer version. I actually downloaded and tested 1.8.4 through 1.8.12 before it worked correctly. So my client is now working on upgrading to a newer version.

Configuring Fluent Bit to send messages to the Netcool Mesage Bus probe

 Background

Fluent Bit is an open source and multi-platform log processor tool which aims to be a generic Swiss knife for logs processing and distribution.

It is included with several distributions of Kubernetes, and is used to pull log messages from multiple sources, modify them as needed, and send the records to one or more output destinations. It is amazingly customizable, so you can do just about any processing you want, with a couple of idiosyncracies, one of which I'll describe here.

The Challenge

What if you have a log message that you want to handle in two different ways:

1. Normalize the fields in the log message for storage in ElasticSearch (or Splunk, etc.).

2. Modify the log message so it has all of the appropriate fields needed for processing by your Netcool environment (fields that you don't necessarily want in your log storage system).

The Solution

Based on all of the unique restrictions in Fluent Bit, what you need to do is create a new copy of the log message, preserving the original so that the original can go through your "standard" processing, and the new message can be processed according to your needs in Netcool.

The specifics of this solution are to use a rewrite_tag FILTER to create a new, distinct copy of the message with a custom tag within the Fluent Bit pipeline, and then configure the appropriate additional FILTERs and OUTPUTs that only Match this new, custom tag. You also need to modify any existing OUTPUTs to exclude this new tag.

Here's a high-level graphic showing what we're going to do:

Our rewrite_tag FILTER is going to match all tags beginning with "kub". This will exclude our new tag, which will be "INC". So after the rewrite_tag filter, there will be two messages in the pipeline: the original plus our new one with our custom "INC" tag. We can the specify the appropriate Match statements in later FILTERs to only match the appropriate tag. So in the ES output above, the Match_Regex statement is:

Match_Regex  ^(?!INC).*

The official name of the above is a "lookahead exclude". Go ahead and try it out at regex101.com if you want. It will match any tag that does NOT begin with "INC", which is the custom tag for our new messages that we want to send tou our HTTP Message Bus probe.

The rewrite_tag FILTER will be custom for your environment, but the following may be close in many cases. For my case, I want to match any message that has a log field containing the string "ERROR writing to". You'll have to analyze your current messages to find the appropriate field and string that you're interested in. But here's my rewrite_tag FILTER stanza:

[FILTER]
    Name rewrite_tag
    Match_Regex ^(?!INC).*
    Rule    $log  ^.*Error\swriting\sto.* INC true

The "Rule" statement is the tricky part here. This statement consists of 4 parts, separated by whitespace:

Rule - the literal string "Rule"
$log - the name of the field you want to search to create a new message, preceded by "$". In this case, we want to search the field named log.
^.*Error\swriting\sto.* - the regular expression we want to match in the specified field. This regular expression CANNOT CONTAIN SPACES. That's why I'm using "\s".
INC - this is the name of the tag to set on the new message. This tag is ONLY used within the Fluent Bit pipeline, so it can literally be anything you want. I chose "INC" because these messages will be sent to the Message Bus proble to eventually create incidents in ServiceNow.
true - this specifies that we want the KEEP the original message. This allows it to continue to be processed as needed.

After you have the rewrite_tag FILTER in place, you will have at least one additional FILTER of type "modify" in your pipeline to allow you to add fields, rename fields, etc. You'll then have an OUTPUT stanza of type "http" to specify the location of the Message Bus probe. Something like the following:

[OUTPUT]
    Name http
    port 80
    Match INC
    host probehost
    uri /probe/webhook/fluentbit
    format json
    json_date_format epoch

The above specifies that the URL that these messages will be sent to is 

http://probehost:80/probe/webhook/fluentbit

In the json that's sent in the body of the POST request, there will be a field named date , and it will be in Unix "epoch" format, which is an integer representing the number of seconds since the beginning of the current epoch (a "normal" Unix/Linux timestamp).

That's it. That's all of the basic configuration needed on the Fluent Bit side.

Extra Credit/TLS Config

If your Message Bus probe is using TLS, you just need to add the following two lines to the above OUTPUT stanza:

    tls On

    tls.verify Off

The first line enables TLS encryption, and the second line is a shortcut that allows the connection to succeed without having to add the appropriate certificates to Fluent Bit - it will accept any certificate presented to it by the Message Bus probe, even a self-signed certificate.