Modify kibana.yml after deploying Kibana with Helm

If you deploy Kibana using the Elastic helm chart with default values, what you'll find is that you don't have any obvious way to modify the kibana.yml file. For example, if you log into the Kibana pod with

kubectl exec --stdin --tty kibana_podname -- /bin/bash

you'll find that there's no editor available (like vi or even ed). You can cat config/kibana.yml, but the comments state that it is auto-generated. So what are you supposed to do to add an a setting to the file? For example, you might need to add a value for xpack.encryptedSavedObjects.encryptionKey so you can configure alerting.

The solution I came up with is a multi-step process:

1. Get the default values.yaml file for the chart and store that in a file with the command:

helm show values elastic/kibana > /tmp/kibana.yaml

2. Edit that file to add a section for kibana.yml under kibanaConfig. Originally, kibanaConfig is empty (set to {}). You need to change it to be something like:

kibanaConfig:

   kibana.yml: |

      xpack.encryptedSaveObject.encryptionKey xxxxxxxxxxxxxxxxxxxx

3. Now (unintuitively at least to me) uninstall the helm chart with:

helm uninstall kibana

3. Then install the helm chart again with:

helm install kibana elastic/kibana -f /tmp/kibana.yaml

And that's it. Your changes will be applied and you're good to go.

I'm pretty sure there's a way to create a configMap and reference it, which would then allow you to just delete the pod to have it re-read the configMap, but I haven't figured out those exact details. Maybe in another post.

Installing .pak Files on WebSphere Application Server 8.5.x

Background

In WAS 7.0 (and possibly earlier), the WebSphere Update Installer was used to install WAS fix packs, which would have a file extension of .pak. Additionally, some other software (IBM Security Identity Manager 6, for example) that runs on WAS decided to package their updates in the same way, with .pak files to be installed with the Update Installer. WAS 8.5 moved to using IBM Installation Manager for its installation and the installation of fix packs. The last version of the WebSphere Update Installer is 7.0.0.45.

Let's say after you installed ISIM 6 on WAS 7, and then later upgraded WAS to 8.5. How do you install an ISIM 6 fixpack onto WAS 8.5?

Solution

You use the WAS 7.0.0.45 Update Installer, of course! 

WebSphere Update Installer is actually a standalone product that isn't reliant on any particular version of WebSphere to be installed. Its version number does its best to throw you off, but it works just fine when run against WAS 8.5 (or even 8.5.5.23 in my latest test).

I couldn't find this spelled out anywhere, so I thought I would share.

Installing additional software on the Rancher docker container

If you read one of my previous posts to install Rancher on a single docker container, you may have found that it doesn't include several commands like ping, netstat, ss, and even apt. And if you run 'uname -a', you might think that the image you're in is Ubuntu, but it's not. It's SUSE Linux (the same people who maintain Rancher), and the package manager there is accessed via the command 'zypper'. So to install several of the tools you know and love, run the following

zypper install net-tools iproute2 bind fping lsof

That's it. Now you have a few more tools for debugging.

Installing Rancher in a Single Docker Container on Ubuntu 20.04

This is MUCH easier than my last couple of posts because this just takes one step after you configure your OS. Rancher is a cloud native (runs on its own K8s/K3s cluster) K8s manager and container orchestration platform. It is a competitor to Red Hat OpenShift and VMWare Tanzu.

This solution is for a DEV/practice environment. 

I've uploaded the script to configure Ubuntu as a gist to Github. So all you need to do is start with a working install of Ubuntu 20.04 desktop (my test systems have been configured with 16 cores and 64GB RAM). Your user must have sudo access (you'll be prompted for the password as the scripts run) and you can run this script:

configureUbuntu.sh

	#!/bin/bash
	#
	# Full list of commands required to install minikube on Ubuntu 20.04
	#
	sudo groupadd docker
	sudo usermod -aG docker $USER
	group=docker
	if [ $(id -gn) != $group ]; then
	exec sg $group "$0 $*"
	fi
	sudo apt update
	sudo apt install -y ca-certificates curl gnupg lsb-release net-tools
	sudo apt-get update
	sudo mkdir -m 0755 -p /etc/apt/keyrings
	curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
	echo \
	"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
	$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
	sudo apt update
	sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
	sudo systemctl status docker
	# verify docker works
	docker run hello-world
	sudo apt install -y curl wget apt-transport-https
	curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
	sudo install minikube-linux-amd64 /usr/local/bin/minikube
	curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
	chmod +x kubectl
	sudo mv kubectl /usr/local/bin/
	kubectl version -o yaml
	curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
	chmod 700 get_helm.sh
	./get_helm.sh
	# Start minikube: Change memory and cpus to whatever you need
	minikube start --addons=ingress default-storageclass storage-provisioner --install-addons=true --kubernetes-version=stable --driver=docker --memory 49152 --cpus 16
	# configure kubectl bash completion
	kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null
	echo 'alias k=kubectl' >>~/.bashrc
	echo 'complete -o default -F __start_kubectl k' >>~/.bashrc
	. ~/.bashrc
	# start new shell in docker group
	exec newgrp docker

view raw configureUbuntu.sh hosted with ❤ by GitHub

Now run this command:

docker run -d --restart=unless-stopped \

  -p 80:80 -p 443:443 \

  --privileged \

  rancher/rancher:latest

Now open your browser to http://localhost and follow the directions. It will instruct you how to get the password, then prompt you to change the password, and you're good to go. You have a local Rancher K3s cluster running in a docker container. From the UI you can probe your cluster configuration, install new applications, etc. One application of interest is:

Monitoring - This is similar to (though not exactly) the kube-prometheus-stack, with Prometheus, Grafana, and several Grafana dashboards configured.

To access the cluster from the CLI, you first need to get the container-id of your rancher container with:

docker ps

Then run:

docker exec -it container-id /bin/bash

At this point you have a root shell with access to the kubectl command.

Another application that will probably interest you is Elasticsearch. Be prepared for a LOT of failure if you try to install this one. I simply could not get it to install, and I could not determine why it failed. I couldn't find any useful logs describing where it was getting hung up. If you can figure it out, please let me know. I will keep on trying.

Update 3/16/2023: I was able to get Elasticsearch installed, and I can verify via curl to port 9200 that it's running, but that's it. I can't get any logs sent to it because the Logging app won't let me configure anything. And while I can install Kibana, I cannot figure out how to access the UI once it's installed. I've tried quite a few different things, but it's not working.

To get Elasticsearch installed, you need to perform some additional steps:

Create a directory like /home/mypv inside the Rancher docker container.

Set the owner of that directory to the user "rancher"

create a PersistentVolume in the Rancher UI to be a HostPath that points to /home/mypv with a size of 30Gi (to match the defaults for the Elasticsearch install)

In the Elasticsearch yaml, change the values of these two keys as listed here:

replicas: 1

minimumMasterNodes: 1

But, like I said, you won't be able to actually do anything with it at this point.

Installing the ELK stack and Fluent-Bit on Minikube on Ubuntu 20.04

 Background

This should be easy, but it took me a couple of days to successfully get it running, so that showed me that I needed to create this post. The problems are:

1. There are a LOT of out-of-date articles out there that are now just wrong (this one was written on 3/14/2023 and will be obsolete at some point; I apologize in advance if you are reading this after that point of obsolescence). It's not the fault of the authors. Components in this space are simply changing very quickly. Event some of the latest HOWTO documentation in the different github repositories is wrong (invalid/deprecated flag used, etc.)

2. The various helm charts include some example yaml files (yay!) that don't work without modification (dammit!).

3. The Fluent Bit helm chart defaults simply do not work with a default Elasticsearch install. Specifically, Elasticsearch requires (and there is no way to disable this) TLS connections with authentication, while the Fluent Bit chart is only set up for an HTTP connection to Elasticsearch with NO authentication.

So those are some of the reasons for this article.

This solution is for a DEV/practice environment. I can't possibly list all of the reasons why. Those reasons start with "it's on minikube" and include "the Elastic password is in plaintext", among many, many others.

Solution

I've uploaded the scripts as gists to Github. So all you need to do is start with a working install of Ubuntu 20.04 desktop (my test systems have been configured with 16 cores and 64GB RAM). Your user must have sudo access (you'll be prompted for the password as the scripts run) and you can run these two scripts in order:

configureUbuntu.sh

	#!/bin/bash
	#
	# Full list of commands required to install minikube on Ubuntu 20.04
	#
	sudo groupadd docker
	sudo usermod -aG docker $USER
	group=docker
	if [ $(id -gn) != $group ]; then
	exec sg $group "$0 $*"
	fi
	sudo apt update
	sudo apt install -y ca-certificates curl gnupg lsb-release net-tools
	sudo apt-get update
	sudo mkdir -m 0755 -p /etc/apt/keyrings
	curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
	echo \
	"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
	$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
	sudo apt update
	sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
	sudo systemctl status docker
	# verify docker works
	docker run hello-world
	sudo apt install -y curl wget apt-transport-https
	curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
	sudo install minikube-linux-amd64 /usr/local/bin/minikube
	curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
	chmod +x kubectl
	sudo mv kubectl /usr/local/bin/
	kubectl version -o yaml
	curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
	chmod 700 get_helm.sh
	./get_helm.sh
	# Start minikube: Change memory and cpus to whatever you need
	minikube start --addons=ingress default-storageclass storage-provisioner --install-addons=true --kubernetes-version=stable --driver=docker --memory 49152 --cpus 16
	# configure kubectl bash completion
	kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null
	echo 'alias k=kubectl' >>~/.bashrc
	echo 'complete -o default -F __start_kubectl k' >>~/.bashrc
	. ~/.bashrc
	# start new shell in docker group
	exec newgrp docker

view raw configureUbuntu.sh hosted with ❤ by GitHub

install-elk-and-fluentbit.sh

	#!/bin/bash
	# install Elastic
	# reference URL:
	# https://www.bogotobogo.com/DevOps/Docker/Docker_Kubernetes_ElasticSearch_with_Helm_minikube.php
	# add helm repo for elastic
	helm repo add elastic https://Helm.elastic.co
	# an example values.yaml for use with minikube, but it didn't work exactly as written for me.
	# curl -O https://raw.githubusercontent.com/elastic/Helm-charts/master/elasticsearch/examples/minikube/values.yaml
	# This is how I created the elasticvalues.yaml file:
	# helm show values elastic/elasticsearch | tee -a elasticvalues.yaml
	#
	# I then edited to increase the Java opts to 512m). The important setting here seems to be "storageClassName: "standard".
	# Yep. That's the trick. I saved the YAML file as elasticvalues.yaml.
	# I also set the password to "passw0rd" to make life easier. Setting the password requires getting the "full" list of values
	# with 'helm show...'
	# enable these addons for minikube.
	minikube addons enable default-storageclass
	minikube addons enable storage-provisioner
	helm install elasticsearch elastic/elasticsearch -f https://gist.githubusercontent.com/franktate/2faaa85e7dfd953ee0115b82d2e989af/raw
	# keep checking the status of the elasticsearch pods. They take several minutes to become Ready.
	echo Sleeping 5 minutes to wait for the install to complete
	sleep 300 # wait 5 minutes
	# Once they're Ready, run the following command. This is just needed to test
	# the status of elasticsearch. It's not required for normal operations.
	kubectl port-forward svc/elasticsearch-master 9200 &
	# now install Kibana
	helm install kibana elastic/kibana
	echo Sleeping 5 minutes to wait for the install to complete
	sleep 300 # wait 5 minutes for the install to complete.
	# provide access to the Kibana UI
	kubectl port-forward deployment/kibana-kibana 5601 &
	# Kibana URL: http://localhost:5601
	# user: elastic
	# get pass with:
	# kubectl get secrets --namespace=default elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d
	# It's actually hard-coded to "passw0rd" in elasticvalues.yaml. Download the file and change it if needed.
	# install metricbeat.
	helm install metricbeat elastic/metricbeat
	# You can verify metricbeat is working by going to https://localhost:9200/_cat/indices?v&pretty
	# and you should see at least one index whose name begins with ".ds-metricbeat"
	echo Sleeping 2 minutes to wait for the install to complete
	sleep 120 # wait 2 minutes for the install to complete.
	# install logstash
	# Specifying this values.yaml file to use the OSS image:
	# https://github.com/elastic/helm-charts/blob/main/logstash/examples/oss/values.yaml
	# The default install looks for a license and other things and causes problems. This one does not.
	helm install logstash elastic/logstash -f https://raw.githubusercontent.com/elastic/helm-charts/main/logstash/examples/oss/values.yaml
	echo Sleeping 5 minutes to wait for the install to complete
	sleep 300 # sleep 5 minutes waiting for the install to really complete. May not take this long.
	# We need filebeat installed and feeding logstash. The OSS example is already configured, so use it.
	helm install filebeat elastic/filebeat -f https://raw.githubusercontent.com/elastic/helm-charts/main/filebeat/examples/oss/values.yaml
	echo Sleeping 30 seconds to wait for the install to complete
	sleep 30 # sleep 30 seconds to wait for the install to really finish
	# To verify that it worked, run:
	# curl --insecure "https://localhost:9200/_cat/_indices?v&pretty"
	# make sure there's at least one index shown whose name begins with ".ds-filebeat-oss"
	# Now that we have the OSS version of Elasticsearch installed, let's install Fluent-Bit
	# Install fluent bit
	helm repo add fluent https://fluent.github.io/helm-charts
	# the fluentbitvalues.yaml file used here was first downloaded with
	# curl https://raw.githubusercontent.com/fluent/helm-charts/main/charts/fluent-bit/values.yaml | tee -a fluentbitvalues.yaml
	# and then modified. The modifications were just to the two "es" [OUTPUT] stanzas
	helm install fluent-bit fluent/fluent-bit -f https://gist.github.com/franktate/0873e0a38234ca8ca57350b6c08a2ef8/raw
	# To verify that it worked, run:
	# curl --insecure "https://localhost:9200/_cat/_indices?v&pretty"
	# You should see a new index whose name begins with "logstash" (really. Seems odd, and is configurable, but that's the default).
	# That's it! You should be good to go.

view raw install-elk-and-fluentbit.sh hosted with ❤ by GitHub

Installing Minikube and Prometheus on Ubuntu 20.04 as of 3/11/2023

Background

You might think it's strange that I've included a specific date in the title of this post, which means that you haven't tried to perform this kind of installation at two points in time some number of months apart. See, EVERYTHING in this space is changing rapidly. The latest and greatest way to install Prometheus in Kubernetes (whether it's actual K8s or minikube or anything else) is to install kube-prometheus-stack (https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) via a helm chart. But the specific details can be changed at any time. None of the many links I found gave me a working installation without modifying the commands at least a little. So I'm hoping this post is useful to at least one person before one or more changes make it obsolete.

Solution

Here's the script that will get everything installed. You can Google any of the commands you want to see why they're in here if you're curious. But if you just need a stinkin' cluster with Prometheus installed, the exact script to do it is below. Some additional links I used to get to this point:

Installing kube-prometheus-stack:

https://github.com/LianDuanTrain/Helm3/blob/master/3%20Helm%20Deep%20Dive/3-11%20Use%20Helm%203%20to%20Install%20Prometheus%20%20on%20MiniKube%20in%2010%20Mins.md

Better instructions that still needed to be cleaned up a bit:

https://getbetterdevops.io/setup-prometheus-and-grafana-on-kubernetes/

	#!/bin/bash
	#
	# Full list of commands required to install minikube and kube-prometheus-stack (Prometheus Operator, Grafana, dashboards, etc.)
	# on Ubuntu 20.04 valid on 3/11/2023. Since kube-prometheus-stack is updated regularly and without warning, there is no guarantee that this will
	# work without modification at any future point in time.
	#
	sudo usermod -aG docker $USER
	group=docker
	if [ $(id -gn) != $group ]; then
	exec sg $group "$0 $*"
	fi
	sudo apt update
	sudo apt install -y ca-certificates curl gnupg lsb-release net-tools
	sudo apt-get update
	sudo mkdir -m 0755 -p /etc/apt/keyrings
	curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
	echo \
	"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
	$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
	sudo apt update
	sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
	sudo systemctl status docker
	docker run hello-world
	sudo apt install -y curl wget apt-transport-https
	curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
	sudo install minikube-linux-amd64 /usr/local/bin/minikube
	curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
	chmod +x kubectl
	sudo mv kubectl /usr/local/bin/
	kubectl version -o yaml
	curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
	chmod 700 get_helm.sh
	./get_helm.sh
	helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
	# Start minikube: Change memory and cpus to whatever you need
	minikube start --addons=ingress --install-addons=true --kubernetes-version=stable --driver=docker --memory 49152 --cpus 16
	# Install kube-prometheus-stack:
	helm install prometheus prometheus-community/kube-prometheus-stack --namespace=prometheus --create-namespace --wait
	# Access Uis:
	# Prometheus
	kubectl --namespace prometheus port-forward svc/prometheus-operated 9090 &
	# Then access via http://localhost:9090
	# Grafana
	kubectl port-forward --namespace prometheus svc/prometheus-grafana 8080:80 &
	# Then access via http://localhost:8080 and use the default grafana user:password of admin:prom-operator.
	# Alert Manager
	kubectl --namespace prometheus port-forward svc/prometheus-kube-prometheus-alertmanager 9093 &
	#Then access via http://localhost:9093
	kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null
	echo 'alias k=kubectl' >>~/.bashrc
	echo 'complete -o default -F __start_kubectl k' >>~/.bashrc
	. ~/.bashrc
	# start new shell in docker group
	exec newgrp docker

view raw install-prom-operator.sh hosted with ❤ by GitHub

Recent versions of the Netcool Message Bus Probe support Kafka

 We are working with a client who needed to send events from their cloud-native application to their legacy on-prem netcool Operations Insight implementation. After researching a bit, we found that their application was already writing the events of interest to a Kafka topic. The only issue was that they had an old version of the Message Bus Probe. So we installed version 21 of the probe and used the included Nokia NFMP files as a starting point to configure the probe to pull the events from this topic so that they could be processed by Netcool. 

Reach out to us if you're using Netcool/Watson AIOps and need some help working through some obstacles.