Troubleshooting Red Hat CodeReady Containers

Background

I've been working with Red Hat CodeReady Containers (CRC) recently, and found that I've had to look all over the place to find even the most basic troubleshooting information. For example, here is the link to the Troubleshooting chapter of the official documentation. Go read it. If you don't think you have time, you're wrong. It will take you about 30 seconds. I'm writing this post to provide a little information that I've found to be useful. It's certainly not everything you need, but it's enough to get you pretty far, and it is infinitely more information than in the link above.

Environment

Here is a diagram that shows my configuration for CRC:

Other than the memory and CPU specs, this is a pretty common configuration for CRC. 

When troubleshooting CRC, your Physical Host Machine and your virtualization software (VMWare Workstation 16 Pro in my case) don't really come into the picture too much. They generally do their job and are transparent to what you're doing, so I'm not touching on those. The systems you're actually going to look ar are your Guest VM, crc VM, and crc pods.

Guest VM

What I'm calling the Guest VM is the system on which you've downloaded and plan to run CRC. So in your case, this could actually be your laptop. But Guest VM is what I'm calling it. This is where most of your troubleshooting will be done if you're having problems getting the crc VM to start, which is what I have encountered most often. I am using libvirt, KVM, and qemu, which is the default/normal configuration on Linux. Information on how these three components work together can be found at this link.

crc VM log file

The most important file to know about is the crc VM log file created by qemu. That file is:

/var/log/libvirt/qemu/crc.log

This is the console output of the crc VM, so it will show you exactly what's happening in the VM as it is booting up.

virsh command

The virsh command is included in the libvirt package. This command allows you to interact with libvirt/KVM/qemu VMs on your system, like the crc VM. The important thing to note is that you need to run this command as root. As root, you can run the following command to see a list of all VMs running:

virsh list --all

virsh is a complete terminal environment, with tons of additional commands that can be run interactively. 

virt-manager

I actually found this before I found the above log file. Now that I know the location of the log file, I've found that this tool isn't as useful, but I wanted to include it anyway.

Install virt-manager on your Guest VM with yum. You can then launch it with the command virt-manager, which will bring up the application window:

You can then click on the crc VM to see the console. There is no way to actually log into the crc VM because you can only log in via the core user's private key (shown later). Googling around, I see that password access has been requested/suggested, but there appears to be no plan to implement it at this time.

crc VM

crc pods log files

If the crc VM is up and running, you can ssh into it with this command:

ssh -i ~/.crc/machines/crc/id_ecdsa core@api.crc.testing

Once inside the crc VM, you can cd to /var/log/pods and you will see one log file for each pod created.

That's all for now

As I said in the beginning, I just included a few tools, but this is more than in the product documentation. CRC and OpenShift are really complex frameworks that rely on tons of components like libvirt, Kubernetes, and tons of other complex components. It is understandable why it's so hard to troubleshoot. However, I personally believe that the development team could include more logging information in the web console itself so that users/operators of the system have access to the data without having to separately open a terminal window.