Troubleshooting

Where to start diagnosing

Given the large number of technologies involved in CAPE installation, configuration and usage, chances are high one or more of them start failing, crashing or simply dying. When this happens, there are a few places you should always look first, since it could help you diagnosing the real problem and avoid wasting time looking at rabbit holes. These places are (no surprise here) the logs generated by each fundamental service involved in CAPE’s execution. While all of them can be checked in their corresponding directories, journalctl can be leveraged to check all of them at once.

Note

Please note the following errors are just random examples used to show how and where to start dealing with CAPE problems. You may never face these errors at all!

Regardless of the error you are facing there are two places where you should start looking: (1) CAPE’s logs (naming inherited from cuckoo) and (2) virtqemu logs (assuming you have installed KVM/QEMU using the kvm_qemu.sh script.)

For example, lets consider the following situation:

CAPE submission (either via web or <submit.py>) apparently works, but the VM is never spawned (and analysis never launched).

Without prior indication, the first place to check are CAPE logs, located in /opt/CAPEv2/log/ (installation directory). The logs corresponding to the analyses are rotated daily, and the current log is cuckoo.log. Checking the contents of /opt/CAPEv2/log/cuckoo.log one could easily spot the culprit:

Error

Error example in <cuckoo.log> file.

../../_images/troubleshooting_error_0.png

Furthermore, the error states something about libvirt. This is a clear indication that the corresponding logs must also be inspected. In this case, the logs of libvirt are stored under the virtqemud service. Whenever something seems wrong regarding the virtual machines, this is the place to look after. It can be inspected with $ sudo journalctl -u virtqemud -r:

Error

Error examples by inspecting vertqemud logs with journalctl.

../../_images/troubleshooting_error_1.png ../../_images/troubleshooting_error_2.png

Additionally, you should always try to see if you are able to manually replicate the error in order to discard technologies and find out which one is failing. Considering the scenario above, when trying to manually spawn the virtual machines:

Error

Error example in Virtual Manager - KVM.

../../_images/troubleshooting_error_3.png

No Internet connection in the guest

There are reasons galore why your guest VM has no Internet connection when an analysis is fired up. Before digging into this problem, please make sure you followed the steps at Network Configuration to set up both the virtual machine and its connections. Furthermore, you should read the Per-Analysis Network Routing chapter in order to know and understand the different routing modes as well as the CAPE Rooter chapter to understand what the Rooter is.

Some considerations:

  1. dirtyline should be the interface that provides your host internet connection like eno1, not a virtual interface like virbr1. This must be configured in the routing.conf configuration file.

  2. Check agent.py is running with elevated privileges within the guest VM.

  3. Make sure you specify the correct STATIC IP in kvm.conf.

  4. Make sure you specified the correct interface in auxiliary.conf.

Also, bear in mind there are already created (and some of them solved) issues about this particular problem. For example:

PCAP Generation

If you are facing problems related to either tcpdump or the PCAP generation, take a look at this issue (#1234).

Note

Make sure the pcap group exists in your system and that the user you use to launch CAPE (presumably the cape user) belongs to it as well as the tcpdump binary.

Make sure the correct path is specified in auxiliary.conf for tcpdump. Check the path of your local installation of tcpdump with:

$ whereis tcpdump

Check permissions of tcpdump binary. cape user must be able to run it. Also check whether you specified the correct interface in auxiliary.conf.

If you are still facing problems and the PCAP is not generating, verify the tcpdump binary belongs to the pcap group and it has the neede capabilities:

$ sudo chgrp pcap /usr/bin/tcpdump
$ sudo setcap cap_net_raw,cap_net_admin=eip /usr/bin/tcpdump

Other issues about this problem: