For this chapter, you will need to have a Palo Alto Networks firewall set up and connected to a management network. It will be helpful if you are able to spin up a syslog server and email relay to reproduce the log forwarding settings we are about to configure. If you can set up or repurpose a Panorama instance, you will be able to follow along with some of the threat correlation examples.

Check out the following link to see the Code in Action video:https://bit.ly/3oTeYZW

In this section, you will learn the steps required to ensure logs are forwarded to an external system. You will also learn how to apply filters so that only specific types of events are forwarded, as well as how to ensure Log forwarding configuration is applied automatically. First, we will look at where and how logs are stored.

Allocating log storage

All NGFW firewalls and Panorama Systems are built from a Linux operating system running proprietary PAN-OS on top. Log files for the system daemons reside in the root partition. They are only accessible via the command line and are included in a Tech Support file for troubleshooting. All logs related to PAN-OS live in the /opt/panlogs partition. Use the following command to review filesystem usage statistics:

reaper@PA-VM> show system disk-space 
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.0G  4.2G  2.5G  64% /
none            3.5G   92K  3.5G   1% /dev
/dev/sda5        16G  2.9G   13G  20% /opt/pancfg
/dev/sda6       8.0G  1.4G  6.3G  18% /opt/panrepo
tmpfs           2.8G  2.4G  420M  86% /dev/shm
cgroup_root     3.5G     0  3.5G   0% /cgroup
/dev/sda8        21G  598M   20G   3% /opt/panlogs

In this example, /dev/sda8 is a partition on the local disk that's used to store logs. Some of the larger hardware platforms may have a secondary hard disk for logging, and on VM firewalls, an additional disk can be installed post-deployment.

The available disk space needs to be shared by all the different log databases, so it is worth reviewing how much space is allocated to each database and tweaking the quotas and expiration periods to optimize them for retention. You can review the current quotas with the following command:

reaper@PA-VM> show system logdb-quota 
Quotas:
      system: 4.00%,  0.629 GB Expiration-period: 0 days
      config: 4.00%,  0.629 GB Expiration-period: 0 days
       alarm: 3.00%,  0.472 GB Expiration-period: 0 days
     traffic: 29.00%, 4.559 GB Expiration-period: 0 days
      threat: 15.00%, 2.358 GB Expiration-period: 0 days
...snipped for brevity...
Disk usage:
traffic: Logs and Indexes: 211M Current Retention: 46 days
threat: Logs and Indexes: 24K Current Retention: 0 days
system: Logs and Indexes: 11M Current Retention: 46 days
config: Logs and Indexes: 21M Current Retention: 46 days
...snipped for brevity...

As you can see, the traffic logs are only assigned 29% of the totally available log space on this particular firewall.

These quotas can be adjusted via the web interface by going to Device > Setup > Management > Logging and Reporting Settings, as shown in the following screenshot. The log databases on the left represent logs that are the direct result of sessions or system events taking place; the column on the right contains the summary databases that are used to compile larger datasets containing statistical data that can be used in reporting:

Figure 1.1 – Logging and reporting settings

As hardware platforms are somewhat limited in terms of their capacity for storing logs, the need to export logs for a longer log retention period may arise quite quickly. A production firewall may see up to 40 GB or more of logs being created daily, thus decreasing log retention to less than a day on smaller platforms. Virtual machines, on the other hand, support having an additional disk added to them, which we will review in the next section.

Adding disk space to a VM firewall

Virtual appliances, both firewalls and Panorama, support local storage expansion by having additional virtual disks added to enlarge their log capacity.

Important note

The primary disk that's assigned to a virtual system cannot be enlarged to accommodate more logs. The partitions are predefined and additional disk space will be left unused.

As shown in the following screenshot, an additional disk can be added that's between 60 GB and 2 TB in size to a firewall VM. Panorama VM can support from 1 to 14 2-TB disks, or one single 24-TB disk in Panorama mode. Panorama systems that are deployed in Legacy mode, which means they were installed in an older version and have since been upgraded, can have a single disk added that's up to 8 TB in size:

Figure 1.2 – Adding disks to a VM

Disks need to be thick provisioned, and the controller must be set to SCSI. Make sure that you shut down the system before adding the new disk. During bootup, the disk will be discovered and mounted as the new /opt/panlogs partition.

The next stage is to enable log forwarding to an external system.

To enable log forwarding to Panorama, the firewall must be connected to a Panorama server. This can be achieved by adding the Panorama IP via Device > Setup > Panorama Settings, as shown in the following screenshot:

Figure 1.3 – Panorama settings on the firewall

Once the firewall has established a connection with Panorama, Panorama sets its external logging destinations to what you specify in the collector group configuration.

As shown in the following screenshot, enabling Enable log redundancy across collectors will ensure each log entry has a copy on a different log collector in the same group. Enabling Forward to all collectors in the preference list will let PA-5200 and PA-7000 devices forward to all collectors in a preference list, managed by Panorama in a round-robin fashion. Otherwise, the default behavior is to send logs to the first available collector in the list:

Figure 1.4 – Collector Group general settings

In the Device Log Forwarding tab, you can select firewall devices and assign a list of collectors that they may send logs to. The first member of a collector group is the primary collector; firewalls will send their logs to this collector for as long as it is available, using the next collector down the list as a fallback collector for redundancy. In the following screenshot, we have two firewalls that have different preferences assigned for the two available collectors. The firewall called PANgurus will send logs to Panorama itself, while the RemoteLAB firewall will send logs to Collector. If one of the log destinations becomes unavailable, the firewalls will fall back to the second collector in the list:

Figure 1.5 – Device log forwarding

In the next section, we will review other useful log forwarding options.

In addition to forwarding logs to Panorama, other server profiles can be set up so that logs can be sent to a third-party log management or SIEM via Simple Network Management Protocol (SNMP). All profiles can be created in the Device > Server Profiles menu.

SNMP trap server profile

As shown in the following screenshot, there are two variations of the SNMP trap profile. SNMP protocol version V2c is very simple and requires only a server IP or FQDN, as well as a community string. This is insecure and should not be deployed in an untrusted network and should only be used if the receiving server is legacy. The V3 protocol version uses both an authentication password and a privacy password. SNMP traps and the privacy password are encrypted using AES 128; the authentication password is sent hashed with SHA1-160. Both passwords need to be between 8 and 256 characters in length. engineID is used as an identification number by the SNMP server and must consist of a hexadecimal format, prefixed by 0x and another 10 to 128 characters. If this is left empty, the firewall will use its serial number as the ID:

Figure 1.6 – SNMP trap profile

Syslog server profile

As you can see in the following screenshot, for syslog, you can select traditional UDP or TCP connections over port 514 or enable ssl encryption via port 6514. These ports can be changed if needed. The format can be changed to either BSD or RFC5424 IETF, and the facility field can be adjusted to accommodate how the receiving server manages incoming messages:

Figure 1.7 – Syslog server settings

An additional feature in the syslog profile is the ability to select the fields to include in forwarded logs for each log database. As shown in the following screenshot, this allows you to choose which fields and in which order they will appear in the forwarded log. This can come in handy if only a limited number of fields are supported, or to weed out unneeded log data. If the syslog server requires some characters to be escaped, you can list them here and define the escape character.

For example, a semicolon (;) may need to be escaped by encapsulating it with single quotes (') for compatibility reasons. Enable escaping and add the required characters, as shown here:

Figure 1.8 – Syslog custom log format

By default, the forwarded syslog message will include the source firewall's FQDN hostname. If the syslog server prefers an IP address or simple hostname, this default value can be changed in Device > Setup > Logging and Reporting Settings > Log Export and Reporting > Syslog HOSTNAME Format.

Email server profile

The Simple Mail Transfer Protocol (SMTP) profile can be configured with a friendly Email Display Name (this display name is injected in the From field in smtp DATA) so that the recipient can easily identify the firewall as the sender, the source email address (so that the relay will accept the sender), and the recipient. Only one email can be added in the To field, so an additional sender field is available to add a second email address. As shown in the following screenshot, since PAN-OS 10.0, the connection can be set to use SMTP over TLS 1.1 or 1.2 for added security. The authentication method can be set to Auto, Login, or Plain: Login with a base64 encoded username and password but send them separately or Plain with a Base64 encoded username and password but send them together. Auto will let the client and server sort out the preferred method of sending the username and password.

Logs can also be customized so that only the relevant fields are forwarded via email:

Figure 1.9 – SMTP profile

Important consideration

Emails can be a great notification method for critical events as most people will have immediate access to incoming emails, but over-abundant use of email notifications may lead to alert fatigue and important messages may go ignored or filtered. Use email notifications sparingly and only for the most critical notifications.

HTTP server profile

The HTTP profile can be used in two separate ways; first, you can add server details, choose a HTTP or HTTPS protocol, set the destination port and TLS version. Second, a certificate profile can be added, and a username and password can also be added.

This profile can be used to simply forward logs via HTTP or the payload can be edited to integrate with third-party API- or HTTP-based services. Several pre-defined payloads are available, so, as illustrated in the following screenshot, ServiceNow tickets can be created as the result of a log file being generated:

Figure 1.10 – Predefined formatting

As shown in the following screenshot, you can also enable Tag Registration. This changes the profile from regular log forwarding to a dynamic tagging role. This type of profile can be used to send dynamic tag registration or deregistration to a remote User-ID Agent (both a firewall and server installed agent) that has XML enabled. See the Log forwarding profile section later in this chapter for more details:

Figure 1.11 – HTTP server profile

For a regular HTTP Server Profile, the GET, PUT, POST, and DELETE methods are available, while for Tag Registration, only GET is supported as a HTTP method currently.

Netflow Profile

The Netflow Profile is the only profile that is assigned directly to an interface, as shown in the following screenshot. Unlike other log forwarding server profiles, no filter can be added to selectively forward certain logs; instead, all session information on an interface is directly streamed to the Netflow server:

Figure 1.12 – Adding a Netflow profile to an interface

Now that we have reviewed the available log forwarding profiles, we'll learn how to use them to forward logs.

Configuring system log forwarding on the firewall

Logs on the firewall fall roughly into two main categories: system logs and session-based logs. Each is made up of several more specific logs. Log forwarding must be configured for each log type individually. The system logs can be configured via Device > Log Settings. As shown in the following screenshot, the available logs are System, Configuration, User-ID, HIP Match, GlobalProtect, and IP Tag:

Figure 1.13 – System log forwarding configuration

Each log type can have multiple profiles associated with it, thus allowing filters and filter-specific actions to be applied. All the profiles are applied, so if a profile exists that sends all the logs of a certain type to Panorama, for example, a second profile for the same log type with a filter does not need to have Panorama checked.

In the preceding screenshot, you can see that we have already set up log forwarding for System log, Configuration logs, and User-ID logs to be sent to Panorama. Additional log forwarding Server Profiles can be added in the same profile, or in different profiles with different filters assigned. The following screenshot shows how to add an additional profile with a Filter set to critical events and one or more log forwarding profiles.

When this profile is added to the system logs, both profiles will be applied at the same time for each log. Log forwarding actions will be applied, depending on the filters that have been set in the profiles: the all system logs profile will always forward all logs to Panorama, while critical system events will match both profiles. The syslog server listed in the second profile will also receive the log:

Figure 1.14 – Adding additional server profiles to the System log forwarding profile

To determine a Filter, if any, you can click the little arrow to the right of the Filter field. Severity filters are preloaded and can simply be clicked. Alternatively, you can open the filter builder to review all the available attributes and values that can be added to the filter:

Figure 1.15 – Filter builder

Now that we've seen how to forward system logs, we will take a closer look at how to forward session-based logs.

Security rules determine how the firewall processes sessions traversing its interfaces. Not only are ports and applications determined by the security rules, but also which security profiles and even which log forwarding profile is applied. This means that you need to attach a Log Forwarding profile to every security rule so that matching sessions are logged to an external system. First, you will need to create a Log Forwarding profile.

Important Note

Creating a Log Forwarding profile named default will automatically add it to every new security rule that is created afterward.

In Objects > Log Forwarding, create a new profile and name it default. This will ensure that this profile will be added to each new security policy moving forward. Additional profiles can be added as needed. Only one Log Forwarding profile can be added per security rule.

The default profile should look somewhat like the profile depicted in the following screenshot:

Figure 1.16 – Default log forwarding profile

A log forwarding profile consists of a set of match lists. Each match list contains the log type that needs to be forwarded, the destination server profiles the logs will be sent to, and optional filters to limit which logs will be forwarded. A typical log forwarding profile will contain all the common logs, such as traffic, threat, and url filtering logs. More specific Match lists can be added and tailored to take a specific action when an event is registered. For example, the following Match List will send out an email and forward the log to a syslog server when a critical threat is detected. This can be leveraged to alert the IT security team of an event and forward the information to an Security Operations Centre (SOC):

Figure 1.17 – Email and syslog log forwarding

Built-In Actions take log forwarding one step further by dynamically taking action on log events that can help protect critical systems from attacks, as we will see in the next section.

Dynamic tagging

The Built-In Actions section of Match List can be used to act dynamically when an event is seen that matches the filter.

The Quarantine checkbox can be used to add a source host to the device's quarantine list (the list can be found via Device > Device Quarantine). These devices can then be matched against the Quarantine attribute in a security policy, as shown in the following screenshot. The default behavior is to block the session that triggered a signature without interfering with other sessions initiated by a potentially malicious client. The advantage of adding this capability to Quarantine is that hosts are placed in a controlled group that can be cordoned off from sensitive resources, and even prevented from establishing a GlobalConnect VPN connection, until an unregister action is triggered or manually removed by an administrator:

Figure 1.18 – Quarantine match on a security policy rule

As shown in the following screenshot, additional actions can be taken that will add a tag (these can be created in Objects > Tags) to a Target:

1. Destination Address
2. Source Address
3. User

   X-Forwarded-For-Address: The address contained in the x-forwarded for header added by a proxy server, indicating the original client IP:

Figure 1.19 – Dynamic tag action

Tags can be added or removed by our Action. The tag can currently be registered to three different systems:

1. Local User-ID: This is the local firewall.
2. Panorama User-ID: This is a Panorama management server that can redistribute the tags via User-ID redistribution.
3. Remote User-ID: This is the User-ID agent that's been installed on a Windows server.

For Panorama and the Remote User-ID, an XML API needs to be enabled in the configuration of the User-ID agents. A timeout can be added so that a tag is removed after a certain amount of time.

As shown in the following screenshot, once tagging has been set up, a Dynamic User Group can be created for tagged users, or an Address Group can be used for tagged addresses. These groups can then be added to security rules so that they can be granted or blocked access to resources:

Figure 1.20 – Dynamic user group and address group

Once the dynamic tags are active and IPs start getting tagged, you can follow their progress in the Monitor > IP-Tag log, as you can see in the following screenshot:

Figure 1.21 – IP-Tag logs

The IP addresses or users that were tagged can be viewed by clicking the more… link next to the dynamic profile. If needed, addresses can be unregistered from this pop up as well:

Figure 1.22 – List of tagged addresses in the dynamic profile

Before the Log Forwarding profiles can effectively start sending logs out and taking dynamic actions, they need to be assigned. We will learn about assigning log forwarding actions in the next section.

Assigning log forwarding actions

The last thing we need to do, before logs can be sent out to external systems, is add the log forwarding profile to all Security Policy rules that process sessions of interest. Log actions are determined by the Security Policy rule a session is handled by, so a log forwarding profile should be added to all security rules.

Important Note

Even rules that have no logging action enabled in the Security Policy Rule Log Settings may still generate logs: any threat event or URL filtering event with an action that is not allowed (that is, alert, block, and so on) will generate a log in the threat or URL filtering log database. The WildFire, Data Filtering, and File Blocking events will also still generate logs.

As shown in the following screenshot, the most common setting is set so that our Security Policy Rule Log Setting is Log at Session End, accompanied by a log forwarding profile.

Log at session end is the default setting that generates a log at the end of a session. This means that all the relevant data pertaining to the session can be recorded in one entry (for example, the number of transmitted bytes and packets, APP-ID, NAT actions, and so on).

Log at session start will create a log entry at every stage of a session, which includes when the SYN packet is received but also every mutation the App-ID may go through, which could be several stages for a child application. This log setting may be useful for troubleshooting but can be very chatty and storage-intensive when applied to regular traffic:

Figure 1.23 – Log forwarding profile in a security policy rule

Now that we have seen what we can do with logs, we will learn how to troubleshoot issues that could arise while forwarding logs.

Each log entry has multiple time indications that not only track when a session started and how long it lasted, but also when it was processed by the log receiver:

• Start Time is the timestamp when the session was accepted by the firewall.
• Generate Time: This is the timestamp for when the log file was generated. A log is generated based on the log action that was set in the security rule:

  Log at session start will Generate Time for each "start" event, such as the session being accepted, but also every time the App-ID changes. In this case, multiple logs will be generated for the same session, all with the same start time, but having a progressively later Generate Time.

  Log at session end will Generate Time when the session has ended.

Receive Time is when the log is received and written to a log database by the logreceiver daemon; this happens when the data plane hands the log over to the management plane to be written into the log database, or in the case of Panorama, when it is received from the firewall on Panorama.

As shown in the following screenshot, the Receive Time property of Panorama is several hours after Generate Time, which could be an indication of several possible issues. There could be a time mismatch between the firewall and Panorama, so make sure both systems are set to the proper time zone and have NTP enabled so that the system's time is kept synchronized. The firewall could have been disconnected from Panorama, so check the System logs to see if the firewall was disconnected at one point:

Figure 1.24 – Time indications in log entries

Debugging log-receiver

To find out if there is an issue on the firewall, run the debug log-receiver statistics command to review the overall health of the log-receiver daemon. If it is receiving an extremely large volume of logs, it may have difficulty writing all of them in a timely manner. It will start queueing logs and if the queue is filled to the limit, logs will be discarded. Take note of Log incoming rate versus Log written rate and keep the Logs discarded (queue full) counters in mind to determine if log generation is too high on the firewall.

For log forwarding issues, review Log Forward discarded (queue full) count and Log Forward discarded (send error) count.

If the data plane is somehow sending corrupted log entries, those will be recorded here as well. If there is an issue with the log partition, you will see the count of Logs not written since disk became unavailable increase:

reaper@PANgurus> debug log-receiver statistics 
Logging statistics
------------------------------ -----------
Log incoming rate:             1/sec
Log written rate:              1/sec
Corrupted packets:             0
Corrupted URL packets:         0
Corrupted HTTP HDR packets:    0
Corrupted HTTP HDR Insert packets: 0
Corrupted EMAIL HDR packets:   0
Logs discarded (queue full):   0
Traffic logs written:          382902
Tunnel logs written:           186
Auth logs written:             0
Userid logs written:           0
GlobalProtect logs written:    5
DECRYPTION logs written:       79
URL logs written:              232716
Wildfire logs written:         2
Anti-virus logs written:       0
Wildfire Anti-virus logs written: 0
Spyware logs written:          0
Spyware-DNS logs written:      0
Attack logs written:           0
Vulnerability logs written:    380
Data logs written:             0
URL cache age out count:       0
URL cache full count:          0
URL cache key exist count:     4049
URL cache wrt incomplete http hdrs count: 0
URL cache rcv http hdr before url count: 0
URL cache full drop count(url log not received): 0
URL cache age out drop count(url log not received): 0
Traffic alarms dropped due to sysd write failures: 0
Traffic alarms dropped due to global rate limiting: 0
Traffic alarms dropped due to each source rate limiting: 0
Traffic alarms generated count:  0
Log Forward count:             249
Log Forward discarded (queue full) count: 0
Log Forward discarded (send error) count: 0
Total logs not written due to disk unavailability: 0
Logs not written since disk became unavailable: 0

The following are some indications of the maximum supported log rates per device. New PAN-OS upgrades may come with new supported log rates, so please check the release notes to ensure there are no changes:

If the total volume of logs is too high, you can review the queue statistics with the debug log-receiver queue_stats command. This ensures that this conclusion is accurate:

reaper@PANgurus> debug log-receiver queue_stats 
Logging statistics
------------------------------ -----------
Log incoming rate:             6/sec
Log written rate:              6/sec
Logs discarded (queue full):   0
Ring buffer entries:           0/16383
Traffic taskqs:
    comp_taskq:                0/64
    disk_flush_taskq:          0/64
    summary_calc_taskq:        0/64
    summary_flush_taskq:       0/1
    bdx_taskqs[ 0]:            0/128
    bdx_taskqs[ 1]:            0/128
    bdx_taskqs[ 2]:            0/128
    bdx_taskqs[ 3]:            0/128
Threat taskqs:
    comp_taskq:                0/64
    disk_flush_taskq:          0/64
    summary_calc_taskq:        0/64
    summary_flush_taskq:       0/1
    bdx_taskqs[ 0]:            0/128
    bdx_taskqs[ 1]:            0/128
    bdx_taskqs[ 2]:            0/128
    bdx_taskqs[ 3]:            0/128

Next, we will learn about reading system resources.

Reading system resources

To verify whether the system is having I/O issues reading and writing to the disk, you can review some parameters in show system resources. To see the output live, add follow to the command and press 1 to see all CPU cores.

The output is similar to the output of top in Linux and will return the load and memory usage of the system, as well as a list of all the running processes and their resource demands.

As shown in the following screenshot, cores 1 and 2 are 100% loaded, and there are two processes taking up 100% of CPU capacity. In this case, this is due to the chassis being a PA-220, which shares a single quad-core CPU between the data plane and the management plane. pan_task processes are packet processing daemons that are pre-spun up, so it is expected that they take up about 100% of the CPU cycles:

Figure 1.25 – Show system resources

For logging issues, ensure the logrcvr process does not appear to be stuck or taking up more resources than expected. The value under the VIRT column represents memory usage in KB and should not run into the GB range.

The wa value next to the CPU readings represents the time spent waiting on disk I/O and should be as close to 0 as possible. A high value, such as over 10%, could indicate a disk problem.

Ideally, there should be a little memory left in both free values, but if you run out of volatile memory, the system will start to swap. Requiring a swap may introduce latency, but when the swap space is depleted, the system will start to degrade severely. If the swap is depleted, look down the list for the process that is taking up all this memory under the VIRT column and consider restarting it using the debug software restart process command, followed by the process name. If these issues present themselves regularly, check if your current PAN-OS requires an upgrade or consider upgrading the firewall to a larger platform that can process more logs.

Using tcpdump

If the system resources and logrcvr stats look healthy, you may be facing an external issue. The connection to the external system may be unreliable or have a lower-than-expected MTU, which could cause logs to be queued up, discarded along the way, or fragmented.

Capturing packets on both ends is usually the best way to find out if something is happening on the network connecting both systems to each other. On the management plane, you can use the tcpdump command followed by a filter. Older PAN-OS limited the default size per packet to 64 bytes, so take note of the capture size and add snaplen 0 to capture whole packets if the capture size is limited.

Unlike tcpdump in Linux systems, there is no need to add an interface as only the management interface (eth0) can be captured. Filters are added between double quotes:

reaper@PANgurus> tcpdump filter "tcp port 514"
Press Ctrl-C to stop capturing
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C 
10 packets captured
10 packets received by filter
0 packets dropped by kernel
reaper@PANgurus> 

The available filters are similar to their Linux tcpdump counterparts; that is, host 10.0.0.1, src net 10.0.0.0/24/24, and tcp port 22. You can add operators to include or exclude attributes, but not port 22.

You can then view packetcapture via CLI using the view-pcap mgmt-pcap mgmt.pcap command. You can add some parameters, such as the following:

• no-dns-lookup yes, to not resolve IP addresses to hostnames
• no-port-lookup yes, to display the actual port number instead of a name
• verbose++ yes, to add more details
• no-timestamp yes, to leave out the timestamp:

reaper@PANgurus> view-pcap no-dns-lookup yes no-port-lookup yes no-timestamp  yes mgmt-pcap mgmt.pcap 
IP 192.168.27.115.47055 > 192.168.27.7.514: Flags [P.], seq 3534228492:3534228807, ack 3260149837, win 115, length 315
IP 192.168.27.7.514 > 192.168.27.115.47055: Flags [.], ack 315, win 8211, length 0
IP 192.168.27.115.47055 > 192.168.27.7.514: Flags [P.], seq 315:632, ack 1, win 115, length 317
IP 192.168.27.7.514 > 192.168.27.115.47055: Flags [.], ack 632, win 8210, length 0
IP 192.168.27.115.47055 > 192.168.27.7.514: Flags [P.], seq 632:949, ack 1, win 115, length 317
IP 192.168.27.7.514 > 192.168.27.115.47055: Flags [.], ack 949, win 8208, length 0
IP 192.168.27.115.47055 > 192.168.27.7.514: Flags [P.], seq 949:1269, ack 1, win 115, length 320
IP 192.168.27.7.514 > 192.168.27.115.47055: Flags [.], ack 1269, win 8207, length 0
IP 192.168.27.115.47055 > 192.168.27.7.514: Flags [P.], seq 1269:1828, ack 1, win 115, length 559
IP 192.168.27.7.514 > 192.168.27.115.47055: Flags [.], ack 1828, win 8212, length 0

The pcap file can also be exported to an SCP or TFTP server so that it can be inspected using a tool such as Wireshark using the tftp export mgmt-pcap from mgmt.pcap to <host> or scp export mgmt-pcap from mgmt.pcap to user@host/path/ commands:

reaper@PANgurus> tftp export mgmt-pcap from mgmt.pcap to 192.168.27.7 
mode set to octet
Connected to 192.168.27.7 (192.168.27.7), port 69
putting /opt/pan/.debug/mgmtpcap/mgmt.pcap to 192.168.27.7:mgmt.pcap [octet]
Sent 2582 bytes in 0.0 seconds [805931 bit/s]

In the next section, we will take a closer look at troubleshooting log forwarding issues to a Panorama log collector.

Troubleshooting forwarding to a log collector

To verify that the firewall is connected to Panorama, you can run the following command to check the status of the log collector agent:

debug management-server log-collector-agent-status

The output will clearly state which log collectors are known and if the firewall is currently connected to them:

reaper@PANgurus> debug management-server log-collector-agent-status 
Logcollector agent status
-------------------------
Serial  IP Address  Connected  Last Disconn Time  Failed conns
-------------------------
0007001 192.168.27.10     no                                 0

Important Note

The firewall will always initiate the connection toward Panorama and additional log collectors. Any commands, updates, or configuration originating from Panorama or a log collector will be backhauled over the connection established by the firewall.

You can then check additional information by running request log-collector-forwarding status. As shown in the following screenshot, it will show timestamps for certain events, such as when it last registered and connected to the log collector, but also the timestamp when the last log was forwarded. A large gap between the current time and the last time a log (most commonly, traffic and a threat) was forwarded, or a very short time between when the last connection was established and the current time, could be an indication of connectivity issues. Take note of Last Seq Num Fwded and Last Seq Num Acked as you will need to compare these log sequence numbers on the log collector:

Figure 1.26 – Log collector forwarding status

On Panorama, you can verify if the log collector is connected and whether it received the latest configuration, in case something changed, using the show log-collector connected command:

reaper@Panorama> show log-collector connected
Serial  CID Hostname  Connected Config Status SW Version         IPv4 - IPv6                                                     
--------------------------------------------------------------
0007001 2   Panorama  yes       In Sync       10.0.3             192.168.27.10 - unknown
Redistribution status:       none
Last commit-all: commit succeeded, current ring version 1
SearchEngine status:     Active
md5sum  updated at ?
  Certificate Status: 
  Certificate subject Name: 
  Certificate expiry at: none
  Connected at: none
  Custom certificate Used: no
  Last masterkey push status: Unknown
  Last masterkey push timestamp:  none

Once you've verified that the collector is connected and in sync, you can check the statuses of the firewalls sending logs to Panorama using show logging-status device <serial of firewall>. You will see the timestamp of the last logs that were received and which sequence number was received:

reaper@Panorama> show logging-status device 012000001
    Type            Last Log Rcvd    Last Seq Num Rcvd       Last Log Generated
Source IP         : Default
Destination IP    : Default
Source Daemon     : unknown
Connection Id      : 012000001-log-collection
Log rate: 1
    config      2020/12/21 00:37:55                3241      2020/12/21 00:17:41
    system      2020/12/21 22:34:55            24693983      2020/12/21 22:34:49
    threat      2020/12/21 22:35:15             7336943      2020/12/21 22:35:10
   traffic      2020/12/21 22:35:15            83646677      2020/12/21 22:35:10
    userid                      N/A                 N/A                       N/A
     iptag      2020/12/21 22:35:15               14127      2020/12/21 22:35:07

You can check the current flow rate of incoming logs by running the debug log-collector log-collection-stats show incoming-logs command.

At the bottom, it will also indicate if there are logs being discarded from devices, which can be an indication that the log rate is too high:

reaper@Panorama> debug log-collector log-collection-stats show incoming-logs
Last time logs received Mon Dec 21 23:00:20 2020
Incoming log rate =   1.10
Detail counts by logtype:
traffic:60936
config:16
system:2650
threat:32723
...
Inbound logger stats:
traffic generation logcount:60936 blkcount:926 addition-rate:0(per sec)
config generation logcount:16 blkcount:2 addition-rate:0(per sec)
system generation logcount:2650 blkcount:638 addition-rate:0(per sec)
threat generation logcount:32723 blkcount:922 addition-rate:0(per sec)
... 
Num of discarded log messages from devices: 0

If there has been a connectivity outage between the firewall and Panorama, the firewall will try to send all its logs starting from the last acked (acknowledged by Panorama that it was received) sequence number. If the disconnection was lengthy, the firewall may need to send a vast volume of logs across, which could lead to more congestion or a long wait time for old logs to catch up and fresh logs to become visible on Panorama. There are a few commands available to control how the firewall will forward its backlog, all of which you can initiate from Panorama.

This command will tell the firewall to stop sending logs:

request log-fwd-ctrl device <FW serial> action stop
scheduled a job with jobid 0. 
0

The following command will tell the firewall to start forwarding logs again, from the last acked log, and buffer any future logs if a disconnect were to happen again. This is the default setting for log forwarding:

request log-fwd-ctrl device <FW serial> action start
scheduled a job with jobid 0. Converted log-fwd-ctrl action to 'start-from-lastack'
0

The following command will instruct the firewall to start sending logs, starting from the latest sequence number, working its way back. This can help you get newer logs faster while allowing yourself some time to catch up on old logs:

request log-fwd-ctrl device <FW serial> action latest
scheduled a job with jobid 0. 
0

The following command will tell the firewall to start forwarding live logs only. It will not attempt to forward the buffered logs. If a disconnect happens again, the firewall will not buffer logs during this outage and will simply continue from the latest log once connectivity is re-established:

request log-fwd-ctrl device <FW serial> action live
scheduled a job with jobid 0. 
0

This will tell the firewall to start regular log forwarding again, starting from the last acked log file. This command can be executed after the action live command to pick up logging, but with buffering enabled:

request log-fwd-ctrl device <FW serial> action start-from-lastack
scheduled a job with jobid 0. 
0

Let's look at a few examples of how these commands could be leveraged during troubleshooting.

If there is an issue with catching up on the buffered logs, due to bandwidth constraints or because the load on Panorama is too high due to a high log ingestion rate, it may be necessary to abandon the buffered logs. The sequence to achieve this is to stop logging and then start logging live. Then after a few minutes, when you've verified that fresh logs are coming in, execute start-from-lastack so that the buffer is active in case there is another short disconnect.

If the log rate is too high at a peak moment, logging could be stopped for a less critical firewall to reduce the load on the log collector and restarted once the peak has faded away. Buffering will ensure no logs are lost.

In some cases, the log rate on the firewall may be so high, or the outage may be so long, that the firewall was unable to keep the oldest logs and the last ack log was purged. In this case, start and start-from-lastack could have trouble restarting as the sequence numbers no longer match up. Here, try latest or run live for a few minutes, followed by start-from-lastack.

Check if your commands are causing the expected log action by executing show logging-status device <FW serial>.

In this chapter, you learned about all the elements that come into play before logs are sent to a third-party logging device or email relay. You learned some troubleshooting steps and how to review whether Panorama and the firewall are in a healthy state. This knowledge will help you set up proper forwarding profiles and leverage these capabilities to aggregate logs on a SIEM solution, or even notify security teams when certain events occur.

In the next chapter, we will take a deeper look at setting up and troubleshooting GlobalProtect.