Understanding log forwarding profiles and best practices

In this section, you will learn the steps required to ensure logs are forwarded to an external system. You will also learn how to apply filters so that only specific types of events are forwarded, as well as how to ensure Log forwarding configuration is applied automatically. First, we will look at where and how logs are stored.

Allocating log storage

All NGFW firewalls and Panorama Systems are built from a Linux operating system running proprietary PAN-OS on top. Log files for the system daemons reside in the root partition. They are only accessible via the command line and are included in a Tech Support file for troubleshooting. All logs related to PAN-OS live in the /opt/panlogs partition. Use the following command to review filesystem usage statistics:

reaper@PA-VM> show system disk-space 
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       7.0G  4.2G  2.5G  64% /
none            3.5G   92K  3.5G   1% /dev
/dev/sda5        16G  2.9G   13G  20% /opt/pancfg
/dev/sda6       8.0G  1.4G  6.3G  18% /opt/panrepo
tmpfs           2.8G  2.4G  420M  86% /dev/shm
cgroup_root     3.5G     0  3.5G   0% /cgroup
/dev/sda8        21G  598M   20G   3% /opt/panlogs

In this example, /dev/sda8 is a partition on the local disk that's used to store logs. Some of the larger hardware platforms may have a secondary hard disk for logging, and on VM firewalls, an additional disk can be installed post-deployment.

The available disk space needs to be shared by all the different log databases, so it is worth reviewing how much space is allocated to each database and tweaking the quotas and expiration periods to optimize them for retention. You can review the current quotas with the following command:

reaper@PA-VM> show system logdb-quota 
Quotas:
      system: 4.00%,  0.629 GB Expiration-period: 0 days
      config: 4.00%,  0.629 GB Expiration-period: 0 days
       alarm: 3.00%,  0.472 GB Expiration-period: 0 days
     traffic: 29.00%, 4.559 GB Expiration-period: 0 days
      threat: 15.00%, 2.358 GB Expiration-period: 0 days
...snipped for brevity...
Disk usage:
traffic: Logs and Indexes: 211M Current Retention: 46 days
threat: Logs and Indexes: 24K Current Retention: 0 days
system: Logs and Indexes: 11M Current Retention: 46 days
config: Logs and Indexes: 21M Current Retention: 46 days
...snipped for brevity...

As you can see, the traffic logs are only assigned 29% of the totally available log space on this particular firewall.

These quotas can be adjusted via the web interface by going to Device > Setup > Management > Logging and Reporting Settings, as shown in the following screenshot. The log databases on the left represent logs that are the direct result of sessions or system events taking place; the column on the right contains the summary databases that are used to compile larger datasets containing statistical data that can be used in reporting:

Figure 1.1 – Logging and reporting settings

As hardware platforms are somewhat limited in terms of their capacity for storing logs, the need to export logs for a longer log retention period may arise quite quickly. A production firewall may see up to 40 GB or more of logs being created daily, thus decreasing log retention to less than a day on smaller platforms. Virtual machines, on the other hand, support having an additional disk added to them, which we will review in the next section.

Adding disk space to a VM firewall

Virtual appliances, both firewalls and Panorama, support local storage expansion by having additional virtual disks added to enlarge their log capacity.

Important note

The primary disk that's assigned to a virtual system cannot be enlarged to accommodate more logs. The partitions are predefined and additional disk space will be left unused.

As shown in the following screenshot, an additional disk can be added that's between 60 GB and 2 TB in size to a firewall VM. Panorama VM can support from 1 to 14 2-TB disks, or one single 24-TB disk in Panorama mode. Panorama systems that are deployed in Legacy mode, which means they were installed in an older version and have since been upgraded, can have a single disk added that's up to 8 TB in size:

Figure 1.2 – Adding disks to a VM

Disks need to be thick provisioned, and the controller must be set to SCSI. Make sure that you shut down the system before adding the new disk. During bootup, the disk will be discovered and mounted as the new /opt/panlogs partition.

The next stage is to enable log forwarding to an external system.