On some distributions, additional support tools are available that help us identify the cause of a denial. These tools have some knowledge of common mistakes (for instance, setting the right context on files to allow the web server to read them). Other distributions require us to use our experience to make proper decisions, supporting us through the distribution mailing lists, bug tracking sites, and other cooperation locations—for example, Internet Relay Chat(IRC).
Troubleshooting with setroubleshoot
In CentOS (andotherRed Hat Enterprise Linux(RHEL)-relateddistributions such as Fedora), additional tools are present that help us troubleshoot denials. The tools work together to catch a denial, look for a plausible solution, and inform the administrator about the denial and its suggested resolutions.
When used on a graphical workstation, denials can even result in popups that ask the administrator to review them immediately. Install thesetroubleshootpackage to get this support. On servers without a graphical environment, administrators can see the information in the system logs or can even configure the system to send out SELinux denial messages via email. Install thesetroubleshoot-serverpackage to get this support.
Under the hood, it is the audit daemon that triggers its audit event dispatcher application (audispd). This application supports plugins, something the SELinux folks gratefully implemented. They built an application calledsedispatchthat will act as a plugin foraudispd. Thesedispatchapplication checks whether the audit event is an SELinux denial and, if so, forwards the events to D-Bus. D-Bus then forwards the events to thesetroubleshootdapplication (or launches the application if it isn’t running yet), which analyzes the denial and prepares feedback for the administrator.
Whenrunning on a workstation,seappletis triggered to show a popup on the administrator workstation. The administrator can then selectShowto view more details. Administrators don’t need a graphical user interface to be informed about SELinux issues, though. You can find analyzed feedback on the filesystem, and in the system logs you can read how to easily reach this information, as illustrated in the following code snippet:
Mar 22 11:40:35 ppubssa3ed setroubleshoot[1544]: SELinux is preventing /usr/sbin/nginx from name_bind access on the tcp_socket port 89. For complete SELinux messages run: sealert -l f2914dba-04ef-44ca-9a0b-0f5e62ec72e4We can look at a complete explanation through the sealert command (as mentioned in the log), as follows:
# sealert -l f2914dba-04ef-44ca-9a0b-0f5e62ec72e4
SELinux is preventing /usr/sbin/nginx from name_bind access on the tcp_socket port 89.
***** Plugin bind_ports (99.5 confidence) suggests ************************
If you want to allow /usr/sbin/nginx to bind to network port 89
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 89
where PORT_TYPE is one of the following: http_cache_port_t, http_port_t, jboss_management_port_t, jboss_messaging_port_t, ntop_port_t, puppet_port_t.
***** Plugin catchall (1.49 confidence) suggests **************************
...Thesealertapplication is a command-line application that parses the information stored by thesetroubleshootdaemon (in/var/lib/setroubleshoot).
This will provide us with a set of options to resolve the denial. In the case of the Apache-related denial shown earlier,sealertgives us one option with a certain confidence score. Depending on the problem, this tool might show multiple options, each with its own confidence figure (that is, how certainsealertis that this is the right resolution).
As we can see from this example, thesetroubleshootapplication itself uses plugins to analyze denials. These plugins (offered through thesetroubleshoot-pluginspackage) look at adenial to check whether they match a particular, well-known use case (for example, when to change an SELinux Boolean or when a target resource has a wrong context) and give feedback tosetroubleshootabout how certain the plugin is so that this denial can be resolved through its recommended method.
Sending emails when SELinux denials occur
Once a systemis fine-tuned and denials no longer occur regularly, administrators can opt to havesetroubleshootdsend emails whenever a new denial comes up. This truly brings SELinux’s host intrusion detection/prevention capabilities on top, as administrators do not need to constantly watch their logs for information. However, keep in mind that this could lead to a sudden burst in emails, which mightresult inDenial of Service(DoS)-like behavior, if many denials are triggered. Administrators should only implement this if their email infrastructurehas rate limiting or otherQuality of Service(QoS) controls in place.
Open/etc/setroubleshoot/setroubleshoot.confin a text editor and locate the[email]section. Update the parameters to match the local mailing infrastructure, as follows:
setroubleshoot.conf
# vi /etc/setroubleshoot/setroubleshoot.conf
[email]
recipients_filepath = /var/lib/setroubleshoot/email_alert_recipients
smtp_port = 25
smtp_host = localhost
from_address = [email protected]
subject = [infra] SELinux Alert for host infra.example.comNext, edit theemail_alert_recipientsfile (as referenced through the recipients_filepathvariable), and add the email addresses that need to be notified when an SELinux alert comes up.
Finally, restart the D-Bus daemon, as follows:
# systemctl restart dbusWhen working on a non-systemd system, use the following command instead:
# service dbus restartThe D-Bus restart is needed as D-Bus manages the setroubleshootd daemon.
Using audit2why
Ifsetroubleshootandsealertare not available in the Linux distribution, we can still get someinformation about a denial. Although it isn’t as extensive as the plugins offered bysetroubleshoot, theaudit2whyutility (which is short foraudit2allow -w) does provide some feedback on a denial. Sadly, it isn’t always right in its deduction.
Let’s try it out against the same denial for which we usedsealert, as follows:
# ausearch -m avc -ts recent | audit2why
type=AVC msg=audit(1584880436.644:385): avc: denied { name_bind } for pid=5119 comm="nginx" src=89 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:reserved_port_t:s0 tclass=tcp_socket permissive=0
Was caused by:
Missing type enforcement (TE) allow rule.
You can use audit2allow to generate a loadable module to allow this access.Theaudit2whyutility here didn’t consider that the context of the target location was wrong, and it suggests that the policy be updated to allow the web server to bind to theunreserved_port_ttype, unlike the information provided bysetroubleshoot, which was more accurate, recommending that the target port be relabeled instead.
As the output of the command mentions, another tool exists calledaudit2allow, which can convert a denial into an SELinux policy.
Interacting with systemd-journal
Alongsidethe Linux audit system, which is used for most SELinux logging and events, we can also gather information through other logging systems. systemd’s journal, for instance, captures SELinux context information with the events and allows administrators to use this information while querying the journal.
For instance, to see the events insystemd-journalthat are generated by an application associated with thesystem_u:system_r:sssd_t:s0context, the following command can be used:
# journalctl _SELINUX_CONTEXT="system_u:system_r:sssd_t:s0"
-- Logs begin at Sun 2020-03-22 10:43:48 UTC, end at Sun 2020-03-22 12:40:12 UTC. --
Mar 22 10:43:51 ppubssa3ed sssd[545]: Starting up
Mar 22 10:43:51 ppubssa3ed sssd[be[implicit_files]][623]: Starting up
Mar 22 10:43:51 ppubssa3ed sssd[nss][630]: Starting upBecausesystemd-journaladds the SELinux context of the originating application, it is harder for malicious applications to generate fake events. Whereas regular system loggers just capturestring events,systemd-journalretrieves the SELinux context from the system. Using the SELinux context, it is easy to group events across different but strongly related applications and have a higher guarantee that events come from a particular application.
When thebash-completionpackage is installed, we can even use it to see which SELinux contexts are present in thesystemd-journallogs, which makes querying the journal logs much easier, as follows:
# journalctl _SELINUX_CONTEXT=<tab><tab>
kernel
system_u:system_r:auditd_t:s0
system_u:system_r:chronyd_t:s0
...To find messages related to nginx, use the embedded grep filter, as follows:
# journalctl -g nginx
-- Logs begin at Sun 2020-03-22 10:43:48 UTC, end at Sun 2020-03-22 12:52:26 UTC. --
Mar 22 11:40:32 ppubssa3ed systemd[1]: Starting The nginx HTTP and reverse proxy server...
Mar 22 11:40:32 ppubssa3ed nginx[1538]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Mar 22 11:40:32 ppubssa3ed nginx[1538]: nginx: [emerg] bind() to 0.0.0.0:89 failed (13: Permission denied)
...
Mar 22 11:40:35 ppubssa3ed setroubleshoot[1544]: SELinux is preventing /usr/sbin/nginx from name_bind access on the tcp_socket port 89. For complete SELinux messages run: sealert -l f2914dba-04ef-44ca-9a0b-0f5e62ec72e4The benefit of the embedded grep filter is that journalctl will still show the multiline messages, whereas actually redirecting the journalctl output through grep would only show the individual lines that match the expression.
Using common sense
Common sense is not easy to document, but reading a denial often leads to the right solution whenwe have some experience with file labels (and what they are used for). If we get a denial about a web server failing to read its files, and the context of the file is (for instance)user_home_t, then that should ring a bell. End user home files, for instance, use theuser_home_tcontext, which is not suitable for system files that the web server reads.
One way to make sure that the context of the target resource is correct is to verify it withmatchpathcon. This utility returns the context as it should be according to the SELinux policy, as follows:
$ matchpathcon /srv/www/html/index.html
/srv/www/html/index.html system_u:object_r:httpd_sys_content_t:s0Performing this for denials related to files and directories might help in finding a proper solution quickly.
Furthermore, many domains have specific manual pages that inform the reader about types commonly used for each domain, as well as how to deal with the domain in more detail (for example, the available booleans, common mistakes made, and so on). These manual pages start with the main service and are suffixed with_selinux, as illustrated here:
$ man ftpd_selinuxIn most cases, the approach to handling denials can be best described as follows:
- Is the target resource label (such as the file label) the right one? Verify this with
matchpathcon, or compare with labels of working (accessible) resources. - Is the source label (the domain) the expected one? An SSH daemon should run in the
sshd_tdomain, not theinit_tdomain. If this is not the case, make sure that the labels of the application itself (such as its executable binary) are correct (again, usematchpathconfor this). - Is the denial one that might be covered by an SELinux boolean? In that case, the policy might already have the appropriate rules in place, only requiring a change in an SELinux boolean value.
setroubleshootdwill report this if it is the case. Usually, the manual page of the domain (such ashttpd_selinux) will also cover the available SELinux Booleans.
To close off this article, common sense will be your most prolific approach to managing SELinux denials, but the aforementioned tools will be of assistance to begin with.
