TheInfiniBandstandardis a relatively recent (in network history) technology that enables very high throughput and very low latency. It accomplishes this by having a verylow overhead on the network layer (protocol) and direct access from user applications to the network level. This direct access also has implications for SELinux, as the Linux kernel is no longer actively involved in the transport of data across an InfiniBand link.
Let’s first look at what InfiniBand looks like, after which we can see how to still apply SELinux controls to its communication flows.
Directly accessing memory
One of themain premises of InfiniBand is to allow user applications to have direct access to the network. By itself, InfiniBand is a popularRemote Direct Memory Access(RDMA) implementation, which has received significantsupport from vendors. We find RDMA actively used in high-performance clusters.
Because of the direct access, controls are only possible while setting up the access approach. Without SELinux, all that is needed to set up and manage InfiniBand communications is to have access to the device file itself. If a process can write to the InfiniBand device, then it can use InfiniBand. By default, these devices are only accessible by therootuser.
The InfiniBanddevices are the network cards orHost Channel Adapters(HCA) and can havemultiple ports. An InfiniBandportis the link or interface that connects to an InfiniBand subnet. The subnet is the high-speed network on which multiple machines (ports) are connected. As with regular networks, InfiniBand switches are used to facilitate communication across a subnet, and routers can be used to connect different subnets with each other.
An InfiniBand subnetis managed by aSubnet Manager(SM). This is a process that coordinates the management of the different ports within the subnet, as well as the partitions.Partitionsin InfiniBand are a way to differentiate between different communications within a subnet, likeVirtual Local Area Networks(VLANs) in moreregular networks. With partitionedcommunication, it is the subnet manager that tells which ports can be used for which partitions of the communication.
Protecting InfiniBand networks
Unlike regularnetworks, where firewalls and switch-level access controls are the norm for preventing unauthorized access, InfiniBand has few protection measures in place. InfiniBand largely assumes that the network is within a trusted environment. However, that does not exclude us from applying more rigid controls over which process can access the InfiniBand network in SELinux.
As the communication flow itself is directly mapped in-memory toward the devices, the Linux kernel does not have any hooks available to do packet-level controls like it can with regular TCP/UDP traffic (using the SECMARK capabilities), or even session-level controls with sockets. Instead, SELinux focuses on two main controls, as visualized in the following diagram:
These two main controls are as follows:
- Controlling who can manage the InfiniBand subnet
- Controlling who can access an InfiniBand partition
To properly govern these controls, the semanage application assigns the right type to the appropriate InfiniBand resource. However, not all SELinux policies already contain the appropriate types, so we need to add those in as well.
Managing the InfiniBand subnet
Let’s startwith managing the InfiniBand network. With InfiniBand on Linux, this is most often accomplished using theopensmapplication. Many InfiniBand adapters have multiple ports, allowing a server to participate in multiple InfiniBand subnets. With SELinux, we can control which domain can manage a subnet by controlling access to the InfiniBand port on a device.
First, we need to assign a label to the InfiniBand port associated with a subnet. To accomplish that, we first need to obtain the right InfiniBand device, create the appropriate label (type), and then assign it to the port.
Let’s start by querying the available InfiniBand-capable devices on the system usingibv_devinfo:
# ibv_devinfo
hca_id: rxe0
transport: InfiniBand (0)
fw_ver: 0.0.0
...
phys_port_cnt:
port: 1
state: PORT_ACTIVE (4)
...Next, we create a type (label) to assign to the port. This type is only used to validate the access from the opensm application to this port. We use the CIL language for this, Developing Policies with SELinux CIL). Create a file with the following content (let’s call it infiniband_subnet.cil):
(typeattribute ibendport_type)
(type local_ibendport_t)
(typeattributeset ibendport_type local_ibendport_t)
(allow opensm_t local_ibendport_t (infiniband_endport (manage_subnet)))In the previous code, we enhance the SELinux policy with a new type calledlocal_ibendport_t, assign it theibendport_typeattribute, and then grant theopensm_tdomain themanage_subnetprivilege within theinfiniband_endportclass.
Let’s load this policy enhancement:
# semodule -i infiniband_subnet.cilFinally, we assign this newly created type to the InfiniBand port:
# semanage ibendport -a -t local_ibendport_t -z rxe0 1This command assigns the local_ibendport_t type to port number 1 of the rxe0 device (as obtained from ibv_devinfo). Once this mapping is in place, we can query it using semanage as well:
# semanage ibendport -l
SELinux IB End Port Type IB Device Name Port Number
local_ibendport_t rxe0 0x1Without any mappings, the command does not display any output.
Note:
Currently, most Linux distributions have not incorporated InfiniBand support within the SELinux policy, requiring us to create our own custom labels. We can expect that distributions will add in default types for InfiniBand resources, and that SELinux support for InfiniBand will be extended with sane defaults.
If we use InfiniBand on an SELinux-enabled system without any port mappings, the initial security context for unlabeled classes will be used as the label for this port, namely, unlabeled_t. It is, however, not recommended to stick to this label, as it is more widely used for unlabeled resources. Granting any privilege to the unlabeled_t type should be limited to highly privileged processes, and its use should be carefully considered to ensure that logging interpretation and SELinux policy rules vis-à-vis InfiniBand resources are clear (through well-documented types).
Controlling access to InfiniBand partitions
While theprevious section focused on allowing the management applicationopensmto manage a subnet, this section will focus on restricting access to the InfiniBand network to the right domains. As mentioned before, an InfiniBand subnet can be divided further into separate networks using InfiniBand partitions.
Originally, thesepartitions are used to allowQuality of Service(QoS) or specific bandwidth and performance requirements on flows. The SM defines thepartitions and its attributes, and applications use aPartition Key(P_Key) to inform the InfiniBand network as regards to which partition certain communications must be done.
SELinux can govern these partitions by creating a mapping between the InfiniBand subnet plus P_Key and an SELinux type. However, as with the subnet management, we need to find the appropriate details first and create an appropriate SELinux type before we can define the mapping.
Let’s start by figuring out the subnet and partition details. Both are managed byopensm. If you do not have access to theopensmconfiguration, then you need to ascertain these details from the (InfiniBand) network administrator.
Within theopensmpartition configuration (/etc/rdma/partitions.conf), the subnet and prefix can be found as follows:
# grep '=0x' /etc/rdma/partitions.conf
Default=0x7fff, rate=3, mtu=4, scope=2, defmember=full;
Default=0x7fff, ipoib, rate=3, mtu=4, scope=2;
rxe0_1=0x0610, rate=7, mtu=4, scope=2, defmember=full;
rxe0_1=0x0610, ipoib, rate=7, mtu=4, scope=2;In thisexample, two partitions are defined. The first one is the default partition, which needs to remain (0x7fff). The second partition with key0x0610is active on therxe0device and port1. It is this second partition that we will protect with SELinux.
Let’s create a new type to assign to this partition. We use the CIL format again to define the policy enhancement, and store these rules in a file calledinfiniband_pkey.cil:
(typeattribute ibpkey_type)
(type local_ibpkey_t)
(typeattributeset ibpkey_type local_ibpkey_t)
(allow unconfined_t local_ibpkey_t (infiniband_pkey (access)))Within this example, we’ve created thelocal_ibpkey_ttype, assigned it to theibpkey_typeattribute, and grantedunconfined_taccess privilege within theinfiniband_pkeyclass.
Let’s load the policy:
# semodule -i infiniband_pkey.cilWe can now create an appropriate mapping to this partition, and limit it to the ff12:: subnet prefix:
# semanage ibpkey -a -t local_ibpkey_t -x ff12:: 0x0610
# semanage ibpkey -l
SELinux IB PKey Type Subnet_Prefix Pkey Number
local_ibpkey_t ff12:: 0x610While we can create separate types for each partition, we can also use an SELinux range to use SELinux category support:
# semanage ibpkey -a -t local_ibpkey_t -r s0-s0:c0.c4 -x ff12:: 0x0610With categories, we can grant access based on the source domain category, something we benefit from with other network protection measures such as labeled networking, which we tackle next.
