Symptom

When Intel VMD is enabled in UEFI, some Linux OSes may not recognize disks connected to the VMD device.

The affected OSes are RHEL 8.7, RHEL 9.0, SLES 15.4 and Ubuntu 20.04.x (Ubuntu kernel version: 5.13) with all Lenovo ThinkSystem V2/V3 servers supporting VMD / VROC.

(where Intel VMD = Intel Volume Management Device) 

Affected configurations

The system may be any of the following Lenovo servers:

• ThinkServer SR660 V2, Type 7D6L, 7DAA, 7DAB, 7DBV, any model                                                      
• ThinkSystem SN550 V2, Type 7Z69, any model                 
• ThinkSystem SR850 V2, Type 7D33/7D32/7D31/7D41, any model
• ThinkSystem SR850 V3, Type 7D97/7D96/7D98, any model
• ThinkSystem SR860 V2, Type 7Z60/7Z59/7D42, any model
• ThinkSystem SR860 V3, Type 7D94/7D93/7D95, any model
• ThinkSystem SD630 v2, Type 7D1K, any model
• ThinkSystem SD650 V3, Type 7D7M, any model
• ThinkSystem SD650-I V3, Type 7D7L, any model
• ThinkSystem SD650-N V2, Type 7D1N, any model               
• ThinkSystem SR250 V2, Type 7D7R, 7D7Q, any model           
• ThinkSystem SR630 V2, Type 7Z70/7Z71, any model
• ThinkSystem SR630 V3, Type 7D72/7D73, any model     
• ThinkSystem SR650 V2, Type 7Z72/7Z73, any model
• ThinkSystem SR650 V3, Type 7D75/7D76, any model          
• ThinkSystem SR670 V2, Type 7Z22/7Z23/7D47, any model       
• ThinkSystem ST250 V2, Type 7D8F, 7D8G, any model           
• ThinkSystem ST50 V2, Type 7D8J, 7D8K, any model            
• ThinkSystem ST650 V2, Type 7Z74/7Z75, any model
• ThinkSystem ST650 V3, Type 7D7B/7D7A, any model
• ThinkSystem ST658 V3, Type 7D7C, any model

The system is configured with at least one of the following:

• Canonical Ubuntu 20.04, any
• RHEL 8.7, any
• RHEL 9.0, any RHEL 9.0
• SLES 15.4, any, any version

This tip is not option specific.

The system has the symptom described above.

Note: This does not imply that the network operating system will work under all combinations of hardware and software.
Please see the compatibility page for more information:

http://www.lenovo.com/us/en/serverproven/

Solution

For RHEL 8.7, updating to errata kernel 4.18.0-425.13.1.el8_7 or later can fix the issue. Please refer to RHSA-2023:0832 - Security Advisory for more information from Red Hat Customer Portal:

https://access.redhat.com/errata/RHSA-2023:0832

Fixes for the rest of the OSes will be included in future versions of each OS. Contact the individual OS vendors for fix schedules.

Workaround

Installing RHEL 8.7 on non-VMD drives

For RHEL 8.7, updating to errata kernel 4.18.0-425.13.1.el8_7 or later can resolve the issue.

• Disable VMD by booting the server into the HII text uEFI by pressing F1 when prompted
• Perform these steps: Select System Settings -> Devices and I/O Ports -> Intel vmd technology-> Enable/Disable Intel VMD : Disabled
• Install RHEL 8.7 and boot into OS
• Install kernel-4.18.0-425.13.1.el8_7 packages
• Enable VMD by booting the server into the HII text uEFI by pressing F1 when prompted
• Perform these steps: Select System Settings -> Devices and I/O Ports -> Intel vmd technology -> Enable/Disable Intel VMD : Enabled

The system can boot into Red Hat Enterprise Linux 4.18.0-425.13.1.el8_7.x86_64 successfully.

Installing any affected Linux distribution changing kernel parameters

Alternatively, appending the kernel parameter "intremap=off" during OS installation can also resolve the issue.

For the other listed OSes, append "intel_iommu=on" during installation or after the OS is installed.

[During installation phase]

1.   When the boot loader (GRUB) menu appears, press 'e' to enter the configuration submenu.
2.   Append 'intel_iommu=on' to the kernel parameter.
3.   Press 'ctrl+x' to boot into the OS installation GUI.
4.   Install OS.
5.   After the installation and system reboot, press 'e' to enter the configuration submenu when the boot loader (GRUB) menu appears.
6.   Append 'intel_iommu=on' to the kernel parameter.
7.   Press 'ctrl+x' to boot into the installed OS.

Note: There is no need to perform steps 5-7 in SLES 15.4 because the installer of SLES 15.4 automatically appends 'intel_iommu=on' to GRUB.cfg if step 2 is performed.

[Up-and-running OS: the following change will take effect for every reboot]

1.   Edit the file '/etc/default/grub'
2.   Append 'intel_iommu=on' to the end of GRUB_CMDLINE_LINUX line.
3.   Run the following command to update GRUB configuration file
   # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Installing any of the affected Linux OSes onto a VMD / VROC RAID

It is not possible to install any of the affected operating systems directly onto an Intel VMD.

Customers wishing to install Red Hat Enterprise Linux 8 have to install at least RHEL 8.8 for the installation to complete.

Please refer to the OS supplier for any other Linux distribution listed in this tip.

Lenovo Server Operating System Compatibility and support

For operating system interoperability refer to the Lenovo OS Interoperability Guide available at

https://lenovopress.lenovo.com/osig

For Linux distributions not listed there, Lenovo will provide best effort support, that is full hardware support. Any operating system related issues have to be discussed with the OS supplier.

Additional Information

The VMD enhancement of RHEL 9.0, SLES 15.4 and Ubuntu 20.04.x (Ubuntu kernel version: 5.13) requires that the IOMMU subsystem is enabled in Linux kernel. The upstream kernel community recently (kernel version: 5.19) supports VMD without appending "intel_iommu=on".

RHEL 9.0, SLES 15.4 and Ubuntu 20.04.x (Ubuntu kernel version: 5.13) do not enable the Intel IOMMU feature by default, so it is required to append 'intel_iommu=on'.

Ubuntu 22.04 enables the Intel IOMMU feature (CONFIG_INTEL_IOMMU_DEFAULT_ON=Y) by default, so it is not required to follow the above-mentioned steps.

The older Linux OSes (RHEL 8.6 or prior, SLES15.3 or prior and Ubuntu 20.04.x with Ubuntu kernel version prior to 5.13) do not include the VMD enhancement, so appending 'intel_iommu=on' is not required.