pve 记录 篇四：康耐信N100小主机玩转指南，显卡SR-IOV

N100 玩转指南

双十一通过站内链接，入手了这个性价比爆棚的N100小主机，正在折腾，所以在这里分享一下折腾经历！

N100-01

• 安装PVE

• 搞定硬件直通

• 十二代CPU的显卡 SR-IOV

安装PVE

pve

制作安装介质

N100-02

• 下载 PVE 最新版 iso

• 制作安装介质

  • GNU Linux

    # dd bs=1M conv=fdatasync if=./proxmox-ve_*.iso of=/dev/XYZ

  • MacOS

    打开Terminal

    转换 .iso 文件到 .dmg 用这个 hdiutil

    # hdiutil convert proxmox-ve_*.iso -format UDRW -o proxmox-ve_*.dmg

    获取磁盘序列

    # diskutil list

    卸载需要写入的U盘

    # diskutil list # diskutil unmountDisk /dev/diskX

    dd 制作启动盘

    # sudo dd if=proxmox-ve_*.dmg bs=1M of=/dev/rdiskX

  • Windows

    • 用 rufus 工具

• 开始安装

  • BIOS 设置U盘启动

  • 进入启动菜单

    • 图形安装方式

    • Console安装方式

开启硬件直通

N100-03

• 添加PVE源 apt list

  echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve.list apt update

• BIOS 相关设置

• 开启VT-d - IOMMU

• VT-x

• 添加 kernel cmdline

  • Grub 设置

    编辑 /etc/default/grub

    GRUB_CMDLINE_LINUX_DEFAULT="quite"

    GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream"

  • kernel 内核模块设置

    #!/bin/sh cat >> /etc/modules-load.d/vfio.conf << EOF vfio vfio_iommu_type1 vfio_pci vfio_virqfd EOF

    • 添加相应参数

      添加 参数到 /etc/modprobe.d/vfio.conf

      echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio.conf

    • 更新 grub & initramfs

      # update-initramfs -u -k all # update-grub # proxmox-boot-tool refresh

    • 检查IOMMU 是否设置成功

      dmesg | grep -e DMAR -e IOMMU

      输出类似如下：

      [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA

      [ 0.007697] ACPI: DMAR 0x0000000079195C18 0000A8 (v01 INTEL EDK2 00000002 01000013)

      [ 0.007741] ACPI: Reserving DMAR table memory at [mem 0x79195c18-0x79195cbf]

      [ 0.130354] DMAR: IOMMU enabled

      [ 0.388873] DMAR: Host address width 39

      [ 0.389091] DMAR: DRHD base: 0x000000fed90000 flags: 0x0

      [ 0.389318] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e

      DMAR: IOMMU enabled 注意输出有这个就是成功了！

    • 由于N100主机只支持 EFI启动，我遇到最大的坑就在这里

      我用的是zfs文件系统，PVE在EFI启动过程使用的是systemd-boot，没有使用grub，所以上面添加 kernel cmdline的方法是不管用的，需要修改loader.conf

    • Systems-boot kernel cmdline 修改

      修改 /etc/kernel/cmdline

      # cat /etc/kernel/cmdline root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction

      # update-initramfs -u -k all

显卡直通

显卡SR-IOV

n100-04

安装内核工具

apt install build-* dkms

下载内核补丁

注意：内核补丁还在测试，不建议用在生产环境

下载github源码 i915-sriov-dkms

进入下载的目录，修改PACKAGE_NAME to i915-sriov-dkms, 修改 PACKAGE_VERSION to 6.2. 保存文件。

目前PVE的内核用的是 6.2

# uname -ar Linux n100 6.2.16-3-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) x86_64 GNU/Linux

拷贝这个目录到 /usr/src/i915-sriov-dkms-6.2

安装dkms模块

dkms -i -m i915-sriov-dkms -v 6.2

安装 sysfsutils

echo "devices/pci0000:00/0000:00:02.0/sriov_numvfs = 7" > /etc/sysfs.conf

修改内核参数

这里vfs就是虚拟显卡的数量，最大多少我没查，这里我设置的是3。

intel_iommu=on i915.enable_guc=3 i915.max_vfs=3

• grub

  修改 /etc/defautl/grub 添加到 GRUB_CMDLINE_LINUX_DEFAULT="quite"这句里面

• Systems-boot

  我用的zfs 所以修改是这种方式

  /etc/kernel/cmdline修改成

  root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction

更新内核和 initramfs

N100-5

update-initramfs -u -k all

proxmox-boot-tool refresh

锁定内核

由于最新的内核没有headers提供，为了避免更新把这个内核顶了，我们要锁定目前编译好的内核！

​ # proxmox-boot-tool kernel pin 6.2.16-3-pve

重启

reboot

检查是否显卡 SR-IOV是否成功

N100-06

lcpci | grep -i vga

输出如下，就是成功了！

00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]

00:02.1 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]

00:02.2 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]

00:02.3 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]

或者如下输出，看到我的已经开启了3个虚拟显卡

# dmesg | grep i915

[ 0.000000] Command line: initrd=EFIproxmox6.2.16-3-pveinitrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction

[ 0.077241] Kernel command line: initrd=EFIproxmox6.2.16-3-pveinitrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction

[ 4.811959] i915 0000:00:02.0: Running in SR-IOV PF mode

[ 4.812864] i915 0000:00:02.0: [drm] VT-d active for gfx access

[ 4.813302] i915 0000:00:02.0: vgaarb: deactivate vga console

[ 4.813362] i915 0000:00:02.0: [drm] Using Transparent Hugepages

[ 4.813758] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem

[ 4.814738] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])

[ 4.815136] i915 0000:00:02.0: Direct firmware load for i915/adlp_dmc.bin failed with error -2

[ 4.817242] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)

[ 4.822141] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/tgl_guc_70.bin version 70.5.1 [ 4.822154] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3 [ 4.827027] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads!

[ 4.827569] i915 0000:00:02.0: [drm] GT0: GUC: submission enabled

[ 4.827574] i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled

[ 4.827921] i915 0000:00:02.0: [drm] GuC RC: enabled

[ 4.828778] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])

[ 4.828956] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized

[ 4.854926] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0

[ 4.856871] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])

[ 4.857105] i915 0000:00:02.0: 3 VFs could be associated with this PF

[ 4.887664] fbcon: i915drmfb (fb0) is primary device

[ 4.972789] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device

[ 5.638917] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem

[ 5.639816] i915 0000:00:02.1: enabling device (0000 -> 0002)

[ 5.640648] i915 0000:00:02.1: Running in SR-IOV VF mode

[ 5.641809] i915 0000:00:02.1: [drm] GT0: GUC: interface version 0.1.0.0

[ 5.643254] i915 0000:00:02.1: [drm] VT-d active for gfx access

[ 5.644017] i915 0000:00:02.1: [drm] Using Transparent Hugepages

[ 5.645371] i915 0000:00:02.1: [drm] GT0: GUC: interface version 0.1.0.0

[ 5.646658] i915 0000:00:02.1: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF

[ 5.647370] i915 0000:00:02.1: HuC firmware PRELOADED

[ 5.650198] i915 0000:00:02.1: [drm] Protected Xe Path (PXP) protected content support initialized

[ 5.650917] i915 0000:00:02.1: [drm] PMU not supported for this GPU.

[ 5.651749] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.1 on minor 1

[ 5.656497] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem

[ 5.657170] i915 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none

[ 5.657938] i915 0000:00:02.2: enabling device (0000 -> 0002)

[ 5.658678] i915 0000:00:02.2: Running in SR-IOV VF mode

[ 5.659806] i915 0000:00:02.2: [drm] GT0: GUC: interface version 0.1.0.0

[ 5.661509] i915 0000:00:02.2: [drm] VT-d active for gfx access

[ 5.662199] i915 0000:00:02.2: [drm] Using Transparent Hugepages

[ 5.663417] i915 0000:00:02.2: [drm] GT0: GUC: interface version 0.1.0.0

[ 5.665322] i915 0000:00:02.2: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF

[ 5.665956] i915 0000:00:02.2: HuC firmware PRELOADED

[ 5.668308] i915 0000:00:02.2: [drm] Protected Xe Path (PXP) protected content support initialized

[ 5.668947] i915 0000:00:02.2: [drm] PMU not supported for this GPU.

[ 5.669666] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.2 on minor 2

[ 5.673931] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem

[ 5.674572] i915 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=none

[ 5.675219] i915 0000:00:02.2: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none

[ 5.675964] i915 0000:00:02.3: enabling device (0000 -> 0002)

[ 5.676640] i915 0000:00:02.3: Running in SR-IOV VF mode

[ 5.677500] i915 0000:00:02.3: [drm] GT0: GUC: interface version 0.1.0.0

[ 5.678512] i915 0000:00:02.3: [drm] VT-d active for gfx access

[ 5.679186] i915 0000:00:02.3: [drm] Using Transparent Hugepages

[ 5.680354] i915 0000:00:02.3: [drm] GT0: GUC: interface version 0.1.0.0

[ 5.681176] i915 0000:00:02.3: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF

[ 5.681765] i915 0000:00:02.3: HuC firmware PRELOADED

[ 5.684137] i915 0000:00:02.3: [drm] Protected Xe Path (PXP) protected content support initialized

[ 5.684777] i915 0000:00:02.3: [drm] PMU not supported for this GPU.

[ 5.685487] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.3 on minor 3

[ 5.686338] i915 0000:00:02.0: Enabled 3 VFs

感谢阅读

虚拟机虚拟显卡搞定，下期再续！

今天就到这里了，至于虚拟机如何使用这个显卡，后续再更新！

作者声明本文无利益相关，欢迎值友理性交流，和谐讨论～