xen pci-passthrogh

【前言】

• 小叮嚀
  • 為了使 Xen patched Kerenl 能支援 PCI Passthrough, 請務必確認以下Xen 相關的核心配置.

    CONFIG_XEN_PCIDEV_FRONTEND=y
    CONFIG_XEN_PCIDEV_BACKEND=y
    CONFIG_XEN_PCIDEV_BACKEND_PASS is not set
    CONFIG_XEN_PCIDEV_BACKEND_VPCI=y
    CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set
    

• 小技巧
  • Binding at Boot-Time.
  • 直接將顯示卡所佔的 PCI_Address 在開機時就隱藏起來,使得一開始便能直接 bind 到該 PCI_Address.
  • 透過 lspci 指令來查詢顯示卡所佔的實體 PCI Address.

    # 台中機器
    /boot/vmlinuz-2.6.26-2-xen-amd64 root=/dev/sda1 ro console=tty0 pciback.permissive pciback.hide=(0000:01:00.0)
    

    # 新竹機器
    /boot/vmlinuz-2.6.26-2-xen-amd64 root=/dev/sda1 ro console=tty0 pciback.permissive pciback.hide=(0000:08:00.0)
    

  • 如果透過 lspci 指令查出顯示卡為 unknown 或是其他不明資訊, 請先更新 PCI ID database 試看看.

    $ update-pciids
    

【Step 0: 流程】

• 本範例將說明如何使用及設定 Xen PCI Passthrough

【Step 1: 首先查看主機上的顯卡資訊】

$ lspci -vv | more

# 台中機器
01:00.0 VGA compatible controller: nVidia Corporation G92 [GeForce 9800 GT] (rev a2)
        Subsystem: ASUSTeK Computer Inc. Device 82a0
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Region 3: Memory at fa000000 (64-bit, non-prefetchable) [size=32M]
        Region 5: I/O ports at dc80 [size=128]
        [virtual] Expansion ROM at fea00000 [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nvidia, nvidiafb

# 新竹機器
08:00.0 VGA compatible controller: nVidia Corporation G92 [GeForce 9800 GT] (rev a2) (prog-if 00 [VGA controller])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at f0000000 (64-bit, non-prefetchable) [size=32M]
	Region 5: I/O ports at 2100 [size=128]
	Capabilities: <access denied>
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nvidia

【Step 2: 產生一台實驗用虛擬機器】

• 設定你想要怎樣規格的虛擬機器.

  $ sudo gedit /etc/xen-tools/xen-tools.conf
  

  45  dir = /home
  128 size   = 1Gb      # Disk image size.
  129 memory = 128Mb    # Memory size
  130 swap   = 128Mb    # Swap size
  133 dist   = lenny    # Default distribution to install.
  134 image  = sparse   # Specify sparse vs. full disk images.
  163 gateway   = 192.168.1.254
  164 netmask   = 255.255.255.0
  165 broadcast = 192.168.1.255
  214 mirror = http://free.nchc.org.tw/debian/
  257 serial_device = hvc0 #default
  

  $ sudo xen-create-image --hostname=xg01 --ip=192.168.1.X --mac=00:16:00:00:00:XX --force
  

【Step 3: PCI Frontend Configuration 設定你的 DomU 】

$ sudo gedit /home/domains/xg01.cfg

9  kernel      = '/boot/vmlinuz-2.6.26-2-xen-amd64'
10 ramdisk     = '/boot/initrd.img-2.6.26-2-xen-amd64'
11 memory      = '128'
12 vcpus       = '1'

14 pci         = ['08:00.0']

19 root        = '/dev/sda2 ro'
20 disk        = [
                  'file:/home/domains/xg01/disk.img,sda2,w',
                  'file:/home/domains/xg01/swap.img,sda1,w',
               ]
29 name        = 'xg01'
 
#
#  Networking
#
34 vif         = [ 'ip=192.168.1.X ,mac=00:16:00:00:00:XX' ]
 
#
#  Behaviour
#
39 on_poweroff = 'destroy'
40 on_reboot   = 'restart'
41 on_crash    = 'restart'

【Step 4: PCI Backend Configuration 設定你的 Dom0 】

• 切換身份為 Super user.

  $ sudo su -
  

• 隱藏 dom0 的PCI 匯流排的位址, 並交由 pciback 模組來控制該匯流排存取權.

  # echo -n "0000:08:00.0" > /sys/bus/pci/drivers/nvidia/unbind
  

• 將 dev_ids 轉到 pciback 成為新的 binding slot.

  # echo -n "0000:08:00.0" > /sys/bus/pci/drivers/pciback/new_slot
  # echo -n "0000:08:00.0" > /sys/bus/pci/drivers/pciback/bind
  

  # cat /sys/bus/pci/drivers/pciback/slots
  

  0000:08:00.0
  

• 警告: 請確認您的顯示卡資源沒有被其 Kernel module 佔有及控制.

  $ ls -al /sys/bus/pci/devices/0000:08:00.0/ | grep driver
  

  driver -> ../../../../bus/pci/drivers/nvidia ---> 此連結應該已經不存在. NVIDIA module 已無該裝置控制權.
  driver -> ../../../../bus/pci/drivers/pciback ---> 控制權應該已經交賦予 pciback module.
  

【Step 5: 硬體定址存取設定】

Permissive Flag

• 說明: 透過 "lspci" 指令來查詢您顯示卡的 PCI address.

  # su clouder
  $ sudo gedit /etc/xen/xend-pci-permissive.sxp
  

  /*** 台中機器 ***/
  (unconstrained_dev_ids
       #('0123:4567:89AB:CDEF')
       ('0000:01:00.0')
  )
  

  /*** 新竹機器 ***/
  (unconstrained_dev_ids
       #('0123:4567:89AB:CDEF')
       ('0000:08:00.0')
  )
  

User-space Quirks

$ sudo gedit /etc/xen/xend-pci-quirks.sxp

• 說明: 請到 PCI_DB 網站:  http://www.pcidatabase.com/ 來查詢您顯示卡的 vendor_id & device_id.

  (pci_ids
     # Entries are formated as follows:  
     #     <vendor>:<device>[:<subvendor>:<subdevice>]
  
     ('10de:0605'   # NVIDIA 9800GT
     )
  )
  

【Step 6: 啟動並登入你的虛擬機器 DomU 】

• 說明: 先用 root 免密碼登入.
• @ Dom0

  $ cd /home/domains
  $ sudo xm create -c xg01.cfg
  

【Step 7: 檢查 DomU 的 PCI Passthrough 有無成功】

• 先用 root 免密碼登入 xg01, 然後新增使用者 clouder.

  # adduser clouder
  

  Adding user `clouder' ...
  Adding new group `clouder' (1000) ...
  Adding new user `clouder' (1000) with group `clouder' ...
  Creating home directory `/home/clouder' ...
  Copying files from `/etc/skel' ...
  Enter new UNIX password:
  Retype new UNIX password:
  passwd: password updated successfully
  Changing the user information for clouder
  Enter the new value, or press ENTER for the default
    Full Name []:
    Room Number []: 
    Work Phone []: 
    Home Phone []: 
    Other []: 
  Is the information correct? [Y/n] y
  

• 先登出 "root" 並且改用使用者 "clouder" 登入,並查看顯卡資訊有無正常顯示.

  $ exit
  $ sudo apt-get update
  $ sudo apt-get install wget pciutils udev
  $ lspci
  

  00:00.0 VGA compatible controller: nVidia Corporation Unknown device 0605 (rev a2)
  

• 更新 PCI ID Database.(當您發現系統無法正常偵測到你的顯示卡時)

  $ sudo update-pciids
  $ lspci
  

  00:00.0 VGA compatible controller: nVidia Corporation G92 [GeForce 9800 GT] (rev a2)
  

• @DomU 查看顯卡資源有無順利分配到 DomU.

  $ dmesg | grep pci
  

  [    0.745795] pcifront pci-0: Installing PCI frontend
  [    0.745872] pcifront pci-0: Creating PCI Frontend Bus 0000:00
  [    0.848291] pci 0000:00:00.0: Boot video device
  

• @Dom0 查看顯卡資源有無順利分配到 DomU.

  $ dmesg | grep pci
  

  # 台中機器
  [    0.718704] ACPI: bus type pci registered
  [    0.902598] pci 0000:00:1f.0: quirk: region 0400-047f claimed by ICH6 ACPI/GPIO/TCO
  [    0.902695] pci 0000:00:1f.0: quirk: region 0480-04bf claimed by ICH6 GPIO
  [    0.925960] pnp 00:01: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c
  [    0.928118] pnp 00:0a: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c
  [    0.928580] pnp 00:0b: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c
  [    0.929060] pnp 00:0c: PNP0c01: calling quirk_system_pci_resources+0x0/0x15c
  [    1.061488] pci 0000:01:00.0: Boot video device
  [    1.061674] Allocate Port Service[0000:00:01.0:pcie00]
  [    1.065543] Allocate Port Service[0000:00:01.0:pcie03]
  [  476.851360] pciback 0000:08:00.0: seizing device
  [  560.262910] pciback: vpci: 0000:08:00.0: assign to virtual slot 0
  

  # 新竹機器
  [    0.655891] ACPI: bus type pci registered
  [    0.673706] pci 0000:00:1f.0: quirk: region f800-f87f claimed by ICH6 ACPI/GPIO/TCO
  [    0.673706] pci 0000:00:1f.0: quirk: region fa00-fa3f claimed by ICH6 GPIO
  [    0.706664] pnp 00:0c: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c
  [    0.706664] pnp 00:0d: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c
  [    0.706664] pnp 00:0e: PNP0c02: calling quirk_system_pci_resources+0x0/0x15c
  [    0.707107] pnp 00:0f: PNP0c01: calling quirk_system_pci_resources+0x0/0x15c
  [    0.836266] pci 0000:08:00.0: Boot video device
  [    0.836266] Allocate Port Service[0000:00:06.0:pcie00]
  [    0.836266] Allocate Port Service[0000:00:06.0:pcie03]
  [    0.836266] Allocate Port Service[0000:00:1c.0:pcie00]
  [    0.836266] Allocate Port Service[0000:00:1c.0:pcie02]
  [    0.836266] Allocate Port Service[0000:00:1c.0:pcie03]
  [    0.836266] Allocate Port Service[0000:00:1c.4:pcie00]
  [    0.836266] Allocate Port Service[0000:00:1c.4:pcie02]
  [    0.836266] Allocate Port Service[0000:00:1c.4:pcie03]
  [    0.836266] Allocate Port Service[0000:00:1c.5:pcie00]
  [    0.840261] Allocate Port Service[0000:00:1c.5:pcie02]
  [    0.840261] Allocate Port Service[0000:00:1c.5:pcie03]
  [ 1891.425509] pciback 0000:08:00.0: seizing device
  [ 2133.589319] pciback: vpci: 0000:08:00.0: assign to virtual slot 0
  

【討論 & 休息】