Solaris Source Insight: PCI system implementation - Part 2

Thu Oct 29 10:46:29 CST 2009

The bus resource enumerated in pci_setup_tree() are stored in pci_bus_res array.

common/sys/pci_impl.h
82 /* pci bus resource maps */
83 struct pci_bus_resource *pci_bus_res;
... ...
93 struct pci_bus_resource {
94 |_______struct memlist *io_avail;|______/* available free io res */
95 |_______struct memlist *io_used;|_______/* used io res */
96 |_______struct memlist *mem_avail;|_____/* available free mem res */
97 |_______struct memlist *mem_used;|______/* used mem res */
98 |_______struct memlist *pmem_avail; /* available free prefetchable mem res */
99 |_______struct memlist *pmem_used; /* used prefetchable mem res */
100 |_______struct memlist *bus_avail;|_____/* available free bus res */
101 |_______|_______|_______/* bus_space_used not needed; can read from regs */
102 |_______dev_info_t *dip;|_______/* devinfo node */
103 |_______void *privdata;||_______/* private data for configuration */
104 |_______uchar_t par_bus;|_______/* parent bus number */
105 |_______uchar_t sub_bus;|_______/* highest bus number beyond this bridge */
106 |_______uchar_t root_addr;|_____/* legacy peer bus address assignment */
107 |_______uchar_t num_cbb;|_______/* # of CardBus Bridges on the bus */
108 |_______boolean_t io_reprogram;|/* need io reprog on this bus */
109 |_______boolean_t mem_reprogram;|_______/* need mem reprog on this bus */
110 |_______boolean_t subtractive;|_/* subtractive PPB */
111 |_______uint_t mem_size;|_______/* existing children required MEM space size */
112 |_______uint_t io_size;||_______/* existing children required I/O space size */
113 };

1. Create pci root bus in device tree
This logic is in create_root_bus_dip(uchar_t bus). Since we may have multiple IOH's on some platforms, there may be multiple root bus dev_info nodes under rootnex node. If the root bus does indeed have PCI-Ex in the path, the "device-type" property will be set to "pciex" and "compatible" set to "pciex_root_complex". Otherwise, only "device-type" is set to "pci".

To judge if root bus has PCI-Ex in the path, the common way is to check MCFG in ACPI tables. However in current code, it is hard-coded. That means the logic enumerates each devices under the root bus (exclude the sub-bus) and checks if any device has PCI_Ex capability in it's configure space.

2. Enumerate devices on bus 0 (exclude sub-buses)
This logic is implemented in enumerate_bus_devs().

intel/io/pci/pci_boot.c
1471 /*
1472 * For any fixed configuration (often compatability) pci devices
1473 * and those with their own expansion rom, create device nodes
1474 * to hold the already configured device details.
1475 */
1476 void
1477 enumerate_bus_devs(uchar_t bus, int config_op)

It is used in multiple cases. The "bus" parameter is the bus about to enumerate and "config_op" can be set to:

CONFIG_NEW: configure the pci bus (called in pci_reprogram() path)
CONFIG_FIX: fix the pci bus (called in add_pci_fixes() path)
CONFIG_INFO: enumerate the pci bus (called in pci_setup_tree() path)

During enumeration, for each function enumerated, process_devfunc() is called with "config_op" passed.

intel/io/pci/pci_boot.c
1825 static void
1826 process_devfunc(uchar_t bus, uchar_t dev, uchar_t func, uchar_t header,
1827 ushort_t vendorid, int config_op)
1828 {

We are interested at two aspects. i) how the pci hierarchy is organized? Especially how multiple IOH's are supported? ii) which properties are created for each kind of pci device node?

1) How multiple IOH's are support?

1904 |_______/* make sure parent bus dip has been created */
1905 |_______if (pci_bus_res[bus].dip == NULL)
1906 |_______|_______create_root_bus_dip(bus);

So if a bus doesn't start from a pci bridge, it should be a seperated root complex.

2) How the pci hierarchy information are stored during enumeration?

common/sys/pci_impl.h
93 struct pci_bus_resource {
94 |_______struct memlist *io_avail;|______/* available free io res */
95 |_______struct memlist *io_used;|_______/* used io res */
96 |_______struct memlist *mem_avail;|_____/* available free mem res */
97 |_______struct memlist *mem_used;|______/* used mem res */
98 |_______struct memlist *pmem_avail; /* available free prefetchable mem res */
99 |_______struct memlist *pmem_used; /* used prefetchable mem res */
100 |_______struct memlist *bus_avail;|_____/* available free bus res */
101 |_______|_______|_______/* bus_space_used not needed; can read from regs */
102 |_______dev_info_t *dip;|_______/* devinfo node */
103 |_______void *privdata;||_______/* private data for configuration */
104 |_______uchar_t par_bus;|_______/* parent bus number */
105 |_______uchar_t sub_bus;|_______/* highest bus number beyond this bridge */
106 |_______uchar_t root_addr;|_____/* legacy peer bus address assignment */
107 |_______uchar_t num_cbb;|_______/* # of CardBus Bridges on the bus */
108 |_______boolean_t io_reprogram;|/* need io reprog on this bus */
109 |_______boolean_t mem_reprogram;|_______/* need mem reprog on this bus */
110 |_______boolean_t subtractive;|_/* subtractive PPB */
111 |_______uint_t mem_size;|_______/* existing children required MEM space size */
112 |_______uint_t io_size;||_______/* existing children required I/O space size */
113 };

"dip" holds the dev_info nodes on the system device tree. PCI nodes were created during enumeration according to their position in the PCI hierarchy. All devices/functions attached on the bus were linked to a list hooked on "privdata". The "par_bus" and "sub_bus" were used to mentain the hierarchy of the PCI system. The io/mem/prefetchable mem/bus resources consumed by the children buses and devices were recorded in the various "memlist".

In general, for each compatible PCI device, enumeration logic (process_devfunc() with CONFIG_INFO op type) will do the following items.

(a) create a dev_info node in the system device tree
If the device is under a bridge in the PCI hierarchy, the created device node will be placed under the node for the bridge; If the device is that integrated in the root complex, the logic will first create a special node under ddi_root_node() which present the root complex and the PCI device node would be under the root complex node.

(b) set the property values compliant with IEEE Std 1275-1994, refer to "PCI bus binding to IEEE Std 1275-1994"

(c) Set the device PM state to D0

(d) If the device is a normal PCI function, it will be put in the list hooked on the pci_bus_resource; if it is a bridge, the responding pci_bus_resouce is updated and the relationship between the parent bus and the secondary bus is built through updating the par_bus, sec_bus members in pci_bus_resource.

(e) Special handling of IOAPIC nodes
If the device/function is an IOAPIC node, a dev_info node will be created as the child of IOAPIC nexus node (located directly under ddi_root_node()). Some properties are created/updated: "vendor-id", "device-id", "device_type"(ioapic), "reg".

(f) for ck8-04 based PCI ISA bridge, some special properties created/updated

(g) set the compatible property values compliant with IEEE Std 1275-1994, refer to "PCI bus binding to IEEE Std 1275-1994". compatible property will be used for driver binding.

Here is how compatible property organized:

"compatible" Construct a list of names in most-specific to least-specific order. The names shall be derived from
values of the Vendor ID, Device ID, Subsystem Vendor ID, Subsystem ID, Revision ID and Class Code
bytes, and shall have the following form, and be placed in the list in the following order:
pciVVVV,DDDD.SSSS.ssss.RR (1)
pciVVVV,DDDD.SSSS.ssss (2)
pciSSSS,ssss (3)
pciVVVV,DDDD.RR (4)
pciVVVV,DDDD (5)
pciclass,CCSSPP (6)
pciclass,CCSS (7)
where:
VVVV is the Vendor ID
DDDD is the Device ID
SSSS is the Subsystem Vendor ID
ssss is the Subsystem ID
RR is the Revision ID
CC is the most-significant byte of the Class Code (base class code, at 0x0b).
SS is the second-most-significant byte of the Class Code (sub-class code, at 0x0a).
PP is the least-significant byte of the Class Code (programming interface, at 0x09).
Entries (1), (2) and (3) shall be included if and only if the Subsystem Vendor ID is non-zero.
Entry (3) is supplied only for backwards compatiblity with versions of the PCI Binding prior to Revision
2.1; new OS binding mechanisms should instead use forms (1) or (2) to select a driver based on the values
of the Subsystem Vendor ID and Subsystem ID.
VVVV, DDDD, SSSS, ssss and RR are lower-case ASCII hexadecimal numbers without leading zeroes.
CC, SS and PP are lower-case ASCII hexadecimal numbers including leading zeroes.

(h) special case for disk controller

2025 |_______/*
2026 |_______ * See if this device is a controller that advertises
2027 |_______ * itself to be a standard ATA task file controller, or one that
2028 |_______ * has been hard coded.
2029 |_______ *
2030 |_______ * If it is, check if any other higher precedence driver listed in
2031 |_______ * driver_aliases will claim the node by calling
2032 |_______ * ddi_compatibile_driver_major. If so, clear pciide and do not
2033 |_______ * create a pci-ide node or any other special handling.
2034 |_______ *
2035 |_______ * If another driver does not bind, set the node name to pci-ide
2036 |_______ * and then let the special pci-ide handling for registers and
2037 |_______ * child pci-ide nodes proceed below.
2038 |_______ */

(i) Add the "reg" and "assigned-addresses" property.
Read the mem/IO/rom resources from BAR and ROM register in PCI configure space. Create the "reg" and "assigned-addresses" properties compliant with IEEE Std 1275-1994. Refer to the spec for decoding of "reg" and "assigned-addresses" properties.

The resources allocation will be recorded in pci_bus_resource[].

(j) Bind the driver with this device node

ndi_devi_bind_driver() is called here.

To be continued ...