Solaris Source Insight: PCI system implementation - Part 3

We went over the first pass of pci_enumeration() during boot. Before go ahead with the second pass, we'd like to get the answer of what happen during the two passes of pci_enumeration.

First, let's track where the first pass of pci_enumeration() is called during boot process.

main()
start_up()
startup_modules()
setup_ddi()
create_devinfo_tree()
impl_setup_ddi()
impl_bus_initialprobe()

In imple_bus_initialprobe(), the bus probe modules will be loaded and the first pass of enumeration will be called. For xpv kernel, if it runs on behalf of domain 0, pci_autoconfig, isa, xpv_autoconfig will be loaded and probed; if on behalf of a PV guest, PCI enumeration module will not load since it has no permission to access the real device. Otherwise, if it is bare metal kernel, pci_autoconfig, isa and acpidev will be loaded and probed. Acpidev is a probe for ACPI name space. It enumerate all objects in the ACPI name space and create device nodes for the objects which it is intereted at. The system devices such as processor, memory, and IOMMU are essential for the system, but there is no existing bus mechanism to enumerate. Acpidev was designed to enumerate such devices.

i86pc/os/ddi_impl.c
2642 /*
2643 * impl_bus_initialprobe
2644 *|_____Modload the prom simulator, then let it probe to verify existence
2645 *|_____and type of PCI support.
2646 */
2647 static void
2648 impl_bus_initialprobe(void)
2649 {
2650 |_______struct bus_probe *probe;
2651
2652 |_______/* load modules to install bus probes */
2653 #if defined(__xpv)
2654 |_______if (DOMAIN_IS_INITDOMAIN(xen_info)) {
2655 |_______|_______if (modload("misc", "pci_autoconfig") < 0) {
2656 |_______|_______|_______panic("failed to load misc/pci_autoconfig");
2657 |_______|_______}
2658
2659 |_______|_______if (modload("drv", "isa") < 0)
2660 |_______|_______|_______panic("failed to load drv/isa");
2661 |_______}
2662
2663 |_______(void) modload("misc", "xpv_autoconfig");
2664 #else
2665 |_______(void) modload("misc", "acpidev");
2666
2667 |_______if (modload("misc", "pci_autoconfig") < 0) {
2668 |_______|_______panic("failed to load misc/pci_autoconfig");
2669 |_______}
2670
2671 |_______if (modload("drv", "isa") < 0)
2672 |_______|_______panic("failed to load drv/isa");
2673 #endif
2674
2675 |_______probe = bus_probes;
2676 |_______while (probe) {
2677 |_______|_______/* run the probe functions */
2678 |_______|_______(*probe->probe)(0);
2679 |_______|_______probe = probe->next;
2680 |_______}
2681 }

Second, let's track the second pass of bus probe.

main()
start_up()
startup_end()
configure()
impl_bus_reprobe()

i86pc/os/ddi_impl.c
2683 /*
2684 * impl_bus_reprobe
2685 *|_____Reprogram devices not set up by firmware.
2686 */
2687 static void
2688 impl_bus_reprobe(void)
2689 {
2690 |_______struct bus_probe *probe;
2691
2692 |_______probe = bus_probes;
2693 |_______while (probe) {
2694 |_______|_______/* run the probe function */
2695 |_______|_______(*probe->probe)(1);
2696 |_______|_______probe = probe->next;
2697 |_______}
2698 }

Okay, before go ahead with checking out what happens during these two points. Let me try to answer two questions in my mind.

(i) How are the drivers bound to the specific device node?
For more detailed description, please refer to "Binding a Driver to a Device" in document of "Writing Device Driver".

http://docs.sun.com/app/docs/doc/819-3196/kernelovr-14?a=view

For short, copy and paste some words below.

Each device node has an associated name property. This property can be assigned either from an external agent, such as the PROM, during system boot or from a driver.conf configuration file. In any case, the name property represents the node name assigned to a device in the device tree. The node name is the name visible in /devices and listed in the prtconf(1M) output.

A device node can have an associated compatible property as well. The compatible property contains an ordered list of one or more possible driver names or driver aliases for the device.

The system uses both the compatible and the name properties to select a driver for the device. The system first attempts to match the contents of the compatible property, if the compatible property exists, to a driver on the system. Beginning with the first driver name on the compatible property list, the system attempts to match the driver name to a known driver on the system. Each entry on the list is processed until the system either finds a match or reaches the end of the list.

If the contents of either the name property or the compatible property match a driver on the system, then that driver is bound to the device node. If no match is found, no driver is bound to the device node.

(ii) After ndi_devi_bind_driver() is called, when will the actual driver module loaded, hence attach() method gets called?
It depends. Some device driver modules are loaded during boot via function such as ndi_devi_online(). Some device node remains BOUND status untill the driver is really needed by the any application. Say, the device is about to be open by an application.

Okay, the following things are done between the two passes of pci_enumerate() (actually for two passes of each bus prober, such as PCI and acpidev)
startup_dodules()==>
( 1) DDI instance intialization
( 2) DDI callback intialization
( 3) Allocate and initialize log_event data structures
( 4) FM initialization
( 5) Initialize IRM subsystem before any drivers are attached
( 6) Load driver.conf file for all major's
( 7) LDI initialization
( 8) Initialize the overall cache file management
( 9) Read cache files
(10) Set up the CPU module subsystem for the boot cpu in the native case, and all physical cpu resource in the xpv dom0 case
(11) Fake a prom tree such that /dev/openprom continues to work
(12) Load all platform specific modules. Retrieve the platform specified modules from /etc/mach and load each of them
[allen@blu-wbg:~]cat /etc/mach
... <legal statement> ...
pcplusmp
xpv_psm
startup_end()==>
(13) Perform tasks that get done after most of the VM initialization has been done but before the clock and other devices get started
(14) Perform CPC initialization for this CPU
(15) Initialize cpu event framework
(16) If tod_module_name is set in /etc/system, load TOD module so that ddi_get_time(9F) etc work
(17) Load "xpvtod" module for xpv kernel
(18) Call configure() to configure the system
(19) Check for disabled drivers and initialize root node
(20) Reprogram devices not set up by firmware (BIOS). The second pass of bus enumeration happens here.

As we could see system keep initializing itself during the two passes of bus probe. Actually, the first pass is a PROM emulator. It does what PROM should do on an IEEE Std 1275 compatible platform. The second pass is called in configure() to reprogram/configure the devices in the system. "pci_reprogram()" is called in the second pass of pci_enumerate(). Let dive into pci_reprogram() and see what's been done in that.

pci_reprogram()
===============
(a) Scan ACPI namespace for _BBN objects, make sure that childless root-bridges appear in devinfo tree.

intel/io/pci/pci_boot.c
238 /*
239 * Scan the ACPI namespace for all top-level instances of _BBN
240 * in order to discover childless root-bridges (which enumeration
241 * may not find; root-bridges are inferred by the existence of
242 * children). This scan should find all root-bridges that have
243 * been enumerated, and any childless root-bridges not enumerated.
244 * Root-bridge for bus 0 may not have a _BBN object.
245 */
246 static void
247 pci_scan_bbn()
248 {
249 |_______void *rv;
250
251 |_______(void) AcpiGetDevices(NULL, pci_process_acpi_device, NULL, &rv);
252 }

Walk the objects in ACPI name space and call pci_process_acpi_device() with "NULL" as parameter. The return value of AcpiGetDevices() is returned in "rv" parameter. If a PCI with _BBN found,

225 |_______if (ACPI_SUCCESS(AcpiEvaluateObjectTyped(hdl, "_BBN",
226 |_______ NULL, &rb, ACPI_TYPE_INTEGER))) {
227 |_______|_______/* PCI with _BBN, process it, go no deeper */
228 |_______|_______if (pci_bus_res[ro.Integer.Value].par_bus == (uchar_t)-1 &&
229 |_______|_______ pci_bus_res[ro.Integer.Value].dip == NULL)
230 |_______|_______|_______create_root_bus_dip((uchar_t)ro.Integer.Value);
231 |_______|_______return (AE_CTRL_DEPTH);
232 |_______}

To be continued ...