When Apple announced the UNIX based Mac OS X lots of sysadmins took notice. For years the Linux crowd had been trying, and failing, to get UNIX to the general desktop audience. I ran Linux for a while on Beige G3 and it required a trip to Open Firmware and a bootconf loader on a floppy to get it going at the time. When I installed my first Rhapsody build, all I had to do was put the Caps Lock key on to get all the UNIXy goodness. The times, they were a changing.
Fast forward several years and Apple is now the highest volume UNIX vendor in the world. Furthermore the base of Mac OS X is open source and Apple uses many OSS projects in Darwin. Apple also has created projects and turned them out into the OSS world. (I'm well aware of the issues that some in the community have with Apple's OSS participation level, it's just not the focus of this article.) Recently Apple has loosed a new beast onto the world and it's knocked many people for a loop.
Welcome, launchd. Seriously.
Read on for more...
Apple's position as the leading UNIX vendor, and as a unified solution provider, gives them some unique opportunities. One of these has been to take a look at some of the traditional mishmashes of UNIX and try to sort them out a bit. Lookupd is one of these efforts and it provides general resolver services for everything from DNS to GID lookups on Mac OS X. Applications don't need to know anything about how the machine is configured, they just ask lookupd for the info. If you look at Mac OS X, and OpenStep before it, you will see that the OS has been striving for this sort of consolidation of services for a while now.
With Tiger, Apple has started to tackle the system of files and processes that start a UNIX system. Before it was often hard to tell how everything got cranked up. Was a daemon started by SystemStarter? Was it started by xinetd? Was it started by RC? Was it started by cron? Was it started by init or mach_init? Was it started by watchdog? Even worse, what if you needed to add a service? It was fairly clear that SystemStarter was what Apple wanted for startup items, but what about launch on demand? Xientd seems obvious here, but not everything can be launched by xinetd. What about service control? How do you easily start and stop services so that the rest of the system is aware of them?
In all honesty, this is a problem that Microsoft had figured out a while ago. There is nothing in Panther and earlier that compares to the Services control panel, or MMC snap-in, on Windows. Mac OS X still doesn't have that easy level of control, but the foundations are now there with launchd and its buddy launchctl. (Incidentally, this is a great opportunity for an industrious AppleScripter to whip out a Studio app.)
Meet the team
When Apple decided to overhaul the 30 year old system for starting and maintaining processes on UNIX it had to think big. launchd can do the jobs of init, mach_init, xinetd, RC, SystemStarter, watchdog, and cron. It is Apple's stated intention that they want to eliminate all of those services in favor of launchd at some point in the future. There are two main programs in the launchd system, launchd and launchctl. Briefly:
launchd manages the daemons at both a system and user level. Similar to xinetd, launchd can start daemons on demand. Similar to watchdogd, launchd can monitor daemons to make sure that they keep running. launchd also has replaced init as PID 1 on Mac OS X and as a result it is responsible for starting the system at boot time.
launchctl is our window into the world of launchd. Using launchctl we can load and unload daemons, start and stop launchd controlled jobs, get system utilization statistics for launchd and its child processes, and set environment settings.
A third, and the most confusing, part of the system are the plist files that define the services to be loaded into launchd with launchctl. Stored in the LaunchAgents or LaunchDaemons of any Library, the XML files have around thirty different keys that can be set, and the syntax is not very pretty. All of the options are laid out in the launchd.plist man page, which hides them nicely by using an obscure name. Even after reading the substantial man page you still have no real clear idea of how to use the keys to construct a valid plist, and that's where thieving from the default Apple plists comes in handy.
Time to dig in...
launchd
When it comes down to it, launchd has two main tasks. The first is to boot the system and the second is to load and maintain services. Let's take a look at the boot part of the job first.
When Mac OS X boots it goes through quite a few steps. Here is a very simplified view of the 10.4 system startup.
1. The BootROM activates, does its thing to the hardware, and then loads BootX.
2. BootX loads the kernel, spins the gear, loads any needed kexts, and then the kernel loads launchd.
3. launchd then runs /etc/rc, scans through /System/Library/LaunchDaemons and /Library/LaunchDaemons and acts on the plists as needed, and finally starts the loginwindow.
Easy huh?
The first one we should look at here is step 2. This is a huge change for Mac OS X, and *NIX in general, in that some flavor of init is not loaded here. Indeed, launchd is now PID 1 on Mac OS X.
In step three, launchd scans through a few different directories for jobs to run. There are two different folders types that are scanned. The LaunchDaemons folders should contain items that will run as root, generally background processes. The LaunchAgents folders should contain jobs, called agent applications, that will run as a user or in the context of userland. Often these can be scripts, other foreground items, and they can even include a user interface. When we get to our example we will be creating a user-specific LaunchAgent job. These directories are all kept in the typical Libraries of Mac OS X.
launchd is very different from SystemStarter in that it may not actually launch all the daemons at boot time. Key to launchd, and similar to xinted, is the idea of launch on demand daemons. When launchd scans through the job plists at boot time it reserves and listens on all of the ports requested by those jobs. If so indicated in the plist by the "OnDemand" key, the daemon is not actually loaded at the time. Rather launchd will listen on the port, start the daemon when needed, and shut it down when it is not. After a daemon is loaded, launchd will keep track of it and make sure it is running if needed. In this way it is like watchdogd, and shares watchdogd's requirement that processes do not attempt to fork or daemonize on their own. If a process goes into the background launchd will lose track of it and attempt to relaunch it.
This is why Tiger boots so fast. The system only has to register the daemons that are to run, not actually launch them. In fact the progress bar that appears is just a placebo that doesn't really show anything other than the passage of time. This explains why the bar never reaches the end before the login window appears sometimes. (For fun you can run it any time you like by executing /usr/libexec/WaitingForLoginWindow.) In addition to incoming requests, there are other ways to define launch on demand and we will take a look at them in just a few moments.
The hardest part to manage during a launchd boot is dependancies. SystemStarter had a very simple system of dependancies that used the "Uses", "Requires", and "Provides" keys in the plist of a startup item. There are two main strategies when creating launch dependancies on 10.4. You can use IPC which will allow the daemons to talk amongst themselves to work it out or you can watch files or paths for changes. Using IPC is much more subtle than the SystemStarter's keys and requires more work from the developer, but it should lead to cleaner and quicker startups. It does however, put dependancies out of the reach of many admins, myself included, as we aren't actually creating daemons. If you have a timed script that needs to be run you can still use a SystemStarter item at this time. Don't get too comfortable with that though as it has been deprecated in 10.4
plist Time
When launchd scans a folder, or a job is submitted with launchctl, it depends on a properly formatted plist file that describes the job to be run. This is, for most sysadmins, is the most difficult area of launchd to grasp. SystemStarter used simple shell scripts to launch daemons. launchd requires the admin to be able to properly format an XML file and while not exceedingly difficult, it can be confusing.
The first stop you should make is the man page for launchd.plist. Give it a good read and don't worry if it makes your eyes bug out. It does that to everyone at first. At the very end it gives an example of a plist for your perusal. This example is very simple but it breaks down some of the keys you will become familiar with. At a minimum there are two required launchd.plist keys. "Label" is the string that defines how launchd will refer to the job, and "ProgramArguments" breaks down how to execute your program. That's it, but it's not enough to generate a useful job. Really we want to toss a few more keys in there to add to the flexibility of the job and to make it more efficient.
Some other useful keys:
Phew! That's not all of them, but I think that these represent a useful selection. Read the man page for all of them and the options to be used. Now that we know about some keys, lets do something neat with them.
In our example were are going to create a very simple user agent application that moves files and folders from one directory to another. It's a very basic sort of idea, but it's also a good building block for more complex file processing.
First we need to come up with the script that our launchd job will execute. For our purposes something simple like:
#!/bin/bash
mv /Users/josh/in/* /Users/josh/out/
exit 0
Now we need to create our launchd job plist. First create ~/Library/LaunchAgents, then go find a launchd plist file in /System/Library/LaunchDeamons that seems pretty similar to what you want to do. For this example I stole the cron plist and copied it to my new LaunchAgents folder. Fire up your editor of choice and start making changes.
In the end we should have something that looks a bit like this:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.afp548.mover</string>
<key>OnDemand</key>
<true/>
<key>Program</key>
<string>/Users/josh/mover</string>
<key>ProgramArguments</key>
<array>
<string>mover</string>
</array>
<key>WatchPaths</key>
<array>
<string>/Users/josh/in</string>
</array>
</dict>
</plist>
Note that in this example I also added a ProgramArguments key. I did this as it makes the job more flexible for future changes. Something to keep in mind with this key is that each flag or argument to be passed to the program must be in its own string of the array. The org.postfix.master.plist file is a good example of this. You can't simply say:
<key>Program</key>
<string>/usr/libexec/postfix/master -e 60</string>
<key>Program</key>
<string>/usr/libexec/postfix/master</string>
<key>ProgramArguments</key>
<array>
<string>master</string>
<string>-e</string>
<string>60</string>
</array>
<key>StartCalendarInterval</key>
<dict>
<key>Hour</key>
<integer>3</integer>
<key>Minute</key>
<integer>15</integer>
</dict>
If all of this seems a bit scary still, there is a nice shareware GUI out there that can handle the file generation for you. Take a look at the Launchd Editor from codepoetry to see what it can do.
Now that we have our executable and plist setup how do we load our new job into launchd? We could logout and back in, but that's a bit intrusive to just fire up a new service. This is where launchd's buddy, launchctl comes into play.
Enter launchctl
One of the big problems before launchd was that there was no easy or consistent way to control system services. SystemStarter was almost cool, but it never seemed to work properly in most cases for controlling services after startup. The problem with the other facilities for service control was that they are strewn across the OS with no central way to manage them. launchctl is Apple's way of fixing this.
On its own, launchctl can take commands from the command line, from standard in, or operate in interactive mode. If you come up with a set of commands that you want to make permanent you can store them in ~/.launchd.conf or /etc/launchd.conf. By using these files you can setup the environment that launchd operates in at either a user or global level. You can use sudo launchctl to make easy changes on a global scale, something that you couldn't do on 10.3 and earlier.
Let's start out with something simple though, loading our job. The syntax is basic, in our case:
launchctl load ~/Library/LaunchAgents/com.afp548.mover.plist
will load our job. We can confirm it's loaded with:
launchctl list
which will show us the launchd jobs that our user has submitted. If you want to see the system job list simply run the same command with sudo.
Now that our job is loaded, try dropping something in your "in" folder. If all is well it will vanish and then appear in your "out" folder in a manner similar to this spiffy movie (H264 codec required). For what it is worth, the folder in the movie has a bunch of loose files in it and it's running on a 400 MHz G4. As you see the response time of launchd is great. Again this is a very simple example, but it's useful and is a great launching pad for bigger tasks. For example launchd could watch a drop folder for graphic files that it processes with sips and then ftp transfers to a web server. The options are practically endless.
To remove a job we use the inverse of the load command...
launchctl unload -w ~/Library/LaunchAgents/com.afp548.mover.plist
Notice that I added the optional "-w" flag this time. This flag adds or removes the "disabled" key from the plist at the appropriate time. If you don't use this when unloading a job it will automatically load the next time you login or reboot, depending on where your job file is located. You use the same "-w" flag when loading the job to remove the disabled key automatically.
Just like lookupd, you can also use launchctl in interactive mode. Simply run it without any arguments and you will be delivered to the launchd prompt. Type "help" to see the list of commands that you can issue. They are the same as if you were calling them directly, but now you don't need to keep typing launchctl over and over again. A standard ctrl-d will escape the launchd shell.
If you want to view the resource usage of any particular job, or launchd as a whole, you can use the "getrusage" command. When using this command you need to specify if you are interested in launchd itself or its children. Again, you use sudo to see the global resource usage. So if I wanted to see the resource usage of all of launchd's kids:
sudo launchctl getrusage children
is all I need. If you change our demo job plist to use a QueueDirectories key rather than the WatchPaths one there is a very high probability that a race condition will appear. This can be really bad as it can run wild and suck up all the CPU time on your system. Which leads us right into our next launchctl subcommand.
Using the "limit" command we can view and set limits on the launchd environment. With no resource specified this displays three columns representing the resources, their soft limits, and finally, their hard limits. Again you can use sudo to define or view these for the system as a whole. The limits you set with launchctl are the same limits you can set in a job plist, but they apply to the entire launchd environment rather than a single job. There are details in the launchd.plist or setrlimit man pages, but I'll list them briefly here:
To set these limits simply use the limit command but add a resource type and name a limit. If you only enter one number it will be used for both the soft and hard limits. If you want them to be different you need to enter both the soft and hard limits in order.
launchctl limit maxfiles 256 512
Will set the maxfiles limit to 512, while retaining the default soft limit of 256 files.
There are other similar launchctl commands that we can use to set logging levels, change the stdout and stderr of launchd, set environment variables, or change the umask of launchd. For example if I wanted to redirect the stdout of just my launchd jobs to a file I could say:
launchctl stdout /Users/josh/Library/Logs/launchd.out
It is important to remember that these commands are job persistant, but not launchd persistent. Meaning that the general launchd settings will not survive a reboot. This is easy to fix by dropping a conf file in the appropriate place. For global settings you should use /etc/launchd.conf and for user settings you should use ~/.launchd.conf. Just enter your launchctl commands, without the launchctl part, in the file one line at a time. If I wanted to combine my two examples above into a permanent setting I would put:
# This is my launchd.conf file.
limit maxfiles 256 512<br>
stdout /Users/josh/Library/Logs/launchd.out