gaoxiangnumber1

15-Interprocess Communication

Please indicate the source: http://blog.csdn.net/gaoxiangnumber1

Welcome to my github: https://github.com/gaoxiangnumber1

15.1 Introduction

Figure 15.1 summarizes the various forms of IPC that are supported by the four systems discussed in this text.
For full-duplex pipes, if the feature can be provided through UNIX domain sockets(Section 17.2), we show “UDS” in the column.
The first ten forms of IPC in Figure 15.1 are restricted to IPC between processes on the same host. The final two rows(sockets and STREAMS) are the only two forms that are supported for IPC between processes on different hosts.

15.2 Pipes

Pipes have two limitations.
1. They are half duplex(i.e., data flows in only one direction).
2. Pipes can be used only between processes that have a common ancestor. A pipe is created by a process, that process calls fork, and the pipe is used between the parent and the child.
FIFOs(Section 15.5) get around the second limitation, and UNIX domain sockets(Section 17.2) get around both limitations.
Every time you type a sequence of commands in a pipeline for the shell to execute, the shell creates a separate process for each command and links the standard output of one process to the standard input of the next using a pipe.

#include <unistd.h>
int pipe(int fd[2]);
Returns: 0 if OK, -1 on error

A pipe is created by calling the pipe function. Two file descriptors are returned through the fd argument: fd[0] is open for reading, and fd[1] is open for writing. The output of fd[1] is the input for fd[0].
POSIX.1 allows for systems to support full-duplex pipes. For these implementations, fd[0] and fd[1] are open for both reading and writing.
Two ways to picture a half-duplex pipe are shown in Figure 15.2. The left half shows the two ends of the pipe connected in a single process. The right half emphasizes that the data in the pipe flows through the kernel.

The fstat function(Section 4.2) returns a file type of FIFO for the file descriptor of either end of a pipe. We can test for a pipe with the S_ISFIFO macro.
Normally, the process that calls pipe then calls fork, creating an IPC channel from the parent to the child, or vice versa. Figure 15.3 shows this scenario.

For a pipe from the parent to the child, the parent closes the read end of the pipe(fd[0]), and the child closes the write end(fd[1]). Figure 15.4 shows the resulting arrangement of descriptors.(For a pipe from the child to the parent, the parent closes fd[1], and the child closes fd[0].)

When one end of a pipe is closed, two rules apply.
1. If we read from a pipe whose write end has been closed, read returns 0 to indicate an end of file after all the data has been read.
2. If we write to a pipe whose read end has been closed, the SIGPIPE signal is generated. If we either ignore the signal or catch it and return from the signal handler, write returns -1 with errno set to EPIPE.
When we’re writing to a pipe(or FIFO), the constant PIPE_BUF specifies the kernel’s pipe buffer size. A write of PIPE_BUF bytes or less will not be interleaved with the writes from other processes to the same pipe(or FIFO). But if multiple processes are writing to a pipe(or FIFO), and if we write more than PIPE_BUF bytes, the data might be interleaved with the data from the other writers. We can determine the value of PIPE_BUF by using pathconf or fpathconf(Figure 2.12).
Figure 15.5 shows the code to create a pipe between a parent and its child and to send data down the pipe.

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>

#define MAX 100

void Exit(char *string)
{
    printf("%s\n", string);
    exit(1);
}

int main()
{
    int fd[2];
    pid_t pid;
    char line[MAX];
    int nread;

    if(pipe(fd) < 0)
    {
        Exit("pipe error");
    }
    if((pid = fork()) < 0)
    {
        Exit("fork error");
    }
    else if(pid > 0)
    {
        close(fd[0]);
        write(fd[1], "gaoxiangnumber1\n", 16);
    }
    else
    {
        close(fd[1]);
        nread = read(fd[0], line, MAX);
        write(STDOUT_FILENO, line, nread);
    }

    exit(0);
}

What’s interesting is to duplicate the pipe descriptors onto standard input or standard output. The child then runs some other program, and that program can either read from its standard input(the pipe that we created) or write to its standard output(the pipe).
Consider a program that displays output that it has created, one page at a time. To avoid writing all the data to a temporary file and calling system to display that file, we want to pipe the output directly to the pager. So, we create a pipe, fork a child process, set up the child’s standard input to be the read end of the pipe, and exec the user’s pager program. Figure 15.6: This example takes a command-line argument to specify the name of a file to display.

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/wait.h>
#include <string.h>

#define MAX 1000
#define DEF_PAGER "/bin/more"

void Exit(char *string)
{
    printf("%s\n", string);
    exit(1);
}

int main(int argc, char **argv)
{
    int nread;
    char line[MAX];
    int fd[2];
    pid_t pid;
    char *pager, *argv0;
    FILE *fp;

    if(argc != 2)
    {
        Exit("Usage: ./a.out pathname");
    }

    if((fp = fopen(argv[1], "r")) == NULL)
    {
        Exit("Can't open this file");
    }
    if(pipe(fd) < 0)
    {
        Exit("pipe error");
    }

    if((pid = fork()) < 0)
    {
        Exit("fork error");
    }
    else if(pid > 0)
    {
        close(fd[0]);
        while(fgets(line, MAX, fp) != NULL)
        {
            nread = strlen(line);
            if(write(fd[1], line, nread) != nread)
            {
                Exit("write error to pipe");
            }
        }
        if(ferror(fp))
        {
            Exit("fgets error");
        }
        close(fd[1]);

        if(waitpid(pid, NULL, 0) < 0)
        {
            Exit("waitpid error");
        }
        exit(0);
    }
    else
    {
        close(fd[1]);
        if(fd[0] != STDIN_FILENO)
        {
            if(dup2(fd[0], STDIN_FILENO) != STDIN_FILENO)
            {
                Exit("dup2 error to stdin");
            }
            close(fd[0]);
        }

        if((pager = getenv("PAGER")) == NULL)
        {
            pager = DEF_PAGER;
        }
        if((argv0 = strrchr(pager, '/')) == NULL)
        {
            argv0++;
        }
        else
        {
            argv0 = pager;
        }

        if(execl(pager, argv0,(char *)0) < 0)
        {
            Exit("execl error for pager");
        }
    }

    exit(0);
}

Before calling fork, we create a pipe. After the fork, the parent closes its read end, and the child closes its write end. The child then calls dup2 to have its standard input be the read end of the pipe. When the pager program is executed, its standard input will be the read end of the pipe.
When we duplicate one descriptor onto another(fd[0] onto standard input in the child), we have to be careful that the descriptor doesn’t already have the desired value. If the descriptor already had the desired value and we called dup2 and close, the single copy of the descriptor would be closed.(Section 3.12: the operation of dup2 when its two arguments are equal.)
Figure 15.7 shows implementation of the five functions TELL_WAIT, TELL_PARENT, TELL_CHILD, WAIT_PARENT, and WAIT_CHILD using pipes.

We create two pipes before fork(Figure 15.8). The parent writes the character “p” across the top pipe when TELL_CHILD is called, and the child writes the character “c” across the bottom pipe when TELL_PARENT is called. The corresponding WAIT_xxx functions do a blocking read for the single character.

Each pipe has an extra reader which doesn’t matter. In addition to the child reading from pfd1[0], the parent has this end of the top pipe open for reading. This doesn’t affect us, since the parent doesn’t try to read from this pipe.

15.3 popen and pclose Functions

The popen and pclose functions handle all work: creating a pipe, forking a child, closing the unused ends of the pipe, executing a shell to run the command, and waiting for the command to terminate.

#include <stdio.h>
FILE *popen(const char *cmdstring, const char *type);
Returns: file pointer if OK, NULL on error
int pclose(FILE *fp);
Returns: termination status of cmdstring, or -1 on error

popen does a fork and exec to execute the cmdstring and returns a standard I/O file pointer.
If type is “r”, the file pointer is connected to the standard output of cmdstring(Figure 15.9).

pclose closes the standard I/O stream, waits for the command to terminate, and returns the termination status of the shell.(the termination status in Section 8.6. The system function in Section 8.13 also returns the termination status.) If the shell cannot be executed, the termination status returned by pclose is as if the shell had executed exit(127).
The cmdstring is executed by the Bourne shell, as in
sh -c cmdstring
The shell expands any of its special characters in cmdstring. This allows us:
fp = popen("ls *.c", "r"); or
fp = popen("cmd 2>&1", "r");
Figure 15.11: using popen to redo the program from Figure 15.6.

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>

#define MAX 1000
#define PAGER "${PAGER:-more}"

void Exit(char *string)
{
    printf("%s\n", string);
    exit(1);
}

int main(int argc, char **argv)
{
    if(argc != 2)
    {
        Exit("Usage: ./a.out pathname");
    }

    char line[MAX];
    FILE *fpin, *fpout;
    if((fpin = fopen(argv[1], "r")) == NULL)
    {
        Exit("Can't open file.");
    }
    if((fpout = popen(PAGER, "w")) == NULL)
    {
        Exit("popen error");
    }
    while(fgets(line, MAX, fpin) != NULL)
    {
        if(fputs(line, fpout) == EOF)
        {
            Exit("fputs error to pipe");
        }
    }
    if(ferror(fpin))
    {
        Exit("fgets error");
    }
    if(pclose(fpout) == -1)
    {
        Exit("pclose error");
    }

    exit(0);
}

The shell command ${PAGER:-more} says to use the value of the shell variable PAGER if it is defined and non-null; otherwise, use the string more.

#include <stdio.h>
#include <stdlib.h>

static pid_t *childpid = NULL;
static int maxfd;

void Exit(char *string)
{
    printf("%s\n", string);
    exit(1);
}

FILE *Popen(const char *cmdstring, const char *type)
{
    if((type[0] != 'r' && type[0] != 'w') || type[1] != 0)
    {
        errno = EINVAL; // Invalid argument
        return NULL;
    }

    if(childpid == NULL)
    {
        maxfd = open_max(); // Figure 2.17
        if((childpid = calloc(maxfd, sizeof(pid_t))) == NULL)
        {
            return NULL;
        }
    }

    int pfd[2];
    pid_t pid;
    FILE *fp;
    if(pipe(pfd) < 0)
    {
        return NULL;
    }
    if(pfd[0] >= maxfd || pfd[1] >= maxfd)
    {
        close(pfd[0]);
        close(pfd[1]);
        errno = EMFILE; // Too many open files
        return NULL;
    }

    if((pid = fork()) < 0)
    {
        return NULL;
    }
    else if(pid == 0)
    {
        if(*type == 'r')
        {
            close(pfd[0]);
            if(pfd[1] != STDOUT_FILENO)
            {
                dup2(pfd[1], STDOUT_FILENO);
                close(pfd[1]);
            }
        }
        else
        {
            close(pfd[1]);
            if(pfd[0] != STDIN_FILENO)
            {
                dup2(pfd[0], STDIN_FILENO);
                close(pfd[0]);
            }
        }
        for(int fd = 0; fd < maxfd; ++fd)
        {
            if(childpid[fd] > 0)
            {
                close(fd);
            }
        }
        execl("/bin/sh", "sh", "-c", cmdstring,(char *)0);
        _exit(127);
    }

    if(*type == 'r')
    {
        close(pfd[1]);
        if((fp = fdopen(pfd[0], type)) == NULL)
        {
            return NULL;
        }
    }
    else
    {
        close(pfd[0]);
        if((fp = fdopen(pfd[1], type)) == NULL)
        {
            return NULL;
        }
    }
    childpid[fileno(fp)] = pid;
    return fp;
}

int Pclose(FILE *fp)
{
    int fd, stat;
    pid_t pid;

    if(childpid == NULL)
    {
        errno = EINVAL;
        return -1;
    }

    fd = fileno(fp);
    if(fd >= maxfd)
    {
        errno = EINVAL;
        return -1;
    }
    if((pid = childpid[fd]) == 0)
    {
        errno = EINVAL;
        return -1;
    }

    childpid[fd] = 0;
    if(fclose(fp) == EOF)
    {
        return -1;
    }
    while(waitpid(pid, &stat, 0) < 0)
    {
        if(errno != EINTR)
        {
            return -1;
        }
    }
    return stat;
}

int main()
{

    exit(0);
}

Each time popen is called, we have to remember the process ID of the child that we create and either its file descriptor or FILE pointer. We choose to save the child’s process ID in the array childpid, which we index by the file descriptor. In this way, when pclose is called with the FILE pointer as its argument, we call the standard I/O function fileno to get the file descriptor and then have the child process ID for the call to waitpid. Since it’s possible for a given process to call popen more than once, we dynamically allocate the childpid array(the first time popen is called), with room for as many children as there are file descriptors.
Note that open_max function from Figure 2.17 can return a guess of the maximum number of open files if this value is indeterminate for the system. We need to be careful not to use a pipe file descriptor whose value is larger than(or equal to) what the open_max function returns.
In popen, if the value returned by open_max happens to be too small, we close the pipe file descriptors, set errno to EMFILE to indicate too many file descriptors are open, and return -1.
In pclose, if the file descriptor corresponding to the file pointer argument is larger than expected, we set errno to EINVAL and return -1.
POSIX.1 requires that popen close any streams that are still open in the child from previous calls to popen. To do this, we go through the childpid array in the child, closing any descriptors that are still open.
What happens if the caller of pclose has established a signal handler for SIGCHLD? The call to waitpid from pclose would return an error of EINTR. Since the caller is allowed to catch this signal(or any other signal that might interrupt the call to waitpid), we simply call waitpid again if it is interrupted by a caught signal.
If the application calls waitpid and obtains the exit status of the child created by popen, we will call waitpid when the application calls pclose, find that the child no longer exists, and return -1 with errno set to ECHILD. This is the behavior required by POSIX.1 in this situation.
popen should never be called by a set-user-ID or set-group-ID program. When it executes the command, popen does the equivalent of
execl("/bin/sh", "sh", "-c", command, NULL);
which executes the shell and command with the environment inherited by the caller. A malicious user can manipulate the environment so that the shell executes commands other than those intended, with the elevated permissions granted by the set-ID file mode.
One thing that popen is suited for is executing filters to transform the input or output of the running command. Consider an application that writes a prompt to standard output and reads a line from standard input. With popen, we can interpose a program between the application and its input to transform the input. Figure 15.13 shows the arrangement of processes in this situation.

Figure 15.14 shows a filter to demonstrate this operation. The filter copies standard input to standard output, converting any uppercase character to lowercase. The reason for fflush standard output after writing a newline is discussed in next section.

#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>

void Exit(char *string)
{
    printf("%s\n", string);
    exit(1);
}

int main()
{
    char ch;
    while((ch = getchar()) != EOF)
    {
        if(isupper(ch))
        {
            ch = tolower(ch);
        }
        if(putchar(ch) == EOF)
        {
            Exit("output error");
        }
        if(ch == '\n')
        {
            fflush(stdout);
        }
    }
    exit(0);
}

We compile this filter into the executable file filter, which we then invoke from the program in Figure 15.15 using popen.

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>

#define MAX 100

void Exit(char *string)
{
    printf("%s\n", string);
    exit(1);
}

int main()
{
    char line[MAX];
    FILE *fpin;

    if((fpin = popen("/home/xiang/Gao/Notes/OS/APUE/Codes/filter", "r")) == NULL)
    {
        Exit("popen error");
    }
    for(;;)
    {
        fputs("$ ", stdout);
        fflush(stdout);
        if(fgets(line, MAX, fpin) == NULL)
        {
            break;
        }
        if(fputs(line, stdout) == EOF)
        {
            Exit("fputs error to pipe");
        }
    }
    if(pclose(fpin) == -1)
    {
        Exit("pclose error");
    }
    putchar('\n');
    exit(0);
}

We need to call fflush after writing the prompt, because the standard output is normally line buffered, and the prompt does not contain a newline.

15.4 Coprocesses

A UNIX system filter is a program that reads from standard input and writes to standard output. A filter becomes a coprocess when the same program generates the filter’s input and reads the filter’s output.
A coprocess normally runs in the background from a shell, and its standard input and standard output are connected to another program using a pipe.
popen gives us a one-way pipe to the standard input or from the standard output of another process, with a coprocess we have two one-way pipes to the other process: one to its standard input and one from its standard output. We want to write to its standard input, let it operate on the data, and then read from its standard output.
Figure 15.16. The process creates two pipes: one is the standard input of the coprocess and the other is the standard output of the coprocess.

The program in Figure 15.17 is a coprocess that reads two numbers from its standard input, computes their sum, and writes the sum to its standard output.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#define MAX 100

void Exit(char *string)
{
    printf("%s\n", string);
    exit(1);
}

int main()
{
    int nread, num1, num2;
    char line[MAX];
    while((nread = read(STDIN_FILENO, line, MAX)) > 0)
    {
        line[nread] = 0;
        if(sscanf(line, "%d%d", &num1, &num2) == 2)
        {
            sprintf(line, "%d\n", num1 + num2);
            int length = strlen(line);
            if(write(STDOUT_FILENO, line, length) != length)
            {
                Exit("write error");
            }
        }
        else
        {
            if(write(STDOUT_FILENO, "invalid args\n", 13) != 13)
            {
                Exit("write error");
            }
        }
    }

    exit(0);
}

We compile this program to the executable file add2. The program in Figure 15.18 invokes the add2 coprocess after reading two numbers from its standard input. The value from the coprocess is written to its standard output.

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <unistd.h>
#include <string.h>

#define MAX 1000

void Exit(char *string)
{
    printf("%s\n", string);
    exit(1);
}

static void sig_pipe(int signo)
{
    printf("Catch SIGPIPE signal\n");
    exit(1);
}

int main()
{
    int nread, fd1[2], fd2[2];
    pid_t pid;
    char line[MAX];

    if(signal(SIGPIPE, sig_pipe) == SIG_ERR)
    {
        Exit("signal error");
    }
    if(pipe(fd1) < 0 || pipe(fd2) < 0)
    {
        Exit("pipe error");
    }

    if((pid = fork()) < 0)
    {
        Exit("foek error");
    }
    else if(pid > 0)
    {
        close(fd1[0]);
        close(fd2[1]);
        while(fgets(line, MAX, stdin) != NULL)
        {
            nread = strlen(line);
            if(write(fd1[1], line, nread) != nread)
            {
                Exit("write error to pipe");
            }
            if((nread = read(fd2[0], line, MAX)) < 0)
            {
                Exit("read error form pipe");
            }
            if(nread == 0)
            {
                printf("child closed pipe\n");
                break;
            }
            line[nread] = 0;
            if(fputs(line, stdout) == EOF)
            {
                Exit("fputs error");
            }
        }
        if(ferror(stdin))
        {
            Exit("fgets error on stdin");
        }
        exit(0);
    }
    else
    {
        close(fd1[1]);
        close(fd2[0]);
        if(fd1[0] != STDIN_FILENO)
        {
            if(dup2(fd1[0], STDIN_FILENO) != STDIN_FILENO)
            {
                Exit("dup2 error to stdin");
            }
            close(fd1[0]);
        }
        if(fd2[1] != STDOUT_FILENO)
        {
            if(dup2(fd2[1], STDOUT_FILENO) != STDOUT_FILENO)
            {
                Exit("dup2 error to stdout");
            }
            close(fd2[1]);
        }
        if(execl("./add2", "add2",(char *)0) < 0)
        {
            Exit("execl error");
        }
    }

    exit(0);
}

We create two pipes, with the parent and the child closing the ends they don’t need. We use two pipes: one for the standard input of the coprocess and one for its standard output. The child then calls dup2 to move the pipe descriptors onto its standard input and standard output, before calling execl.
If we kill the add2 coprocess while the program in Figure 15.18 is waiting for our input and then enter two numbers, the signal handler is invoked when the program writes to the pipe that has no reader.(Exercise 15.4.)
In the coprocess add2(Figure 15.17), we use UNIX I/O system calls: read and write. Figure 15.19 use standard I/O.

If we invoke this new coprocess from the program in Figure 15.18, it no longer works. The problem is the default standard I/O buffering. When the program in Figure 15.19 is invoked, the first fgets on the standard input causes the standard I/O library to allocate a buffer and choose the type of buffering. Since the standard input is a pipe, the standard I/O library defaults to fully buffered. The same thing happens with the standard output. While add2 is blocked reading from its standard input, the program in Figure 15.18 is blocked reading from the pipe. We have a deadlock.
We can change the program in Figure 15.19 by adding the following four lines before the while loop. These lines cause fgets to return when a line is available and cause printf to do an fflush when a newline is output(Section 5.4).

if(setvbuf(stdin, NULL, _IOLBF, 0) != 0)
    err_sys("setvbuf error");
if(setvbuf(stdout, NULL, _IOLBF, 0) != 0)
    err_sys("setvbuf error");

If we aren’t able to modify the program that we’re piping the output into, other techniques are required. E.g., if we use awk(1) as a coprocess from our program, the following won’t work:

#! /bin/awk -f
{ print $1 + $2 }

The reason this won’t work is the standard I/O buffering, but we cannot modify the executable of awk in any way to change the way the standard I/O buffering is handled.
The solution is to make the coprocess being invoked(awk in this case) think that its standard input and standard output are connected to a terminal. That causes the standard I/O routines in the coprocess to line buffer these two I/O streams, similar to what we did with the explicit calls to setvbuf previously.

15.5 FIFOs

FIFOs are also called named pipes. Unnamed pipes can be used only between related processes when a common ancestor has created the pipe. With FIFOs, unrelated processes can exchange data.
Chapter 4: A FIFO is a type of file. One of the encodings of the st_mode member of the stat structure(Section 4.2) indicates that a file is a FIFO. We can test for this with the S_ISFIFO macro.

#include <sys/stat.h>
int mkfifo(const char *path, mode_t mode);
int mkfifoat(int fd, const char *path, mode_t mode);
Both return: 0 if OK, −1 on error

The pathname for a FIFO exists in the file system.
The specification of the mode argument is the same as for the open function(Section 3.3).
The rules for the user and group ownership of the new FIFO are the same as in Section 4.6.
The mkfifoat function can be used to create a FIFO in a location relative to the directory represented by the fd file descriptor argument. There are three cases:
1. path is absolute: fd is ignored and mkfifoat behaves like mkfifo.
2. path is relative and fd is a valid file descriptor for an open directory: the pathname is evaluated relative to this directory.
3. path is relative and fd = AT_FDCWD: the pathname is evaluated starting in the current working directory.
After we use mkfifo/mkfifoat to create a FIFO, we use normal file I/O functions(open, close, read, write, unlink) to work with FIFOs.
Linux support the mkfifo(1) command and we can create a FIFO using a shell command and then access it with the normal shell I/O redirection. When we open a FIFO, the nonblocking flag(O_NONBLOCK) affects what happens.
1. Without O_NONBLOCK: an open for read-only blocks until some other process opens the FIFO for writing; an open for write-only blocks until some other process opens the FIFO for reading.
2. With O_NONBLOCK: an open for read-only returns immediately; an open for write-only returns -1 with errno = ENXIO if no process has the FIFO open for reading.
As with pipes, if we write to a FIFO that no process has open for reading, the SIGPIPE signal is generated. When the last writer for a FIFO closes the FIFO, an end of file is generated for the reader of the FIFO.
It is common to have multiple writers for a given FIFO and we must concern atomic writes if we don’t want the writes from multiple processes to be interleaved. As with pipes, the constant PIPE_BUF specifies the maximum amount of data that can be written atomically to a FIFO.
Two uses for FIFOs.
1. Used by shell commands to pass data from one shell pipeline to another without creating intermediate temporary files.
2. Used as rendezvous points in client-server applications to pass data between the clients and the servers.

Using FIFOs to Duplicate Output Streams

FIFOs can be used to duplicate an output stream in a series of shell commands. This prevents writing the data to an intermediate disk file, like using pipes to avoid intermediate disk files. But pipes can be used only for linear connections between processes, a FIFO has a name and it can be used for nonlinear connections.
Consider a procedure that needs to process a filtered input stream twice.

With a FIFO and the UNIX program tee(1), we can accomplish this procedure without using a temporary file. The tee program copies its standard input to both its standard output and the file named on its command line.

mkfifo fifo1
prog3 < fifo1 &
prog1 < infile | tee fifo1 | prog2

We create the FIFO and then start prog3 in the background, reading from the FIFO. We then start prog1 and use tee to send its input to both the FIFO and prog2. Figure 15.21 shows the process arrangement.

Client-Server Communication Using a FIFO

Another use for FIFOs is to send data between a client and a server. If we have a server that is contacted by many clients, each client can write its request to a well-known FIFO(the pathname of the FIFO is known to all clients) that the server creates. Figure 15.22 shows this arrangement.

Since there are multiple writers for the FIFO, the requests sent by the clients to the server need to be less than PIPE_BUF bytes in size. This prevents any interleaving of the client writes.
Problem is how to send replies back from the server to each client. A single FIFO can’t be used, as the clients would never know when to read their response versus responses for other clients. Solution is for each client to send its process ID with the request. The server then creates a unique FIFO for each client, using a pathname based on the client’s process ID, shown in Figure 15.23.

This arrangement works, but the server can’t tell whether a client crashes. The server must catch SIGPIPE, since it’s possible for a client to send a request and terminate before reading the response, leaving the client-specific FIFO with one writer(the server) and no reader.
Figure 15.23: If the server opens its well-known FIFO read-only(since it only reads from it) each time the number of clients goes from 1 to 0, the server will read an end of file on the FIFO. To prevent the server from handling this case, a trick is to have the server open its well-known FIFO for read-write(Exercise 15.10.)

15.6 XSI IPC

15.6.1 Identifiers and Keys

Each IPC structure(message queue, semaphore, or shared memory segment) in the kernel is referred to by a non-negative integer identifier. To send a message to or fetch a message from IPC structure, all we need know is its identifier. When a given IPC structure is created and then removed, the identifier associated with that structure continually increases until it reaches the maximum positive value for an integer, and then wraps around to 0.
The identifier is an internal name for an IPC object. Cooperating processes need an external name to be able to rendezvous using the same IPC object. So, an IPC object is associated with a key that acts as an external name.
Whenever an IPC structure is being created, a key of type key_t(a long integer in

#include <sys/ipc.h>
key_t ftok(const char *path, int id);
Returns: key if OK,(key_t)-1 on error

path must refer to an existing file.
Only the lower 8 bits of id are used.
The key created by ftok is formed by taking parts of the st_dev and st_ino fields in the stat structure(Section 4.2) corresponding to the given pathname and combining them with the project ID. If two pathnames refer to two different files, then ftok usually returns two different keys for the two pathnames. But both i-node numbers and keys are stored in long integers, information loss can occur when creating a key, so, two different pathnames can generate the same key if the same project ID is used.
The three get functions(msgget, semget, and shmget) all have two similar arguments: a key and an integer flag. A new IPC structure is created if key is IPC_PRIVATE or key is not currently associated with an IPC structure of the particular type and the IPC_CREAT bit of flag is specified.
To reference an existing queue, key must equal the key that was specified when the queue was created, and IPC_CREAT must not be specified. It’s impossible to specify IPC_PRIVATE to reference an existing queue since this key value creates a new queue. To reference an existing queue that was created with a key of IPC_PRIVATE, we must know the associated identifier and then use that identifier in the other IPC calls, bypassing the get function.
If we want to create a new IPC structure, making sure that we don’t reference an existing one with the same identifier, we must specify a flag with both IPC_CREAT and IPC_EXCL bits set. Doing this causes an error return of EEXIST if the IPC structure already exists.

15.6.2 Permission Structure

Each IPC structure associates with an ipc_perm structure that defines the permissions and owner and includes at least the following members:

struct ipc_perm
{
    uid_t   uid;        /* owner’s effective user ID */
    gid_t   gid;        /* owner’s effective group ID */
    uid_t   cuid;   /* creator’s effective user ID */
    gid_t   cgid;   /* creator’s effective group ID */
    mode_t  mode;   /* access modes */
    ...
};

All the fields are initialized when the IPC structure is created. We can modify the uid, gid, and mode fields by calling msgctl, semctl, or shmctl later. To change these values, the calling process must be either the creator of the IPC structure or the superuser.
There is no execute permission for any of the IPC structures. Message queues and shared memory use the terms read and write, but semaphores use the terms read and alter. Figure 15.24:

15.6.3 Configuration Limits

You can display the IPC-related limits by running ipcs -l.

$ ipcs -l
------Shared Memory Limits --------
max number of segments = 4096
max seg size(kbytes) = 18014398509465599
max total shared memory(kbytes) = 18446744073642442748
min seg size(bytes) = 1

------Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

------Messages Limits --------
max queues system wide = 15825
max size of message(bytes) = 8192
default max size of queue(bytes) = 16384

15.6.4 Advantages and Disadvantages

First problem: The IPC structures are system-wide and do not have a reference count. E.g., if we create a message queue, place some messages on the queue, and then terminate, the message queue and its contents are not deleted. They remain in the system until specifically read or deleted by msgrcv() or msgctl(), by executing the ipcrm(1) command, or by the system being rebooted.
Pipe is completely removed when the last process to reference it terminates. For FIFO, though the name stays in the file system until explicitly removed, any data left in a FIFO is removed when the last process to reference the FIFO terminates.
Second problem: These IPC structures are not known by names in the file system and we can’t access them or modify their properties with the functions in Ch 3 and 4. Because these IPC don’t use file descriptors, we can’t use the multiplexed I/O functions(select and poll) with them which makes it harder to use more than one of these IPC structures at a time or to use any of these IPC structures with file or device I/O. E.g., we can’t have a server wait for a message to be placed on one of two message queues without some form of busy-wait loop.

Figure 15.25:
1. “Connectionless” means the ability to send a message without calling some form of open function first.
2. Since all these forms of IPC are restricted to a single host, all are reliable.
3. “Flow control” means that the sender is put to sleep if there is a shortage of system resources(buffers) or if the receiver can’t accept any more messages. When the flow control condition subsides(i.e., when there is room in the queue), the sender should automatically be awakened.

15.7 Message Queues

A message queue is a linked list of messages stored within the kernel and identified by a message queue identifier(queue ID).
Each queue has the following msqid_ds structure associated with it:

struct msqid_ds
{
    struct ipc_perm msg_perm;   /* Section 15.6.2 */
    msgqnum_t       msg_qnum;   /* # of messages on queue */
    msglen_t        msg_qbytes; /* max # of bytes on queue */
    pid_t           msg_lspid;  /* pid of last msgsnd() */
    pid_t           msg_lrpid;  /* pid of last msgrcv() */
    time_t          msg_stime;  /* last-msgsnd() time */
    time_t          msg_rtime;  /* last-msgrcv() time */
    time_t          msg_ctime;  /* last-change time */
    ...
};

This structure defines the current status of the queue. systems include additional fields not covered by the standard.

Figure 15.26 lists the system limits that affect message queues. “derived” means a limit is derived from other limits. E.g., the maximum number of messages is based on the maximum number of queues and the maximum amount of data allowed on the queues. The maximum number of queues is based on the amount of RAM installed in the system. The queue maximum byte size limit further limits the maximum size of a message to be placed on a queue.
1. New messages are added to the end of a queue by msgsnd. Every message has a positive long integer type field, a non-negative length, and the actual data bytes, all of them are specified to msgsnd when the message is added to a queue.
2. We can fetch messages based on type field from a queue by msgrcv.

#include <sys/msg.h>
int msgget(key_t key, int flag);
Returns: message queue ID if OK, -1 on error

msgget: create a new queue or open an existing queue.
When a new queue is created, the following members of the msqid_ds structure are initialized.
• ipc_perm is initialized as described in Section 15.6.2. The mode member of this structure is set to the corresponding permission bits of flag. These permissions are specified with the values from Figure 15.24.
• msg_qnum, msg_lspid, msg_lrpid, msg_stime, and msg_rtime are all set to 0.
• msg_ctime is set to the current time.
• msg_qbytes is set to the system limit.
On success, msgget returns the non-negative queue ID that is used with the other three message queue functions.

#include <sys/msg.h>
int msgctl(int msqid, int cmd, struct msqid_ds *buf);
Returns: 0 if OK, -1 on error

msgctl() and related functions for semaphores and shared memory(semctl and shmctl) are the ioctl-like functions for XSI IPC(i.e., the garbage-can functions).
cmd specifies the command to be performed on the queue specified by msqid.
1. IPC_STAT
  Fetch the msqid_ds structure for this queue and store it in the structure pointed to by buf.
2. IPC_SET
  Copy the following fields from the structure pointed to by buf to the msqid_ds structure associated with this queue: msg_perm.uid, msg_perm.gid, msg_perm.mode, and msg_qbytes.
  This command can be executed only by a process whose effective user ID equals msg_perm.cuid or msg_perm.uid or by a process with superuser privileges. Only the superuser can increase the value of msg_qbytes.
3. IPC_RMID
  Remove the message queue from the system and any data on the queue immediately. Any other process still using the message queue will get an error of EIDRM on its next attempted operation on the queue.
  This command can be executed only by a process whose effective user ID equals msg_perm.cuid or msg_perm.uid or by a process with superuser privileges.

#include <sys/msg.h>
int msgsnd(int msqid, const void *ptr, size_t nbytes, int flag);
Returns: 0 if OK, -1 on error

msgsnd: place data onto a message queue. Each message is composed of a positive long integer type field, a non-negative length(nbytes), and the data bytes(corresponding to the length). Messages are always placed at the end of the queue.
ptr points to a long integer that contains the positive integer message type, and it is immediately followed by the message data. There is no message data if nbytes is 0. If the largest message we send is 512 bytes, we can define the following structure: ptr is a pointer to a my_message structure. The message type can be used by the receiver to fetch messages in an order other than first in, first out.

struct my_message
{
    long type;      /* positive message type */
    char text[512]; /* message data of length nbytes */
};

flag: If the message queue is full, flag = IPC_NOWAIT causes msgsnd to return immediately with an error of EAGAIN; otherwise, we are blocked until there is room for the message, the queue is removed from the system(return error of EIDRM(identifier removed)), or a signal is caught and the signal handler returns(return error of EINTR).
Since a reference count is not maintained with each message queue, the removal of a queue generates errors on the next queue operation by processes still using the queue. Semaphores handle this removal in the same fashion. In contrast, when a file is removed, the file’s contents are not deleted until the last open descriptor for the file is closed.
When msgsnd returns successfully, the msqid_ds structure associated with the message queue is updated to indicate the process ID that made the call(msg_lspid), the time that the call was made(msg_stime), and one more message is on the queue(msg_qnum).

#include <sys/msg.h>
ssize_t msgrcv(int msqid, void *ptr, size_t nbytes, long type, int flag);
Returns: size of data portion of message if OK, -1 on error

msgrcv: retrieve messages from a queue.
ptr points to a long integer(where the message type of the returned message is stored) followed by a data buffer for the actual message data.
nbytes is the size of the data buffer. When the returned message is larger than nbytes and if the MSG_NOERROR bit in flag is
1. set: the message is truncated without any notification given to us.
2. not set: an error of E2BIG is returned and the message stays on the queue.
type specifies which message we want.
1. = 0: The first message on the queue is returned.
2. > 0: The first message on the queue whose message type equals type is returned.
3. < 0: The first message on the queue whose message type is the lowest value less than or equal to the absolute value of type is returned.
A nonzero type is used to read the messages in an order other than FIFO. E.g., type could be a priority value if the application assigns priorities to the messages.
Another use of type is to contain the process ID of the client if a single message queue is being used by multiple clients and a single server(if a process ID fits in a long integer).
When a message of the specified type is not available, flag
1. = IPC_NOWAIT: msgrcv return -1 with errno set to ENOMSG.
2. != IPC_NOWAIT: the operation blocks until a message of the specified type is available, the queue is removed from the system(return -1 with errno set to EIDRM), or a signal is caught and the signal handler returns(return -1 with errno set to EINTR).
When msgrcv succeeds, the kernel updates the msqid_ds structure associated with the message queue to indicate the caller’s process ID(msg_lrpid), the time of the call(msg_rtime), and that one less message is on the queue(msg_qnum).

Example-Timing Comparison of Message Queues and Full-Duplex Pipes

If we need a bidirectional flow of data between a client and a server, we can use message queues or full-duplex pipes(available through the UNIX domain sockets mechanism(Section 17.2)).

Figure 15.27 shows a timing comparison of message queues, full-duplex(STREAMS) pipes, and UNIX domain sockets on Solaris. The tests consisted of a program that created the IPC channel, called fork, and then sent about 200 megabytes of data from the parent to the child. The data was sent using 100,000 calls to msgsnd, with a message length of 2,000 bytes for the message queue, and 100,000 calls to write, with a length of 2,000 bytes for the full-duplex pipe and UNIX domain socket. The times are all in seconds.
Message queues are not much faster than other forms of IPC. When we consider the problems in using message queues(Section 15.6.4), we shouldn’t use them for new applications.

15.8 Semaphores

A semaphore is a counter used to provide access to a shared data object for multiple processes.
To obtain a shared resource, a process needs to do the following:
1. Test the semaphore that controls the resource.
2. If the value of the semaphore is positive, the process can use the resource. The process decrements the semaphore value by 1, indicating that it has used one unit of the resource. When the process is done with the resource, the semaphore value is incremented by 1. If any other processes are asleep waiting for the semaphore, they are awakened.
3. Otherwise, if the value of semaphore is 0, the process goes to sleep until the semaphore value is greater than 0. When the process wakes up, it returns to step 1.
To implement semaphores correctly, the test of a semaphore’s value and the decrementing of this value must be an atomic operation. So, semaphores are normally implemented inside the kernel.
A binary semaphore controls a single resource and its value is initialized to 1. A semaphore can be initialized to any positive value that indicates how many units of the shared resource are available for sharing.
XSI semaphores are complicate with three features.
1. A semaphore is not a single non-negative value, we have to define a semaphore as a set of one or more semaphore values. When we create a semaphore, we specify the number of values in the set.
2. The creation of a semaphore(semget) is independent of its initialization(semctl).
3. Since all forms of XSI IPC remain in existence even when no process is using them, a program may terminates without releasing the semaphores it has been allocated. The undo feature is supposed to handle this.
The kernel maintains a semid_ds structure for each semaphore set, systems can define additional members in the semid_ds structure.

struct semid_ds
{
    struct ipc_perm sem_perm;   /* see Section 15.6.2 */
    unsigned short  sem_nsems;  /* # of semaphores in set */
    time_t          sem_otime;  /* last-semop() time */
    time_t          sem_ctime;  /* last-change time */
    ...
};

Each semaphore is represented by an anonymous structure containing at least the following members:

struct
{
    unsigned short  semval;     /* semaphore value, always >= 0 */
    pid_t           sempid;     /* pid for last operation */
    unsigned short  semncnt;    /* processes awaiting semval > curval */
    unsigned short  semzcnt;    /* processes awaiting semval == 0 */
    ...
};

Figure 15.28 lists the system limits that affect semaphore sets.

#include <sys/sem.h>
int semget(key_t key, int nsems, int flag);
Returns: semaphore ID if OK, -1 on error

semget: obtain a semaphore ID.
When a new set is created, the following members of the semid_ds structure are initialized.
• ipc_perm is initialized as described in Section 15.6.2. The mode member of this structure is set to the corresponding permission bits of flag. These permissions are specified with the values from Figure 15.24.
• sem_otime is set to 0.
• sem_ctime is set to the current time.
• sem_nsems is set to nsems.
nsems is the number of semaphores in the set. If a new set is created, we must specify nsems. If we are referencing an existing set, we can specify nsems as 0.

#include <sys/sem.h>
int semctl(int semid, int semnum, int cmd, ... /* union semun arg */);
Returns: (see following)

arg is optional, depending on cmd; if present, it is of type semun which is a union of various command-specific arguments.

union semun
{
    int             val;        /* for SETVAL */
    struct semid_ds*    buf;        /* for IPC_STAT and IPC_SET */
    unsigned short* array;  /* for GETALL and SETALL */
};

The cmd argument specifies one of the following ten commands to be performed on the set specified by semid. The five commands that refer to one particular semaphore value use semnum to specify one member of the set. The value of semnum is [0, nsems-1].
1. IPC_STAT
  Fetch the semid_ds structure for this set, storing it in the structure pointed to by arg.buf.
2. IPC_SET
  Set the sem_perm.uid, sem_perm.gid, and sem_perm.mode fields from the structure pointed to by arg.buf in the semid_ds structure associated with this set. This command can be executed only by a process whose effective user ID equals sem_perm.cuid or sem_perm.uid or by a process with superuser privileges.
3. IPC_RMID
  Remove the semaphore set from the system. This removal is immediate. Any other process still using the semaphore will get an error of EIDRM on its next attempted operation on the semaphore. This command can be executed only by a process whose effective user ID equals sem_perm.cuid or sem_perm.uid or by a process with superuser privileges.
4. GETVAL
  Return the value of semval for the member semnum.
5. SETVAL
  Set the value of semval for the member semnum. The value is specified by arg.val.
6. GETPID
  Return the value of sempid for the member semnum.
7. GETNCNT
  Return the value of semncnt for the member semnum.
8. GETZCNT
  Return the value of semzcnt for the member semnum.
9. GETALL
  Fetch all the semaphore values in the set. These values are stored in the array pointed to by arg.array.
10. SETALL
  Set all the semaphore values in the set to the values pointed to by arg.array.
For all GET commands other than GETALL, the function returns the corresponding value; for the remaining commands, the return value is 0 if the call succeeds. On error, it sets errno and returns -1.

#include <sys/sem.h>
int semop(int semid, struct sembuf semoparray[], size_t nops);
Returns: 0 if OK, -1 on error

semoparray is a pointer to an array of semaphore operations, represented by sembuf structures:

struct sembuf
{
    unsigned short  sem_num;    /* member in set (0, 1, ..., nsems-1) */
    short           sem_op;     /* operation (negative, 0, or positive) */
    short           sem_flg;        /* IPC_NOWAIT, SEM_UNDO */
};

nops specifies the number of operations(elements) in the array.
The operation on each member of the set is specified by the corresponding sem_op value that can be negative, 0, or positive. Semaphores’ undo flag corresponds to the SEM_UNDO bit in the corresponding sem_flg member.
1. sem_op is positive. This case corresponds to the returning of resources by the process. The value of sem_op is added to the semaphore’s value.
  If the undo flag is specified, sem_op is also subtracted from the semaphore’s adjustment value for this process.
2. sem_op is negative, we want to obtain resources that the semaphore controls.
  -1-If the semaphore’s value is greater than or equal to the absolute value of sem_op(the resources are available), the absolute value of sem_op is subtracted from the semaphore’s value. This guarantees the resulting semaphore value is greater than or equal to 0. If the undo flag is specified, the absolute value of sem_op is also added to the semaphore’s adjustment value for this process.
  -2-If the semaphore’s value is less than the absolute value of sem_op(the resources are not available), the following conditions apply.
  a. If IPC_NOWAIT is specified, semop returns with an error of EAGAIN.
  b. If IPC_NOWAIT is not specified, the semncnt value for this semaphore is incremented(since the caller is about to go to sleep), and the calling process is suspended until one of the following occurs.
  i. The semaphore’s value becomes greater than or equal to the absolute value of sem_op(i.e., some other process has released some resources). The value of semncnt for this semaphore is decremented(since the calling process is done waiting), and the absolute value of sem_op is subtracted from the semaphore’s value. If the undo flag is specified, the absolute value of sem_op is also added to the semaphore’s adjustment value for this process.
  ii.The semaphore is removed from the system. In this case, the function returns an error of EIDRM.
  iii. A signal is caught by the process, and the signal handler returns. In this case, the value of semncnt for this semaphore is decremented(since the calling process is no longer waiting), and the function returns an error of EINTR.
3. sem_op is 0, this means that the calling process wants to wait until the semaphore’s value becomes 0.
  -1-If the semaphore’s value is currently 0, the function returns immediately.
  -2-If the semaphore’s value is nonzero, the following conditions apply.
  a. If IPC_NOWAIT is specified, return is made with an error of EAGAIN.
  b. If IPC_NOWAIT is not specified, the semzcnt value for this semaphore is incremented(since the caller is about to go to sleep), and the calling process is suspended until one of the following occurs.
  i. The semaphore’s value becomes 0. The value of semzcnt for this semaphore is decremented(since the calling process is done waiting).
  ii.The semaphore is removed from the system. In this case, the function returns an error of EIDRM.
  iii. A signal is caught by the process, and the signal handler returns. In this case, the value of semzcnt for this semaphore is decremented(since the calling process is no longer waiting), and the function returns an error of EINTR.
The semop function operates atomically; it does either all the operations in the array or none of them.

Semaphore Adjustment on exit

As we mentioned earlier, it is a problem if a process terminates while it has resources allocated through a semaphore. Whenever we specify the SEM_UNDO flag for a semaphore operation and we allocate resources(a sem_op value less than 0), the kernel remembers how many resources we allocated from that particular semaphore(the absolute value of sem_op). When the process terminates, either voluntarily or involuntarily, the kernel checks whether the process has any outstanding semaphore adjustments and, if so, applies the adjustment to the corresponding semaphore.
If we set the value of a semaphore using semctl, with either the SETVAL or SETALL commands, the adjustment value for that semaphore in all processes is set to 0.

Example-Timing Comparison of Semaphores, Record Locking, and Mutexes

If we are sharing a single resource among multiple processes, we can use one of three techniques to coordinate access. We can use a a semaphore, record locking, or a mutex that is mapped into the address spaces of both processes. It’s interesting to compare the timing differences between the three techniques.
With a semaphore, we create a semaphore set consisting of a single member and initialize the semaphore’s value to 1. To allocate the resource, we call semop with a sem_op of -1; to release the resource, we perform a sem_op of +1. We also specify SEM_UNDO with each operation, to handle the case of a process that terminates without releasing its resource.
With record locking, we create an empty file and use the first byte of the file(which need not exist) as the lock byte. To allocate the resource, we obtain a write lock on the byte; to release it, we unlock the byte. The record locking properties guarantee that if a process terminates while holding a lock, the kernel automatically releases the lock.
To use a mutex, we need both processes to map the same file into their address spaces and initialize a mutex at the same offset in the file using the PTHREAD_PROCESS_SHARED mutex attribute. To allocate the resource, we lock the mutex; to release the resource, we unlock the mutex. If a process terminates without releasing the mutex, recovery is difficult unless we use a robust mutex(recall the pthread_mutex_consistent function discussed in Section 12.4.1).
Figure 15.29 shows the time required to perform these three locking techniques on Linux. In each case, the resource was allocated and then released 1,000,000 times. This was done simultaneously by three different processes. The times in Figure 15.29 are the totals in seconds for all three processes.

On Linux, record locking is faster than semaphores, but mutexes in shared memory outperform both semaphores and record locking. If we’re locking a single resource and don’t need all the fancy features of XSI semaphores, record locking is preferred over semaphores. The reasons are that it is much simpler to use, it is faster(on this platform), and the system takes care of any lingering locks when a process terminates.
Even though using a mutex in shared memory is the fastest option on this platform, we still prefer to use record locking, unless performance is the primary concern. There are two reasons for this. First, recovery from process termination is more difficult using a mutex in memory shared among multiple processes. Second, the process-shared mutex attribute isn’t universally supported yet. In older versions of the Single UNIX Specification, it was optional. Although it is still optional in SUSv4, it is now required by all XSI-conforming implementations.

15.9 Shared Memory

Shared memory allows two or more processes to share a given region of memory. This is the fastest form of IPC because the data does not need to be copied between client and server.
We should synchronize access to a given region among multiple processes: if the server is placing data into a shared memory region, the client shouldn’t try to access the data until the server is done. Semaphores, record locking or mutexes are used to synchronize shared memory access.
The XSI shared memory differs from memory-mapped files in that there is no associated file. The XSI shared memory segments are anonymous segments of memory. The kernel maintains a structure with at least the following members for each shared memory segment.

struct shmid_ds
{
    struct ipc_perm shm_perm;       /* see Section 15.6.2 */
    size_t          shm_segsz;      /* size of segment in bytes */
    pid_t           shm_lpid;       /* pid of last shmop() */
    pid_t           shm_cpid;       /* pid of creator */
    shmatt_t            shm_nattch;     /* number of current attaches */
    time_t          shm_atime;      /* last-attach time */
    time_t          shm_dtime;      /* last-detach time */
    time_t          shm_ctime;      /* last-change time */
    ...
};

The type shmatt_t is defined to be an unsigned integer at least as large as an unsigned short. Figure 15.30 lists the system limits that affect shared memory.

#include <sys/shm.h>
int shmget(key_t key, size_t size, int flag);
Returns: shared memory ID if OK, -1 on error

shmget() obtains a shared memory identifier.
Section 15.6.1 described the rules for converting the key into an identifier and whether a new segment is created or an existing segment is referenced. When a new segment is created, the following members of the shmid_ds structure are initialized.
• The ipc_perm structure is initialized as described in Section 15.6.2. The mode member of this structure is set to the corresponding permission bits of flag. These permissions are specified with the values from Figure 15.24.
• shm_lpid, shm_nattch, shm_atime, and shm_dtime are all set to 0.
• shm_ctime is set to the current time.
• shm_segsz is set to the size requested.
The size parameter is the size of the shared memory segment in bytes.
1. If a new segment is being created, we must specify its size and the contents of the segment are initialized with zeros. Systems usually round up this size to a multiple of the system’s page size, if an application specifies size as a value other than an integral multiple of the system’s page size, the remainder of the last page will be unavailable for use.
2. If we are referencing an existing segment, we can specify size as 0.

#include <sys/shm.h>
int shmctl(int shmid, int cmd, struct shmid_ds *buf );
Returns: 0 if OK, -1 on error

cmd specifies one of the following five commands to be performed on the segment specified by shmid
1. IPC_STAT
  Fetch the shmid_ds structure for this segment, storing it in the structure pointed to by buf.
2. IPC_SET
  Set the following three fields from the structure pointed to by buf in the shmid_ds structure associated with this shared memory segment: shm_perm.uid, shm_perm.gid, and shm_perm.mode. This command can be executed only by a process whose effective user ID equals shm_perm.cuid or shm_perm.uid or by a process with superuser privileges.
3. IPC_RMID
  Remove the shared memory segment set from the system. Since an attachment count is maintained for shared memory segments(the shm_nattch field in the shmid_ds structure), the segment is not removed until the last process using the segment terminates or detaches it. Regardless of whether the segment is in use, the segment’s identifier is immediately removed so that shmat can no longer attach the segment. This command can be executed only by a process whose effective user ID equals shm_perm.cuid or shm_perm.uid or by a process with superuser privileges.
4. SHM_LOCK
  Lock the shared memory segment in memory. This command can be executed only by the superuser.
5. SHM_UNLOCK
  Unlock the shared memory segment. This command can be executed only by the superuser.

#include <sys/shm.h>
void *shmat(int shmid, const void *addr, int flag);
Returns: pointer to shared memory segment if OK, -1 on error

shmat: attaches a shared memory segment to calling process’s address space.
The address in the calling process at which the segment is attached depends on the addr argument and whether the SHM_RND bit is specified in flag. If:
1. addr = 0: the segment is attached at the first available address selected by the kernel. This is the recommended technique.
2. addr != 0 and SHM_RND is not specified: the segment is attached at the address given by addr.
3. addr != 0 and SHM_RND is specified: the segment is attached at the address given by (addr - addr % SHMLBA). SHM_RND stands for “round”. SHMLBA stands for “low boundary address multiple” and is always a power of 2. What the arithmetic does is round the address down to the next multiple of SHMLBA.
If the SHM_RDONLY bit is specified in flag, the segment is attached as read-only; otherwise, the segment is attached as read-write.
The value returned by shmat is the address at which the segment is attached, or -1 if an error occurred. If shmat succeeds, the kernel will increment the shm_nattch counter in the shmid_ds structure associated with the shared memory segment.

#include <sys/shm.h>
int shmdt(const void *addr);
Returns: 0 if OK, -1 on error

shmdt: detach a shared memory segment. This does not remove the identifier and its associated data structure from the system. The identifier remains in existence until some process specifically removes it by calling shmctl with a command of IPC_RMID.
addr is the value that was returned by a previous call to shmat. If successful, shmdt will decrement the shm_nattch counter in the associated shmid_ds structure.
Figure 15.31 shows a program that prints some information on where system places various types of data.

#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <sys/ipc.h>
#include <sys/types.h>
#include <sys/shm.h>

#define ARRAY_SIZE 40000
#define MALLOC_SIZE 100000
#define SHM_SIZE 100000
#define SHM_MODE 0600 /* user read&write */

char array[ARRAY_SIZE]; /* uninitialized data = bss */

void Exit(char *string)
{
    printf("%s\n", string);
    exit(1);
}

int main()
{
    int shmid;
    char *ptr, *shmptr;

    printf("array[] from %p to %p\n", (void *)&array[0],
        (void *)&array[ARRAY_SIZE]);
    printf("stack around %p\n", (void *)&shmid);

    if((ptr = malloc(MALLOC_SIZE)) == NULL)
    {
        Exit("malloc error");
    }
    printf("malloced from %p to %p\n", (void *)ptr, (void *)ptr+MALLOC_SIZE);

    if((shmid = shmget(IPC_PRIVATE, SHM_SIZE, SHM_MODE)) < 0)
    {
        Exit("shmget error");
    }
    if((shmptr = shmat(shmid, 0, 0)) == (void *)-1)
    {
        Exit("shmat error");
    }
    printf("shared memory attached from %p to %p\n", (void *)shmptr,
        (void *)shmptr+SHM_SIZE);

    if (shmctl(shmid, IPC_RMID, 0) < 0)
    {
        Exit("shmctl error");
    }

    exit(0);
}

$ ./a.out
array[] from 0x6020c0 to 0x60bd00
stack around 0x7fff957b146c
malloc from 0x9e3010 to 0x9fb6b0
shared memory attached from 0x7fba578ab000 to 0x7fba578c36a0

Figure 15.32 shows a picture of this, the shared memory segment is placed below the stack.

The mmap function(Section 14.8) can be used to map portions of a file into the address space of a process. Difference with shmat() is that the memory segment mapped with mmap is backed by a file, whereas no file is associated with a shared memory segment.

Example-Memory Mapping of /dev/zero

Shared memory can be used between unrelated processes; if the processes are related, some systems provide a different technique. Linux support the mapping of character devices into the address space of a process.
The device /dev/zero is an infinite source of 0 bytes when read, it accepts and ignores any data that is written to it. When it is memory mapped:
1. An unnamed memory region is created whose size is the second argument to mmap, rounded up to the nearest page size on the system.
2. The memory region is initialized to 0.
3. Multiple processes can share this region if a common ancestor specifies the MAP_SHARED flag to mmap.

The program opens the /dev/zero device and calls mmap with a size of a long integer. Once the region is mapped, we can close the device. The process then creates a child. Since MAP_SHARED was specified in the call to mmap, writes to the memory-mapped region by one process are seen by the other process. If we specified MAP_PRIVATE instead, this example wouldn’t work.
The parent and the child then alternate running, incrementing a long integer in the shared memory-mapped region, using the synchronization functions from Section 8.9. The memory-mapped region is initialized to 0 by mmap. The parent increments it to 1, then the child increments it to 2, then the parent increments it to 3, and so on.
Advantage of using /dev/zero in this manner: An actual file need not exist before we call mmap to create the mapped region. Mapping /dev/zero automatically creates a mapped region of the specified size.
Disadvantage: It works only between related processes. With related processes, it is simpler and more efficient to use threads. No matter which technique is used, we still need to synchronize access to the shared data.

Example-Anonymous Memory Mapping

Linux provide anonymous memory mapping(similar to the /dev/zero feature). To use it, we specify the MAP_ANON or MAP_ANONYMOUS flag to mmap and specify the file descriptor as -1. The resulting region is anonymous(since it’s not associated with a pathname through a file descriptor) and creates a memory region that can be shared with descendant processes.
We make 3 changes in Figure 15.33 to use this facility:
(a) remove the open of /dev/zero,
(b) remove the close of fd, and
(c) change the call to mmap to the following:

if((area = mmap(0, SIZE, PROT_READ | PROT_WRITE, MAP_ANON | MAP_SHARED, -1, 0)) == MAP_FAILED)

15.10 POSIX Semaphores

The POSIX semaphore interfaces were meant to address several deficiencies with the XSI semaphore interfaces:
• The POSIX semaphore interfaces allow for higher-performance systems compared to XSI semaphores.
• The POSIX semaphore interfaces are simpler to use: there are no semaphore sets, and several of the interfaces are patterned after familiar file system operations. Although there is no requirement that they be implemented in the file system, some systems do take this approach.
• The POSIX semaphores behave more gracefully when removed. Recall that when an XSI semaphore is removed, operations using the same semaphore identifier fail with errno set to EIDRM. With POSIX semaphores, operations continue to work normally until the last reference to the semaphore is released.
POSIX semaphores are available in two flavors: named and unnamed. They differ in how they are created and destroyed, but otherwise work the same. Unnamed semaphores exist in memory only and require that processes have access to the memory to be able to use the semaphores. This means they can be used only by threads in the same process or threads in different processes that have mapped the same memory extent into their address spaces. Named semaphores, in contrast, are accessed by name and can be used by threads in any processes that know their names.
To create a new named semaphore or use an existing one, we call the sem_open function.

#include <semaphore.h>
sem_t *sem_open(const char *name, int oflag, ... /* mode_t mode, unsigned int value */ );
Returns: Pointer to semaphore if OK, SEM_FAILED on error

When using an existing named semaphore, we specify only two arguments: the name of the semaphore and a zero value for the oflag argument. When the oflag argument has the O_CREAT flag set, we create a new named semaphore if it does not yet exist. If it already exists, it is opened for use, but no additional initialization takes place.
When we specify the O_CREAT flag, we need to provide two additional arguments. The mode argument specifies who can access the semaphore. It can take on the same values as the permission bits for opening a file: user-read, user-write, user-execute, group-read, group-write, group-execute, other-read, other-write, and other-execute.
The resulting permissions assigned to the semaphore are modified by the caller’s file creation mask(Sections 4.5 and 4.8). Note, however, that only read and write access matter, but the interfaces don’t allow us to specify the mode when we open an existing semaphore. systems usually open semaphores for both reading and writing.
The value argument is used to specify the initial value for the semaphore when we create it. It can take on any value from 0 to SEM_VALUE_MAX(Figure 2.9).
If we want to ensure that we are creating the semaphore, we can set the oflag argument to O_CREAT|O_EXCL. This will cause sem_open to fail if the semaphore already exists.
To promote portability, we must follow certain conventions when selecting a semaphore name.
• The first character in the name should be a slash(/). Although there is no requirement that an implementation of POSIX semaphores uses the file system, if the file system is used, we want to remove any ambiguity as to the starting point from which the name is interpreted.
• The name should contain no other slashes to avoid implementation-defined behavior. E.g., if the file system is used, the names /mysem and //mysem would evaluate to the same filename, but if the implementation doesn’t use the file system, the two names could be treated as different(consider what would happen if the implementation hashed the name to an integer value used to identify the semaphore).
• The maximum length of the semaphore name is implementation defined. The name should be no longer than _POSIX_NAME_MAX(Figure 2.8) characters, because this is the minimum acceptable limit to the maximum name length if the implementation does use the file system.
The sem_open function returns a semaphore pointer that we can pass to other semaphore functions when we want to operate on the semaphore. When we are done with the semaphore, we can call the sem_close function to release any resources associated with the semaphore.

#include <semaphore.h>
int sem_close(sem_t *sem);
Returns: 0 if OK, -1 on error

If our process exits without having first called sem_close, the kernel will close any open semaphores automatically. Note that this doesn’t affect the state of the semaphore value-if we have incremented its value, this doesn’t change just because we exit. Similarly, if we call sem_close, the semaphore value is unaffected. There is no mechanism equivalent to the SEM_UNDO flag found with XSI semaphores.
To destroy a named semaphore, we can use the sem_unlink function.

#include <semaphore.h>
int sem_unlink(const char *name);
Returns: 0 if OK, -1 on error

The sem_unlink function removes the name of the semaphore. If there are no open references to the semaphore, then it is destroyed. Otherwise, destruction is deferred until the last open reference is closed.
Unlike with XSI semaphores, we can only adjust the value of a POSIX semaphore by one with a single function call. Decrementing the count is analogous to locking a binary semaphore or acquiring a resource associated with a counting semaphore.
Note that there is no distinction of semaphore type with POSIX semaphores. Whether we use a binary semaphore or a counting semaphore depends on how we initialize and use the semaphore. If a semaphore only ever has a value of 0 or 1, then it is a binary semaphore.
When a binary semaphore has a value of 1, we say that it is “unlocked;” when it has a value of 0, we say that it is “locked.” To decrement the value of a semaphore, we can use either the sem_wait or sem_trywait function.

#include <semaphore.h>
int sem_trywait(sem_t *sem);
int sem_wait(sem_t *sem);
Both return: 0 if OK, -1 on error

With the sem_wait function, we will block if the semaphore count is 0. We won’t return until we have successfully decremented the semaphore count or are interrupted by a signal. We can use the sem_trywait function to avoid blocking. If the semaphore count is zero when we call sem_trywait, it will return -1 with errno set to EAGAIN instead of blocking.
A third alternative is to block for a bounded amount of time. We can use the sem_timedwait function for this purpose.

#include <semaphore.h>
#include <time.h>
int sem_timedwait(sem_t *restrict sem,
const struct timespec *restrict tsptr);
Returns: 0 if OK, -1 on error

The tsptr argument specifies the absolute time when we want to give up waiting for the semaphore. The timeout is based on the CLOCK_REALTIME clock(recall Figure 6.8). If the semaphore can be decremented immediately, then the value of the timeout doesn’t matter-even though it might specify some time in the past, the attempt to decrement the semaphore will still succeed. If the timeout expires without being able to decrement the semaphore count, then sem_timedwait will return -1 with errno set to ETIMEDOUT.
To increment the value of a semaphore, we call the sem_post function. This is analogous to unlocking a binary semaphore or releasing a resource associated with a counting semaphore.

#include <semaphore.h>
int sem_post(sem_t *sem);
Returns: 0 if OK, -1 on error

If a process is blocked in a call to sem_wait(or sem_timedwait) when we call sem_post, the process is awakened and the semaphore count that was just incremented by sem_post is decremented by sem_wait(or sem_timedwait).
When we want to use POSIX semaphores within a single process, it is easier to use unnamed semaphores. This only changes the way we create and destroy the semaphore. To create an unnamed semaphore, we call the sem_init function.

#include <semaphore.h>
int sem_init(sem_t *sem, int pshared, unsigned int value);
Returns: 0 if OK, -1 on error

The pshared argument indicates if we plan to use the semaphore with multiple processes. If so, we set it to a nonzero value. The value argument specifies the initial value of the semaphore.
Instead of returning a pointer to the semaphore like sem_open does, we need to declare a variable of type sem_t and pass its address to sem_init for initialization. If we plan to use the semaphore between two processes, we need to ensure that the sem argument points into the memory extent that is shared between the processes.
When we are done using the unnamed semaphore, we can discard it by calling the sem_destroy function.

#include <semaphore.h>
int sem_destroy(sem_t *sem);
Returns: 0 if OK, -1 on error

After calling sem_destroy, we can’t use any of the semaphore functions with sem unless we reinitialize it by calling sem_init again.
One other function is available to allow us to retrieve the value of a semaphore. We call the sem_getvalue function for this purpose.

#include <semaphore.h>
int sem_getvalue(sem_t *restrict sem, int *restrict valp);
Returns: 0 if OK, -1 on error

On success, the integer pointed to by the valp argument will contain the value of the semaphore. Be aware, however, that the value of the semaphore can change by the time that we try to use the value we just read. Unless we use additional synchronization mechanisms to avoid this race, the sem_getvalue function is useful only for debugging. The sem_getvalue function is not supported by Mac OS X 10.6.8.

Example

One of the motivations for introducing the POSIX semaphore interfaces was that they can be made to perform significantly better than the existing XSI semaphore interfaces. It is instructive to see if this goal was reached in existing systems, even though these systems were not designed to support real-time applications.
In Figure 15.34, we compare the performance of using XSI semaphores(without SEM_UNDO) and POSIX semaphores when 3 processes compete to allocate and release the semaphore 1,000,000 times on two platforms(Linux 3.2.0 and Solaris 10).

In Figure 15.34, we can see that POSIX semaphores provide only a 12% improvement over XSI semaphores on Solaris, but on Linux the improvement is 94%(almost 18 times faster)! If we trace the programs, we find that the Linux implementation of POSIX semaphores maps the file into the process address space and performs individual semaphore operations without using system calls.

Example

Recall from Figure 12.5 that the Single UNIX Specification doesn’t define what happens when one thread locks a normal mutex and a different thread tries to unlock it, but that error-checking mutexes and recursive mutexes generate errors in this case. Because a binary semaphore can be used like a mutex, we can use a semaphore to create our own locking primitive to provide mutual exclusion.
Assuming we were to create our own lock that could be locked by one thread and unlocked by another, our lock structure might look like

struct slock
{
    sem_t*  semp;
    char        name[_POSIX_NAME_MAX];
};

The program in Figure 15.35 shows an implementation of a semaphore-based mutual exclusion primitive.

We create a name based on the process ID and a counter. We don’t bother to protect the counter with a mutex, because if two racing threads call s_alloc at the same time and end up with the same name, using the O_EXCL flag in the call to sem_open will cause one to succeed and one to fail with errno set to EEXIST, so we just retry if this happens. Note that we unlink the semaphore after opening it. This destroys the name so that no other process can access it and simplifies cleanup when the process ends.

15.11 Client-Server Properties

Let’s detail some of the properties of clients and servers that are affected by the various types of IPC used between them. The simplest type of relationship is to have the client fork and exec the desired server. Two half-duplex pipes can be created before the fork to allow data to be transferred in both directions. Figure 15.16 is an example of this arrangement. The server that is executed can be a set-user-ID program, giving it special privileges. Also, the server can determine the real identity of the client by looking at its real user ID.(Recall from Section 8.10 that the real user ID and real group ID don’t change across an exec.)
With this arrangement, we can build an open server.(We show an implementation of this client-server mechanism in Section 17.5.) It opens files for the client instead of the client calling the open function. This way, additional permission checking can be added, above and beyond the normal UNIX system user/group/other permissions. We assume that the server is a set-user-ID program, giving it additional permissions(root permission, perhaps). The server uses the real user ID of the client to determine whether to give it access to the requested file. This way, we can build a server that allows certain users permissions that they don’t normally have.
In this example, since the server is a child of the parent, all the server can do is pass back the contents of the file to the parent. Although this works fine for regular files, it can’t be used for special device files, E.g.. We would like to be able to have the server open the requested file and pass back the file descriptor. Whereas a parent can pass a child an open descriptor, a child cannot pass a descriptor back to the parent(unless special programming techniques are used, which we cover in Chapter 17).
We showed the next type of server in Figure 15.23. The server is a daemon process that is contacted using some form of IPC by all clients. We can’t use pipes for this type of client-server arrangement. A form of named IPC is required, such as FIFOs or message queues. With FIFOs, we saw that an individual per-client FIFO is also required if the server is to send data back to the client. If the client-server application sends data only from the client to the server, a single well-known FIFO suffices.(The System V line printer spooler used this form of client-server arrangement. The client was the lp(1) command, and the server was the lpsched daemon process. A single FIFO was used, since the flow of data was only from the client to the server. Nothing was sent back to the client.)
Multiple possibilities exist with message queues.
1. A single queue can be used between the server and all the clients, using the type field of each message to indicate the message recipient. E.g., the clients can send their requests with a type field of 1. Included in the request must be the client’s process ID. The server then sends the response with the type field set to the client’s process ID. The server receives only the messages with a type field of 1(the fourth argument for msgrcv), and the clients receive only the messages with a type field equal to their process IDs.
2. Alternatively, an individual message queue can be used for each client. Before sending the first request to a server, each client creates its own message queue with a key of IPC_PRIVATE. The server also has its own queue, with a key or identifier known to all clients. The client sends its first request to the server’s well-known queue, and this request must contain the message queue ID of the client’s queue. The server sends its first response to the client’s queue, and all future requests and responses are exchanged on this queue.
One problem with this technique is that each client-specific queue usually has only a single message on it: a request for the server or a response for a client. This seems wasteful of a limited systemwide resource(a message queue), and a FIFO can be used instead. Another problem is that the server has to read messages from multiple queues. Neither select nor poll works with message queues.
Either of these two techniques using message queues can be implemented using shared memory segments and a synchronization method(a semaphore or record locking).
The problem with this type of client-server relationship(the client and the server being unrelated processes) is for the server to identify the client accurately. Unless the server is performing a nonprivileged operation, it is essential that the server know who the client is. This is required, E.g., if the server is a set-user-ID program.
Although all these forms of IPC go through the kernel, there is no facility provided by them to have the kernel identify the sender.
With message queues, if a single queue is used between the client and the server(so that only a single message is on the queue at a time, E.g.), the msg_lspid of the queue contains the process ID of the other process. But when writing the server, we want the effective user ID of the client, not its process ID. There is no portable way to obtain the effective user ID, given the process ID.(Naturally, the kernel maintains both values in the process table entry, but other than rummaging around through the kernel’s memory, we can’t obtain one, given the other.)
We’ll use the following technique in Section 17.2 to allow the server to identify the client. The same technique can be used with FIFOs, message queues, semaphores, and shared memory. For the following description, assume that FIFOs are being used, as in Figure 15.23. The client must create its own FIFO and set the file access permissions of the FIFO so that only user-read and user-write are on. We assume that the server has superuser privileges(or else it probably wouldn’t care about the client’s true identity), so the server can still read and write to this FIFO. When the server receives the client’s first request on the server’s well-known FIFO(which must contain the identity of the client-specific FIFO), the server calls either stat or fstat on the client-specific FIFO.
The server assumes that the effective user ID of the client is the owner of the FIFO(the st_uid field of the stat structure). The server verifies that only the user-read and user-write permissions are enabled. As another check, the server should look at the three times associated with the FIFO(the st_atime, st_mtime, and st_ctime fields of the stat structure) to verify that they are recent(no older than 15 or 30 seconds, E.g.). If a malicious client can create a FIFO with someone else as the owner and set the file’s permission bits to user-read and user-write only, then the system has other fundamental security problems.
To use this technique with XSI IPC, recall that the ipc_perm structure associated with each message queue, semaphore, and shared memory segment identifies the creator of the IPC structure(the cuid and cgid fields). As with the example using FIFOs, the server should require the client to create the IPC structure and have the client set the access permissions to user-read and user-write only. The times associated with the IPC structure should also be verified by the server to be recent(since these IPC structures hang around until explicitly deleted).
We’ll see in Section 17.3 that a far better way of doing this authentication is for the kernel to provide the effective user ID and effective group ID of the client. This is done by the socket subsystem when file descriptors are passed between processes.

15.12 Summary

We’ve detailed numerous forms of interprocess communication: pipes, named pipes(FIFOs), the three forms of IPC commonly called XSI IPC(message queues, semaphores, and shared memory), and an alternative semaphore mechanism provided by POSIX.
Semaphores are really a synchronization primitive, not true IPC, and are often used to synchronize access to a shared resource, such as a shared memory segment. With pipes, we looked at the implementation of the popen function, at coprocesses, and at the pitfalls that can be encountered with the standard I/O library’s buffering.
After comparing the timing of message queues versus full-duplex pipes, and semaphores versus record locking, we can make the following recommendations: learn pipes and FIFOs, since these two basic techniques can still be used effectively in numerous applications. Avoid using message queues and semaphores in any new applications. Full-duplex pipes and record locking should be considered instead, as they are far simpler. Shared memory still has its use, although the same functionality can be provided through the use of the mmap function(Section 14.8).
In the next chapter, we will look at network IPC, which can allow processes to communicate across machine boundaries.
Exercises(Redo)

Please indicate the source: http://blog.csdn.net/gaoxiangnumber1

Welcome to my github: https://github.com/gaoxiangnumber1

你可能感兴趣的:(github,ipc)

如何在 Fork 的 GitHub 项目中保留自己的修改并同步上游更新？github_fork_update iBaoxing github
如何在Fork的GitHub项目中保留自己的修改并同步上游更新？在GitHub上Fork了一个项目后，你可能会对项目进行一些修改，同时原作者也在不断更新。如果想要在保留自己修改的基础上，同步原作者的最新更新，很多人会不知所措。本文将详细讲解如何在不丢失自己改动的情况下，将上游仓库的更新合并到自己的仓库中。问题描述假设你在GitHub上Fork了一个项目，并基于该项目做了一些修改，随后你发现原作者对
下载github patch到本地小米人er 我的博客 git patch
以下是几种从GitHub上下载以.patch结尾的补丁文件的方法：通过浏览器直接下载打开包含该.patch文件的GitHub仓库。在仓库的文件列表中找到对应的.patch文件。点击该文件，浏览器会显示文件的内容，在页面的右上角通常会有一个“Raw”按钮，点击它可以获取原始文件内容。然后在浏览器中使用快捷键（如Ctrl+S或者Command+S）将原始文件保存到本地，选择保存的文件名并确保后缀为.p
509. 斐波那契数(每日一题) lzyprime
lzyprime博客(github)创建时间：2021.01.04qq及邮箱：2383518170leetcode笔记题目描述斐波那契数，通常用F(n)表示，形成的序列称为斐波那契数列。该数列由0和1开始，后面的每一项数字都是前面两项数字的和。也就是：F(0)=0，F(1)=1F(n)=F(n-1)+F(n-2)，其中n>1给你n，请计算F(n)。示例1：输入：2输出：1解释：F(2)=F(1)+
GitHub上克隆项目 bigbig猩猩 github
从GitHub上克隆项目是一个简单且直接的过程，它允许你将远程仓库中的项目复制到你的本地计算机上，以便进行进一步的开发、测试或学习。以下是一个详细的步骤指南，帮助你从GitHub上克隆项目。一、准备工作1.安装Git在克隆GitHub项目之前，你需要在你的计算机上安装Git工具。Git是一个开源的分布式版本控制系统，用于跟踪和管理代码变更。你可以从Git的官方网站（https://git-scm.
git - Webhook让部署自动化大猪大猪
我们现在有一个需求，将项目打包上传到gitlab或者github后，程序能自动部署，不用手动地去服务器中进行项目更新并运行，如何做到？这里我们可以使用gitlab与github的挂钩，挂钩的原理就是，每当我们有请求到gitlab与github服务器时，这时他俩会根据我们配置的挂钩地扯进行访问，webhook挂钩程序会一直监听着某个端口请求，一但收到他们发过来的请求，这时就知道用户有请求提交了，这时
libyuv之linux编译 jaronho Linux linux 运维服务器
文章目录一、下载源码二、编译源码三、注意事项1、银河麒麟系统（aarch64）（1）解决armv8-a+dotprod+i8mm指令集支持问题（2）解决armv9-a+sve2指令集支持问题一、下载源码到GitHub网站下载https://github.com/lemenkov/libyuv源码，或者用直接用git克隆到本地，如：gitclonehttps://github.com/lemenko
pyecharts——绘制柱形图折线图 2224070247 信息可视化 python java 数据可视化
一、pyecharts概述自2013年6月百度EFE(ExcellentFrontEnd）数据可视化团队研发的ECharts1.0发布到GitHub网站以来，ECharts一直备受业界权威的关注并获得广泛好评，成为目前成熟且流行的数据可视化图表工具，被应用到诸多数据可视化的开发领域。Python作为数据分析领域最受欢迎的语言，也加入ECharts的使用行列，并研发出方便Python开发者使用的数据
node.js学习小猿L node.js node.js 学习 vim
node.js学习实操及笔记温故node.js，node.js学习实操过程及笔记~node.js学习视频node.js官网node.js中文网实操笔记githubcsdn笔记为什么学node.js可以让别人访问我们编写的网页为后续的框架学习打下基础，三大框架vuereactangular离不开node.jsnode.js是什么官网：node.js是一个开源的、跨平台的运行JavaScript的运行
你可能遗漏的一些C#/.NET/.NET Core知识点追逐时光者 C#.NET DotNetGuide编程指南 c#.net .netcore microsoft
前言在这个快速发展的技术世界中，时常会有一些重要的知识点、信息或细节被忽略或遗漏。《C#/.NET/.NETCore拾遗补漏》专栏我们将探讨一些可能被忽略或遗漏的重要知识点、信息或细节，以帮助大家更全面地了解这些技术栈的特性和发展方向。拾遗补漏GitHub开源地址https://github.com/YSGStudyHards/DotNetGuide/blob/main/docs/DotNet/D
python怎么将png转为tif_png转tif weixin_39977276
发国外的文章要求图片是tif，cmyk色彩空间的。大小尺寸还有要求。比如网上大神多，找到了一段代码，感谢！https://www.jianshu.com/p/ec2af4311f56https://github.com/KevinZc007/image2Tifimportjava.awt.image.BufferedImage;importjava.io.File;importjava.io.Fi
windows下python opencv ffmpeg读取摄像头实现rtsp推流拉流图像处理大大大大大牛啊 opencv实战代码讲解视觉图像项目 windows python opencv
windows下pythonopencvffmpeg读取摄像头实现rtsp推流拉流整体流程1.下载所需文件1.1下载rtsp推流服务器1.2下载ffmpeg2.开启RTSP服务器3.opencv读取摄像头并调用ffmpeg进行推流4.opencv进行拉流5.opencv异步拉流整体流程1.下载所需文件1.1下载rtsp推流服务器下载RTSP服务器下载页面https://github.com/blu
ChatGPT 高效学习套路揭秘：让知识获取事半功倍的秘诀 kkai人工智能 chatgpt 人工智能学习媒体 ai
最近这段时间，AI热潮因ChatGPT的火爆再次掀起。如今，网上大部分内容都在调侃AI，但很少有人探讨如何正经使用ChatGPT做事情。作为一名靠搜索引擎和GitHub自学编程的开发者，第一次和ChatGPT深度交流后，我就确信：ChatGPT能够极大提高程序员学习新技术的效率。使用ChatGPT一个月后，我越发感受到它的颠覆性。因此，我想从工作和学习的角度，分享它的优势及我的一些使用技巧，而非娱
LeetCode github集合，附CMU大神整理笔记 Wesley@ LeetCode github
GithubLeetCode集合本人所有做过的题目都写在一个java项目中，同步到github中了，算是见证自己的进步。github目前同步的题目是2020-09-17日之后写的题。之前写过的题会陆续跟新到github中。目前大概400个题目Github项目链接：https://github.com/sunliancheng/leetcode_github附上一份优秀的教材整合：这是卡内基梅隆(C
[Swift]LeetCode943. 最短超级串 | Find the Shortest Superstring 黄小二哥 swift
★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★➤微信公众号：山青咏芝（shanqingyongzhi）➤博客园地址：山青咏芝（https://www.cnblogs.com/strengthen/）➤GitHub地址：https://github.com/strengthen/LeetCode➤原文地址：https://www.cnblogs.com/streng
[Swift]LeetCode767. 重构字符串 | Reorganize String weixin_30591551 swift runtime
★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★➤微信公众号：山青咏芝（shanqingyongzhi）➤博客园地址：山青咏芝（https://www.cnblogs.com/strengthen/）➤GitHub地址：https://github.com/strengthen/LeetCode➤原文地址：https://www.cnblogs.com/streng
VUE3 + xterm + nestjs实现web远程终端或连接开启SSH登录的路由器和交换机。焚木灵 node.js vue
可远程连接系统终端或开启SSH登录的路由器和交换机。相关资料：xtermjs/xterm.js:Aterminalfortheweb(github.com)后端实现(NestJS)：1、安装依赖：npminstallnode-ssh@nestjs/websockets@nestjs/platform-socket.io2、我们将创建一个名为RemoteControlModule的NestJS模块，
metaRTC8.0，一个全新架构的webRTC SDK库 metaRTC webrtc 音视频
概述metaRTC8.0是metaRTC开源以来架构变化最大的一个版本，是metaIPC3.0等高性能的基础。metaRTC8.0是一个全新架构版本，并非在metaRTC7.0版本上简单升级，在QOS/语音对讲/内存占用/视频文件录制读取等方面新增多个模块，在弱网对抗/语音对讲/内存优化等效果上有显著提升。metaRTC8.0在一年多的开发中进行了近200次迭代，metaRTC8.0社区版计划在2
golang实现从服务器下载文件到本地指定目录余生逆风飞翔 golang 服务器开发语言
一、连接服务器，采用sftp连接模式packagemiddlewaresimport("fmt""time""github.com/pkg/sftp""golang.org/x/crypto/ssh")//建立服务器连接funcConnect(user,password,hoststring,portint)(*sftp.Client,error){var(auth[]ssh.AuthMethod
Nginx之ngx_http_proxy_connect_module模块小米bb Nginx nginx http 运维
近期由于项目需要使用到https正向代理，而nginx官方模块仅支持做http正向代理，一番百度学习后发现了该模块，故今日记录下此笔记供大家一起学习交流ngx_http_proxy_connect_module模块主要用于隧道SSL请求的代理服务器GitHub地址：http://www.github.com/chobits/ngx_http_proxy_connect_modulenginx配置：
编译Windows平台的Nginx+ngx_http_proxy_connect_module Grovvy_Deng windows nginx http
编译Windows平台的Nginx+ngx_http_proxy_connect_module背景：由于公司的正向出局代理是windows机器。机器上的Squid不稳定，打算替换成nginx+ngx_http_proxy_connect_module实现。通过几天痛苦的尝试，最后参考了github大神项目通过在线CICD工具编译window平台可用的ng。步骤：获取git可识别的patch由于CI
FlagEmbedding 吉小雨 python库 python
FlagEmbedding教程FlagEmbedding是一个用于生成文本嵌入（textembeddings）的库，适合处理自然语言处理（NLP）中的各种任务。嵌入（embeddings）是将文本表示为连续向量，能够捕捉语义上的相似性，常用于文本分类、聚类、信息检索等场景。官方文档链接：FlagEmbedding官方GitHub一、FlagEmbedding库概述1.1什么是FlagEmbeddi
[移动端自动化] AppAgent介绍大卫软件测试自动化
AUITestAgentAUITestAgent/README_zh.mdatmain·bz-lab/AUITestAgent(github.com)1/MobileAgentX-PLUG/MobileAgent:Mobile-Agent:ThePowerfulMobileDeviceOperationAssistantFamily(github.com)2/AppAgentAppAgent/RE
Gobelieve 架构 weixin_34099526 数据库 golang json
Gobelievegithub地址声明:转简书JackieF的文章,为了自己方便copy了一份,加一些自己的东西.链接：https://www.jianshu.com/p/8121d6e85282IMCore主要分三大块:im客户连接服务器（可分布式部署，暂无负载均衡模块)imr路由查询服务器（主要解决im分布式部署的问题）ims存储服务器(主从部署)基础模块1.数据包协议包：header(12)
Github 2024-09-12 Go开源项目日报Top10 老孙正经胡说 github golang 开源 Github趋势分析开源项目 Python Golang
根据GithubTrendings的统计，今日(2024-09-12统计)共有10个项目上榜。根据开发语言中项目的数量，汇总情况如下：开发语言项目数量Go项目10C项目1Terraform：基础设施即代码的开源工具创建周期：3626天开发语言：Go协议类型：OtherStar数量：40393个Fork数量：9397次关注人数：40393人贡献人数：358人OpenIssues数量：1943个Git
利用frp实现内网穿透（需要云主机） heiyumiao 网络工具 github windows linux
详细实现教程：https://www.iplaysoft.com/frp.htmlhttps://blog.csdn.net/u013144287/article/details/78589643/项目官网地址：https://github.com/fatedier/frp/releases按照系统下载相应的程序包什么是frp内网穿透工具有很多，其中Frp(FastReverseProxy)是比较
PAT Advanced 1015. Reversible Primes (C语言实现) OliverLew
我的PAT系列文章更新重心已移至Github，欢迎来看PAT题解的小伙伴请到GithubPages浏览最新内容。此处文章目前已更新至与GithubPages同步。欢迎star我的repo。题目Areversibleprimeinanynumbersystemisaprimewhose"reverse"inthatnumbersystemisalsoaprime.Forexampleinthedec
CMU 15-445/645 Lab2-B+Tree Index yyy_3y CMU-15/445 b树数据结构 CMU15-445 数据库
0.写在前面GitHub同步更新https://github.com/kaniel-outis/CMU15-445Lab2的地址：https://15445.courses.cs.cmu.edu/fall2020/project2/本文主要总结一下在写Lab2需要的基础知识以及Task的解决思路（不公开代码，如果有问题可以留言）。Lab2的主要内容是B+tree的定义和Insert、Delete操
discuz discuz_admincp.php 讲解,Discuz! 1.5-2.5 命令执行漏洞分析(CVE-2018-14729) weixin_39740419 discuz 讲解
0x00漏洞简述漏洞信息8月27号有人在GitHub上公布了有关Discuz1.5-2.5版本中后台数据库备份功能存在的命令执行漏洞的细节。漏洞影响版本Discuz!1.5-2.50x01漏洞复现官方论坛下载相应版本就好。0x02漏洞分析需要注意的是这个漏洞其实是需要登录后台的，并且能有数据库备份权限，所以比较鸡肋。我这边是用Discuz!2.5完成漏洞复现的，并用此进行漏洞分析的。漏洞点在：so
python logging模块默认日志级别_一看就懂，Python 日志 logging 模块详解及应用路易·罗莎 python logging模块默认日志级别
日志概述百度百科的日志概述：Windows网络操作系统都设计有各种各样的日志文件，如应用程序日志，安全日志、系统日志、Scheduler服务日志、FTP日志、WWW日志、DNS服务器日志等等，这些根据你的系统开启的服务的不同而有所不同。我们在系统上进行一些操作时，这些日志文件通常会记录下我们操作的一些相关内容，这些内容对系统安全工作人员相当有用。比如说有人对系统进行了IPC探测，系统就会在安全日志
探索未来，大规模分布式深度强化学习——深入解析IMPALA架构汤萌妮Margaret
探索未来，大规模分布式深度强化学习——深入解析IMPALA架构scalable_agent项目地址:https://gitcode.com/gh_mirrors/sc/scalable_agent在当今的人工智能研究前沿，深度强化学习（DRL）因其在复杂任务中的卓越表现而备受瞩目。本文要介绍的是一个开源于GitHub的重量级项目：“ScalableDistributedDeep-RLwithImp
html 周华华 html
js 1，数组的排列 var arr=[1,4,234,43,52,]; for(var x=0;x<arr.length;x++){ for(var y=x-1;y<arr.length;y++){ if(arr[x]<arr[y]){ &
【Struts2 四】Struts2拦截器 bit1129 struts2拦截器
Struts2框架是基于拦截器实现的，可以对某个Action进行拦截，然后某些逻辑处理，拦截器相当于AOP里面的环绕通知，即在Action方法的执行之前和之后根据需要添加相应的逻辑。事实上，即使struts.xml没有任何关于拦截器的配置，Struts2也会为我们添加一组默认的拦截器，最常见的是，请求参数自动绑定到Action对应的字段上。 Struts2中自定义拦截器的步骤是：
make:cc 命令未找到解决方法 daizj linux 命令未知 make cc
安装rz sz程序时，报下面错误： [root@slave2 src]# make posix cc -O -DPOSIX -DMD=2 rz.c -o rz make: cc：命令未找到 make: *** [posix] 错误 127 系统：centos 6.6 环境：虚拟机错误原因：系统未安装gcc，这个是由于在安
Oracle之Job应用周凡杨 oracle job
最近写服务，服务上线后，需要写一个定时执行的SQL脚本，清理并更新数据库表里的数据，应用到了Oracle 的 Job的相关知识。在此总结一下。一：查看相关job信息 1、相关视图 dba_jobs all_jobs user_jobs dba_jobs_running 包含正在运行
多线程机制朱辉辉33 多线程
转至http://blog.csdn.net/lj70024/archive/2010/04/06/5455790.aspx 程序、进程和线程：程序是一段静态的代码，它是应用程序执行的蓝本。进程是程序的一次动态执行过程，它对应了从代码加载、执行至执行完毕的一个完整过程，这个过程也是进程本身从产生、发展至消亡的过程。线程是比进程更小的单位，一个进程执行过程中可以产生多个线程，每个线程有自身的
web报表工具FineReport使用中遇到的常见报错及解决办法（一）老A不折腾 web报表 finereport java报表报表工具
FineReport使用中遇到的常见报错及解决办法（一）这里写点抛砖引玉，希望大家能把自己整理的问题及解决方法晾出来，Mark一下，利人利己。出现问题先搜一下文档上有没有，再看看度娘有没有，再看看论坛有没有。有报错要看日志。下面简单罗列下常见的问题，大多文档上都有提到的。 1、address pool is full：含义：地址池满，连接数超过并发数上
mysql rpm安装后没有my.cnf 林鹤霄没有my.cnf
Linux下用rpm包安装的MySQL是不会安装/etc/my.cnf文件的，至于为什么没有这个文件而MySQL却也能正常启动和作用，在这儿有两个说法，第一种说法，my.cnf只是MySQL启动时的一个参数文件，可以没有它，这时MySQL会用内置的默认参数启动，第二种说法，MySQL在启动时自动使用/usr/share/mysql目录下的my-medium.cnf文件，这种说法仅限于r
Kindle Fire HDX root并安装谷歌服务框架之后仍无法登陆谷歌账号的问题 aigo root
原文：http://kindlefireforkid.com/how-to-setup-a-google-account-on-amazon-fire-tablet/ Step 4: Run ADB command from your PC On the PC, you need install Amazon Fire ADB driver and instal
javascript 中var提升的典型实例 alxw4616 JavaScript
// 刚刚在书上看到的一个小问题,很有意思.大家一起思考下吧 myname = 'global'; var fn = function () { console.log(myname); // undefined var myname = 'local'; console.log(myname); // local }; fn() // 上述代码实际上等同于以下代码 m
定时器和获取时间的使用百合不是茶时间的转换定时器
定时器:定时创建任务在游戏设计的时候用的比较多 Timer();定时器 TImerTask();Timer的子类由 Timer 安排为一次执行或重复执行的任务。定时器类Timer在java.util包中。使用时，先实例化，然后使用实例的schedule(TimerTask task, long delay)方法，设定
JDK1.5 Queue bijian1013 java thread java多线程 Queue
JDK1.5 Queue LinkedList： LinkedList不是同步的。如果多个线程同时访问列表，而其中至少一个线程从结构上修改了该列表，则它必须保持外部同步。（结构修改指添加或删除一个或多个元素的任何操作；仅设置元素的值不是结构修改。）这一般通过对自然封装该列表的对象进行同步操作来完成。如果不存在这样的对象，则应该使用 Collections.synchronizedList 方
http认证原理和https bijian1013 http https
一.基础介绍在URL前加https://前缀表明是用SSL加密的。你的电脑与服务器之间收发的信息传输将更加安全。 Web服务器启用SSL需要获得一个服务器证书并将该证书与要使用SSL的服务器绑定。 http和https使用的是完全不同的连接方式，用的端口也不一样,前者是80，后
【Java范型五】范型继承 bit1129 java
定义如下一个抽象的范型类，其中定义了两个范型参数，T1，T2 package com.tom.lang.generics; public abstract class SuperGenerics<T1, T2> { private T1 t1; private T2 t2; public abstract void doIt(T
【Nginx六】nginx.conf常用指令(Directive) bit1129 Directive
1. worker_processes 8; 表示Nginx将启动8个工作者进程，通过ps -ef|grep nginx,会发现有8个Nginx Worker Process在运行 nobody 53879 118449 0 Apr22 ? 00:26:15 nginx: worker process
lua 遍历Header头部 ronin47 lua header 遍历　
local headers = ngx.req.get_headers() ngx.say("headers begin", "<br/>") ngx.say("Host : ", he
java-32.通过交换a,b中的元素，使[序列a元素的和]与[序列b元素的和]之间的差最小(两数组的差最小)。 bylijinnan java
import java.util.Arrays; public class MinSumASumB { /** * Q32.有两个序列a,b，大小都为n,序列元素的值任意整数，无序. * * 要求：通过交换a,b中的元素，使[序列a元素的和]与[序列b元素的和]之间的差最小。 * 例如: * int[] a = {100,99,98,1,2,3
redis 开窍的石头 redis
在redis的redis.conf配置文件中找到# requirepass foobared 把它替换成requirepass 12356789 后边的12356789就是你的密码打开redis客户端输入config get requirepass 返回 redis 127.0.0.1:6379> config get requirepass 1) "require
[JAVA图像与图形]现有的GPU架构支持JAVA语言吗？ comsci java语言
无论是opengl还是cuda，都是建立在C语言体系架构基础上的，在未来，图像图形处理业务快速发展，相关领域市场不断扩大的情况下，我们JAVA语言系统怎么从这么庞大，且还在不断扩大的市场上分到一块蛋糕，是值得每个JAVAER认真思考和行动的事情
安装ubuntu14.04登录后花屏了怎么办 cuiyadll ubuntu
这个情况，一般属于显卡驱动问题。可以先尝试安装显卡的官方闭源驱动。按键盘三个键：CTRL + ALT + F1 进入终端，输入用户名和密码登录终端：安装amd的显卡驱动 sudo apt-get install fglrx 安装nvidia显卡驱动 sudo ap
SSL 与数字证书的基本概念和工作原理 darrenzhu 加密 ssl 证书密钥签名
SSL 与数字证书的基本概念和工作原理 http://www.linuxde.net/2012/03/8301.html SSL握手协议的目的是或最终结果是让客户端和服务器拥有一个共同的密钥，握手协议本身是基于非对称加密机制的，之后就使用共同的密钥基于对称加密机制进行信息交换。 http://www.ibm.com/developerworks/cn/webspher
Ubuntu设置ip的步骤 dcj3sjt126com ubuntu
在单位的一台机器完全装了Ubuntu Server，但回家只能在XP上VM一个，装的时候网卡是DHCP的，用ifconfig查了一下ip是192.168.92.128,可以ping通。转载不是错： Ubuntu命令行修改网络配置方法 /etc/network/interfaces打开后里面可设置DHCP或手动设置静态ip。前面auto eth0，让网卡开机自动挂载. 1. 以D
php包管理工具推荐 dcj3sjt126com PHP Composer
http://www.phpcomposer.com/ Composer是 PHP 用来管理依赖（dependency）关系的工具。你可以在自己的项目中声明所依赖的外部工具库（libraries），Composer 会帮你安装这些依赖的库文件。中文文档入门指南下载安装包列表 Composer 中国镜像
Gson使用四（TypeAdapter） eksliang json gson Gson自定义转换器 gsonTypeAdapter
转载请出自出处：http://eksliang.iteye.com/blog/2175595 一.概述 Gson的TypeAapter可以理解成自定义序列化和返序列化二、应用场景举例例如我们通常去注册时（那些外国网站），会让我们输入firstName，lastName,但是转到我们都
JQM控件之Navbar和Tabs gundumw100 html xml css
在JQM中使用导航栏Navbar是简单的。只需要将data-role="navbar"赋给div即可： <div data-role="navbar"> <ul> <li><a href="#" class="ui-btn-active&qu
利用归并排序算法对大文件进行排序 iwindyforest java 归并排序大文件分治法 Merge sort
归并排序算法介绍，请参照Wikipeida zh.wikipedia.org/wiki/%E5%BD%92%E5%B9%B6%E6%8E%92%E5%BA%8F 基本思想：大文件分割成行数相等的两个子文件，递归（归并排序）两个子文件，直到递归到分割成的子文件低于限制行数低于限制行数的子文件直接排序两个排序好的子文件归并到父文件直到最后所有排序好的父文件归并到输入
iOS UIWebView URL拦截啸笑天 UIWebView
本文译者：candeladiao，原文：URL filtering for UIWebView on the iPhone说明：译者在做app开发时，因为页面的javascript文件比较大导致加载速度很慢，所以想把javascript文件打包在app里，当UIWebView需要加载该脚本时就从app本地读取，但UIWebView并不支持加载本地资源。最后从下文中找到了解决方法，第一次翻译，难免有
索引的碎片整理SQL语句 macroli sql
SET NOCOUNT ON DECLARE @tablename VARCHAR (128) DECLARE @execstr VARCHAR (255) DECLARE @objectid INT DECLARE @indexid INT DECLARE @frag DECIMAL DECLARE @maxfrag DECIMAL --设置最大允许的碎片数量,超过则对索引进行碎片
Angularjs同步操作http请求with $promise qiaolevip 每天进步一点点学习永无止境 AngularJS 纵观千象
// Define a factory app.factory('profilePromise', ['$q', 'AccountService', function($q, AccountService) { var deferred = $q.defer(); AccountService.getProfile().then(function(res) {
hibernate联合查询问题 sxj19881213 sql Hibernate HQL 联合查询
最近在用hibernate做项目，遇到了联合查询的问题，以及联合查询中的N+1问题。针对无外键关联的联合查询，我做了HQL和SQL的实验，希望能帮助到大家。（我使用的版本是hibernate3.3.2） 1 几个常识：（1）hql中的几种join查询，只有在外键关联、并且作了相应配置时才能使用。（2）hql的默认查询策略，在进行联合查询时，会产
struts2.xml wuai struts
<?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE struts PUBLIC "-//Apache Software Foundation//DTD Struts Configuration 2.3//EN" "http://struts.apache