Linux内核IPC源码——共享内存

介绍

我看的是linux-4.2.3的源码。参考了《边干边学——Linux内核指导》（鬼畜的书名）第16章内容，他们用的是2.6.15的内核源码。

现在linux中可以使用共享内存的方式有两种

1. POSIX的shm_open()在/dev/shm/下打开一个文件，用mmap()映射到进程自己的内存地址

2. System V的shmget()得到一个共享内存对象的id，用shmat()映射到进程自己的内存地址

POSIX的实现是基于tmpfs的，函数都写在libc里，没什么好说的，主要还是看System V的实现方式。在System V中共享内存属于IPC子系统。所谓ipc，就是InterProcess Communication即进程间通信的意思，System V比前面的Unix增加了3中进程间通信的方式，共享内存、消息队列、信号量，统称IPC。主要代码在以下文件中

• ipc/shm.c

• include/linux/shm.h

• ipc/util.c

• ipc/util.h

• include/linux/ipc.h

同一块共享内存在内核中至少有3个标识符

1. IPC对象id（IPC对象是保存IPC信息的数据结构）

2. 进程虚拟内存中文件的inode，即每个进程中的共享内存也是以文件的方式存在的，但并不是显式的。可以通过某个vm_area_struct->vm_file->f_dentry->d_inode->i_ino表示

3. IPC对象的key。如果在shmget()中传入同一个key可以获取到同一块共享内存。但由于key是用户指定的，可能重复，而且也很少程序写之前会约定一个key，所以这种方法不是很常用。通常System V这种共享内存的方式是用于有父子关系的进程的。或者用ftok()函数用路径名来生成一个key。

首先看一下在内核中表示一块共享内存的数据结构，在include/linux/shm.h中

/* */是内核源码的注释，// 是我的注释

1. struct shmid_kernel /* private to the kernel */

2. {

3. struct kern_ipc_perm shm_perm; // 权限，这个结构体中还有一些重要的内容，后面会提到

4. struct file *shm_file; // 表示这块共享内存的内核文件，文件内容即共享内存的内容

5. unsigned long shm_nattch; // 连接到这块共享内存的进程数

6. unsigned long shm_segsz; // 大小，字节为单位

7. time_t shm_atim; // 最后一次连接时间

8. time_t shm_dtim; // 最后一次断开时间

9. time_t shm_ctim; // 最后一次更改信息的时间

10. pid_t shm_cprid; // 创建者进程id

11. pid_t shm_lprid; // 最后操作者进程id

12. struct user_struct *mlock_user;

14. /* The task created the shm object. NULL if the task is dead. */

15. struct task_struct *shm_creator;

16. struct list_head shm_clist; /* list by creator */

17. };

再看一下struct shmid_kernel中存储权限信息的shm_perm，在include/linux/ipc.h中

1. /* used by in-kernel data structures */

2. struct kern_ipc_perm

3. {

4. spinlock_t lock;

5. bool deleted;

6. int id; // IPC对象id

7. key_t key; // IPC对象键值，即创建共享内存时用户指定的

8. kuid_t uid; // IPC对象拥有者id

9. kgid_t gid; // 组id

10. kuid_t cuid; // 创建者id

11. kgid_t cgid;

12. umode_t mode;

13. unsigned long seq;

14. void *security;

15. };

为啥有这样一个struct呢？因为这些权限、id、key是IPC对象都有的属性，所以比如表示semaphore的结构struct semid_kernel中也有一个这样的struct kern_ipc_perm。然后在传递IPC对象的时候，传的也是struct kern_ipc_perm的指针，再用container_of这样的宏获得外面的struct，这样就能用同一个函数操作3种IPC对象，达到较好的代码重用。

接下来我们看一下共享内存相关函数。首先它们都是系统调用，对应的用户API在libc里面，参数是相同的，只是libc中的API做了一些调用系统调用需要的日常工作（保护现场、恢复现场之类的），所以就直接看这个系统调用了。

声明在include/linux/syscalls.h中

1. asmlinkage long sys_shmat(int shmid, char __user *shmaddr, int shmflg);

2. asmlinkage long sys_shmget(key_t key, size_t size, int flag);

3. asmlinkage long sys_shmdt(char __user *shmaddr);

4. asmlinkage long sys_shmctl(int shmid, int cmd, struct shmid_ds __user *buf);

定义在ipc/shm.c中

shmget

1. SYSCALL_DEFINE3(shmget, key_t, key, size_t, size, int, shmflg)

2. {

3. struct ipc_namespace *ns;

4. static const struct ipc_ops shm_ops = {

5. .getnew = newseg,

6. .associate = shm_security,

7. .more_checks = shm_more_checks,

8. };

9. struct ipc_params shm_params;

11. ns = current->nsproxy->ipc_ns;

13. shm_params.key = key;

14. shm_params.flg = shmflg;

15. shm_params.u.size = size;

17. return ipcget(ns, &shm_ids(ns), &shm_ops, &shm_params);

18. }

首先看到这个函数定义可能会很奇怪，不过这个SYSCALL_DEFINE3的宏展开来最后形式肯定和.h文件中声明的一样，即还是long sys_shmget(key_t key, size_t size, int flag)这个宏是为了修一个bug，纯粹黑科技，这里不提它。

然后这里实际调用的函数是ipcget()。为了统一一个ipc的接口也是煞费苦心，共享内存、信号量、消息队列三种对象创建的时候都会调用这个函数，但其实创建的逻辑并不在这里。而在shm_ops中的三个函数里。

namespace

顺便提一下其中的current->nsproxy->ipc_ns。这个的类型是struct ipc_namespace。它是啥呢？我们知道，共享内存这些进程间通信的数据结构是全局的，但有时候需要把他们隔离开，即某一组进程并不知道另外的进程的共享内存，它们只希望在组内共用这些东西，这样就不会与其他进程冲突。于是就煞费苦心在内核中加了一个namespace。只要在clone()函数中加入CLONE_NEWIPC标志就能创建一个新的IPC namespace。

那么这个IPC namespace和我们的共享内存的数据结构有什么关系呢，可以看一下结构体

1. struct ipc_ids {

2. int in_use;

3. unsigned short seq;

4. struct rw_semaphore rwsem;

5. struct idr ipcs_idr;

6. int next_id;

7. };

9. struct ipc_namespace {

10. atomic_t count;

11. struct ipc_ids ids[3];

12. ...

13. };

比较重要的是其中的ids，它存的是所用IPC对象的id，其中共享内存都存在ids[2]中。而在ids[2]中真正负责管理数据的是ipcs_idr，它也是内核中一个煞费苦心弄出来的id管理机制，一个id可以对应任意唯一确定的对象。把它理解成一个数组就好。它们之间的关系大概如下图所示。

1. [0] struct kern_ipc_perm <==> struct shmid_kernel

2. struct ipc_namespace => struct ipc_ids => struct idr => [1] struct kern_ipc_perm <==> struct shmid_kernel

3. [2] struct kern_ipc_perm <==> struct shmid_kernel

回到shmget

好的，我们回头来看看shmget()究竟干了啥，首先看一下ipcget()

1. int ipcget(struct ipc_namespace *ns, struct ipc_ids *ids,

2. const struct ipc_ops *ops, struct ipc_params *params)

3. {

4. if (params->key == IPC_PRIVATE)

5. return ipcget_new(ns, ids, ops, params);

6. else

7. return ipcget_public(ns, ids, ops, params);

8. }

如果传进来的参数是IPC_PRIVATE（这个宏的值是0）的话，无论是什么mode，都会创建一块新的共享内存。如果非0，则会去已有的共享内存中找有没有这个key的，有就返回，没有就新建。

首先看一下新建的函数newseg()

1. static int newseg(struct ipc_namespace *ns, struct ipc_params *params)

2. {

3. key_t key = params->key;

4. int shmflg = params->flg;

5. size_t size = params->u.size;

6. int error;

7. struct shmid_kernel *shp;

8. size_t numpages = (size + PAGE_SIZE - 1) >> PAGE_SHIFT;

9. struct file *file;

10. char name[13];

11. int id;

12. vm_flags_t acctflag = 0;

14. if (size < SHMMIN || size > ns->shm_ctlmax)

15. return -EINVAL;

17. if (numpages << PAGE_SHIFT < size)

18. return -ENOSPC;

20. if (ns->shm_tot + numpages < ns->shm_tot ||

21. ns->shm_tot + numpages > ns->shm_ctlall)

22. return -ENOSPC;

24. shp = ipc_rcu_alloc(sizeof(*shp));

25. if (!shp)

26. return -ENOMEM;

28. shp->shm_perm.key = key;

29. shp->shm_perm.mode = (shmflg & S_IRWXUGO);

30. shp->mlock_user = NULL;

32. shp->shm_perm.security = NULL;

33. error = security_shm_alloc(shp);

34. if (error) {

35. ipc_rcu_putref(shp, ipc_rcu_free);

36. return error;

37. }

39. sprintf(name, "SYSV%08x", key);

40. if (shmflg & SHM_HUGETLB) {

41. struct hstate *hs;

42. size_t hugesize;

44. hs = hstate_sizelog((shmflg >> SHM_HUGE_SHIFT) & SHM_HUGE_MASK);

45. if (!hs) {

46. error = -EINVAL;

47. goto no_file;

48. }

49. hugesize = ALIGN(size, huge_page_size(hs));

51. /* hugetlb_file_setup applies strict accounting */

52. if (shmflg & SHM_NORESERVE)

53. acctflag = VM_NORESERVE;

54. file = hugetlb_file_setup(name, hugesize, acctflag,

55. &shp->mlock_user, HUGETLB_SHMFS_INODE,

56. (shmflg >> SHM_HUGE_SHIFT) & SHM_HUGE_MASK);

57. } else {

58. /*

59. * Do not allow no accounting for OVERCOMMIT_NEVER, even

60. * if it's asked for.

61. */

62. if ((shmflg & SHM_NORESERVE) &&

63. sysctl_overcommit_memory != OVERCOMMIT_NEVER)

64. acctflag = VM_NORESERVE;

65. file = shmem_kernel_file_setup(name, size, acctflag);

66. }

67. error = PTR_ERR(file);

68. if (IS_ERR(file))

69. goto no_file;

71. id = ipc_addid(&shm_ids(ns), &shp->shm_perm, ns->shm_ctlmni);

72. if (id < 0) {

73. error = id;

74. goto no_id;

75. }

77. shp->shm_cprid = task_tgid_vnr(current);

78. shp->shm_lprid = 0;

79. shp->shm_atim = shp->shm_dtim = 0;

80. shp->shm_ctim = get_seconds();

81. shp->shm_segsz = size;

82. shp->shm_nattch = 0;

83. shp->shm_file = file;

84. shp->shm_creator = current;

85. list_add(&shp->shm_clist, ¤t->sysvshm.shm_clist);

87. /*

88. * shmid gets reported as "inode#" in /proc/pid/maps.

89. * proc-ps tools use this. Changing this will break them.

90. */

91. file_inode(file)->i_ino = shp->shm_perm.id;

93. ns->shm_tot += numpages;

94. error = shp->shm_perm.id;

96. ipc_unlock_object(&shp->shm_perm);

97. rcu_read_unlock();

98. return error;

100. no_id:

101. if (is_file_hugepages(file) && shp->mlock_user)

102. user_shm_unlock(size, shp->mlock_user);

103. fput(file);

104. no_file:

105. ipc_rcu_putref(shp, shm_rcu_free);

106. return error;

107. }

这个函数首先几个if检查size是不是合法的参数，并且检查有没有足够的pages。然后调用ipc_rcu_alloc()函数给共享内存数据结构shp分配空间。然后把一些参数写到shp的shm_perm成员中。然后sprintf下面那个大的if-else是为表示共享内存内容的file分配空间。再然后ipc_addid()是一个比较重要的函数，它把刚才新建的这个共享内存的数据结构的指针加入到namespace的ids里，即可以想象成加入到数组里，并获得一个可以找到它的id。这里的id并不完全是数组的下标，因为要避免重复，所以这里有一个简单的机制来保证生成的id几乎是unique的，即ids里面有个seq变量，每次新加入共享内存对象时都会加1，而真正的id是这样生成的SEQ_MULTIPLIER * seq + id。然后初始化一些成员，再把这个数据结构的指针加到当前进程的一个list里。这个函数的工作就基本完成了。

接下来我们再看一下如果创建时传入一个已有的key，即ipcget_public()的逻辑

1. static int ipcget_public(struct ipc_namespace *ns, struct ipc_ids *ids,

2. const struct ipc_ops *ops, struct ipc_params *params)

3. {

4. struct kern_ipc_perm *ipcp;

5. int flg = params->flg;

6. int err;

8. /*

9. * Take the lock as a writer since we are potentially going to add

10. * a new entry + read locks are not "upgradable"

11. */

12. down_write(&ids->rwsem);

13. ipcp = ipc_findkey(ids, params->key);

14. if (ipcp == NULL) {

15. /* key not used */

16. if (!(flg & IPC_CREAT))

17. err = -ENOENT;

18. else

19. err = ops->getnew(ns, params);

20. } else {

21. /* ipc object has been locked by ipc_findkey() */

23. if (flg & IPC_CREAT && flg & IPC_EXCL)

24. err = -EEXIST;

25. else {

26. err = 0;

27. if (ops->more_checks)

28. err = ops->more_checks(ipcp, params);

29. if (!err)

30. /*

31. * ipc_check_perms returns the IPC id on

32. * success

33. */

34. err = ipc_check_perms(ns, ipcp, ops, params);

35. }

36. ipc_unlock(ipcp);

37. }

38. up_write(&ids->rwsem);

40. return err;

41. }

逻辑非常简单，先去找有没有这个key。没有的话还是创建一个新的，注意ops->getnew()对应的就是刚才的newseg()函数。如果找到了就判断一下权限有没有问题，没有问题就直接返回IPC id。

可以再看下ipc_findkey()这个函数

1. static struct kern_ipc_perm *ipc_findkey(struct ipc_ids *ids, key_t key)

2. {

3. struct kern_ipc_perm *ipc;

4. int next_id;

5. int total;

7. for (total = 0, next_id = 0; total < ids->in_use; next_id++) {

8. ipc = idr_find(&ids->ipcs_idr, next_id);

10. if (ipc == NULL)

11. continue;

13. if (ipc->key != key) {

14. total++;

15. continue;

16. }

18. rcu_read_lock();

19. ipc_lock_object(ipc);

20. return ipc;

21. }

23. return NULL;

24. }

逻辑也很简单，注意到ids->ipcs_idr就是之前提到的Interger ID Managenent机制，里面存的就是shmid和对象一一对应的关系。然后这里可以看到ids->in_use表示的是共享内存的个数，由于中间的有些可能删掉了，所以total在找到一个不为空的共享内存的时候才++。然后我们也可以看到，这里对重复的key并没有做任何处理。所以我们在编程的时候也应该避免直接约定用某一个数字当key。

shmat

接下来我们看一下shmat()，它的逻辑全在do_shmat()中，所以我们直接看这个函数。

1. long do_shmat(int shmid, char __user *shmaddr, int shmflg, ulong *raddr,

2. unsigned long shmlba)

3. {

4. struct shmid_kernel *shp;

5. unsigned long addr;

6. unsigned long size;

7. struct file *file;

8. int err;

9. unsigned long flags;

10. unsigned long prot;

11. int acc_mode;

12. struct ipc_namespace *ns;

13. struct shm_file_data *sfd;

14. struct path path;

15. fmode_t f_mode;

16. unsigned long populate = 0;

18. err = -EINVAL;

19. if (shmid < 0)

20. goto out;

21. else if ((addr = (ulong)shmaddr)) {

22. if (addr & (shmlba - 1)) {

23. if (shmflg & SHM_RND)

24. addr &= ~(shmlba - 1); /* round down */

25. else

26. #ifndef __ARCH_FORCE_SHMLBA

27. if (addr & ~PAGE_MASK)

28. #endif

29. goto out;

30. }

31. flags = MAP_SHARED | MAP_FIXED;

32. } else {

33. if ((shmflg & SHM_REMAP))

34. goto out;

36. flags = MAP_SHARED;

37. }

39. if (shmflg & SHM_RDONLY) {

40. prot = PROT_READ;

41. acc_mode = S_IRUGO;

42. f_mode = FMODE_READ;

43. } else {

44. prot = PROT_READ | PROT_WRITE;

45. acc_mode = S_IRUGO | S_IWUGO;

46. f_mode = FMODE_READ | FMODE_WRITE;

47. }

48. if (shmflg & SHM_EXEC) {

49. prot |= PROT_EXEC;

50. acc_mode |= S_IXUGO;

51. }

53. /*

54. * We cannot rely on the fs check since SYSV IPC does have an

55. * additional creator id...

56. */

57. ns = current->nsproxy->ipc_ns;

58. rcu_read_lock();

59. shp = shm_obtain_object_check(ns, shmid);

60. if (IS_ERR(shp)) {

61. err = PTR_ERR(shp);

62. goto out_unlock;

63. }

65. err = -EACCES;

66. if (ipcperms(ns, &shp->shm_perm, acc_mode))

67. goto out_unlock;

69. err = security_shm_shmat(shp, shmaddr, shmflg);

70. if (err)

71. goto out_unlock;

73. ipc_lock_object(&shp->shm_perm);

75. /* check if shm_destroy() is tearing down shp */

76. if (!ipc_valid_object(&shp->shm_perm)) {

77. ipc_unlock_object(&shp->shm_perm);

78. err = -EIDRM;

79. goto out_unlock;

80. }

82. path = shp->shm_file->f_path;

83. path_get(&path);

84. shp->shm_nattch++;

85. size = i_size_read(d_inode(path.dentry));

86. ipc_unlock_object(&shp->shm_perm);

87. rcu_read_unlock();

89. err = -ENOMEM;

90. sfd = kzalloc(sizeof(*sfd), GFP_KERNEL);

91. if (!sfd) {

92. path_put(&path);

93. goto out_nattch;

94. }

96. file = alloc_file(&path, f_mode,

97. is_file_hugepages(shp->shm_file) ?

98. &shm_file_operations_huge :

99. &shm_file_operations);

100. err = PTR_ERR(file);

101. if (IS_ERR(file)) {

102. kfree(sfd);

103. path_put(&path);

104. goto out_nattch;

105. }

107. file->private_data = sfd;

108. file->f_mapping = shp->shm_file->f_mapping;

109. sfd->id = shp->shm_perm.id;

110. sfd->ns = get_ipc_ns(ns);

111. sfd->file = shp->shm_file;

112. sfd->vm_ops = NULL;

114. err = security_mmap_file(file, prot, flags);

115. if (err)

116. goto out_fput;

118. down_write(¤t->mm->mmap_sem);

119. if (addr && !(shmflg & SHM_REMAP)) {

120. err = -EINVAL;

121. if (addr + size < addr)

122. goto invalid;

124. if (find_vma_intersection(current->mm, addr, addr + size))

125. goto invalid;

126. }

128. addr = do_mmap_pgoff(file, addr, size, prot, flags, 0, &populate);

129. *raddr = addr;

130. err = 0;

131. if (IS_ERR_VALUE(addr))

132. err = (long)addr;

133. invalid:

134. up_write(¤t->mm->mmap_sem);

135. if (populate)

136. mm_populate(addr, populate);

138. out_fput:

139. fput(file);

141. out_nattch:

142. down_write(&shm_ids(ns).rwsem);

143. shp = shm_lock(ns, shmid);

144. shp->shm_nattch--;

145. if (shm_may_destroy(ns, shp))

146. shm_destroy(ns, shp);

147. else

148. shm_unlock(shp);

149. up_write(&shm_ids(ns).rwsem);

150. return err;

152. out_unlock:

153. rcu_read_unlock();

154. out:

155. return err;

156. }

首先检查shmaddr的合法性并进行对齐，即调整为shmlba的整数倍。如果传入addr是0，前面检查部分只会加上一个MAP_SHARED标志，因为后面的mmap会自动为其分配地址。然后从那一段两行的注释开始，函数通过shmid尝试获取共享内存对象，并进行权限检查。然后修改shp中的一些数据，比如连接进程数加一。然后是通过alloc_file()创建真正的要做mmap的file。在mmap之前还要对地址空间进行检查，检查是否和别的地址重叠，是否够用。实际的映射工作就在do_mmap_pgoff()函数中做了。

shmdt

1. SYSCALL_DEFINE1(shmdt, char __user *, shmaddr)

2. {

3. struct mm_struct *mm = current->mm;

4. struct vm_area_struct *vma;

5. unsigned long addr = (unsigned long)shmaddr;

6. int retval = -EINVAL;

7. #ifdef CONFIG_MMU

8. loff_t size = 0;

9. struct file *file;

10. struct vm_area_struct *next;

11. #endif

13. if (addr & ~PAGE_MASK)

14. return retval;

16. down_write(&mm->mmap_sem);

18. /*

19. * This function tries to be smart and unmap shm segments that

20. * were modified by partial mlock or munmap calls:

21. * - It first determines the size of the shm segment that should be

22. * unmapped: It searches for a vma that is backed by shm and that

23. * started at address shmaddr. It records it's size and then unmaps

24. * it.

25. * - Then it unmaps all shm vmas that started at shmaddr and that

26. * are within the initially determined size and that are from the

27. * same shm segment from which we determined the size.

28. * Errors from do_munmap are ignored: the function only fails if

29. * it's called with invalid parameters or if it's called to unmap

30. * a part of a vma. Both calls in this function are for full vmas,

31. * the parameters are directly copied from the vma itself and always

32. * valid - therefore do_munmap cannot fail. (famous last words?)

33. */

34. /*

35. * If it had been mremap()'d, the starting address would not

36. * match the usual checks anyway. So assume all vma's are

37. * above the starting address given.

38. */

39. vma = find_vma(mm, addr);

41. #ifdef CONFIG_MMU

42. while (vma) {

43. next = vma->vm_next;

45. /*

46. * Check if the starting address would match, i.e. it's

47. * a fragment created by mprotect() and/or munmap(), or it

48. * otherwise it starts at this address with no hassles.

49. */

50. if ((vma->vm_ops == &shm_vm_ops) &&

51. (vma->vm_start - addr)/PAGE_SIZE == vma->vm_pgoff) {

53. /*

54. * Record the file of the shm segment being

55. * unmapped. With mremap(), someone could place

56. * page from another segment but with equal offsets

57. * in the range we are unmapping.

58. */

59. file = vma->vm_file;

60. size = i_size_read(file_inode(vma->vm_file));

61. do_munmap(mm, vma->vm_start, vma->vm_end - vma->vm_start);

62. /*

63. * We discovered the size of the shm segment, so

64. * break out of here and fall through to the next

65. * loop that uses the size information to stop

66. * searching for matching vma's.

67. */

68. retval = 0;

69. vma = next;

70. break;

71. }

72. vma = next;

73. }

75. /*

76. * We need look no further than the maximum address a fragment

77. * could possibly have landed at. Also cast things to loff_t to

78. * prevent overflows and make comparisons vs. equal-width types.

79. */

80. size = PAGE_ALIGN(size);

81. while (vma && (loff_t)(vma->vm_end - addr) <= size) {

82. next = vma->vm_next;

84. /* finding a matching vma now does not alter retval */

85. if ((vma->vm_ops == &shm_vm_ops) &&

86. ((vma->vm_start - addr)/PAGE_SIZE == vma->vm_pgoff) &&

87. (vma->vm_file == file))

88. do_munmap(mm, vma->vm_start, vma->vm_end - vma->vm_start);

89. vma = next;

90. }

92. #else /* CONFIG_MMU */

93. /* under NOMMU conditions, the exact address to be destroyed must be

94. * given */

95. if (vma && vma->vm_start == addr && vma->vm_ops == &shm_vm_ops) {

96. do_munmap(mm, vma->vm_start, vma->vm_end - vma->vm_start);

97. retval = 0;

98. }

100. #endif

102. up_write(&mm->mmap_sem);

103. return retval;

104. }

接下来是shmdt()，这个函数非常简单，找到传入的shmaddr对应的虚拟内存数据结构vma，检查它的地址是不是正确的，然后调用do_munmap()函数断开对共享内存的连接。注意此操作并不会销毁共享内存，即使没有进程连接到它也不会，只有手动调用shmctl(id, IPC_RMID, NULL)才能销毁。

shmctl()总体就是一个switch语句，大多数做的是读取信息的或者设置标志位的工作，这里不赘述。