Linux Block IO子系统分析总结
1.read/write/fsync与fread/fwrite/fflush的关系和区别
可以用下图来表示：
总结如下：
read/write/fsync：
- Linux底层操作；
- 内核调用， 涉及到进程上下文的切换，即用户态到核心态的转换，这是个比较消耗性能的操作。
fread/fwrite/fflush：
- c语言标准规定的io流操作，建立在read/write/fsync之上
- 在用户层， 又增加了一层缓冲机制，用于减少内核调用次数，但是增加了一次内存拷贝
Musl libc中的实现：
setbuf:
fflush:
可见，fflush并非和直接的系统调用相对应，它的底层是通过write系统调用实现的。
而fsync是有实实在在的系统调用对应的
补充：
对于输入设备，调用fsync/fflush将清空相应的缓冲区，其内数据将被丢弃;
对于输出设备或磁盘文件，fflush只能保证数据到达内核缓冲区，并不能保证数据到达物理设备, 因此应该在调用fflush后，调用fsync(fileno(stream))，确保数据存入磁盘。
fflush函数和fsync函数对比如下：
1.fflush接受一个参数FILE *.
fflush(FILE *);
fflush是libc.a中提供的方法，是用来将流中未写的数据传送到内核。如果参数为null，将导致所有流冲洗。
fsync接受的时一个Int型的文件描述符。
fsync(int fd);
fsync是系统提供的系统调用。将数据写到磁盘上
2.fflush/fsync 功能区别
fflush:是把C库中的缓冲调用write函数写到磁盘[其实是写到内核的缓冲区]。
fsync：是把内核缓冲刷到磁盘上。
c库缓冲-----fflush---------〉内核缓冲--------fsync-----〉磁盘
进一步引申到sync()、fflush()、fsync()这3个函数的别
a、三者的用途不一样：
sync,是同步整个系统的磁盘数据的.
fsync是同步打开的一个文件到缓冲区数据到磁盘上.
fflush是刷新打开的流的.
b、同样是同步，但三者的同步等级不一样：.
sync, 将缓冲区数据写回磁盘, 保持同步.(无参数)
fsync, 将缓冲区的数据写到文件中.(有一个参数 int fd)
fflush, 将文件流里未写出的数据立刻写出
验证：
非O_DIRECT方式打开，不call fsync.
#include
#include
#include
#include
#include
#include
#include
//O_DIRECT
#define __USE_GNU 1
#include
#include
#include
#include
int main(int argc,char **argv)
{
int fdno;
//fdno = open("./fuck.bin", O_DIRECT|O_RDWR);
fdno = open("./fuck.bin", O_RDWR);
if(fdno < 0)
{
printf("%s line %d, open file failure.\n", __func__, __LINE__);
return -1;
}
lseek(fdno, 0, SEEK_SET);
int len = lseek(fdno,0,SEEK_END);
printf("%s line %d, len = %d.\n", __func__, __LINE__, len);
len = len + (getpagesize() - len%getpagesize());
printf("%s line %d, len = %d.\n", __func__, __LINE__, len);
unsigned char *p = malloc(len + getpagesize());
unsigned char *q = (unsigned char *)((unsigned long)p&(~(getpagesize()-1)));
p = q;
if(p == NULL)
{
printf("%s line %d, malloc failure.\n", __func__, __LINE__);
return -1;
}
memset(p, 0x00, len);
lseek(fdno, 0, SEEK_SET);
int reallen = read(fdno, p,len);
printf("%s line %d, reallen %d read status %s\n", __func__, __LINE__, reallen, strerror(errno));
reallen = write(fdno, p,len);
printf("%s line %d, reallen %d write status %s\n", __func__, __LINE__, reallen, strerror(errno));
//fsync(fdno);
close(fdno);
return 0;
} 打开块设备调试跟踪器
echo 1 > /proc/sys/vm/block_dump
可以看到，是通过flush线程的随即会刷写入磁盘的。
这次我们加上fsync调用：
#include
#include
#include
#include
#include
#include
#include
//O_DIRECT
#define __USE_GNU 1
#include
#include
#include
#include
int main(int argc,char **argv)
{
int fdno;
//fdno = open("./fuck.bin", O_DIRECT|O_RDWR);
fdno = open("./fuck.bin", O_RDWR);
if(fdno < 0)
{
printf("%s line %d, open file failure.\n", __func__, __LINE__);
return -1;
}
lseek(fdno, 0, SEEK_SET);
int len = lseek(fdno,0,SEEK_END);
printf("%s line %d, len = %d.\n", __func__, __LINE__, len);
len = len + (getpagesize() - len%getpagesize());
printf("%s line %d, len = %d.\n", __func__, __LINE__, len);
unsigned char *p = malloc(len + getpagesize());
unsigned char *q = (unsigned char *)((unsigned long)p&(~(getpagesize()-1)));
p = q;
if(p == NULL)
{
printf("%s line %d, malloc failure.\n", __func__, __LINE__);
return -1;
}
memset(p, 0x00, len);
lseek(fdno, 0, SEEK_SET);
int reallen = read(fdno, p,len);
printf("%s line %d, reallen %d read status %s\n", __func__, __LINE__, reallen, strerror(errno));
reallen = write(fdno, p,len);
printf("%s line %d, reallen %d write status %s\n", __func__, __LINE__, reallen, strerror(errno));
fsync(fdno);
close(fdno);
return 0;
} 
可以看到,fsync直接通过同名系统调用将数据会刷到磁盘上的。
fdatasync:类似于fsync，但是它只更新文件的数据部分，而除数据外，fsync还更新文件的元数据部分。
#include
#include
#include
#include
#include
#include
#include
//O_DIRECT
#define __USE_GNU 1
#include
#include
#include
#include
int main(int argc,char **argv)
{
int fdno;
//fdno = open("./fuck.bin", O_DIRECT|O_RDWR);
fdno = open("./fuck.bin", O_RDWR);
if(fdno < 0)
{
printf("%s line %d, open file failure.\n", __func__, __LINE__);
return -1;
}
lseek(fdno, 0, SEEK_SET);
int len = lseek(fdno,0,SEEK_END);
printf("%s line %d, len = %d.\n", __func__, __LINE__, len);
len = len + (getpagesize() - len%getpagesize());
printf("%s line %d, len = %d.\n", __func__, __LINE__, len);
unsigned char *p = malloc(len + getpagesize());
unsigned char *q = (unsigned char *)((unsigned long)p&(~(getpagesize()-1)));
p = q;
if(p == NULL)
{
printf("%s line %d, malloc failure.\n", __func__, __LINE__);
return -1;
}
memset(p, 0x00, len);
lseek(fdno, 0, SEEK_SET);
int reallen = read(fdno, p,len);
printf("%s line %d, reallen %d read status %s\n", __func__, __LINE__, reallen, strerror(errno));
reallen = write(fdno, p,len);
printf("%s line %d, reallen %d write status %s\n", __func__, __LINE__, reallen, strerror(errno));
//fsync(fdno);
fdatasync(fdno);
close(fdno);
return 0;
} 关于用法，一段外文讲解：
int fflush(FILE *stream);
If stream points to an output stream or an update stream in which the most recent operation was not input, the fflush function causes any unwritten data for that stream to be delivered to the host environment to be written to the file; otherwise, the behavior is undefined.
如果 stream 指向输出流或者更新流（update stream），并且这个更新流最近执行的操作不是输入，那么 fflush 函数将把这个流中任何待写数据传送至宿主环境（host environment）写入文件。否则，它的行为是未定义的。
(宿主环境可以理解为操作系统或内核).
所以如果 stream 指向输入流（如 stdin），那么 fflush 函数的行为是不确定的。故而使用fflush(stdin)是不正确的。
The function fflush forces a write of all user-space buffered data for the given output or update stream via the stream underlying write function. The open status of the stream is unaffected. If the stream argument is NULL, fflush flushes all open output streams.
Note that fflush() only flushes the user space buffers provided by the C library. To ensure that the data is physically stored on disk the kernel buffers must be flushed too, e.g. with sync(2) or fsync(2).
int fsync(int fd);
fsync copies all in-core parts of a file to disk, and waits until the device reports that all parts are on stable storage. It also updates metadata stat information. It does not necessarily ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync on the file descriptor of the directory is also needed.
fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) where that file resides. The call blocks until the device reports that the transfer has completed. It also flushes metadata information associated with the file (see stat(2)).
1.提供者fflush是libc.a中提供的方法，fsync是系统提供的系统调用。
2.原形fflush接受一个参数FILE *.fflush(FILE *);fsync接受的时一个Int型的文件描述符。fsync(int fd);
3.功能fflush:是把C库中的缓冲调用write函数写到磁盘[其实是写到内核的缓冲区]。fsync：是把内核缓冲刷到磁盘上。
4.fsync 将文件相关的所有更改都发送到disk device。 这个调用是阻塞的，直到disk通知此函数传输完成。此函数也会将该文件的文件信息flush到disk。
5.fsync最终将缓冲的数据更新到文件里。
所以可以看出fflush和fsync的调用顺序应该是：
c库缓冲-----fflush---------〉内核缓冲--------fsync-----〉磁盘
Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync() on a file descriptor for the directory is also needed.
fdatasync() is similar to fsync(), but does not flush modified metadata unless that metadata is needed in order to allow a subsequent data retrieval to be correctly handled. For example, changes to st_atime or st_mtime (respectively, time of last access and time of last modification; see stat(2)) do not not require flushing because they are not necessary for a subsequent data read to be handled correctly. On the other hand, a change to the file size (st_size, as made by say ftruncate(2)), would require a metadata flush.
The aim of fdatasync(2) is to reduce disk activity for applications that do not require all metadata to be synchronised with the disk.
fdatasync与fsync的区别在于fdatasync不会flush文件的metadata信息。这个是为了减少对磁盘的操作
Notes
If the underlying hard disk has write caching enabled , then the data may not really be on permanent storage when fsync() / fdatasync() return. 如果是O_DIRECT模式，do_direct_IO函数在调用链中执行下面的函数，得到用户态PAGE对应的物理PAGE。
可以同时用,O_DIRECT和O_SYNC,因为虽然O_DIRECT没有走PAGE CACHE,但是在 data sync的时候,并不是通过page cache找到buffer head,而是直接找到buffer head,方法是通过 address_space->private_list找到的
