NFS Client in Linux Kernel - Open

1. Callstack for Opening for a file

open() => nfs4_file_open() => nfs4_atomic_open() => nfs4_do_open() => _nfs4_do_open()

_nfs4_do_open分析:

1. nfs4_get_state_owner()获得open owner对象
2. 设置claim，他有几种可能CLAIM_NULL(文件用文件名描述)，CLAIM_FH(文件用current filehandle描述)， CLAIM_PREVIOUS(用于reclaim open)
3. nfs4_opendata_alloc()分配nfs4_opendata
4. _nfs4_open_and_get_state()发送OPEN NFS命令
   4.1 _nfs4_proc_open, 发送OPEN NFS命令
   4.2 nfs4_opendata_to_nfs4_state() =>_nfs4_opendata_to_nfs4_state(), 创建open state,并将OPEN NFS命令得到的open state id，跟新这里。
5. open state存在ctx->state，其中ctx是nfs_open_context，它存储在filp->private_data。对文件open以后，就可以通过filp->private_data得到ctx，并得到ctx->state->open_stateid，通过这个stateid向NFS Server发送READ/WRITE/LOCK操作。

2. NFS协议中的Open owner

NFS协议规定发送OPEN时候要发送一个open owner，标识这个open的owner。在Server看来，clientid可以区分是来自哪个client的，open owner可以区分不同进程的open请求。
摘自[nfs4.1协议]https://tools.ietf.org/html/rfc5661
NFS协议中Open owner和Lock owner都用下面这个结构体表达。

   struct state_owner4 {
           clientid4       clientid;
           opaque          owner;
   };

1. clientid,Server分配的，唯一标识一个和server建立session的client
2. 字符串+字符串的长度,唯一标识client中的一个进程
3. client标识+进程标识，唯一确定一个lock的owner。

### 2.1 Linux Kernel中对Open owner的设置
static inline void encode_openhdr(struct xdr_stream *xdr, const struct nfs_openargs *arg)
{
...
    *p++ = cpu_to_be32(24);
    p = xdr_encode_opaque_fixed(p, "open id:", 8);
    *p++ = cpu_to_be32(arg->server->s_dev);
    *p++ = cpu_to_be32(arg->id.uniquifier);
    xdr_encode_hyper(p, arg->id.create_time);
}

Lock owner大小24字节:

1. 8字节的固定字符，"lock id:"
2. 4字节的s_dev，表示是本地哪个文件系统。
3. 4字节的id。由ida_get_new函数创建，在本地文件系统中唯一。
4. 8字节的create_time
5. 以上三点，保证了在一个client内Lock owner是唯一的。
6. 再加上clientid，可以保证所有和Server建立session连接的client来自的lock owner是唯一确定。

3. Linux Kernel对一个打开实体Owner的描述

struct nfs4_state_owner {
    struct nfs_server    *so_server;
    struct list_head     so_lru;
    unsigned long        so_expires;
    struct rb_node       so_server_node;
    struct rpc_cred      *so_cred;   //以这个为key,插入到红黑树里
    spinlock_t       so_lock;
    atomic_t         so_count;
    unsigned long        so_flags;
    struct list_head     so_states;
    struct nfs_seqid_counter so_seqid;
    seqcount_t       so_reclaim_seqcount;
    struct mutex         so_delegreturn_mutex;
};

3.1 创建nfs4_state_owner

nfs4_get_state_owner()

1. 首先通过so_cred为key，在红黑树里搜索
2. 如果没有搜到，创建一个

4. Linux Kernel对一个打开实体的描述

这个数据结构专门描述open state，一个实体会被多个Owner打开，所以struct nfs4_state和struct nfs4_state_owner是一对多的关系

struct nfs4_state {
    struct list_head open_states;   //对于同一个inode,所有open state存在inode->open_states
    struct list_head inode_states;  //对于同一个owner,所有的open state存在owner->so_states
    struct list_head lock_states;   //这个open state下所有的lock state

    struct nfs4_state_owner *owner; /* Pointer to the open owner */
    struct inode *inode;        /* Pointer to the inode */

    unsigned long flags;        /* Do we hold any locks? */
    spinlock_t state_lock;      /* Protects the lock_states list */

    seqlock_t seqlock;      /* Protects the stateid/open_stateid */
    nfs4_stateid stateid;       /* Current stateid: may be delegation */
    nfs4_stateid open_stateid;  /* OPEN stateid */

    /* The following 3 fields are protected by owner->so_lock */
    unsigned int n_rdonly;      /* Number of read-only references */
    unsigned int n_wronly;      /* Number of write-only references */
    unsigned int n_rdwr;        /* Number of read/write references */
    fmode_t state;          /* State on the server (R,W, or RW) */
    atomic_t count;
};

4.1 创建nfs4_state

nfs4_get_open_state

1. 通过owner和inode的组合，试图寻找一下
2. 如果找不到创建一个和inode关联

4. open stateid for Read/Write/Lock/Setattr/Close

这些操作都是依靠open stateid来向Server交互的。
参见下列函数的实现

• encode_read()
• encode_write()
• encode_setattr()
• encode_close()
• encode_lock()
• encode_locku()

static void encode_read(struct xdr_stream *xdr, const struct nfs_pgio_args *args,
            struct compound_hdr *hdr)
{
    __be32 *p;
    encode_op_hdr(xdr, OP_READ, decode_read_maxsz, hdr);
    encode_nfs4_stateid(xdr, &args->stateid); //NFS Server需要通过stateid找到file owner
    p = reserve_space(xdr, 12);
    p = xdr_encode_hyper(p, args->offset);
    *p = cpu_to_be32(args->count);
}

调用栈

//open for file
nfs4_file_open <= sys_open
  open_context // nfs4_atomic_open
    nfs4_do_open
      _nfs4_do_open
        nfs4_recover_expired_lease
        //如果当前Dentry访问过,则claim = NFS4_OPEN_CLAIM_FH
        nfs4_opendata_alloc
        _nfs4_open_and_get_state
          _nfs4_proc_open
            nfs4_run_open_task //send OPEN request to server

nfs4_open_prepare
  //如果claim == NFS4_OPEN_CLAIM_FH，将OPEN修改成OPEN_NOATTR

//open for dir
nfs_opendir <= sys_open
  if (filp->f_path.dentry == filp->f_path.mnt->mnt_root)
    __nfs_revalidate_inode
      getattr

//read dir
nfs_readdir <= sys_getdents
  readdir_search_pagecache
    find_cache_page
      get_cache_page
        => nfs_readdir_filler
          nfs_readdir_xdr_to_array
            nfs_readdir_xdr_filler
              readdir //nfs4_proc_readdir
                _nfs4_proc_readdir //send READDIR
      nfs_readdir_search_array