How to create DBFS file system

   DBFS是oracle Exadata提供的一种的集群文件系统，他是通过创建一个表空间，并将该表空间通过一系列操作，挂载成为文件系统，为Exadata各个节点提供文件系统服务。
   DBFS一般都是在客户初次部署的时候，由oracle远程工程师通过固定的脚本，自动化部署全部的操作系统数据库的同时，将其建立。
   这里说到的就是，在之前客户没有规划DBFS文件系统，但是后期打算手动添加的方法。
  我的OS oracle linux

1.创建fuse用户组，并将oracle添加到这个组内。(注：dcli -g ~/dbs_group -l表示后面的命令式在~/dbs_group中列出的全部服务器上执行)

(root)# dcli -g ~/dbs_group -l root usermod -a -G fuse oracle

2.创建/etc/fuse.conf file并添加 user_allow_other. 赋予相应的权限到这个文件.

(root)# dcli -g ~/dbs_group -l root "echo user_allow_other > /etc/fuse.conf"
(root)# dcli -g ~/dbs_group -l root chmod 644 /etc/fuse.conf

3.在所有的服务器上创建一个空的目录，用来作为DBFS文件系统的挂载点。

(root)# dcli -g ~/dbs_group -l root mkdir /dbfs_direct

  更改目录的属组，确保oracle可以正常访问与修改.

(root)# dcli -g ~/dbs_group -l root chown oracle:dba /dbfs_direct

4.To pick up the additional group (fuse) membership for the oracle user on Linux or the workaround above on Solaris, Clusterware must be restarted. For example, to restart Clusterware on all nodes at the same time (non-rolling), you can use the following commands as root:

(root)# dcli -g ~/dbs_group -l root /u01/app/11.2.0/grid/bin/crsctl stop crs
(root)# dcli -g ~/dbs_group -l root /u01/app/11.2.0/grid/bin/crsctl start crs

Note that the "crsctl stop cluster -all" syntax may not be used as it leaves ohasd running and Solaris database hosts require it to be restarted for the workaround to take effect.

5.创建一个数据库表空间，作为DBFS文件系统的存储空间. 如何创建表空间，请参考 Note 1191144.1
创建一个表空间（表空间名dbfsts）为将来存储文件使用，可以根据你的需求创建：大小，是否扩展和bigfile or small file等，完全随意

SQL> sqlplus / as sysdba
SQL> create bigfile tablespace dbfsts datafile '+DBFS_DG' size 32g autoextend on next 8g maxsize 300g NOLOGGING EXTENT MANAGEMENT LOCAL AUTOALLOCATE  SEGMENT SPACE MANAGEMENT AUTO ;

6.创建用户，该用户是后续管理DBFS使用的

SQL> create user dbfs_user identified by dbfs_passwd default tablespace dbfsts quota unlimited on dbfsts;
SQL> grant create session, create table, create view, create procedure, dbfs_role to dbfs_user;

7.通过调用 dbfs_create_filesystem.sql 脚本，创建文件系统

(oracle)$ cd $ORACLE_HOME/rdbms/admin
(oracle)$ sqlplus dbfs_user/dbfs_passwd
SQL> start dbfs_create_filesystem dbfsts FS1

This script takes two arguments:
dbfsts: tablespace for the DBFS database objects    （注：该参数输入之前创建的表空间名）
FS1: filesystem name, this can be any string and will appear as a directory under the mount point （注：这个是在后续挂载完成之后，出现在挂件点下的文件夹名称）
---脚本分析---
查看 dbfs_create_filesystem 脚本
$ cd $ORACLE_HOME
$ ls dbfs*
dbfs_create_filesystem_advanced.sql  dbfs_create_filesystem.sql  dbfs_drop_filesystem.sql
$more  dbfs_create_filesystem.sql
=================
define ts_name      = &1
define fs_name      = &2
@@dbfs_create_filesystem_advanced.sql &ts_name &fs_name nocompress nodeduplicate noencrypt non-partition
undefine ts_name
undefine fs_name
===================

$ more  dbfs_create_filesystem_advanced.sql
这个脚本主要是赋值，通过一个procedure来执行dbms_dbfs_sfs.createFilesystem包，创建一个文件系统
dbms_dbfs_sfs 的用法及定义，请参看如下链接
http://docs.oracle.com/cd/E11882_01/appdev.112/e25788/d_dbfs_sfs.htm#ARPLS72180
The DBMS_DBFS_SFS package provides an interface to operate a SecureFile-based store (SFS) for the content interface described in the DBMS_DBFS_CONTENT package.

可以通过如下命令去确认是否已经安装这个package
SQL>select OBJECT_NAME from all_objects t where object_type like 'PACKAGE%' and object_name like 'DBMS_DBFS_SFS%';
OBJECT_NAME
------------------------------
DBMS_DBFS_SFS
DBMS_DBFS_SFS
DBMS_DBFS_SFS_ADMIN
DBMS_DBFS_SFS_ADMIN
---脚本分析---

---至此，已经完成DBFS文件系统的安装，下面是对DBFS的挂载和管理---

有两种方式来挂载之前创建的表空间为DBFS
第一种方法是通过安装rpm包，使用dbfs_client命令来实现DBFS的挂载，但是不能使用wallet的功能（注意这里的wallet不是数据库中的wallet，它只是保存 dbfs_user/dbfs_passwd 的一个加密文件）
第二种方法是使用 Oracle Wallet来存储密码和使用mount命令

8. The second option is to use the Oracle Wallet to store the password and make use of the mount command. The wallet directory ($HOME/dbfs/wallet in the example here) may be any oracle-writable directory (creating a new, empty directory is recommended). All commands in this section should be run by the oracle user unless otherwise noted.）

On Linux DB nodes, set the library path on all nodes using the commands that follow (substitute proper RDBMS ORACLE_HOMEs):
(root)# dcli -g dbs_group -l root mkdir -p /usr/local/lib
(root)# dcli -g dbs_group -l root ln -s /u01/app/oracle/product/11.2.0/dbhome_1/lib/libnnz11.so /usr/local/lib/libnnz11.so
(root)# dcli -g dbs_group -l root ln -s /u01/app/oracle/product/11.2.0/dbhome_1/lib/libclntsh.so.11.1 /usr/local/lib/libclntsh.so.11.1
(root)# dcli -g dbs_group -l root ln -s /lib64/libfuse.so.2 /usr/local/lib/libfuse.so.2
(root)# dcli -g dbs_group -l root 'echo /usr/local/lib >> /etc/ld.so.conf.d/usr_local_lib.conf'
(root)# dcli -g dbs_group -l root ldconfig

9. Create a new TNS_ADMIN directory ($HOME/dbfs/tnsadmin) for exclusive use by the DBFS mount script.

(oracle)$ dcli -g dbs_group -l oracle mkdir -p $HOME/dbfs/tnsadmin

Create the $HOME/dbfs/tnsadmin/tnsnames.ora file with the following contents on the first node. This example presumes that the name of the DBFS repository database is fsdb and the instance on the first node is named fsdb1. If your RDBMS ORACLE_HOME is not /u01/app/oracle/product/11.2.0/dbhome_1, then change the PROGRAM and ENVS settings accordingly).

fsdb.local =
  (DESCRIPTION =
      (ADDRESS =
        (PROTOCOL=BEQ)
        (PROGRAM=/u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle)                                       <<<<<<注意修改这个路径
        (ARGV0=oraclefsdb1)
        (ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=BEQ)))')
        (ENVS='ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1,ORACLE_SID=fsdb1') <<<<<<注意修改实例名
      )
  (CONNECT_DATA=(SID=fsdb1))                                                                                                             <<<<<<注意修改实例名
)

On other nodes, create similar entries (all using the name "fsdb.local") and change all occurrences of fsdb1 to the appropriate instance name to match the instance name running on the node where that tnsnames.ora file resides. The tnsnames.ora file on each node will be slightly different so that each tnsnames.ora file references the instance running locally on that node.

10. On each node, create the $HOME/dbfs/tnsadmin/sqlnet.ora file with the same contents on each node after making the proper substitution for :

WALLET_LOCATION =
  (SOURCE=(METHOD=FILE)
          (METHOD_DATA=(DIRECTORY=<HOMEDIR_PATH_HERE>/dbfs/wallet))   <<<<<< 注意，这里要使用全路径，不能使用变量
  )
SQLNET.WALLET_OVERRIDE = TRUE

Ensure you substitute the correct path for the DIRECTORY attribute. You may not use variables in this path - it must be a literal full path.

Copy the file to all nodes using dcli:
(oracle)$ dcli -g ~/dbs_group -l oracle -d $HOME/dbfs/tnsadmin -f $HOME/dbfs/tnsadmin/sqlnet.ora

11. Create a wallet directory on one database server as the oracle user. For example:

(oracle)$ mkdir -p $HOME/dbfs/wallet

Create an empty auto-login wallet:

(oracle)$ mkstore -wrl $HOME/dbfs/wallet -create

Add the necessary credentials to the wallet. The credentials can be specific for the connect string used as shown here:

(oracle)$ mkstore -wrl $HOME/dbfs/wallet -createCredential fsdb.local dbfs_user dbfs_passwd

Copy the wallet files to all database nodes.

(oracle)$ dcli -g ~/dbs_group -l oracle mkdir -p $HOME/dbfs/wallet
(oracle)$ dcli -g ~/dbs_group -l oracle -d $HOME/dbfs/wallet -f $HOME/dbfs/wallet/ewallet.p12
(oracle)$ dcli -g ~/dbs_group -l oracle -d $HOME/dbfs/wallet -f $HOME/dbfs/wallet/cwallet.sso

12. Ensure that the TNS entry specified above (fsdb.local in the example) is defined and working properly (checking with "TNS_ADMIN=/home/oracle/dbfs/tnsadmin tnsping fsdb.local" is a good test).

(oracle)$ dcli -g ~/dbs_group -l oracle "export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1; TNS_ADMIN=$HOME/dbfs/tnsadmin /u01/app/oracle/product/11.2.0/dbhome_1/bin/tnsping fsdb.local | grep OK"
dm01db01: OK (20 msec)
dm01db02: OK (30 msec)

下载脚本 mount-dbfs.sh 到一个DB节点上的临时位置   (如 /tmp/mount-dbfs.sh). 运行 dos2unix防止文件格式不兼容。

For Linux, run this:
(root)# dos2unix /tmp/mount-dbfs.sh

mount-dbfs.sh 脚本是后续挂载DBFS文件系统的时候都要调用的，其中涉及一下参数，需要根据实际环境进行设置

主要是以下参数需要设置
DBNAME
MOUNT_POINT
DBFS_USER
ORACLE_HOME (should be the RDBMS ORACLE_HOME directory)
LOGGER_FACILITY (used by syslog to log the messages/output from this script)
MOUNT_OPTIONS
DBFS_PASSWD (used only if WALLET=false)                                    <<<<<如果 WALLET=true，这个设置与否都无所谓，当调用的时候会去查看wallert文件
DBFS_PWDFILE_BASE (used only if WALET=false)
WALLET (must be true or false)                                                           <<<<<如果 WALLET=true，会使用 DBFS_PASSWD 去连接并挂载DBFS
TNS_ADMIN (used only if WALLET=true)
DBFS_LOCAL_TNSALIAS

13. 修改后将文件拷贝到每个服务器的适当目录下，这里就使用GI_HOME/crs/script，并使用root设置相应的权限

(root)# dcli -g ~/dbs_group -l root -d /u01/app/11.2.0/grid/crs/script -f /tmp/mount-dbfs.sh
(root)# dcli -g ~/dbs_group -l root chown oracle:dba /u01/app/11.2.0/grid/crs/script/mount-dbfs.sh
(root)# dcli -g ~/dbs_group -l root chmod 750 /u01/app/11.2.0/grid/crs/script/mount-dbfs.sh

14. 通过以下脚本，来实现将DBFS作为一种资源注册到CRS，以使用CRS来管理DBFS。使用数据库的owner（既oracle）来执行以下脚本，根据实际情况适当修改ORACLE_HOME（其为grid home） 和DBNAME（数据库名），还可以注册多个DBFS文件系统到CRS，不过这里就不讲了，有机会再更新。

---注：这个脚本只要在一个节点执行就可以了

##### start script add-dbfs-resource.sh
#!/bin/bash
ACTION_SCRIPT=/u01/app/11.2.0/grid/crs/script/mount-dbfs.sh
RESNAME=dbfs_mount
DBNAME=fsdb
DBNAMEL=`echo $DBNAME | tr A-Z a-z`
ORACLE_HOME=/u01/app/11.2.0/grid
PATH=$ORACLE_HOME/bin:$PATH
export PATH ORACLE_HOME
crsctl add resource $RESNAME \
  -type local_resource \
  -attr "ACTION_SCRIPT=$ACTION_SCRIPT, \
         CHECK_INTERVAL=30,RESTART_ATTEMPTS=10, \
         START_DEPENDENCIES='hard(ora.$DBNAMEL.db)pullup(ora.$DBNAMEL.db)',\
         STOP_DEPENDENCIES='hard(ora.$DBNAMEL.db)',\
         SCRIPT_TIMEOUT=300"
##### end script add-dbfs-resource.sh
Then run this as the Grid Infrastructure owner (typically oracle) on one database server only:
(oracle)$ sh ./add-dbfs-resource.sh

15.通过Oracle Clusterware来管理 DBFS挂载与卸载

资源注册成功后，可以通过crsctl  stat res dbfs_mount  来查看到 dbfs_mount 资源，其所有的节点状态是 OFFLINE:

(oracle)$ <GI_HOME>/bin/crsctl stat res dbfs_mount -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
dbfs_mount
               OFFLINE OFFLINE      dscbac05
               OFFLINE OFFLINE      dscbac06

                    
17.执行crsctl start resource dbfs_mount，将会挂载DBFS到全部的节点:

(oracle)$ <GI_HOME>/bin/crsctl start resource dbfs_mount
CRS-2672: Attempting to start 'dbfs_mount' on 'dscbac05'
CRS-2672: Attempting to start 'dbfs_mount' on 'dscbac06'
CRS-2676: Start of 'dbfs_mount' on 'dscbac06' succeeded
CRS-2676: Start of 'dbfs_mount' on 'dscbac05' succeeded
(oracle)$ <GI_HOME>/bin/crsctl stat res dbfs_mount -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
dbfs_mount
               ONLINE  ONLINE       dscbac05
               ONLINE  ONLINE       dscbac06

        
18.挂载成功后，你可以通过df -h查看到DBFS文件系统的信息 Also, the default startup for this resource is "restore" which means that if it is online before Clusterware is stopped, it will attempt to come online after Clusterware is restarted. For example:

(oracle)$ df -h /dbfs_direct
Filesystem            Size  Used   Avail Use% Mounted on
dbfs                      1.5M   40K  1.4M   3% /dbfs_direct
To unmount DBFS on all nodes, run this as the oracle user:
(oracle)$ <GI_HOME>/bin/crsctl stop res dbfs_mount

19.DBFS配置过程中，遇到一段插曲
通过crsctl显示状态是online，但是文件系统不能被访问，当访问的时候，会遇到df: `/dbfs_direct': Input/output error错误，后来应用patch 13340960，问题解决。

[oracle@dm03db01 tmp]$ /u01/app/11.2.0.3/grid/bin/crsctl stat res dbfs_mount -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
dbfs_mount
               ONLINE  ONLINE       dm03db01
               ONLINE  ONLINE       dm03db02
               ONLINE  ONLINE       dm03db03
               ONLINE  ONLINE       dm03db04

[oracle@dm03db01 tmp]$ /u01/app/11.2.0.3/grid/crs/script/mount-dbfs.sh status
Checking status now
Check -- ONLINE

[oracle@dm03db01 tmp]$ cd /dbfs_direct
-bash: cd: /dbfs_direct: Input/output error
[oracle@dm03db01 tmp]$ df -h /dbfs_direct
df: `/dbfs_direct': Input/output error

详情请参考oracle官方文档：
Configuring DBFS on Oracle Database Machine (Doc ID 1054431.1)
Accessing Dbfs Using Oracle Wallet Fails With Input/output error (Doc ID 1358194.1)