Linux那些事儿之我是SCSI硬盘(7)从应用层走来的ioctl

2007年过去了,这一年里明星们一如既往,大牌们继续做领军人物,而希望上位的小辈也使尽手段.该恋爱的恋爱,该炒作的炒作,该整容的整容.功成名就的就做慈善,有待提高的就造绯闻.而我该做的,是继续写我的blog,继续说Linux那些鸟事儿,继续说那些无聊的函数,在说完了sd_probe之后,我们要接触一些新的函数了,首先推出的是ioctl,具体到sd模块中就是sd_ioctl.

当我们向scsi磁盘发送命令的时候,这个函数多半会被调用.现在让我们用kdb来演示一下:先在kdb中设置断点,sd_ioctl,

Entering kdb (current=0xffff81022adcf140, pid 4074) on processor 6 due to KDB_ENTER()

[6]kdb> bp sd_ioctl

Instruction(i) BP #0 at 0xffffffff880b1de2 ([sd_mod]sd_ioctl)

    is enabled globally adjust 1

[6]kdb> go

然后执行一个scsi命令,比如INQUIRY命令.

[root@lfg2 tedkdb]# sg_inq /dev/sdg

不用说,电光火石之间kdb提示符就打印了出来,

Entering kdb (current=0xffff81022cdae760, pid 4095) on processor 5 due to Breakpoint @ 0xffffffff880b1de2

[5]kdb> bt

Stack traceback for pid 4095

0xffff81022cdae760     4095     4044  1    5   R  0xffff81022cdaea40 *sg_inq

rsp                rip                Function (args)

 ======================= <debug>

0xffff81022f619fd8 0xffffffff880b1de2 [sd_mod]sd_ioctl

 ======================= <normal>

0xffff81022d7f7d40 0xffffffff803101a0 blkdev_driver_ioctl+0x63

0xffff81022d7f7d80 0xffffffff80310810 blkdev_ioctl+0x65b

0xffff81022d7f7da0 0xffffffff80278825 __handle_mm_fault+0x6d3

0xffff81022d7f7db0 0xffffffff8029a1ea may_open+0x65

0xffff81022d7f7e10 0xffffffff80322a48 __up_read+0x7a

0xffff81022d7f7e40 0xffffffff80249b4f up_read+0x9

0xffff81022d7f7e50 0xffffffff80482d6d do_page_fault+0x48a

0xffff81022d7f7e60 0xffffffff8027acff __vma_link+0x52

0xffff81022d7f7ea0 0xffffffff802b58d5 block_ioctl+0x1b

0xffff81022d7f7eb0 0xffffffff8029da98 do_ioctl+0x2c

0xffff81022d7f7ee0 0xffffffff8029dd6d vfs_ioctl+0x247

0xffff81022d7f7f30 0xffffffff8029dde9 sys_ioctl+0x5f

0xffff81022d7f7f80 0xffffffff80209efc tracesys+0xdc

[5]kdb>

实际上走的路径是一个系统调用ioctl,或者说从sys_ioctl进来的,最终走到了sd_ioctl.

关于sd_ioctl,来自drivers/scsi/sd.c:

    641 /**

    642  *      sd_ioctl - process an ioctl

    643  *      @inode: only i_rdev/i_bdev members may be used

    644  *      @filp: only f_mode and f_flags may be used

    645  *      @cmd: ioctl command number

    646  *      @arg: this is third argument given to ioctl(2) system call.

    647  *      Often contains a pointer.

    648  *

    649  *      Returns 0 if successful (some ioctls return postive numbers on

    650  *      success as well). Returns a negated errno value in case of error.

    651  *

    652  *      Note: most ioctls are forward onto the block subsystem or further

    653  *      down in the scsi subsytem.

    654  **/

    655 static int sd_ioctl(struct inode * inode, struct file * filp,

    656                     unsigned int cmd, unsigned long arg)

    657 {

    658         struct block_device *bdev = inode->i_bdev;

    659         struct gendisk *disk = bdev->bd_disk;

    660         struct scsi_device *sdp = scsi_disk(disk)->device;

    661         void __user *p = (void __user *)arg;

    662         int error;

    663

    664         SCSI_LOG_IOCTL(1, printk("sd_ioctl: disk=%s, cmd=0x%x/n",

    665                                                 disk->disk_name, cmd));

    666

    667         /*

    668          * If we are in the middle of error recovery, don't let anyone

    669          * else try and use this device.  Also, if error recovery fails, it

    670          * may try and take the device offline, in which case all further

    671          * access to the device is prohibited.

    672          */

    673         error = scsi_nonblockable_ioctl(sdp, cmd, p, filp);

    674         if (!scsi_block_when_processing_errors(sdp) || !error)

    675                 return error;

    676

    677         /*

    678          * Send SCSI addressing ioctls directly to mid level, send other

    679          * ioctls to block level and then onto mid level if they can't be

680          * resolved.

    681          */

    682         switch (cmd) {

    683                 case SCSI_IOCTL_GET_IDLUN:

    684                 case SCSI_IOCTL_GET_BUS_NUMBER:

    685                         return scsi_ioctl(sdp, cmd, p);

    686                 default:

    687                         error = scsi_cmd_ioctl(filp, disk, cmd, p);

    688                         if (error != -ENOTTY)

    689                                 return error;

    690         }

    691         return scsi_ioctl(sdp, cmd, p);

    692 }

继续设置断点,scsi_cmd_ioctl和scsi_ioctl,会发现,调用的是scsi_cmd_ioctl.

Instruction(i) breakpoint #2 at 0xffffffff802ee554 (adjusted)

0xffffffff802ee554 scsi_cmd_ioctl:         int3

 

Entering kdb (current=0xffff81022e1a01c0, pid 3583) due to Breakpoint @ 0xffffffff802ee554

kdb> bt

Stack traceback for pid 3583

0xffff81022e1a01c0     3583     3425  1    0   R  0xffff81022e1a0490 *sg_inq

rsp                rip                Function (args)

 ======================= <debug>

0xffffffff805dafa0 0xffffffff88080dcc [sd_mod]sd_ioctl

0xffffffff805dafd8 0xffffffff802ee554 scsi_cmd_ioctl

 ======================= <normal>

0xffff8102112d7d70 0xffffffff88080e68 [sd_mod]sd_ioctl+0x9c

0xffff8102112d7db0 0xffffffff802ec9b2 blkdev_driver_ioctl+0x3a

0xffff8102112d7dc0 0xffffffff802ed00d blkdev_ioctl+0x627

0xffff8102112d7e40 0xffffffff80239e1b up_read+0x9

0xffff8102112d7e50 0xffffffff80435872 do_page_fault+0x48a

0xffff8102112d7ea0 0xffffffff802641f6 vma_link+0x3a

0xffff8102112d7ee0 0xffffffff80295155 block_ioctl+0x1b

0xffff8102112d7ef0 0xffffffff8027ec3b do_ioctl+0x1b

0xffff8102112d7f00 0xffffffff8027ee7e vfs_ioctl+0x20e

0xffff8102112d7f30 0xffffffff8027eeef sys_ioctl+0x5f

0xffff8102112d7f80 0xffffffff80209c8c tracesys+0xdc

其中的那来自block/scsi_ioctl.c

    520 int scsi_cmd_ioctl(struct file *file, struct gendisk *bd_disk, unsigned int cmd, void __user *arg)

    521 {

    522         request_queue_t *q;

    523         int err;

    524

    525         q = bd_disk->queue;

    526         if (!q)

    527                 return -ENXIO;

    528

    529         if (blk_get_queue(q))

    530                 return -ENXIO;

    531

    532         switch (cmd) {

    533                 /*

    534                  * new sgv3 interface

    535                  */

    536                 case SG_GET_VERSION_NUM:

    537                         err = sg_get_version(arg);

    538                         break;

    539                 case SCSI_IOCTL_GET_IDLUN:

    540                         err = scsi_get_idlun(q, arg);

    541                         break;

    542                 case SCSI_IOCTL_GET_BUS_NUMBER:

    543                         err = scsi_get_bus(q, arg);

    544                         break;

    545                 case SG_SET_TIMEOUT:

    546                         err = sg_set_timeout(q, arg);

    547                         break;

    548                 case SG_GET_TIMEOUT:

    549                         err = sg_get_timeout(q);

    550                         break;

    551                 case SG_GET_RESERVED_SIZE:

    552                         err = sg_get_reserved_size(q, arg);

    553                         break;

554                 case SG_SET_RESERVED_SIZE:

    555                         err = sg_set_reserved_size(q, arg);

    556                         break;

    557                 case SG_EMULATED_HOST:

    558                         err = sg_emulated_host(q, arg);

    559                         break;

    560                 case SG_IO: {

    561                         struct sg_io_hdr hdr;

    562

    563                         err = -EFAULT;

    564                         if (copy_from_user(&hdr, arg, sizeof(hdr)))

    565                                 break;

    566                         err = sg_io(file, q, bd_disk, &hdr);

    567                         if (err == -EFAULT)

    568                                 break;

    569

    570                         if (copy_to_user(arg, &hdr, sizeof(hdr)))

    571                                 err = -EFAULT;

    572                         break;

    573                 }

    574                 case CDROM_SEND_PACKET: {

    575                         struct cdrom_generic_command cgc;

    576                         struct sg_io_hdr hdr;

    577

    578                         err = -EFAULT;

    579                         if (copy_from_user(&cgc, arg, sizeof(cgc)))

    580                                 break;

    581                         cgc.timeout = clock_t_to_jiffies(cgc.timeout);

    582                         memset(&hdr, 0, sizeof(hdr));

    583                         hdr.interface_id = 'S';

    584                         hdr.cmd_len = sizeof(cgc.cmd);

    585                         hdr.dxfer_len = cgc.buflen;

    586                         err = 0;

    587                         switch (cgc.data_direction) {

    588                                 case CGC_DATA_UNKNOWN:

    589                                         hdr.dxfer_direction = SG_DXFER_UNKNOWN;

    590                                         break;

    591                                 case CGC_DATA_WRITE:

    592                                         hdr.dxfer_direction = SG_DXFER_TO_DEV;

    593                                         break;

    594                                 case CGC_DATA_READ:

    595                                         hdr.dxfer_direction = SG_DXFER_FROM_DEV;

596                                         break;

    597                                 case CGC_DATA_NONE:

    598                                         hdr.dxfer_direction = SG_DXFER_NONE;

    599                                         break;

    600                                 default:

    601                                         err = -EINVAL;

    602                         }

    603                         if (err)

    604                                 break;

    605

    606                         hdr.dxferp = cgc.buffer;

    607                         hdr.sbp = cgc.sense;

    608                         if (hdr.sbp)

    609                                 hdr.mx_sb_len = sizeof(struct request_sense);

    610                         hdr.timeout = cgc.timeout;

    611                         hdr.cmdp = ((struct cdrom_generic_command __user*) arg)->cmd;

    612                         hdr.cmd_len = sizeof(cgc.cmd);

    613

    614                         err = sg_io(file, q, bd_disk, &hdr);

    615                         if (err == -EFAULT)

    616                                 break;

    617

    618                         if (hdr.status)

    619                                 err = -EIO;

    620

    621                         cgc.stat = err;

    622                         cgc.buflen = hdr.resid;

    623                         if (copy_to_user(arg, &cgc, sizeof(cgc)))

    624                                 err = -EFAULT;

    625

    626                         break;

    627                 }

    628

    629                 /*

    630                  * old junk scsi send command ioctl

    631                  */

    632                 case SCSI_IOCTL_SEND_COMMAND:

    633                         printk(KERN_WARNING "program %s is using a deprecated SCSI ioctl, please convert it to SG_IO/        n", current->comm);

634                         err = -EINVAL;

    635                         if (!arg)

    636                                 break;

    637

    638                         err = sg_scsi_ioctl(file, q, bd_disk, arg);

    639                         break;

    640                 case CDROMCLOSETRAY:

    641                         err = blk_send_start_stop(q, bd_disk, 0x03);

    642                         break;

    643                 case CDROMEJECT:

    644                         err = blk_send_start_stop(q, bd_disk, 0x02);

    645                         break;

    646                 default:

    647                         err = -ENOTTY;

    648         }

    649

    650         blk_put_queue(q);

    651         return err;

    652 }

通过kdb的跟踪,你会发现传递进来的是cmd是SG_IO.换言之,switch-case这一大段最终会定格在560行这个SG_IO里.那么这里涉及到一个结构体就很重要了,来自include/scsi/sg.h中的sg_io_hdr.

     83 typedef struct sg_io_hdr

     84 {

     85     int interface_id;           /* [i] 'S' for SCSI generic (required) */

     86     int dxfer_direction;        /* [i] data transfer direction  */

     87     unsigned char cmd_len;      /* [i] SCSI command length ( <= 16 bytes) */

     88     unsigned char mx_sb_len;    /* [i] max length to write to sbp */

     89     unsigned short iovec_count; /* [i] 0 implies no scatter gather */

     90     unsigned int dxfer_len;     /* [i] byte count of data transfer */

     91     void __user *dxferp;        /* [i], [*io] points to data transfer memory

     92                                               or scatter gather list */

     93     unsigned char __user *cmdp; /* [i], [*i] points to command to perform */

     94     void __user *sbp;           /* [i], [*o] points to sense_buffer memory */

     95     unsigned int timeout;       /* [i] MAX_UINT->no timeout (unit: millisec) */

     96     unsigned int flags;         /* [i] 0 -> default, see SG_FLAG... */

     97     int pack_id;                /* [i->o] unused internally (normally) */

     98     void __user * usr_ptr;      /* [i->o] unused internally */

     99     unsigned char status;       /* [o] scsi status */

    100     unsigned char masked_status;/* [o] shifted, masked scsi status */

    101     unsigned char msg_status;   /* [o] messaging level data (optional) */

    102     unsigned char sb_len_wr;    /* [o] byte count actually written to sbp */

    103     unsigned short host_status; /* [o] errors from host adapter */

    104     unsigned short driver_status;/* [o] errors from software driver */

    105     int resid;                  /* [o] dxfer_len - actual_transferred */

    106     unsigned int duration;      /* [o] time taken by cmd (unit: millisec) */

    107     unsigned int info;          /* [o] auxiliary information */

    108 } sg_io_hdr_t;  /* 64 bytes long (on i386) */

这其中,cmdp指针指向的不是别人,正是命令本身也.

有过Linux下编程经验的人一定不会对copy_from_user和copy_to_user这两个骨灰级的函数陌生吧,内核空间和用户空间传递数据的两个经典函数,更远咱就不说了,当年2.4内核那会儿课堂上老师让写一个简单的字符驱动的时候就用上这两个函数了.而这里具体来说,先是把arg里的玩意儿copy到hdr中,然后调用sg_io完成实质性的工作,然后再把hdr里的内容copy回到arg中.

进一步跟踪sg_io,来自block/scsi_ioctl.c:

    225 static int sg_io(struct file *file, request_queue_t *q,

    226                 struct gendisk *bd_disk, struct sg_io_hdr *hdr)

    227 {

    228         unsigned long start_time, timeout;

    229         int writing = 0, ret = 0;

    230         struct request *rq;

    231         char sense[SCSI_SENSE_BUFFERSIZE];

    232         unsigned char cmd[BLK_MAX_CDB];

    233         struct bio *bio;

    234

    235         if (hdr->interface_id != 'S')

    236                 return -EINVAL;

    237         if (hdr->cmd_len > BLK_MAX_CDB)

    238                 return -EINVAL;

    239         if (copy_from_user(cmd, hdr->cmdp, hdr->cmd_len))

    240                 return -EFAULT;

    241         if (verify_command(file, cmd))

    242                 return -EPERM;

    243

    244         if (hdr->dxfer_len > (q->max_hw_sectors << 9))

    245                 return -EIO;

    246

    247         if (hdr->dxfer_len)

    248                 switch (hdr->dxfer_direction) {

    249                 default:

    250                         return -EINVAL;

    251                 case SG_DXFER_TO_DEV:

    252                         writing = 1;

    253                         break;

    254                 case SG_DXFER_TO_FROM_DEV:

    255                 case SG_DXFER_FROM_DEV:

    256                         break;

    257                 }

    258

    259         rq = blk_get_request(q, writing ? WRITE : READ, GFP_KERNEL);

    260         if (!rq)

    261                 return -ENOMEM;

    262

    263         /*

    264          * fill in request structure

    265          */

    266         rq->cmd_len = hdr->cmd_len;

    267         memset(rq->cmd, 0, BLK_MAX_CDB); /* ATAPI hates garbage after CDB */

    268         memcpy(rq->cmd, cmd, hdr->cmd_len);

    269

    270         memset(sense, 0, sizeof(sense));

    271         rq->sense = sense;

    272         rq->sense_len = 0;

    273

    274         rq->cmd_type = REQ_TYPE_BLOCK_PC;

    275

    276         timeout = msecs_to_jiffies(hdr->timeout);

    277         rq->timeout = (timeout < INT_MAX) ? timeout : INT_MAX;

    278         if (!rq->timeout)

    279                 rq->timeout = q->sg_timeout;

    280         if (!rq->timeout)

    281                 rq->timeout = BLK_DEFAULT_TIMEOUT;

    282

    283         if (hdr->iovec_count) {

    284                 const int size = sizeof(struct sg_iovec) * hdr->iovec_count;

    285                 struct sg_iovec *iov;

    286

    287                 iov = kmalloc(size, GFP_KERNEL);

    288                 if (!iov) {

    289                         ret = -ENOMEM;

    290                         goto out;

    291                 }

    292

    293                 if (copy_from_user(iov, hdr->dxferp, size)) {

    294                         kfree(iov);

    295                         ret = -EFAULT;

    296                         goto out;

    297                 }

    298

    299                 ret = blk_rq_map_user_iov(q, rq, iov, hdr->iovec_count,

    300                                           hdr->dxfer_len);

    301                 kfree(iov);

    302         } else if (hdr->dxfer_len)

    303                 ret = blk_rq_map_user(q, rq, hdr->dxferp, hdr->dxfer_len);

    304

    305         if (ret)

    306                 goto out;

    307

    308         bio = rq->bio;

    309         rq->retries = 0;

    310

    311         start_time = jiffies;

    312

    313         /* ignore return value. All information is passed back to caller

    314          * (if he doesn't check that is his problem).

    315          * N.B. a non-zero SCSI status is _not_ necessarily an error.

    316          */

    317         blk_execute_rq(q, bd_disk, rq, 0);

    318

    319         /* write to all output members */

    320         hdr->status = 0xff & rq->errors;

    321         hdr->masked_status = status_byte(rq->errors);

    322         hdr->msg_status = msg_byte(rq->errors);

    323         hdr->host_status = host_byte(rq->errors);

    324         hdr->driver_status = driver_byte(rq->errors);

    325         hdr->info = 0;

    326         if (hdr->masked_status || hdr->host_status || hdr->driver_status)

    327                 hdr->info |= SG_INFO_CHECK;

    328         hdr->resid = rq->data_len;

    329         hdr->duration = ((jiffies - start_time) * 1000) / HZ;

    330         hdr->sb_len_wr = 0;

    331

    332         if (rq->sense_len && hdr->sbp) {

    333                 int len = min((unsigned int) hdr->mx_sb_len, rq->sense_len);

    334

    335                 if (!copy_to_user(hdr->sbp, rq->sense, len))

    336                         hdr->sb_len_wr = len;

337         }

    338

    339         if (blk_rq_unmap_user(bio))

    340                 ret = -EFAULT;

    341

    342         /* may not have succeeded, but output values written to control

    343          * structure (struct sg_io_hdr).  */

    344 out:

    345         blk_put_request(rq);

    346         return ret;

    347 }

不难看出这里hdr实际上扮演了两个角色,一个是输入,一个是输出.而我们为了得到信息所采取的手段依然是blk_execute_rq,即仍然是以提交request的方式,在blk_execute_rq之后,实际上信息已经保存到了rq的各个成员里边,而这之后的代码就是把信息从rq的成员中转移到hdr的成员中.这种情况就好比,我去翠宫饭店游泳,我花900块钱办了一张游泳卡,半年有效期的那种,总共可以游30次,于是每次我去游泳就得凭这张卡进入,等我出来的时候,工作人员会把卡还给我,而卡上有30个格子,每有一次工作人员会划掉一格.于是hdr这个变量的作用就相当于我这张卡,进来出去都要和它打交道,并且进来出去时的状态是不一样的.

那么基本上这个函数在做什么我们是很清楚了的,我们再来关注一些细节.首先结合struct sg_io_hdr这个伟大的结构体来看代码.

interface_id,必须得是大”S”.表示Scsi Generic.从历史渊源来说表征当年那个叫做sg的模块.而与之对应的是另一个叫做pg的模块(parallel port generic driver,并行端口通用驱动),也会有interface_id这么一个变量,它的这个变量则被设置为”P”.

dxfer_direction,这个就表示数据传输方向.比如对于写命令这个变量可以取值为SG_DXFER_TO_DEV,对于读命令这个变量可以取值为SG_DXFER_FROM_DEV.

cmd_len就不用说了,表征该scsi命令的长度.它必须小于等于16.因为scsi命令最多就是16个字节.这就是为什么237行判断是否大于16.(BLK_MAX_CDB被定义为16.)

dxfer_len就是数据传输阶段到底会传输多少字节.max_hw_sectors表示单个request最大能传输多少个sectors,这个是硬件限制.一个sector是512个字节,所以244行要左移9位,即乘上512.

259行,blk_get_request基本上可以理解为申请一个struct request结构体.

然后268行,把cmd复制给rq->cmd.而cmd只是刚才239行从hdr->cmdp中copy过来的.

需要注意的是274行这里设置了rq->cmd_type是REQ_TYPE_BLOCK_PC,block那边会按不同的命令类型进行不同的处理.

283行,iovec_count,它和dxferp是分不开的.如果iovec_count为0,dxferp表示用户空间内存地址,如果iovec_count大于0,那么dxferp实际指向了一个scatter-gather数组,这个数组中的每一个成员是一个sg_iovec_t结构体变量.struct sg_iovec定义于include/scsi/sg.h:

     76 typedef struct sg_iovec /* same structure as used by readv() Linux system */

     77 {                       /* call. It defines one scatter-gather element. */

     78     void __user *iov_base;      /* Starting address  */

     79     size_t iov_len;             /* Length in bytes  */

     80 } sg_iovec_t;

而blk_rq_map_user和blk_rq_map_user_iov的作用是一样的,建立用户数据和request之间的映射.如今几乎每个人都听说过Linux中所谓的零拷贝特性,而传说中神奇的零拷贝这一刻离我们竟是如此之近.调用这两个函数的目的就是为了使用零拷贝技术.零拷贝的作用自不必说,改善性能,提高效率,这与十七大倡导的科学发展观是吻合的.然而,这其中涉及的这潭水未免太深了,我们就不去掺合了.

在执行完blk_execute_rq之后就简单了.

resid表示还剩下多少字节没有传输.

duration表示从scsi命令被发送到完成之间的时间,单位是毫秒.

sbp指向用于写Scsi sense buffer的user memory.

sb_len_wr表示实际写了多少个bytes到sbp指向的user memory中.命令如果成功了基本上就不会写sense buffer,这样的话sb_len_wr就是0.

mx_sb_len则表征能往sbp中写的最大的size.人民大学东门外办假学生证的哥们儿都知道,sb_len_wr<=mx_sb_len.

那么最后,我们实际上就可以大致了解了从应用层来讲是如何给scsi设备发送命令的.sg_inq实际上触发的是ioctl的系统调用,经过几次辗转反侧,最终sd_ioctl会被调用.而sd_ioctl会调用scsi核心层提供的函数,sg_io,最终走的路线依然是blk_execute_rq,而关于这个函数最终如何与usb-storage牵上手的,我们在block层那边对scsi命令进行分析时已经详细的介绍过了.