RHEL6: Server panicked in 'redirfs' module
环境
- Red Hat Enterprise Linux 6
- 3rd party [redirfs]
问题
- Actual panic message can vary significantly.
One of the examples of panic message is:
Raw
kernel BUG at /build/DEEPSECURITY_DSA/9.5_SP1/rhel6_64/en/Source/src/output/RedHat_EL6.x86_64/vfs-filter/2.6.32-431.el6.x86_64/3rd_party/redirfs-0.10/rfs_inode.c:114!
Another examples:
Raw
general protection fault: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/block/dm-129/range
CPU 5
Modules linked in:
Pid: 8321, comm: mnt_monitor Tainted: G W -- ------------ 2.6.32-754.6.3.el6.x86_64
RIP: 0010:[] [] redirfs_detach_data_file+0x4f/0xd0 [redirfs]
Or
Raw
BUG: unable to handle kernel paging request at 00000001635e1065
IP: [] redirfs_detach_data_file+0x4f/0xd0 [redirfs]
决议
- Panic has occurred in 'redirfs' module which is a third party code. For further investigation please contact vendor who has provided this module.
根源
- Kernel has panicked in third party redirfs module.
诊断步骤
System status when it panicked:
Raw
KERNEL: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-504.3.3.el6.x86_64/vmlinux
DUMPFILE: /cores/retrace/tasks/299866732/crash/vmcore [PARTIAL DUMP]
CPUS: 4
DATE: Fri May 8 09:24:41 2015
UPTIME: 11:45:43
LOAD AVERAGE: 1.87, 1.41, 0.91
TASKS: 1058
NODENAME: localhost
RELEASE: 2.6.32-504.3.3.el6.x86_64
VERSION: #1 SMP Fri Dec 12 16:05:43 EST 2014
MACHINE: x86_64 (2700 Mhz)
MEMORY: 16 GB
PANIC: "kernel BUG at /build/DEEPSECURITY_DSA/9.5_SP1/rhel6_64/en/Source/src/output/RedHat_EL6.x86_64/vfs-filter/2.6.32-431.el6.x86_64/3rd_party/redirfs-0.10/rfs_inode.c:114!"
PID: 1918
COMMAND: "glusterfs"
TASK: ffff8804368c9540 [THREAD_INFO: ffff88043638e000]
CPU: 2
STATE: TASK_RUNNING (PANIC)
2) There are 3 third party modules inserted in this system:
Raw
mod: cannot find or load object file for dsa_filter module
mod: cannot find or load object file for redirfs module
mod: cannot find or load object file for gsch module
crash> mod -t
NAME TAINTS
dsa_filter P(U)
redirfs (U)
gsch (U)
3) Actual panic messages:
Raw
------------[ cut here ]------------
kernel BUG at /build/DEEPSECURITY_DSA/9.5_SP1/rhel6_64/en/Source/src/output/RedHat_EL6.x86_64/vfs-filter/2.6.32-431.el6.x86_64/3rd_party/redirfs-0.10/rfs_inode.c:114! <<<<<--<<<<<--- Note this, it is coming from 'redirfs' code, which is third party
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 2
Modules linked in: gsch(U) redirfs(U) fuse nfs lockd fscache auth_rpcgss nfs_acl sunrpc nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ip6table_filter ip6_tables iptable_filter ip_tables ipv6 dsa_filter(P)(U) ppdev parport_pc parport microcode e1000 vmware_balloon i2c_piix4 i2c_core sg shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom mptspi mptscsih mptbase scsi_transport_spi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Pid: 1918, comm: glusterfs Tainted: P --------------- 2.6.32-504.3.3.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[] [] rfs_inode_put+0x69/0x70 [redirfs]
RSP: 0018:ffff88043638f4c8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8803494fa528 RCX: ffff8803ae051518
RDX: 0000000000000034 RSI: ffff8803494fa528 RDI: ffff8803494fa528
RBP: ffff88043638f4d8 R08: ffff88043804d1d0 R09: dead000000200200
R10: ffff8802a6506a68 R11: ffff8803bcba5138 R12: ffff8802c5a895d0
R13: ffff8802c5a89670 R14: ffff8804358fef80 R15: ffff88043638f538
FS: 00007f99cfde7700(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fc5606e2000 CR3: 00000004375cd000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process glusterfs (pid: 1918, threadinfo ffff88043638e000, task ffff8804368c9540)
Stack:
ffff8802c5a895d0 ffff8803494fa528 ffff88043638f508 ffffffffa03bee68
ffff88043638f508 ffff8803ae0514e8 ffff88043638f578 ffff8803ae0514e8
ffff88043638f528 ffffffffa03c25b4 ffff8802c5a895d0 ffff88043638f578
Call Trace:
[] rfs_dentry_rem_rinode+0x68/0x80 [redirfs]
[] rfs_dcache_rinode_del+0x14/0xa0 [redirfs]
[] rfs_d_iput+0x129/0x240 [redirfs]
[] dentry_iput+0x7c/0x100
[] d_kill+0x31/0x60
[] __shrink_dcache_sb+0x366/0x3c0
[] ? apic_timer_interrupt+0xe/0x20
[] ? apic_timer_interrupt+0xe/0x20
[] shrink_dcache_memory+0x129/0x1e0
[] shrink_slab+0x12a/0x1a0
[] zone_reclaim+0x356/0x570
[] get_page_from_freelist+0x6ac/0x870
[] __alloc_pages_nodemask+0x113/0x8d0
[] ? tcp_transmit_skb+0x4b4/0x8b0
[] ? __kmalloc_node+0x4d/0x60
[] ? __alloc_skb+0x7a/0x180
[] alloc_pages_current+0xaa/0x110
[] tcp_sendmsg+0x677/0xa20
[] sock_aio_write+0x19b/0x1c0
[] ? fuse_dev_write+0x0/0x70 [fuse]
[] ? sock_aio_write+0x0/0x1c0
[] do_sync_readv_writev+0xfb/0x140
[] ? autoremove_wake_function+0x0/0x40
[] ? cap_file_permission+0x0/0x10
[] ? security_file_permission+0x16/0x20
[] do_readv_writev+0xd6/0x1f0
[] vfs_writev+0x46/0x60
[] sys_writev+0x51/0xb0
[] ? dput+0x9a/0x150
[] gsch_writev_hook_fn+0x9a/0x150 [gsch]
[] system_call_fastpath+0x16/0x1b
Code: e8 4d bc ff ff 48 8d 7b 10 e8 74 3b 00 00 48 8b 3d 95 79 00 00 48 89 de e8 05 46 db e0 f0 ff 0d 7e 79 00 00 48 83 c4 08 5b c9 c3 <0f> 0b eb fe 0f 1f 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 00 00
RIP [] rfs_inode_put+0x69/0x70 [redirfs]
RSP
Second vmcore:
1) System status when it panicked:
Raw
crash> sys
KERNEL: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-504.3.3.el6.x86_64/vmlinux
DUMPFILE: /cores/retrace/tasks/534341116/crash/vmcore [PARTIAL DUMP]
CPUS: 4
DATE: Fri May 8 22:41:57 2015
UPTIME: 13:15:07
LOAD AVERAGE: 0.25, 0.21, 0.60
TASKS: 437
NODENAME: localhost
RELEASE: 2.6.32-504.3.3.el6.x86_64
VERSION: #1 SMP Fri Dec 12 16:05:43 EST 2014
MACHINE: x86_64 (2700 Mhz)
MEMORY: 16 GB
PANIC: "kernel BUG at /build/DEEPSECURITY_DSA/9.5_SP1/rhel6_64/en/Source/src/output/RedHat_EL6.x86_64/vfs-filter/2.6.32-431.el6.x86_64/3rd_party/redirfs-0.10/rfs_inode.c:114!"
2) Third party modules:
Raw
crash> mod -t
NAME TAINTS
dsa_filter P(U)
redirfs (U)
gsch (U)
3) Log messages:
Raw
[22782(glusterfs)]: gsch_mount_hook_fn(10.0.0.2:/FEPSPublic,/fepspublic,fuse.glusterfs,0,000000000062f1c0) done
gsch_flt_add_mnt(/opt/tmp @ 10.153.242.59:/SOAPEE[65735546(fuse.glusterfs)]) done: 0
[22819(glusterfs)]: gsch_mount_hook_fn(10.0.0.2:/SOAPEE,/opt/tmp,fuse.glusterfs,0,000000000062f180) done
gsch_flt_add_mnt(/ffm @ 10.153.242.59:/xfervol[65735546(fuse.glusterfs)]) done: 0
[22856(glusterfs)]: gsch_mount_hook_fn(10.0.0.2:/xfervol,/ffm,fuse.glusterfs,0,000000000062f180) done
shrink_slab: shrink_dcache_memory+0x0/0x1e0 negative objects to delete nr=-74363813888
shrink_slab: shrink_dcache_memory+0x0/0x1e0 negative objects to delete nr=-37181906944
shrink_slab: shrink_dcache_memory+0x0/0x1e0 negative objects to delete nr=-74363813888
shrink_slab: shrink_dcache_memory+0x0/0x1e0 negative objects to delete nr=-75525748480
------------[ cut here ]------------
kernel BUG at /build/DEEPSECURITY_DSA/9.5_SP1/rhel6_64/en/Source/src/output/RedHat_EL6.x86_64/vfs-filter/2.6.32-431.el6.x86_64/3rd_party/redirfs-0.10/rfs_inode.c:114!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 2
Modules linked in: gsch(U) redirfs(U) fuse nfs lockd fscache auth_rpcgss nfs_acl sunrpc nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ip6table_filter ip6_tables iptable_filter ip_tables ipv6 dsa_filter(P)(U) ppdev parport_pc parport microcode e1000 vmware_balloon sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom mptspi mptscsih mptbase scsi_transport_spi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Pid: 21985, comm: perl Tainted: P --------------- 2.6.32-504.3.3.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[] [] rfs_inode_put+0x69/0x70 [redirfs]
RSP: 0018:ffff8801d9b97508 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff880372d24ea8 RCX: ffff880431d9b900
RDX: 000000000000002e RSI: ffff880372d24ea8 RDI: ffff880372d24ea8
RBP: ffff8801d9b97518 R08: ffff8803975c2610 R09: dead000000200200
R10: 0000000000000001 R11: 0000000000000000 R12: ffff8801df678db0
R13: ffff8801df678e50 R14: ffff880437182f40 R15: ffff8801d9b97578
FS: 00007fc1a4037700(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fc15811e000 CR3: 000000040510b000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process perl (pid: 21985, threadinfo ffff8801d9b96000, task ffff88031d74b500)
Stack:
ffff8801df678db0 ffff880372d24ea8 ffff8801d9b97548 ffffffffa0303e68
ffff8801d9b97548 ffff880431d9b8d0 ffff8801d9b975b8 ffff880431d9b8d0
ffff8801d9b97568 ffffffffa03075b4 ffff8801df678db0 ffff8801d9b975b8
Call Trace:
[] rfs_dentry_rem_rinode+0x68/0x80 [redirfs]
[] rfs_dcache_rinode_del+0x14/0xa0 [redirfs]
[] rfs_d_iput+0x129/0x240 [redirfs]
[] dentry_iput+0x7c/0x100
[] d_kill+0x31/0x60
[] __shrink_dcache_sb+0x366/0x3c0
[] ? apic_timer_interrupt+0xe/0x20
[] ? apic_timer_interrupt+0xe/0x20
[] shrink_dcache_memory+0x129/0x1e0
[] shrink_slab+0x12a/0x1a0
[] zone_reclaim+0x356/0x570
[] get_page_from_freelist+0x6ac/0x870
[] ? blk_queue_bio+0x11c/0x610
[] __alloc_pages_nodemask+0x113/0x8d0
[] alloc_pages_current+0xaa/0x110
[] __page_cache_alloc+0x87/0x90
[] __do_page_cache_readahead+0xdb/0x210
[] ra_submit+0x21/0x30
[] ondemand_readahead+0x11d/0x250
[] page_cache_async_readahead+0x90/0xc0
[] generic_file_aio_read+0x503/0x700
[] do_sync_read+0xfa/0x140
[] ? autoremove_wake_function+0x0/0x40
[] ? vfs_write+0x132/0x1a0
[] ? security_file_permission+0x16/0x20
[] vfs_read+0xb5/0x1a0
[] sys_read+0x51/0x90
[] system_call_fastpath+0x16/0x1b
Code: e8 4d bc ff ff 48 8d 7b 10 e8 74 3b 00 00 48 8b 3d 95 79 00 00 48 89 de e8 05 f6 e6 e0 f0 ff 0d 7e 79 00 00 48 83 c4 08 5b c9 c3 <0f> 0b eb fe 0f 1f 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 00 00
RIP [] rfs_inode_put+0x69/0x70 [redirfs]
RSP
One more example of crash:
Raw
crash> sys
KERNEL: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-754.6.3.el6.x86_64/vmlinux
DUMPFILE: /cores/retrace/tasks/122044735/crash/vmcore [PARTIAL DUMP]
CPUS: 6
DATE: Sat Jun 1 14:01:46 2019
UPTIME: 175 days, 13:00:51
LOAD AVERAGE: 3.30, 3.34, 3.29
TASKS: 1440
NODENAME: localhost
RELEASE: 2.6.32-754.6.3.el6.x86_64
VERSION: #1 SMP Tue Sep 18 10:29:08 EDT 2018
MACHINE: x86_64 (2400 Mhz)
MEMORY: 40 GB
PANIC: "general protection fault: 0000 [#1] SMP "
crash> mod -t
NAME TAINTS
redirfs (U)
ampavflt (U)
ampnetworkflow (U)
vmci (U)
vsock (U)
Backtrace:
Raw
crash> bt
PID: 8321 TASK: ffff880242835520 CPU: 5 COMMAND: "mnt_monitor"
#0 [ffff880a2a1df8e0] machine_kexec at ffffffff81040f1b
#1 [ffff880a2a1df940] crash_kexec at ffffffff810d6722
#2 [ffff880a2a1dfa10] oops_end at ffffffff8155e2e0
#3 [ffff880a2a1dfa40] die at ffffffff8100f95b
#4 [ffff880a2a1dfa70] do_general_protection at ffffffff8155ddc2
#5 [ffff880a2a1dfaa0] general_protection at ffffffff8155d235
[exception RIP: redirfs_detach_data_file+79]
RIP: ffffffffa0310a0f RSP: ffff880a2a1dfb58 RFLAGS: 00010206
RAX: 4c4a45bc7b1016ee RBX: ffff880a2ab2f000 RCX: ffffffffa02601c0
RDX: ffff880141182f80 RSI: ffff880242b468c0 RDI: ffff880a2ab2f000
RBP: ffff880a2a1dfb88 R8: ffff880a2a1dfc38 R9: 0000000000000000
R10: ffff88000bbcef20 R11: 0000000000000020 R12: ffff880a2ab2f000
R13: ffff88020ac69538 R14: ffff88020acd8cc0 R15: ffff88020ac69528
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#6 [ffff880a2a1dfb90] rfs_dentry_rem_data at ffffffffa030c2ec [redirfs]
#7 [ffff880a2a1dfbe0] rfs_dcache_rem at ffffffffa030ee23 [redirfs]
#8 [ffff880a2a1dfc10] rfs_dcache_walk at ffffffffa030f3eb [redirfs]
#9 [ffff880a2a1dfc90] rfs_info_rem at ffffffffa030a521 [redirfs]
#10 [ffff880a2a1dfcc0] rfs_info_rem_include at ffffffffa030ab3f [redirfs]
#11 [ffff880a2a1dfd10] rfs_root_rem_include at ffffffffa0309bc2 [redirfs]
#12 [ffff880a2a1dfd50] redirfs_rem_path at ffffffffa0308fb7 [redirfs]
#13 [ffff880a2a1dfd90] redirfs_rem_paths at ffffffffa0309148 [redirfs]
#14 [ffff880a2a1dfdd0] rfs_flt_paths_store at ffffffffa03116ea [redirfs]
#15 [ffff880a2a1dfe20] rfs_flt_store at ffffffffa0311920 [redirfs]
#16 [ffff880a2a1dfe60] sysfs_write_file at ffffffff8121fb8c
#17 [ffff880a2a1dfec0] vfs_write at ffffffff811a155a
#18 [ffff880a2a1dff00] sys_write at ffffffff811a2091
#19 [ffff880a2a1dff50] system_call_fastpath at ffffffff81564357
转载至https://access.redhat.com/solutions/1443703