Greetings,

I have a test system I use for testing deployments and when I am not
using it, it runs Boinc. It is a Scientific Linux 6.4 fully updated box.
Recently (last ~3 weeks) I have started getting the same kernel panic.
Sometimes it will be multiple times in a single day and other times it
will be days before the next one (it just had a 5 day uptime). But the
kernel panic looks pretty much the same. It is a complaint about a hung
task plus information about the ext4 file system. I have run the
smartmon tool against both drives (2 drives setup in a hardware RAID
mirror) and both drives checkout fine. I ran a fsck against the /
partition and everything looked fine (on this text box there is only /
and swap partitions). I even took out a drive at a time and had the same
crashes (though this could be an indicator that both drives are bad). I
am wondering if my RAID card is going bad.

When the crash happens I still have the SSH prompt, however, I can only
do basic things like navigating directories and sometimes reading files.
Writing to a file seems to hang, using tab-autocomplete will frequently
hang, running most programs (even `init 6` or `top`) will hang.

It crashed again last night, and I am kind of stumped. I would greatly
appreciate others thoughts and input on what the problem might be.

Thanks!
~Stack~

Dec  4 02:25:09 testbox kernel: INFO: task jbd2/cciss!c0d0:273 blocked
for more than 120 seconds.
Dec  4 02:25:09 testbox kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  4 02:25:09 testbox kernel: jbd2/cciss!c0 D 0000000000000000     0
 273      2 0x00000000
Dec  4 02:25:09 testbox kernel: ffff8802142cfb30 0000000000000046
ffff8802138b5800 0000000000001000
Dec  4 02:25:09 testbox kernel: ffff8802142cfaa0 ffffffff81012c59
ffff8802142cfae0 ffffffff810a2431
Dec  4 02:25:09 testbox kernel: ffff880214157058 ffff8802142cffd8
000000000000fb88 ffff880214157058
Dec  4 02:25:09 testbox kernel: Call Trace:
Dec  4 02:25:09 testbox kernel: [<ffffffff81012c59>] ? read_tsc+0x9/0x20
Dec  4 02:25:09 testbox kernel: [<ffffffff810a2431>] ?
ktime_get_ts+0xb1/0xf0
Dec  4 02:25:09 testbox kernel: [<ffffffff810a2431>] ?
ktime_get_ts+0xb1/0xf0
Dec  4 02:25:09 testbox kernel: [<ffffffff81119e10>] ? sync_page+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8150e953>] io_schedule+0x73/0xc0
Dec  4 02:25:09 testbox kernel: [<ffffffff81119e4d>] sync_page+0x3d/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8150f30f>] __wait_on_bit+0x5f/0x90
Dec  4 02:25:09 testbox kernel: [<ffffffff8111a083>]
wait_on_page_bit+0x73/0x80
Dec  4 02:25:09 testbox kernel: [<ffffffff81096de0>] ?
wake_bit_function+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8112f115>] ?
pagevec_lookup_tag+0x25/0x40
Dec  4 02:25:09 testbox kernel: [<ffffffff8111a4ab>]
wait_on_page_writeback_range+0xfb/0x190
Dec  4 02:25:09 testbox kernel: [<ffffffff8125d42d>] ? submit_bio+0x8d/0x120
Dec  4 02:25:09 testbox kernel: [<ffffffff8111a56f>]
filemap_fdatawait+0x2f/0x40
Dec  4 02:25:09 testbox kernel: [<ffffffffa004de59>]
jbd2_journal_commit_transaction+0x7e9/0x1500 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffff8100975d>] ?
__switch_to+0x13d/0x320
Dec  4 02:25:09 testbox kernel: [<ffffffff81081b5b>] ?
try_to_del_timer_sync+0x7b/0xe0
Dec  4 02:25:09 testbox kernel: [<ffffffffa0054148>]
kjournald2+0xb8/0x220 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffff81096da0>] ?
autoremove_wake_function+0x0/0x40
Dec  4 02:25:09 testbox kernel: [<ffffffffa0054090>] ?
kjournald2+0x0/0x220 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Dec  4 02:25:09 testbox kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Dec  4 02:25:09 testbox kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Dec  4 02:25:09 testbox kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Dec  4 02:25:09 testbox kernel: INFO: task master:1058 blocked for more
than 120 seconds.
Dec  4 02:25:09 testbox kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  4 02:25:09 testbox kernel: master        D 0000000000000000     0
1058      1 0x00000080
Dec  4 02:25:09 testbox kernel: ffff88021535d948 0000000000000082
ffff88021535d8d8 ffffffff81065c75
Dec  4 02:25:09 testbox kernel: ffff880028216700 ffff88021396b578
ffff880214336ad8 ffff880028216700
Dec  4 02:25:09 testbox kernel: ffff88021396baf8 ffff88021535dfd8
000000000000fb88 ffff88021396baf8
Dec  4 02:25:09 testbox kernel: Call Trace:
Dec  4 02:25:09 testbox kernel: [<ffffffff81065c75>] ?
enqueue_entity+0x125/0x410
Dec  4 02:25:09 testbox kernel: [<ffffffff810a2431>] ?
ktime_get_ts+0xb1/0xf0
Dec  4 02:25:09 testbox kernel: [<ffffffff811b62b0>] ? sync_buffer+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8150e953>] io_schedule+0x73/0xc0
Dec  4 02:25:09 testbox kernel: [<ffffffff811b62f0>] sync_buffer+0x40/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8150f1ba>]
__wait_on_bit_lock+0x5a/0xc0
Dec  4 02:25:09 testbox kernel: [<ffffffff811b62b0>] ? sync_buffer+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8150f298>]
out_of_line_wait_on_bit_lock+0x78/0x90
Dec  4 02:25:09 testbox kernel: [<ffffffff81096de0>] ?
wake_bit_function+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff811b5819>] ?
__find_get_block+0xa9/0x200
Dec  4 02:25:09 testbox kernel: [<ffffffff811b6496>] __lock_buffer+0x36/0x40
Dec  4 02:25:09 testbox kernel: [<ffffffffa004d293>]
do_get_write_access+0x493/0x520 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa004d471>]
jbd2_journal_get_write_access+0x31/0x50 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa00996d8>]
__ext4_journal_get_write_access+0x38/0x80 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074b23>]
ext4_reserve_inode_write+0x73/0xa0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074b9c>]
ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa004c495>] ?
jbd2_journal_start+0xb5/0x100 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074e90>]
ext4_dirty_inode+0x40/0x60 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffff811ac48b>]
__mark_inode_dirty+0x3b/0x160
Dec  4 02:25:09 testbox kernel: [<ffffffff8119c742>]
file_update_time+0xf2/0x170
Dec  4 02:25:09 testbox kernel: [<ffffffff8118bbf2>] pipe_write+0x302/0x6a0
Dec  4 02:25:09 testbox kernel: [<ffffffff8118109a>]
do_sync_write+0xfa/0x140
Dec  4 02:25:09 testbox kernel: [<ffffffff81096da0>] ?
autoremove_wake_function+0x0/0x40
Dec  4 02:25:09 testbox kernel: [<ffffffff81186b94>] ?
cp_new_stat+0xe4/0x100
Dec  4 02:25:09 testbox kernel: [<ffffffff8122904b>] ?
selinux_file_permission+0xfb/0x150
Dec  4 02:25:09 testbox kernel: [<ffffffff8121bf26>] ?
security_file_permission+0x16/0x20
Dec  4 02:25:09 testbox kernel: [<ffffffff81181398>] vfs_write+0xb8/0x1a0
Dec  4 02:25:09 testbox kernel: [<ffffffff81181c91>] sys_write+0x51/0x90
Dec  4 02:25:09 testbox kernel: [<ffffffff810dc685>] ?
__audit_syscall_exit+0x265/0x290
Dec  4 02:25:09 testbox kernel: [<ffffffff8100b072>]
system_call_fastpath+0x16/0x1b
Dec  4 02:25:09 testbox kernel: INFO: task boinc_client:1080 blocked for
more than 120 seconds.
Dec  4 02:25:09 testbox kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  4 02:25:09 testbox kernel: boinc_client  D 0000000000000000     0
1080      1 0x00000080
Dec  4 02:25:09 testbox kernel: ffff8802149558c8 0000000000000086
ffff8802152ded00 0000000000000000
Dec  4 02:25:09 testbox kernel: ffff880214955848 ffffffff81119c1e
00000000000135a0 0000000000180121
Dec  4 02:25:09 testbox kernel: ffff880214343ab8 ffff880214955fd8
000000000000fb88 ffff880214343ab8
Dec  4 02:25:09 testbox kernel: Call Trace:
Dec  4 02:25:09 testbox kernel: [<ffffffff81119c1e>] ?
find_get_page+0x1e/0xa0
Dec  4 02:25:09 testbox kernel: [<ffffffff8109708e>] ?
prepare_to_wait+0x4e/0x80
Dec  4 02:25:09 testbox kernel: [<ffffffffa004d09d>]
do_get_write_access+0x29d/0x520 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffff81096de0>] ?
wake_bit_function+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffffa004d471>]
jbd2_journal_get_write_access+0x31/0x50 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa00996d8>]
__ext4_journal_get_write_access+0x38/0x80 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074b23>]
ext4_reserve_inode_write+0x73/0xa0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074b9c>]
ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa004c495>] ?
jbd2_journal_start+0xb5/0x100 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074e90>]
ext4_dirty_inode+0x40/0x60 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffff811ac48b>]
__mark_inode_dirty+0x3b/0x160
Dec  4 02:25:09 testbox kernel: [<ffffffff8119c742>]
file_update_time+0xf2/0x170
Dec  4 02:25:09 testbox kernel: [<ffffffff811832f0>] ?
__sb_start_write+0x80/0x120
Dec  4 02:25:09 testbox kernel: [<ffffffffa0075d10>] ?
ext4_da_get_block_prep+0x0/0x3c0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffff811b845b>]
__block_page_mkwrite+0x3b/0x140
Dec  4 02:25:09 testbox kernel: [<ffffffffa0078001>]
ext4_page_mkwrite+0x121/0x360 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffff81142f90>] do_wp_page+0x640/0x920
Dec  4 02:25:09 testbox kernel: [<ffffffff81143a6d>]
handle_pte_fault+0x2cd/0xb50
Dec  4 02:25:09 testbox kernel: [<ffffffff812253c4>] ?
inode_has_perm+0x54/0xa0
Dec  4 02:25:09 testbox kernel: [<ffffffff8114452a>]
handle_mm_fault+0x23a/0x310
Dec  4 02:25:09 testbox kernel: [<ffffffff810474e9>]
__do_page_fault+0x139/0x480
Dec  4 02:25:09 testbox kernel: [<ffffffff81071e2d>] ?
wait_consider_task+0x9d/0xb20
Dec  4 02:25:09 testbox kernel: [<ffffffff8109715c>] ?
remove_wait_queue+0x3c/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff81072a2f>] ? do_wait+0x17f/0x240
Dec  4 02:25:09 testbox kernel: [<ffffffff81072b9e>] ? sys_wait4+0xae/0x100
Dec  4 02:25:09 testbox kernel: [<ffffffff81513bfe>] do_page_fault+0x3e/0xa0
Dec  4 02:25:09 testbox kernel: [<ffffffff81510fb5>] page_fault+0x25/0x30
Dec  4 02:25:09 testbox kernel: INFO: task hsgamma_FGRP2_1:6123 blocked
for more than 120 seconds.
Dec  4 02:25:09 testbox kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  4 02:25:09 testbox kernel: hsgamma_FGRP2 D 0000000000000002     0
6123   1080 0x00020084
Dec  4 02:25:09 testbox kernel: ffff88021606b9a8 0000000000000086
0000000000000000 ffff880214083000
Dec  4 02:25:09 testbox kernel: ffff88021606ba18 ffff8802164824c0
ffff88021606ba18 ffff88021bcc1600
Dec  4 02:25:09 testbox kernel: ffff88021385faf8 ffff88021606bfd8
000000000000fb88 ffff88021385faf8
Dec  4 02:25:09 testbox kernel: Call Trace:
Dec  4 02:25:09 testbox kernel: [<ffffffffa004d09d>]
do_get_write_access+0x29d/0x520 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffff81096de0>] ?
wake_bit_function+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffffa004d471>]
jbd2_journal_get_write_access+0x31/0x50 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa00996d8>]
__ext4_journal_get_write_access+0x38/0x80 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074b23>]
ext4_reserve_inode_write+0x73/0xa0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074b9c>]
ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa004c495>] ?
jbd2_journal_start+0xb5/0x100 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074e90>]
ext4_dirty_inode+0x40/0x60 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffff811ac48b>]
__mark_inode_dirty+0x3b/0x160
Dec  4 02:25:09 testbox kernel: [<ffffffff8119c742>]
file_update_time+0xf2/0x170
Dec  4 02:25:09 testbox kernel: [<ffffffff8111c1e0>]
__generic_file_aio_write+0x230/0x490
Dec  4 02:25:09 testbox kernel: [<ffffffff81010c7e>] ?
copy_user_generic+0xe/0x20
Dec  4 02:25:09 testbox kernel: [<ffffffff8111c4c8>]
generic_file_aio_write+0x88/0x100
Dec  4 02:25:09 testbox kernel: [<ffffffffa006efb1>]
ext4_file_write+0x61/0x1e0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffff8118109a>]
do_sync_write+0xfa/0x140
Dec  4 02:25:09 testbox kernel: [<ffffffff81096da0>] ?
autoremove_wake_function+0x0/0x40
Dec  4 02:25:09 testbox kernel: [<ffffffff8122904b>] ?
selinux_file_permission+0xfb/0x150
Dec  4 02:25:09 testbox kernel: [<ffffffff8121bf26>] ?
security_file_permission+0x16/0x20
Dec  4 02:25:09 testbox kernel: [<ffffffff81181398>] vfs_write+0xb8/0x1a0
Dec  4 02:25:09 testbox kernel: [<ffffffff81181c91>] sys_write+0x51/0x90
Dec  4 02:25:09 testbox kernel: [<ffffffff810dc685>] ?
__audit_syscall_exit+0x265/0x290
Dec  4 02:25:09 testbox kernel: [<ffffffff8104d830>]
sysenter_dispatch+0x7/0x2e
Dec  4 02:25:09 testbox kernel: INFO: task hsgamma_FGRP2_1:6124 blocked
for more than 120 seconds.
Dec  4 02:25:09 testbox kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  4 02:25:09 testbox kernel: hsgamma_FGRP2 D 0000000000000001     0
6124   1080 0x00020080
Dec  4 02:25:09 testbox kernel: ffff880215b4b8c8 0000000000000086
0000000000000000 ffff8802139e2640
Dec  4 02:25:09 testbox kernel: ffff880215b4b858 ffffffff81166ebe
00000000000135a0 ffff8802139e2640
Dec  4 02:25:09 testbox kernel: ffff88021431baf8 ffff880215b4bfd8
000000000000fb88 ffff88021431baf8
Dec  4 02:25:09 testbox kernel: Call Trace:
Dec  4 02:25:09 testbox kernel: [<ffffffff81166ebe>] ?
kmem_getpages+0x15e/0x170
Dec  4 02:25:09 testbox kernel: [<ffffffffa004d09d>]
do_get_write_access+0x29d/0x520 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffff81096de0>] ?
wake_bit_function+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffffa004d471>]
jbd2_journal_get_write_access+0x31/0x50 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa00996d8>]
__ext4_journal_get_write_access+0x38/0x80 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074b23>]
ext4_reserve_inode_write+0x73/0xa0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074b9c>]
ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa004c495>] ?
jbd2_journal_start+0xb5/0x100 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074e90>]
ext4_dirty_inode+0x40/0x60 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffff811ac48b>]
__mark_inode_dirty+0x3b/0x160
Dec  4 02:25:09 testbox kernel: [<ffffffff8119c742>]
file_update_time+0xf2/0x170
Dec  4 02:25:09 testbox kernel: [<ffffffff811832f0>] ?
__sb_start_write+0x80/0x120
Dec  4 02:25:09 testbox kernel: [<ffffffffa0075d10>] ?
ext4_da_get_block_prep+0x0/0x3c0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffff811b845b>]
__block_page_mkwrite+0x3b/0x140
Dec  4 02:25:09 testbox kernel: [<ffffffffa0078001>]
ext4_page_mkwrite+0x121/0x360 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffff81142f90>] do_wp_page+0x640/0x920
Dec  4 02:25:09 testbox kernel: [<ffffffff81143a6d>]
handle_pte_fault+0x2cd/0xb50
Dec  4 02:25:09 testbox kernel: [<ffffffff8100988e>] ?
__switch_to+0x26e/0x320
Dec  4 02:25:09 testbox kernel: [<ffffffff8114452a>]
handle_mm_fault+0x23a/0x310
Dec  4 02:25:09 testbox kernel: [<ffffffff810474e9>]
__do_page_fault+0x139/0x480
Dec  4 02:25:09 testbox kernel: [<ffffffff8150fc93>] ?
do_nanosleep+0x93/0xc0
Dec  4 02:25:09 testbox kernel: [<ffffffff8109c0d4>] ?
hrtimer_nanosleep+0xc4/0x180
Dec  4 02:25:09 testbox kernel: [<ffffffff8109af20>] ?
hrtimer_wakeup+0x0/0x30
Dec  4 02:25:09 testbox kernel: [<ffffffff8109bf04>] ?
hrtimer_start_range_ns+0x14/0x20
Dec  4 02:25:09 testbox kernel: [<ffffffff81012c59>] ? read_tsc+0x9/0x20
Dec  4 02:25:09 testbox kernel: [<ffffffff810a1507>] ?
getnstimeofday+0x57/0xe0
Dec  4 02:25:09 testbox kernel: [<ffffffff81513bfe>] do_page_fault+0x3e/0xa0
Dec  4 02:25:09 testbox kernel: [<ffffffff81510fb5>] page_fault+0x25/0x30
Dec  4 02:25:09 testbox kernel: INFO: task pickup:6125 blocked for more
than 120 seconds.
Dec  4 02:25:09 testbox kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  4 02:25:09 testbox kernel: pickup        D 0000000000000000     0
6125   1058 0x00000080
Dec  4 02:25:09 testbox kernel: ffff880215b29968 0000000000000082
ffff880215b298e8 ffffffff8112fe00
Dec  4 02:25:09 testbox kernel: ffff880200000000 ffffea0004013060
ffff880214ad8aa8 00000007f83efcc7
Dec  4 02:25:09 testbox kernel: ffff880214337058 ffff880215b29fd8
000000000000fb88 ffff880214337058
Dec  4 02:25:09 testbox kernel: Call Trace:
Dec  4 02:25:09 testbox kernel: [<ffffffff8112fe00>] ?
__lru_cache_add+0x40/0x90
Dec  4 02:25:09 testbox kernel: [<ffffffff810a2431>] ?
ktime_get_ts+0xb1/0xf0
Dec  4 02:25:09 testbox kernel: [<ffffffff811b62b0>] ? sync_buffer+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8150e953>] io_schedule+0x73/0xc0
Dec  4 02:25:09 testbox kernel: [<ffffffff811b62f0>] sync_buffer+0x40/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8150f1ba>]
__wait_on_bit_lock+0x5a/0xc0
Dec  4 02:25:09 testbox kernel: [<ffffffff81065c75>] ?
enqueue_entity+0x125/0x410
Dec  4 02:25:09 testbox kernel: [<ffffffff811b62b0>] ? sync_buffer+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8150f298>]
out_of_line_wait_on_bit_lock+0x78/0x90
Dec  4 02:25:09 testbox kernel: [<ffffffff81096de0>] ?
wake_bit_function+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff811b5819>] ?
__find_get_block+0xa9/0x200
Dec  4 02:25:09 testbox kernel: [<ffffffff811b6496>] __lock_buffer+0x36/0x40
Dec  4 02:25:09 testbox kernel: [<ffffffffa004d293>]
do_get_write_access+0x493/0x520 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa004d471>]
jbd2_journal_get_write_access+0x31/0x50 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa00996d8>]
__ext4_journal_get_write_access+0x38/0x80 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074b23>]
ext4_reserve_inode_write+0x73/0xa0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074b9c>]
ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffffa004c495>] ?
jbd2_journal_start+0xb5/0x100 [jbd2]
Dec  4 02:25:09 testbox kernel: [<ffffffffa0074e90>]
ext4_dirty_inode+0x40/0x60 [ext4]
Dec  4 02:25:09 testbox kernel: [<ffffffff811ac48b>]
__mark_inode_dirty+0x3b/0x160
Dec  4 02:25:09 testbox kernel: [<ffffffff8119c955>] touch_atime+0x195/0x1a0
Dec  4 02:25:09 testbox kernel: [<ffffffff8118c265>] pipe_read+0x2d5/0x4e0
Dec  4 02:25:09 testbox kernel: [<ffffffff811811da>] do_sync_read+0xfa/0x140
Dec  4 02:25:09 testbox kernel: [<ffffffff81096da0>] ?
autoremove_wake_function+0x0/0x40
Dec  4 02:25:09 testbox kernel: [<ffffffff8122900f>] ?
selinux_file_permission+0xbf/0x150
Dec  4 02:25:09 testbox kernel: [<ffffffff8121bf26>] ?
security_file_permission+0x16/0x20
Dec  4 02:25:09 testbox kernel: [<ffffffff81181ac5>] vfs_read+0xb5/0x1a0
Dec  4 02:25:09 testbox kernel: [<ffffffff81181c01>] sys_read+0x51/0x90
Dec  4 02:25:09 testbox kernel: [<ffffffff810dc685>] ?
__audit_syscall_exit+0x265/0x290
Dec  4 02:25:09 testbox kernel: [<ffffffff8100b072>]
system_call_fastpath+0x16/0x1b
Dec  4 02:25:09 testbox kernel: INFO: task data_collect_v4:6140 blocked
for more than 120 seconds.
Dec  4 02:25:09 testbox kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  4 02:25:09 testbox kernel: data_collect_ D 0000000000000002     0
6140   1080 0x00000084
Dec  4 02:25:09 testbox kernel: ffff880216d9bcc8 0000000000000082
ffff880216d9bc70 0000000000000286
Dec  4 02:25:09 testbox kernel: ffff880215f83810 ffff880200000000
7ffffffffffffffe ffff880216d9be38
Dec  4 02:25:09 testbox kernel: ffff88021385e638 ffff880216d9bfd8
000000000000fb88 ffff88021385e638
Dec  4 02:25:09 testbox kernel: Call Trace:
Dec  4 02:25:09 testbox kernel: [<ffffffff810a2431>] ?
ktime_get_ts+0xb1/0xf0
Dec  4 02:25:09 testbox kernel: [<ffffffff81119e10>] ? sync_page+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8150e953>] io_schedule+0x73/0xc0
Dec  4 02:25:09 testbox kernel: [<ffffffff81119e4d>] sync_page+0x3d/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8150f30f>] __wait_on_bit+0x5f/0x90
Dec  4 02:25:09 testbox kernel: [<ffffffff8111a083>]
wait_on_page_bit+0x73/0x80
Dec  4 02:25:09 testbox kernel: [<ffffffff81096de0>] ?
wake_bit_function+0x0/0x50
Dec  4 02:25:09 testbox kernel: [<ffffffff8112f115>] ?
pagevec_lookup_tag+0x25/0x40
Dec  4 02:25:09 testbox kernel: [<ffffffff8111a4ab>]
wait_on_page_writeback_range+0xfb/0x190
Dec  4 02:25:09 testbox kernel: [<ffffffff8112e1b1>] ?
do_writepages+0x21/0x40
Dec  4 02:25:09 testbox kernel: [<ffffffff8111a5fb>] ?
__filemap_fdatawrite_range+0x5b/0x60
Dec  4 02:25:09 testbox kernel: [<ffffffff8111a678>]
filemap_write_and_wait_range+0x78/0x90
Dec  4 02:25:09 testbox kernel: [<ffffffff811b1d5e>]
vfs_fsync_range+0x7e/0xe0
Dec  4 02:25:09 testbox kernel: [<ffffffff811b1e2d>] vfs_fsync+0x1d/0x20
Dec  4 02:25:09 testbox kernel: [<ffffffff811b1e6e>] do_fsync+0x3e/0x60
Dec  4 02:25:09 testbox kernel: [<ffffffff811b1ec0>] sys_fsync+0x10/0x20
Dec  4 02:25:09 testbox kernel: [<ffffffff8100b072>]
system_call_fastpath+0x16/0x1b
Dec  4 02:27:09 testbox kernel: INFO: task jbd2/cciss!c0d0:273 blocked
for more than 120 seconds.
Dec  4 02:27:09 testbox kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  4 02:27:09 testbox kernel: jbd2/cciss!c0 D 0000000000000000     0
 273      2 0x00000000
Dec  4 02:27:09 testbox kernel: ffff8802142cfb30 0000000000000046
ffff8802138b5800 0000000000001000
Dec  4 02:27:09 testbox kernel: ffff8802142cfaa0 ffffffff81012c59
ffff8802142cfae0 ffffffff810a2431
Dec  4 02:27:09 testbox kernel: ffff880214157058 ffff8802142cffd8
000000000000fb88 ffff880214157058
Dec  4 02:27:09 testbox kernel: Call Trace:
Dec  4 02:27:09 testbox kernel: [<ffffffff81012c59>] ? read_tsc+0x9/0x20
Dec  4 02:27:09 testbox kernel: [<ffffffff810a2431>] ?
ktime_get_ts+0xb1/0xf0
Dec  4 02:27:09 testbox kernel: [<ffffffff810a2431>] ?
ktime_get_ts+0xb1/0xf0
Dec  4 02:27:09 testbox kernel: [<ffffffff81119e10>] ? sync_page+0x0/0x50
Dec  4 02:27:09 testbox kernel: [<ffffffff8150e953>] io_schedule+0x73/0xc0
Dec  4 02:27:09 testbox kernel: [<ffffffff81119e4d>] sync_page+0x3d/0x50
Dec  4 02:27:09 testbox kernel: [<ffffffff8150f30f>] __wait_on_bit+0x5f/0x90
Dec  4 02:27:09 testbox kernel: [<ffffffff8111a083>]
wait_on_page_bit+0x73/0x80
Dec  4 02:27:09 testbox kernel: [<ffffffff81096de0>] ?
wake_bit_function+0x0/0x50
Dec  4 02:27:09 testbox kernel: [<ffffffff8112f115>] ?
pagevec_lookup_tag+0x25/0x40
Dec  4 02:27:09 testbox kernel: [<ffffffff8111a4ab>]
wait_on_page_writeback_range+0xfb/0x190
Dec  4 02:27:09 testbox kernel: [<ffffffff8125d42d>] ? submit_bio+0x8d/0x120
Dec  4 02:27:09 testbox kernel: [<ffffffff8111a56f>]
filemap_fdatawait+0x2f/0x40
Dec  4 02:27:09 testbox kernel: [<ffffffffa004de59>]
jbd2_journal_commit_transaction+0x7e9/0x1500 [jbd2]
Dec  4 02:27:09 testbox kernel: [<ffffffff8100975d>] ?
__switch_to+0x13d/0x320
Dec  4 02:27:09 testbox kernel: [<ffffffff81081b5b>] ?
try_to_del_timer_sync+0x7b/0xe0
Dec  4 02:27:09 testbox kernel: [<ffffffffa0054148>]
kjournald2+0xb8/0x220 [jbd2]
Dec  4 02:27:09 testbox kernel: [<ffffffff81096da0>] ?
autoremove_wake_function+0x0/0x40
Dec  4 02:27:09 testbox kernel: [<ffffffffa0054090>] ?
kjournald2+0x0/0x220 [jbd2]
Dec  4 02:27:09 testbox kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Dec  4 02:27:09 testbox kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Dec  4 02:27:09 testbox kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Dec  4 02:27:09 testbox kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Dec  4 02:27:09 testbox kernel: INFO: task master:1058 blocked for more
than 120 seconds.
Dec  4 02:27:09 testbox kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  4 02:27:09 testbox kernel: master        D 0000000000000000     0
1058      1 0x00000080
Dec  4 02:27:09 testbox kernel: ffff88021535d948 0000000000000082
ffff88021535d8d8 ffffffff81065c75
Dec  4 02:27:09 testbox kernel: ffff880028216700 ffff88021396b578
ffff880214336ad8 ffff880028216700
Dec  4 02:27:09 testbox kernel: ffff88021396baf8 ffff88021535dfd8
000000000000fb88 ffff88021396baf8
Dec  4 02:27:09 testbox kernel: Call Trace:
Dec  4 02:27:09 testbox kernel: [<ffffffff81065c75>] ?
enqueue_entity+0x125/0x410
Dec  4 02:27:09 testbox kernel: [<ffffffff810a2431>] ?
ktime_get_ts+0xb1/0xf0
Dec  4 02:27:09 testbox kernel: [<ffffffff811b62b0>] ? sync_buffer+0x0/0x50
Dec  4 02:27:09 testbox kernel: [<ffffffff8150e953>] io_schedule+0x73/0xc0
Dec  4 02:27:09 testbox kernel: [<ffffffff811b62f0>] sync_buffer+0x40/0x50
Dec  4 02:27:09 testbox kernel: [<ffffffff8150f1ba>]
__wait_on_bit_lock+0x5a/0xc0
Dec  4 02:27:09 testbox kernel: [<ffffffff811b62b0>] ? sync_buffer+0x0/0x50
Dec  4 02:27:09 testbox kernel: [<ffffffff8150f298>]
out_of_line_wait_on_bit_lock+0x78/0x90
Dec  4 02:27:09 testbox kernel: [<ffffffff81096de0>] ?
wake_bit_function+0x0/0x50
Dec  4 02:27:09 testbox kernel: [<ffffffff811b5819>] ?
__find_get_block+0xa9/0x200
Dec  4 02:27:09 testbox kernel: [<ffffffff811b6496>] __lock_buffer+0x36/0x40
Dec  4 02:27:09 testbox kernel: [<ffffffffa004d293>]
do_get_write_access+0x493/0x520 [jbd2]
Dec  4 02:27:09 testbox kernel: [<ffffffffa004d471>]
jbd2_journal_get_write_access+0x31/0x50 [jbd2]
Dec  4 02:27:09 testbox kernel: [<ffffffffa00996d8>]
__ext4_journal_get_write_access+0x38/0x80 [ext4]
Dec  4 02:27:09 testbox kernel: [<ffffffffa0074b23>]
ext4_reserve_inode_write+0x73/0xa0 [ext4]
Dec  4 02:27:09 testbox kernel: [<ffffffffa0074b9c>]
ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
Dec  4 02:27:09 testbox kernel: [<ffffffffa004c495>] ?
jbd2_journal_start+0xb5/0x100 [jbd2]
Dec  4 02:27:09 testbox kernel: [<ffffffffa0074e90>]
ext4_dirty_inode+0x40/0x60 [ext4]
Dec  4 02:27:09 testbox kernel: [<ffffffff811ac48b>]
__mark_inode_dirty+0x3b/0x160
Dec  4 02:27:09 testbox kernel: [<ffffffff8119c742>]
file_update_time+0xf2/0x170
Dec  4 02:27:09 testbox kernel: [<ffffffff8118bbf2>] pipe_write+0x302/0x6a0
Dec  4 02:27:09 testbox kernel: [<ffffffff8118109a>]
do_sync_write+0xfa/0x140
Dec  4 02:27:09 testbox kernel: [<ffffffff81096da0>] ?
autoremove_wake_function+0x0/0x40
Dec  4 02:27:09 testbox kernel: [<ffffffff81186b94>] ?
cp_new_stat+0xe4/0x100
Dec  4 02:27:09 testbox kernel: [<ffffffff8122904b>] ?
selinux_file_permission+0xfb/0x150
Dec  4 02:27:09 testbox kernel: [<ffffffff8121bf26>] ?
security_file_permission+0x16/0x20
Dec  4 02:27:09 testbox kernel: [<ffffffff81181398>] vfs_write+0xb8/0x1a0
Dec  4 02:27:09 testbox kernel: [<ffffffff81181c91>] sys_write+0x51/0x90
Dec  4 02:27:09 testbox kernel: [<ffffffff810dc685>] ?
__audit_syscall_exit+0x265/0x290
Dec  4 02:27:09 testbox kernel: [<ffffffff8100b072>]
system_call_fastpath+0x16/0x1b
Dec  4 02:27:09 testbox kernel: INFO: task boinc_client:1080 blocked for
more than 120 seconds.
Dec  4 02:27:09 testbox kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec  4 02:27:09 testbox kernel: boinc_client  D 0000000000000000     0
1080      1 0x00000080
Dec  4 02:27:09 testbox kernel: ffff8802149558c8 0000000000000086
ffff8802152ded00 0000000000000000
Dec  4 02:27:09 testbox kernel: ffff880214955848 ffffffff81119c1e
00000000000135a0 0000000000180121
Dec  4 02:27:09 testbox kernel: ffff880214343ab8 ffff880214955fd8
000000000000fb88 ffff880214343ab8
Dec  4 02:27:09 testbox kernel: Call Trace:
Dec  4 02:27:09 testbox kernel: [<ffffffff81119c1e>] ?
find_get_page+0x1e/0xa0
Dec  4 02:27:09 testbox kernel: [<ffffffff8109708e>] ?
prepare_to_wait+0x4e/0x80
Dec  4 02:27:09 testbox kernel: [<ffffffffa004d09d>]
do_get_write_access+0x29d/0x520 [jbd2]
Dec  4 02:27:09 testbox kernel: [<ffffffff81096de0>] ?
wake_bit_function+0x0/0x50
Dec  4 02:27:09 testbox kernel: [<ffffffffa004d471>]
jbd2_journal_get_write_access+0x31/0x50 [jbd2]
Dec  4 02:27:09 testbox kernel: [<ffffffffa00996d8>]
__ext4_journal_get_write_access+0x38/0x80 [ext4]
Dec  4 02:27:09 testbox kernel: [<ffffffffa0074b23>]
ext4_reserve_inode_write+0x73/0xa0 [ext4]
Dec  4 02:27:09 testbox kernel: [<ffffffffa0074b9c>]
ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
Dec  4 02:27:09 testbox kernel: [<ffffffffa004c495>] ?
jbd2_journal_start+0xb5/0x100 [jbd2]
Dec  4 02:27:09 testbox kernel: [<ffffffffa0074e90>]
ext4_dirty_inode+0x40/0x60 [ext4]
Dec  4 02:27:09 testbox kernel: [<ffffffff811ac48b>]
__mark_inode_dirty+0x3b/0x160
Dec  4 02:27:09 testbox kernel: [<ffffffff8119c742>]
file_update_time+0xf2/0x170
Dec  4 02:27:09 testbox kernel: [<ffffffff811832f0>] ?
__sb_start_write+0x80/0x120
Dec  4 02:27:09 testbox kernel: [<ffffffffa0075d10>] ?
ext4_da_get_block_prep+0x0/0x3c0 [ext4]
Dec  4 02:27:09 testbox kernel: [<ffffffff811b845b>]
__block_page_mkwrite+0x3b/0x140
Dec  4 02:27:09 testbox kernel: [<ffffffffa0078001>]
ext4_page_mkwrite+0x121/0x360 [ext4]
Dec  4 02:27:09 testbox kernel: [<ffffffff81142f90>] do_wp_page+0x640/0x920
Dec  4 02:27:09 testbox kernel: [<ffffffff81143a6d>]
handle_pte_fault+0x2cd/0xb50
Dec  4 02:27:09 testbox kernel: [<ffffffff812253c4>] ?
inode_has_perm+0x54/0xa0
Dec  4 02:27:09 testbox kernel: [<ffffffff8114452a>]
handle_mm_fault+0x23a/0x310
Dec  4 02:27:09 testbox kernel: [<ffffffff810474e9>]
__do_page_fault+0x139/0x480
Dec  4 02:27:09 testbox kernel: [<ffffffff81071e2d>] ?
wait_consider_task+0x9d/0xb20
Dec  4 02:27:09 testbox kernel: [<ffffffff8109715c>] ?
remove_wait_queue+0x3c/0x50
Dec  4 02:27:09 testbox kernel: [<ffffffff81072a2f>] ? do_wait+0x17f/0x240
Dec  4 02:27:09 testbox kernel: [<ffffffff81072b9e>] ? sys_wait4+0xae/0x100
Dec  4 02:27:09 testbox kernel: [<ffffffff81513bfe>] do_page_fault+0x3e/0xa0
Dec  4 02:27:09 testbox kernel: [<ffffffff81510fb5>] page_fault+0x25/0x30