のくす牧場
コンテンツ
牧場内検索
カウンタ
総計:122,149,151人
909,660,826頁
昨日:1,085人
2,981頁
今日:414人
800頁
最近の注目
人気の最安値情報

    2011/12/26 朝 raid5 の 4台のうち、2台がfail

    • 状況
      • /dev/md1 - /dev/sd[abcd]3 の4台。sd[cd]3 の2台が応答無し。
      • とりあえず再起動してしまった。
      • 起動時の自動構築で、
        md: bind<sdb3>
        md: bind<sdc3>
        md: bind<sdd3>
        md: bind<sda3>
        md: kicking non-fresh sdd3 from array!
        md: unbind<sdd3>
        md: export_rdev(sdd3)
        md: kicking non-fresh sdc3 from array!
        md: unbind<sdc3>
        md: export_rdev(sdc3)
        md/raid:md1: device sda3 operational as raid disk 0
        md/raid:md1: device sdb3 operational as raid disk 1
        md/raid:md1: allocated 4282kB
        md/raid:md1: not enough operational devices (2/4 failed)
        md/raid:md1: failed to run raid set.
        となり、自動では復旧せず。
    • 復旧の定石を1つずつ試す
      • mdadm --assemble /dev/md1
        mdadm: /dev/sdc3 reports being an active member for /dev/md1, but a --re-add fails.
        mdadm: not performing --add as that would convert /dev/sdc3 in to a spare.
        mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdc3" first.
        • active member だが組み込めない。エラーが出るのでsuper blockを消してスペアとして組み込め、とのこと。しかし、そうするとデータが消えてしまう(1台だけ障害の時は良いが、今回は2台なのでraid5は破壊される)
        • もう少しあがいてみる。
      • mdadm --assemble --force /dev/md1
        • 頼みの --force だが、エラーメッセージも何もなくダメ。
    • mdadm --exmain
      • /dev/sd[ab]3 と /dev/sd[cd]3 はアレイ情報は一致するものの、bitmap の timestamp が異なる。(non-fresh とはこのこと?)
        # mdadm --examine /dev/sdc3
        /dev/sdc3:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x1
             Array UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
                   Name : ss.sea-mew.jp:1  (local to host ss.sea-mew.jp)
          Creation Time : Tue Nov 15 07:12:05 2011
             Raid Level : raid5
           Raid Devices : 4
        
         Avail Dev Size : 3770710016 (1798.01 GiB 1930.60 GB)
             Array Size : 11312126976 (5394.04 GiB 5791.81 GB)
          Used Dev Size : 3770708992 (1798.01 GiB 1930.60 GB)
            Data Offset : 2048 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 60b8ea23:9a222fbc:d76d8768:05ccd494
        
        Internal Bitmap : 2 sectors from superblock
            Update Time : Sat Dec 24 15:58:09 2011
               Checksum : dc138c95 - correct
                 Events : 96263
            Layout : left-symmetric
         Chunk Size : 512K 
    
       Device Role : Active device 2
       Array State : AAAA ('A' == active, '.' == missing)
    # mdadm --examine /dev/sda3
    /dev/sda3:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x1
         Array UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
               Name : ss.sea-mew.jp:1  (local to host ss.sea-mew.jp)
      Creation Time : Tue Nov 15 07:12:05 2011
         Raid Level : raid5
       Raid Devices : 4
    
     Avail Dev Size : 3770710016 (1798.01 GiB 1930.60 GB)
         Array Size : 11312126976 (5394.04 GiB 5791.81 GB)
      Used Dev Size : 3770708992 (1798.01 GiB 1930.60 GB)
        Data Offset : 2048 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 12385c17:a56889da:4ee31373:1af57fd0
    
    Internal Bitmap : 2 sectors from superblock
        Update Time : Mon Dec 26 06:25:40 2011
           Checksum : 3416718c - correct
             Events : 121832
    
             Layout : left-symmetric
         Chunk Size : 512K
    
       Device Role : Active device 0
       Array State : AA.. ('A' == active, '.' == missing)
    • とにかく assemble してくれない (--forceでもダメ)
    • bitmap があるからダメなのかもしれないが詳細が分からない。
      • inactive 状態で bitmap の消し方も分からない
    • 一か八か create してみる
      # mdadm --create /dev/md1 --assume-clean --level 5 --chunk 512 --layout left-asymmetric /dev/sd[abcd]3
      mdadm: no raid-devices specified.
      • 失敗か。エラーの意味は分からない。
    • もう一度 assemble してみる
      # mdadm --assemble --force /dev/md1
      mdadm: forcing event count in /dev/sdd3(3) from 121830 upto 121832
      mdadm: clearing FAULTY flag for device 3 in /dev/md1 for /dev/sdd3
      mdadm: Marking array /dev/md1 as 'clean'
      mdadm: /dev/md1 has been started with 3 drives (out of 4).
      # cat /proc/mdstat
      Personalities : [raid1] [raid6] [raid5] [raid4]
      md1 : active raid5 sda3[0] sdd3[4] sdb3[1]
            5656063488 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UU_U]
            bitmap: 14/15 pages [56KB], 65536KB chunk
      • あれ、/dev/sdc3 を無視して /dev/sdd3 を使って(bitmap countをずらして)構築できた。
    • mount してみる
      • 1分ほどなにやらやっていたが、mountできた。ファイルも無事っぽい。
    • /dev/sdc3 も加えてやる
      # mdadm /dev/md1 --add /dev/sdc3
      mdadm: re-added /dev/sdc3
      # cat /proc/mdstat
      Personalities : [raid1] [raid6] [raid5] [raid4]
      md1 : active raid5 sdc3[3] sda3[0] sdd3[4] sdb3[1]
            5656063488 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UU_U]
            [>....................]  recovery =  0.0% (213120/1885354496) finish=294.8min speed=106560K/sec
            bitmap: 14/15 pages [56KB], 65536KB chunk
    • 結果オーライか? 詳細は不明だが、とりあえずログを書き記す。
      • manage /dev/sdd3 の結果も残っていたのでコピー。sdd3 の timestamp は sda3 と同じなので、最初から sdc3 を諦めて sdd3 を組み込めば(mdadm /dev/md1 --add /dev/sdd3)復旧していたかも。
        # mdadm --misc --examine /dev/sdc3
        /dev/sdc3:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x1
             Array UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
                   Name : ss.sea-mew.jp:1  (local to host ss.sea-mew.jp)
          Creation Time : Tue Nov 15 07:12:05 2011
             Raid Level : raid5
           Raid Devices : 4
        
         Avail Dev Size : 3770710016 (1798.01 GiB 1930.60 GB)
             Array Size : 11312126976 (5394.04 GiB 5791.81 GB)
          Used Dev Size : 3770708992 (1798.01 GiB 1930.60 GB)
            Data Offset : 2048 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 60b8ea23:9a222fbc:d76d8768:05ccd494
        
        Internal Bitmap : 2 sectors from superblock
            Update Time : Sat Dec 24 15:58:09 2011
               Checksum : dc138c95 - correct
                 Events : 96263
        
                 Layout : left-symmetric
             Chunk Size : 512K
        
           Device Role : Active device 2
           Array State : AAAA ('A' == active, '.' == missing)
        # mdadm --misc --examine /dev/sdd3
        /dev/sdd3:
                 Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x1
             Array UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
                   Name : ss.sea-mew.jp:1  (local to host ss.sea-mew.jp)
          Creation Time : Tue Nov 15 07:12:05 2011
             Raid Level : raid5
           Raid Devices : 4
        
         Avail Dev Size : 3770710016 (1798.01 GiB 1930.60 GB)
             Array Size : 11312126976 (5394.04 GiB 5791.81 GB)
          Used Dev Size : 3770708992 (1798.01 GiB 1930.60 GB)
            Data Offset : 2048 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 836b04b2:678be417:44125f0f:8b816314
        
        Internal Bitmap : 2 sectors from superblock
            Update Time : Mon Dec 26 06:23:05 2011
               Checksum : ec478290 - correct
                 Events : 121830
        
                 Layout : left-symmetric
             Chunk Size : 512K
        
           Device Role : Active device 3
           Array State : AA.A ('A' == active, '.' == missing)
        # mdadm --misc --examine /dev/sda3
        /dev/sda3:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x1
             Array UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
                   Name : ss.sea-mew.jp:1  (local to host ss.sea-mew.jp)
          Creation Time : Tue Nov 15 07:12:05 2011
             Raid Level : raid5
           Raid Devices : 4
        
         Avail Dev Size : 3770710016 (1798.01 GiB 1930.60 GB)
             Array Size : 11312126976 (5394.04 GiB 5791.81 GB)
          Used Dev Size : 3770708992 (1798.01 GiB 1930.60 GB)
            Data Offset : 2048 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 12385c17:a56889da:4ee31373:1af57fd0
        
        Internal Bitmap : 2 sectors from superblock
            Update Time : Mon Dec 26 06:25:40 2011
               Checksum : 3416718c - correct
                 Events : 121832
        
                 Layout : left-symmetric
             Chunk Size : 512K
        
           Device Role : Active device 0
           Array State : AA.. ('A' == active, '.' == missing)
        # mdadm --misc --examine /dev/sdb3
        /dev/sdb3:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x1
             Array UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
                   Name : ss.sea-mew.jp:1  (local to host ss.sea-mew.jp)
          Creation Time : Tue Nov 15 07:12:05 2011
             Raid Level : raid5
           Raid Devices : 4
        
         Avail Dev Size : 3770710016 (1798.01 GiB 1930.60 GB)
             Array Size : 11312126976 (5394.04 GiB 5791.81 GB)
          Used Dev Size : 3770708992 (1798.01 GiB 1930.60 GB)
            Data Offset : 2048 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : a7d682d4:ffb9b5df:c68a59f9:bcaec1f8
        
        Internal Bitmap : 2 sectors from superblock
            Update Time : Mon Dec 26 06:25:40 2011
               Checksum : a4f0c297 - correct
                 Events : 121832
        
                 Layout : left-symmetric
             Chunk Size : 512K
        
           Device Role : Active device 1
           Array State : AA.. ('A' == active, '.' == missing)
        # mdadm --misc --examine /dev/sdc3
        /dev/sdc3:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x1
             Array UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
                   Name : ss.sea-mew.jp:1  (local to host ss.sea-mew.jp)
          Creation Time : Tue Nov 15 07:12:05 2011
             Raid Level : raid5
           Raid Devices : 4
        
         Avail Dev Size : 3770710016 (1798.01 GiB 1930.60 GB)
             Array Size : 11312126976 (5394.04 GiB 5791.81 GB)
          Used Dev Size : 3770708992 (1798.01 GiB 1930.60 GB)
            Data Offset : 2048 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 60b8ea23:9a222fbc:d76d8768:05ccd494
        
        Internal Bitmap : 2 sectors from superblock
            Update Time : Sat Dec 24 15:58:09 2011
               Checksum : dc138c95 - correct
                 Events : 96263
        
                 Layout : left-symmetric
             Chunk Size : 512K
        
           Device Role : Active device 2
           Array State : AAAA ('A' == active, '.' == missing)
        # mdadm --manage /dev/md1 --add /dev/sdc3
        mdadm: /dev/sdc3 reports being an active member for /dev/md1, but a --re-add fails.
        mdadm: not performing --add as that would convert /dev/sdc3 in to a spare.
        mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdc3" first.
        # mdadm --manage /dev/md1 --examine-bitmap
        mdadm: --examine-bitmap would set mdadm mode to "misc", but it is already set to "manage".
        # mdadm --manage /dev/md1 --examine-bitmap /dev/sda3
        mdadm: --examine-bitmap would set mdadm mode to "misc", but it is already set to "manage".
        # mdadm --examine-bitmap /dev/sda3
                Filename : /dev/sda3
                   Magic : 6d746962
                 Version : 4
                    UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
                  Events : 121832
          Events Cleared : 96263
                   State : OK
               Chunksize : 64 MB
                  Daemon : 5s flush period
              Write Mode : Normal
               Sync Size : 1885354496 (1798.01 GiB 1930.60 GB)
                  Bitmap : 28769 bits (chunks), 2608 dirty (9.1%)
        # mdadm --examine-bitmap /dev/sdc3
                Filename : /dev/sdc3
                   Magic : 6d746962
                 Version : 4
                    UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
                  Events : 96263
          Events Cleared : 96263
                   State : OK
               Chunksize : 64 MB
                  Daemon : 5s flush period
              Write Mode : Normal
               Sync Size : 1885354496 (1798.01 GiB 1930.60 GB)
                  Bitmap : 28769 bits (chunks), 11 dirty (0.0%)
        # mdadm --grow /dev/md1 --bitmap=none
        mdadm: no bitmap found on /dev/md1
        # mdadm --examine-bitmap /dev/sdc3
                Filename : /dev/sdc3
                   Magic : 6d746962
                 Version : 4
                    UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
                  Events : 96263
          Events Cleared : 96263
                   State : OK
               Chunksize : 64 MB
                  Daemon : 5s flush period
              Write Mode : Normal
               Sync Size : 1885354496 (1798.01 GiB 1930.60 GB)
                  Bitmap : 28769 bits (chunks), 11 dirty (0.0%)
        # mdadm --examine-bitmap /dev/sda3
                Filename : /dev/sda3
                   Magic : 6d746962
                 Version : 4
                    UUID : bf4ae5e2:3ffa4800:dbd7d44f:0f56ffd6
                  Events : 121832
          Events Cleared : 96263
                   State : OK
               Chunksize : 64 MB
                  Daemon : 5s flush period
              Write Mode : Normal
               Sync Size : 1885354496 (1798.01 GiB 1930.60 GB)
                  Bitmap : 28769 bits (chunks), 2608 dirty (9.1%)
        # mdadm --manage /dev/md1 --add /dev/sdc3
        mdadm: /dev/sdc3 reports being an active member for /dev/md1, but a --re-add fails.
        mdadm: not performing --add as that would convert /dev/sdc3 in to a spare.
        mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdc3" first.
        # mdadm -v --manage /dev/md1 --add /dev/sdc3
        mdadm: /dev/sdc3 reports being an active member for /dev/md1, but a --re-add fails.
        mdadm: not performing --add as that would convert /dev/sdc3 in to a spare.
        mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdc3" first.
    • 追記
      • /var/log/messages に、/dev/sdc の SMART self check fail のエラーが以前から出ていたっぽい
        Dec 25 05:54:00 ss smartd[2440]: Device: /dev/sdc [SAT], not capable of SMART self-check
        Dec 25 05:54:00 ss smartd[2440]: Device: /dev/sdc [SAT], failed to read SMART Attribute Data
        Dec 25 05:55:09 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 25 05:55:09 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 25 05:55:09 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        • /dev/sd[cd] は同じHDDだが、sdd の方はエラーが出ていない
      • 異常発生時sdd に問題発生か
        Dec 26 06:24:00 ss smartd[2440]: Device: /dev/sdc [SAT], not capable of SMART self-check
        Dec 26 06:24:00 ss smartd[2440]: Device: /dev/sdc [SAT], failed to read SMART Attribute Data
        Dec 26 06:24:39 ss kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
        Dec 26 06:24:39 ss kernel: ata4.00: failed command: FLUSH CACHE EXT
        Dec 26 06:24:39 ss kernel: ata4.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
        Dec 26 06:24:39 ss kernel:         res 40/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
        Dec 26 06:24:39 ss kernel: ata4.00: status: { DRDY }
        Dec 26 06:24:39 ss kernel: ata4: hard resetting link
        Dec 26 06:24:45 ss kernel: ata4: link is slow to respond, please be patient (ready=0)
        Dec 26 06:24:49 ss kernel: ata4: COMRESET failed (errno=-16)
        Dec 26 06:24:49 ss kernel: ata4: hard resetting link
        Dec 26 06:24:55 ss kernel: ata4: link is slow to respond, please be patient (ready=0)
        Dec 26 06:25:00 ss kernel: ata4: COMRESET failed (errno=-16)
        Dec 26 06:25:00 ss kernel: ata4: hard resetting link
        Dec 26 06:25:05 ss kernel: ata4: link is slow to respond, please be patient (ready=0)
        Dec 26 06:25:35 ss kernel: ata4: COMRESET failed (errno=-16)
        Dec 26 06:25:35 ss kernel: ata4: limiting SATA link speed to 1.5 Gbps
        Dec 26 06:25:35 ss kernel: ata4: hard resetting link
        Dec 26 06:25:40 ss kernel: ata4: COMRESET failed (errno=-16)
        Dec 26 06:25:40 ss kernel: ata4: reset failed, giving up
        Dec 26 06:25:40 ss kernel: ata4.00: disabled
        Dec 26 06:25:40 ss kernel: ata4.00: device reported invalid CHS sector 0
        Dec 26 06:25:40 ss kernel: ata4: EH complete
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 53 0b 33 80 00 00 08 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 1393243008
        Dec 26 06:25:40 ss kernel: __ratelimit: 50 callbacks suppressed
        Dec 26 06:25:40 ss kernel: md/raid:md1: read error not correctable (sector 1256926080 on sdd3).
        Dec 26 06:25:40 ss kernel: md/raid:md1: Disk failure on sdd3, disabling device.
        Dec 26 06:25:40 ss kernel: md/raid:md1: Operation continuing on 2 devices.
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Write(10): 2a 00 53 0b 32 80 00 01 00 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 1393242752
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 53 0b 33 88 00 00 38 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 1393243016
        Dec 26 06:25:40 ss kernel: md/raid:md1: read error not correctable (sector 1256926088 on sdd3).
        Dec 26 06:25:40 ss kernel: md/raid:md1: read error not correctable (sector 1256926096 on sdd3).
        Dec 26 06:25:40 ss kernel: md/raid:md1: read error not correctable (sector 1256926104 on sdd3).
        Dec 26 06:25:40 ss kernel: md/raid:md1: read error not correctable (sector 1256926112 on sdd3).
        Dec 26 06:25:40 ss kernel: md/raid:md1: read error not correctable (sector 1256926120 on sdd3).
        Dec 26 06:25:40 ss kernel: md/raid:md1: read error not correctable (sector 1256926128 on sdd3).
        Dec 26 06:25:40 ss kernel: md/raid:md1: read error not correctable (sector 1256926136 on sdd3).
        Dec 26 06:25:40 ss smartd[2440]: Device: /dev/sdd [SAT], not capable of SMART self-check
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 11 7d 68 00 00 00 08 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 293431296
        Dec 26 06:25:40 ss kernel: md/raid:md1: read error not correctable (sector 157114368 on sdd3).
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 16 2c 14 00 00 00 08 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 371987456
        Dec 26 06:25:40 ss kernel: md/raid:md1: read error not correctable (sector 235670528 on sdd3).
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 24 38 18 00 00 00 08 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 607655936
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 28 e6 c4 00 00 00 08 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 686212096
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 36 f2 c8 00 00 00 08 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 921880576
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 3b a1 74 00 00 00 08 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 1000436736
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 49 ad 78 00 00 00 08 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 1236105216
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 4e 5c 24 00 00 00 08 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 1314661376
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Unhandled error code
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
        Dec 26 06:25:40 ss kernel: sd 3:0:0:0: [sdd] CDB: Write(10): 2a 00 08 00 07 e8 00 00 01 00
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 134219752
        Dec 26 06:25:40 ss kernel: end_request: I/O error, dev sdd, sector 134219752
        Dec 26 06:25:40 ss kernel: md: super_written gets error=-5, uptodate=0
        Dec 26 06:25:40 ss kernel: md/raid1:md0: Disk failure on sdd1, disabling device.
        Dec 26 06:25:40 ss kernel: md/raid1:md0: Operation continuing on 2 devices.
        Dec 26 06:25:40 ss kernel: I/O error in filesystem ("md1") meta-data dev md1 block 0xe0c16280       ("xlog_iodone") error 5 buf count 32768
        Dec 26 06:25:40 ss kernel: xfs_force_shutdown(md1,0x2) called from line 904 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa02f412b
        Dec 26 06:25:40 ss kernel: Filesystem "md1": Log I/O Error Detected.  Shutting down filesystem: md1
        Dec 26 06:25:40 ss kernel: I/O error in filesystem ("md1") meta-data dev md1 block 0xe0c163c0       ("xlog_iodone") error 5 buf count 32768
        Dec 26 06:25:40 ss kernel: Please umount the filesystem, and rectify the problem(s)
        Dec 26 06:25:40 ss kernel: xfs_force_shutdown(md1,0x2) called from line 904 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa02f412b
        Dec 26 06:25:40 ss kernel: I/O error in filesystem ("md1") meta-data dev md1 block 0xe0c162c0       ("xlog_iodone") error 5 buf count 32768
        Dec 26 06:25:40 ss kernel: xfs_force_shutdown(md1,0x2) called from line 904 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa02f412b
        Dec 26 06:25:40 ss kernel: I/O error in filesystem ("md1") meta-data dev md1 block 0xe0c16300       ("xlog_iodone") error 5 buf count 32768
        Dec 26 06:25:40 ss kernel: xfs_force_shutdown(md1,0x2) called from line 904 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa02f412b
        Dec 26 06:25:40 ss kernel: I/O error in filesystem ("md1") meta-data dev md1 block 0xe0c16340       ("xlog_iodone") error 5 buf count 32768
        Dec 26 06:25:40 ss kernel: xfs_force_shutdown(md1,0x2) called from line 904 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa02f412b
        Dec 26 06:25:40 ss kernel: I/O error in filesystem ("md1") meta-data dev md1 block 0xe0c16380       ("xlog_iodone") error 5 buf count 32768
        Dec 26 06:25:40 ss kernel: xfs_force_shutdown(md1,0x2) called from line 904 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa02f412b
        Dec 26 06:25:40 ss smartd[2440]: Sending warning via mail to root ...
        Dec 26 06:25:40 ss kernel: xfs_force_shutdown(md1,0x2) called from line 672 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa02f6492
        Dec 26 06:25:40 ss kernel: I/O error in filesystem ("md1") meta-data dev md1 block 0xe0c16400       ("xlog_iodone") error 5 buf count 32768
        Dec 26 06:25:40 ss kernel: xfs_force_shutdown(md1,0x2) called from line 672 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa02f6492
        Dec 26 06:25:40 ss kernel: I/O error in filesystem ("md1") meta-data dev md1 block 0xe0c16400       ("xlog_iodone") error 5 buf count 32768
        Dec 26 06:25:40 ss kernel: xfs_force_shutdown(md1,0x2) called from line 904 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa02f412b
        Dec 26 06:25:40 ss kernel: I/O error in filesystem ("md1") meta-data dev md1 block 0xe0c16440       ("xlog_iodone") error 5 buf count 32768
        Dec 26 06:25:40 ss kernel: xfs_force_shutdown(md1,0x2) called from line 904 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa02f412b
        Dec 26 06:25:40 ss smartd[2440]: Warning via mail to root: successful
        Dec 26 06:25:40 ss smartd[2440]: Device: /dev/sdd [SAT], failed to read SMART Attribute Data
        Dec 26 06:25:40 ss smartd[2440]: Sending warning via mail to root ...
        Dec 26 06:25:40 ss smartd[2440]: Warning via mail to root: successful
        Dec 26 06:25:45 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:25:45 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:25:45 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:25:45 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:25:45 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:25:45 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:25:46 ss kernel: Filesystem "md1": xfs_log_force: error 5 returned.
        Dec 26 06:27:16 ss kernel: Filesystem "md1": xfs_log_force: error 5 returned.
        Dec 26 06:28:46 ss kernel: Filesystem "md1": xfs_log_force: error 5 returned.
        Dec 26 06:30:01 ss kernel: nfsd: non-standard errno: 5
        Dec 26 06:30:10 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:30:10 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:30:10 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:30:10 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:30:10 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:30:10 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:30:16 ss kernel: Filesystem "md1": xfs_log_force: error 5 returned.
        Dec 26 06:31:46 ss kernel: Filesystem "md1": xfs_log_force: error 5 returned.
        Dec 26 06:33:16 ss kernel: Filesystem "md1": xfs_log_force: error 5 returned.
        Dec 26 06:34:46 ss kernel: Filesystem "md1": xfs_log_force: error 5 returned.
        Dec 26 06:35:09 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:35:09 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:35:09 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:35:09 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:35:09 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:35:09 ss kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
        Dec 26 06:35:41 ss rpc.mountd[1859]: authenticated mount request from 192.168.12.23:722 for /storage/video (/storage/video)
        Dec 26 06:35:41 ss rpc.mountd[1859]: /storage/video is not a directory or regular file
        Dec 26 06:36:16 ss kernel: Filesystem "md1": xfs_log_force: error 5 returned.
      • この後再起動してしまう

    トップ   凍結 差分 バックアップ 添付 複製 名前変更 リロード   新規 一覧 単語検索 最終更新   ヘルプ   最終更新のRSS 1.0 最終更新のRSS 2.0 最終更新のRSS Atom
    Counter: 1737, today: 1, yesterday: 0
    最終更新: 2011-12-26 (月) 08:49:20 (JST) (1810d) by 牧場長