If you want to sleep well …
Posted by Matthias Rogel on 25. January 2012
… and you are a DBA and you use ASM and you happen to run 11.2 ( but not yet 220.127.116.11 ) …
then, you probably want to apply patch 10422126 ( see http://sve.to/2011/12/22/database-11-2-bug-causes-huge-number-of-alert-log-entries/ and MOS Doc ID 1274852.1 )
I ( and our storage admin ) didn’t sleep well last night😦
The huge number of logging entries Svetoslav is talking about started for us at 6 pm yesterday, generating about 500 MB / minute.
.... WARNING: Read Failed. group:2 disk:1 AU:98945 offset:1474560 size:8192 path:/dev/oracleasm/disks/RAID5_SPB incarnation:0xe9508f84 synchronous result:'I/O error' subsys:System iop:0x2b54bc0f8390 bufp:0x2f41b4000 osderr:0x434c5344 osderr1:0x0 ....
At midnight, around 180 GB was written, so our partition holding the alert log was full. New Connections received ORA-03113.
No signs of any error in the alert of ASM, no signs of any error from the storage, no signs of weird behaviour of the database (beside extremely logging the same error over and over again)
We cleaned all log- and alert-directories, restarted the database and everything looks ok again (like the last 5 months)
Tomorrow, we will apply patch 10422126.