mirror of
https://github.com/schoebel/mars
synced 2024-12-16 03:35:03 +00:00
8e2de8288d
Some primary appeared to have died, and was rebooted. In the meantime, the old secondary was forcefully switched to primary. Afterwards, the old primary = new secondary got stuck because 2 versionlinks, which had been _produced_ by _himself_, were missing, but they were present at the new primary = old secondary! How could this happen? All transaction logfiles were fully present and correct everywhere. However, the old primary kern.log showed that a problem with the RAID system must have existed. In addition, the RAID controller errorlog also reported some problems which appeared to have healed. Problem analysis shows the following possibility: The transaction logger can continue to write data, even via fsync(), while the _writeback_ of other parts of the /mars filesystem (e.g. symlink updates) got stuck for a long time due to an IO problem. Usually, slow or even missing symlink updates are no problem because upon recovery after a reboot, everything is healed by transaction replay (possibly replaying much more data than really necessary, but this does not affect semantics, and it is even advantageous when RAID disks might contain defective data). There is one exception: after a logrotate, the corresponding new versionlink should appear after a small time. Otherwise, the above mentioned scenario could emerge. We use sync_filesystem() to ensure that any versionlink update to a _new_ versionlink is either guaranteed to become persistent, or (in case of IO problems) the mars_light thread will hang, which will be (hopefully) noticed soon by monitoring. |
||
|---|---|---|
| .. | ||
| mars_light.c | ||
| mars_proc.c | ||
| mars_proc.h | ||
| README | ||
| strategy.h | ||
| sy_generic.c | ||
| sy_net.c | ||
This is ugly code from a first prototype trial. It will disappear as soon as the new strategy layer has been implemented. The new system will use an interpreter for automated pattern matching and triggering actions, in a much more generic fashion than hand-crafted here.