Detection

    • DB::Write, DB::Put, DB::Delete, DB::SingleDelete, DB::DeleteRange, DB::Merge
    • DB::IngestExternalFile
    • DB::CompactRange
    • Status::Severity::kFatalError - The DB is in read-only mode. The only way to recover is to close the DB, remedy the underlying cause of the error, and then re-open the DB.
    • Status::Severity::kUnrecoverableError - This is the highest severity and indicates a corruption in the database. It may be possible to close and re-open the DB, but the contents of the database may no longer be correct.In addition to the above, a notification callback EventListener::OnBackgroundError will be called as soon as the background error is encountered.

    Recovery

    • The EventListener::OnBackgroundError callback can override the error status if it determines that its not serious enough to stop further writes to the DB. It can do so by setting the bg_error parameter. Doing so can be risky, as RocksDB may not be able to guarantee the consistency of the DB. Check the BackgorundErrorReason and severity of the error before overriding it.
    • Automatic recovery from background errors. This is done by polling the system to ensure the underlying error condition is resolved, and then following the same steps as DB::Resume() to restore write capability. Notification callbacks and EventListener::OnErrorRecoveryCompleted are called at the start and end of the recovery process respectively, to inform the user of the status. At present, the automatic recovery is only supported for ENOSPC error from the filesystem. If the error happens during WAL sync, recovery is done only if 2PC is not in use. In the future, we will add more cases.