RedHat – Rebuild corrupted rpmdb

The “rpmdb” is broken and not able to run any of the rpm commands.

#rpm -qa
rpmdb: Thread/process 3392/140287890028448 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
rpmdb: Thread/process 3392/140287890028448 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages database in /var/lib/rpm
Even the yum commands failing
#yum repolist
rpmdb: Thread/process 3392/140287890028448 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed
There was some rouge rpmquery command which is in “D” state which I have managed to kill it. 
#ps aux | grep rpm
root      3391  0.0  0.0 108188  1444 ?        Ss   21:44   0:00 sh /opt/splunkforwarder/etc/apps/Splunk_TA_nix_scripts/bin/rpmqueryAll.sh
root      3392  0.5  0.4 174580 16256 ?        D    21:44   0:01 rpmquery -a --queryformat="%{NAME}","%{VERSION}","%{RELEASE}","%{INSTALLTIME:date}"\n
root      4581  0.0  0.0 103328   904 pts/0    S+   21:50   0:00 grep rpm

#kill -9 3391
#kill -9 3392
But the issue still remains
  1. Delete all files in /var/spool/up2date
    # cd /var/spool/up2date
    # rm *
    # rm .*
    
  2. Ensure there are no processes with the RPM database files open. One way to do this is using lsof & ps as the root user:
    # ps -aux | grep -e rpm -e yum -e up2date
    # lsof | grep /var/lib/rpm
    
  3. As always it best to start by taking a backup before attempting to repair.

    # cd /var/lib
    # tar -zcvf /var/preserve/rpmdb-$(date +%Y-%m-%d_%H-%M-%S).tar.gz rpm
  4. If, and only if, there is no RPM database activity going on it is safe to delete the lock files.
    WARNING: If the system is in multi-user mode it is possible that a process may open the RPM database at any time. Therefore, the recommended method for clearing stale locks is the reboot which cleans up the locks prior to entering multi-user mode.

    # cd /var/lib/rpm
    # rm -f __db*      # to avoid stale locks
    # /usr/lib/rpm/rpmdb_verify Packages
    

    If the last command shown in method 2 returns without any error, then we can be sure about the integrity of the rpm database. You can also issue the following command to be sure of the success of the rpmdb_verify command.

    $ /usr/lib/rpm/rpmdb_verify Packages
    $ echo $?
    0
    

    A return value of 0 from echo $? means that the last command has been executed successfully.

  5. If the above verify reports success, go to step 8.
  6. If the above verify reports any errors, a dump and load of the database is required.
    # mv Packages Packages.orig
    # /usr/lib/rpm/rpmdb_dump Packages.orig | /usr/lib/rpm/rpmdb_load Packages

    7. Verify the newly loaded Packages file, as well as rpm output.

    # /usr/lib/rpm/rpmdb_verify Packages
    # rpm -qa >/dev/null    # Validating there are no errors
    # rpm -qa               # Validating list of installed RPMs
    

    8. If no errors and everything looks good, rebuild the RPM indexes

    # rpm -vv --rebuilddb
    

    9. Once again verify RPM database:

    # cd /var/lib/rpm
    # /usr/lib/rpm/rpmdb_verify Packages
    

Note: If the above recovery steps fail or throw errors, restoring /var/lib/rpm directory from a recent system backup may be your best option. Remember to not overwrite the tar backup as it may still be valuable.

#rpm -qa | grep 2.6.32-696.1.1
kernel-headers-2.6.32-696.1.1.el6.x86_64
kernel-devel-2.6.32-696.1.1.el6.x86_64
kernel-firmware-2.6.32-696.1.1.el6.noarch
kernel-2.6.32-696.1.1.el6.x86_64
Advertisements

One thought on “RedHat – Rebuild corrupted rpmdb

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s