Mr Tan in ocean land

Recovering MySQL After an Ubuntu 22.04 → 24.04 Upgrade Broke the MySQL Installation

Written by

mrtan@ocean

in

,

Your mysql versions may not be competible with your present versions.

Upgrading Ubuntu from 22.04 LTS to 24.04 LTS should normally be a straightforward process.

However, if the system has a customized MySQL installation or a MySQL version coming from a different repository, the Ubuntu upgrade can leave MySQL packages and the existing MySQL data directory in an inconsistent state.

This is exactly what happened to me.

The purpose of this article is to document what happened, how I recovered the MySQL installation, and—most importantly—what I would do differently next time.

⚠️ Important: Back up your MySQL data before attempting any recovery.

Do not select an option to wipe the MySQL data directory unless you are absolutely certain that you have a complete and verified backup. Removing the data directory can make recovery considerably more difficult or impossible.

What happened?

I was upgrading a server from:

Ubuntu 22.04 LTS → Ubuntu 24.04 LTS

During the upgrade process, I was presented with a choice involving the MySQL installation.

The system offered an option to upgrade MySQL to a newer version or retain the existing customized MySQL 8.0 installation.

I chose to upgrade.

During the process, I was also presented with an option to wipe the existing MySQL data.

I did not choose to wipe the data.

That turned out to be extremely important because my existing databases were still present under:

/var/lib/mysql

After the upgrade, however, MySQL would no longer start.

I also could not cleanly upgrade or downgrade the MySQL installation.

The result was essentially a mismatch between:

  • the installed MySQL packages,
  • the MySQL server binaries,
  • the package configuration,
  • and the existing MySQL data directory.

At this point, repeatedly trying to upgrade or downgrade MySQL was not helping.

I needed to recover the installation without destroying the existing data.

Step 1 — Back up the MySQL data directory

Before removing anything, I made a copy of the entire MySQL data directory.

sudo cp -R /var/lib/mysql /var/lib/mysql_backup

This gave me a second copy of the existing database files.

Why this matters

MySQL upgrades can modify the data directory as part of the upgrade process. MySQL’s documentation explicitly recommends protecting the data with a backup before performing an upgrade. (MySQL)

For a production environment, I would strongly recommend going beyond a simple filesystem copy and maintaining a proper logical or physical backup strategy.

The key principle is:

Do not start experimenting with package removal until you have protected the data.

Step 2 — Purge the broken MySQL installation

Because the installed MySQL packages were already in a broken state, I decided to remove the broken packages rather than continue trying to switch between incompatible installations.

sudo apt purge mysql-server mysql-client mysql-common mysql-server-core-* mysql-client-core-*

Then:

sudo apt autoremove

And:

sudo apt-get autoclean

The objective here was to get rid of the broken package installation while preserving the existing database data.

Do not confuse removing MySQL packages with deliberately deleting /var/lib/mysql.

The database data is the valuable part.

Step 3 — Repair the partially installed packages

After removing the broken packages, I repaired the Debian/Ubuntu package system.

First:

sudo apt clean

Then:

sudo apt-get update --fix-missing

Next:

sudo dpkg --configure -a

And:

sudo apt-get install -f

These commands are useful when dpkg or apt has been left with partially configured packages or unresolved dependencies.

At this stage, the goal is not yet to get MySQL running.

The goal is to get the Linux package management system back into a consistent state.

Step 4 — Update and upgrade the Ubuntu packages

I then ran:

sudo apt update

followed by:

sudo apt upgrade

However, MySQL still did not start.

That was expected in my situation because the MySQL server installation itself had not yet been properly restored.

Step 5 — Reinstall the MySQL 8.0 client/core package

I then installed the MySQL 8.0 client core package:

sudo apt install mysql-client-core-8.0

I also repaired any remaining incomplete package configuration:

sudo dpkg --configure -a

Then:

sudo apt-get install -f

After that, I reloaded the systemd configuration:

sudo systemctl daemon-reload

Step 6 — Synchronize the package system again

I ran another update and upgrade:

sudo apt update

sudo apt upgrade

At this point, the packages were in a much cleaner state.

Step 7 — Start MySQL

Finally, I enabled MySQL to start automatically:

sudo systemctl enable mysql

Then started it:

sudo systemctl start mysql

And checked its status:

sudo systemctl status mysql

MySQL was running again.

That was the moment I knew the recovery had worked.

Step 8 — Check the MySQL data permissions

I also checked the ownership and permissions of the MySQL data directory.

If necessary:

sudo chown -R mysql:mysql /var/lib/mysql

And:

sudo chmod -R 750 /var/lib/mysql

Then:

sudo systemctl restart mysql

And finally:

sudo systemctl status mysql

Give MySQL some time to recover

One thing I learned from this incident is that a successful systemctl start mysql does not necessarily mean everything is immediately ready.

Depending on the size and state of the database, MySQL may need some time to perform recovery or initialization work.

So after getting the service running, I would not immediately assume everything is fine.

I would check:

sudo systemctl status mysql

and inspect the MySQL error log if anything looks suspicious.

For example:

sudo journalctl -u mysql

You should also verify that the actual databases are present and accessible.

What I learned

The biggest lesson from this incident is:

Don’t treat a MySQL version change like a normal application upgrade.

MySQL is not simply an executable that can always be replaced with a newer executable.

The MySQL server interacts closely with its data directory, data dictionary, system tables, storage engine metadata, configuration, and installed packages.

MySQL’s own documentation describes an in-place upgrade as replacing the old binaries and then starting the new server against the existing data directory. The upgrade process can make changes to that data directory. (MySQL)

This is why blindly switching between versions can be dangerous.

Don’t assume you can simply downgrade

Another important lesson is that:

Downgrading MySQL is not necessarily the reverse of upgrading MySQL.

MySQL documents specific supported downgrade paths. Depending on the versions involved, a logical dump/restore or restoring a backup may be required rather than simply installing the older package and pointing it at the newer data directory. (MySQL)

For example, MySQL’s documentation explicitly recommends restoring a backup taken before an upgrade when the desired downgrade is not supported. (MySQL)

Therefore, my recovery strategy was essentially:

Protect the data → remove the broken packages → repair the package system → restore the appropriate MySQL package installation → start MySQL → verify the existing data.

An important distinction: package recovery vs. database recovery

This distinction is worth understanding.

There are actually two separate problems:

Problem 1 — Broken Ubuntu/Debian packages

Symptoms may include:

dpkg was interrupted

or:

package is not configured

or dependency errors.

The relevant tools are things like:

dpkg --configure -a
apt-get install -f
apt clean
apt update
apt upgrade

Problem 2 — MySQL cannot open or upgrade the data directory

This is much more serious.

For example, MySQL may fail because the data directory has been modified by a newer server version, because of incompatible metadata, configuration problems, permissions, storage-engine problems, or other upgrade-related issues.

In that situation, do not start randomly deleting files from /var/lib/mysql.

Preserve the data first.

What I would do differently next time

If I were performing this Ubuntu upgrade again, my procedure would be:

1. Confirm the current MySQL version

mysql --version

2. Identify where MySQL came from

Check the installed packages:

dpkg -l | grep mysql

And check the configured repositories.

This is important because Ubuntu’s own repositories and the official MySQL APT repository can provide different MySQL release lines. The MySQL documentation also recommends checking which MySQL release series is selected in the MySQL APT repository before upgrading. (MySQL)

3. Make a proper database backup

Do this before upgrading Ubuntu.

4. Make a copy of /var/lib/mysql

As an additional safety measure:

sudo cp -R /var/lib/mysql /var/lib/mysql_backup

5. Check MySQL upgrade compatibility

Do not assume:

8.0 → newer version → 8.0

is always reversible.

Check the supported upgrade and downgrade paths for the exact versions involved.

6. Upgrade MySQL separately from the Ubuntu upgrade

Where possible, avoid changing two major components at the same time.

For a production server, I would prefer:

Backup
   ↓
Verify backup
   ↓
Check MySQL compatibility
   ↓
Upgrade MySQL
   ↓
Verify MySQL
   ↓
Upgrade Ubuntu
   ↓
Verify MySQL again

rather than:

Upgrade Ubuntu
      +
Upgrade MySQL
      +
Change MySQL repository
      +
Change MySQL version

all in one operation.

My recovery checklist

For anyone facing a similar situation, this is the short version:

1. STOP
   Do not wipe /var/lib/mysql.

2. BACKUP
   Make a copy of the MySQL data directory.

3. IDENTIFY
   Determine which MySQL version and repository you are actually using.

4. PURGE
   Remove the broken MySQL packages if the package installation itself is corrupted.

5. REPAIR
   dpkg --configure -a
   apt-get install -f

6. RESTORE
   Install the intended MySQL package/version.

7. CHECK PERMISSIONS
   Make sure MySQL owns its data directory.

8. START
   systemctl start mysql

9. VERIFY
   systemctl status mysql
   journalctl -u mysql

10. CHECK DATA
    Confirm your databases and tables are actually accessible.

11. BACKUP AGAIN
    Once MySQL is healthy, create a fresh verified backup.

Final lesson

The most important lesson I took away from this incident is simple:

Before upgrading MySQL, protect the data and understand the upgrade path.

A MySQL version change can involve much more than replacing packages. The server may upgrade internal metadata and other components of the data directory. MySQL itself recommends backing up before an upgrade, and unsupported downgrades may require restoring a pre-upgrade backup rather than simply reinstalling the older version. (MySQL)

In my case, I was fortunate that I did not wipe /var/lib/mysql.

The broken MySQL packages could be removed and reconstructed. The existing database files were the part that needed to be protected.

If you are about to upgrade Ubuntu on a server running MySQL, make the database backup first.

Sources

I would publish this as a “MySQL recovery after Ubuntu upgrade” article rather than simply a diary of what happened. That makes it much more useful to someone who is desperately searching Google at 2 a.m. because mysql.service suddenly refuses to start.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts