Author: mrtan@ocean

  • How to Fix Outdated WP fail2ban Filters on DigitalOcean WordPress 1 click app installation after some time.

    Fix Outdated WP fail2ban Filters on DigitalOcean WordPress

    How to Fix Outdated WP fail2ban Filters on DigitalOcean WordPress

    Step-by-step guide for a DigitalOcean one-click WordPress installation on Ubuntu.

    Example warning

    /etc/fail2ban/filter.d/wordpress-extra.conf is obsolete (version 4.3.0.8).

    WP fail2ban’s WordPress plugin and Ubuntu’s server-side Fail2ban filters are separate. Updating the plugin does not necessarily update the copies under /etc/fail2ban/filter.d/.

    1. Why this happens

    The WordPress plugin emits security events, while server-level Fail2ban reads logs, matches them using filter files, and applies bans through configured firewall actions. The plugin ships filter files in its own directory; Fail2ban uses server-side copies.

    After a plugin update, the server-side filters may remain old. WP fail2ban version 5 also changed its filter layout, so an old wordpress-extra.conf may no longer be needed in a standard setup. Do not assume this without checking your jail configuration.

    Official references: WP fail2ban maintenance guide and version 5 upgrade notes.

    2. Step-by-step instructions

    Step 1 — Connect to the Droplet and check Fail2ban

    Run in SSH
    sudo systemctl status fail2ban --no-pager
    sudo fail2ban-client status

    These commands show whether Fail2ban is running and which jails are active.

    Step 2 — Find the plugin’s supplied filter files

    Run in SSH
    sudo find /var/www -path '*/wp-fail2ban/filters.d/*.conf' -print 2>/dev/null

    A common plugin path is /var/www/html/wp-content/plugins/wp-fail2ban/filters.d/. Use the path actually returned by your server.

    Step 3 — Back up the server filter directory

    Run in SSH before editing files
    sudo cp -a /etc/fail2ban/filter.d "/root/fail2ban-filter-backup-$(date +%F-%H%M%S)"

    This preserves the current filters so you can recover them if needed.

    Step 4 — List the installed WordPress filters

    Run in SSH
    sudo ls -l /etc/fail2ban/filter.d/wordpress*.conf

    Step 5 — Check whether the obsolete extra filter is referenced

    Run in SSH
    sudo grep -Rni 'wordpress-extra' /etc/fail2ban

    If there is no output, no file under /etc/fail2ban appears to reference it. If output appears, review the referenced jail before removing or changing the filter.

    Step 6 — Compare the hard and soft filters

    These commands assume the plugin is in the standard directory. If Step 2 returned another path, replace the source path accordingly.

    Compare hard filter
    sudo diff -u \
      /var/www/html/wp-content/plugins/wp-fail2ban/filters.d/wordpress-hard.conf \
      /etc/fail2ban/filter.d/wordpress-hard.conf
    Compare soft filter
    sudo diff -u \
      /var/www/html/wp-content/plugins/wp-fail2ban/filters.d/wordpress-soft.conf \
      /etc/fail2ban/filter.d/wordpress-soft.conf

    No output means the files are identical. Differences mean they are not identical; review the plugin version and release notes before replacing them.

    Step 7 — Update the hard and soft filters (only after comparing)

    Important: Only run this step if the plugin’s filter files are present and appropriate for your installed release. Review any customisations first. Do not blindly copy a filter from an unrelated release.
    Install source filters into Fail2ban
    sudo install -m 644 \
      /var/www/html/wp-content/plugins/wp-fail2ban/filters.d/wordpress-hard.conf \
      /etc/fail2ban/filter.d/wordpress-hard.conf
    
    sudo install -m 644 \
      /var/www/html/wp-content/plugins/wp-fail2ban/filters.d/wordpress-soft.conf \
      /etc/fail2ban/filter.d/wordpress-soft.conf

    Adjust the source path if your plugin directory differs. WP fail2ban releases may change which filter files are required.

    Step 8 — Remove the old extra filter only if it is not needed

    If the grep in Step 5 produced no output, and you confirmed your installed release and jails no longer need the file, back it up separately:

    Back up the old file
    sudo cp -a /etc/fail2ban/filter.d/wordpress-extra.conf /root/wordpress-extra.conf.backup

    Then, only if you confirmed it is unnecessary:

    Remove obsolete file
    sudo rm /etc/fail2ban/filter.d/wordpress-extra.conf

    Step 9 — Test the configuration

    Run before reloading
    sudo fail2ban-client -t

    Do not reload if the test fails. Read the error and fix the missing filter or invalid configuration first.

    Step 10 — Reload and verify

    Run only if the test passed
    sudo fail2ban-client reload
    sudo systemctl status fail2ban --no-pager
    sudo fail2ban-client status

    If your WordPress jail is actually named wordpress, inspect it with:

    Optional — use your actual jail name
    sudo fail2ban-client status wordpress

    3. Common problems

    • “No such file or directory”: Use Step 2 to locate the real plugin path. Do not guess paths.
    • Warning remains: Check the plugin version and its release notes. The Site Health check may itself have a version-detection issue.
    • Configuration test fails: Do not reload. Review the reported jail/filter error and restore the backup if necessary.
    • No bans appear: Confirm a jail is enabled, reading the expected log, matching events, and applying a working firewall action.
    • allowipv6 warning: This is separate from obsolete filters. The default auto setting is often appropriate; don’t disable IPv6 merely to hide the warning.

    4. Final check

    In WordPress, open Tools → Site Health and check whether the obsolete-filter warning has disappeared. If it remains, confirm the installed WP fail2ban version and compare the server filters with the matching release.

    Official documentation: WP fail2ban maintenance.

    Safety reminder: Do not delete filters or alter jails solely to silence a warning. The goal is compatible filter files, a valid configuration, and active jails that protect the server without locking out legitimate users.

    Copy buttons use your browser’s clipboard permission. If copying is blocked, select the command text and copy it manually.
  • Ubuntu LTS versions and their counterpart PHP and MySQL versions.

    Ubuntu LTS versions and their counterpart PHP and MySQL versions.

    Introduction

    The following table lists the main Ubuntu LTS releases from 18.04 to 26.04 and their typical PHP and MySQL versions from the official Ubuntu repositories.

    Ubuntu LTS versionCodenameDefault PHP versionDefault MySQL version
    18.04 LTSBionic BeaverPHP 7.2MySQL 5.7
    20.04 LTSFocal FossaPHP 7.4MySQL 8.0
    22.04 LTSJammy JellyfishPHP 8.1MySQL 8.0
    24.04 LTSNoble NumbatPHP 8.3MySQL 8.0
    26.04 LTSResolute RaccoonPHP 8.5MySQL 8.4

    Ubuntu support

    Ubuntu LTS releases generally receive 5 years of standard security maintenance, with extended security updates available through Ubuntu Pro. Here are the support dates for each major LTS release from 16.04 to 26.04, according to Canonical’s official release lifecycle.

    Ubuntu LTS versionCodenameReleasedStandard support endsExtended security maintenance (Ubuntu Pro) endsLegacy support ends
    16.04 LTSXenial XerusApr 2016Apr 2021May 2026May 2031
    18.04 LTSBionic BeaverApr 2018May 2023May 2028May 2033
    20.04 LTSFocal FossaApr 2020May 2025May 2030May 2035
    22.04 LTSJammy JellyfishApr 2022May 2027May 2032May 2037
    24.04 LTSNoble NumbatApr 2024May 2029May 2034May 2039
    26.04 LTSResolute RaccoonApr 2026May 2031May 2036May 2041

    What if you have outdated versions

    Ubuntu can still receive updates after its standard support period ends, but it depends on the Ubuntu version and the type of update.

    For a server running WordPress, PHP and MySQL, this is important because an outdated Ubuntu release can continue to run normally even after it stops receiving regular security patches.

    What happens after Ubuntu support ends?

    Update typeAfter standard support ends
    Regular security updatesStop under standard support
    Bug fixes and maintenance updatesGenerally stop
    Extended security updatesAvailable for eligible releases through Ubuntu Pro
    Application updates (PHP, MySQL, WordPress)Depend on the application’s own support and repository
    Upgrade to a newer Ubuntu LTSStill possible; recommended for long-term maintenance

    Conclusion

    Assuming you have a digitalocean droplet, then You have 3 choices when your ubuntu version is less than version 24, First upgrade them. There are some risks in this option like incompetible, so best to do it step by step:

    1. backup and live snapshot your server.
    2. clone a server and update clone server noting all the problems and solutions.
    3. update ubuntu version one at a time, DO NOT skip order. That is you NEED to upgrade ubuntu 16 to 20 first, then 20 to 22, 22 to 24 and lastly 24 to 26. you can live snapshot before every update as a fail-safe step.
    4. keep you backup and snapshot for 7 days and check every function of your server like payment gateway, online shop function etc.
    5. This is the recommended way

    Secondly, you can keep your server as it is, and your site will be vulnarable to hacking. being hack would mean your site data and activities will be halted to compromised.

    Thirdly, you can setup a new server and export your wordpress site data and database to new server. you would need to sollow following steps to setup new LAMP with wordpress cluster as new digital one click app always setup cluster wordpress which I’m not included towards.

    Fourly, you can check out Extended Security Maintenance (ESM), which is a service provided by Canonical for Ubuntu that gives you access to ongoing security updates and maintenance for specific older versions of Ubuntu after they have reached their end of life. This is particularly useful for users who need to maintain legacy systems but still require security updates.

    How to Enable ESM in Ubuntu:

    1. Check for ESM Availability: Ensure your version of Ubuntu supports ESM. You can check the status of your Ubuntu version here.
    2. Register Your System (if necessary): You might need to register your system with an Ubuntu Advantage account to access ESM, which is often free for personal use. You can register with:sudo ua attach <your-token>
    3. Enable ESM: Once registered, run the following command to enable ESM:sudo apt-get install ubuntu-advantage-tools sudo ua enable esm-infra
    4. Update Your Package Lists: After enabling ESM, update your package lists to retrieve the available security updates:sudo apt update
    5. Upgrade Packages: To apply the available security updates, you can run:sudo apt upgrade

    Recommendation

    It is advisable to regularly check for available updates and patches, as enabling ESM can help ensure your system remains secure even after its standard support period ends.

  • Critical PHP Vulnerability: What WordPress Owners Need to Do

    Critical PHP Vulnerability: What WordPress Owners Need to Do

    What happened?

    Introduction

    On 21 October 2026, the PHP team released urgent fixes for two critical vulnerabilities, CVE-2026-109618 and CVE-2026-109619. Both could let an attacker run harmful code on a server without logging in. The PHP team advises updating immediately. (PHP advisory)

    If your WordPress site runs on a DigitalOcean server or droplets, it could be at risk until PHP is updated.

    Why it matters

    Think of WordPress as a shop and PHP as the worker who handles every customer request. The flaw is in that worker, not in the shop. An attacker could send specially crafted requests that trick the server into running their instructions.

    If exploited, the possible results include:

    • The site going down or behaving incorrectly
    • Unauthorised access to the server
    • Exposure of data stored on the server
    • Changed files or the server being used for other attacks

    These are possible outcomes, not proof that your site has been attacked.

    WordPress itself is not broken. This is a PHP issue, so no separate WordPress update is needed to fix it. Keep WordPress, themes and plugins updated anyway. (WordPress announcement)

    Which versions are affected

    • Fixed: PHP 8.2, 8.3, 8.4 and 8.5 receive the patches.
    • Not fixed: PHP 8.1 and earlier are no longer supported and will not get these fixes from the PHP project.

    What to do

    If a developer or hosting company manages your server, send them these steps. Otherwise, follow them yourself.

    1. Back up first. Take a DigitalOcean snapshot of your Droplet, plus a recent backup of your WordPress files and database. This gives you a way back if something goes wrong.
    2. Check your PHP version. In WordPress, go to Tools → Site Health → Info → Server and note the PHP version. If it is 8.1 or older, updating Ubuntu will not solve the problem. You will need to move to a supported version (see below).
    3. Install the security update. Ubuntu 24.04 uses PHP 8.3 by default, and Ubuntu released its own fix on 21 October 2026 (notice USN-8948-1). Apply the available system updates, and review what will change before confirming. Ubuntu’s version numbers may differ from PHP’s own, so check the Ubuntu notice rather than the version number alone. If you installed PHP from a third-party repository, check that source for the fix instead.
    4. Restart the PHP service. The update only takes effect once the service your site actually uses (PHP-FPM or Apache) has restarted. Expect a brief interruption.
    5. Test your site. Check that key pages load, you can log in to the dashboard, and forms or payments work. Then recheck the PHP version in Site Health.

    If you cannot confirm that the PHP serving your site has the fix, assume it is still vulnerable and ask your administrator to verify.

    If you run PHP 8.1 or older

    Plan an upgrade to a supported PHP version. Test your themes and plugins for compatibility first, and keep your backup ready, because older plugins can break on newer PHP.

    *You may try to upgrade your ubuntu core accordingly which will upgrade your php version. However you must follow the step by step guide to upgrade ubuntu from version 18 to 20, then to 22, 24 and lastly 26. as you upgrade your ubuntu version, each version will carry a updated php version with the software.

    Why act now

    Once flaws become public, attackers scan websites automatically. Small sites are targeted too. A site can look normal while still running vulnerable software, so don’t wait for visible problems.

    Updating PHP is a key step, but it doesn’t replace regular maintenance of WordPress, plugins, server settings and backups.

    References

    This is general guidance. Server setups differ, so confirm which PHP installation your site uses before making changes.

  • Upgrading an Outdated Ubuntu VPS: The Step-by-Step Guide

    Upgrading an Outdated Ubuntu VPS: The Step-by-Step Guide

    Learn how to safely upgrade your server OS, PHP, and MySQL database without breaking your website.

    Upgrading an outdated server (Virtual Private Server) can feel intimidating, but if you follow the right sequence, you won’t crash your website or lose your data.

    Think of your VPS like a house:

    • Ubuntu is the foundation and walls.
    • PHP is the plumbing and electricity.
    • MySQL is the fragile filing cabinet storing all your precious data.

    If you try to move the filing cabinet while breaking down walls, everything will shatter. That is why the order of operations is critical.


    The Golden Rule: OS & PHP First, MySQL Second

    Always upgrade your operating system and web scripting environment first, while keeping your database untouched. Once the house is structurally stable on the newest version, you upgrade the database step-by-step.

    [Phase 1: Foundation]               [Phase 2: Database]
    Ubuntu 20.04 (PHP 7.4)             MySQL 5.7
              ↓                                 ↓
    Ubuntu 22.04 (PHP 8.1/8.3)         MySQL 8.0
              ↓                                 ↓
    Ubuntu 26.04 (PHP 8.3)             MySQL 8.4 → 9.3 → 9.7

    Step-by-Step Upgrade Process

    Step 1: Take a Full Server Snapshot (Backup)

    Crucial prerequisite before running any terminal commands

    Before typing a single command, log in to your VPS provider dashboard (DigitalOcean, AWS, Linode, etc.) and create a full Snapshot / Backup. If anything breaks during the upgrade, you can restore your entire server with one click.

    Step 2: Upgrade Ubuntu OS & PHP (20.04 LTS → 22.04 LTS → 26.04 LTS)

    Do not touch or upgrade MySQL during this phase

    Update your current packages first, then upgrade the operating system release by release. Never skip LTS versions.

    1. Update current packages:
      sudo apt update && sudo apt dist-upgrade -y
    2. Upgrade Ubuntu 20.04 LTS to 22.04 LTS:
      sudo do-release-upgrade
    3. Install PHP 8.3:
      sudo add-apt-repository ppa:ondrej/php -y
      sudo apt update
      sudo apt install php8.3 php8.3-cli php8.3-fpm php8.3-mysql -y
    4. Upgrade to the next LTS version (24.04 / 26.04 LTS):
      Repeat sudo do-release-upgrade to move to the next long-term support release while keeping PHP configured at 8.3.

    Step 3: Verify PHP and Web Services

    Ensure your websites can run on PHP 8.3

    Check that PHP is running correctly on version 8.3:

    php -v

    Ensure your web server (Nginx or Apache) points to the new php8.3-fpm socket and that your websites load properly before touching the database.

    Step 4: Upgrade MySQL via the Sequential Version Path

    Path: 5.7 → 8.0 → 8.4 → 9.3 → 9.7

    Major MySQL updates change how data is stored. You cannot jump directly from 5.7 to 9.x; doing so will corrupt your tables. You must step through each major milestone:

    • Step 4A: MySQL 5.7 → 8.0
      Run the MySQL shell upgrade checker to verify syntax compatibility, then upgrade packages to 8.0.
    • Step 4B: MySQL 8.0 → 8.4 LTS
      Upgrade to 8.4 (the Long Term Support release for MySQL 8).
    • Step 4C: MySQL 8.4 → 9.3 Innovation Release
      Transition into the MySQL 9.x series.
    • Step 4D: MySQL 9.3 → 9.7
      Finalize the upgrade to version 9.7.

    After each MySQL step, run the system status check to verify database health:

    sudo systemctl status mysql

    Key Takeaways for Beginners

    1. Never jump OS or Database major versions. Moving incrementally (20.04 → 22.04 → 26.04 for OS; 5.7 → 8.0 → 8.4 for MySQL) ensures automatic data migration scripts run smoothly.
    2. Freeze database changes while upgrading. Do not make major database updates at the same time you are updating OS libraries.
    3. Always test PHP compatibility. PHP 7.4 code often uses functions deprecated in PHP 8.3, so ensure your website scripts (like WordPress plugins or themes) are updated to support PHP 8.3.
  • Recovering MySQL After an Ubuntu 22.04 → 24.04 Upgrade Broke the MySQL Installation

    Your mysql versions may not be competible with your present versions.

    Upgrading Ubuntu from 22.04 LTS to 24.04 LTS should normally be a straightforward process.

    However, if the system has a customized MySQL installation or a MySQL version coming from a different repository, the Ubuntu upgrade can leave MySQL packages and the existing MySQL data directory in an inconsistent state.

    This is exactly what happened to me.

    The purpose of this article is to document what happened, how I recovered the MySQL installation, and—most importantly—what I would do differently next time.

    ⚠️ Important: Back up your MySQL data before attempting any recovery.

    Do not select an option to wipe the MySQL data directory unless you are absolutely certain that you have a complete and verified backup. Removing the data directory can make recovery considerably more difficult or impossible.


    What happened?

    I was upgrading a server from:

    Ubuntu 22.04 LTS → Ubuntu 24.04 LTS

    During the upgrade process, I was presented with a choice involving the MySQL installation.

    The system offered an option to upgrade MySQL to a newer version or retain the existing customized MySQL 8.0 installation.

    I chose to upgrade.

    During the process, I was also presented with an option to wipe the existing MySQL data.

    I did not choose to wipe the data.

    That turned out to be extremely important because my existing databases were still present under:

    /var/lib/mysql
    

    After the upgrade, however, MySQL would no longer start.

    I also could not cleanly upgrade or downgrade the MySQL installation.

    The result was essentially a mismatch between:

    • the installed MySQL packages,
    • the MySQL server binaries,
    • the package configuration,
    • and the existing MySQL data directory.

    At this point, repeatedly trying to upgrade or downgrade MySQL was not helping.

    I needed to recover the installation without destroying the existing data.


    Step 1 — Back up the MySQL data directory

    Before removing anything, I made a copy of the entire MySQL data directory.

    sudo cp -R /var/lib/mysql /var/lib/mysql_backup
    

    This gave me a second copy of the existing database files.

    Why this matters

    MySQL upgrades can modify the data directory as part of the upgrade process. MySQL’s documentation explicitly recommends protecting the data with a backup before performing an upgrade. (MySQL)

    For a production environment, I would strongly recommend going beyond a simple filesystem copy and maintaining a proper logical or physical backup strategy.

    The key principle is:

    Do not start experimenting with package removal until you have protected the data.


    Step 2 — Purge the broken MySQL installation

    Because the installed MySQL packages were already in a broken state, I decided to remove the broken packages rather than continue trying to switch between incompatible installations.

    sudo apt purge mysql-server mysql-client mysql-common mysql-server-core-* mysql-client-core-*
    

    Then:

    sudo apt autoremove
    

    And:

    sudo apt-get autoclean
    

    The objective here was to get rid of the broken package installation while preserving the existing database data.

    Do not confuse removing MySQL packages with deliberately deleting /var/lib/mysql.

    The database data is the valuable part.


    Step 3 — Repair the partially installed packages

    After removing the broken packages, I repaired the Debian/Ubuntu package system.

    First:

    sudo apt clean
    

    Then:

    sudo apt-get update --fix-missing
    

    Next:

    sudo dpkg --configure -a
    

    And:

    sudo apt-get install -f
    

    These commands are useful when dpkg or apt has been left with partially configured packages or unresolved dependencies.

    At this stage, the goal is not yet to get MySQL running.

    The goal is to get the Linux package management system back into a consistent state.


    Step 4 — Update and upgrade the Ubuntu packages

    I then ran:

    sudo apt update
    

    followed by:

    sudo apt upgrade
    

    However, MySQL still did not start.

    That was expected in my situation because the MySQL server installation itself had not yet been properly restored.


    Step 5 — Reinstall the MySQL 8.0 client/core package

    I then installed the MySQL 8.0 client core package:

    sudo apt install mysql-client-core-8.0
    

    I also repaired any remaining incomplete package configuration:

    sudo dpkg --configure -a
    

    Then:

    sudo apt-get install -f
    

    After that, I reloaded the systemd configuration:

    sudo systemctl daemon-reload
    

    Step 6 — Synchronize the package system again

    I ran another update and upgrade:

    sudo apt update
    
    sudo apt upgrade
    

    At this point, the packages were in a much cleaner state.


    Step 7 — Start MySQL

    Finally, I enabled MySQL to start automatically:

    sudo systemctl enable mysql
    

    Then started it:

    sudo systemctl start mysql
    

    And checked its status:

    sudo systemctl status mysql
    

    MySQL was running again.

    That was the moment I knew the recovery had worked.


    Step 8 — Check the MySQL data permissions

    I also checked the ownership and permissions of the MySQL data directory.

    If necessary:

    sudo chown -R mysql:mysql /var/lib/mysql
    

    And:

    sudo chmod -R 750 /var/lib/mysql
    

    Then:

    sudo systemctl restart mysql
    

    And finally:

    sudo systemctl status mysql
    

    Give MySQL some time to recover

    One thing I learned from this incident is that a successful systemctl start mysql does not necessarily mean everything is immediately ready.

    Depending on the size and state of the database, MySQL may need some time to perform recovery or initialization work.

    So after getting the service running, I would not immediately assume everything is fine.

    I would check:

    sudo systemctl status mysql
    

    and inspect the MySQL error log if anything looks suspicious.

    For example:

    sudo journalctl -u mysql
    

    You should also verify that the actual databases are present and accessible.


    What I learned

    The biggest lesson from this incident is:

    Don’t treat a MySQL version change like a normal application upgrade.

    MySQL is not simply an executable that can always be replaced with a newer executable.

    The MySQL server interacts closely with its data directory, data dictionary, system tables, storage engine metadata, configuration, and installed packages.

    MySQL’s own documentation describes an in-place upgrade as replacing the old binaries and then starting the new server against the existing data directory. The upgrade process can make changes to that data directory. (MySQL)

    This is why blindly switching between versions can be dangerous.


    Don’t assume you can simply downgrade

    Another important lesson is that:

    Downgrading MySQL is not necessarily the reverse of upgrading MySQL.

    MySQL documents specific supported downgrade paths. Depending on the versions involved, a logical dump/restore or restoring a backup may be required rather than simply installing the older package and pointing it at the newer data directory. (MySQL)

    For example, MySQL’s documentation explicitly recommends restoring a backup taken before an upgrade when the desired downgrade is not supported. (MySQL)

    Therefore, my recovery strategy was essentially:

    Protect the data → remove the broken packages → repair the package system → restore the appropriate MySQL package installation → start MySQL → verify the existing data.


    An important distinction: package recovery vs. database recovery

    This distinction is worth understanding.

    There are actually two separate problems:

    Problem 1 — Broken Ubuntu/Debian packages

    Symptoms may include:

    dpkg was interrupted
    

    or:

    package is not configured
    

    or dependency errors.

    The relevant tools are things like:

    dpkg --configure -a
    apt-get install -f
    apt clean
    apt update
    apt upgrade
    

    Problem 2 — MySQL cannot open or upgrade the data directory

    This is much more serious.

    For example, MySQL may fail because the data directory has been modified by a newer server version, because of incompatible metadata, configuration problems, permissions, storage-engine problems, or other upgrade-related issues.

    In that situation, do not start randomly deleting files from /var/lib/mysql.

    Preserve the data first.


    What I would do differently next time

    If I were performing this Ubuntu upgrade again, my procedure would be:

    1. Confirm the current MySQL version

    mysql --version
    

    2. Identify where MySQL came from

    Check the installed packages:

    dpkg -l | grep mysql
    

    And check the configured repositories.

    This is important because Ubuntu’s own repositories and the official MySQL APT repository can provide different MySQL release lines. The MySQL documentation also recommends checking which MySQL release series is selected in the MySQL APT repository before upgrading. (MySQL)

    3. Make a proper database backup

    Do this before upgrading Ubuntu.

    4. Make a copy of /var/lib/mysql

    As an additional safety measure:

    sudo cp -R /var/lib/mysql /var/lib/mysql_backup
    

    5. Check MySQL upgrade compatibility

    Do not assume:

    8.0 → newer version → 8.0
    

    is always reversible.

    Check the supported upgrade and downgrade paths for the exact versions involved.

    6. Upgrade MySQL separately from the Ubuntu upgrade

    Where possible, avoid changing two major components at the same time.

    For a production server, I would prefer:

    Backup
       ↓
    Verify backup
       ↓
    Check MySQL compatibility
       ↓
    Upgrade MySQL
       ↓
    Verify MySQL
       ↓
    Upgrade Ubuntu
       ↓
    Verify MySQL again
    

    rather than:

    Upgrade Ubuntu
          +
    Upgrade MySQL
          +
    Change MySQL repository
          +
    Change MySQL version
    

    all in one operation.


    My recovery checklist

    For anyone facing a similar situation, this is the short version:

    1. STOP
       Do not wipe /var/lib/mysql.
    
    2. BACKUP
       Make a copy of the MySQL data directory.
    
    3. IDENTIFY
       Determine which MySQL version and repository you are actually using.
    
    4. PURGE
       Remove the broken MySQL packages if the package installation itself is corrupted.
    
    5. REPAIR
       dpkg --configure -a
       apt-get install -f
    
    6. RESTORE
       Install the intended MySQL package/version.
    
    7. CHECK PERMISSIONS
       Make sure MySQL owns its data directory.
    
    8. START
       systemctl start mysql
    
    9. VERIFY
       systemctl status mysql
       journalctl -u mysql
    
    10. CHECK DATA
        Confirm your databases and tables are actually accessible.
    
    11. BACKUP AGAIN
        Once MySQL is healthy, create a fresh verified backup.
    

    Final lesson

    The most important lesson I took away from this incident is simple:

    Before upgrading MySQL, protect the data and understand the upgrade path.

    A MySQL version change can involve much more than replacing packages. The server may upgrade internal metadata and other components of the data directory. MySQL itself recommends backing up before an upgrade, and unsupported downgrades may require restoring a pre-upgrade backup rather than simply reinstalling the older version. (MySQL)

    In my case, I was fortunate that I did not wipe /var/lib/mysql.

    The broken MySQL packages could be removed and reconstructed. The existing database files were the part that needed to be protected.

    If you are about to upgrade Ubuntu on a server running MySQL, make the database backup first.


    Sources

    I would publish this as a “MySQL recovery after Ubuntu upgrade” article rather than simply a diary of what happened. That makes it much more useful to someone who is desperately searching Google at 2 a.m. because mysql.service suddenly refuses to start.