Success: Upgrading Magento Open Source 2.3.5 to 2.4.4 (Experiments & Research)

YouTube · April 19, 2022, 11:15am

Introduction

For those of you who had a chance to join me during the recent Live Stream, thank you for hanging out. It was good being able to exchange ideas with you in real-time. For those who missed it, you can watch the recording below.

Attempt 1 (Failure)

I’ve labelled this an “attempt”, as we ran into a catastrophic failure at the end. Therefore, I had to rollback the server to before I started. This is why we take backups!

At the end of the video, I promised to follow up with a progress update. Well, here is is.

Attempt 2 (Success!)

Important. Read before copying.

Do not blindly copy/paste everything you see here. (This post is for research purposes only):

Do not think of this as a “Tutorial”

These are the notes I made during my successful attempt

These notes are specific to my scenario

Try to understand my thought process when debugging your own upgrade

There is definitely room for optimisation here

Ensure you test your changes after each step, no matter what that step was. I did not document my tests.

This time around, I figured I’d tackle the upgrade in stages. So rather than jump straight from 2.3.5 to 2.4.4, I’d instead go from 2.3.5 to 2.3.7-p3 to 2.4.4.

1. Upgrading Magento 2.3.5 > 2.3.7-p3

1.1 Preparing the work space

Logged in as Super Admin (not magento user)

Updated Ubuntu OS

sudo apt update
sudo apt upgrade -y

Set a 4G swapfile, because my sandbox server memory isn’t great

sudo fallocate -l 4G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile

Figured I’d make sure the Magento folder ownership was correct.

sudo chown -R magento:apache /var/www/html/

1.2 Installing Magento Composer Plugin

Switched to Magento User and navigated to Magento Root Directoy

sudo su magento
cd /var/www/html/

Manually updateded composer.json due to PHP conflict issues

vi composer.json and updated 2 entries.

"friendsofphp/php-cs-fixer": "~2.16.0", (changed from 2.13.0)
"magento/magento2-functional-testing-framework": "2.3.9", (changed from ^2.7)

The reason I determined these 2 files were a problem, was because each time I ran composer update, it would have an issue with these lines. So, I referenced composer.json for v2.3 on the Magento GitHub and swapped the version numbers for what I found there. I didn’t want to change everything just in case. So, I did it one at a time until no errors display during composer update.

Added the composer plugin

require magento/composer-root-update-plugin ~2.0 --no-update

What’s interesting here is that the Magento Docs state that if you run PHP 7.3 or greater, then you need to install version 2.0 of this module. However, in one of the steps coming up, I’m forced to downgrade it to 1.0. No idea why.

Ran composer update

php -d memory_limit=4G /usr/local/bin/composer update

Then flushed the caches and ran Setup Upgrade

redis-cli FLUSHALL
rm -rf var/cache/* var/page_cache/* generated/code/*
bin/magento setup:upgrade

1.3 Upgrading to 2.3.7-p3

Now I upgraded from 2.3.5 to 2.3.7-p3. Note the --interactive-root-conflicts tag that I used. This was a really helpful tag and solved many issues. I just pressed “Y” at each prompt.

composer require-commerce magento/product-community-edition 2.3.7-p3 --no-update --interactive-root-conflicts

Ran composer update

php -d memory_limit=4G /usr/local/bin/composer update

Then flushed the caches and ran Setup Upgrade

redis-cli FLUSHALL
rm -rf var/cache/* var/page_cache/* generated/code/*
bin/magento setup:upgrade

Win! I was now on Magento 2.3.7-p3. Tests all OK.

2.1 Upgrading Magento 2.3.7-p3 > 2.4.4

Now it was time to attempt changing branches. This was a little more technical, because a variety of applications needed installing/upgrading and configuring (e.g. PHP and Elasticsearch).

2.2 Preparing the work space (again)

Exited from Magento User back to Super Admin

exit

Upgraded Ubuntu 18 to 20. I did this because I needed access to the latest repositories so that I could install PHP and upgrade MySQL.

sudo do-release-upgrade -y

After the server restarted, I logged in as Admin

Then I had to set the swapfile again (as the this is reset on reboot)

sudo fallocate -l 4G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile

2.3 Update composer from 1.x to 2.x

At this point, Magento 2.3.7-p3 or greater requires composer 2.x. So, I updated it.

sudo composer self-update

2.4 Upgrade PHP 7.3 to 7.4

I installed PHP 7.4 and all required modules

sudo apt install php7.4 libapache2-mod-php7.4 php7.4-mysql php7.4-soap php7.4-bcmath php7.4-xml php7.4-mbstring php7.4-gd php7.4-common php7.4-cli php7.4-curl php7.4-intl php7.4-zip php7.4-mbstring php7.4-gettext zip unzip -y

Then switched from PHP 7.3 to 7.4

sudo update-alternatives --config php
sudo a2dismod php7.3
sudo a2enmod php7.4

Then updated the configuration file, changing all 4 values:

sudo vi /etc/php/7.4/apache2/php.ini

realpath_cache_size = 10M
realpath_cache_ttl = 7200
memory_limit = 2G
date.timezone = Europe/London

Then a quick restart for the changes to take affect

sudo systemctl restart apache2

2.5 Install Elasticsearch

Essentially, I used the documentation from one of my older posts. But I swapped out the following version numbers (that latest supported version at the time of doing this):

JDK from 8 to 11
Elasticsearch 7.6 to 7.16

I moved to the Super Admin home directory. This is because I’m about to download some files.

cd ~

Then installed JDK 11

sudo apt install openjdk-11-jdk -y

Then went through the motions of installed and running Elasticsearch. Note, I did not bother configuring Elasticsearch during this test because it runs fine out the box.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.16.0-amd64.deb
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.16.0-amd64.deb.sha512
shasum -a 512 -c elasticsearch-7.16.0-amd64.deb.sha512
sudo dpkg -i elasticsearch-7.16.0-amd64.deb
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo systemctl start elasticsearch

Now that Elasticsearch was installed, I needed to log into the backend and update the engine to Elasticsearh 7.0+ (Stores > Settings > Configuration > Catalog > Catalog > Catalog Search)

Time to switch back to Magento User and navigate to the Magento Root Directory

sudo su magento
cd /var/www/html/

Followed by a quick Reindex and confirmation that Magento was now on Elasticsearch

bin/magento indexer:reindex
bin/magento config:show catalog/search/engine

2.6 Update Cron Jobs

The old cron jobs needed updating for 2 reasons:

They refer to an old PHP version
2.3.7 has deprecated a couple of the lines. So, this results in jamming up your log files with a bunch of errors

So, I loaded up the crontab and deleted all the lines referring to Magento

crontab -e

Then ran the Magento script to reinstall the latest correct version

bin/magento cron:install

2.7 Reinstalling Magento Composer Plugin

Remember earlier, when I installed this plugin and then it got downgraded to 1.0? Well, it’s time to update it again to 2.0

composer require magento/composer-root-update-plugin ~2.0 --no-update

Ran composer update

php -d memory_limit=4G /usr/local/bin/composer update

Then flushed the caches and ran Setup Upgrade

redis-cli FLUSHALL
rm -rf var/cache/* var/page_cache/* generated/code/*
bin/magento setup:upgrade

Ok. So I ran into 2 errors whilst running the upgrade command. One went away when I ran the upgrade command again. The other required me to change a variable in MySQL. I did this by logging into phpMyAdmin as root. Then searched for this entry in variables (log_bin_trust_function_creators). Changing it from OFF to ON. No idea what this actually does, but it solved the issues. Requires more research.

2.8 Upgrading to 2.4.4

So far, so good I guess. Time to update from 2.3.7-p3 to 2.4.4. Note, how I used the --interactive-root-conflicts tag again. Pressing “Y” at every prompt.

composer require-commerce magento/product-community-edition 2.4.4 --no-update --interactive-root-conflicts

I also needed upgrade all of my Sample Data (as my Sandbox instance has Sample Data preinstalled already

composer require magento/module-bundle-sample-data:100.4.* magento/module-widget-sample-data:100.4.* magento/module-theme-sample-data:100.4.* magento/module-catalog-sample-data:100.4.* magento/module-customer-sample-data:100.4.* magento/module-cms-sample-data:100.4.*  magento/module-catalog-rule-sample-data:100.4.* magento/module-sales-rule-sample-data:100.4.* magento/module-review-sample-data:100.4.* magento/module-tax-sample-data:100.4.* magento/module-sales-sample-data:100.4.* magento/module-grouped-product-sample-data:100.4.* magento/module-downloadable-sample-data:100.4.* magento/module-msrp-sample-data:100.4.* magento/module-configurable-sample-data:100.4.* magento/module-product-links-sample-data:100.4.* magento/module-wishlist-sample-data:100.4.* magento/module-swatches-sample-data:100.4.* magento/sample-data-media:100.4.* magento/module-offline-shipping-sample-data:100.4.* --no-update

Ran composer update

php -d memory_limit=4G /usr/local/bin/composer update

Then flushed the caches and ran Setup Upgrade

redis-cli FLUSHALL
rm -rf var/cache/* var/page_cache/* generated/code/*
bin/magento setup:upgrade

Win! I was now on Magento 2.4.4. Tests all OK.

Summary

And that’s that. As I mentioned at the beginning. These are simply my notes, that I’m sharing with you. If I had time, I’d start over to ensure all of my notes are correct. Use at your own risk. Let me know if this helped you.

Useful References

Update (21st April 2022)

I’ve upgraded 2 of my other servers/projects from 2.3.x to 2.4.4 now. The steps I’ve documented here have been invaluable in getting me through it.

In each case, I’ve had to make a couple of minor tweaks (everyone will be different in the real world) and it’s been fine.

The only exception to this is that some of my 3rd party module are not compatible with 2.4.4 yet. So, I could only upgrade to 2.4.3-p2.

The standout things that I took away were:

Ensure that you only update the composer from v1 to v2 after requiring the composer-root-update-plugin (as indicated in the steps above).
--interactive-root-conflicts is an truly awesome tag to use once your on the new composer-root-update-plugin. Beause it prompts you to check your decencies. This seems to negate the times where I’ve had to manually mess with composer.json. This tool should make upgrades less “hit and miss” in the future

digitalstartup · April 19, 2022, 9:52pm

Want more Live Streams? Like this post to let me know.

live4soccer7 · April 20, 2022, 5:37pm

Great to see you back at it again! I hope all has been well.

digitalstartup · April 20, 2022, 6:27pm

Thank you. Just trying to do everything in bitesize chunks

live4soccer7 · April 20, 2022, 6:39pm

I hear ya. Balancing life, work, and hobbies can be quite a tough task.

digitalstartup · April 26, 2022, 2:00pm

2 posts were split to a new topic: Is it safe to disable Redis Commands?

Stephen.Ro · April 26, 2022, 2:07pm

Appreciated and thanks for your help always, thank you Craig.

live4soccer7 · May 7, 2022, 5:44pm

@digitalstartup I see in Magento’s documentation they list opensearch 1.2 as a requirement. Is this something that is only required if you don’t want to use elasticsearch and can then be ignored?

digitalstartup · May 7, 2022, 11:06pm

From what I understand, you choose one or the other. I believe they made Opensearch an option, because of some sort of licensing dispute or something like that. It’s a slightly older fork of Elasticsearch. No idea which one is better, so I’m just sticking with Elasticsearch.

Also, you may want to fact-check this, I think Opensearch support is only for the “Commerce” version right now (not OpenSource)?

live4soccer7 · May 9, 2022, 5:23am

Thanks for the reply! I was thinking along the same lines.

live4soccer7 · May 11, 2022, 5:49pm

@digitalstartup As you may or may not recall, but I had a journey to select a host and chose cloudways. They have been fantastic with the exception of one aspect which I have been affected by twice now. They are not quick to update Magento Dependencies all the time.

Right now they have a max Elasticsearch version of 7.9, where the latest magento patch and release require 7.16. I have been bugging them for a month now with the same answer that they are testing.

What’s your take on upgrading to 2.4.3-p2 and just keeping 7.9 until 7.16 is available on cloudways?

digitalstartup · May 11, 2022, 11:09pm

@live4soccer7 Ah, that’s a bit frustrating

I don’t see any issue with sticking with 2.4.3 (so long as you keep patching it with the security updates). 2.4.3 doesn’t reach End of Life until November 2022, so there’s plenty of time to Cloudways to pull their finger out.

For me, it’s not Cloudways that are stopping me from using 2.4.4. It’s actual some 3rd party extensions I use that don’t support the latest version yet. So, I’ll be sitting on 2.4.3-p2 for a while too.

As for using Elasticsearch 7.9 vs 7.16. I think that’s more of a “strong recommendation” rather than “it’ll break if you don’t upgrade it”.

Ref:

live4soccer7 · May 11, 2022, 11:46pm

@digitalstartup Starting at 2.4.3-p2 they have made 7.16 a requirement too where in p1 it was not so.

Localjohn · February 7, 2023, 5:55pm

Although code for these features is bundled with quarterly releases of the Magento core code, several of these projects (for example, Inventory Management and Progressive Web Applications (PWA) Studio) are also released independently. Bug fixes for Sinuano Noche projects are documented in the separate, project-specific release information available in each project’s documentation.