Introduction to E-commerce website testing – Non-Functional Testing

How to effectively test your e-commerce website? We will give you insights into the best practices of efficient e-commerce testing, this time more about Non-Functional Testing.

In our last blog post, we discussed aspects and ideas on how to handle functional testing of your e-commerce webshop. Have a brief look at the intro, for it is general. Now it is time to go to non-functional aspects of testing. This is one of the most neglected areas of software development. To some extent, it is understandable, because people focus on the functionalities they want to give to their users. They focus on what the shop is doing. And forget how the shop is doing it. This can hit back big time. So you need to think about non-functional requirements and testing from the very beginning.

• UX Tests
• Compatibility testing
• SEO and social media integration
• Localization and internalization
• Load & Performance testing
• Security testing
• Scalability, operations, and maintenance

Think about this:

• Amazon’s tests showed they would lose $1.6 BILLION every year if they slowed down by just one second
• 79% of online shoppers say they won’t go back to a website if they’ve had trouble with load speed.
• Google will reduce the number of crawlers on your webpage if the load speed exceeds 2 seconds
• A strong UX design can improve conversion rates by 400%
• Mobile users are 5 times more likely to abandon a task if a site isn’t optimized for mobile

So just imagine what you get when your webshop is 1) not well designed and 2) slow in loading and 3) confusing to navigate. I guess you have done the math yourself. And the funny thing is: the flip side of the coin is that you can make your bullet-proof functionalities worthless by having non-functional issues. The three of the aspects mentioned above are all non-functional issues.

So far it is clear that you need at least some non-functional testing for your e-commerce website. It is not so easy to decide to what extent you should go. But, a proper risk analysis should give you answers. As said in our previous blog it is a question of what you sell – if you want to sell old books from your garage and set up a small shop for it, then probably some basic security checks would be fine. On the other hand, if you run a multi-million-turnover shop, then you should go seriously and step-by-step, letting no stone unturned.

The three most important things you should focus on when doing non-functional testing are: Usability, Performance, and Security. And some more things, as we will see below.

UX tests

In a couple of facts stated above, you already got an impression of the importance of your UX. And just about now, around 2020, customer experience is taking over as the PRIMARY decision criteria for buying a product. It becomes even more important than product characteristics and price. Go figure.

UX is something you shouldn’t actually do afterward – after you have designed your shop and its workflows. UX should be the primary driver in the design, something you start with. It can give you lots of answers about what your users want, how they behave and this will steer your development efforts.

Use A/B testing for important design decisions. You might get some input upfront by asking users and sending questionnaires, but people react differently and more precisely when they see it.

Especially take care of

• Landing page
• User interaction flows

Compatibility testing

• different browsers
• different screen sizes
• responsiveness

SEO and social media integration

• everybody knows the importance of SEO nowadays. So proper design and further advancing of your SEO are necessary.
• once installed, you should test your SEO results and rankings regularly
• plenty of tools offer you possibilities to test your SEO – use them, there are very good ones out there
• don’t let your social media links lead customers to nowhere. Test all social medial links, including the ones that go out with your standard email reply, for instance.

Localization and internalization

If you are running an international shop or your country is multi-lingual, then you have to make sure your localization and language settings work just fine. Some users are very sensitive when they get the default settings of their website in the “wrong” language. Think about Switzerland – a small country with three language zones. You don’t want a user in Geneva to get your shop opened in German per default…

It is a specific kind of testing and it might require some traveling and VPN-ing.

When faking your location on your smartphone, in order to test internalization, take care to use both VPN connections and fake locations

Load & Performance testing

Enough reasons to have a good performance, right? Load & Performance testing can be a subject on its own.

Load and performance testing is a complex affair. You can pretty easily run into any kind of problems, like:

• false-positive results when testing with a high load, but a limited spectrum of data
• false-negative results when parametrizing wrongly and favoriting less used scenarios
• testing during limited periods of time and not knowing the picture of what happens in the system when being under stress for a longer period of time (data spillovers, slow DB lockings, the crash of your services, etc)
• testing a wrong configuration, different from your final production architecture (this is a sweet and frequent one…)
• bad scalability of your test environment, when you try to project results from the test environment into production
• running into any kind of unforeseeable issues like over-using your cache, unpredictable stub behavior, etc.

Proper planning, designing, and parameterization of your LPT (load and performance tests) are essential.

Take care of:

• Test scenarios. You need to choose the most relevant scenario, in the order and frequency of how they appear in the production. Do a lot of analysis and data crunching when doing this, because it often goes sideways.
• Parameterization. Also, you need to parametrize your test scenarios. Does search from the landing page happens 10 times per second or 50 times per second. Let your beliefs and hopes aside when doing this, trust the data.
• Tools. LPT can’t be done without specialized tools. Take care of which you use, for some of them, are quite costly (not to mention which ones), and require lots of experience to be able to work properly with them.
• Test environment. Make sure the architecture of your test environment corresponds to your production. Having a one-service test environment vs. four-service production doesn’t give you any valuable answer. Architectures need to match to get reliable answers, and environment powers can be scaled.
• Test data is necessary to generate enough variation. This is a common mistake – you take a handful of data (like 10 different products on the test environment vs. thousands in the production) and all your requests during LPT are fetched from the cache… After that, you actually know less about your system than at the beginning and it gives you false confidence. Use a large variety of data, which is in accordance with your production data. But, don’t just copy your production data, you are violating GDPR…
• Try to attack your system from many angles with different scenarios and loads, to get a feeling of how your system behaves under different scenarios.

Security testing

Once again, the range of security analysis and testing depends heavily on how large and prominent your shop is. For a garage sale one-off shop, you probably don’t need to do a full spectrum, and if you use a built-in functionality of a major provider, then your security will probably be just fine. However, if you run a large shop with a high turnover, security is something you need to take very seriously. People might (just might…) swallow bad performance and UX, but will never forget a security breach. The annoyance about a slow page is one thing, but a break into your system and stealing of your personal data is really unforgivable. Not to mention the stealing of credit card numbers. You can put yourself very deep into …. troubles here, with very tough legal consequences too.

So, do take security rather seriously, if you run a considerable e-commerce business.

• It’s not enough to run a penetration test tool over your page.
• You need to do a deeper security risk assessment by analyzing shop architecture, data flows, and business processes. Using OWASP’s top 10 evaluation of your risk, define and weight your security risks – and you have some, for sure.
• Thereafter you will have a clear list of your security testing priorities. It will probably include – as already said – not only Injection, but also authentification, data exposure, and others. Most probably, some manual hacking and reverse engineering will be necessary to do these tests. See https://owasp.org/www-project-top-ten/
• Once again, it is not enough to use a – luckily widely available – penetration test tool and run it over your webpage. There are potentially much greater holes in your security that you don’t see, and only a serious security risk assessment, combined with skilled ethical hacking, can uncover these holes.

Scalability, operations, and maintenance

Also, these kinds of tests are often forgotten, and the range of them depends again heavily on what kind of an e-commerce system you are running. If you use a standard plugin from a major hosted provider, you don’t need to take too much care of it.

But, if you have a complex architecture of your e-commerce landscape, then you need to take care of how easily you can scale your shop (think about the keyword: Christmas!), is it run to operate (monitoring, alarming, CI, deployments, etc) and maintain.

Plan these tests early, because they are very difficult – if not meaningless – once you have built the system completely. If you have a plan of what to do early on, you can run almost all the tests during your development cycle without much effort:

• Are the code and configurations easily deployable? well, you will have hundreds of chances to improve it during your development cycle, because you will do hundreds of deployments during the development.
• Monitoring? The same. During development, and especially during load and performance testing you will have to have sound monitoring to look into your system. You need to build it up together with the rest of the infrastructure.
• Scalability? Again, the same – during the LPT test you can practice your scalability, add and remove services, servers, instances, etc.