Clever use of testing outputs in your Development Pipeline

RIGHT

Andreas Mautz

  • first program in 1996
  • first magento shop in 2008
  • 3 years of QA in big projects before
  • CTO @ webvisum
  • FireGento Board Member
  • @mautz_et_tong
Welcome 1st testfest (as speaker) BOTTOM
First things first: I am not a native speaker, so if anything is unclear because of "words" interrupt
me. Otherwise I please you to ask all topic related stuff at the end.
Why am I supposed to speak about testing (Spoiler alert: yes, this is a talk about testing, I have seen the slides before)?
I do Magento since 10 years for a living and own a small agency in cologne.
I have learned to test stuff the hard way, multiple times.
So, what is this talk about

AGENDA:

  • Is this relevant for you?
  • What do you need to start?
  • What will you learn?
Speaker are not experts. They are storytellers. I have read this on twitter, so it must be true. So, the last Point is a story about my learnings

Is this relevant for you?

RIGHT

It depends

But you are here, sooo

yes!

This is relevant for

Developers

  • testing experience is a must
  • testing saves time and costs
  • testing improves quality

DevOps People

  • test chaining
  • pipeline design
  • threshold configuration
RIGHT
By now, everything is widely documentated test your personal diff with grumphp get warm by testing on your setup but test with your company rules / framework rules / wild mix focused on the technical part of testing, I say: developers writing tests and test their own code, This is not about the consence, which test may fail, which on e should break a prod deployment.

What do you need to start?

How long will it take you to start?

It depends

Preparation and TTFR*

*TTFR = Time To First (Test) Result

RIGHT
Fast and cheap: don't test at all Good and fast: start local testing with grumphp and start blaming yourself Good and cheap: framework (when it is done) - use given tools only

What will you learn?

Or better: What have I learned?

hopefully, some starting points or improvements for your testing pipeline

1. start a diet

  • Test locally and save pipeline runs
  • Test every case once in your pipeline
  • Set thresholds and work against them
  • Constantly work on your numbers
RIGHT
diet, test as less as possible in the project pipeline but as much as needed The faster your pipeline is, the better use grumphp for local pre-commit testing you delta test each part of your software once in the pipeline there is no need for codesniffing a module twice whitout a change beeing made

2. divide and conquer

  • separate framework (core) testing, third party modules and local stuff (if you have it, otherwise, test own modules like third party)
  • run each test tool parallel on a single job for saving time
RIGHT
Your core version does not change as often as the code is changed by a third party or the code is changed. If so, turn the next one over. we build weekly automated magento core builds which are tested vanilla -> all versions that we use plus one ahead if there is one we build automated project builds including current magento version and third party modules (nighlys) we use grumphp to test every commit with different static tests on code smell (phpcs, phpmd, psalm) 10 jobs with 30 minutes serial: 300 minutes, 5 hours. 10 jobs with 30 minutes parallel with 5 runner: 60 minutes

3. Avoid building the tower of babel

  • generalize the setups for all project members / company devs / ...
  • have a common test context (Where to test what)
  • imitate your productive environment as close as possible
regex all the stuff: make a quality number out of summaries, reports and return 0,1,2

4. Use the force

  • Get the main information out of each report (regex FTW)
  • Create a comprehensible ground
  • Read and discuss these results and measurements
runned test are just fire and forget. Informations are lost (beside gitlab job history) regex all the stuff: make a quality number out of summaries, reports and return 0,1,2 if you want to measure things over time e.g detecting trends, you have to collect and compare against time discuss the complete image of all test, after running the tests separately, dig deeper into on topic if needed
  • Repetingly rethink your testing behaviour
    • clever: - configurable gitlab params for finetuning projects an make projects better and better without hazing anybody - Don't think of traffic lights in testing aspects, when you start. You won't have a code coverage of 100% overall in the beginning. define guard rails - seperate the test run from the analysis - - run tests with summary - collect summaries - evaluate them
    Don't forget: raising code quality is doing a part of a mental health program in your company

    Questions?

    Thank you!

    RIGHT
    https://www.youtube.com/embed/VM-2OVNt-eQ?controls=0

    SOURCES