Testing, verification and test automation

[monperrus]

Wikipedia references:

• https://en.wikipedia.org/wiki/Software_testing
• https://en.wikipedia.org/wiki/Unit_testing
• https://en.wikipedia.org/wiki/Regression_testing
• https://en.wikipedia.org/wiki/Code_coverage

"Automated testing is the act of conducting specific tests via automation (e.g. a set of regression tests) as opposed to conducting them manually, while test automation refers to automating the process of tracking and managing the different tests." source

[bbaudry]

Interesting tool.
Detailed failure reports and hassle free assertions
Scott test reporter
https://github.com/dodie/scott

[monperrus]

From @AG00:

There is a relationship between desired properties (requirements), tests (verification) and test results. Those may be present in a hierarchy of components/subsystems/libraries. In addition, there is a relationship between planning activities on the system (product) being developed and its desired properties. Further, with time there will be more than one system/product configuration to consider and more than one version in use with different installations/deployments/customers. These relationships quickly becomes complex to understand and keep track of. As changes are being made (code or documents) and automatic tests are run, it would be good to have the information model changes generated rather than manually written. The information model can also be used to convince a customer (or the devops team itself) that a complex/high-level system property is achieved.

[bbaudry]

Testcontainers to test different environments on the developer's machine
https://www.testcontainers.org/

[bbaudry]

A post about the challenges to get global metrics for code coverage
https://massol.myxwiki.org/xwiki/bin/view/Blog/GlobalLocalCoverage

[bbaudry]

Security-based testing
https://www.inspec.io/

[bbaudry]

DeFlaker: Automatically Detecting Flaky Tests
https://www.jonbell.net/icse18-deflaker.pdf

[bbaudry]

Property-based testing in Java
https://github.com/quicktheories/QuickTheories

[monperrus]

Cross Browser Testing, Selenium Testing, and Mobile Testing
https://saucelabs.com/

[monperrus]

Perfecto - Cloud-based Devops Testing
https://www.perfecto.io/

[monperrus]

Capturing test history
https://stackoverflow.com/questions/37657130/how-to-capture-the-junit-test-result-of-project-in-database

[monperrus]

Automated Tool for Prioritizing JUnit Test Suites
https://github.com/kanonizo/kanonizo

[bbaudry]

mutation testing
https://github.com/hcoles/pitest

[bbaudry]

automatic unit test amplification
https://github.com/STAMP-project/dspot
https://arxiv.org/pdf/1811.08330.pdf

[bbaudry]

automatic detection of pseudo-tested methods
https://github.com/STAMP-project/pitest-descartes
https://arxiv.org/pdf/1807.05030.pdf

[bbaudry]

mutation testing in JS
https://stryker-mutator.io/

[monperrus]

Property-based testing in OCaml
https://github.com/c-cube/qcheck

@robertfeldt you may want to subscribe to this issue.

[monperrus]

Chris Parnin's Lecture on software bots
https://docs.google.com/presentation/d/1JClXhqNX8ufzmxoDpVPCLRDlqu4tUkvbJgwYICw70kM/edit?usp=sharing

[bbaudry]

Testing of Microservices

https://labs.spotify.com/2018/01/11/testing-of-microservices/

[bbaudry]

Amazon codeguru
It’s like having a distinguished engineer on call, 24x7
https://aws.amazon.com/codeguru/

[matsskoglunds]

Postman/Newman https://www.getpostman.com/

[bbaudry]

New testing course from TU Delft
https://sttp.site/

[monperrus]

Rultor is a DevOps team assistant. It helps your programmers and release managers automate routine operations (merge, deploy and release), with an easy-to-use intuitive chatbot interface.
https://github.com/yegor256/rultor

[bbaudry]

A series of recent posts about test and automation
http://blog.testingcurator.com/2020/04/12/testing-bits-april-5th-april-11th-2020/

[monperrus]

ScalaTest: https://github.com/scalatest/scalatest

[monperrus]

Jasmine is a behavior-driven development framework for testing JavaScript code.
https://jasmine.github.io/

[monperrus]

Catch2: modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++
https://github.com/catchorg/Catch2

[bbaudry]

Commonly used test data
https://github.com/search?p=2&q=dQw4w9WgXcQ&type=Code

[monperrus]

Commercial product to accelerate tests with an automated dependency graph
https://yourbase.io/

[bbaudry]

Automated Cucumber-JVM test pack for DECS built on the Serenity BDD framework
https://github.com/UKHomeOffice/hocs-serenity-automation

[bbaudry]

Testing at Electronic Arts
SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning
https://arxiv.org/pdf/2203.05566.pdf

[bbaudry]

Property-based testing in language X
For example, in javascript
https://marmelab.com/blog/2019/04/18/property-based-testing-js.html

[monperrus]

An interesting essay / demo / tutorial topic: Automated migration of Junit4 to Junit5

[monperrus]

Suggestion: Testing for GraphQL APIs and Engines.

[bbaudry]

An interesting and very original topic for a presentation / demo in software testing:
code coverage in production

[monperrus]

Topic suggestion: in depth analysis of a mocking library in language X (eg Mockito)

[bbaudry]

fuzz testing is an essential topic for testing and security

[monperrus]

Robot Framework is a generic open source automation framework. It can be used for test automation and robotic process automation (RPA). https://robotframework.org/

[bbaudry]

Predictive Test Selection cuts testing time up to 70% by using machine learning to select and run only tests that are likely to provide useful feedback during test runs
https://gradle.com/gradle-enterprise-solutions/predictive-test-selection/

[bbaudry]

Load-Test-As-Code. https://gatling.io/

[bbaudry]

Instancio is a Java library that automatically creates and populates objects for your unit tests.
https://github.com/instancio/instancio

[monperrus]

Mimicking Production Behavior with Generated Mocks 2022

[monperrus]

Mutagen and cargo-mutants are two relevant mutation testing tools for Rust.

Credits @VIGGEEN
cc/ @ErikNatanael

[monperrus]

following up on Jepsen
JepREST: Functional tests for distributed REST applications.

[bbaudry]

A command-line benchmarking tool for rust
https://github.com/sharkdp/hyperfine

[bbaudry]

BDD with Rust https://github.com/cucumber-rs/cucumber

[bbaudry]

A library for generating fake data in Rust.
https://github.com/cksac/fake-rs

[bbaudry]

The art and craft of test-driven development (Increment, 2019)

[monperrus]

Integrity Test Framework - a sophisticated integration test automation framework for Java
https://github.com/integrity-tf/integrity

[monperrus]

Newman is a command-line collection runner for Postman
https://github.com/postmanlabs/newman

[bbaudry]

Framework for web testing, and capture&replay
https://github.com/microsoft/playwright

[monperrus]

Kurtosis is a composable build system for multi-container test environments.
https://docs.kurtosis.com/