Do they even test this?!
That’s the question every tester dreads to hear, because it usually means we’ve let something really embarrassing slip into a release.
The real answer is, “Yes, we do,” though that doesn’t offer much comfort if you’re facing issues in production. Still, during quieter moments, people sometimes ask less rhetorically what kind of testing takes place in MariaDB. Let’s dive into that.
When a pull request (or an internal patch outside the PR system) is pushed into the MariaDB/server repository on Github, it is picked by the MariaDB server CI, Buildbot. The total number of the builders in the Buildbot is reported to be over a hundred, dynamic, and regularly updated, with new builders added and old ones removed when their operating systems reach EOL. The exact set of builders used depends on the MariaDB version the patch is intended for, but it will contain at least several dozen of builders, including standard release builds and instrumented binaries (ASAN, MSAN, UBSAN). In addition to the Buildbot, the patch is also built in AppVeyor.
After building the binaries, the Buildbot and AppVeyor run MTR tests. MTR is the official MariaDB test framework and a collection of tests, mainly designed for functional regression testing. While the number of test files or queries or lines aren’t particularly meaningful characteristics, those curious can count them themselves as MTR is a part of MariaDB source code and is included into official DEB/RPM packages and binary tarballs. It’s sufficient to say that the test set provides decent coverage, especially when run in full capacity, using different test “protocols” (in addition to executing plain SQL queries, it has several other modes, converting everything into prepared statements, views, stored procedures, using the cursors), platform-dependent variations, debug-only facilities, and environment specifics like different SSL versions or FIPS support. MariaDB unit tests are also run via MTR. The test collection is actively maintained, because the general MariaDB server policy is that every code patch must be accompanied by a test case covering it. During code review, the reviewer ensures the patch includes sufficient MTR tests. Exceptions occur when changes can’t be tested with MTR, but they are not common.
