(Lack of) Test Discovery in C

2023-08-10

I’ve been doing some coding in C lately, a change from my usual C++ ways. As part of those projects, I’ve been writing tests. I didn’t want to use any testing frameworks, so I kept the test code simple:

// Each test returns 0 on success, otherwise it has failed.
int testA(void) {
    // ...
}

int testB(void) {
    // ...
}

int testC(void) {
    // ...
}

int main(void) {
    int ret = 0;

    if (testA()) ret = 1;
    if (testB()) ret = 1;
    if (testC()) ret = 1;

    return ret;
}

This program would get called when running make test. On success, the program returns 0 and the make command succeeds. Otherwise, the program returns 1 and the command fails.

There were some problems with this approach. Mainly, when adding new test cases I need to remember to add function calls to them in main. If I forget, the test case will not get called and I would not know. It has happened that I added a test case and only two days later realized it was never executed. Fortunately, it did pass when I added it later, so there were no undiscovered bugs there, but it’s still too much of a risk not being aware that some of my tests are not being run.

I decided that I should try to address this problem somehow. Ideally, I could just define a new test function and have it automatically called from main along with all the other ones.

I went looking through existing C test frameworks to see how they accomplished this, but they seem to also require the user to manually add new tests to a list of functions somewhere. I don’t know if they warn you if a test never gets called.

I also found one very interesting code snippet that does test discovery using __attribute__((section ("..."))) and similar compiler features. However, I decided that that was too advanced and non-portable.

In C++, you can implement test discovery by abusing constructors of global variables. This is how GoogleTest does it with its TEST() macros. ^{And in OrbLang, it would be even easier to accomplish the same thing, and with no runtime overhead.}

But I’m not using C++ here, so I started reading up on how the C preprocessor works and what you can do with it. It turns out that it is (intentionally) very limited. I came up with two alternative sorta-good solutions.

Solution 1

I accidentally discovered that marking a function as static and compiling with GCC or Clang with the -Wunused-function flag (included by -Wall) emits a warning if that function is unused. Compiling with -Werror turns that warning into a compile error, so you are forced to fix it.

This is not test discovery, it does not automatically add calls to new test functions in main, but it does protect you from forgetting to call a new test.

The modified test code would look like:

static int testA(void) {
    // ...
}

static int testB(void) {
    // ...
}

static int testC(void) {
    // ...
}

int main(void) {
    int ret = 0;

    if (testA()) ret = 1;
    if (testB()) ret = 1;
    if (testC()) ret = 1;

    return ret;
}

and you’d compile it with gcc -Wall -Werror .... It’s simple, it’s only a small change to how tests are written, but you need to remember to add static to each new test function.

If, for whatever reason, you want not to be warned about a static function, mark it as inline in addition to static.

Solution 2

This solution actually does automatically call all test cases. The idea was to create a linked-list, conceptually at least, between all of the test cases during preprocessing. That is easier said than done and this solution is not perfect, but here it is in all its glory:

typedef int (*TestFunc)(void);

#define PREV_TEST NULL
TestFunc prevTest = NULL;

int testA(void) {
    prevTest = PREV_TEST;
#undef PREV_TEST
#define PREV_TEST testA

    // ...
}

int testB(void) {
    prevTest = PREV_TEST;
#undef PREV_TEST
#define PREV_TEST testB

    // ...
}

int testC(void) {
    prevTest = PREV_TEST;
#undef PREV_TEST
#define PREV_TEST testC

    // ...
}

int main(void) {
    int ret = 0;

    TestFunc f = PREV_TEST;
    while (f != NULL) {
        if (f()) ret = 1;
        f = prevTest;
    }

    return ret;
}

Each test function sets a global variable prevTest to point to the previous test declared in code. This is done at runtime.

During preprocessing, the value of PREV_TEST gets modified to refer to the name of the current test function. Thus, the next function will know what to set prevTest to.

main will end up calling the final function referred to by PREV_TEST. main needs to be defined after all of the test functions have been defined.

Using this trick, you get a hacky way for automated test discovery in C. When adding new tests, you need to remember to copy those first three lines into the function and update the third one with the name of your new function. You can also freely reorder the tests without having to modify anything inside them.

It’s not perfect and if you forget to update the third line you can end up with one or more test cases being skipped altogether. But it’s the best I could come up with. C macros don’t allow for creating and calling new preprocessor directives (if they emit #define ..., it will not be invoked). And __func__ creates a string literal, which cannot be used in lieu of the function name.

The final flaw is that the tests will be called in reverse - testC, testB, testA.

Conclusion

If I was better at C, maybe I could come up with a more practical approach. For now, I think I’ll stick to the first solution. It’s simple and all I really needed was to not forget to call any of the tests.

Link dump

https://jadlevesque.github.io/PPMP-Iceberg

https://marc.info/?l=boost&m=118835769257658&w=2

http://www.throwtheswitch.org/unity

https://cunit.sourceforge.net

https://github.com/google/googletest

https://gist.github.com/nickrolfe/ffc9b1c02381b9dc17c975b98db42172