Introspection APIs

[eschaton]

I've been doing ANSI C projects for fun the past couple of years, and I'll often start out writing my tests using XCTest in Xcode out of convenience, and then migrating them to Check for wider platform support. (I worked on Xcode for a couple decades before retiring a couple of years ago, and was responsible for integrating testing support into Xcode way back in Xcode 2.1…)

Like other IDEs, Xcode's test infrastructure is at least somewhat amenable to a degree of integration with Check. However, to get better integration than just catching failures, Check will need some additional APIs beyond getting a TestResult collection after a run. In particular, I would need to be able to iterate over the Suite and Test instances in a Suite before running them, and to be able to run an individual Test or Suite.

Having these APIs would allow me to create XCTestSuite and XCTestCase subclasses that can (at setup time) generate further synthetic subclasses and methods at runtime that can be used to map Xcode's runtime test specifiers to the specific Check Suite and Test to invoke, and then report their state back to Xcode in a way it'll present as success or failure with proper source annotation. (None of this should require an Xcode plug-in or anything of the sort, it should all be doable within the XCTest API if Check itself has the necessary API.)

[brarcher]

Sounds interesting!

I'm not that familiar with using XCode. If you think adding introspection APIs would enable running unit tests and mapping the results to source code in XCode, give it a try. As long as other supported platforms for Check work just fine too it should be ok.

[eschaton]

Thanks! Xcode (lowercase "c") is an IDE so in addition to editing and building your code, it also has features like presenting the results of tests and running both all and a subset of tests or an individual test for rapid iteration scenarios like test-driven development.

Adding full support for Check to Xcode would require participation from my former coworkers in Xcode engineering, since Xcode doesn't expose that kind of plug-in API. However, because Xcode's testing framework, XCTest, is written in a dynamic OO language (Objective-C), I can add a substantial amount of support for reporting test results and making it possible to re-run tests after they've run once as long as I can actually introspect the structure of the tests before they've been run.

[eschaton]

Here are the APIs that I think would be useful for this:

size_t srunner_nsuites(SRunner * sr);
Suite **srunner_suites(SRunner * sr);

const char *suite_name(Suite * s);
size_t suite_ntcases(Suite * s);
TCase **suite_tcases(Suite * s);

const char *tcase_name(TCase * tc);
size_t tcase_ntests(TCase * tc);
TTest **tcase_tests(TCase * tc);
size_t tcase_ntags(TCase * tc);
const char **tcase_tags(TCase * tc);

They should be self-explanatory; the tags APIs aren't something I'd use in Xcode integration, but I could see someone else wanting them and included them for completeness. The string-returning APIs return pointers into Check-managed memory, the array-returning APIs return newly-allocated arrays that need to be freed with free(3) but whose contents don't need to be, just like with srunner_results().

I also wonder if the result of tcase_tests() should be an array of TTest * or of some other struct that can be interrogated for its associated TTest * and its signal, iterations, etc., basically the parameters to _tcase_add_test() that get packaged into the TF that's actually part of tc->tflst. Maybe exposing TF, opaquely with accessors, would be the right thing for the array returned by tcase_tests()?

[brarcher]

I think the APIs sound reasonable. Regarding tcase_tests(), what's returned should have some path to getting at the other details like you brought up (signal, iterations, etc). Exposing TF could work with some additional helper methods could work.