I recently finished a project which required me to modify an annotation macro for Scala using the Scala Macros Paradise project. Because I was extensively modifying a macro that we were already using, I wanted to write tests to prevent breaking our current workflow and to check that the new portion of the macro worked properly. Immediately, I ran into a problem: how could I test something whose proper behavior could be to throw a compiler error? After some research, however, I discovered there are quite a few ways to test both the successful and the error cases of macros. To compare them, I wrote a very simple macro that appends
" Hello World" onto a String val.
When the above macro works correctly, it should expand this:
@Example val test: String = "A welcome"
val test: String = "A welcome" + " Hello World"
Of course, if the annotated code compiles, the simplest test is to just use it.
The above test shows that the
Example macro appends
" Hello World" to an annotated String val, but it offers no guarantees regarding correct compiler errors or the exact expansion.
shouldNot compile, and
shouldNot typeCheck, which at least allows for some simple compilation tests.
shouldNot typeCheck catches macro-caused compiler errors because the macros from Scala Macros Paradise are expanded after the parsing phase of the compiler and before type checking. Any macro errors are caught by the time the type checking phase is complete.
ScalaTest’s approach is good enough for simple compilation checks, but if there’s more than one way for macro compilation to fail, there’s no way to test that the correct compiler error was thrown. That’s where
illTyped from shapeless comes in handy.
illTyped is implemented as a macro which checks at compile time that the given String does not type check and that the error given matches the expected error. Once again, the phase of the compiler that performs the macro expansion allows type checking to capture the macro-generated compiler error. Both ScalaTest’s approach and
illTyped check the macro’s compilation when the test is compiled. However, ScalaTest automatically modifies the test result based on the compilation errors, while
illTyped simply makes the test fail to compile if the check fails.
Unfortunately, if the given code generates more than one compiler error,
illTyped only reports one.
illTyped also doesn’t capture important details like the position of the compiler error. If the annotated value is an entire class, the position of the error can be quite important to the user of the macro. The Scala Macros Paradise project checks for multiple errors and the correct position of errors by compiling files and comparing error output against expected error output in one of its own tests.
After testing for correct results when the macro compiles and making sure that it throws the expected compiler errors, the only thing left to check is that it actually expands correctly. If you use the scalameta paradise version of macros, it’s easy to check that two syntax trees are structurally equal which allows you to easily make sure the expansion of your macro happens as expected.
There are different advantages to each type of macro test, and I was able to write a test suite for the macro I recently modified, directly testing its correct use and checking for compiler errors with
illTyped, that made it easy to update without worrying about breaking something.
All of the code examples shown in this blog post can be found on github at https://github.com/TrudyFirestone/sample-macro-tests.