Testing Macros in Scala

I recently finished a project which required me to modify an annotation macro for Scala using the Scala Macros Paradise project. Because I was extensively modifying a macro that we were already using, I wanted to write tests to prevent breaking our current workflow and to check that the new portion of the macro worked properly. Immediately, I ran into a problem: how could I test something whose proper behavior could be to throw a compiler error? After some research, however, I discovered there are quite a few ways to test both the successful and the error cases of macros. To compare them, I wrote a very simple macro that appends " Hello World" onto a String val.

@compileTimeOnly("enable macro paradise to expand macro annotations")
class Example extends StaticAnnotation {
  def macroTransform(annottees: Any*): Any = macro ExampleMacro.impl
}

object ExampleMacro {
  def impl(c: Context)(annottees: c.Expr[Any]*): c.Expr[Any] = {
    import c.universe._
    val result: List[Tree] = annottees.map(_.tree).toList match {
      case q"$mods val $tname: String = $expr" :: Nil =>
        val helloWorld = " Hello World"
        List(q"$mods val $tname: String = $expr + $helloWorld")
      case value @ q"$mods val $tname = $expr" :: Nil =>
        c.error(value.head.pos, s"$tname must have an explicit type")
        value
      case value @ q"$mods val $tname: $tpt = $expr" :: Nil =>
        c.error(value.head.pos, s"$tname must be of type String, not $tpt")
        value
      case tree =>
        c.error(tree.head.pos, "@Example can only be used with val of type String")
        tree
    }
    c.Expr[Any](Block(result, Literal(Constant(()))))
  }
}

When the above macro works correctly, it should expand this:

@Example val test: String = "A welcome"

to

val test: String = "A welcome" + " Hello World"

Of course, if the annotated code compiles, the simplest test is to just use it.

"Example macro" should {
  "append Hello World" in {
    @Example val x: String = "Welcome and"
    x must be equalTo "Welcome and Hello World"
  }
}

The above test shows that the Example macro appends " Hello World" to an annotated String val, but it offers no guarantees regarding correct compiler errors or the exact expansion.

ScalaTest provides should compile, shouldNot compile, and shouldNot typeCheck, which at least allows for some simple compilation tests.

"Example macro" should "not compile when applied to a def" in {
  """@Example def x: String = "a"""" shouldNot compile
  """@Example def x: String = "a"""" shouldNot typeCheck
}

it should "not compile when applied to a val that doesn't have an explicit type" in {
  """@Example val y = "Welcome and"""" shouldNot compile
  """@Example val y = "Welcome and"""" shouldNot typeCheck
}

it should "not compile when applied to a val of a non String type" in {
  """@Example val y: Int = 1""" shouldNot compile
  """@Example val y: Int = 1""" shouldNot typeCheck
}

it should "compile when applied to a String val" in {
  """@Example val x: String = "Welcome and"""" should compile
}

shouldNot typeCheck catches macro-caused compiler errors because the macros from Scala Macros Paradise are expanded after the parsing phase of the compiler and before type checking. Any macro errors are caught by the time the type checking phase is complete.

ScalaTest’s approach is good enough for simple compilation checks, but if there’s more than one way for macro compilation to fail, there’s no way to test that the correct compiler error was thrown. That’s where illTyped from shapeless comes in handy.

"Example macro" should {

  "Give a compiler error when applied to a def" in {
    illTyped(
      "@Example def x: String = \"Welcome and\"",
      "@Example can only be used with val of type String"
    )
    ok
  }

  "Give a compiler error when applied to a val that doesn't have an explicit type" in {
    illTyped(
      "@Example val y = \"Welcome and\"",
      "y must have an explicit type"
    )
    ok
  }

  "Give a compiler error when applied to a val of a non String type" in {
    illTyped(
      "@Example val y: Int = 1",
      "y must be of type String, not Int"
    )
    ok
  }

}

illTyped is implemented as a macro which checks at compile time that the given String does not type check and that the error given matches the expected error. Once again, the phase of the compiler that performs the macro expansion allows type checking to capture the macro-generated compiler error. Both ScalaTest’s approach and illTyped check the macro’s compilation when the test is compiled. However, ScalaTest automatically modifies the test result based on the compilation errors, while illTyped simply makes the test fail to compile if the check fails.

Unfortunately, if the given code generates more than one compiler error, illTyped only reports one. illTyped also doesn’t capture important details like the position of the compiler error. If the annotated value is an entire class, the position of the error can be quite important to the user of the macro. The Scala Macros Paradise project checks for multiple errors and the correct position of errors by compiling files and comparing error output against expected error output in one of its own tests.

After testing for correct results when the macro compiles and making sure that it throws the expected compiler errors, the only thing left to check is that it actually expands correctly. If you use the scalameta paradise version of macros, it’s easy to check that two syntax trees are structurally equal which allows you to easily make sure the expansion of your macro happens as expected.

There are different advantages to each type of macro test, and I was able to write a test suite for the macro I recently modified, directly testing its correct use and checking for compiler errors with illTyped, that made it easy to update without worrying about breaking something.

All of the code examples shown in this blog post can be found on github at https://github.com/TrudyFirestone/sample-macro-tests.