A couple years ago I wrote an experimental strongly statically typed DSL for testing a web-app in Kotlin. It was actually composed of two parts: the DSL generator, and the generated DSL itself. It didn’t survive as the solution we were looking for, but it was an informative experiment, and I think there’s potential in the approach I took with it.

At the time, I was one a few engineers that investigated frameworks for writing automated end-to-end tests for web apps. The target we wanted to test had the bulk of its UI written for Angular. Our high level requirements were:

  • Plays nicely with Angular, SPAs and page transitions (ie. our app)
  • Interface with javascript well (for flexibility; what we are testing has native javascript support after all)
  • Allow non-programmers to write tests (for manual tests)
  • Encourages stable tests (not brittle)

One of the things we considered when looking at frameworks was looking at how selectors were handled. We had two big concerns:

  1. Some items on the page didn’t have a good unique path (e.g. lots of indices into lists needed, nameless div soup, etc.)
  2. UI changes can break selectors

What do I mean by break selectors? Well, a selector of #my-form .btn-primary will work perfectly fine until someone adds another button of with the class of btn-primary somewhere under that form, perhaps unwittingly in a nested component. Even seeming harmless structural page changes break tests - but you don’t find out until you run all the tests.

Could I make a DSL?

My desire to squash #2 drove this Kotlin experiment. What if we could parse the angular templates, and with some extra logic for the more dynamic parts, generate all the selectors we would need? Assigning names based off various template information (like button text, tag name, id, aria info or other adjacent tags) would make it more human friendly. Using a statically typed DSL for it would make it even more human friendly (auto-complete in IDEs) and it could make them compile time checks. That drastically shortens the loop for figuring out when UI changes can break tests due to selector changes.

My master plan was basically this: the process used to build the UI also builds the e2e testing DSL. Then if the tests compile we know that the UI changes didn’t break them from a structural standpoint.

Kotlin is Good for DSLs

As I was on a Kotlin kick, I wanted to investigate using to build such a framework. I knew it could compile down to javascript, so it could really just be a facade over the existing testing frameworks we investigated. And I knew that it had pretty good features for writing DSLs. So an investigation seemed warranted.

I tried to play to Kotlin’s strengths for building DSLs (see their docs on this topic to make the following snippet less confusing). To sum up those benefits (probably poorly):

  • You can define function that when called look like named code blocks (ie. keyword { ... })
  • That syntax passes the block { ... } as a closure to the function
  • Those function calls can make other function calls to be available only within the block it consumes
  • This is all strongly & statically typed

The doc pages uses this HTML building DSL as an example of the power of that pattern:

import com.example.html.* // see declarations below

fun result() =
    html {
        head {
            title {+"XML encoding with Kotlin"}
        }
        body {
            h1 {+"XML encoding with Kotlin"}
            p  {+"this format can be used as an alternative markup to XML"}

            // an element with attributes and text content
            a(href = "http://kotlinlang.org") {+"Kotlin"}

            // mixed content
            p {
                +"This is some"
                b {+"mixed"}
                +"text. For more see the"
                a(href = "http://kotlinlang.org") {+"Kotlin"}
                +"project"
            }
            p {+"some text"}

            // content generated by
            p {
                for (arg in args)
                    +arg
            }
        }
    }

In said example, each function that takes a block (e.g. html, head, body, h1, p, a, b) is really composed of two things: the function itself and the type that provides method definitions available to be called inside the block (e.g you can call p from within a body block but not a head block).

I made a Kotlin DSL

Armed with that knowledge, I set out to write a basic template parser (which was easy because it was AngularJS, so I could use an HTML parser to generate objects that I could easily inspect). With some automated DSL generation logic, and some extension points to add in extra manual logic, I was able to generate DSL classes that allowed for stuff like this:

import org.example.test.testingFeature;
import org.example.test.test;
import org.example.test.Sections;

import org.example.extensions.*;

testingFeature("Recommendations");

test("Recommend a favorited item", Sections.REGULAR_USER) { 
    login("username", "password")
    onPage(HOME){
        click(nav.favorites)
    }
    onPage(FAVORITES){
        targeting(item(0)) {
            fill(email, "bobTester@example.com")
            fill(message, "Hey Bob, buy this for your kid.")
            click(sendRecommendation)
            waitFor(successMessage)
        }
    }
}

How my DSL Works

There were three main types of standard classes that get extended by generated code:

  1. The Structure class - it let’s you declare where on the site you’re performing checks
  2. The Page class - it let’s you test conditions on the page, or scope certain checks against children elements
  3. The Component class - it lets you test conditions on that component, or scope certain checks against children elements.

Each of these three types follow a similar pattern: 1. They can have custom extension code 2. They have methods for checking items handed to them (pages, components, HTML things…) 3. Their subclasses have methods/fields describing their structure, which you pass to the their check/action methods

In addition to those three main types that get extended, there’s a handful of types that the HTML bits that make up components (and pages without components) - buttons, links, fields, etc. The unifying aspect is that they all know what their selector is (and components know theirs as well).

So here’s my commentary on the previous code:

import org.example.test.testingFeature; 
import org.example.test.test;
import org.example.test.Sections;

// Import our custom extensions
// Structure.login is one such extension point
import org.example.extensions.*;

// Where are we testing? This is more or less equivalent to a class name - it names a group of tests
testingFeature("Recommendations");

// `RegularUser` is a subclass of `Structure`, `Sections.REGULAR_USER` is the factory for it.
// The `test` method runs the factory and runs the block against the created instance
// ie the implicit `this` inside the block here (some 20 lines) is an instance of `RegularUser` 
test("Recommend a favorited item", Sections.REGULAR_USER) { 
    // Login custom code inserted through an extension point for the `Structure` class
    // It is statically resolved
    login("username", "password")
    
    // `onPage` is super-class method (belonging to the `Structure` class, not `RegularUser`)
    // It constructs the page from the factory given, and waits for the url to show up before advancing
    // `HOME` constructs an instance of `Home`
    onPage(HOME){
    
        // `click` is a check/action method of the `Page` class.
        // `nav` is a field on the `Home` class - it's class extends the `Component` class
        // `nav.favorites` is a field on the `Nav` component class, it's a link
        click(nav.favorites)
    }
    
    // `FAVORITES` constructs an instance of `Favorite`, a subclass of `Page`
    onPage(FAVORITES){
    
        // `targeting` is a super-class method (declared on `Page`, not `Favorite`)
        // `items(index: int): RecommendedItem` is a method that generates a `RecommendedItem` instance
        // `RecommendedItem` is a subclass of `Component`
        // All calls in the block passed to targeting are scoped to the components
        targeting(items(0)) {
        
            // `fill` is a method on `Component`
            // `email` and `message` are fields of `RecommendedItem`
            fill(email, "bobTester@example.com")
            fill(message, "Hey Bob, buy this for your kid.")
            
            // `click` is also a method of `Component`
            // `sendRecommendation` is a field on `RecommendedItem`
            click(sendRecommendation)
            
            // `waitFor` is also a method of `Component`
            // `successMessage` is a field on `RecommendedItem`
            waitFor(sendSuccessful)
        }
    }
}

Here’s what some of the base code could look like - the structure, page and component base classes, along with the necessary glue code to the js framework (CodeceptJS) that we’re targeting.

package org.example.test.base;

// external definitions of CodeceptJS functions
external fun Feature(featureName: String)
external fun Scenario(testName: String, test: (CodeceptHelper) -> Unit)
external class CodeceptHelper {
    // obviously not all here
    fun click(selector: String)
    fun waitInUrl(selector: String, sec: int)
    fun fillText(selector: String, value: String)
    fun waitForElement(selector: String, sec: int)
}

/** shim for Feature **/
fun testingFeature(feature: String){
    Feature(feature)
}

/** shim for Scenario **/
fun <T : org.example.role.Role> runTest(name: String, factory: Factory<T>, test: T.() -> Unit){
    Scenario(name) { I -> 
        val role = factory.create(I)
        role.test()
    }
}

interface Factory<T: BrowserWrapper> {
    fun create(helper: CodeceptHelper): T
}

interface PageFactory<P: PageBase> {
    fun create(): Page<P>
}

interface Url {
    fun getUrl(): String
}

open class Page<P: PageBase>(val page: P, val url: String): Url {
    override fun getUrl(): String {
        return this.url
    }
}


interface Selectable {
    fun selector() : String
}
interface LinkLike : Selectable {}
interface ButtonLike: Selectable {}
interface FieldLike: Selectable {}
interface TextLike: Selectable {}

open class Link(val selectorPath: String, val parent: Selectable?) : ChainedSelector(selectorPath, parent), LinkLike
open class Button(val selectorPath: String, val parent: Selectable?) : ChainedSelector(selectorPath, parent), ButtonLike
open class Field(val selectorPath: String, val parent: Selectable?) : ChainedSelector(selectorPath, parent), FieldLike
open class Text(val selectorPath: String, val parent: Selectable?) : ChainedSelector(selectorPath, parent), TextLike


open class ChainedSelector(val selectorPath: String, val parent: Selectable?) : Selectable {
    override fun selector() {
        return (parent?.selector() ?: "") + selectorPath
    }
}


@DslMarker       // Used to prevent mixing calls from different nested closures
annotation class StatefulWrapper

@StatefulWrapper // Helper base closures when nested can't accidentally call methods from the outer one
open class HelperBase(val helper: CodeceptHelper) {}

open class ComponentLikeBase(helper: CodeceptHelper) : HelperBase(helper) {
    fun click(target: Button) {
        helper.click(target.selector())
    }
    fun fill(target: Field, text: String) {
        helper.fillField(target.selector())
    }
    fun waitFor(target: Selectable){
        helper.waitForElement(target.selector(), 5)
    }
}

/** Subclasses should fill selectorPath with a constant in their call to super**/
open class Component (selectorPath: String, helper: CodeceptHelper, parent: Selectable?) : ComponentLikeBase(helper), Selectable{
    override fun selector() {
        return (parent?.selector() ?: "") + selectorPath
    }
}

/** Subclasses should generate a factory class/singleton **/
open class PageComponent (helper: CodeceptHelper) : ComponentLikeBase(helper){
    abstract fun url();
}

/** Subclasses should generate a factory class/singleton **/
open class Structure(helper: CodeceptHelper) : HelperBase(helper) {
    fun <T: PageComponent> onPage(factory: PageFactory<T>, onPage: T.() -> Unit){
        val page = factory.create()
        page.onPage()
    }
}

Generating the DSL

So I gave an example of the base code for the DSL, and what it would look like with generated classes, but how does one actually generate the DSL classes? And how do you can you add extension points?

Extension Points

It’s actually rather easy to implement extension points with Kotlin. In Kotlin you can define “functions with receivers”. What does that mean? It’s best explained with an example:

package com.example.extensions;

// this is a function with receiver
fun String.hasContent() : boolean {
    return this.trim().length() > 0
}

// this is not (but uses the above defintion
fun stringHasContent(value: String) : boolean {
    return value.hasContent();
}

then elsewhere you can easily import it and use it:

import com.example.extensions.hasContent;

val value = "abc";
val usable = value.hasContent()

So then adding extension points is very easy - just import all your extension code in a package you can glob import into your tests. That’s what the line import org.example.extensions.* in our DSL example does. And it’s an easy solution because you can keep that code in a separate source root and in a separate package which makes managing manual source code vs generated source code dead simple.

Components

Components are pretty simple as well - you parse a component template and you spit out a definition. Ideally you have some way of distinguishing component from pages as you parse, but it’s not strictly necessary (e.g myWidget.component.{html,js} instead of myWidget.{html,js}) The hardest part about this is what tags from your component template get included and how to include them. Here are the notes on my basic approach:

  • What tags gets included
    • Leaf tags (no children) - get turned into fields
    • Repeating tag - get turned into methods accepting an index
      • If Generate a unique class per repeating series to contain the inner structure if it has children tags

  • Naming the methods / fields - rules starting with highest precedence

    • Use the ID of the tag
    • Use the name of the tag
    • Use the text of a button
    • Use the aria-role of a button
    • If it has a label, use the label text
    • Use the simple dynamic source variable ({{someValue}} is simple, {{someValue * 3}} is not)

  • Generating selectors

    • Use direct descendants (always, to avoid accidentally matching children of a nested children components)
    • Keep track of all selectors used during code-gen and emit an error if you can’t make something unique

  • Handling errors - let the compiler handle it:

    • It’s okay to duplicate names
    • Buttons and fields need names. If no name is generated, generate an invalid identifier

So lets walk through an example of what might get generated. First the example HTML:

// my-favorite-item.component.html
<my-favorite-item>
    <h4>{{itemName}}</h1>
    <form>
        Share it with a friend
        <input type="text" ng-model="email" name="email" />
        <label for="email">Friend's Email</label>
        <input type="textarea" ng-model="message" name="message" />
        <label for="email">Message</label>
        <button type="submit">Send Message</button>
        <div ng-if="{{sendSuccessful}}">Success</div>
    </form>
</my-favorite-item>

And now the generated code:

package org.example.test.gen.components;

import org.example.test.base.*;

class MyFavoritItemComponent(selectorPath: String, parent: Selectable?, helper: CodeceptHelper) : Component(selectorPath, helper, parent), Selectable {
    val itemName = Text(">h4", this)
    val email = Field(">form>input[name='email']", this);
    val message = Field(">form>input[name='message']")
    val send = Button(">form>button", this);
    val successMessage = Text(">form>div", this);
}

Not that bad!

Pages

Pages are like components, but usually don’t have parents. If your pages are actually components being displayed wrapper of sorts, that get trickier, but can usually be addressed at the structure level not page level. With that, let’s pretend there is no common nav wrapper, but that we instead include it in each template.

So here’s an example, where somehow we learned that this page is paginated

// display-favorites.page.html
<display-favorites>
    <regular-nav id="nav" selected="favorites"></regular-nav>
    <h1>Favorited Items</h1>
    <my-favorite-item ng-repeat="item in items"></my-favorite-item>
    <div id="pagination">
        <span name="summaryText">Showing items {{items[0].index}} to {{items[items.length - 1].index}}</span>
        <button onClick="...">Previous</button>
        <a ng-repeat="pageNum in visiblePageNums" href="" onClick="...">{{pageNum}}</a>
        <button onClick="...">Next</button>
    </div>
    <common-footer></common-footer>
</display-favorites>

And the generated code:

package org.example.test.gen.pages;

import org.example.test.base.*;
import org.example.test.gen.components.RegularNavComponent;
import org.exmaple.test.gen.components.MyFavoritItemComponent;

class DisplayFavoritesPage(helper: CodeceptHelper): PageComponent("display-favorites", helper) {
    val nav = RegularNav(">#nav", this, helper);
    fun items(index: Int) : MyFavoritItemComponent {
        //Kotlin has string interpolation
        return MyFavoritItemComponent(">my-favorite-item:nth-of-type(${index+1})", this, helper) 
    }
    val summaryText = Text(">#pagination>span", this, helper);
    val previous = Button(">#pagination>button:nth-of-type(1)");
    fun visiblePageNums(index: Int) : Link {
        return Link(">#pagination>a:nth-of-type(${index+1}", this)
    }
    val next = Button(">#pagination>button:nth-of-type(2)")
}

Structures

Structures are the hardest. They contain the navigation information! You have to somehow extract that programatically to generate the structure pages! Your best bet is to load the all the js code post compilation in a headless browser and then inject a script that dumps the routing information. You will have an easier time if your routes are defined as json and then included in your frontend with only a small js loader, but that might not be possible.

Ideally, you’ll be able to generate some sort of state tree. You’ll need 3 bits of information * Parameters for each route, and for each parameter * parameter name * parameter type * The template of the route * The name of the page

Then you can define the routes piece by piece - one class for branch in the nav tree. Here’s an example of a couple bits of a full generated nav tree. And the route at the top is the one that extends the base structure class. You can have as many “structures” as you have navigation modes (e.g. perhaps one mode for regular users, and another for admins?);

import org.example.test.base.*;
import org.example.test.gen.pages.*;

// The favorites page is paginated, but has a default page (the first page)
// So implement the Getter interface to represent the default
class RegularUserFavorites(parent: RegularUser) : PageFactory<DisplayFavoritesPage> {
    override fun create(): Page<DisplayFavoritesPage> {
        return getPaginated(0)
    }
    
    fun getPaginated(pageIndex: Int) : Page<DisplayFavoritesPage> {
        return Page(DisplayFavoritesPage(this.parent.helper), "/favorites/${pageIndex+1}")
    }
}

class RegularUser(: Structure {
    // A static url needs only the constructor and reference to the path in the tree if it has no children
    // StaticPageFactory implement
    val HOME = StaticGetter(::HomePage, "/home", this) // factory being passed the constructor for home & this 1, this
    
    // Any children page that have their own children need special classes
    val FAVORITES = RegularUserFavorites(this)
    
    // ... more elided
}

And there you have it. That’s all for the basics of generating a DSL.

Drawbacks

There are of course a couple of huge drawbacks:

  • Kotlin is a non-standard layer of javascript. A lot of js package interop is going to have to be done by hand.
  • This example is only usable for toy projects - it’ll require much more work to account for the various ideosynchronsies of your particular project
  • This doesn’t generalize well - there’s going to be so much that is unique to your project that would make it very hard to build up a common library base for building these abstractions
  • It’s probably another language in your codebase

With all that said, I hope someone takes this approach someday. It would provide a very strong communication channel about potential breakage caused by UI code changes. You wouldn’t need to worry about selectors breaking - if they did break, you’d get a compile time error, not a test runtime error.

This doesn’t account for many sources of runtime errors (brittle checks, not actually waiting for events correctly, slow servers, etc.) But I have other ideas to tackle those.