Recipe for using the “expectations” testing tool, based on CodeaUnit.
We’ll cover the basic concepts with a small but complete test example and discuss its main components. Then we’ll look at some detailed Mini Recipes. To skip to the specific examples, use these links:
local _ = require('./expectations')
function _:featureTestSuite()
_:describe("Recipe Test Suite", function()
_:test("Equality Test", function()
local f = "Foo"
local b = "Bar"
local result = f..b
_:expect(result).is("FooBar")
end)
_:test("Expected to Fail", function()
local f = "Foo"
local b = "Bar"
local result = b..f
_:expect(result).is("FooBar")
end)
_:test("Totaling", function()
local price = 100
local tax = 6
local total = price + tax
_:expect(total).is(106)
end)
end)
end
_:execute()
Scanning that file, we see that we require ‘expectations’, saving the return in the local variable _. Then we have one feature, which we describe, containing three tests, each with one expectation. We’ll examine those in detail just below.
When we run the above file, we get this output:
Feature: Recipe Test Suite
Actual: BarFoo, Expected: FooBar. Test: 'Expected to Fail'.
Recipe Test Suite, 3 Tests: Pass 2, Fail 1, Ignored 0.
The output lists the feature and its description, “Recipe Test Suite”. Then, for every test that fails, an error message is produced, showing information about the specific failure, and including the name of the failing test, in our case “Expected to Fail”.
All else is there for the other ten percent of times when checking a single value or table for equality isn’t quite the thing.
Don’t let all the details daunt you: it’s really quite easy and nice.
Now let’s step softly through the details:
local _ = require('./expectations')
Requiring the ‘expectations’ file returns an object containing all of Expectations’s code. Conventionally we assign the return to _ for convenience and neatness in writing our tests. You can assign it to anything you like, and use that as a prefix instead of underbar, if you wish.
On a system without require, such as SLua is likely to be, you may have to use a weaker feature, such as include or copy the file into your source manually. We’ll update this article when the facts are known, gods willing.
function _:featureTestSuite()
We typically test one feature of our code at a time. A feature might be an entire class, or one aspect of a more complex class. A feature can even be a single function. Sometimes I even test aspects of the SLua language or its libraries, just to ensure that I understand how things work. I keep the tests, quite often, as a reminder.
All actual testing code must be in a function, prefixed by _: and with a name beginning with “feature”. That is how Expectations finds our tests, in order to run them.
_:describe("Recipe Test Suite", function()
...
end)
We describe our feature as above. All our tests and explanations go inside the function that is the second parameter to describe. It may be best not to think about this and just to create these lines and fill them in.
What happens, if you want the details, is that Expectations finds the feature function and calls it. Inside it there are one or more calls to describe, typically one. The describe function is executed, and its job is to initialize all the counters and then safely execute what is inside the describe’s function, primarily tests:
_:test("Equality Test", function()
local f = "Foo"
local b = "Bar"
local result = f..b
_:expect(result).is("FooBar")
end)
We recommend writing many small tests, rather than a few large ones. Experiment to see what works best for you, but larger tests tend to be more fragile, requiring more updating, and they can get quite difficult to write. If all your small pieces are working, your larger pieces’ tests will likely be fewer and easier to write.
With these testing tools, we speak of a test as having three parts, arrange, act, and assert. In the test above the assignments to f and b are arrange, the computation of result is the act, the thing we want to test, in this case, our understanding of the .. operator.
And finally, we make an assertion, which we express with expect and some additional verbs:
_:expect(result).is("FooBar")
We enclose whatever we want to check in expect(), in this case, result. We could have said:
_:expect(f..b).is("FooBar")
but it is often better to compute the test result as a local and then refer to it.
Finally we come to the actual check, in this case is:
_:expect(result).is("FooBar")
That does what it says on the box. It checks whether the thing in expect is in fact the string “FooBar”. As we’ll see below, there are a few different checks:
| word | meaning |
|---|---|
| _:expect(result).is(expected) | result == expected |
| _:expect(result).isnt(expected) | result ~= expected |
| _:expect(aTable).has(element) | aTable includes element |
| _:expect(aFunction()).throws(message) | aFunction call errors with message |
We’ll show examples of each of those below.
require in the Expectations object and assign it to _._:feature... methods._:describe, providing a feature description string and a function defining the tests._:test, providing a test name, and a function to be performed, typically including arrange, act, and assert (_:expect)._:expect is followed by a verb expressing the aspect of the result to be checked, equality (is), inequality (isnt), containment (has), or exception (throws)This may seem like a lot, but in fact it’s mostly just boilerplate, and what we get for our effort is some very comprehensive checking and reporting. Here is just part of the output from my current test suite for Expectations itself:
Feature: Expect
Actual: 55, Expected: 54. [expected failure] Test: 'expect_is_fail'.
Actual: 555, Expected: 54 +/- 1. Test: 'not close_enough'.
Actual: table[d]=nil, Expected: 4. Test: 'table_missing_element'.
Actual: table[b]=22, Expected: 2. Test: 'table_wrong_element'.
Ignored test: 'too difficult'
Actual: table[qqq]=42, Unexpected key [qqq]. Test: 'table_extra_element'.
Actual: 100 was unexpected. [better not be 100] Test: 'isnt'.
Actual: 97 should not be within 4 of 100 [stay away from 100] Test: 'isnt_epsilon'.
Actual: 97 should not be within 4 of 100 Test: 'isnt_epsilon_quiet'.
Table: table: 0x000000012b0166b0 did not contain 9. [checking table] Test: 'has fail'.
Expected throw: attempt to call a nil value. Test: 'throw should fail'.
Error: /Users/jr/Desktop/repo/test-suite.lua:140: attempt to call a nil value. Test: 'throws unexpectedly'.
Expect, 16 Tests: Pass 4, Fail 11, Ignored 1.
The errors above are all intentional, because I need to see that the right info comes out on a failure. When Expectations goes into the Downloads, I’ll make sure that you can get the current test suite for it as well.
I think you’ll find that you use the is verb the great bulk of the time, because we typically know the exact answer that we want. But here are some specials:
Sometimes, with real numbers, we don’t know the precise answer and just want to make sure that our result is close:
_:test("Approximate value", function()
local sqrt2 = math.sqrt(2)
_:expect(sqrt2).is(1.414, 0.00001)
end)
We can provide a second parameter to the is verb, expressing the allowable error in a real number approximation. By the way, the test above fails, saying:
Actual: 1.4142135623730951, Expected: 1.414 +/- 0.00001.
Test: 'Approximate value'.
Of course, if we had said 0.001 instead of 0.00001, or provided more digits, the test would have passed.
To determine whether a result string, we can check to see whether it contains any desired substring, including, of course, the whole string we expect:
_:test("String has value", function()
_:expect('vorpal blade').has('vorpal blade')
end)
Similarly sometimes we want to know whether a specific value has appeared in a table. We use has for that question as well:
_:test("Table has value", function()
local tab = {1,3,4,5}
_:expect(tab).has(2)
end)
There is no 2 in the table and we get:
Table: table: 0x000000015a0177c0 did not contain 2.
Test: 'Table has value'.
The unfortunate table id is down to SLua. Tables do not have a standard way of naming them. Fortunately, the test name will let us isolate which table has caused the problem … and that’s why we typically only use one assertion per test. But we can do otherwise:
In this test, for some reason, we check two tables:
_:test("Named expectations", function()
local t1 = {1, 2, 3, 4}
local t2 = {1, 20, 3, 4}
_:expect(t1).has(2)
_:expect(t2).has(2)
end)
The output is a bit confusing, because we can’t tell which table is the offender:
Table: table: 0x00000001258176d0 did not contain 2.
Test: 'Named expectations'.
We get the test name, but that’s all. To identify particular assertions, we can give each assertion its own name, optionally:
_:test("Named expectations", function()
local t1 = {1, 2, 3, 4}
local t2 = {1, 20, 3, 4}
_:expect(t1, "table t1").has(2)
_:expect(t2, "table t2").has(2)
end)
Now we get this output:
Table: table: 0x000000012b0176d0 did not contain 2.
[table t2] Test: 'Named expectations'.
The optional second parameter of expect is a string which will be displayed in square brackets if the test fails.
When we find ourselves doing this, it can be a sign that our tests are too large, but you get to find your own balance. Personally, I try to avoid needing named expectations.
We can compare tables for equality:
_:test("Table Equality", function()
local v_result = {x=10, y=20, z=3}
local v_expect = {x=10, y=20, z=30}
_:expect(v_result).is(v_expect)
end)
Actual: table[z]=3, Expected: 30. Test: 'Table Equality'.
The output will list each unmatched element, missing elements, or extra elements. Here is a more complex example:
_:test("Table Equality", function()
local v_result = {x=10, y=21, Z=3}
local v_expect = {x=10, y=20, z=30}
_:expect(v_result).is(v_expect)
end)
The output:
Actual: table[y]=21, Expected: 20. Test: 'Table Equality'.
Actual: table[z]=nil, Expected: 30. Test: 'Table Equality'.
Actual: table[Z]=3, Unexpected key [Z]. Test: 'Table Equality'.
This one is a bit tricky. We can indicate that a function call should throw an error:
_:test("Exception", function()
local function might_throw()
error("be more positive")
end
_:expect(might_throw).throws('more pos')
end)
Note that we name the function in the expect, we do not call it. The throws verb will call it and check to be sure the parameter string occurs in the error message.
There is no way to provide the parameters to the function in the expect, at least at this writing. We can do that by providing a function that calls our function that should throw, like this:
_:test("Exception with parameters", function()
local function throw_on_negative(parm)
if parm > 0 then
error("parm was negative")
end
end
local function should_throw()
throw_on_negative(-3)
end
_:expect(should_throw).throws('was neg')
end)
Since the check for negative is wrong in throw_on_negative, this test should fail. And it does:
Expected throw: was neg. Test: 'Exception with parameters'.
Sometimes we might want something that Expectations does not offer. One example is checking vectors in some code I’ve been working on. Expectations can check individual values for being approximately equal, but cannot check a whole vector for being close enough. Here’s a nearly sensible test for vector scaling:
_:test("vector scaling", function()
v1 = vector(100.0, 200.0, 300.0)
vs = v1 * 1.1
_:expect(vs).is(vector(110.0, 220.0, 330.0))
end)
This test fails, like this:
Actual: table[y]=220.00000000000003, Expected: 220.
Test: 'vector scaling'.
Actual: table[x]=110.00000000000001, Expected: 110.
Test: 'vector scaling'.
We need to accommodate the rounding. We could write out three checks every time we do this but we’ll probably need to check vectors for being close enough more than this once, so we write this:
function verify_vectors(check, original)
_:expect(check.x).is( original.x, 0.001)
_:expect(check.y).is( original.y, 0.001)
_:expect(check.z).is( original.z, 0.001)
end
And say this:
_:test("vector scaling", function()
v1 = vector(100.0, 200.0, 300.0)
vs = v1 * 1.1
vexpect = vector(110.0, 220.0, 330.0)
verify_vectors(vs, vexpect)
end)
And, of course, this test passes as expected. For greater flexibility we could pass in the allowable delta, but the test above was all I needed in my current code.
So a common thing to do, when we have something special to check, and need to do it more than once, is to write a helper function that calls the basic expect checkers.
Often, something in our script will just break. Expectations will give us an error message on a test that throws an unexpected error and then continue with other tests. For example:
_:test("Do something useful", function()
local s = no_such_function()
_:expect(s).is(42)
end)
That test calls a function that doesn’t exist, due to a typo or other oversight. We get this result:
Error: /Users/jr/Desktop/repo/expect_recipe.lua:71:
attempt to call a nil value.
Test: 'Do something useful'.
Often a series of tests all want to use some of the same objects for testing. In my Bezier-related tests, I use the same sample Bezier curves over and over. Expectations allows us to set up some variables before each test executes, with _:setup(). Let’s change our first two tests to use setup just as a trivial example. We start with this:
_:test("Equality Test", function()
local f = "Foo"
local b = "Bar"
local result = f..b
_:expect(result).is("FooBar")
end)
_:test("Expected to Fail", function()
local f = "Foo"
local b = "Bar"
local result = b..f
_:expect(result).is("FooBar")
end)
Imagine that instead of just f and b we had a lot of work to do to get those values. We do this:
local f
local b
_:setup(function()
f = "Foo"
b = "Bar"
end)
_:test("Equality Test", function()
local result = f..b
_:expect(result).is("FooBar")
end)
_:test("Expected to Fail", function()
local result = b..f
_:expect(result).is("FooBar")
end)
Our tests run the same but now we have saved all that extra typing of parts of the setup. Note that the variables need to be defined inside describe, and outside the setup function, so that both setup and our tests can see them. And a wise programmer would probably not call them f and b but something more distinctive. A quick rename refactoring gives us this:
local foo_string
local bar_string
_:setup(function()
foo_string = "Foo"
bar_string = "Bar"
end)
_:test("Equality Test", function()
local result = foo_string..bar_string
_:expect(result).is("FooBar")
end)
_:test("Expected to Fail", function()
local result = bar_string..foo_string
_:expect(result).is("FooBar")
end)
Rarely—in my experience almost never—we might want to tear down what we’ve built up with setup and a test. An example might be a series of tests that build up and use a very large table. Since the local variables in the feature site will persist over all the tests, we might want to release the storage after each test runs. Simple enough with teardown:
_:teardown(function()
foo_string = nil
var_string = nil
end)
Each _:describe can have its own setup and teardown. There can be more than one describe in a single feature function, although one per feature is more common.
Sometimes we really don’t want to run a test for a while for some reason. For me this happens in at least these situations:
Instead of _:test, we can say _:ignore, like this:
_:ignore("time the search function", function()
local t_start = os.clock()
for i = 1,1_000_000 do
search('value_not_present')
end
local t_end = os.clock()
_:expect(t_end-t_start).is(30, 5)
end)
We could, of course just delete the test, or comment it out. But when we ignore it, we get this result:
Ignored test: 'time the search function'
Recipe Test Suite, 10 Tests: Pass 2, Fail 7, Ignored 1.
Each ignored test is listed in the test results and counted in the final summary. Using ignore lets me tuck a test away without forgetting about it, You might find that useful as well.
Currently I develop most of my SLua code using a luau compiler installed on my Mac, editing with Sublime Text 4. That luau supports require, so I require in the framework and begin writing tests. Currently—and this might change—I keep the tests and the classes or functions I’m creating in the same file, because Sublime Text isn’t really good with lots of tabs. When Second Life finally releases SLua to the main grid, my process will change and if it’s interesting, I’ll write it up.
If I didn’t have a reasonably compatible luau on my Mac, and was developing in the Second Life viewer, I’d still use Sublime as my external script editor, and would probably run the tests and code in SL. We don’t yet know about the memory constraints with SLua, but surely in production mode we’ll want to keep the test code separate from the working code. The process for doing that remains to be figured out after we know what SL is going to give us.
I believe that it is also possible to develop luau using some Roblox code, but at this writing I do not know how to do that. When I learn, I’ll document it here.
Working with so many tests did not come to me as second nature, but over time I’ve found that my best work happens when I create a small test for the next bit of feature that I want, see the test fail, then make it work, in very small steps. It feels slow but it is fast.
“Slow is smooth, smooth is fast” may have originated with the military, but I find that it is true in my work. When I’m at my best, I write a small test, then a small bit of code to make it work, repeat. The tests make sure that everything keeps working as I change things. I progress smoothly in small steps with few reversions and very little debugging.
You might be concerned that the tests themselves might be slow. If one’s tests are slow, one needs better tests. My current real project using this tool runs about 900 tests in 75 milliseconds, on my Mac. Test time is not an obstacle, and so I run them whenever I want to, which allows me to make very small changes, one at a time, very quickly.
I hope that you’ll give Expectations a try and find it of value. I’d love to hear from you either way.
Safe paths!