Source   Edit  

testing

bookmark

TLDR

• idiomatic nim tests
  • tests dir should contain
    • optional config.nims

    # now importing custom modules in your test file will work
    switch("path", "$projectDir/../where/you/keep/your/source/code")
    --path:"$projectDir/../.." # IMO preferred

    • optional nim.cfg
    • optional file (any name, e.g. skip) with tests to skip (1 per line, # comments ignored)
    • subdirs for test categories, e.g. root/tests/mymodule/\*.nim
      • test files prefixed with t e.g. tsomefile.nim

gotchas

• testament
  • dont add leading tabs before discard """ stuff here """ when using output type directives
    • e.g. for formatting; remember """ raw strings keep tabs, thus wont match against output
• valgrind
  • Use --mm:orc -d:useMalloc with Valgrind in order to avoid false positives.

links

• other
  • nimscript test file
  • example testament tests
  • testament src
• high impact
  • status fuzz testing
  • testament: preferred testing tool
  • testament: unit test boilerplate
  • profiling and debugging
  • nimble test docs
  • valgrind dynamic analysis toolset
• niche
  • dr nim
  • unit tests (prefer testament)
  • Z3 proof engine

testament

• advanced automatic unittest runner with support for:
  • process isolation
  • test case stats & html reports
  • cross-compile support
  • dry runs & logging
  • etc
• you can apt install valgrind to check for mem leaks in test

testament html reports

• first run some tests to generate test data for the parser
• then create html reports with testament html

testament running tests

• you can target all, individual, categories or globs of test files

testament all
testament run somefile
testament pattern "tests/**/**/*.nim"

testament writing tests

• a single test consists of two sections
  • test specs (optional): as defined by this file
    • type TSpec and proc parseSpec determine fields and validation rules, respectively
      • focus on parseSpec as it massages spec keys into Tspec fields
  • test code (required): just normal nim code!
    • test lifecycle: parse spec -> validate spec > execute code against spec
    • the default is to fail a test if any error is thrown
      • however you can change this behavior, e.g.
        • "compile" only | "run" and compile | "reject" tests that dont throw expected errors
        • reject test if stdout fails sparsely match with expected stdout
  • examples that reflect specs & code
    • unittests examples
    • expected output with tests grouped in blocks tarray
    • expected errors inline with the code
    • expected errors based on exit code + output substitution
    • expected errors based on errormsg thrown on specific line in specific file
    • successful execution without a spec section
    • successful compilation
    • tracking regressions
    • async http servers
    • async file read/write
    • multitasking
    • targeting specific os/architectures
    • targeting specific backends, e.g. javascript
      • simpler javascript example
    • benchmarking

testament options

• --print results to console
• --simulate dry runs
• --failing log only
• --targets "c cpp js objc"
• --nim:/some/path
• --directory:/initial/dir
• --colors:on/off
• --backendLogging:on/off
• --skipFrom:/some/file # throws if file not found

testament env vars

• NIM_TESTAMENT_BATCH runs batchable tests together

testament specs

• FYI
  • if file|line|column is specified, either msg | errormsg | nimout (if line)
    • set to a non empty value
    • appear before file option
  • if exitcode is set forces action == run
  • if errormsg is set forces action == reject
  • can use config.nims/nim.cfg instead of cmd if only changing nimc flags
    • cmd is required for nimscripts; e.g. cmd: "nim e --hints:on -d:testing $options $file"
• action sets test validation to compile | (compile &) run | reject
• batchable can run in batch mode; i.e. all tests with the same dependency set can be batched
• column @see FYI section
• disabled if true | OS/ARCH matches this string; can be specified multiple times
  • win | linux | bsd | osx | unix
  • littlendian | bigendian |
  • cpu8/16/32/64
  • travis | appveyor | azure
  • any OS/CPU value defined in compiler/platform
• errormsg: expected in stdout
• exitcode: test should return
• ccodecheck can be passed multiple times
• cmd default nim $target --hints:on -d:testing --nimblePath:build/deps/pkgs $options $file
• file that will stderr/out a msg | errormsg | nimout; @see FYI section
• input: stdinput for test
• joinable: will be run with other tests
• line in file that will stderr/out the string set in error(msg) options; @see FYI section
• matrix of ; delimited switches each being a group of test scenarios
• maxcodesize test permitted to compile to
• nimout '''multi line output''' each line sparsely matched against COMPILER (not test!!) output
• nimoutFull nimout is the full output or a subset
• output test prints to stdout for validation <--- likely what you want for testing test output
• outputsub test prints that must be included in the full stdout
• sortoutput before validating against stdout
• targets default "c" accepts space separated backends
• timeout in seconds after which test considered failed
• valgrind is in path and should check for memory leaks

nimble tests

• nimble can be used for running tests via tasks
• you can override the standard test command in your $project.nimble file

task test, "Runs the test suite":
  exec "nim r tests/tester"

unittests

• the nim std library comes with its own unit test framework
  • unittests provides a familiar test interface
• however, testament is preferred so we'll be skipping std/unittest