vim-UT

Introduction

UT is another Unit Testing Framework for Vim, which main particularity is to fill the quickfix window with the assertion failures.

Features

Main features

• Assertion failures are reported in the quickfix window
• Meant to be simple to use
  • Simple assertion syntax (:Assert expression, :AssertEquals(ref, expr)...)
  • Supports :Comments
  • Automatic registration of test functions
    • All the s:Test* functions of a suite are executed (almost) independently (i.e., a critical :Assert! failure will stop the Test of the function, and lh#UT will proceed to the next s:Test function
  • A suite == a file
  • Several s:TestXxx() per suite
• Callstack is decoded and expanded in the quickfix window on uncaught exceptions.

Other features

• Lightweight and simple to use: there is only one command defined, all the other definitions are kept in an autoload plugin.
• Fixtures:
  • optional s:Setup(), s:Teardown() executed before and after each test
  • optional s:BeforeAll(), s:AfterAll() executed once before and after all tests from a suite
• Supports banged :Assert! to stop processing a given test on failed assertions
• s:LocalFunctions(), s:variables, and l:variables are supported
• Buffer content can be set -- with :SetBufferContent
• Buffer content can be tested -- with :AssertBufferMatches
• Count successful tests and failed assertions
• Command to exclude, or specify the tests to play => :UTPlay, UTIgnore
• Add :debug before an assertion to debug its evaluation
• Short-cuts to run the Unit Tests associated to a given vim script; Relies on: Let-Modeline/local_vimrc/Project to set g:UTfiles (space separated list of glob-able paths), and on lh-vim-lib#path
• Helper scripts are provided to help integration with vimrunner+rspec. See examples of use in lh-vim-lib and lh-brackets.
• Takes advantage of BuildToolsWrapper's :COpen command if installed

Demo

Here is a little screencast to see how things are displayed when errors are detected.

Notes:

• the diff-mode is entered with D -- sorry, I wasn't able to display the keys typed in this screencast.
• the signs and balloons are automatically added by vim-compil-hints

Usage

Start-up

• Create a new vim script, it will be a Unit Testing Suite.

• One of the first lines must contain

  UTSuite Some intelligible name for the suite

• Then you are free to directly assert anything you wish as long as it is a valid vim expression, e.g.

  Assert 1 > 2
  Assert 1 > 0
  Assert s:foo > s:Bar(g:var + 28) / strlen("foobar")
  
  debug AssertTxt (s:foo > s:Bar(g:var+28)
        \, s:foo." isn't bigger than s:Bar(".g:var."+28)")
  AssertEquals!('a', 'a')
  AssertDiffers('a', 'a')
  let dict = {}
  AssertIs(dict, dict)
  AssertIsNot(dict, dict)
  AssertMatch('abc', 'a')
  AssertRelation(1, '<', 2)
  AssertThrows 0 + [0]

  tests/lh/README.vim|| SUITE <[lh#UT] Demonstrate assertions in README>
  tests/lh/README.vim|27 error| assertion failed: 1 > 2
  tests/lh/README.vim|31 error| assertion failed: s:foo > s:Bar(g:var + 28) / strlen("foobar")
  tests/lh/README.vim|33 error| assertion failed: -1 isn't bigger than s:Bar(5+28)
  tests/lh/README.vim|37 error| assertion failed: 'a' is not different from 'a'
  tests/lh/README.vim|40 error| assertion failed: {} is not identical to {}
  ...
  

• or to define as many independent tests as you wish. A test is a function with a name starting with s:Test. Even if a test critically fails, the next test will be executed, e.g.

  function s:Test1()
    let var = SomeFunction()
    Assert! type(var) == type(0)
    Assert var < 42
    Assert! var > 0
  
    " Some other code that won't be executed if the previous assertion failed
    " /*the wiki does not recognizes vim comments*/
    let i = var / 42.0
    Comment "This comment may never be displayed if {var} is negative or not a number"
  endfunction
  
  function s:Test2()
    Assert s:what != Ever()
  endfunction

• Now run :UTRun on your test script (filename), and ... debug your failed assertions.

Fixtures

Code can be executed before and after each test function with the optional special functions:

• s:Setup(): set-up function executed before each test
• s:Teardown(): clean-up function executed after each test
• s:BeforeAll(): set-up function execute once before all tests from a suite
• s:AfterAll(): clean-up function execute once after all tests from a suite

Test on buffers

Most :AssertXxx commands are dedicated to unit testing vim functions. A function returns a result and we test whether its value equals, differs, matches, is...

Since V2, it's now possible set the content of a buffer before transforming it, and to test whether the new buffer state is as expected.

Within the quickfix-window, we can hit D on the message associated to a buffer matching failure in order to display, in a new tabpage, the expected content alongside the expected content in diff-mode.

The diff-mode can be exited by hitting q from any scratch buffer involded.

silent! call lh#window#create_window_with('new') " work around possible E36
try
    SetBufferContent a/file/name.txt
    %s/.*/\U&/
    AssertBufferMatch a/file/NAME.txt
finally
    bw
endtry

Or, with :let=<< syntax

silent! call lh#window#create_window_with('new') " work around possible E36
try
    SetBufferContent << trim EOF
    1
    3
    2
    EOF

    %sort

    AssertBufferMatch << trim EOF
    1
    4
    3
    EOF
finally
    silent bw!
endtry

which results into

tests/lh/README.vim|78 error| assertion failed: Observed buffer does not match Expected reference:
|| ---
|| +++
|| @@ -1,3 +1,3 @@
||  1
|| -4
|| +2
||  3

Note: :SetBufferContent and :AssertBufferMatch with << [trim] EOF syntax can only be used within s:TestXxx functions.

Examples

See:

• tests/lh/UT.vim for a classical test,
• tests/lh/UT-fixtures.vim for a test with fixtures.
• tests/lh/UT-buf.vim for tests on buffer content.

To Do

• Check UT works fine under windows (where paths have spaces, etc.), and on UTF-8 files

• Simplify s:errors functions

• Support Embedded comments like for instance:

  Assert 1 == 1 " 1 must value 1

• Ways to test buffers produced

• Find a way to prevent the potential script scope pollution

• Add a summary at the end of the execution

Design Choices

• The assertions supported by this plugin are expected to be made in a Unit Testing file, they are not to be used in regular VimL scripts as a Design by Contract tool. Check my DbC framework in lh-vim-lib, or even Thomas Link's plugin, it is much more suited for that kind of assertions.

• In order to be able to produce the quickfix entries, the plugin first parses the Unit Test file to complete all :Assert occurrences with extra information about the line number where the assertion is made. Incidentally, this permits to reduce plugin footprint: none of :AssertXXX commands are actual commands.

Installation

• Requirements: Vim 7.+, lh-vim-lib v5.1.0

• With vim-addon-manager, install vim-UT (this is the preferred method because of the dependencies)

  ActivateAddons UT

• or with vim-flavor, which also handles dependencies

  flavor 'LucHermitte/vim-UT'
  

• or you can clone the git repositories

  git clone git@github.com:LucHermitte/lh-vim-lib.git
  git clone git@github.com:LucHermitte/vim-UT.git

• or with Vundle/NeoBundle:

  Bundle 'LucHermitte/lh-vim-lib'
  Bundle 'LucHermitte/vim-UT'

Other Tests related plugins for Vim

• Tom Link's tAssert plugin, and spec_vim plugin,
• Staale Flock's vimUnit plugin,
• Meikel Brandmeyer's vimTAP plugin,
• Ben Fritz's vim-2html-test plugin,
• Ingo Karkat's runVimTests plugin,
• See also Paul Mucur article's: Testing Vim Plugins on Travis CI with RSpec and Vimrunner,
• Andrew Radev's vimrunner,
• Kana's vim-spec
• Dhruva Sagar's vim-testify