- About
- "runtest" shell script
- .tst file runtest command
- .tst file directives
- Predefined variables
- Creating test files that can run on z/CMS as well as under Hercules
This directory contains test cases that are run by Hercules console commands. High-level test cases are run under an operating system, for example OS/j.
Test cases are run after Hyperion has been built, but before it is installed. "make check" runs the entire regression test reference.
With current Hyperion as of 2016-06-06, there are no longer any special considerations for testing crypto.
A test case is run stand-alone under hercules from a script specified with the -r flag. The script first sets up a program in core storage either by the r console command or by loading a core image by loadcore. It then issues the runtest command to start Hercules by a restart interrupt, the restart new PSW having been set up in the previous step. The script regains control when all CPUs have entered the stopped state. The remainder of the script inspects storage, registers, etc., to display data that are later inspected. The session log is then processed to check the results against known good values.
You can create test cases in several ways:
- Entirely by hand using only a text editor. Many test cases have been built this way, but this usage has been superseded by use of an assembler that can generate core images directly.
- By using Harold Grovesteen's excellent SATK's ASMA ("A Small Mainframe Assembler"), which is an assembler that implements a subset of HLASM's functionality, but has the ability to generate a core image directly. Thus no post processing is required. Typically, you will write by hand a
.tstfile to load the.corefile and do whatever is required. - By a real /360 assembler, such as
IFOXorHLASM. The object deck is converted torconsole commands or a core file. This offers the full assembly language with a powerful macro capability, but at the expense of some post processing.
Validating the test case (specifying the desired results; the oracle in testing parlance) is the most difficult step if you do not have access to z/CMS. There may be ambiguities in the Principles Of Operation, or perhaps more likely, you may misinterpret the manual. As IBM does not publish the suite the developers use to validate a new CPU model, you will need to find a friend who has access to "real iron". This real iron could be z/PDT.
There are two routes to obtaining an object module and validation script for z/CMS: From an ASSEMBLE file using an assembler or from a .tst file by the cvttst.rexx utility (very much still in the development phase).
By convention, the following file type extensions are used:
.tst
Files that have a tst extension are to be run by the "runtest" or "runtest.cmd" shell script. They are run and the Hercules console output is then inspected for correctness by an automated rexx script. You can create .tst files either "by hand", with the ASMA (SATK); or by assembly on a /360 system (VM, MVS, DOS), but then TEXT decks need to be processed into .tst files.
.subtst
Files that have a subtst extension are secondary helper scripts called by a primary tst script. They are used when more than one primary tst script consists of code that is virtually identical to another. Rather than duplicate the same code in multiple tst files, the common code is placed into a subtst file instead and "called" by each of the primary tst scripts (via the 'script' command). Think of them as tst script "subroutines".
.assemble
Source files for test cases that are designed for assembly with HLASM. On z/VM with HLASM, the test cases can be prepared by BLDHTC EXEC (bldhtc.exec) on VM or by bldhtc.rexx on the workstation. ASMTST XEDIT (asmtst.xedit) is a sample macro to perform the build step from within the XEDIT session.
On VM/370, you can use the system assembler to assemble, but you will likely do the conversion on the workstation.
http://vm.marist.edu/~pipeline/zops.copy contains a stacked copy file for many new operation codes.
.listing
An assembler listing file produced by HLASM.
.asm
Source files for the .tst test cases that are designed for assembly by
ASMA (Harold Grovesteen's SATK).
To clone: git@github.com:s390guy/SATK.git.
When instructed accordingly, the assembler will produce a core image file
(.core) that can be loaded with the loadcore console command.
This makefile snippet can automate the process. You must modify it with the location of the SATK and Hyperion build directories in your system:
.SUFFIXES: .text .tst .assemble .rexx .asm .core
USD:=/usr/data/src
APATH:=${USD}/SATK/tools
%.core: %.asm
ASMPATH=. ${APATH}/asma.py -l $*.list -i $@ $<
While ASMA so obviously is superior to hand assembly, it does have a few limitations relative to HLASM that you should be aware of:
-
Always check the documentation of the assembler instructions you intend to use to be sure they are indeed implemented. Notable omissions are `AMODE` and `RMODE`. -
No floating point constants. `D'...'` assembles as if `FD'...'` were specified, which is correct only for `D'0'`. Use hexadecimal constants to assemble floating point constants. The file `dc-float.asm` contains, in no particular order, a number of constants suitable for cut-and-paste. This file was generated from the HLASM listing file from an assembly of float constants. -
Input is variable length ASCII. Continuation is specified by a backslash at the end of a line. The alternate format for macro instructions is not supported.
.list
An assembler listing file produced by ASMA.
.txt
Files that have a .txt extension are manually executed test scripts
invoked via the Hercules script command. Their output must be
manually eyeballed to ensure they executed correctly. Usually they
end by loading a disabled wait PSW. If the PSW's instruction address
is 0 then the test case likely completed successfully.
.core
Files that have the extension .core are binary core image files loaded
into main storage by the Hercules loadcore console command. They can
be created by many means; one of them is ASMA as described above.
The runtest UNIX shell script (runtest.cmd batch file on Windows)
is used to run one or more (usually all) automated .tst scripts to
verify proper Hercules functionality.
On UNIX, assuming that your are building outside the source tree,
perhaps in the "atom" directory, the source directory might be
../hyperion. Use the following command to run and verify the .tst test
cases:
../hyperion/tests/runtest
"make check" in the directory where you built Hercules will find stuff wherever you configured it.
On Windows the batch file runtest.cmd runs the test cases. Its first
few parameters are identical (almost) to the parameters accepted by
the UNIX runtest shell script, but supports many additional parameters
as well. Enter runtest /? or runtest --help for more information.
In either case, runtest first creates a composite set of test cases
(allTests.testin in the current working directory) and then invokes
Hercules to run them using the configuration file tests.conf and the
.rc file just created. When it completes, the console output
(allTests.out) is then inspected by redtest.rexx and the results are
reported to the user.
The runtest return code is the number of failed test cases. A return code of 0 indicates success (no failed test cases). Any other return code indicates failure (one or more failed test cases).
The tail end of an early sample output is:
...
Test SSKE S/390 nextlast. 3 OK compares. All pass.
Test SSKE S/390 lastpage. 3 OK compares. All pass.
Test timeout. 2 OK compares. All pass.
Done 138 tests. All OK.
The Linux runtest shell script accepts these flags:
-d <word>
Specify the path to the test directory. The default is ${dirname $0}, perhaps ../hyperion/tests
-e <word>
Specify the default extension for the input test files. tst is the default. The -e flag is active for all -f flags until the next -e flag, if any.
-f <word>
Specify the input file name perhaps including an extension. The default is -f *.tst to get the current behaviour. Multiple -f flags are supported. The default extension is appended if the word does not contain a period.
-h <word>
Specify the object directory that contains the hercules executable. The
default is parent of -d if there is no hercules executable in the
current directory (that is, in-source build).
-l <word>
Specify the name of a module to load dynamically into hercules. This is passed to Hercules.
-p <word>
Specify the library path for loadable modules, i.e., device managers. This is passed to Hercules.
-q
Pass "quiet" to redtest to suppress details about test cases.
-r <number>
Repeat the composite test script n times. There is no need to specify -x as hercules terminates without an explicit quit when in daemon mode.
-t <number>
Passed to hercules (timeout factor)
-v <word>
Set variable for redtest.rexx (in fact, arguments to redtest.rexx). Use -v ptrsize=4 when running a 32 bit executable on an 8 bit system if no valid config.h can be found. You can also set private variables this way (but such variables will be unset when running "make check"). Variable names are case insensitive but their values are not. -v quiet will make redtest more terse.
-w <word>
Specify the work file name. The default is allTests.
-x
Do not append the quit command to the composite test script.
The test a .tst script defines is normally begun via a special runtest
script command which supports optional arguments indicating how the
test should be started as well as the maximum amount of time it should
run. Unlike the pause command, runtest waits only until all CPUs
are in the stopped state:
runtest [restart|start|<oldpsw>] [timeout] [# comment...]
<oldpsw> ::= external | svc | program | machine | io
The first argument is optional. If specified, it must be either
restart, start, or an interrupt type. It specifies how the test
should be started: via the Hercules restart console command or the
start console command. The default is to start the test via the
restart command. When an interrupt type is specified, the old psw for
that type is copied to the restart new PSW before the restart command is
issued to fire up the test case where it left off taking an interrupt
for which the new PSW was disabled.
The second argument, which is also optional, defines the maximum amount of time (specified in a whole or fractional number of seconds) that the test is allowed to run. This is a safety feature to prevent runaway tests from running forever due to a bug in the test or within Hercules itself. Under normal circumstances all tests should end automatically the very moment they are done. If no timeout is specified, the default is 30 seconds.
Please also note that timeout values are multiplied by the test
timeout factor value which is specified by the -t switch on the
Hercules command. The test timeout factor value allows for running
tests on systems that may be slower than the system they were
originally designed for. Refer to the documentation describing
Hercules's supported command line arguments.
The runtest script command is almost always immediately followed
by Hercules console commands to display storage or registers, etc, as
well as several of the below .tst file directives to verify the test
case ran correctly.
.tst files are executed as Hercules .rc files, so they contain Hercules
console commands.
A number of "loud" comments (without a blank immediately following it)
are interpretted by redtest.rexx as directives. All testing directives
are case sensitive. Using *testcase for example accomplishes nothing.
* Hercules "loud" comment. There must be a blank after the
asterisk for it to be ignored. If it is not followed by a
blank it will interpretted as a test directive.
*Testcase Identify beginning of a test case.
*Compare An 'r', 'v' or 'abs' command follows and the output should
be stored for later comparison. The '*Compare' directive
triggers the storing of messages in the message queue for
later comparison by the '*Hmsg' and other directives. If
not given then any subsequent '*Hmsg' directives will fail
due to the stored message queue being empty.
*Want The expected r command output. A string identifying the
comparison may be enclosed in double quotes before the
compare data (e.g. `*Want "CRC-32" DF00E74C`)
*Gpr A previous gpr console command should have been issued to
display the general registers. The specified register is
compared with the hexadecimal data specified after the
register number. Specify a decimal register number with no
leading zero. The comment "#address" may be specified to
notify the CMS test driver that the register contains a
storage address.
*Cr Same thing as *Gpr except for control registers.
*Key Compare the specified key against the key displayed in the
previous r command.
*Prefix Compare the specified value to the contents of the prefix
register displayed by a previous pr command.
*Program A program check is expected, for example to verify that a
privileged operation causes an 0c2. Since program checks
typically cause a disabled wait state psw to be loaded,
to prevent an unwanted "unexpected wait state" error from
being detected instead this directive MUST be specified
BEFORE the 'runtest' script command that starts the test.
*Hmsg Specifies a complete informational message string to be
compared against the last stored message (except for any
messages that are processed by redtest.rexx plus messages
HHC00007I and HHC01603I). A number is optional before the
message identifier to specify the last but one message, the
last but two message, etc. For example, if a test expects
three messages then you would use:
*Hmsg 2 First expected message...
*Hmsg 1 Second expected message...
*Hmsg Third expected message...
The stored messages queue does not begin storing messages
until the '*Compare' directive is first given.
*Info Synonym for *Hmsg. Use if an informational message is
expected.
*Error Synonym for *Hmsg. Use if an error message is expected.
*Explain Specify the explanatory text to write when the next
comparison fails. You can specify as many *Explains as you
like for multiline messages. The explain array is cleared
once a test has been done, whether OK or not.
*Message Print a message on the output.
*If Specify a condition to test (see later). The expression is
evaluated by the Rexx interpreter. If the condition holds,
the following directives are processed up to the matching
*Else or *Fi. If the condition does not hold, directives
are ignored except that nested *If directives are parsed.
*Else Flip the setting of the matching *If condition, unless
nested in a suppressed *If.
*Fi End an *If. The process/ignore setting for directives is
unstacked.
*Done End of a test case. Specify "nowait" if the test case does
not end by loading a disabled PSW, for example, when it
tests just console commands without running any code.
*Timeout A timeout is expected. Do not use.
logicimm.tst shows examples of all directives, except *Key, *Prefix,
and *Program; see sske.tst for an example of *Key and *Program. See
pr.tst for *Prefix. See mainsize.tst for conditional processing.
Predefined variables are specified as arguments to redtest.rexx as well
as being extracted from the HHC01417I features messages. The runtest
script (not to be confused with the runtest command above) sets the value
of the $ptrsize variable to either 4 or 8 by directly passing the value
to redtest.rexx on the command line. Any number of user defined variables
can also be defined the same way by simply passing their name and value
on the command line to redtest.rexx. Refer to runtest's help display
for how to do this.
Other variables are defined by redtest.rexx itself based on the Hercules
HHC01417I startup and other messages. For example, the variable $cmpxchg1
is set to 1 (true) if "cmpxchg1" appears in message HHC01417I Machine dependent assists:. The variable $locking_model is set to error if the message
HHC01417I Using Error-Checking Mutex Locking Model is seen, and so on.
Variables are either of boolean, numeric or string type and are read only.
A variable's name is case insensitive but its value is not. $foo, $FOO,
$Foo, etc, all refer to the same variable. Predefined variables are meant
to be use in *If directives to adjust test cases according the defined
value of a given variable or variables, since one user's build of Hercules
(or the environment in which it runs) might not be the same as another's.
Designing you test cases to take this into consideration is encouraged.
The current list of pre-defined runtime variables are as follows:
Boolean:
$can_esa390_mode
$can_s370_mode
$can_zarch_mode
$CCKD_BZIP2
$cmpxchg1
$cmpxchg16
$cmpxchg4
$cmpxchg8
$CONFIG_INCLUDE
$CONFIG_SYMBOLS
$externalgui
$fetch_dw
$HAO
$HDL
$HET_BZIP2
$HTTP
$IEEE
$IPV6
$NLS
$regex
$rexx_ErrPrefix
$rexx_MsgLevel
$rexx_MsgPrefix
$rexx_Resolver
$rexx_supported
$rexx_SysPath
$shared_devices
$SIGABEND
$sqrtl
$store_dw
$syncio
$SYSTEM_SYMBOLS
$ZLIB
Numeric:
$max_cpu_engines
$ptrsize
String:
$hatomics atomicIntrinsics, C11, msvcIntrinsics, syncBuiltins
$keepalive Basic, Full, Partial
$libraries Shared, Static
$locking_model Error, Normal, Recursive
$platform Linux, Windows, etc.
$rexx_Extensions .REXX;.rexx;.REX;.rex;.CMD;.cmd;.RX;.rx
$rexx_Mode Command, Subroutine
$rexx_package OORexx, Regina
$rexx_RexxPath /foo:/bar:/foobar (or C:\Foo;C:\Bar;C:\FooBar)
$rexx_SOURCE WindowsNT, LINUX, etc.
$rexx_VERSION REXX-ooRexx_4.2.0(MT)_64-bit 6.04 22 Feb 2014
$threading_model Fish, POSIX
Examples of if/else variable usage:
mainsize 2147483648b
*If $ptrsize = 4
*Error 1 HHC01430S Error in function configure_storage(2G): Cannot allocate memory
*Error HHC02388E Configure storage error -1
*Else # 64 bit
*Info HHC17003I MAIN storage is 2G (mainsize); storage is not locked
*Fi
*If $HET_BZIP2
...(do test here)...
*Else
*Message SKIPPING: Testcase MyTest
*Message REASON: No BZIP2 support
*Fi
*If \$rexx_supported
*Message SKIPPING: Testcase Example
*Message REASON: No Hercules Rexx support.
*Else
*If $rexx_package = ''
*Message SKIPPING: Testcase Example
*Message REASON: Rexx is not installed.
*Else
...(do test here)...
*Fi
*Fi
Running a test case on z/CMS is accomplished by DOTEST EXEC. It merges the object deck for the test case with a pipeline program that interprets the runtest directives, which are extracted from the object deck.
A test case to run on z/CMS can be generated in two ways:
-
From assembler source. The object deck is uploaded to z/CMS and used directly. The Hercules (bare iron) test case is created from the object deck by
bldhtc.rexx. -
From a hand-built
.tstscript. The test script is converted to a TEXT deck bycvttst.rexx.
There are a number of differences when a test case is run on CMS rather than on the bare iron:
-
The machine is not all yours. Well, it is, but you may not wish to exercise that prerogative. You'd likely want CMS to remain after running the test case.
-
GPR15 is the base register for an emulated main storage the size of the program (see agf.assemble). The test case is entered with the return address in GPR14.
-
The test case must not modify GPR11 (or it must save and restore it).
-
Likewise, the test case must not modify GPR14 (or it must save and restore it).
-
The
*Gprdirective to verify an address must add the comment#addressso that the value can be rebased.
The following applies to .tst scripts that are generated from assembler
source:
- When the script is finished, it should first check whether GPR14 is nonzero and branch back rather than load a disabled PSW.
The following applies when cvttst.rexx generates an object deck from a
hand-built .tst script:
-
Loading a disabled PSW is replaced by BR 14, so there is no reason to write dual path code in hand-built .tst scripts.
-
Loading control register 0 is suppressed, but the driver notes the fact that the AFPR enable bit is on and turns it on for the duration of the test.
-
The test program must not reference storage beyond the highest address specified in any
rconsole command (either loading data or displaying it). The highest address is set as the control section size.
Here is an example of how to code a dual-path return in an assembler program:
ltr 14,14 Have a return address?
bnzr 14 return if so
LPSWE WAITPSW Load wait PSW
WAITPSW dc x'0002000180000000',ad(0) OK wait state PSW
Some programs contain a series of tests where the CPU is stopped after
each test. This should be emulated by a branch back using GPR14, having
stored the resume address in the SVC old PSW. This applies only to
tests that are assembled from assembler source; cvttst.rexx knows how to
deal with loading a disabled PSW.
swap equ * entry by bas 9,swap
ltr 14,14 Running on CMS?
jz bare
* Under z/CMS in 31-bit mode. Simulate a SVC 255
la 10,0(,9) Strip amode bit
st 10,svcold+12
basr 14,14 Return to driver
br 9 Return to caller
bare equ *
svc 255 We are in problem state
br 9
The test cases are TEXT decks on CMS. When the .tst script is built from
assembler source, the object deck is a by product; it should be uploaded
to CMS without change. The test case is loaded at a page boundary by CMS.
The Rexx script cvttst.rexx builds an object deck from a .tst script
that is built by hand (notably the ieee floating point test cases).
This process makes several changes to the program:
-
References to the PSA are rebased to use GPR15 as a base register. For
LA, this is done only when the displacement is above the first instruction in the program (which is obtained from the restart new PSW field). -
A
LPSW[E]for a disabled wait is replaced byBR 14to return. -
LCTLto enable advanced floating point is replaced by a no operation and a directive is emitted to have the test driver enable this flag and disable it again after the tests have all been run. -
The addresses in the various new PSW fields in the PSA are marked as relocatable when the PSW is not a disabled wait. This will relocate the restart new PSW and make sure that the test case fires up the right place. For other new PSWs this is of little practical application since the real interrupts will be fielded by CMS.
-
Distinguishing between an instruction and a constant is in general an untractable problem, but it is assumed that there is one
rcommand for each instruction and for each constant. Clearly, a length that is not one of 2, 4, 6 must refer to a constant. If the first two bits of the data are not congruent with the length of the area, it must be a constant. Further, data where the first byte is either '00'x or 'ff'x is also a constant.This leaves negative short precision constants. If the displacement part of the value is less than '200'x it is taken as a constant. I'll likely have to invent better algorithms.
cvttst is still very much experimental.