======================================================================
Title:		S2 language documentation
Version:	2006-09-15
======================================================================

This document describes the current state of S2, a general-purpose test
client with SRM2 protocol support.  There are README files in every
S2 distribution which should be read first.  Last (but not least)
the PCRE library has very extensive and exhaustive documentation on
pattern matching.

Overview
~~~~~~~~
S2 is a simple general-purpose test client with the SRM 2.1 and 2.2
protocol support.  S2 interprets a tree-like language, which was also
named S2.

The S2 tree consists of branches.  Every branch is indented by 0
or more spaces which, together with its row position, defines its
relationship to the other branches.  This indentation is then followed
by an optional set of parameters common to all branches, followed
by a specific <ACTION> with clearly defined syntax and semantics.
Branches located at the same indentation can be connected by three
different primitives.  No connection at all (which stands for parallel
execution), the OR and AND connectives.  The S2 language also supports
parallel and sequential (OR and AND) repeats.

The evaluation of an S2 tree starts with its root (the first
branch with no indentation).  If an <ACTION> of a branch executes
successfully, the evaluation process continues with the execution of
its child.  Unsuccessful execution results in a search and evaluation
of the first OR branch indented by the same number of spaces as the
unsuccessful <ACTION>.  The OR and AND branches are left-associative
and are evaluated in the short-circuit style.

The parameters common to all branches include the evaluation level
which decides success or failure of a branch's execution, timeout
and PCRE library options.

Regular expressions can be used as parameters of <ACTION>s where
a response is expected.  This is the most powerful feature of the
S2 language.

S2 also supports simple preprocessor directives; support for macros, 
expression evaluation and further <ACTION>s is planned.


Abbreviations
~~~~~~~~~~~~~
LHS     left hand side
RHS     right hand side
PCRE	Perl Compatible Regular Expressions


1 Syntax
~~~~~~~~
The grammar uses some notation from ASF+SDF specification, a
specification formalism developed at the University of Amsterdam and
the Centrum voor Wiskunde en Informatica.  The following notational
conventions apply

- S* defines zero or more repetitions of a symbol (non-terminal or 
  literal) S;
- S+ defines one or more repetitions of a symbol S;
- {S sep}* defines zero or more repetitions of a symbol S separated 
  by the literal sep;
- {S sep}+ defines one or more repetitions of a symbol S separated 
  by the literal sep;
- (S) defines grouping of two or more symbols `S';
- [s] represents any one of the literals in the string (a character 
  class) s;
- [^s] represents any character not in the string s;
- \t is the horizontal tabulation character;
- \n is the newline character.


1.1) Branches
~~~~~~~~~~~~~
The start symbol of the grammar is <S>.  Comments start optional whitespace,
the '%' character [as in (La)TeX] and continue up to the next newline
('\n') character.

  <S> :: (<BRANCH> '\n' | <PREPROCESSOR> '\n')*

  <BRANCH> :: <OFFSET> <BRANCH_PREFIX>? <ACTION> ('\n' <OFFSET> <BRANCH>)*
; The RHS <OFFSET> must be at least one space character higher than the LHS <OFFSET>.

  <OFFSET> :: [ ]*

  <BRANCH_PREFIX> :: <COND>? <REPEAT>? <BRANCH_OPT>*

  <COND> :: '||' | '&&' | ';;'

  <REPEAT> :: '>' <X> <PAR_SEQ> <Y> | 'WHILE'
  
  <PAR_SEQ> :: ' ' | '||' | '&&' | ';;'
  
  <X> :: <DQ_I>
; repeat start value

  <Y> :: <DQ_I>
; repeat end value

  <BRANCH_OPT> :: <EVAL> | <TIMEOUT> | <MATCH_OPTS>

  <EVAL> :: 'eval' '=' <I_ENV>
; default <I_ENV> value is 0, note the this must be a constant or $ENV{<VAR>}
; maximum exit value of <ACTION> to start the evaluation of RHS <BRANCH>

  <TIMEOUT> :: 'timeout' '=' <TIMEOUT_MICROSEC>
; default <TIMEOUT_MICROSEC> value is 0, note the this must be a constant or $ENV{<VAR>}

  <TIMEOUT_MICROSEC> :: <N_ENV>

  <MATCH_OPT> :: 'match' '=' <MATCH_OPTS>

  <MATCH_OPTS> :: [01hlicEsxXmNU8?BZN-]+


1.2) Preprocessor
~~~~~~~~~~~~~~~~~
  <PREPROCESSOR> :: <IF_ELSE_ENDIF> | <DEFINE> | <INCLUDE> | <REQUIRE>

  <IF_ELSE_ENDIF> :: <IF> '\n' <BRANCH>* (<ELSE> '\n' <BRANCH>*)? <ENDIF>

  <IF> :: '#' 'if' <IF_COND> | '#' 'ifdef' <IFDEF_CONF> | '#' 'ifndef' <IFDEF_CONF>

  <ELSE> :: '#' 'else'

  <ENDIF> :: '#' 'endif'

  <IF_COND> :: <I_ENV>

  <IFDEF_COND> :: <DEFINE_VAR>

  <DEFINE> :: '#' 'define' <DEFINE_VAR>

  <DEFINE_VAR> :: <DQ_PARAM>

  <INCLUDE> :: '#' 'include' <INCLUDE_FILE>
  
  <INCLUDE_FILE> :: <DQ_PARAM>
; Note <INCLUDE_FILE> is <DQ_PARAM>
; (with the exception of $ENV{<VAR>} it is not dynamically evaluated)

  <REQUIRE> :: '#' 'require' <S2_VERSION>
  
  <S2_VERSION> :: <DQ_PARAM>

1.3) Actions
~~~~~~~~~~~~
  <ACTION> :: <ASSIGN> | <DEFUN> | <FUN> |
              <NOP> | <SETENV> | <SLEEP> | <SRM2> | <SYSTEM>

Syntax of the individual actions is described in the Semantics section.


1.4 Lexicon
~~~~~~~~~~~
1.4.1 Static
~~~~~~~~~~~~
  <INTEGER> :: ('-' | '+')? <NATURAL>

  <DNATURAL> :: [1-9][0-9]*

  <NATURAL> :: <DNATURAL> |  '0x'[0-9A-Fa-f]+ | 0[0-7]*
; a natural number (decimal, hexadecimal and    octal numbers are supported)

  <BR_PARAM> :: [^}]*
; any string terminated by an unquoted '}' character (not in <BR_PARAM>)
; <TAG>s in <BR_PARAM> are evaluated.

  <VAR> :: <ID>

  <ID> :: [A-Za-z_][A-Za-z0-9_]*

  <STR> :: <CHAR>*

  <CHAR> :: [\x00-\xFF]

  <HEX_STR> :: [0-9A-F]+

  <ARG> :: <DQ_PARAM>

1.4.1.1 <DQ_PARAM>
~~~~~~~~~~~~~~~~~~
  <DQ_PARAM> :: [^ \t]* | '"' [^"]* '"'

; a parameter optionally enclosed by double-quotes:
; e.g.:\ this\ is\ a\ DQ_PARAM\ with\ escaped\ spaces
; "e.g.: this is a DQ_PARAM"
; <TAG>s in <DQ_PARAM> are evaluated.

TODO: <DQ_PARAM> syntax of unquoted TAGs, e.g.:
  ASSIGN v $MATCH{. a}

1.4.2 Dynamic
~~~~~~~~~~~~~
The following non-terminals appearing in <DQ_PARAM> and <BR_PARAM>
are evaluated before the values are used.

  <ENV> :: '$ENV{' <BR_STR> '}'

  <TAG> :: '${' <BR_STR> '}' | 
           '$ENV{' <BR_STR> '}' |
           '$I{' <DQ_I> '}' |
           '$EVAL{' <BR_STR> '}' |
           '$EXPR{' <EXPR> '}' |
           '$RND{' <EXPR> '}' |
           '$DATE{' <BR_STR> '}' |
           '$PRINTF{' <PRINTF_FORMAT> (' ' <PRINTF_ARG>)* '}' |
           '$MD5{' <BR_STR> '}' |
           '$DEFINED{' <BR_STR> '}' |
           '$MATCH{' <PATTERN> <SUBJECT> '}' |
           '$INT{' <EXPR> '}' |
           '$FUN{' <FUN_NAME> {<FUN_ARG> ' '}* '}' |
           '$MAP{' <FUN_NAME> {<FUN_ARG> ' '}* '}' |
           '$SEQ{' <SEQ_FROM> <SEQ_TO> '}' |
           '$INTERLEAVE{' <INTERLEAVE_COUNT> {<INTERLEAVE_ARG> ' '}* '}' |
           '$SPLIT{' <BR_STR> '}'

  <I_ENV> :: <INTEGER> | <ENV>
; <ENV> must evaluate to <INTEGER>

  <N_ENV> :: <NATURAL> | <ENV>
; <ENV> must evaluate to <NATURAL>

  <DQ_I> :: <DQ_PARAM>
; <DQ_PARAM> must evaluate to <INTEGER>

  <DQ_N> :: <DQ_PARAM>
; <DQ_PARAM> must evaluate to <NATURAL>

  <DQ_STR> :: <DQ_PARAM>
; <DQ_PARAM> always evaluates to <STR>

  <BR_I> :: <BR_PARAM>
; <BR_PARAM> must evaluate to <INTEGER>

  <BR_N> :: <BR_PARAM>
; <BR_PARAM> must evaluate to <NATURAL>

  <BR_STR> :: <BR_PARAM>
; <BR_PARAM> always evaluates to <STR>


2 Semantics
~~~~~~~~~~~
2.1 Definitions
~~~~~~~~~~~~~~~
Execution and evaluation
~~~~~~~~~~~~~~~~~~~~~~~~
1) (Partial) Execution
~~~~~~~~~~~~~~~~~~~~~~
A branch executes as TRUE (success), if execution value of <ACTION>
of this branch is less or equal to <EVAL>; otherwise it executes as
FALSE (failure).  It is simple to force execution of a parent's child
(RHS <BRANCH>) by increasing the value of <EVAL>.  Execution value
of a branch is the exit value of <ACTION>.

2) (Complete) Evaluation
~~~~~~~~~~~~~~~~~~~~~~~~
A branch evaluates to TRUE (success), if all its sub-branches execute
as TRUE.  If any of its sub-branches executes as FALSE, it evaluates
to FALSE (failure).  The (complete) evaluation value of a branch
is the execution value of the last sub-branch.

3) A virtual branch
~~~~~~~~~~~~~~~~~~~
Virtual branches are created from other branches by means of connecting
them to other branches placed on the same level of indentation.
The connectives can be '||' and '&&' operators, an empty connective
which stands for parallel execution or ';;' which stands for
unconditional sequential execution.  A branch is a virtual branch.

Branch execution/evaluation values
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
0: success
1: warning(s)
2: function/branch evaluation failure(s)
3: script could not be executed/timeout
4: assertion failed error (internal error)
5: system error (malloc failed, etc.)

2.2 Basic concept
~~~~~~~~~~~~~~~~~
General model
-------------
<branch 1a>
  <branch 2a>
  <branch 2b>
    <branch 3>
<branch 1b>
<branch 1c>

Indentation of branches is crucial.  <BRANCH>es indented by the same
amount of whitespace (starting at the identical column) are evaluated
in parallel unless special '||', '&&' or ';;' operators described in
the following sections are used.  A branch having a parent is evaluated
only if execution of this parent succeeded (i.e. a conditional
sequential execution).  For example, <branch 3> is only executed
if execution of <branch 2b> succeeded. (Consequently, execution of
<branch 1a> must have succeeded as well.)  Branches <branch 1a>,
<branch 1b> and <branch 1c> are evaluated in parallel, and so are
branches <branch 2a> and <branch 2b>.

Empty lines are ignored.

Conditional sequential execution, the '||' (OR) operator
--------------------------------------------------------
<branch 1>
|| <branch 2>

<branch 2> is evaluated only if evaluation of <branch 1> fails.

The virtual branch created by connecting <branch 1> and <branch 2>
by the '||' operator evaluates to TRUE if <branch 1> evaluates to
TRUE, otherwise it evaluates to evaluation of <branch 2>.

The evaluation value of a virtual branch connected by the '||'
operator is the evaluation value of <branch 1> if it evaluates to
TRUE, otherwise the evaluation value of <branch 2>.

Conditional sequential execution, the '&&' (AND) operator
---------------------------------------------------------
<branch 1>
&& <branch 2>

<branch 2> is evaluated only if the evaluation of <branch 1>
succeeds.

There is no semantic difference between the following two trees:

tree 1         | tree 2
----------------------------------------
<branch 1>     | <branch 1>
  <branch 2>   | && <branch 2>

As tree 1 and tree 2 are semantically equivalent, the evaluation
value of a virtual branch connected by the '&&' operator is equal
to the evaluation value of <branch 1> of tree 1.

The aim of the '&&' operator is to provide a "join" primitive.  For 
example, the following trees (1 and 2) are semantically equivalent:

tree 1 (not using '&&')      | tree 2 (using the '&&' primitive)
----------------------------------------------------------------
<branch 1>                   | <branch 1>
  <branch 2a>                |   <branch 2a>
    <branch 3a>              |     <branch 3a>
      <branch 2c>            |
        <branch 3c>          |
  || <branch 2b>             |   || <branch 2b>
    <branch 3b>              |     <branch 3b>
      <branch 2c>            |   && <branch 2c>
        <branch 3c>          |     <branch 3c>
  <branch 2d>                |   <branch 2d>

Unconditional sequential execution, the ';;' operator
-----------------------------------------------------
<branch 1>
;; <branch 2>

<branch 2> is evaluated sequentially irrespective of 
<branch 1> evaluation result.

The virtual branch created by connecting <branch 1> and <branch 2>
by the ';;' evaluates to evaluation of <branch 2>.

The evaluation value of a virtual branch connected by the ';;'
operator is the evaluation value of <branch 2>.

Parallel execution
------------------
Branches not connected by any of the above conditions are evaluated in
parallel.

<branch 1>
<branch 2>

A virtual branch connected by the parallel (empty) connective evaluates
to TRUE, if all of the branches evaluate to TRUE.  Otherwise it
evaluates to FALSE.

The evaluation value of a virtual branch connected by the parallel
(empty) connective is the the maximum evaluation value of these
branches.


Repeatition
~~~~~~~~~~~
There are 3 different repeat operators.  However, they all share the
following characteristic:

  >X Y

From X (down)to Y inclusive.  For example, >0 0 repeats the branch
just once.

The sequential '&&' repeat operator
-----------------------------------
  >X&&Y <branch>

A virtual branch created by the use of the sequential '&&' repeat
operator evaluates to TRUE, if all of <branch> iterations
evaluate to TRUE.  The evaluation of the '&&' repeat loop
ends with its first unsuccessful evaluation and it evaluates 
to FALSE.

The evaluation value is the value of the last iteration.

The sequential '||' repeat operator
-----------------------------------
  >X||Y <branch>

A virtual branch created by the use of the sequential '||' repeat
operator evaluates to FALSE, if all of <branch> iterations evaluate
to FALSE.  The evaluation of the '||' repeat loop ends with its first
successful evaluation and it evaluates to TRUE.

The evaluation value is the value of the last iteration.

The sequential ';;' repeat operator
-----------------------------------
  >X;;Y <branch>

A virtual branch created by the use of the unconditional sequential
';;' repeat operator evaluates to the evaluation value of the last
<branch> iteration.

The evaluation value is the value of the last iteration.

The parallel repeat operator
----------------------------
  >X Y <branch>

A virtual branch created by the use of the parallel repeat operator
evaluates to TRUE, if all of <branch> iterations evaluate to TRUE.
The evaluation of the parallel repeat loop ends when all its iterations
complete (successfully or otherwise).  If any of the iterations
evaluates to FALSE, the virtual branch evaluates to FALSE.

The evaluation value of the branch is the maximum of its completed
iterations.

The WHILE loop
--------------
  WHILE <branch>

Repeat execution of <branch> while it evaluates to TRUE.


2.3 Actions
~~~~~~~~~~~
There are several kinds of branch actions.


2.3.1 ASSIGN
~~~~~~~~~~~~
  <ASSIGN> :: 'ASSIGN' <OVERWRITE>? ( <VAR> <VAR_VAL> )+
  <OVERWRITE> :: 'overwrite' '=' <DQ_I>
  <VAR_VAL> :: <DQ_STR>

If <VAR> is not set, assign variable <VAR> value <VAR_VAL>.  If <VAR>
is set, assign variable <VAR> value <VAR_VAL> only if <OVERWRITE>
is unspecified or <DQ_I> evaluates to non-zero.


2.3.2 Function definition
~~~~~~~~~~~~~~~~~~~~~~~~~
  <DEFUN> :: 'DEFUN' <FUN_NAME> ({<FUN_PARAM> ' '}* ':'? | {<FUN_PARAM_REF> ' '}*)?
  <FUN_NAME> :: <DQ_STR>
  <FUN_PARAM> :: <DQ_STR>
  <FUN_PARAM_REF> :: <DQ_STR>


2.3.3 Function call
~~~~~~~~~~~~~~~~~~~
  <DEFUN> :: 'FUN' <FUN_NAME> ({<FUN_ARG> ' '}* ':'? | {<FUN_ARG_REF> ' '}*)?
  <FUN_ARG> :: <DQ_STR>
  <FUN_ARG_REF> :: <DQ_STR>

<FUN_ARG> are passed by value, <FUN_ARG_REF> are passed by reference.


2.3.4 NOP
~~~~~~~~~~
  <NOP> :: 'NOP' <NOP_VAL>?
  <NOP_VAL> :: <DQ_I>

The simplest action of all is the <NOP> action which is a "no 
operation" action.  It has an optional parameter --- execution value.
If this parameter is omitted, the default value 0 is used.


2.3.5 SETENV
~~~~~~~~~~~~
  <SETENV> :: 'SETENV' <OVERWRITE>? ( <VAR> <VAR_VAL> )+
  <OVERWRITE> :: 'overwrite' '=' <DQ_I>
  <VAR_VAL> :: <DQ_STR>

If <VAR> is not set, assign environment variable <VAR> value <VAR_VAL>.
If <VAR> is set, assign environment variable <VAR> value <VAR_VAL>
only if <OVERWRITE> is unspecified or <DQ_I> evaluates to non-zero.


2.3.6 SLEEP
~~~~~~~~~~~
  <SLEEP> :: 'SLEEP' <SLEEP_OPT>?
  <SLEEP_OPT> :: <SLEEP_SEC> <SLEEP_NANOSEC>?
  <SLEEP_SEC> :: <DQ_N>
  <SLEEP_NANOSEC> :: <DQ_N>

Sleep for a specified amount of seconds and/or nanoseconds.


2.3.9 SRM2 actions
~~~~~~~~~~~~~~~~~~~
All SRM2 methods are implemented.  Please see the
`testing/scripts/protos/srm/{2.1,2.2}/template' directories for syntax.
Also, please refer to the `How to create new SRM 2.2 scenarios' section
of the README document.


2.3.10 SYSTEM
~~~~~~~~~~~~~
  <SYSTEM> :: 'SYSTEM' <SYSTEM_OPT>? <SYSTEM_COMMAND>
  <SYSTEM_OPT> :: 'out' '=' <DQ_STR>
  <SYSTEM_COMMAND> :: <STR>

System call with an optional argument (out=<regex>) allowing the
capture and pattern matching of a program's output.


2.3.11 TEST
~~~~~~~~~~~
  <TEST> :: 'TEST' <EXPR>
  <EXPR> :: <STR>

Exit with the status determined by <EXPR>.  TEST <EXPR> executes
as TRUE unless it is an empty string or 0.


2.4 Pattern matching
~~~~~~~~~~~~~~~~~~~~
There are various options which affect pattern matching.  The meaning
column of the following two tables is for the option set to value 1.

option | long name | meaning
-------+--------------------------------------------------------------
h      | hbsplit   | compare head and body of a message separately
l      | linesplit | pattern is matched line-by-line against
       |           | the subject string; every pattern line has
       |           | to match against the subject string

hbsplit:
~~~~~~~~
head and body are separated by one of the following sequences:
1) '\n\n'
2) '\n\r\n'
In both cases, head contains the first newline '\n' character, and
body starts with the first character following the second '\n' 
character.

linesplit:
~~~~~~~~~~
This option is particularly useful for line-by-line matching when the
order of lines in the subject string is not important, but every line
of the pattern string has to be contained in the subject string.  It is 
also very useful in combination with the multiline option to assert 
newlines within the subject string by the ^ and $ characters.

Note that when the `notempty' option (N) is set in this mode and the
last line of a pattern is terminated by the newline '\n' character,
this particular pattern can never match as its last line is actually an
empty string.

option | internal option  | long name       | PCRE name              |
-------+-------------------------------------------------------------+
i      | i                | caseless        | PCRE_CASELESS          |
c      | ---------------- | complete        | PCRE_ANCHORED + \z     |
E      | ---------------- | dollar_endonly  | PCRE_DOLLAR_END_ONLY   |
s      | s                | dotall          | PCRE_DOTALL            |
x      | x                | extended        | PCRE_EXTENDED          |
X      | X                | extra           | PCRE_EXTRA             |
m      | m                | multiline       | PCRE_MULTILINE         |
n      | ---------------- | no_auto_capture | PCRE_NO_AUTO_CAPTURE   |
U      | U                | ungreedy        | PCRE_UNGREEDY          |
8      | ---------------- | utf8            | PCRE_UTF8              |
?      | ---------------- | utf8_no_check   | PCRE_NO_UTF8_CHECK     |
-------+------------------+-----------------+------------------------+
B      | ---------------- | notbol          | PCRE_NOTBOL            |
Z      | ---------------- | noteol          | PCRE_NOTEOL            |
N      | ---------------- | notempty        | PCRE_NOTEMPTY          |

The option names have been chosen so that they do not collide with the
options of the pcretest file in the PCRE library distribution (v5.0).

caseless:
~~~~~~~~~
Do caseless matching.

complete:
~~~~~~~~~
Whole-pattern matching.  Pattern must match the "subject string" from
its start to the end.  Equivalent to prefixing the pattern by \A and
appending \z (not just \Z!).

dollar_endonly:
~~~~~~~~~~~~~~~
$ not to match newline at end.  If this options is set, a dollar
metacharacter in the pattern matches only at the end of the subject
string.  Without this option, a dollar also matches immediately before
the final character if it is a newline (but not before any other
newlines).  This option is ignored if `multiline' mode is set.

dotall:
~~~~~~~
Dot `.' matches anything including newlines.  Without it, newlines
are excluded.  A negative class such as [^a] always matches a newline
character, independent of the setting of this option.

extended:
~~~~~~~~~
If this option is set, whitespace data characters in the pattern
are totally ignored except when escaped or inside a character
class. Whitespace does not include the VT character (code 11).  In
addition, characters between an unescaped # outside a character class
and the next newline character, inclusive, are also ignored.

This option makes it possible to include comments inside
complicated patterns.  Note, however, that this applies only to data
characters. Whitespace characters may never appear within special
character sequences in a pattern, for example within the sequence (?(
which introduces a conditional subpattern.

extra:
~~~~~~
This option was invented in order to turn on additional functionality
of PCRE that is incompatible with Perl, but it is currently of very
little use. When set, any backslash in a pattern that is followed by
a letter that has no special meaning causes an error, thus reserving
these combinations for future expansion. By default, as in Perl, a
backslash followed by a letter with no special meaning is treated as
a literal.

multiline:
~~~~~~~~~~
^ and $ match newlines within data.  By default, PCRE treats the
subject string as consisting of a single line of characters (even if
it actually contains newlines).  The "start of line" metacharacter
(^) matches only at the start of the string, while the "end of line"
metacharacter ($) matches only at the end of the string, or before a
terminating newline (unless `dollar_endonly' is set).

When `multiline' option is set, the "start of line" and "end of line"
constructs match immediately following or immediately before any
newline in the subject string, respectively, as well as at the very
start and end.  If there are no "\n" characters in a subject string,
or no occurrences of ^ or $ in a pattern, setting `multiline' has no
effect.

no_auto_capture:
~~~~~~~~~~~~~~~~
If this option is set, it disables the use of numbered capturing
parentheses in the pattern.  Any opening parenthesis that is
not followed by ? behaves as if it were followed by ?: but named
parentheses can still be used for capturing (and they acquire numbers
in the usual way).

ungreedy:
~~~~~~~~~
This option inverts the "greediness" of the quantifiers so that they
are not greedy by default, but become greedy if followed by "?".
It can also be set by a (?U) option setting within the pattern.

utf8:
~~~~~
This option causes PCRE to regard both the pattern and the subject
as strings of UTF-8 characters instead of single-byte character
strings.  However, it is available only when PCRE is built to
include UTF-8 support.  If not, the use of this option provokes an
error.  Note that UTF-8 support must also be enabled enabled during 
PCRE library compilation.

no_utf8_check:
~~~~~~~~~~~~~~
When utf8 is set, the validity of the pattern as a UTF-8 string is
automatically checked. If an invalid UTF-8 sequence of bytes is found,
pcre_compile() returns an error. If you already know that your pattern
is valid, and you want to skip this check for performance reasons, you
can set the `no_utf8_check' option. When it is set, the effect of
passing an invalid UTF-8 string as a pattern is undefined. It may cause
the S2 interpreter to crash.

notbol:
~~~~~~~
This option specifies that first character of the subject string is not
the beginning of a line, so the circumflex metacharacter (^) should
not match before it.  Setting this without `multiline' (at compile
time) causes circumflex never to match.  This option affects only the
behaviour of the circumflex metacharacter.  It does not affect \A.

noteol:
~~~~~~~
This option specifies that the end of the subject string is not the end
of a line, so the dollar metacharacter should not match it nor (except
in multiline mode) a newline immediately before it. Setting this
without `multiline' (at compile time) causes dollar never to match.
This option affects only the behaviour of the dollar metacharacter. It
does not affect \Z or \z.

notempty:
~~~~~~~~~
An empty string is not considered to be a valid match if this option is
set.  If there are alternatives in the pattern, they are tried. If all
the alternatives match the empty string, the entire match fails.

There are three additional options which can be used to set the required
behaviour.

option | long name | meaning
-------+-----------+------------------------------------------------
0      | none      | all options disabled
1      | all       | all options enabled
-      | switch    | invert the meaning of options which follow

Regular expressions
~~~~~~~~~~~~~~~~~~~
For detailed information about the syntax of regular expressions
supported please refer to PCRE library manuals especially to
pcrepattern.3.  A test script writer should especially go through
the sections "Simple assertions", "Internal option settings",
"Subpatterns", "Named subpatterns", "Repetition (making quantifiers not
greedy)", "Atomic grouping and possessive quantifiers", and "Conditional
subpatterns" as they are not generally well known features of regular
expressions.

Atomic grouping is important where performance of test script is an
issue.

Named regular expressions allow parsing values from received messages 
into variables for use in all child branches (and subsequent lines of a 
message to be matched).

e.g.: (?P<named>.)
      matches one character and stores it into a special variable ${named}
      for use in all subsequent lines ('TXT' message) and children.

Caution must be taken when defining and using values of named regular
expressions on the same line.  Consider the following line:

e.g.: 
char *expected = "ab(?P<named_regex>.*)ef.*(?P=named_regex)"
char *received = "abcdefghcdef"

Named regular expression "named_regex" contains "cd" and is
subsequently used on the same line to match the second occurrence of
the string "cd".  The `expected' string is a valid regular expression
supported by the PCRE library.

S2 uses ${name_string} syntax to refer to variables instantiated by
the use of named regular expressions.  These variables are written
into internal structures during matching.  See the following example:

e.g.: ${named_regex}some_text(?P<named_regex>...)some_text${named_regex} 
  Provided we have already a variable "named_regex" with value "12345",
  the line is evaluated before matching as 
  =>           12345some_text(?P<named_regex>...)some_text12345
  and the value of the variable ${named_regex} 12345 is overwritten by the
  value of (?P<named_regex>...) after matching completes.  Provided we 
  want to use the value of (?<named_regex>) on the same line, the 
  following regular expression has to be given
  =>           12345some_text(?P<named_regex>...)some_text(?P=named_regex)


2.5 Variables
~~~~~~~~~~~~~
2.5.1 Script variables
~~~~~~~~~~~~~~~~~~~~~~
Script variables are defined by the ASSIGN action, or by using
named regular expressions as described in the previous section.
All variables are global variables apart from variables defined in
parallel processes and functions.  A function call and a parallel
process both create a separate local variable scope.

Script variable values are accessed by a number of ways:

1) The ${0<VAR>} tag refers to a global variable <VAR>.  If this
   variable does not exist, the tag expands as an empty string, a
   warning message is issued and the branch executes (at the very best)
   as 1, i.e. warning (see Branch execution/evaluation values section).
   Also, please see 4).

2) The ${<VAR>} tag refers to a local variable <VAR>.  If this variable
   does not exist, parent's variable scope(s) is (are) searched.  If
   this variable is not defined in the topmost parent's variable scope,
   global variable scope is searched.  The behaviour of this tag is then
   equivalent to 1), i.e. the ${0<VAR>} tag.

3) The ${<DNATURAL><VAR>}.  <DNATURAL> local variable scope(s) is skipped
   before the search for a variable <VAR> starts.  The search method is
   then equivalent to the search method in 2). 

4) All of the above tags can be used with a special prefix (-), which tells
   the variable evaluation engine not to issue any warnings and not to
   change execution value of the branch.  The following is possible:
   ${-0<VAR>}/${-<DNATURAL><VAR>}/${-<VAR>} to refer to global/parent's/local
   variable <VAR> and ignore <VAR>'s non-existence.

While 1) and 3) are often useful and result in very compact
descriptions, the reader is encouraged to use them with caution,
sparingly and use return-by-reference function parameters/arguments.

2.5.2 Environment variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Environment variables are accessible by the $ENV{<VAR>} tag.

2.5.3 Special variables
~~~~~~~~~~~~~~~~~~~~~~~
${?} reference to parent's execution value.  If a branch doesn't have
     a parent, ${?} expands to 0.

${!} reference to evaluation value of a branch that was evaluated immediately
     before the current branch and is indented by the same number 
     of spaces as the current branch.  If such a branch doesn't exist ${!}
     expands to 0.

For example:

NOP 2
&& SYSTEM echo "Not evaluated."
|| SYSTEM echo ${!}

shall print the number 2.

$I{<BR_I>} 
  refrerence to the internal counter of a repeat operator.
  The value is always between <X> any <Y>.  <BR_I> denotes branch
  nesting.  0 refers to the innermost repeat operator, 1 refers
  to the second innermost repeat operator...  If <BR_I> refers to
  a non-existent repeat operator, 0 is returned and a warning message
  issued.


2.5 Tags
~~~~~~~~
2.5.1 String evaluation
~~~~~~~~~~~~~~~~~~~~~~~
  '$EVAL{' <BR_STR> '}'

<BR_STR> is evaluated twice, rather than just once if no $EVAL tag was used.
For example, if <BR_STR> is "${v1}", the value of v1 is "${one}" and the value
of one is "1", the result of using <BR_STR> is "${one}", whereas the result of
using $EVAL{${v1}} is "1".


2.5.2 Expression evaluation
~~~~~~~~~~~~~~~~~~~~~~~~~~~
  '$EXPR{' <EXPR> '}'

  <EXPR> :: <EXPR> '||' <EXPR> | <EXPR> '&&' <EXPR> |
            <EXPR> '|'  <EXPR> | <EXPR> '^'  <EXPR> | <EXPR> '&' <EXPR> |
            <EXPR> '==' <EXPR> | <EXPR> '!=' <EXPR> |
            <EXPR> '<'  <EXPR> | <EXPR> '<=' <EXPR> | <EXPR> '>' <EXPR> | <EXPR> '>=' <EXPR> |
            <EXPR> '<<' <EXPR> | <EXPR> '>>' <EXPR> |
            <EXPR> '+'  <EXPR> | <EXPR> '-'  <EXPR> |
            <EXPR> '*'  <EXPR> | <EXPR> '/'  <EXPR> | <EXPR> '%' <EXPR> |
            '+' <EXPR> | '-' <EXPR> | '!' <EXPR> | '~' <EXPR> |
            '(' <EXPR> ')' | <DQ_I> | <DQ_STR>


  LOWEST
  ==============================
  ||
  &&
  |
  ^
  &
  == !=
  < <= > >=
  << >>
  + -
  * / %
  + - ! ~                (unary)
  ()
  ==============================
  HIGHEST

  C-like operator priority table 

2.5.3 Random number
~~~~~~~~~~~~~~~~~~~
  '$RND{' <EXPR> '}'

Returns a random number between 0 and <EXPR>-1.

2.5.4 Date
~~~~~~~~~~
  '$DATE{' <BR_STR> '}'

Interpreted sequences in <BR_STR> are:

  %%     a literal %
  %C     century  (year  divided  by  100  and  truncated  to an integer)
         [00-99]
  %d     day of month (01..31)
  %D     date (mm/dd/yy)
  %e     day of month, blank padded ( 1..31)
  %F     same as %Y-%m-%d
  %H     hour (00..23)
  %I     hour (01..12)
  %j     day of year (001..366)
  %k     hour ( 0..23)
  %l     hour ( 1..12)
  %m     month (01..12)
  %M     minute (00..59)
  %n     a newline
  %N     nanoseconds (000000000..999999999)
  %R     time, 24-hour (hh:mm)
  %s     seconds since 00:00:00 1970-01-01 UTC
  %S     second (00..60); the 60 is necessary to accommodate a leap second
  %t     a horizontal tab
  %T     time, 24-hour (hh:mm:ss)
  %u     day of week (1..7);  1 represents Monday
  %V     week number of year with Monday as first day of week (01..53)
  %w     day of week (0..6);  0 represents Sunday
  %y     last two digits of year (00..99)
  %Y     year (1970...)

If <STR> is empty, the output is +%Y-%m-%d@%H:%M:%S.%N, where %N is cut to 6 digits.

2.5.5 Formatted output
~~~~~~~~~~~~~~~~~~~~~~
  '$PRINTF{' <PRINTF_FORMAT> (' ' <PRINTF_ARG>)* '}'
  <PRINTF_FORMAT> :: <DQ_STR>
  <PRINTF_ARGS> :: <DQ_STR>

The tag is replaced by a formatted output.  This tag conforms to 
IEEE Std 1003.1-2001 (``POSIX.1'').
  
2.5.6 MD5 sum of a string
~~~~~~~~~~~~~~~~~~~~~~~~~
  '$MD5{' <BR_STR> '}'

The tag is replaced by an MD5 sum of <BR_STR>.

2.5.7 Test on an existence of a variable
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  '$DEFINED{' <BR_STR> '}'

The tag is replaced by 1 if variable <BR_STR> is defined, 0 otherwise.


2.3.8 Regular expression based matching
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  '$MATCH{' <PATTERN> <SUBJECT> '}'
  <PATTERN> :: <DQ_STR>
  <SUBJECT> :: <DQ_STR>

PCRE match <SUBJECT> against <PATTERN>.  Return 1 if match succeeded,
return 0 if matching fails.  Please refer the section dedicated to pattern
matching for more information.

2.3.9 Truncating to an integer value
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  '$INT{' <EXPR> '}'

Warning message is issued if conversion is not possible.

2.3.10 In-situ function call expansion
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  '$FUN{' <FUN_NAME> {<FUN_ARG> ' '}* '}'

Any <FUN_PARAM_REF> defined by DEFUN shall be expanded in-situ separated
by a space ' '.

2.3.11 In-situ function call multiple times
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  '$MAP{' <FUN_NAME> {<FUN_ARG> ' '}* '}'

Any <FUN_PARAM_REF> defined by DEFUN shall be expanded in-situ separated
by a space ' '.  <FUN_NAME> is called <FUN_PARAM>s/<FUN_ARG>s times.  The
number of arguments passed in each iteration is <FUN_PARAM>s.

2.3.12 Interleaving arguments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  '$INTERLEAVE{' <INTERLEAVE_COUNT> {<INTERLEAVE_ARG> ' '}* '}'
  <INTERLEAVE_COUNT> :: <DQ_N>
  <INTERLEAVE_ARG> :: <DQ_STR>

2.3.13 Generating a sequence of numbers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  '$SEQ{' <SEQ_FROM> <SEQ_TO> '}'
  <SEQ_FROM> :: <EXPR>
  <SEQ_TO> :: <EXPR>

Produces a sequence of numbers from <SEQ_FROM> to <SEQ_TO> separated 
by spaces.  If <SEQ_FROM> > <SEQ_TO> it counts from <SEQ_TO> down to
<SEQ_FROM>.  At least one number is always output, i.e. $SEQ{0 0}
produces 0.

2.3.14 Split a string into separate tokens
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  '$SPLIT{' <BR_STR> '}'

<BR_STR> is evaluated in exactly the same maner if no $SPLIT was used,
however, the result is split into separate tokens separated by whitespace.


3 Caveats
~~~~~~~~~
3.1 Comments
~~~~~~~~~~~~
While putting comments next to branches is allowed in most cases, it is bad
practice and highly discouraged (consider the SYSTEM command).