1. 28 Sep, 2021 1 commit
  2. 07 Jun, 2020 3 commits
  3. 06 Jun, 2020 2 commits
    • Charles Ferguson's avatar
      Add support for the JunitXML file in RISC OS or unix format. · 0ac50625
      Charles Ferguson authored
      Previously the JUnixXML filename had to be given in native format.
      That's not what the other RISC OS tools expect, so now we try to work
      out which one it is and convert to native.
      0ac50625
    • Charles Ferguson's avatar
      Add support for creating multiple files. · 42a338d2
      Charles Ferguson authored
      It is necessary to test that multiple files (or dirs) were created
      for some operations. This is now supported by the test tool, allowing
      it to check whether the files were created and delete them afterward.
      This doesn't support the operations on the created files like comparisons
      because there are multiple, but it is part of the way.
      42a338d2
  4. 04 May, 2020 1 commit
    • Charles Ferguson's avatar
      Update the test.pl script to handle variable expansions together. · 54f75dd8
      Charles Ferguson authored
      When the variable expansions butted up against one another they would
      not be expanded properly. The escape checking broke the regex, and
      because the version of perl we run with on RISC OS does not support
      the look behind form, we must go to extreme lengths to parse the
      expansion.
      54f75dd8
  5. 29 Apr, 2020 1 commit
  6. 15 Feb, 2020 2 commits
    • Charles Ferguson's avatar
      Add support for declaring what we capture; Improved upgrade tool. · ce5ff198
      Charles Ferguson authored
      The capturing can now declare explicitly that we capture stdout
      and stderr or any combination. This is more useful for those cases
      where we're running something that isn't a C program on RISC OS, but
      even on Unix-like systems, this may help to explicitly ignore
      the stderr output (as I do in Norcroft CC testing).
      ce5ff198
    • Charles Ferguson's avatar
      Add timings in JUnit output; update Arg# to be empty if unused. · 9b6c33f4
      Charles Ferguson authored
      The JUnit XML output now include the durations of the operations
      so that we can see where time has been spent.
      
      The argument numbers are able to substitute as empty, rather than
      substituting as themselves. This makes it possible to include or
      excludes parameters when they are not provided.
      9b6c33f4
  7. 26 Jan, 2020 1 commit
  8. 26 Dec, 2019 5 commits
    • Charles Ferguson's avatar
      Add test cases for the test code. · 696368c7
      Charles Ferguson authored
      Hopefully these will keep me from breaking the basic functionality
      of the tool.
      696368c7
    • Charles Ferguson's avatar
      Revamp the replacements handling to be more sed-like. · ab57ebbe
      Charles Ferguson authored
      In a number of the test expectations it is necessary to look for
      certain lines and only those lines, or to filter out the content
      that we don't care about, and which might change independant of
      what's being tested. Essentially being able to process a simple
      filtering on the output beyond just regular expressions is
      needed for some cases. Specifically this was necessary to ensure
      that Pyromaniac's expectations weren't affected as much by
      unrelated changes in other parts of the sources.
      
      Since invoking `sed` is likely to be much more involved/error
      prone when it comes to RISC OS, and of course the point is to
      make it easy to include testing for a tool without having to
      rely on anything other than perl... and perl can be `sed`.
      
      The replacement scripts are similar in principle to sed, but
      do not work in the complete way that sed scripts do. The format
      is just intended to make the testing comparisons easy and not
      to implement sed.
      
      Supported conditions:
      
      * Line number and range matching.
      * Regular expression matching.
      
      Actions:
      
      * Regular expression replacements.
      * Print lines (fast exit, keeping line)
      * Delete lines (fast exit, skipping line)
      * Quit (complete the script)
      
      Additionally, replacement scripts may include other replacement
      scripts. Don't create infinite loops - it'll end badly.
      
      This appears to work pretty well in the cases that have been
      tested.
      ab57ebbe
    • Charles Ferguson's avatar
      Replace the command substitution to maintain the characters input. · 1533c7b4
      Charles Ferguson authored
      The command substitution wasn't particularly reliable and meant that
      it was commonly necessary to include an extra level of shell escaping
      in the commands supplied, which was entirely against the point of
      writing the commands as they would be expected to be executed on
      RISC OS.
      
      The replacements still haven't been tested on RISC OS, but I think
      they replace a lot of the behaviour that was present previously
      and caused odd effects. The pyromaniac invocation tests are
      significantly simpler now.
      1533c7b4
    • Charles Ferguson's avatar
      Fix for rogue ';' on the end of the CDATA output in JUnitXML. · 70a8c8b2
      Charles Ferguson authored
      The ';' looks like it was a mistakenly input, thinking it was the end
      of a perl line. Oops.
      70a8c8b2
    • Charles Ferguson's avatar
      Add 'Absent' and 'Removes' statements. · 9534a867
      Charles Ferguson authored
      The 'Absent' checks that a file does not exists at the end of the run.
      The 'Removes' checks that a file that has been created by the test
      tool is removed at the end of the run.
      9534a867
  9. 21 Oct, 2019 2 commits
  10. 13 Oct, 2019 2 commits
    • Charles Ferguson's avatar
      Add support for environment variables. · d06b17a4
      Charles Ferguson authored
      Environment variables can now be set within the test and group
      configuration, allowing the tests to configure outside of the
      command line parameters. This probably won't be used often, but
      it gives a nice way to add tests for things that aren't handled
      by the command line.
      d06b17a4
    • Charles Ferguson's avatar
      Add colouring to the test results. · 451e5f60
      Charles Ferguson authored
      When running a number of tests, it's useful to have the failures jump out
      because they're in red. Crashe's appear in purple, to make them stand out
      more.
      451e5f60
  11. 17 Jul, 2019 1 commit
  12. 03 Jul, 2019 1 commit
    • Charles Ferguson's avatar
      Leave variables alone if they have no substitute. · a20da1a1
      Charles Ferguson authored
      Previously it was impossible to use Var$Name because the $N would be
      classed as a variable - without a substitution, and this would cause
      a failure to replace the content.
      
      So it's allowed now. And characters which would be expected to be
      regular RISC OS characters are now escaped if they have a special
      meaning on unix-like systems.
      a20da1a1
  13. 28 Jun, 2019 1 commit
    • Charles Ferguson's avatar
      Add ability to override settings in the test group. · e51f33c7
      Charles Ferguson authored
      Being able to override the settings for the test group means that we
      can provide a much nice interface to some tests - for a couple of cases
      we can remove the defaults, as well as set them to other values, as
      that may matter for some of the tests (eg an Expect on a group could
      be overridden on a couple of places where there is no Expect file).
      e51f33c7
  14. 26 Jun, 2019 1 commit
    • Charles Ferguson's avatar
      Add argument passing for the test.pl script; allow capture replacements. · 07703d94
      Charles Ferguson authored
      We can now pass arbitrary arguments to the scripts, which can use them
      in different ways. This is intended to be used with the templating system
      to perform different types of runs without defining all the parameters
      repeatedly.
      
      Captured values in the regular expressions used for replacements can be
      added to the replacements which allows us to reorder things if we wish.
      07703d94
  15. 25 Jun, 2019 2 commits
    • Charles Ferguson's avatar
      Add support for splitting the test scripts with includes. · 7f933171
      Charles Ferguson authored
      The tests.pl script has been updated so that it can include other
      test scripts. The intention is to make it easier to build up the
      structure of the testing from blocks that test related, or large
      functionality, rather than it all being in one file together.
      7f933171
    • Charles Ferguson's avatar
      Allow comments in replacements file; allow unmodified replacements. · dc59effa
      Charles Ferguson authored
      The replacements file contains a number of different regex's to apply.
      If there are a few of them, adding comments to explain them is very
      useful. Blank lines # prefixed lines are now ignored.
      
      The modifier on the end of the line is optional, but wasn't actually
      being checked as optional.
      dc59effa
  16. 16 Jun, 2019 2 commits
  17. 15 Jun, 2019 1 commit
  18. 14 Jun, 2019 1 commit
  19. 10 Jun, 2019 2 commits
  20. 08 Jun, 2019 2 commits
  21. 28 May, 2019 1 commit
    • Charles Ferguson's avatar
      Add script to upgrade all the uses of the tool in my test code. · b92495ae
      Charles Ferguson authored
      I'm using the test tool in RISC OS builds so it's handy to ensure
      that it's up to date everywhere. This script will upgrade all the
      versions in the known directories so that we are using the same
      version. We try to find a match in our history and then copy the
      latest version to the destination if we find a match. If we don't,
      we report it and then move on - those files can be dealt with
      manually.
      b92495ae
  22. 27 May, 2019 2 commits
    • Charles Ferguson's avatar
      Add binary checker. · bf0e1ce5
      Charles Ferguson authored
      The linker needed to check that the output files were valid, so this
      change creates a checker for binary files. Only words, bytes and strings
      can be checked, but this should be enough for the immediate needs.
      bf0e1ce5
    • Charles Ferguson's avatar
      Initial import of the RISC OS tool testing tool. · 5f7f860c
      Charles Ferguson authored
      The tool will run through a 'tests.txt' description, executing a tool
      and reporting on the output. It is used as part of the RISC OS CI
      to check that the systems have not regressed.
      5f7f860c