GitLab

JUnitXML will now encode the newlines in attributes so that we get
well-formed XML.

The XML Lint parser is now a completely separate configuration which
should allow other parsers to be dropped in there soon.

New python processor. Works faster.

The Bash parser is dog slow for the xmllint JUnit XML generation. This
appears to be the many file manipulations and messing around parsing
the content. On the server it is taking about 15 minutes to process the
1000 errors (down from 50 minutes to process them, after the code was
optimised and 1000 were fixed).

Python setup reports some deprecation warnings that we can safely ignore
for the purposes of the environment setup.

Invocation tests would not produce the right output if the input for
those tests contained spaces at the start of the lines or had escaped
characters in them. This was caused by misused 'read' commands, without
the -r switch, and not clearing IFS beforehand.

Additionally, add the 'dim' colouring option to our 'say' function.

On entering the python environment, the errors are reported more
obviously now when we are already in a virtualenv.

Rather than doing the CDATA inline, this is now handled by a function,
which is a lot tidier.

Replacing some sed operations with bash replacements helps.
Moving around a fast reject so we don't waste time is also helpful.

The timestamp format was not globally set because the function was being
called as a subprocess, so the output wasn't affecting the parent
environment.

Rather than writing out simple strings many times into different files,
we now amalgamate most of those lines into a simpler line in a single
values file. This means that we're not creating lots of additional files
that we don't need to - which hopefully also speeds up the process.

Shell JUnit attributes are now escaped (at least for the common two
that would have special characters in).

The format of the XMLLint messages are now handled better - they were
only being processed when they were the regular parser problems. We
now handle the DTD processing as well.

The XML_CATALOG_FILES mean that we can process the lint operations
on DTDs without requiring network activity. This makes it possible
to use local DTDs without having to rely on external resources.

We need to always process the output from the xml-lint tool to check that
there was nothing reported. In some cases (entities being misused) the
xml-lint tool has been reporting success, when actually it has been
failing to parse (and has been saying so in the output). This is now
reported as a failure.

Reorganising the pylint code appears to have broken it - the result in
the TestCase was not being passed appropriately.

The common JUnitXML library provides almost the same functions as the
original lint processor, so we can remove the code that is common.

When there are problems, the JUnitXML file can report them more nicely,
I hope.

It is possible to detect the languages relatively easily. This can be
used either as a diagnostic, or - as I intend - to allow the CI scripts
to be set up appropriately for the languages that are present.

The extra parameters on the junitxml_result command are now documented.

Timestamp on the TestSuite is apparently required, and the classname
on the TestCase, is also required in some cases. In the latter case,
we just force the classname to be 'lint'.

The error message would have been reported as the literal string
'$message'. Oops.

Although the JUnit XML format is not really that well defined, there
are a few things that seem to be required to make files get processed.
This isn't always in line with the schemas that are out there, but
we have to work with what actually works.

Firstly, the `id` on the testsuite element is set incrementing from
the start, regardless of the original values. This should ensure that
anyone relying on it gets a sane answer if they look up based on those
numbers.

Secondly, where times are present, they are propagated to the top
of the testsuite (as `time` is also a required attribute on some
schemas). This only happens if we can be sure that the times
are given on all the testcases.

Finally, we leave an annotation about the classname being required,
or possibly not, in some cases. We don't force a value as yet, but
that might have to happen in the future.

In order to get a useful number of errors reported for the linting,
we now give a separate testcase for each of the errors that is reported,
or just a dummy test case where no errors are found.

In order to ensure that we always write out a class name, which is
required by some versions of the JUnit schema, we write out the
classname as 'test' if it has not been set. It's not perfect, but
it helps deal with some systems that require it to be present.

The CI tests can now include JUnit XML output to describe the problems
encountered by the linting process. This is just based on the YAML
processor, so it's not particularly complex.

Generation of the XML Junit output was breaking because of UTF-8
not being used for the output in Python 3.

The virtualenv might be broken - for example if the version of
python had been upgraded by brew underneath you. Detecting such a
case and recovering without the user needing to perform surgery
is very useful for the developer, as they just want the environment
to work. That is, I want the system to do what is necessary to make
the command I ran happen.

We now check if the environment is working and if not, remove the
whole thing.

No need to include 'clean' on the end of the version string.

The perl-lint helper will process the perl critic output and
generate Junit XML files.

The Junit output would write out durations or times that were negative
or invalid when there was no duration recorded. This made for some
pretty unpleasant reading for the JunitXML output. These have now
been omitted where they are not relevant.

The output written to stderr was not coming out in the right place,
due to the interleaving. Changing to a stdout is clearer.

On the OSX system, the ci-vars was not reporting successfully for the
paths we were handling.

If a language is not defined, it would previously have defaulted to
enabled. The 'all_enabled' setting allows the default to be switched
to 'false' so that we must explicitly set the enabled setting for
each language, rather than explicitly disable them.

The Pip list output now breaks the behaviour of the uninstaller due
to outputting different messages - in particular the lines '----'
and the headings 'Package', 'Version'. We try to strip these from
the output so that we do not fail the environment setup.

Version numbers for the branch version are now better managed,
resolving to a more sensible set of values when multiple gitflow-like
branches are in use. This should mean that branch testing and the
like will be much more obvious in the future.

I only really use bash, and since the tool has begun complaining about
there being no #! at the start of the bash.d files, I'm just forcing
to bash for now.

Trying to make the libraries a little more obvious.

The language files were called 'lang.<name>', which meant that we
had files that thought they were yaml formatted in the editor. Changing
the language files to be 'lang-<name>.sh' means that they're formatted
properly.

XML linting is now able to be performed with the xmllint tool. This isn't
isolated, so it isn't guaranteed to work, but it's good enough for now.

We also ignore the artifacts directory and the logging directory so that
we can be sure that the outputs of the CI don't interfere with the CI
itself.