Commits (20)
  • Charles Ferguson's avatar
    9299a45c
  • Charles Ferguson's avatar
    Add 'output_filter' function to try to give consistent output. · 7f35d750
    Charles Ferguson authored
    The output from the lint and test might look a little different, but
    it is possible to combine the output so that we get something
    consistent being written out. The output_filter is hoped to do that.
    7f35d750
  • Charles Ferguson's avatar
    Update perl-env-setup to search modules manually. Use perl -f. · 68c0dc70
    Charles Ferguson authored
    The module search used to use a 'use' statement to find the modules
    and then read the '$VERSION' variable. This would allow code to be
    executed (which would be a bad thing), and means that we're at the
    whim of what the module does to '$VERSION'. In this case, the issue
    lies with DBI, which calls 'eval' on the version number, turning
    1.630 into 1.63, which isn't the same thing.
    
    Now, we try to locate the file and extract the version number
    from it as assigned to a variable. If we can do this then all is
    well and we use that. If we cannot, then we fall back to using
    'require', which will then do the code execution, but should at
    least find the file.
    
    The -f option disables the reading of the site customisations,
    which could affect the operation of the code.
    68c0dc70
  • Charles Ferguson's avatar
    Fix the fish translations of the environment syntax. · 07c84e68
    Charles Ferguson authored
    The environment we got back from the 'local::lib' tool is being
    translated into 'fish' commands so that the environment is set up
    properly for the fish sell. However, it was broken. The expressions
    that were given had incorrect escaping, so they ended up not working
    properly, and the regular user environment was used instead.
    
    The sed translations should now be fixed. Needs checking on OSX, but
    it certainly works on Linux.
    07c84e68
  • Charles Ferguson's avatar
    Update the tools used by perl to be defined in the perl script. · 25e127e3
    Charles Ferguson authored
    The tools were fixed installing specific versions, but that's not a
    lot of good if you're trying to allow the environment to be generic.
    The code has been pinched from the Python common code, which allows
    the configuration to be used to set up the versions of the tools
    that are to be used.
    25e127e3
  • Charles Ferguson's avatar
    Try to fix the usage of sed with fish and the exported variables on OSX. · 492b09b0
    Charles Ferguson authored
    On OSX there are different limitations, such as not being able to have
    empty sub-expressions in the regular expressions passed to sed.
    Additionally, having a different version of local::lib installed which
    understands fish, and generates the exports differently broke the
    automated translater. Working with 'any version' is hard, but I think
    we may be nearly there.
    492b09b0
  • Charles Ferguson's avatar
    Small change to the output message to stop warning. · dc648b89
    Charles Ferguson authored
    The ~ character in double quotes was being reported as a warning
    that it would not be expanded, from shellcheck. We know it won't
    be, as we're just printing a message. Putting it in single quotes
    should remove the warning.
    dc648b89
  • Charles Ferguson's avatar
    More updates to fix use of sed, this time on linux. · 3b7c0a3f
    Charles Ferguson authored
    The {} usage on OSX appears to work just fine, but on Linux, it
    reports that the values in the braces are badly formed. This time I
    think it's the GNU tool that's right, and escaping the braces seems
    to fix the issue.
    
    Additionally we handle the escaping of the replaced strings a little
    better.
    3b7c0a3f
  • Charles Ferguson's avatar
    Add support for common directories for artifacts. Merge JUnit files. · c29d75a1
    Charles Ferguson authored
    We now have common directories for the artifacts produced, the logs
    we generate and the environment files that we use. These fixed
    directories mean that an automation system can more readily rely on
    the locations provided.
    
    We have a tidier mechanism for reporting the results in XML, by
    writing multiple files, and then merging them. We could still do a
    lot more with this, but it works pretty well right now.
    
    This change covers all the components, but it has only been tested
    with the Python code really.
    c29d75a1
  • Charles Ferguson's avatar
    Add fixup for nosetests XML output. · 95d2f280
    Charles Ferguson authored
    The XML output for nosetests does not separate the test module from
    the class. A fixup hasa been added to try to make these results more
    useful.
    95d2f280
  • Charles Ferguson's avatar
    Fix for perl test runs. · 551eb49f
    Charles Ferguson authored
    Perl test runs would fail due to problems with the deletion of the
    xml files - of which there were none. Similarly, we didn't support
    version numbers of modules that were not integers, without quotes.
    551eb49f
  • Charles Ferguson's avatar
    Fix for output filter not preserving spaces. · d5bb3678
    Charles Ferguson authored
    The filter was using bare 'read' which was not working with the
    leading spaces. Must use -r and clear IFS for this to work.
    d5bb3678
  • Charles Ferguson's avatar
    Fix for 'if' statement incorrect syntax for 'fish'. · 748146e1
    Charles Ferguson authored
    The fish code that checked for the file needing to be deleted was
    using the bash syntax, not the fish syntax.
    748146e1
  • Charles Ferguson's avatar
    Initial implementation of 'clean' action. · 65a5762c
    Charles Ferguson authored
    The clean action will clear up the project so that there are no files
    left behind which should be removed from the environment. At present
    we only have the CI specific files being removed, but the framework
    for the other languages to be cleaned is also present (unused).
    65a5762c
  • Charles Ferguson's avatar
    Add cleaning of python files to 'clean' action. · 21dc1c91
    Charles Ferguson authored
    When python files are cleaned, we will clear out all the .pyc, .pyo
    files, and any __pycache__ directories.
    21dc1c91
  • Charles Ferguson's avatar
    Add support for expanding to match language files. · 753e8b8d
    Charles Ferguson authored
    We can expand the language names to match the extensions and the
    magic recognition for those files. This means than many definitions
    become a lot easier to set, as we may just specify which language
    files to match.
    753e8b8d
  • Charles Ferguson's avatar
    Update perl and python to use the {language} specification. · 54b6fa03
    Charles Ferguson authored
    The '{language}' specification meanst hat files that don't have an
    extension can still be processed by the tools. This lets us name
    files without extensions in order to make them implementation
    agnostic, and still have them prcessed by the ci system.
    54b6fa03
  • Charles Ferguson's avatar
    Add basic shell file processing. · b9713b17
    Charles Ferguson authored
    Shell files are not normally checked, and the checking we perform
    is only 'shellcheck' but that's fine for us because it gives us
    a simple way to ensure that the scripts will actually do what we
    expect.
    
    However, there is currently no way to restrict the shellcheck
    version to one that is known. This may need to be addressed in
    the future, but for now we use the system installed version.
    b9713b17
  • Charles Ferguson's avatar
    Update the output format so that mixed languages look tidier. · 2f4509bc
    Charles Ferguson authored
    The shell + python output looked very untidy because the indentation
    was very wrong. Really we could do with using one of the exec formats
    to handle the output instead of merely piping outputs. But for now
    this does the job.
    2f4509bc
  • Charles Ferguson's avatar
    Ensure that the 'perl-env' and 'python-env' command exit with the RC. · dc496318
    Charles Ferguson authored
    When the perl-env and python-env tools are executed with a command
    line passed to them, they would not terminate with the correct return
    code. This meant that their use as part of a test environment (although
    I'm not keen on that - they will go away eventually) would not be
    possible.
    
    Fortunately, it's just a matter of returning the correct code immediately
    after the command runs. Or I could just have used 'set -e', but didn't
    in the end.
    dc496318
...@@ -27,6 +27,24 @@ function bool() { ...@@ -27,6 +27,24 @@ function bool() {
return 1 return 1
} }
##
# Trim leading and trailing whitespace from a string.
#
# @param $1 String to trim
function trim() {
local var="$1"
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
echo -n "$var"
}
## ##
# Read a configuration parameter from the project's config. # Read a configuration parameter from the project's config.
# #
...@@ -49,12 +67,12 @@ function config() { ...@@ -49,12 +67,12 @@ function config() {
local key="$1" local key="$1"
local default="${2:-}" local default="${2:-}"
if [[ ! -f "${root}/project.config" ]] ; then if [[ ! -f "${config_file}" ]] ; then
# No configuration file # No configuration file
echo "${default}" echo "${default}"
elif grep -q "^${key}: \?" "${root}/project.config" ; then elif grep -q "^${key}: \?" "${config_file}" ; then
# Key found, so we can return it # Key found, so we can return it
grep "^${key}: \?" "${root}/project.config" | sed "s/^${key}: *//" || true grep "^${key}: \?" "${config_file}" | sed "s/^${key}: *//" || true
else else
# Key not found in configuration file # Key not found in configuration file
echo "${default}" echo "${default}"
...@@ -65,8 +83,66 @@ function config() { ...@@ -65,8 +83,66 @@ function config() {
## ##
# Expand a set of filenames from wildcards. # Expand a set of filenames from wildcards.
# #
# This can be either:
# * A wildcarded filename in shell style.
# Examples: *.py foo/*.py
#
# * A language name in braces to restrict to that type, followed by a directory to search within
# Examples: {python} {python}foo
#
# @param $@ filenames with wildcards # @param $@ filenames with wildcards
#
# @return filename per line on stdout
function expand_filenames() { function expand_filenames() {
local specs="$*"
local lang=''
local dir=''
local exts
local magic
local find_match
local spec
for spec in $specs ; do
if [[ "${spec:0:1}" == '{' ]] ; then
# This is a language specification
lang="${spec:1}"
lang="${lang%%\}*}"
dir="${spec#*\}}"
exts="$(call_if_function "${lang}_match_extensions")"
magic="$(call_if_function "${lang}_match_magic")"
# Construct the specification for find
find_match=()
for ext in $exts ; do
if [[ "${#find_match[@]}" != '0' ]] ; then
find_match+=('-o')
fi
find_match+=('-name' "*.$ext")
done
# List all the requested extensions
if [[ "${#find_match[@]}" != '0' ]] ; then
find "./$dir" -type f "${find_match[@]}" \
| (grep -E -v '(/\.|^\.//*ci/)' || true)
fi
# List all the files matching requested magic
if [[ "${magic}" != '' ]] ; then
if [[ "${#find_match[@]}" != '0' ]] ; then
find_match=('!' '(' "${find_match[@]}" ')')
else
find_match=()
fi
# Find files that do not match the extensions, are not in the
# special or hidden directories, and which match the magic
# output.
find "./$dir" -type f -size -100k "${find_match[@]}" -print0 \
| (grep -E -Z -z -v '(/\.|^\.//*ci/)' || true) \
| xargs -0 file -N -n -F':::' /dev/null \
| sed -E "/${magic}/! d; s/::: .*//"
fi
else
( (
# If nothing matches a wildcard string, return empty # If nothing matches a wildcard string, return empty
shopt -s nullglob shopt -s nullglob
...@@ -74,10 +150,13 @@ function expand_filenames() { ...@@ -74,10 +150,13 @@ function expand_filenames() {
# Note: Cannot use 'shopt -s globstar' to allow '**' to be subdirectory match. # Note: Cannot use 'shopt -s globstar' to allow '**' to be subdirectory match.
# It isn't supported on OSX. # It isn't supported on OSX.
eval echo "$*" eval echo "$spec"
) )
fi
done | sed '/^$/d ; s!//*!/!g ; s!^\./!!' | sort -u
} }
## ##
# Find the tool configuration for a given action. # Find the tool configuration for a given action.
# #
...@@ -173,6 +252,20 @@ function tool_config_expansion() { ...@@ -173,6 +252,20 @@ function tool_config_expansion() {
return 0 return 0
} }
##
# Delete an entire directory tree, as safely as possible.
#
# @param $@ Directories to remove.
function rmtree() {
local dirs="$@"
if [[ "$(uname -s)" == 'Darwin' ]] ; then
# On OSX, '--one-file-system' does not exist.
rm -rf "${dirs[@]}"
else
rm -rf --one-file-system "${dirs[@]}"
fi
}
## ##
# Check if a named function exists # Check if a named function exists
# #
...@@ -199,6 +292,86 @@ function call_if_function() { ...@@ -199,6 +292,86 @@ function call_if_function() {
} }
##
# Report that we have started processing a file.
#
# @param $1 filename being looked at
# @param $2 message to display
function report_file() {
local filename="$1"
local message="$2"
echo "$filename: $message"
}
##
# Report that we have failed for some reason.
#
# @param $1 filename being looked at
# @param $2 message to display
function report_failure() {
local filename="$1"
local message="${2:-FAILED}"
echo "$filename: $message"
}
##
# Report that we have succeeded
#
# @param $1 filename being looked at
# @param $2 message to display
function report_success() {
local filename="$1"
local message="${2:-Success}"
echo "$filename: $message"
}
##
# Operation output filter - a pipe that reports the status of the operations.
#
# @param $1 message for starting output (or '+' to indent with no message)
# @param $2 message for ending output (when something printed)
# @param $3 message for ending output (when nothing printed)
function output_filter() {
local start="$1"
local end_failure="$2"
local end_success="$3"
local line
local nlines=0
local indent
if [[ "$start" == '' ]] ; then
indent=' '
else
indent=' '
fi
while IFS= read -r line ; do
if [[ "$nlines" == '0' ]] ; then
if [[ "$start" != '' && "$start" != '+' ]] ; then
echo " $start"
fi
fi
echo "$indent$line"
nlines=$(( nlines+1 ))
done
if [[ "$nlines" == '0' ]] ; then
if [[ "$end_success" != '' ]] ; then
echo " $end_success"
fi
else
if [[ "$end_failure" != '' ]] ; then
echo " $end_failure"
fi
fi
}
function all_setup_test() { function all_setup_test() {
local running="${1:-tests}" local running="${1:-tests}"
...@@ -232,6 +405,63 @@ function all_setup_docs() { ...@@ -232,6 +405,63 @@ function all_setup_docs() {
docs_total=0 docs_total=0
} }
function all_setup_clean() {
echo '>>> Cleaning code'
}
##
# Merge all the test result files into a single file,
# and clear out the original files.
function merge_test_results() {
local n=0
local test_files=()
local tf
while [[ "$n" -lt "$tests_total" ]] ; do
tf="${artifact_dir}/test-results-$n.xml"
if [[ -f "$tf" ]] ; then
test_files+=("$tf")
fi
n=$(( n + 1 ))
done
# Merge all those files together
if [[ "${#test_files[@]}" == 0 ]]; then
echo "No JUnit XML files generated"
else
echo "Merging ${#test_files[@]} files into '${artifact_dir}/test-results.xml'"
"${scripts}/junit-xml" --fix-nosetests \
--output "${artifact_dir}/test-results.xml" \
"${test_files[@]}"
# And clear up the files that we now no-longer need.
rm "${test_files[@]}"
fi
}
function all_run_clean() {
rmtree "${artifact_dir:-ARTIFACT_DIR_UNSET}"
}
function all_process_tests() {
merge_test_results
}
function all_process_coverage() {
# If the file writing fails, it isn't fatal
merge_test_results || true
}
function all_process_clean() {
# Cleaning the log and artifact directory must come last so that
# if the language needs to do something, it can use the existing
# environment (if it needs to), and write logs (although they
# would be cleared away).
rmtree "${log_dir:-LOG_DIR_UNSET}"
rmtree "${environment_dir:-ENVIRONMENT_DIR_UNSET}"
}
function all_end_test() { function all_end_test() {
if [[ "$tests_failed" != "0" ]] ; then if [[ "$tests_failed" != "0" ]] ; then
...@@ -275,6 +505,11 @@ function all_end_docs() { ...@@ -275,6 +505,11 @@ function all_end_docs() {
exit 0 exit 0
} }
function all_end_clean() {
echo '<<< Cleaned'
exit 0
}
## ##
# Provide help messages. # Provide help messages.
...@@ -444,7 +679,14 @@ function run_action() { ...@@ -444,7 +679,14 @@ function run_action() {
exit 1 exit 1
fi fi
# Ensure the directories we'll use are present and (where necessary)
# cleaned.
rmtree "${artifact_dir:-ARTIFACT_DIR_UNSET}" "${log_dir:-LOG_DIR_UNSET}"
mkdir -p "${environment_dir}" "${artifact_dir}" "${log_dir}"
"all_setup_${action}" "all_setup_${action}"
call_if_function "all_run_${action}"
for lang in $languages ; do for lang in $languages ; do
if active "$lang" "$action" ; then if active "$lang" "$action" ; then
...@@ -465,6 +707,14 @@ function run_action() { ...@@ -465,6 +707,14 @@ function run_action() {
call_if_function "${lang}_process_${action}" call_if_function "${lang}_process_${action}"
fi fi
done done
call_if_function "all_process_${action}"
# Remove any directories we did not use
for dir in "${environment_dir}" "${artifact_dir}" "${log_dir}" ; do
if [[ -d "$dir" ]] ; then
rmdir "$dir" 2> /dev/null || true
fi
done
# Report the results of the action, and call exit appropriately. # Report the results of the action, and call exit appropriately.
"all_end_${action}" "all_end_${action}"
...@@ -478,4 +728,25 @@ for file in $(expand_filenames "${scripts}"'/common.*') ; do ...@@ -478,4 +728,25 @@ for file in $(expand_filenames "${scripts}"'/common.*') ; do
source "${scripts}/common.${file##*.}" source "${scripts}/common.${file##*.}"
done done
actions="test coverage lint docs" actions="test coverage lint docs clean"
# General configuration for all tools.
config_file="${root}/project.config"
# The 'environment' directory is where we store the necessary tools and
# resources to make it possible to isolate the actions.
export environment_dir="${root:-ROOT_UNSET}/$(config environment_dir '.env')"
# The 'artifact' directory is where the output of the action is stored.
# Results of tests, or built objects, would go here.
# This directory might be the content that was preserved.
export artifact_dir="${root:-ROOT_UNSET}/$(config artifact_dir 'artifacts')"
# The 'log' directory is where files related to the action, but not
# its direct output, would be placed.
# Transient logs, and data used as part of the operation, but not its
# direct output, would go here.
# This directory might be the content that would be used for diagnosing
# problems, but not retained once the lifetime of the action had been
# reached.
export log_dir="${root:-ROOT_UNSET}/$(config log_dir 'ci-logs')"
...@@ -5,6 +5,21 @@ ...@@ -5,6 +5,21 @@
set -eo pipefail set -eo pipefail
# Defaults for the tools we shall use
perl_tool_coverage='cover'
perl_tool_lint='perlcritic'
# Default versions of the tools
perl_toolversion_cover='1.25'
perl_toolversion_perlcritic='1.126'
# And the definitions of what we need in requirements
perl_tool_coverage_1_25='Devel::Cover@1.25'
perl_tool_perlcritic_1_126="
PPI@1.220
Perl::Critic@1.126
Perl::Critic::Pulp@93
"
## ##
# Is perl active for this action? # Is perl active for this action?
...@@ -17,7 +32,7 @@ function perl_active() { ...@@ -17,7 +32,7 @@ function perl_active() {
if [[ "$action" == 'test' || "$action" == 'coverage' ]] ; then if [[ "$action" == 'test' || "$action" == 'coverage' ]] ; then
files="$(config -e test_perl_files '*-test.pl')" files="$(config -e test_perl_files '*-test.pl')"
elif [[ "$action" == 'lint' ]] ; then elif [[ "$action" == 'lint' ]] ; then
files="$(config -e lint_perl_files '*.pl *.pm')" files="$(config -e lint_perl_files '{perl}')"
else else
# Unrecognised type # Unrecognised type
return 1 return 1
...@@ -31,6 +46,16 @@ function perl_active() { ...@@ -31,6 +46,16 @@ function perl_active() {
} }
##
# Matching functions for filename expansion.
function perl_match_magic() {
echo -n 'Perl script'
}
function perl_match_extensions() {
echo -n 'pl pm'
}
## ##
# Help messages # Help messages
function perl_help_lint() { function perl_help_lint() {
...@@ -54,8 +79,21 @@ function perl_switch_lint_checks() { ...@@ -54,8 +79,21 @@ function perl_switch_lint_checks() {
# #
# @param $@ CPAN requirements to install (either filenames, or names of modules preceeded by a '+') # @param $@ CPAN requirements to install (either filenames, or names of modules preceeded by a '+')
function perl_environment() { function perl_environment() {
"${scripts}/perl-env-setup" -e "${root}/perllib" "$@" < /dev/null # Taking input from /dev/null means that the input isn't a tty, so we avoid
source "${scripts}/perl-env" -e "${root}/perllib" # printing messages.
"${scripts}/perl-env-setup" -l "${log_dir}/cpan_output.txt" \
-e "${environment_dir}/perllib" "$@" < /dev/null
source "${scripts}/perl-env" -e "${environment_dir}/perllib"
}
function perl_tool_requirements() {
local action="$1"
local reqs
reqs="$(tool_config_expansion perl "$action")"
local req
for req in $reqs ; do
echo -n " +$req"
done
} }
function perl_environment_test() { function perl_environment_test() {
...@@ -66,19 +104,23 @@ function perl_environment_test() { ...@@ -66,19 +104,23 @@ function perl_environment_test() {
} }
function perl_environment_coverage() { function perl_environment_coverage() {
perl_environment +PadWalker@2.2 +Devel::Cover@1.25 \ perl_environment +PadWalker@2.2 \
"${root}/cpan.txt" \ "${root}/cpan.txt" \
"${root}/cpan-test.txt" \ "${root}/cpan-test.txt" \
"${root}/cpan-coverage.txt" \ "${root}/cpan-coverage.txt" \
"${scripts}/cpan-test.txt" \ "${scripts}/cpan-test.txt" \
"${scripts}/cpan-coverage.txt" "${scripts}/cpan-coverage.txt" \
$(perl_tool_requirements coverage)
} }
function perl_environment_lint() { function perl_environment_lint() {
perl_environment +PadWalker@2.2 +Perl::Critic@1.126 +PPI@1.220 \ perl_environment +PadWalker@2.2 \
"${root}/cpan.txt" \ "${root}/cpan.txt" \
"${root}/cpan-test.txt" \
"${root}/cpan-lint.txt" \ "${root}/cpan-lint.txt" \
"${scripts}/cpan-lint.txt" "${scripts}/cpan-test.txt" \
"${scripts}/cpan-lint.txt" \
$(perl_tool_requirements lint)
} }
...@@ -98,18 +140,7 @@ function perl_setup_coverage() { ...@@ -98,18 +140,7 @@ function perl_setup_coverage() {
ignore_options="${ignore_options},+ignore_re,${pattern}" ignore_options="${ignore_options},+ignore_re,${pattern}"
done done
export PERL5OPT="MDevel::Cover=-db,cover_db$ignore_options,-silent,1" export PERL5OPT="MDevel::Cover=-db,${log_dir}/cover_db$ignore_options,-silent,1"
# Clear away anything left.
if [[ "$(uname -s)" == 'Darwin' ]] ; then
# On OSX, '--one-file-system' does not exist.
rm -rf cover_db test_results
else
rm -rf --one-file-system cover_db test_results
fi
# Ensure that the results directory exists.
mkdir test_results
} }
function perl_setup_lint() { function perl_setup_lint() {
...@@ -119,8 +150,7 @@ function perl_setup_lint() { ...@@ -119,8 +150,7 @@ function perl_setup_lint() {
function perl_run_test() { function perl_run_test() {
for test in $(config -e test_perl_files '*-test.pl') ; do for test in $(config -e test_perl_files '*-test.pl') ; do
echo " Test '$test'..." if perl "$test" 2>&1 | output_filter "Test '$test'..." ; then
if perl "$test" | sed 's/^/ /' ; then
tests_passed=$(( tests_passed + 1 )) tests_passed=$(( tests_passed + 1 ))
else else
echo " Test '$test' failed" echo " Test '$test' failed"
...@@ -136,18 +166,23 @@ function perl_run_coverage() { ...@@ -136,18 +166,23 @@ function perl_run_coverage() {
} }
function perl_run_lint() { function perl_run_lint() {
for file in $(config -e lint_perl_files '*.pl *.pm') ; do local file
if ! perl -c "$file" 2>&1 | sed 's/^/ /'; then local message="Compiling Perl files"
for file in $(config -e lint_perl_files '{perl}') ; do
if ! perl -c "$file" 2>&1 | output_filter "$message" ; then
lint_failed=$(( lint_failed+1 )) lint_failed=$(( lint_failed+1 ))
else else
lint_passed=$(( lint_passed+1 )) lint_passed=$(( lint_passed+1 ))
fi fi
message="+"
lint_total=$(( lint_total+1 ))
done done
if ! perlcritic --profile perlcriticrc \ if ! perlcritic --profile "${root}/perlcriticrc" \
--color \ --color \
$(config -e lint_perl_files '*.pl *.pm') \ $(config -e lint_perl_files '{perl}') \
2>&1 | sed 's/^/ /'; then 2>&1 | output_filter "Linting Perl files"; then
lint_failed=$(( lint_failed+1 )) lint_failed=$(( lint_failed+1 ))
else else
lint_passed=$(( lint_passed+1 )) lint_passed=$(( lint_passed+1 ))
...@@ -165,18 +200,18 @@ function perl_process_coverage() { ...@@ -165,18 +200,18 @@ function perl_process_coverage() {
echo '--- Generating HTML coverage report ---' echo '--- Generating HTML coverage report ---'
cover -report html_basic \ cover -report html_basic \
-outputdir test_results/coverage \ -outputdir "${artifact_dir}/perl-coverage" \
+ignore_re 'test-' \ +ignore_re 'test-' \
cover_db > test_results/Coverage.txt "${log_dir}/cover_db" > "${artifact_dir}/perl-coverage.txt"
echo '--- Generating text coverage report ---' echo '--- Generating text coverage report ---'
cover -report text \ cover -report text \
-outputdir test_results/coverage \ -outputdir "${artifact_dir}/perl-coverage" \
+ignore_re 'test-' \ +ignore_re 'test-' \
cover_db > test_results/CoverageFull.txt "${log_dir}/cover_db" > "${artifact_dir}/perl-coveragefull.txt"
# And read the amount of code covered # And read the amount of code covered
coverage_percentage=$(grep Total test_results/Coverage.txt | awk 'END { print $3 }') coverage_percentage=$(grep Total "${artifact_dir}/perl-coverage.txt" | awk 'END { print $3 }')
echo "Perl Coverage: ${coverage_percentage}%" echo "Perl Coverage: ${coverage_percentage}%"
} }
......
...@@ -72,14 +72,20 @@ function python_active() { ...@@ -72,14 +72,20 @@ function python_active() {
if [[ "$action" == 'test' || "$action" == 'coverage' ]] ; then if [[ "$action" == 'test' || "$action" == 'coverage' ]] ; then
files="$(config -e test_python_files '*_test.py')" files="$(config -e test_python_files '*_test.py')"
elif [[ "$action" == 'lint' ]] ; then elif [[ "$action" == 'lint' ]] ; then
files="$(config -e lint_python_files '*.py')" files="$(config -e lint_python_files '{python}')"
elif [[ "$action" == 'docs' ]] ; then elif [[ "$action" == 'docs' ]] ; then
files="$(config -e docs_python_files '*.py')" files="$(config -e docs_python_files '{python}')"
elif [[ "$action" == 'clean' ]] ; then
files="$(config -e docs_python_files '{python}')"
files="$files $(config -e lint_python_files '{python}') "
files="$files $(config -e test_python_files '*_test.py')"
else else
# Unrecognised type # Unrecognised type
return 1 return 1
fi fi
files="$(trim "$files")"
if [[ -n "$files" ]] ; then if [[ -n "$files" ]] ; then
return 0 return 0
fi fi
...@@ -88,6 +94,16 @@ function python_active() { ...@@ -88,6 +94,16 @@ function python_active() {
} }
##
# Matching functions for filename expansion.
function python_match_magic() {
echo -n 'Python script'
}
function python_match_extensions() {
echo -n 'py'
}
## ##
# Help messages # Help messages
function python_help_any() { function python_help_any() {
...@@ -108,9 +124,9 @@ function python_switch_any_3() { ...@@ -108,9 +124,9 @@ function python_switch_any_3() {
# @param $@ Requirements files to install # @param $@ Requirements files to install
function python_environment() { function python_environment() {
# Set up and enter the Virtual env. # Set up and enter the Virtual env.
"${scripts}/python-env-setup" ${PYTHON_SWITCH} -e "${root}/${VENV_DIR}" \ "${scripts}/python-env-setup" ${PYTHON_SWITCH} -e "${environment_dir}/${VENV_DIR}" \
"$@" < /dev/null "$@" < /dev/null
source "${scripts}/python-env" -e "${root}/${VENV_DIR}" source "${scripts}/python-env" -e "${environment_dir}/${VENV_DIR}"
} }
function python_tool_requirements() { function python_tool_requirements() {
...@@ -159,11 +175,6 @@ function python_setup_test() { ...@@ -159,11 +175,6 @@ function python_setup_test() {
function python_setup_coverage() { function python_setup_coverage() {
python_setup_test "$@" python_setup_test "$@"
# Clear out any previous coverage data.
if [ -f .coverage ]; then
rm -f .coverage
fi
} }
function python_setup_lint() { function python_setup_lint() {
...@@ -180,9 +191,10 @@ function python_run_test() { ...@@ -180,9 +191,10 @@ function python_run_test() {
local switches="$*" local switches="$*"
for test in $(config -e test_python_files '*_test.py') ; do for test in $(config -e test_python_files '*_test.py') ; do
echo " Test '$test'..." if "$PYTHON_TOOL" "$test" \
if "$PYTHON_TOOL" "$test" -v $switches \ --with-xunit --xunit-file "${artifact_dir}/test-results-${tests_total}.xml" \
2>&1 | sed 's/^/ /' ; then -v $switches 2>&1 \
| output_filter "Test '$test'..." ; then
tests_passed=$(( tests_passed + 1 )) tests_passed=$(( tests_passed + 1 ))
else else
echo " Test '$test' failed" echo " Test '$test' failed"
...@@ -193,11 +205,18 @@ function python_run_test() { ...@@ -193,11 +205,18 @@ function python_run_test() {
} }
function python_run_coverage() { function python_run_coverage() {
# We need to set the 'COVERAGE_FILE' variable so that the output from the
# coverage is written to the log directory, rather than the root of the
# project. This is also needed when processing the results.
COVERAGE_FILE="${log_dir}/python-coverage" \
python_run_test --with-coverage --cover-branches python_run_test --with-coverage --cover-branches
} }
function python_run_lint() { function python_run_lint() {
if ! pylint --reports no --rcfile pylintrc $(config -e lint_python_files '*.py') ; then if ! pylint --reports no \
--rcfile "${root}/pylintrc" \
$(config -e lint_python_files '{python}') 2>&1 \
| output_filter "Linting Python files" ; then
lint_failed=$(( lint_failed+1 )) lint_failed=$(( lint_failed+1 ))
else else
lint_passed=$(( lint_passed+1 )) lint_passed=$(( lint_passed+1 ))
...@@ -208,7 +227,9 @@ function python_run_lint() { ...@@ -208,7 +227,9 @@ function python_run_lint() {
function python_run_docs() { function python_run_docs() {
if ! "${scripts}/python-build-docs" --initial-tag "$(config version_git_tag none)" \ if ! "${scripts}/python-build-docs" --initial-tag "$(config version_git_tag none)" \
--major-version "$(config version 1.0)" \ --major-version "$(config version 1.0)" \
--paths "$(config docs_python_files '*.py')" ; then --paths "$(config docs_python_files '{python}')" \
--output-dir "${artifact_dir}/python-docs" \
| output_filter ; then
docs_failed=$(( docs_failed+1 )) docs_failed=$(( docs_failed+1 ))
else else
docs_passed=$(( docs_passed+1 )) docs_passed=$(( docs_passed+1 ))
...@@ -216,6 +237,39 @@ function python_run_docs() { ...@@ -216,6 +237,39 @@ function python_run_docs() {
docs_total=$(( docs_total+1 )) docs_total=$(( docs_total+1 ))
} }
function python_run_clean() {
local files=''
local lastdir=''
local thisdir
files="$(config -e docs_python_files '*.py')"
files="$files $(config -e lint_python_files '{python}') "
files="$files $(config -e test_python_files '*_test.py')"
lastdir=''
for file in $files ; do
file="./$file"
if [[ "${file: -3}" == '.py' ]] ; then
if [[ -f "${file//.py/.pyc}" ]] ; then
rm "${file//.py/.pyc}"
fi
if [[ -f "${file//.py/.pyo}" ]] ; then
rm "${file//.py/.pyo}"
fi
# Only check each directory once.
if [[ "${file:0:${#lastdir} + 1}" != "${lastdir}/" ]] || \
[[ "${file:${#lastdir} + 1}" =~ / ]] ; then
thisdir="$(dirname "$file")"
if [[ -d "${thisdir}/__pycache__" ]] ; then
rmtree "${thisdir:-THISDIR_NOT_SET}/__pycache__"
fi
lastdir="$thisdir"
fi
fi
done
}
function python_process_test() { function python_process_test() {
: # Nothing to do : # Nothing to do
...@@ -233,8 +287,22 @@ function python_process_coverage() { ...@@ -233,8 +287,22 @@ function python_process_coverage() {
ignore_options="${ignore_options},${pattern}" ignore_options="${ignore_options},${pattern}"
done done
coverage_percentage=$(coverage report --omit "${ignore_options}" \ # Generate the text report
| tail -1 \ COVERAGE_FILE="${log_dir}/python-coverage" \
coverage report --omit "${ignore_options}" \
> "${artifact_dir}/python-coverage.txt"
# And the HTML report
COVERAGE_FILE="${log_dir}/python-coverage" \
coverage html --dir "${artifact_dir}/python-coverage" \
--omit "${ignore_options}"
# And the XML report
COVERAGE_FILE="${log_dir}/python-coverage" \
coverage xml -o "${artifact_dir}/coverage.xml" \
--omit "${ignore_options}"
coverage_percentage=$(tail "${artifact_dir}/python-coverage.txt" \
| sed -n 's/^.*[^0-9]\([0-9][0-9\.]*\)%.*$/\1/ p' \ | sed -n 's/^.*[^0-9]\([0-9][0-9\.]*\)%.*$/\1/ p' \
| awk 'END { print int($1) }') | awk 'END { print int($1) }')
echo "Python coverage: ${coverage_percentage}%" echo "Python coverage: ${coverage_percentage}%"
......
#!/bin/false /bin/bash
##
# Common functions used by the Shell code.
#
set -eo pipefail
##
# Is shell active for this action?
#
# @param $1 the action we are attempting to perform
function shell_active() {
local action="$1"
local files=''
if [[ "$action" == 'test' ]] ; then
return 1
elif [[ "$action" == 'lint' ]] ; then
files="$(config -e lint_shell_files '{shell}')"
elif [[ "$action" == 'coverage' ]] ; then
return 1
elif [[ "$action" == 'docs' ]] ; then
return 1
elif [[ "$action" == 'clean' ]] ; then
return 1
else
# Unrecognised type
return 1
fi
files="$(trim "$files")"
if [[ -n "$files" ]] ; then
return 0
fi
return 1
}
##
# Matching functions for filename expansion.
function shell_match_magic() {
echo -n 'Shell script'
}
function shell_match_extensions() {
echo -n 'sh'
}
function shell_setup_lint() {
# Do we have shelllcheck installed
if ! shellcheck -V > /dev/null 2>&1 ; then
echo "ShellCheck is not installed" >&1
exit 1
fi
}
function shell_run_lint() {
echo " Linting Shell files"
for file in $(config -e lint_shell_files '{shell}') ; do
if ! shellcheck "$file" 2>&1 \
| output_filter " $(report_failure "$file")"; then
lint_failed=$(( lint_failed+1 ))
else
report_success " $file"
lint_passed=$(( lint_passed+1 ))
fi
lint_total=$(( lint_total+1 ))
done
}
#!/usr/bin/env python
"""
Process the JUnit XML files.
The intention of this script is to take number of JUnit XML script containing an
arbitrary number of test suites each, and put them all together in a single
file.
"""
import argparse
import sys
import xml.etree.ElementTree as ET
import copy
def expect_int(value):
"""
Expect an integer to be passed, but if it isn't, we'll return None rather than
generate exceptions.
"""
if value is not None:
try:
value = int(value)
except ValueError:
value = None
return value
def expect_float(value):
"""
Expect an float to be passed, but if it isn't, we'll return None rather than
generate exceptions.
"""
if value is not None:
try:
value = float(value)
except ValueError:
value = None
return value
def sum_or_none(iterable):
"""
Expect to be able to sum the values, but return 'None' if all the values were 'None'.
"""
total = None
try:
for value in iterable:
if value is not None:
if total is None:
total = value
else:
total += value
except ValueError:
total = None
return total
class TestXML(object):
def __init__(self, xmlfile=None):
if xmlfile is not None:
tree = ET.parse(xmlfile)
root = tree.getroot()
if root.tag == 'testsuites':
testsuites = root.findall('./testsuite')
self.suites = [TestSuite(suite_node) for suite_node in testsuites]
elif root.tag == 'testsuite':
self.suites = [TestSuite(root)]
else:
# Cannot interpret it, so giving up
self.suites = None
else:
self.suites = []
def __add__(self, other):
if not isinstance(other, TestXML):
raise TypeError("Cannot add a TextXML to a '%r'" % (other,))
new = TestXML(None)
new.suites = copy.deepcopy(self.suites)
new.suites.extend(other.suites)
return new
def __iadd__(self, other):
if not isinstance(other, TestXML):
raise TypeError("Cannot add a TextXML to a '%r'" % (other,))
self.suites.extend(other.suites)
return self
def xml(self):
root = ET.Element('testsuites')
# pylint: disable=C0326
n_tests = sum_or_none(suite.n_tests for suite in self.suites)
n_disabled = sum_or_none(suite.n_disabled for suite in self.suites)
n_skip = sum_or_none(suite.n_skip for suite in self.suites)
n_errors = sum_or_none(suite.n_errors for suite in self.suites)
n_failures = sum_or_none(suite.n_failures for suite in self.suites)
if n_tests is not None:
root.set('tests', str(n_tests))
if n_disabled is not None:
root.set('disabled', str(n_disabled))
if n_errors is not None:
root.set('errors', str(n_errors))
if n_skip is not None:
root.set('skipped', str(n_skip))
if n_failures is not None:
root.set('failures', str(n_failures))
for suite in self.suites:
root.append(suite.xml())
return ET.ElementTree(root)
class TestSuite(object):
"""
<testsuite name="" <!-- Full (class) name of the test for non-aggregated testsuite documents.
Class name without the package for aggregated testsuites documents. Required -->
tests="" <!-- The total number of tests in the suite, required. -->
disabled="" <!-- the total number of disabled tests in the suite. optional -->
errors="" <!-- The total number of tests in the suite that errored. An errored test is one that had
an unanticipated problem, for example an unchecked throwable; or a problem with the
implementation of the test. optional -->
failures="" <!-- The total number of tests in the suite that failed. A failure is a test which the code
has explicitly failed by using the mechanisms for that purpose. e.g., via an
assertEquals. optional -->
hostname="" <!-- Host on which the tests were executed. 'localhost' should be used if the hostname
cannot be determined. optional -->
id="" <!-- Starts at 0 for the first testsuite and is incremented by 1 for each following
testsuite -->
package="" <!-- Derived from testsuite/@name in the non-aggregated documents. optional -->
skipped="" <!-- The total number of skipped tests. optional -->
time="" <!-- Time taken (in seconds) to execute the tests in the suite. optional -->
timestamp="" <!-- when the test was executed in ISO 8601 format (2014-01-21T16:17:18).
Timezone may not be specified. optional -->
>
"""
def __init__(self, suite):
self.name = suite.attrib.get('name', None)
self.package = suite.attrib.get('package', None)
self.hostname = suite.attrib.get('hostname', None)
# id is actually a string
self.id = suite.attrib.get('id', None)
self.time = expect_float(suite.attrib.get('time', None))
self.timestamp = suite.attrib.get('timestamp', None)
# PHPunit also reports 'file'
self.file = suite.attrib.get('file', None)
# Counts, which might not be specified
self.n_tests = expect_int(suite.attrib.get('tests', None))
self.n_disabled = expect_int(suite.attrib.get('disabled', None))
self.n_errors = expect_int(suite.attrib.get('errors', None))
self.n_failures = expect_int(suite.attrib.get('failures', None))
# nose xunit generates 'skip'
# specs I've seen use 'skipped'
# nose2 junitxml documentation says it generates 'skips'
# nose2 junitxml implementation uses 'skipped'
# mocha junit generates 'skipped'
# tap-xunit generates 'skipped'
# xcpretty does not output skipped counts
# xmlrunner unittest xml reporting generates 'skipped'
self.n_skip = expect_int(suite.attrib.get('skip', suite.attrib.get('skipped', None)))
# PHPunit reports assertions
self.n_assertions = expect_int(suite.attrib.get('assertions', None))
self.testcases = suite.findall('./testcase')
self.testcases = copy.deepcopy(self.testcases)
if self.n_tests is None:
# We don't know the number of tests, so we will need to count them
# And we cannot determine which tests were disabled as there is no
# indication as far as can be seen.
self.n_tests = (self.n_disabled or 0) + len(self.testcases)
if self.n_errors is None:
# We don't know the number of errors, so we will need to count them
self.n_errors = len(suite.findall('./testcase/error'))
if self.n_skip is None:
# We don't know the number of errors, so we will need to count them
self.n_skip = len(suite.findall('./testcase/skipped'))
if self.n_failures is None:
# We don't know the number of errors, so we will need to count them
self.n_failures = len(suite.findall('./testcase/failure'))
self.system_out = suite.find('./system-out')
if self.system_out is not None:
self.system_out = copy.deepcopy(self.system_out)
self.system_err = suite.find('./system-err')
if self.system_err is not None:
self.system_err = copy.deepcopy(self.system_err)
self.properties = suite.find('./properties')
if self.properties is not None:
self.properties = copy.deepcopy(self.properties)
def xml(self):
suite = ET.Element('testsuite')
if self.n_tests is not None:
suite.set('tests', str(self.n_tests))
if self.n_skip is not None:
suite.set('skipped', str(self.n_skip))
if self.n_errors is not None:
suite.set('errors', str(self.n_errors))
if self.n_disabled is not None:
suite.set('disabled', str(self.n_disabled))
if self.n_failures is not None:
suite.set('failures', str(self.n_failures))
if self.n_assertions is not None:
suite.set('assertions', str(self.n_assertions))
if self.name is not None:
suite.set('name', self.name)
if self.file is not None:
suite.set('file', self.file)
if self.package is not None:
suite.set('package', self.package)
if self.hostname is not None:
suite.set('hostname', self.hostname)
if self.id is not None:
suite.set('id', self.id)
if self.time is not None:
suite.set('time', str(self.time))
if self.timestamp is not None:
suite.set('timestamp', self.timestamp)
if self.properties is not None:
suite.append(self.properties)
# NOTE: In the testcases, 'classname' is the field that is in the specification.
# PHPunit uses 'class'.
suite.extend(self.testcases)
if self.system_out is not None:
suite.append(self.system_out)
if self.system_err is not None:
suite.append(self.system_err)
return suite
def fix_nosetests(txml):
"""
The 'nosetests' tool writes out a file with the 'testsuite.name' set to 'nosetests',
and the 'testcase.classname' set to '<package>.<class>'. This function attempts to
fix up such cases, so that 'testsuite.name' is the package, and 'testcase.classname'
is the class alone.
"""
for suite in txml.suites:
if suite.name == 'nosetests':
# The fix could be applied.
name_is_common = None
test_package = None
for case in suite.testcases:
# at present the testcases are actually still XML Nodes.
classname = case.attrib.get('classname', None)
if classname and '.' in classname:
tname, cname = classname.split('.', 1)
if not test_package:
test_package = tname
name_is_common = True
else:
if test_package != tname:
name_is_common = False
break
if name_is_common:
# We can make the replacement.
for case in suite.testcases:
classname = case.attrib.get('classname', None)
if classname and '.' in classname:
tname, cname = classname.split('.', 1)
case.attrib['classname'] = cname
suite.name = test_package
def main():
usage = "junit-xml <options> {<junit-files>}*"
parser = argparse.ArgumentParser(usage=usage,
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--output', type=str,
help="Specify the JUnit output file", default='test-results.xml')
parser.add_argument('--fix-nosetests',
help="Move the package name to the test suite", default=False, action='store_true')
parser.add_argument('files', nargs='+',
help='JUnit files to merge together')
options = parser.parse_args()
txml = TestXML()
for xmlfile in options.files:
txml += TestXML(xmlfile)
if options.fix_nosetests:
fix_nosetests(txml)
xml = txml.xml()
xml.write(options.output, encoding='UTF-8', xml_declaration=True)
return 0
if __name__ == '__main__':
sys.exit(main())
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
# #
_PERLLIB_SHELL=false _PERLLIB_SHELL=false
_PERLLIB_ENVDIR=perllib _PERLLIB_ENVDIR=".env/perllib"
_PERLLIB_COMMAND=() _PERLLIB_COMMAND=()
function _perllib_parse() { function _perllib_parse() {
...@@ -56,6 +56,8 @@ else ...@@ -56,6 +56,8 @@ else
# Commands were given; execute them in the environment # Commands were given; execute them in the environment
source "${_PERLLIB_ENVDIR}/bin/activate_perllib" source "${_PERLLIB_ENVDIR}/bin/activate_perllib"
"${_PERLLIB_COMMAND[@]}" "${_PERLLIB_COMMAND[@]}"
# And terminate with the return code
exit $?
elif "$_PERLLIB_SHELL" ; then elif "$_PERLLIB_SHELL" ; then
echo "Dropping to shell" echo "Dropping to shell"
# Commands were not given, but a shell was requested # Commands were not given, but a shell was requested
...@@ -74,7 +76,7 @@ if [ "\$__PERLLIB_MAGIC" = "$__magic_value" ] ...@@ -74,7 +76,7 @@ if [ "\$__PERLLIB_MAGIC" = "$__magic_value" ]
rm "\$me" rm "\$me"
else else
set -l hours_ago (math \\( (date +%s) - (stat --format '%Z' "\$me") \\) / \\( 60 \\* 60 \\)) set -l hours_ago (math \\( (date +%s) - (stat --format '%Z' "\$me") \\) / \\( 60 \\* 60 \\))
if test \$hours_ago -ge 2 ; then if test \$hours_ago -ge 2
# This script hasn't been used for over 2 hours - something has # This script hasn't been used for over 2 hours - something has
# gone wrong, and it's not going to be invoked. It should really # gone wrong, and it's not going to be invoked. It should really
# have been run within moments. # have been run within moments.
......
...@@ -16,6 +16,9 @@ set -eo pipefail ...@@ -16,6 +16,9 @@ set -eo pipefail
# This is a Camel symbol, to indicate this is Perl. # This is a Camel symbol, to indicate this is Perl.
environment_icon="🐪 " environment_icon="🐪 "
# The logfile we will write to
log_file='output_cpan.txt'
## ##
# Convert a given path into one that starts from the root. # Convert a given path into one that starts from the root.
...@@ -75,6 +78,7 @@ Optional arguments: ...@@ -75,6 +78,7 @@ Optional arguments:
-v verbose; be more noisy about what is being done -v verbose; be more noisy about what is being done
-e <env-dir> path to the directory in which to create the -e <env-dir> path to the directory in which to create the
environment (default 'perllib') environment (default 'perllib')
-l <log-file> path to the file into which the log is written
-h show this help message and exit -h show this help message and exit
Positional arguments: Positional arguments:
...@@ -113,11 +117,39 @@ function check_module() { ...@@ -113,11 +117,39 @@ function check_module() {
debug "Checking for '$module' @ '$version'" debug "Checking for '$module' @ '$version'"
if ! perl -e "use ${module}; if ! perl -f -e "# Find the module
my \$leaf = '${module}.pm';
\$leaf =~ s/::/\\//g;
my \$filename;
# Try to find the file by name first - this has less
# risk in executing code, and may be faster.
for my \$dir (@INC) {
if (-r \"\$dir/\$leaf\") {
\$filename = \"\$dir/\$leaf\";
last;
}
}
my \$modversion;
if (defined \$filename) {
if (open(my \$fh, '<', \$filename)) {
my \$nlines = 0;
while (<\$fh>) {
if (/\\\$VERSION *= *(?:[\"']([0-9\\.]+)[\"']|([0-9\.]+))/) {
\$modversion = \$1 // \$2;
last;
}
last if (\$nlines++ > 100);
}
}
}
if (!defined \$modversion) {
require ${module};
\$modversion = \$${module}::VERSION;
}
if ('${version}' ne '' && if ('${version}' ne '' &&
\$${module}::VERSION ne '${version}') { \"\$modversion\" ne '${version}') {
die \"Version should be ${version}, not \$${module}::VERSION\\n\"; die \"Version should be ${version}, not \$modversion\\n\";
}" > /dev/null 2>&1 ; then }" > /dev/null ; then
return 1 return 1
fi fi
...@@ -183,7 +215,7 @@ function install_requirements() { ...@@ -183,7 +215,7 @@ function install_requirements() {
# to complete. # to complete.
local still_sorting=true local still_sorting=true
local iterations=20 local iterations=20
while $still_sorting && [ "$iterations" -gt 0 ] ; do while $still_sorting && [ "$iterations" -gt 0 ] && [ "${#required_defs[@]}" != 0 ] ; do
still_sorting=false still_sorting=false
local -a new_required_defs local -a new_required_defs
new_required_defs=() new_required_defs=()
...@@ -226,29 +258,29 @@ function install_requirements() { ...@@ -226,29 +258,29 @@ function install_requirements() {
debug " Now required defs: ${required_defs[@]}" debug " Now required defs: ${required_defs[@]}"
done done
echo "Ready to install CPAN modules:" >> output_cpan.txt echo "Ready to install CPAN modules:" >> "${log_file}"
for module in "${required_defs[@]}" ; do for module in "${required_defs[@]}" ; do
echo " $module" >> output_cpan.txt echo " $module" >> "${log_file}"
done done
echo $'\nInstalling modules...\n' >> output_cpan.txt echo $'\nInstalling modules...\n' >> "${log_file}"
if [ "${#required_defs[@]}" != '0' ] ; then if [ "${#required_defs[@]}" != '0' ] ; then
if ! cpanm --skip-satisfied -n "${required_defs[@]}" >> output_cpan.txt 2>&1 ; then if ! cpanm --skip-satisfied -n "${required_defs[@]}" >> "${log_file}" 2>&1 ; then
echo "CPAN installation failed" >&2 echo "CPAN installation failed" >&2
sed 's/^/ /' < output_cpan.txt sed 's/^/ /' < "${log_file}"
exit 1 exit 1
fi fi
fi fi
echo $'\nChecking installation...\n' >> output_cpan.txt echo $'\nChecking installation...\n' >> "${log_file}"
failed=false failed=false
for module in "${required_defs[@]}" ; do for module in "${required_defs[@]}" ; do
if ! check_module "${module}" ; then if ! check_module "${module}" 2>> "${log_file}"; then
echo " $module was not installed properly" >> output_cpan.txt echo " $module was not installed properly" >> "${log_file}"
echo "ERROR: $module was not installed properly" >&2 echo "ERROR: $module was not installed properly" >&2
failed=true failed=true
else else
echo " $module ok" >> output_cpan.txt echo " $module ok" >> "${log_file}"
fi fi
done done
...@@ -256,11 +288,11 @@ function install_requirements() { ...@@ -256,11 +288,11 @@ function install_requirements() {
echo "Environment setup failed." >&2 echo "Environment setup failed." >&2
exit 1 exit 1
fi fi
echo $'\nInstalled ok!' >> output_cpan.txt echo $'\nInstalled ok!' >> "${log_file}"
} }
# Parse the command line # Parse the command line
while getopts ":he:v" opt ; do while getopts ":he:l:v" opt ; do
case $opt in case $opt in
v) v)
debug=true debug=true
...@@ -268,6 +300,9 @@ while getopts ":he:v" opt ; do ...@@ -268,6 +300,9 @@ while getopts ":he:v" opt ; do
e) e)
environment="$(abspath "$OPTARG")" environment="$(abspath "$OPTARG")"
;; ;;
l)
log_file="$(abspath "$OPTARG")"
;;
h) h)
usage usage
exit exit
...@@ -278,7 +313,7 @@ done ...@@ -278,7 +313,7 @@ done
# Our temporary directory # Our temporary directory
tmpdir="$(mktemp -d -t perl-env.XXXXXXXX)" tmpdir="$(mktemp -d -t perl-env.XXXXXXXX)"
if [ "$?" != '0' -o "$tmpdir" == '' ] ; then if [[ "$?" != '0' || "$tmpdir" == '' ]] ; then
echo "Cannot create temporary directory. That would be bad." >&2 echo "Cannot create temporary directory. That would be bad." >&2
exit 1 exit 1
fi fi
...@@ -298,7 +333,7 @@ tmpinput="${tmpdir}/inputrequirements.txt" ...@@ -298,7 +333,7 @@ tmpinput="${tmpdir}/inputrequirements.txt"
# Determine the modules we're going to install # Determine the modules we're going to install
echo > "${tmpinput}" echo > "${tmpinput}"
if [ -f "${root}/cpan.txt" ] ; then if [[ -f "${root}/cpan.txt" ]] ; then
cat "${root}/cpan.txt" >> "${tmpinput}" cat "${root}/cpan.txt" >> "${tmpinput}"
fi fi
for file in "${@:$OPTIND}" ; do for file in "${@:$OPTIND}" ; do
...@@ -315,7 +350,7 @@ sed -E "s/^#.*//; /^ *$/d;" "${tmpinput}" \ ...@@ -315,7 +350,7 @@ sed -E "s/^#.*//; /^ *$/d;" "${tmpinput}" \
# Ensure that our environment is set up properly # Ensure that our environment is set up properly
mkdir -p "${environment}" mkdir -p "${environment}"
eval "$(perl -M"local::lib=${environment}")" eval "$(SHELL=/bin/bash perl -f -M"local::lib=${environment}" 2> /dev/null)"
if [ $? != 0 ] ; then if [ $? != 0 ] ; then
echo "Failed to create local::lib environment" echo "Failed to create local::lib environment"
exit 1 exit 1
...@@ -332,16 +367,20 @@ while [[ "$environment_name" == 'perllib' || ...@@ -332,16 +367,20 @@ while [[ "$environment_name" == 'perllib' ||
done done
echo "CPAN output" > output_cpan.txt echo "CPAN output" > "${log_file}"
# Ensure that we have a version of CPAN # Ensure that we have a version of CPAN
if ! check_module App::cpanminus ; then if ! check_module App::cpanminus 2>> "${log_file}" ; then
echo "==== cpanminus install ====" >> output_cpan.txt echo "==== cpanminus install ====" >> "${log_file}"
cpan -T App::cpanminus >> output_cpan.txt 2>&1 cpan -T App::cpanminus >> "${log_file}" 2>&1
fi
if ! check_module ExtUtils::MakeMaker@7.10 2>> "${log_file}" ; then
echo "==== ExtUtils::MakeMaker install ====" >> "${log_file}"
cpanm -n ExtUtils::MakeMaker@7.10 >> "${log_file}" 2>&1
fi fi
if ! check_module Module::Build@0.4222 ; then if ! check_module Module::Build@0.4222 2>> "${log_file}" ; then
echo "==== Module::Build install ====" >> output_cpan.txt echo "==== Module::Build install ====" >> "${log_file}"
cpanm -n Module::Build@0.4222 >> output_cpan.txt 2>&1 cpanm -n Module::Build@0.4222 >> "${log_file}" 2>&1
fi fi
debug "Requirements from input:" debug "Requirements from input:"
...@@ -420,7 +459,7 @@ function _perllib_setup() { ...@@ -420,7 +459,7 @@ function _perllib_setup() {
local oldval local oldval
local tmpfile="$(mktemp -t perllib.XXXXXXXX)" local tmpfile="$(mktemp -t perllib.XXXXXXXX)"
SHELL=/bin/bash perl -Mlocal::lib="$PERLLIB_ENV" > "${tmpfile}" SHELL=/bin/bash perl -f -Mlocal::lib="$PERLLIB_ENV" > "${tmpfile}"
for var in "${PERLLIB_ENVVARS[@]}" ; do for var in "${PERLLIB_ENVVARS[@]}" ; do
oldvar="_OLD_PERLLIB_$var" oldvar="_OLD_PERLLIB_$var"
...@@ -509,7 +548,7 @@ function _perllib_setup ...@@ -509,7 +548,7 @@ function _perllib_setup
set -l oldval set -l oldval
set -l tmpfile (mktemp -t perllib.XXXXXXXX) set -l tmpfile (mktemp -t perllib.XXXXXXXX)
env SHELL=/bin/bash perl -Mlocal::lib="$PERLLIB_ENV" > "$tmpfile" env SHELL=/bin/bash perl -f -Mlocal::lib="$PERLLIB_ENV" > "$tmpfile"
for var in $PERLLIB_ENVVARS for var in $PERLLIB_ENVVARS
set oldvar "_OLD_PERLLIB_$var" set oldvar "_OLD_PERLLIB_$var"
...@@ -522,12 +561,21 @@ function _perllib_setup ...@@ -522,12 +561,21 @@ function _perllib_setup
end end
if [ "$var" = 'PATH' ] if [ "$var" = 'PATH' ]
eval (sed -E '/^\(export \|\)'$var'=/ ! d; # The path variable needs to have the colons translated to spaces,
s/^\(export \|\)'$var'="\(.*\)"\(;\|\)/set -gx "$var" \2/; # and the path parameters as a whole un-quoted, as the PATH is
s/:/ /g' "$tmpfile") # an array in fish.
eval (sed -E '/^(export )?'$var'=/ ! d;
s/export '$var';//g;
s/^(export )?'$var'="(.*)";?/set -gx "'$var'" \2/;
s/\$\{'$var':\+:\$\{'$var'\}\}/'":\$$var"'/;
s/:/ /g;' "$tmpfile")
else else
eval (sed -E '/^\(export \|\)'$var'=/ ! d; # Non-PATH variables only need to be translated to the format
s/^\(export \|\)'$var'="\(.*\)"\(;\|\)/set -gx "$var" "\2"/;' "$tmpfile") # with quotes around the string, and colons retained.
eval (sed -E '/^(export )?'$var'=/ ! d;
s/export '$var';//g;
s/^(export )?'$var'="(.*)";?/set -gx "'$var'" "\2"/;
s/\$\{'$var':\+:\$\{'$var'\}\}/'":\$$var"'/;' "$tmpfile")