Plumhall automated testing
Plumhall C test suite
Plumhall C test suite is now in the automated testing. The results log the fulltest.sum file and the list of skipped or failed tests. Changes to the test suite should be done in the Plumhall repository. Due to the licensing constraints of Plumhall, this repository is only accessible to MIPS employees in the office. Please limit changes to flags.h, envsuite, and scripts in the scripts directory.
Process automated testing uses to run Plumhall
The automated testing builds LLVM and runs various test suites including Plumhall. This process assumes you have a previously built version of LLVM which the automated test process will have already built. Also, this builds Plumhall using little endian, PIC code and the GCC assembler. You'll need to change the parameters passed to the run-plumhall script and the QEMU variables if this is not how you're running Plumhall.
- Configure the environment
base_dir=$(pwd) export CLANG_ALT_DRIVER="$base_dir/clang-alt-driver/src/mips_linux_gnu_gcc" export PLUMHALL_SRC="$base_dir/plumhall/cvs10a_full/cvs10a" export PLUMHALL_DST="$base_dir/plumhall-results" export QEMU="/mips/tools/mips/swpkgs/bin/run-on-qemu" export QEMUFLAGS="-p -EL"
- Checkout Plumhall test suite
git clone -b master git://dmz-portal.mips.com/plumhall.git
- Checkout LLVM/Clang alternate driver
git clone -b master git://dmz-portal.mips.com/llvm_driver.git clang-alt-driver
- Point the LLVM/Clang alternate driver at LLVM build and MIPS GCC directories.
./clang-alt-driver/src/DriverConfig.py -l /path/to/LLVM/build/Debug+Asserts/directory/or/the/install/directory -g /mips/arch/overflow/codesourcery/mips-linux-gnu/pro/release/2011.03-95/Linux
- Run Plumhall
plumhall/scripts/run-plumhall '-mips32r2 -fPIC -EL'
The flags.h file in Plumhall controls which tests are run and which are skipped. A bug should be created in Bugzilla for any tests that are skipped in Plumhall. When the test starts to work, then the skip flag should be removed from Plumhall and the Bug closed. The flags.h file is a C header file. Comments are C style comments. Tests are skipped by using #define and the correct define.
Skipping a test:
- Create a new bug in Bugzilla for the test being skipped
- Checkout the Plumhall repository
- Edit the plumhall/plumhall/cvs10a_full/cvs10a/dst.mips/flags.h file
- Find the SKIP #define for the test in the flags.h file and uncomment it if found. It may be in the file more than once.
- If it isn't in the file, add a #define for it at the bottom of the flags.h file with the other skipped tests.
- Add a comment with a link to the Bugzilla bug number
- Save the flags.h file
- Commit the change to the Plumhall repository
- Push the Plumhall repository back to the server
Plumhall debugging rules
The contract allows us to send parts of the plum hall test suite to outside contractors with the following rules:
- The whole driver can be sent out. In other words, what is in the dst part of it.
- Single test files can be sent and debugged. After resolving the issues, the test files should be deleted.
When a test fails, there are these self contained files that are created by the suite. These single files usually should be all you need.
Generally, send only what is needed to recreate that particular problem.
Be careful with this because the tests are watermarked. That is, each customer that has plum hall gets a slightly different version of it so that each copy can be uniquely identified.
That is, if the source gets out , Plum Hall can possibly figure out that it was our copy.
If a bug needs to be submitted to clang, then a new test needs to be made creatively that does not look at all like the original but causes the bug to happen too.
If this can't be done for some reason or it's too much work, it's possible to file a clang bug and just reference the plum hall test that fails.
Apple has some version of plum hall but does not run it anymore.