Compiler Test Plan
What we refer to as the LLVM compiler is actually two projects: Clang - a C/C++ front end, LLVM a generic machine language compiler. It has been possible in the past to even use LLVM with the gcc front end but this is not currently a supported optionfor MIPS.
Bringing up the compiler
The first step in bringup the new LLVM compiler is to make sure that it is as bug free as we can make reasonably make it.
This requires a lot of testing.
We should never let customers find bugs that we could have found but didn't because we were too lazy to test. Finding these bugs later in customer code is very difficult and costly in terms of management, customer support and developer resources. Given the large amount of C/C++ code already available, it should be possible to head off the majority problems before customers ever find them.
Compiler bugs generally eat the entire development schedule if proper testing is not done. The most senior people will be occupied with 90% of their time tending to this and forward progress on the compiler stops.
Gcc has been in development for over 25 years now and while it has many shortcomings in terms of modern compiler technology, it "works" and probably 95% of the C/C++ code in he world (except for windows platforms) is built with it. This includes linux, android, mac os, iphone os, etc.
We need to be as solid as gcc.
There are two major parts to this bring up process, bringing up the C and C++ compilers.
For the C++ part of this we work, we are hiring a contractor in Russia, Simon Atanysyan, that I have worked with in the past when he was employed by Sun. The C++ part is very complicated to bring up and get right and it's virtually impossible to find someone with his skill set. I have been working with Paul Dutton in legal to set this up but we now have a way to do this and are finalizing the contract with Kelly Services in Russia. Simon will have other duties as well which will include Clang maintenace in some cases.
Another part of this matrix is the various testing that can be done.
1. Clang/LLVM has builtin test suites. There are two of these, a small and a full test. We pass most of this now and what is left are bugs in Clang and not in our compiler. All bugs have been filed with Clang.
2. Commerical test suites. For the most part, these test for language conformance but some like ACE cover more complex situations. We currently have the Plum hall C suite and have tested the compiler with that it fixed bugs/reported issues found. We are in the process of purchasing the C++ and C++ library suites. I have been working with Paul Dutton to rewrite the agreements with Plum Hall so that they are more practical for our current work setup, especially with contractors in different countries etc. These suites typically have very strict rules as to how they can be used and by whom. We now have a workable contract that Plum Hall agrees with and we are finalizing it and hope to have something for Ravi to sign in a few days.
We have budgetted to purchase the Nullstone test suite, which is specifically for optimization and will purchase this next.
We have budgetted to reactivate our license for Perenial test suite but will not make the purchase until we have time to go through it. The suites are generally licensed with updates so there is no point in starting that clock ticking before we need to. I managed to get us reactiviated without having to pay the full license fees.
There are several more test suite products I am interested to buy but there was not enough budget to make those purchases this year:
ACE C/C++ test suites (this is not only a conformance but also a stress
test ala GCC dejagnu)
Modena Test++, Opt++ suites Glen McCluskey C++0x, C++ and C99 test suites. CodeSourcery C++ ABI Test Suite
You can't overtest a compiler. Even if a suite finds one bug, it can pay for itself several times over and maybe even be the reason for some big sale being made.
Bugs are not only code that executes incorrectly but can be regressions in space or speed.
We should really buy these additional suites as soon as we have finished passing those we are already budgeted for.
3. GCC DejaGnu test suite.
This is a very difficult set of 60,000 tests that cover both C and C++. This is from 20 years of gcc bugs and compiler testing. Even for a gcc port this is a difficult suite to pass but is even more complex for a different compiler to pass it. The suite not only addresses many compiler issues but serves to define and test many gcc extensions which we need to support.
We have run the c/gcc and c++/g++ tests.
We have gone through the gcc suite results and fixed and/or addressed urgent issues. RT/RK and Simon will be working on resolving issues.
Some issues are Clang issues, some our MIPS LLVM bugs, some generic LLVM bugs, some complex C/C++/Gcc extension issues.
4. BOOST. As soon as we can pass the plum hall C++ tests we need to try to pass BOOST. This is an important modern C++ class library and as well is an excellent test of C++. Clang/X86 claims to pass BOOST. It was an important milestone in Clang when they were able to make this claim.
5. Recurse the compiler. Nominally, the Clang/LLVM compiler is built with gcc. We should be able to compile the compiler using itself, and then compile the compiler using that resultant compiler and get the same result. This is called recursing the compiler and is a very important test for compilers since a compiler is a very complex application and tends to use quite a bit of the language. Clangand LLVM make heavy use of sophisticated parts of C++.
As part of this process, we will have built a native MIPS linux hosted compiler.
Traditionlly this recursing should be done even on bare metal targets since the compiler requires only minimal IO functionality like file IO which can even be simulated over TCP.
6. EEMBC tests. Important as a general test and for benchmarking.
7. Spec tests. Important as a general test and for benchmarking.
8. Build the open source world. Open source projects all have their own test suites so it is possible to build and "make check" virtually any open source, including linux, various RTOS's , and hundreds or thousand or more apps etc.
We need a MIPS linux hosted compiler to do this on any large scale since otherwise some custom modification of make files may be necessary.
9. Function as the first phase of gcc building. The first phase of building a new gcc compiler requires some kind of C compiler. We should be able to use Clang/LLVM for this step.
10. Our own test library. Bugs found, internally generated tests will go here.
Build and Test Requirements.
We have a portal machine, dmz-portal.mips.com. Connected to this are various source repositories.
Check ins to the repository as well as varous time scheduled jobs will cause builds and run of the above tests.
Tests/Builds need to be done with various options:
1) All main optimization for speed levels, -O0, -O1, -O2, -O3 2) Optimization for space 3) Special common optimizations, no RTII, no exceptions, others. 4) Random optimization combinations. Many compiler bugs arise from the interaction of various optimization combinations that are not tested as a predefined combination as in something like -O1 for example. 5) Debug , i.e -g 6) Target - linux, bare metal, RTOS, etc., QEMU linux, QEMU bare metal,
Debug needs to matrix with all of 1-4, 6. i.e. -O1 -g, -O2 -g
Ideally all of this would be run on every checkin but this is not possible in practice.
Generally testing is dividing into various levels, i.e. mini test, nightly, weekly, release, etc.
Generally these is some kind of mini test run on every checkin to the repository. Buildbot software will cause build/tests to be done.
Results will be stored in some kind of database.
Visual examination will be done by using off the shelf open software that dipslays buildbot results.
Test results include:
1) failed tests 2) regression in time 3) regression in space
I think that 90% of what the benchmarking team from marketing needs will need will be satisfied with the testing that we are doing.
Requirements for servers
These jobs can all be run on Electric Cloud or whatever other system we might chose.
We should be able to submit jobs and get the results back.
In essence some kind of buildbot slave can run on Electric Cloud.
We are not interested to use any kind of workflow tools that are on Electric CLoud. The "make" accelerator is possible if it works seamlessly. We may need to do some testing to convince ourselves that it is a true "gnu make" drop in replacement.