Bibliography for “Practical Testing of a C99 Compiler Using Output Comparison”

Flash Sheridan


This is the bibliography from my article, “Practical Testing of a C99 Compiler Using Output Comparison,” published in Software: Practice and Experience (http://doi.wiley.com/10.1002/spe.812; a pre-print is available at http://pobox.com/~flash/Practical_Testing_of_C99.pdf.) The journal version of the bibliography is in reference order, and contains items discussed in the article but not directly relevant to compiler testing. The online version is sorted alphabetically, with items on topics other than compiler testing separated; it will be updated if more articles on compiler testing appear. The emphasis, as the title of the article indicates, is on practical testing of C/C++ compilers.

The literature on compiler testing is surprisingly scant. There is substantial literature on the theoretical design of compilers which would provably not need testing, but the audience for such work is largely disjoint from that for the testing of compilers for widely-used languages which will have a substantial user base. There are also a number of articles on the automated generation of test code, but given that there is now a substantial base of real Open Source software, this is less useful than formerly.

This article, and PalmSource’s testing, was firmly directed towards shipping a high-quality, but imperfect, compiler which would be of practical use to the developer community. Producing an inherently bug-free compiler for a theoretically desirable language was not an option. The goal was to catch as high a proportion of serious bugs as possible in a useful compiler for two widely-used languages, C99 and C++98.

The best available bibliography is over a decade old, by Dr. C.J. Burgess of the University of Bristol; it was a posting to the comp.compilers Usenet newsgroups, below [Burgess]. Bailey & Davidson [Bailey & Davidson] is an academic article on the testing of function calls, somewhat similar to Lindig’s Quest [Lindig]; it contains the interesting observations that “the state-of-the-art in compiler testing is inadequate” (p. 1040), and that in their experience, the ratio of failed tests to bugs was approximately one thousand to one (p. 1041). The standard work on compiler theory is Compilers: Principles, Techniques and Tools [Aho et al], commonly known as the Dragon book. It is a good general introduction, but had little direct relevance to our testing, except for some extra caution in including examples of spaghetti code; other standard compiler texts which were consulted, but did not have significant sections on testing, are omitted from the bibliography. A Retargetable C Compiler: Design and Implementation [Fraser & Hanson] contains a brief section on the authors’ experience with testing their compiler, with some practical advice on the importance of regression test cases; difficulties in using lcc’s regression tests for other compilers are discussed above, in the section on emulated-execution output correctness testing. An updated and alphabetized version of this bibliography will be made available at http://pobox.com/~flash/compiler_testing_bibliography.html.



Compiler Testing

  1. Bailey, Mark W. and Davidson, Jack W., “Automatic Detection and Diagnosis of Faults in Generated Code for Procedure Calls”, IEEE Transactions on Software Engineering, volume 29, issue 11, 2003. An abstract is available online, at http://csdl.computer.org/comp/trans/ts/2003/11/e1031abs.htm, as is an earlier version of the full paper, http://big-oh.cs.hamilton.edu/~bailey/pubs/techreps/TR-2001-1.pdf
  2. Bhattacharya, Soumyabrata “ANSI C Test suites,” comp.compilers, http://compilers.iecc.com/comparch/article/94-10-060, 1994.
  3. Burgess, C.J. , “Bibliography for Automatic Test Data Generation for Compilers,” comp.compilers, http://compilers.iecc.com/comparch/article/93-12-064, 1993.
  4. DejaGnu http://www.gnu.org/software/dejagnu/, 1993-.
  5. Delta, a tool for test failure minimization, Wilkerson, Daniel and McPeak, Scott, http://delta.tigris.org, 2003-5. Based on [Zeller]. See also [Open Source Quality Project].
  6. Fraser, Christopher and Hanson, David, A Retargetable C compiler: Design and Implementation, ISBN: 0-8053-1670-1, Benjamin/Cummings Publishing, 1995, §19.5 pp. 531–3.
  7. Jones, Derek “Who Guards the Guardians?” (a study of the coverage of the Perennial Validation Suite), http://www.knosof.co.uk/whoguard.html, 1993.
  8. Kahan, William Sumner, Thos, et al., Paranoia Floating Point Test, http://www.netlib.org/paranoia/paranoia.c, 1983-5.
  9. lcc, A Retargetable Compiler for ANSI C, http://www.cs.princeton.edu/software/lcc/; described in A Retargetable C Compiler: Design and Implementation, Hanson, David R. and Fraser, Christopher W., ISBN: 0-8053-1670-1, Benjamin/Cummings Publishing 1995.
  10. Lindig, Christian, “Random Testing of the Translation of C Function Calls”, http://www.st.cs.uni-sb.de/~lindig/src/quest. Proceedings of the Sixth International Workshop on Automated Debugging, ISBN 1-59593-050-7, Association for Computing Machinery 2005.
  11. Modena Test++ Suite, http://modena.us/testpp.htm.
  12. Open Source Quality Project http://osq.cs.berkeley.edu/.
  13. Perennial Validation Suites http://peren.com/pages/products_set.htm.
  14. Plum Hall C and C++ Validation Test Suites http://plumhall.com/suites.html.
  15. Sheridan, Flash, “Practical Testing of a C99 Compiler Using Output Comparison,” Software: Practice and Experience, http://dx.doi.org/10.1002/spe.812, 2007. A pre-print is available at http://pobox.com/~flash/Practical_Testing_of_C99.pdf. A list of bugs discovered using the techniques in the article is at http://pobox.com/~flash/FlashsOpenSourceBugReports.html.
  16. Small Device C Compiler (SDCC), Dutta, Sandeep et al., http://sdcc.sourceforge.net/, 1999-.
  17. Tydeman, Fred, C99 FPCE Test Suite, http://www.tybor.com/readme.1st, 1995-2006.
  18. Zeller, A.: “Yesterday, my program worked. Today, it does not. Why?”, Software Engineering - ESEC/FSE'99: 7th European Software Engineering Conference, ISSN 0302-9743, volume 1687 of Lecture Notes in Computer Science, pp. 253-267, 1999.

Source Code Useful for Compiler Testing (Primarily C/C++)

Other




Copyright © 2002-2007, Access Systems Americas, Inc. PalmSource, Palm OS and Palm Powered, and certain other trade names, trademarks and logos are trademarks which may be registered in the United States, France, Germany, Japan, the United Kingdom and other countries, and are either owned by PalmSource, Inc. or its affiliates, or are licensed exclusively to PalmSource, Inc. by Palm Trademark Holding Company, LLC. All other brands, trademarks and service marks used herein are or may be trademarks of, and are used to identify other products or services of, their respective owners. All rights reserved.