Project title: A study on dissimilarities between test case executions

Note: In students' assignments and project reports, if any part (including sentences, figures/diagrams, tables, definitions, descriptions, data, and so on) are copied from other people's work (including both published and unpublished papers, reports, Web articles, etc), or if any idea is from other people, such work or people must be acknowledged explicitly. If you directly "copy and paste" sentences from other people's work to your report, you must use quotation marks to enclose the copied sentences and make a citation immediately. It is otherwise plagiarism.

1. Size of a group: minimum four, maximum six.

2. Progress report: Each group is required to give a 3-minute oral report to the tutor in Week 4 lab, and submit a two-page progress report. All team members are required to attend. The progress report should cover the following points:
i. Every team member's name, email address and student number.
ii. What is your progress so far?
iii. Did you encounter any problems?
iv. Your future plan and timetable.
v. A summary of the paper "On the correlation between the effectiveness of metamorphic relations and dissimilarities of test case executions" (available under "Reading material -> Research papers" in Moodle.)

(1) Progress report:

(2) Final presentation:

Submit a hardcopy of your presentation slides to the lecturer/tutor immediately before your group's presentation. It is not compulsory for all team members to speak, but all team members must stand in front of the class during their group's presentation. If a student is absent from his/her group presentation, the student will receive 0 presentation marks.

(3) Did you use a folder to enclose all submitted materials when you submit the final report in week 12 lecture?

(4) Technical value and writing of the final project report

6. Technical tasks

Step 1: Paper reading

Conduct in-depth study of the following paper (available in Moodle site under "Reading material -> Research papers").

[1] Y. Cao, Z. Q. Zhou, and T. Y. Chen, "On the correlation between the effectiveness of metamorphic relations and dissimilarities of test case executions," in Proceedings of the 13th International Conference on Quality Software (QSIC'13). IEEE, 2013, pp. 153-162.

Write a one-page summary of the above paper using your own word --- you are not allowed to copy any sentence from the paper when writing the summary.

Read the following related papers, summarize each of them, and explain their relationship and difference to [1] (one page each):

[2] "Comparing white-box and black-box test prioritization".

[3] "Measurable Metrics for Qualitative Guidelines of Metamorphic Relation". In COMPSAC 2015,

[4] Metamorphic Relations Ranking for Reducing Testing Cost in Scientific Software. Master Thesis.

Step 2: Conduct an empirical study similar to [1] by using different subject programs.

Identify 9 (or more) subject programs, each of which has a size similar to (or larger than) the grep, sed, or bash programs listed in Table 1 of [1]. These programs should have characteristics similar to the above grep, sed, and bash programs in the sense that (1) each of them have multiple faulty versions (preferably with real faults, but systematically seeded faults are also acceptable) and (2) you can apply the GCOV tool to record the coverage data for each test case execution. Do the following:

(1) For each subject program, identify 10 or more metamorphic relations (MRs).

(2) For each MR, generate a large number of (minimum 1000) source test cases. You can use existing test suites or test case generation tools or random generation for this purpose. Different MRs can share the same set of source test cases.

(3) Conduct experiments using the faulty versions of the subject programs, the MRs, and the test cases, in concert with GCC and the GCOV tool, to validate the statement "there is a significant strong positive correlation between BCMD and the fault-detection effectiveness of MRs [1, p161]." You can also use tools other than GCC and GCOV for this study.

(4) Report your findings.

Step 3: Try other distance metrics other than those reported in [1], and conduct similar empirical study to see which one is better

Frequently asked question: Where can I get the subject programs and their faulty versions?

Answer: From all possible sources, such as from open source projects or your past software development projects. Another source is the Software-artifact Infrastructure Repository (SIR).