Seminar: Advanced Software Engineering - Spring 2020

Project instructions

Version 1.4 (April 27th, 2020): Added a table with the description of the final decision values from reviewers.

Version 1.3 (April 21st, 2020): Added some details about the submission and reviewing process.

Version 1.2 (April 16th, 2020): Added the instructions for report submission and changed some deadlines.

Version 1.1 (March 15th, 2020): Changed the deadline for the focus selection.

Version 1.0 (February 18th, 2020): First version.

As a project for the seminar, you are asked to write a report about the literature review you will conduct during the seminar period.

📝 Report

Your report will be the result of a literature review you will conduct for an assigned topic. You are asked to select a main topic, and then to come up with a subtopic on which focus. You will receive all the indications to review the literature and produce a report, including how to write a document in LaTeX, during the lectures.

🔎 Search related work

You can use the main sources for scientific research papers, such as Google Scholar, IEEE Xplore, and ACM Digital Library. Other options are looking through the proceeding of the main conferences in software engineering, such as ICSE, FSE, ASE, MSR, ICPC, etc.

Starting from a target paper, you can look for related works backward in the past, i.e., snowballing, following the references cited in the text. On the contrary, you can use search engines functionalities to look to the papers that have cited the target paper after its publication.

You must always report the process you adopted to find the related work.

📐 Format

The report has to be written by using LaTeX and IEEE double-column template.

You have 6 pages for the review and 2 more for the references.

The paper has to be written in correct and understandable English.

🕵️ Anti-plagiarism check

Your work must be original. As instructors, we are required to run an anti-plagiarism tool against the documents you provide. The University of Zurich provides PlagScan, which is a tool comparing the text of documents with plenty of resources in the web and scientific work.

Please, consider this to not incur into serious issues.

💡 Topics

Software Engineering for Machine Learning

Machine learning is more and more integrated into the activities of companies. Even if it is classifiable as a software, the traditional software engineering processes are not enough to deal with it. The companies need to adopt a specific workflow to let new roles, such as data scientists, and data processes to work effectively together with the legacy operations and software.

Reference:

Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. “Software Engineering for Machine Learning: A Case Study.” In: IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 2019, pp. 291-300.

Agile Methodologies in Education

Agile methodologies are used to lighten the development process of software. However, they proved to be effective also in other production environments. Recently, teachers of any grade started to bring these methodologies into classrooms.

Reference:

Pasquale Salza, Paolo Musmarra, and Filomena Ferrucci. “Agile Methodologies in Education: A Review.” In: Agile and Lean Concepts for Teaching and Learning: Bringing Methodologies from Industry to the Classroom. Springer, 2019, pp. 25-45.

Flaky Tests

Flaky tests are a worrying phenomenon in continuous integration. They consists in non-deterministic results of the test running on the same version of the software under test. Their reasons are not perfectly clear, and their presence can make the testing phase totally unreliable.

Reference:

Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. “An Empirical Analysis of Flaky Tests.” In: ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). 2014, pp. 643-653.

Fuzz Testing

Let us consider a function that asks the user to have a choice between 0, 1, or 2, then stored into an integer variable. What if the user transmits 3? It would be possible if the switch case were not been implemented securely. 3 is a possible input for the function that impact on security.

Fuzzing is an automated testing technique that “randomly” generates test inputs for programs to expose faults in a highly effective manner.

Reference:

Valentin J.M. Manes, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. “The Art, Science, and Engineering of Fuzzing: A Survey.” In: IEEE Transactions on Software Engineering. 2020.

⏱️ Sprints

Sprint 1 - Registration

Events:

Working time: from 20.02.2020 to 26.02.2020 (1 week)

After the kick-off lecture, where we present the organization of the seminar together with the description of the topic options, you have to register officially. The deadline to complete this is the day before the second lecture.

You are required to register by sending an e-mail, indicating a ranking of preference of the topics you would like to discover. Please, also indication some motivations for your choices.

We will try to balance the assignments of topics, preferably accustoming students’ needs. You will be notified during the lecture of the day after the deadline what your final assignment is.

Sprint 2 - Focus choice

Events:

Working time: from 27.02.2020 to 18.03.2020 (2 weeks)

After having chosen the main topic, you will have to choose a subtopic on which to focus, especially if the main topic is too broad to deal with. You can notify your choice and motivation by sending an e-mail. We will briefly discuss choices during the lecture that will take place during the day after the deadline.

Sprint 3 - Literature review

Events:

Working time: from 12.03.2020 to 18.04.2020 (5 weeks)

You will have almost five weeks to produce your literature review report.

You are asked to conduct your research and report the results in the form of a report. At the end of this sprint, you will submit the PDF file to EasyChair to start the reviewing process.

Please, you are requested to send an anonymous version of your document, i.e., your name must not appear on it.

Sprint 4 - Report reviewing

Events:

Working time: from 19.04.2020 to 03.05.2020 (3 weeks)

Your report will follow a simulated blind peer reviewing process. You will not know the name of your reviewers and the reviewers for the same report will knot know each other. After the submission, you have three days for paper bidding. You are asked to select your preferences for reviewing, according to the list of titles and abstracts you find on EasyChair.

Each of you has to review a total of three papers in a week. After that, one week is reserved to have a discussion between reviewers to reach an agreement. The day after that, the notifications for revision, if any, will reach the authors of the papers.

Sprint 5 - Report revision and presentation

Events:

Working time: from 04.05.2020 to 11.05.2020 (1 week)

You have to take into account the revision indications you received from the reviewers of your paper. You have 1 week to submit the final version of the report by e-mail.

You have to show the work you concluded with a presentation. It will be part of the evaluation.

📄 Submission and reviewing process

Here you find the instructions for the submission and reviewing process. We are going to simulate the process that happens for a real conference. You will act both as an author and program committee member (reviewer). Each of you will be asked to review a total of three reports. We will also simulate a bidding process to indicate your preferences for the reviews. Once received the three reviews, we will start a discussion period in which a senior program committee member from the s.e.a.l. group will act as the moderator to reach a final decision. The senior member will also provide a metareview that will be delivered, together with the three reviews, to the author of the report. Finally, the author will use the feedback to produce the final version of the report.

Please, notice that we will use a double-blind reviewing process: as a reviewer, you cannot know the identity of the reports’ authors and neither the names of the other reviewers.

📄 Report submission

I will provide the link for the conference on EasyChair. This link will be used for the whole process. The first step will be to submit the PDF of your paper.

To do this, you will have to create an account and provide your information as an author.

Please, note that you have to use your full name and UZH account.

Then, you will be asked to insert the title, abstract, and at least three keywords to help identifying the topic of your report. Finally, the last field is to upload the PDF file.

Please, be careful to hide your name on the PDF to satisfy the blindness requirement.

I will manually manage the submission time to guarantee that everyone has correctly uploaded the document, without any blindness violation.

👉 Paper bidding period

Once I will have received all the expected submission, the bidding period will start. You will first receive an invitation to act as a program committee member for the conference so that you can join the reviewing process. Basically, you will be enabled to operate on EasyChair also as a reviewer, with the same account you used as an author. You have to indicate your preferences, which will help me deciding for the final assignment. Each of you will receive a total of three reviews to complete. I will do my best to satisfy your preferences, but also to balance the level of seniority of the members, since there are both BSc and MSc students.

☑️ Reviewing period

Once you will have received your three assigned papers, you will have about one week to produce a review for each of them

You will find a form on EasyChair to write your reviews. My suggestion is to read the paper and take down your comments (digitally, or with a pen if you print it). Then you write your review in a document. Once ready, you copy and paste your reviews on EasyChair.‌

Do not forget that you’ll be evaluated also (20%) on the level of contribution you will apply during the reviewing time.

As indications for the review, provide an overall evaluation and explain your rating:

  1. start with a brief summary;
  2. list the “points in favor” (pros) of the report. Mention parts, arguments, ideas, etc. that you think are a strong point of the report;
  3. in the same way, list the “points against” (cons). Try to provide constructive feedback;
  4. summarize your review in one paragraph, highlighting the most important points;
  5. you can provide minor comments about grammar, typos, or any other minor issues. These things should not influence your decision, but help the author to revise the paper for the final submission.

Please, be as factual as possible in your review, free of judgment, and do not use opinion as an argument. A statement similar to “I like X”, “I do not think”, “I assume” has no room in a professional review. Instead, base your response on (counter-) arguments.

Be positive and fair to each other!

🗣️ Discussion period

Once received all the reviews, we will start with the discussion period. A senior member from the s.e.a.l. group will join the discussion for each of the papers. Your goal is to reach a consensus at the end, and produce the final version of your review. Indeed, you are allowed to revise your original review, and successive versions, including the score.

An effective way to join the conversation would be that of reading the other two reviews and comment the points on which you agree, and those where disagree.

The following table describes the possible decision values.

Decision Description
😖 Major revision The reviewers have major concerns about your report. You should definitively revise your report before the final submission.
😔 Revision You should carefully consider the reviews and thoroughly revise your report.
🙂 Good Good report, try to consider the reviews to make improvements for the final submission.
🥳 Great Great report, only minor changes are required.

Consider that the author of the paper will receive all of your anonymous reviews that s/he will use to revise the final version of the report. Moreover, the senior program committee member will write a metareview that will accompany the ones you will write.

✉️ Notification to the authors and final submission

Finally, every author will receive a notification about the decision about the reports. Every one will be asked for some modifications to the report, if any. Once ready, you can send me an e-mail with the final version that has to include the source code (LaTeX) as well.