CS2103/T - Admin: tP: Practical Exam

tP: Practical Exam Dry Run (PE-D)

PE-D Overview

What: The latest release of the v1.3 period is subjected to a round of peer acceptance/system testing, also called the Practical Exam (PE) Dry Run as this round of testing will be similar to the graded Practical Exam that will be done at v1.4.

When, where: uses a 40 minute slot at the start of week 11 lecture slot (to be done online).

Grading: The PE dry run affects your grade in the following ways.

If you scored less than half of the marks in the PE, we will consider your performance in PE dry run as well when calculating the PE marks. You can earn up to 2 bonus marks in this manner.
PE dry run is a way for you to practice for the actual PE.
Taking part in the PE dry run will earn you participation points.
There is no penalty for bugs reported in your product. Every bug you find is a win-win for you and the team whose product you are testing.

Why:

To train you to do manual testing, bug reporting, bug bug fixing, communicating with users/testers/developers, evaluating products etc.
To help you improve your product before the final submission.

PE-D Preparation

Ensure that you have accepted the invitation to join the GitHub org used by the module. Go to https://github.com/nus-cs2103-AY2223S1 to accept the invitation.
Ensure you have access to a computer that is able to run module projects e.g. has the right Java version.
Ensure you can run CATcher on your computer. You should have done this when you smoke-tested CATcher earlier in the semester.

Have a good screen grab tool with annotation features so that you can quickly take a screenshot of a bug, annotate it, and post in the issue tracker.
You can use Ctrl+V to paste a picture from the clipboard into a text box in a bug report.
[Optional] Have a good screen recording tool if you plan to use screen recording clips as part of your bug reports. Ensure that your screen recording tool can create small files as CATcher doesn't allow files bigger than 10Mb.
As the CATcher support for uploading screen recordings is new and limited, use it only if strictly necessary -- use screenshots for other cases.
Download the product to be tested.

PE Dry Run (at v1.3)

After you have been notified which team to test (likely to be in the morning of PE-D day), download the latest JAR file from the team's releases page.

PE (at v1.4)

After you have been notified of the download location, download the zip file that bears your name. The password required to unzip it will be given to you at the start of the PE.

Testing tips

Use easy-to-remember patterns in test data. For example, if you use 12345678 as a phone number while testing and it appears as 2345678 somewhere else in the UI, you can easily spot that the first digit has gone missing. But if you used a random number instead, detecting that bug won't be as easy. Similarly, if you use Alice Bee, Benny Lee, Charles Pereira as test data (note how the names start with letters A, B, C), it will be easy to detect if one goes missing, or they appear in the incorrect order.

Go wide before you go deep. Do a light testing of all features first. That will give you a better idea of which features are likely to be more buggy. Spending equal time for all features or testing in the order the features appear in the UG is not always the best approach.

PE-D During the session

Use the CATcher Web app for reporting bugs. More instructions will be given during the PE-D e.g., which session to use.

Use MS Teams (not Zoom) to contact prof if you need help during the session. Use Zoom chat only if you don't get a response via MS Teams.

How many bugs to report?

Report as many bugs as you can find during the given time. Take longer if you need (unlike the PE, PE-D is not timed strictly). If you can't find many bugs at this stage when the product is largely untested, you are unlikely to be able to find enough bugs in the better-tested final submission later. In that case, all the more reasons to spend more time and find more bugs now.

Insincere bug reports will not count for credit i.e., don't submit 'fake' bug reports to increase the bug count.

The median number of bugs reported in the previous semester's PE-D was 9. Someone reporting just a 2-3 bugs is usually a sign of a half-hearted attempt rather than lack of bugs to find. If you really can't find bugs, at least submit suggestions for improvements.

PE and PE-D are manual testing sessions. Using test automation tools or scripting is not allowed.

Test the product and report bugs as described below, when the prof informs you to begin testing.

Testing instructions for PE and PE-D

a) Launching the JAR file

Get the jar file to be tested:

PE Dry Run (at v1.3)

Download the latest JAR file from the team's releases page, if you haven't done this already.

PE (at v1.4)

Download the zip file from the given location (to be given to you at least a few hours before the PE), if you haven't done that already.
Unzip the downloaded zip file with the password (to be given to you at the start of the PE, via LumiNUS gradebook). This will give you another zip file with the name suffix _inner.zip.
Unzip the inner zip file. This will give you the JAR file and other PDF files needed for the PE. Warning: do not run the JAR file while it is still inside the zip file.
Ignore the padding_file found among the extracted files. Its only purpose is to mask the true size of the JAR file so that someone cannot guess which team they will be testing based on the zip file size.

Put the JAR file in an empty folder in which the app is allowed to create files (i.e., do not use a write-protected folder).
In rare cases, the team could have submitted a ZIP file instead of a JAR file. In that case, unzip that file into the target folder.
Open a command window. Run the java -version command to ensure you are using Java 11.
Do this again even if you did this before, as your OS might have auto-updated the default Java version to a newer version.
Check the UG to see if there are extra things you need to do before launching the JAR file e.g., download another file from somewhere
You may visit the team's releases page on GitHub if they have provided some extra files you need to download.
Launch the jar file using the java -jar command rather than double-clicking (reason: to ensure the jar file is using the same java version that you verified above). Use double-clicking as a last resort.
If you are on Windows, use the DOS prompt or the PowerShell (not the WSL terminal) to run the JAR file.
If the JAR file name has spaces in it, remember to surround it with double quotes.
e.g., java -jar "Name With Spaces.jar"

If the product doesn't work at all: If the product fails catastrophically e.g., cannot even launch, or even the basic commands crash the app, do the following:

Check the UG of the team, to see if there are extra things you need to do before launching the JAR.
Confirm that you are using Java 11 and using the java -jar command to run the JAR, as explained in points above.
Contact our head TA via MS Teams (name: Kim Hyeongcheol, NUSNET: dcskh) and give him
(a) a screenshot of the error message, and
(b) your GitHub username.
Wait for him to give you a fallback team to test.
Contact the prof (via MS Teams) if you didn't get a response from Kim within 5 minutes.
Delete bug reports you submitted for the previous team (if any), using CATcher.
You should not go back to testing the previous team after you've been given a fallback team to test.

b) What to test

PE Dry Run (at v1.3)

Test the product based on the User Guide available from their GitHub website https://{team-id}.github.io/tp/UserGuide.html.
Do system testing first i.e., does the product work as specified by the documentation?. If there is time left, you can do acceptance testing as well i.e., does the product solve the problem it claims to solve?.

PE (at v1.4)

Test based on the Developer Guide (Appendix named Instructions for Manual Testing) and the User Guide. The testing instructions in the Developer Guide can provide you some guidance but if you follow those instructions strictly, you are unlikely to find many bugs. You can deviate from the instructions to probe areas that are more likely to have bugs.
As before, do both system testing and acceptance testing but give priority to system testing as those bugs can earn you more credit.
Be careful when copying commands from the UG (PDF version) to the software as some PDF viewers can affect the pasted text. If that happens, you might want to open the UG in a different PDF viewer.

c) What bugs to report?

PE Dry Run (at v1.3)

You may report functionality bugs, UG bugs, and feature flaws.

You can also post suggestions on how to improve the product.
Be diplomatic when reporting bugs or suggesting improvements. For example, instead of criticising the current behavior, simply suggest alternatives to consider.

PE (at v1.4)

Report functionality bugs:

Do not post suggestions but if the product is missing a critical functionality that makes the product less useful to the intended user, it can be reported as a bug of type Type.FeatureFlaw. The dev team is allowed to reject bug reports framed as mere suggestions or/and lacking in a convincing justification as to why the omission of that functionality is problematic.

You may also report UG bugs.

d) How to report bugs

Post bugs as you find them (i.e., do not wait to post all bugs at the end) because bug reports created/modified after the allocated time will not count.

Using CATcher

Launch CATcher, and login to the correct profile (when CATcher asks, consent to creating a new repo):
- PE Dry Run: CS2103/T PE Dry run
- PE: CS2103/T PE
Post bugs using CATcher.

Not using CATcher

Issues created for PE-D and PE need to be in a precise format for our grading scripts to work. Incorrectly-formatted responses will have to discarded. Therefore, you are not allowed to use the GitHub interface for PE-D and PE activities, unless you have obtained our permission first.

Post bug reports in the following repo you created earlier:
- PE Dry Run: ped
- PE: pe
The whole description of the bug should be in the issue description i.e., do not add comments to the issue.

e) Bug report format

Each bug should be a separate issue i.e., do not report multiple problems in the same bug report.
If there are multiple bugs in the same report, the dev team will select only one of the bugs in the report and discard the others.
When reporting similar bugs, it is safer to report them as separate bugs because there is no penalty for reporting duplicates while putting multiple bugs in the same report can reduce your bug count (see the previous point). But as submitting multiple bug reports take extra time, if you are quite sure they will be considered as duplicates by the dev team later, you can report them together, to save time.
Write good quality bug reports; poor quality or incorrect bug reports will not earn credit.
Use a descriptive title.
Give a good description of the bug with steps to reproduce, expected, actual, and screenshots. If the receiving team cannot reproduce the bug, you will not be able to get credit for it.
Assign exactly one severity.* label to the bug report. Bug reports without a severity label are considered severity.Low (lower severity bugs earn lower credit)

Bug Severity labels:

severity.VeryLow : A flaw that is purely cosmetic and does not affect usage e.g., a typo/spacing/layout/color/font issues in the docs or the UI that doesn't affect usage. Only cosmetic problems should have this label.
severity.Low : A flaw that is unlikely to affect normal operations of the product. Appears only in very rare situations and causes a minor inconvenience only.
severity.Medium : A flaw that causes occasional inconvenience to some users but they can continue to use the product.
severity.High : A flaw that affects most users and causes major problems for users. i.e., only problems that make the product almost unusable for most users should have this label.

When applying for documentation bugs, replace user with reader.

Assign exactly one type.* label to the issue.

Type labels:

type.FunctionalityBug: A functionality does not work as specified/expected.
type.FeatureFlaw: Some functionality missing from a feature delivered in v1.4 in a way that the feature becomes less useful to the intended target user for normal usage. i.e., the feature is not 'complete'. In other words, an acceptance-testing bug that falls within the scope of v1.4 features.
These issues are counted against the product design aspect of the project. Therefore, other design problems (e.g., low testability, mismatches to the target user/problem, project constraint violations etc.) can be put in this category as well.
Features that work as specified by the UG but should have been designed to work differently (from the end-user's point of view) fall in this category too.
type.DocumentationBug: A flaw in the documentation e.g., a missing step, a wrong instruction, typos

If you need to include < or > symbols in your bug report, use HTML equivalents for 'less than' and 'greater than' instead (i.e., < and > e.g., x < y instead of x < y).
Reason: CATcher and GitHub strips out content wrapped in < and >, for security reasons.

PE-D After the session

The relevant bug reports will be transferred to your issue tracker within a day after the session is over. Once you have received the bug reports for your product, you can decide whether you will act on reported issues before the final submission v1.4. For some issues, the correct decision could be to reject or postpone to a version beyond v1.4.
Reminder: There is no penalty for any of the bugs you received in the PE-D.

Dealing with "What the h___ the tester was thinking?" type bug reports

Some bug reports will make you angry because they seem baseless, wrong, offensive etc. It's still possible to get value from such bug reports though:

After you got over the initial indignation, dig deeper to see if there's even the slightest possibility that there is a bug. For example, consider this scenario:
1. The tester claims a certain command doesn't work.
2. All your team members tried the exact same command and it works as advertised. What the h___ the tester is trying to pull here?
3. In reality, the error is actually caused by a duplicate entry in the database resulting from a previous command; the tester didn't mention that command in the bug report (because s/he didn't realize the two are connected).
The reported bug might be non-existent but the tester's screw up can indicate other areas to improve. For example, the tester reports a missing feature that is clearly mentioned as 'not implemented' in the UG, but perhaps the UG can be improved to make that fact harder to miss?
What exactly about the bug report that makes you angry? Remind yourself not to do the same offence when you report bugs yourself in future.

Use the pain of dealing with this kind of bug reports as an opportunity to develop the following mindset:

(a) The product is guilty until proven innocent: If the bug report has even a hint of something amiss with the product, it's your (not the tester's) responsibility to try and prove if it is really a problem or not. Why? because finding a bug is a win for you -- as you can then fix it and thereby avoid the humiliation of releasing a buggy product.

(b) A crappy bug report is better than no bug report: If the bug actually exists, it is better to have some indication about it than none at all. In a real project, a tester that fails to find bugs can cause more harm to your career than a tester who finds bugs but doesn't report them well.

(c) I used to get angry at bug reports, but not any more: If you work hard, take pride in the quality of your work, it's no wonder that you get angry when others find faults with your work incorrectly. Aim to move past that phase where you take bug reports personally. The sooner you can tackle any sh*tty bug report calmly and objectively, the sooner you'll rise to the 'professional' software engineer level.

If you have received stray bug reports (i.e., bug reports that don't seem to be about your project), do let us know ASAP (email the prof).
You can navigate to the original bug report (via the back-link provided in the bug report given to you) and post in that issue thread to communicate with the tester who reported the bug e.g. to ask for more info, etc. However, the tester is not obliged to respond. Note that simply replying to the bug report in your own repo will not notify the tester.
- Do not argue with the tester to try to convince that person that your way is correct/better. If at all, you can gently explain the rationale for the current behavior but do not waste time getting involved in long arguments. If you think the suggestion/bug is unreasonable, just thank the tester for their view and discontinue to discussion.
Aim to do a systematic triaging of issues received. Some suggestions:
- Close duplicate issues.
- Use labels (create new labels if necessary) to,
  - differentiate bugs from the rest (e.g., feature suggestions/flaws).
  - indicate priority of the bugs that need fixing.
- Assign each bug to the person who should fix it.
You may ignore type/severity.* labels given by the tester. They will not affect you or the tester either way -- they were there just for the testers to practice. You may apply your own type/severity labels if you wish.
In particular, beware of simply following the type.* given by the tester; it is your job to decide the correct type of the issue. e.g., What the tester labeled as a bug might actually be a feature flaw.
If a bug report is simply a feature suggestion, you can take note of it and close it (to reduce clutter in the issue tracker, and to make it easy for the teaching team to track your progress on dealing with PE-D issues). Similarly, you can close PE-D issues not relevant to v1.3.

Note that listing bugs as 'known bugs' in the UG or specifying unreasonable constraints in the UG to make bugs 'out of scope' will not exempt those bugs from the final grading. That is, PE testers can still earn credit for reporting those bugs and you will still be penalized for them.
However, a product is allowed to have 'known limitations' (e.g., a daily expense tracking application meant for students is unable to handle expenses larger than $999) as long as they don't degrade the product's use within the intended scope. They will not be penalized.

Even bugs inherited from AB3 need to be fixed. As mentioned in a previous week, even bugs you inherited from AB3 need fixing (because "we inherited it from the previous dev team" is not a valid excuse to leave a bug unfixed). If you are unsure if something is such a bug that need fixing, please post in the forum.

Identify bugs you missed in the PE-D: Visit the issue tracker of the team you tested and see bugs reported by others who tested the same product. Identify bugs you missed (if any). That knowledge might help you find similar bugs in your own product as well as find more bugs during the PE.

tP: Practical Exam (PE)

Remote or F2F? You can choose between the following two modes:

Remote mode: This is the recommended mode. Proctored via Zoom. You'll need to join the Zoom session from a quiet place (i.e., conducive to an exam) at which you can set up a Zoom device for proctoring.
F2F mode: Attend the PE at the lecture venue (iCube auditorium). Choose only if you are unable to use the remote mode (reason: higher the number of F2F attendees, higher the risk of WiFi speed issues and GitHub throttling issues).

We'll use a LumiNUS survey to collect your preferred mode.

PE Overview

The upfront objective of the PE is to increase the rigor of project grading. Assessing most aspects of the project involves an element subjectivity. As the project counts for a large percentage of the final grade, it is not prudent to rely on evaluations of tutors alone as there can be significant variations between how different tutors assess projects. That is why we collect more data points via the PE so as to minimize the chance of your project being affected by evaluator-bias.
PE mainly evaluates your testing skills, done as the following two-parts:
1. You will be given a chance to find bugs in a different software. Furthermore, you will be given an opportunity to ~~defend your bug reports against any possible objections~~ interact with the developers (anonymously) to refine the bug report further e.g., correct the severity level if the level you chose initially was incorrect. If you report possible bugs that turn out to be actual bugs, you earn marks (provided the product actually had bugs in the first place).
2. Your product will be subjected to a rigorous testing and you will be given a chance to point out any inaccuracies in the bugs reported. You will lose marks for any bugs that turned out to be real bugs, but only if your work has more bugs than a certain bar.
The above two can lead to high-rigor, evaluation of your testing skills. The alternative is to rely solely on other easy-to-measure metrics (e.g., the number of test cases, test coverage, test LoC etc.) which we don't think as good representations of your actual testing skills. The ultimate objective of the PE is not even the higher rigor of grading. Because of the PE, you will realize that any bugs are very likely to be detected, which means you will work extra hard to avoid bugs; and THAT is the real benefit. Ultimately, what we want is for you to internalize a higher standard for testing and a lower tolerance for bugs in your own code -- something that can have a longer-lasting impact on your future careers, beyond the mere module grade.
Problem: There is no way we can carry out the above-mentioned two-part evaluation at a high-level of rigor if using tutors as testers, or using an automated testing script. e.g., some tutors might not have the motivation to try hard enough to find bugs, and it will be hard to find tutors willing to spend many hours testing products so near to their own exams.
Solution: Get the two parts of the evaluation to feed each other by getting student to test each others' products.
The fact that you are testing products created by your classmates and objecting to bugs reported by your classmates can makes this a rather 'unpleasant' experience. You might feel like being pitted against each other, or as if you are forced to bring down each other. But as you read above, it is a necessary evil for this evaluation to be even possible. Given the actual goal is to get you to create products with very few bugs, we think switching off the 'collaborative learning' mode for just a few days is a price worth paying to achieve that goal. After all, the PE is an evaluation activity (not a learning activity) and happens after the regular learning period is over.
You are not taking marks from someone else -- at least, don't think of it that way. The point of contention is 'is this really a bug?' which is independent of the people involved. Furthermore, the reward for detecting a bug and the penalty for having a bug in your code are calculated independently.
Still, none of us likes it when others point out problems of our work. Some of us don't even like pointing out problems of others' work. But we just have to learn not to take bug reports personally. Another important intended outcome is to be able to report bugs in a way that doesn't feel like you are attacking or trying to sabotage the dev team.
PE also evaluates aspects other than testing e.g., your product evaluation skills, effort estimation skills etc. When evaluating those aspects in particular, they are not graded solely based on peer ratings. Rather, PE data are cross-validated with tutors' grades to identify cases that need further investigation. When peer inputs are used for grading, they are usually combined with tutors' grades with appropriate weight for each. In some cases ratings from team members are given a higher weight compared to ratings from other peers, if that is appropriate.
Grading:
- Your performance in the practical exam will affect your final grade and your peers', as explained in Admin: Project Grading section.
- As such, we have put in measures to identify and penalize insincere/random evaluations.
- Also see:

PE Preparation

It's similar to,

In addition, it is recommended that you read the guidelines the dev team will follow when responding to your bug reports later, given in the panel below.

PE Phase 1: Bug Reporting

When: Last lecture slot of the semester (Fri, Nov 11th). Remember to join 15-30 minutes earlier than usual lecture start time. The Zoom link will be given to you closer to the day.

PE Phase 1 will conducted under exam conditions. We will be following the SoC's E-Exam SOP, combined with the deviations/refinements given below. Any non-compliance will be dealt with similar to a non-compliance in the final exam.
Note: Those opting for the F2F mode can ignore any Zoom-related points in the instructions below.

Proctoring will be done via Zoom. No admission if the following requirements are not met.
- You need two Zoom devices (PC: chat, audio ~~video~~, Phone: video, ~~audio~~), unless you have an external web cam for your PC.
- Add your [PE_seat_number] in front of the first name of your Zoom display name, in your Zoom devices. Seat numbers can be found in here about 2 days before the PE. e.g.,
  - [M48] John Doe (M18 is the seat number)
  - [M48][PC] John Doe (for the PC, if using a phone as well)
- Set your camera so that all the following are visible:
  1. your face (side view, no mask)
  2. your hands
  3. the work area (i.e., the table top)
  4. the computer screen
Join the Zoom waiting room 15-30 minutes before the start time. Admitting you to the Zoom session can take some time.
If on F2F mode: Arrive at lecture venue (ICube Auditorium) around 10-15 minutes early. If you haven't been to this venue before, plan ahead as the venue is located away from the main SoC area.
In case of Zoom outage, we'll fall back on MS Teams (MST). Make sure you have MST running and have joined the MST Team for the class.
Recording the screen is not required.
You are allowed to use head/ear phones. But no talking allowed (unless you are talking to the invigilator) -- so, no talking/singing to your self as this can be mistaken for a rule violation.
Only one screen is allowed (for both remote mode and F2F mode). If you want to use the secondary monitor, you should switch off the primary monitor. The screen being used should be fully visible in the Zoom camera view.
If using a second device for Zoom proctoring, the screen of that device should only be used for Zoom.
Do not use the public chat channel to ask questions from the prof. If you do, you might accidentally reveal which team you are testing.
Do not use more than one CATcher instance at the same time. Our grading scripts will red-flag you if you use multiple CATcher instances in parallel.
Use MS Teams (not Zoom) private messages to communicate with the prof. Zoom sessions are invigilated by tutors, not the prof.
Do not view video Zoom feeds of others while the testing is ongoing. Keep the video view minimized.
During the bug reporting periods (i.e., PE Phase 1 - part I and PE Phase 1 - part II), do not use websites/software not in the list given below. If you need to visit a different website or use another software, please ask for permission first.
- Website: LumiNUS
- Website/software: MSTeams (only to communicate with the prof of Tech support)
- Website: Module website (e.g., to look up PE info)
- Software: CATcher, any text editor, any screen grab/recording software
- Software: PDF reader (to read the UG/DG or other references such as the textbook)
- Software: A text editor (to keep track of commands you tried)
Do not visit GitHub in PE Phase 1 - part I unless you are going there to download a file the team has provided and is needed for testing. You may visit GitHub during part II and part III.
Do not use any other software running in the background e.g., Telegram chat.
This is a manual testing session. Do not use any test automation tools or custom scripts.

PE Phase 1 - Part I Product Testing [60 minutes]

Bonus marks for high accuracy rates!

You will receive bonus marks if a high percentage (e.g., some bonus if >50%, a substantial bonus if >70%,) of your bugs are accepted as reported (i.e., the eventual type.* and severity.* of the bug match the values you chose initially and the bug is accepted by the team).

Take away: Aim for the correct severity/type etc. rather than the one that gives you most marks, as the former can end up earning you more marks in the end anyway.

Test the product and report bugs as described below. You may report both product bugs and documentation bugs during this period.

Testing instructions for PE and PE-D

a) Launching the JAR file

Get the jar file to be tested:

PE Dry Run (at v1.3)

Download the latest JAR file from the team's releases page, if you haven't done this already.

PE (at v1.4)

Download the zip file from the given location (to be given to you at least a few hours before the PE), if you haven't done that already.
Unzip the downloaded zip file with the password (to be given to you at the start of the PE, via LumiNUS gradebook). This will give you another zip file with the name suffix _inner.zip.
Unzip the inner zip file. This will give you the JAR file and other PDF files needed for the PE. Warning: do not run the JAR file while it is still inside the zip file.
Ignore the padding_file found among the extracted files. Its only purpose is to mask the true size of the JAR file so that someone cannot guess which team they will be testing based on the zip file size.

Put the JAR file in an empty folder in which the app is allowed to create files (i.e., do not use a write-protected folder).
In rare cases, the team could have submitted a ZIP file instead of a JAR file. In that case, unzip that file into the target folder.
Open a command window. Run the java -version command to ensure you are using Java 11.
Do this again even if you did this before, as your OS might have auto-updated the default Java version to a newer version.
Check the UG to see if there are extra things you need to do before launching the JAR file e.g., download another file from somewhere
You may visit the team's releases page on GitHub if they have provided some extra files you need to download.
Launch the jar file using the java -jar command rather than double-clicking (reason: to ensure the jar file is using the same java version that you verified above). Use double-clicking as a last resort.
If you are on Windows, use the DOS prompt or the PowerShell (not the WSL terminal) to run the JAR file.
If the JAR file name has spaces in it, remember to surround it with double quotes.
e.g., java -jar "Name With Spaces.jar"

If the product doesn't work at all: If the product fails catastrophically e.g., cannot even launch, or even the basic commands crash the app, do the following:

Check the UG of the team, to see if there are extra things you need to do before launching the JAR.
Confirm that you are using Java 11 and using the java -jar command to run the JAR, as explained in points above.
Contact our head TA via MS Teams (name: Kim Hyeongcheol, NUSNET: dcskh) and give him
(a) a screenshot of the error message, and
(b) your GitHub username.
Wait for him to give you a fallback team to test.
Contact the prof (via MS Teams) if you didn't get a response from Kim within 5 minutes.
Delete bug reports you submitted for the previous team (if any), using CATcher.
You should not go back to testing the previous team after you've been given a fallback team to test.

b) What to test

PE Dry Run (at v1.3)

Test the product based on the User Guide available from their GitHub website https://{team-id}.github.io/tp/UserGuide.html.
Do system testing first i.e., does the product work as specified by the documentation?. If there is time left, you can do acceptance testing as well i.e., does the product solve the problem it claims to solve?.

PE (at v1.4)

Test based on the Developer Guide (Appendix named Instructions for Manual Testing) and the User Guide. The testing instructions in the Developer Guide can provide you some guidance but if you follow those instructions strictly, you are unlikely to find many bugs. You can deviate from the instructions to probe areas that are more likely to have bugs.
As before, do both system testing and acceptance testing but give priority to system testing as those bugs can earn you more credit.
Be careful when copying commands from the UG (PDF version) to the software as some PDF viewers can affect the pasted text. If that happens, you might want to open the UG in a different PDF viewer.

c) What bugs to report?

PE Dry Run (at v1.3)

You may report functionality bugs, UG bugs, and feature flaws.

You can also post suggestions on how to improve the product.
Be diplomatic when reporting bugs or suggesting improvements. For example, instead of criticising the current behavior, simply suggest alternatives to consider.

PE (at v1.4)

Report functionality bugs:

Do not post suggestions but if the product is missing a critical functionality that makes the product less useful to the intended user, it can be reported as a bug of type Type.FeatureFlaw. The dev team is allowed to reject bug reports framed as mere suggestions or/and lacking in a convincing justification as to why the omission of that functionality is problematic.

You may also report UG bugs.

d) How to report bugs

Post bugs as you find them (i.e., do not wait to post all bugs at the end) because bug reports created/modified after the allocated time will not count.

Using CATcher

Launch CATcher, and login to the correct profile (when CATcher asks, consent to creating a new repo):
- PE Dry Run: CS2103/T PE Dry run
- PE: CS2103/T PE
Post bugs using CATcher.

Not using CATcher

Post bug reports in the following repo you created earlier:
- PE Dry Run: ped
- PE: pe
The whole description of the bug should be in the issue description i.e., do not add comments to the issue.

e) Bug report format

Each bug should be a separate issue i.e., do not report multiple problems in the same bug report.
If there are multiple bugs in the same report, the dev team will select only one of the bugs in the report and discard the others.
When reporting similar bugs, it is safer to report them as separate bugs because there is no penalty for reporting duplicates while putting multiple bugs in the same report can reduce your bug count (see the previous point). But as submitting multiple bug reports take extra time, if you are quite sure they will be considered as duplicates by the dev team later, you can report them together, to save time.
Write good quality bug reports; poor quality or incorrect bug reports will not earn credit.
Use a descriptive title.
Give a good description of the bug with steps to reproduce, expected, actual, and screenshots. If the receiving team cannot reproduce the bug, you will not be able to get credit for it.
Assign exactly one severity.* label to the bug report. Bug reports without a severity label are considered severity.Low (lower severity bugs earn lower credit)

Bug Severity labels:

severity.VeryLow : A flaw that is purely cosmetic and does not affect usage e.g., a typo/spacing/layout/color/font issues in the docs or the UI that doesn't affect usage. Only cosmetic problems should have this label.
severity.Low : A flaw that is unlikely to affect normal operations of the product. Appears only in very rare situations and causes a minor inconvenience only.
severity.Medium : A flaw that causes occasional inconvenience to some users but they can continue to use the product.
severity.High : A flaw that affects most users and causes major problems for users. i.e., only problems that make the product almost unusable for most users should have this label.

When applying for documentation bugs, replace user with reader.

Assign exactly one type.* label to the issue.

Type labels:

type.FunctionalityBug: A functionality does not work as specified/expected.
type.FeatureFlaw: Some functionality missing from a feature delivered in v1.4 in a way that the feature becomes less useful to the intended target user for normal usage. i.e., the feature is not 'complete'. In other words, an acceptance-testing bug that falls within the scope of v1.4 features.
These issues are counted against the product design aspect of the project. Therefore, other design problems (e.g., low testability, mismatches to the target user/problem, project constraint violations etc.) can be put in this category as well.
Features that work as specified by the UG but should have been designed to work differently (from the end-user's point of view) fall in this category too.
type.DocumentationBug: A flaw in the documentation e.g., a missing step, a wrong instruction, typos

If you need to include < or > symbols in your bug report, use HTML equivalents for 'less than' and 'greater than' instead (i.e., < and > e.g., x < y instead of x < y).
Reason: CATcher and GitHub strips out content wrapped in < and >, for security reasons.

When in doubt, choose the lower severity: If the severity of a bug seems to be smack in the middle of two severity levels, choose the lower severity (unless much closer to the higher one than the lower one).

Reason: The teaching team follow the same policy when adjudicating disputed severity levels in the last phase of the PE.
As the tester, you might feel like you are throwing away marks by choosing a lower priority; but the lower priority has a lower risk of being disputed by the dev team, giving you (and the dev team) have a better chance of earning bonus marks for accuracy (which will be substantial).
To make your case stronger, state in the bug report why you think the bug might even qualify for a higher severity, while you actually chose the lower one.
In this section, there will be a mass identity check

Mass identity check

When the invigilator announces an identity check,

remove mask (if any)
turn towards the camera
move closer to the camera (but do not adjust the camera position) -- no need to show the student card
hold that pose until the invigilator has taken a screenshot and asks you to go back to the exam activity.

Please comply quickly as non-compliance can delay the exam for everyone. This is expected to take no more than 15 seconds.

PE Phase 1 - Part II Evaluating Documents [30 minutes]

This slot is for reporting documentation bugs only. You may report bugs related to the UG and the DG.
Only the content of the UG/DG PDF files (not the online version) should be considered.
For each bug reported, cite evidence and justify. For example, if you think the explanation of a feature is too brief, explain what information is missing and why the omission hinders the reader.
Do not report bugs that are not contained within in the UG and DG pdf files (e.g., bugs in the README.md).

You may visit the team's project on GitHub during this portion, for the purpose of verifying the accuracy of documentation e.g., to check if a diagram matches the code. You are also allowed to download and open the team's code in a code editor.

PE Phase 1 - Part III Overall Evaluation [15 minutes]

To be submitted via TEAMMATES. You are recommended to complete this during the PE session itself, but you have until the end of the day to submit (or revise) your submissions.
The TEAMMATES email containing the submission link should have reached you the day before the PE. If you didn't receive it by then, you can request it to be resent from this page.

Important questions included in the evaluation:

Q Quality of the product design,
Evaluate based on the User Guide and the actual product behavior.

Criterion	Unable to judge	Low	Medium	High
`target user`	Not specified			Clearly specified and narrowed down appropriately
`value proposition`	Not specified	The value to target user is low. App is not worth using	Some small group of target users might find the app worth using	Most of the target users are likely to find the app worth using
`optimized for target user`		Not enough focus for CLI users	Mostly CLI-based, but cumbersome to use most of the time	Feels like a fast typist can be more productive with the app, compared to an equivalent GUI app without a CLI
`feature-fit`		Many of the features don't fit with others	Most features fit together but a few may be possible misfits	All features fit together to for a cohesive whole

Q Compared to AddressBoook-Level3 (AB3), the overall quality of the UG you evaluated is,
Evaluate based on fit-for-purpose, from the perspective of a target user. For reference, the AB3 UG is here.

Significantly lower
Slightly lower
Similar
Higher

Q Compared to AB3, the overall quality of the DG you evaluated is,
Evaluate based on fit-for-purpose from the perspective of a new team member trying to understand the product's internal design by reading the DG. For reference, the AB3 DG is here.

Significantly lower
Slightly lower
Similar
Higher

Q If the implementation effort required to create AB3 from scratch is 10, the estimated implementation effort of this team is, [0..20] e.g., if you give 8, that means the team's effort is about 80% of that spent on creating AB3. We expect most typical teams to score near to 10.

Do read the DG appendix named Effort, if any.
Consider implementation work only (i.e., exclude testing, documentation, project management etc.)
Do not give a high value just to be nice. Your responses will be used to evaluate your effort estimation skills.

Q [Optional] Concerns or any noteworthy observations about the product you evaluated

PE Phase 2: Developer Response

Deadline: Mon, Nov 14th 2359

This phase is for you to respond to the bug reports you received.

This is not a bug; it's fine the way it is! Everytime you say that in response to a legitimate bug report, you are saying something about your own standard of work as a software engineer. Don't set that standard too low, which goes totally against what we are trying to achieve in this module.

Yes, that needs fixing! For each bug report you receive, if you think a software engineer who takes pride in their own work would say "yes, that needs fixing", accept it graciously as a bug, even if you can come up with some argument for it not being a bug. Your professional integrity is more valuable than a measly fraction of a mark that you might lose.
Even when you still want to defend the behavior being labelled as problematic, instead of pretending that the behavior was a deliberate choice to begin with, you can say something like,

"Thx for raising this. Indeed it didn't occur to us. But now that we have thought about it, we still feel .."

Some bugs are 'expected'. Given the short time you had for the tP and your inexperience in SE team projects, this work is not expected to be totally bug free. The grading scheme factors that in already -- i.e., your grade will not suffer if you accept a few bugs in this phase.

Bonus marks for high accuracy rates!

You will receive bonus marks if a high percentage (e.g., some bonus if >60% substantial bonus >80%) of bugs are accepted as triaged (i.e., the eventual type.*, severity.*, and response.* of the bug match the ones you chose).

It's not bargaining!

When the tester and the dev team cannot reach a consensus, the teaching team will be select either the dev team position or the tester position as the final state of the bug, whichever appear to be closer to being reasonable. The teaching team will not come up with our own position, or choose a middle ground.

Hence, do not be tempted to argue for an unreasonable position in the hope that you'll receive something less than asked but still in your favor e.g., if the tester chose severity.High but you think it should be severity.Medium, don't argue for severity.VeryLow in the hope that the teaching team will decide a middle ground of severity.Low or severity.Medium. It's more likely that the teaching team will choose the tester's position as yours seems unreasonable.

More importantly, this is not a bargaining between two parties; it's an attempt to determine the true nature of the bug, and your ability to do so (which is an important skill).

Duration: The review period will start around 1 day after the PE and will last for 2-3 days (exact times will be announced later). However, you are recommended to finish this task ASAP, to minimize cutting into your exam preparation work.

Bug reviewing is recommended to be done as a team as some of the decisions need team consensus.

Instructions for Reviewing Bug Reports

Don't freak out if there are lot of bug reports. Many can be duplicates and some can be false positives. In any case, we anticipate that all of these products will have some bugs and our penalty for bugs is not harsh. Furthermore, it depends on the severity of the bug. Some bug may not even be penalized.
Nit-picking is a good sign: If you receive a lot of nit-picking type of bugs that make you roll your eyes, it means testers were unable to find more serious bugs. That's a good thing.
Not exactly zero-sum: As mentioned earlier, the penalty for having a specific bug is not the same as the reward for reporting that bug (it's not a zero-sum game). For example, the reward for testers will be higher (because we don't expect the products to have that many bugs after they have gone through so much prior testing)

Penalty for a minor bug (e.g., ) is unlikely to make a difference in your final grade, especially given that the penalty applies only if you have more than a certain amount of bugs.

For example, in a typical case a developer might be assigned 5+ severity.VeryLow bugs before the penalty even starts affecting their marks.

Accordingly, we hope you'll accept bug reports graciously (rather than fight tooth-and-nail to reject every bug report received) if you think the bug is within the ballpark of 'reasonable'. Those minor bugs are really not worth stressing/fighting over.

Using CATcher

CATcher does not come with a UG, but the UI is fairly intuitive (there are tool tips too). Do post in the forum if you need any guidance with its usage.
Also note that CATcher hasn't been battle-tested for this phase, in particular, w.r.t. multiple team members editing the same issue concurrently. It is ideal if the team members get together and work through the issues together. If you think others might be editing the same issues at the same time, use the Sync button at the top to force-sync your view with the latest data from GitHub.

Go to CATcher Web app, and login to the profile CS2103/T PE. It will show all the bugs assigned to your team, divided into three sections:
1. Issues Pending Responses - Issues that your team has not processed yet.
2. Issues Responded - Your job is to get all issues to this category.
3. Faulty Issues - e.g., Bugs marked as duplicates of each other, or causing circular duplicate relationships. Fix the problem given so that no issues remain in this category.
Respond to the bug reports shown.

Not using CATcher

You must use CATcher. You are strictly prohibited from editing PE bug reports using the GitHub Web interface as it can render bug reports unprocessable by CATcher, sometimes in an irreversible ways, and can affect the entire class. Please contact the prof if you are unable to use CATcher for some reason.

If a bug seems to be for a different product (i.e. wrongly assigned to your team), let us know ASAP.
If the bug is reported multiple times,
- Mark all copies EXCEPT one as duplicates of the one left out (let's call that one the original) using the A Duplicate of tick box.
- For each group of duplicates, all duplicates should point to one original i.e., no multiple levels of duplicates, and no cyclical duplication relationships.
- If the duplication status is eventually accepted, all duplicates will be assumed to have inherited the type.* and severity.* from the original.
If you cannot reproduce the bug based on the info given by the tester you are still expected to make a reasonable attempt to go beyond the information provided by the tester to reproduce the bug, if there is clear evidence of something wrong.
For example, the screenshot in the bug report clearly shows an error message that should not appear, but you can't reproduce the error message based on the info given by the tester. Perhaps the error was caused by something else the tester did although the tester didn't realize it is connected to the error. In this case, based on the error message, you might be in a better position to figure out the real cause of the error. If you don't, the decision can go against you in a later phase if either the tester or the moderator figures out how to reproduce the error and the moderator decides that it is something you should have been able to figure out yourself.

Apply one of these labels (if missing, we assign: response.Accepted)

Response Labels:

response.Accepted: You accept it as a valid bug.
response.NotInScope: It is a valid issue but not something the team should be penalized for e.g., it was not related to features delivered in v1.4.
response.Rejected: What tester treated as a bug is in fact the expected and correct behavior (from the user's point of view), or the tester was mistaken in some other way. Note: Disagreement with the bug severity/type given by the tester is not a valid reason to reject the bug.
response.CannotReproduce: You are unable to reproduce the behavior reported in the bug after multiple tries.
response.IssueUnclear: The issue description is not clear. Don't post comments asking the tester to give more info. The tester will not be able to see those comments because the bug reports are anonymous.

Only the response.Accepted bugs are counted against the dev team. While response.NotInScope are not counted against the dev team, they can earn a small amount of consolation marks for the tester. The other three do not affect marks of either the dev team or the tester, except when calculating bonus marks for accuracy.

If you disagree with the original bug type assigned to the bug, you may change it to the correct type.

Type labels:

type.FunctionalityBug: A functionality does not work as specified/expected.
type.FeatureFlaw: Some functionality missing from a feature delivered in v1.4 in a way that the feature becomes less useful to the intended target user for normal usage. i.e., the feature is not 'complete'. In other words, an acceptance-testing bug that falls within the scope of v1.4 features.
These issues are counted against the product design aspect of the project. Therefore, other design problems (e.g., low testability, mismatches to the target user/problem, project constraint violations etc.) can be put in this category as well.
Features that work as specified by the UG but should have been designed to work differently (from the end-user's point of view) fall in this category too.
type.DocumentationBug: A flaw in the documentation e.g., a missing step, a wrong instruction, typos

If you disagree with the original severity assigned to the bug, you may change it to the correct level.

Bug Severity labels:

severity.VeryLow : A flaw that is purely cosmetic and does not affect usage e.g., a typo/spacing/layout/color/font issues in the docs or the UI that doesn't affect usage. Only cosmetic problems should have this label.
severity.Low : A flaw that is unlikely to affect normal operations of the product. Appears only in very rare situations and causes a minor inconvenience only.
severity.Medium : A flaw that causes occasional inconvenience to some users but they can continue to use the product.
severity.High : A flaw that affects most users and causes major problems for users. i.e., only problems that make the product almost unusable for most users should have this label.

When applying for documentation bugs, replace user with reader.

If you need the teaching team's inputs when deciding on a bug (e.g., if you are not sure if the UML notation is correct), post in the forum. Remember to quote the issue number shown in CATcher (it appears at the end of the issue title).

Guidelines for bug triaging

General:

If a bug report contains multiple bugs (i.e., despite instructions to the contrary, a tester included multiple bugs in a single bug report), you have to choose one bug and ignore the others. If there are valid bugs, choose from valid bugs. Among the choices available, choose the one with the highest severity (in your opinion). In your response, mention which bug you chose.
What bugs can be considered duplicates? It is up to the dev team to prove conclusively that a bug is a duplicate. If the proof is not convincing enough, they will be considered as 'not duplicates'. Only the following cases can be considered duplicates:
(a) The exact same bug reported multiple times.
(b) Multiple buggy behaviors that are actually caused by the same defect and cannot be fixed independently (i.e., fixing one fixes the others automatically).

In real projects, similar bugs (e.g., the same typo in multiple places) tend to get combined into a single issue/PR; in the PE, we have to keep independently-fixable things as separate bugs, to avoid complications in grading. After all, having the same typo in two places is not exactly the same as having it in only one place.
If an independently-fixable yet similar problem appears in more than five distinct places, get our permission to combine them as one bug (in which case we'll require you to increase the severity to match the frequency of the bug).

How to prove that something is response.NotInScope: In general, a flaw (e.g., a missing feature, a sub-optimal design of a feature, a known bug) can be considered NotInScope if rectifying it is less important (based on the value/effort considerations) than the work that has been done already (because it is fine to delay lower priority work until future iterations).
In addition, the following (at least one) need to be satisfied:
- The UG specifies it as not supported or coming in a future version.
- The user cannot attempt to use the missing feature or when the user does so, the software fails gracefully, possibly with a suitable error message i.e., the software should not crash.

~~You can reject bugs that you from AB3.~~ Even bugs inherited from AB3 are counted.
If you can't reproduce the bug based on the exact steps given in the bug report but there is some indication of 'something wrong' (e.g., the bug report cites a behavior that shouldn't happen under normal use), you are expected to make a reasonable effort to reproduce the bug yourself by experimenting with the information provided in the bug report. Reason: it will go against you if claim that the bug is not reproducible and yet someone else (the tester or a tutor) manages to conclusively prove the existence of the bug in a later phase.

Functionality bugs

Problems caused by extreme user behaviors:
- If the problems can only happen in case of deliberate sabotage (e.g., user entered a 30-digit telephone number), in which case it should not be considered a bug.
  However, if such input can be entered by a user mistake, they should not cause harm e.g., such mistakes should not crash the app, corrupt the data, or make it unusable.
- Problems caused by integer overflows -- apply the guideline in the previous point.
Problems caused by very long input values: When a user input is unusually long e.g., a very long name, a very large number, it can cause problems e.g., the UI layout can get messed up, some part of it might get cut off.
- These can be considered cosmetic issues (i.e., severity.VeryLow) of type.FunctionalityBug.
  However, if the problem can hinder the user (e.g., not seeing the last part of a very long name might not hinder the user but it does hinder the user if only the first few characters of the name is shown), the severity can be Low or higher.
- It is also fine to restrict the size/length of inputs as long as the limits are reasonable. For example, limiting the phone number to 8 digits is not reasonable unless you are targeting users whose telephone numbers are not longer than 8 digits.
Use of symbols in input values: It is acceptable to disallow certain characters in input values if there is a justification (e.g., because using those symbols in an input value makes the command harder to parse), but they can still be considered FeatureFlaw bugs if they cause inconvenience to the user. For example, disallowing s/o in a person name because / is used as a command delimiter can cause a major problem if the input is expected to match the legal name of the person.
Mismatch between the UG and the feature: If the feature behavior needs to be changed, it is either a type.FunctionalityBug or type.FeatureFlaw. But if it the UG that needs to be updated, it is a type.DocumentationBug.
Handling manual edits to the data file: AB3 UG specifies the current level of support for manually editing the data file. At least that level of support should be supported in the new product as well.

Feature flaws

Missing features and problems in how a feature is designed are considered feature flaws i.e., type.FeatureFlaw.
Feature flaws can be claimed as NotInScope, if they qualify as per rules explained above, except for these cases:
- if fixing the feature flaw is essential for the app to be reasonably useful
- if the feature is implemented to work in a certain way but it could have been implemented to work in a better way (from the end-user's point of view) without much additional effort
Bugs related to duplicate detection: Duplicate detection (e.g., detecting if two persons in the address book are the same) is not trivial; often, detecting only the exact string/value matches is not enough. For example, John Doe and john doe are likely to be the same person. Similarly, extra white space (e.g., the user typed an extra space between the two names) is unlikely to mean they are two different persons. Typically, it is best if you can give a warning in such near match cases so that the user can make the final decision.
If you app has a duplicate detection feature, make sure its limitations are made clear to the user so that users are not led to believe that duplicates are being detected while many potential duplicate cases go undetected. Otherwise, it can be considered a type.FeatureFlaw.
Overzealous input validation: It is better to warn rather than to block when inputs are not compliant with the expected format, unless accepting such inputs can hinder the operations of the software. Allowing such flexibility can in turn allow the user to use the software in ways you didn't even anticipate while overzealous rejection of inputs can annoy the user:
Example 1: While your software allows only one phone number in input values, a user might want to input 1234 5678 (HP) 1111-3333 (Office) -- blocking that input might not add any value but allowing it does.
Example 2: A user might want to enter an appointment in the past, just for record keeping purposes.
Such overzealous input blocking can be considered a type.FeatureFlaw.
However, it is fine (and recommended) to show a warning for such inputs to guard against the deviation being a mistake rather than intentional.
Lack of proper handling (either blocking or warning) for potentially invalid inputs can be considered a type.FeatureFlaw bug too. Not detecting easy-to-detect incorrect flags (e.g., user entered /t but it should be /tag) is a FeatureFlaw too.
Specificity of error message: Error messages can be correct but not specific enough (e.g., it says the input is 'invalid' without giving the reason, or gives too many possible reasons without pointing out the specific reason). These cases can be considered type.FeatureFlaw.
Calling an invalid value a 'format error' and vice versa is a severity.Low bug e.g., if a date input is required to be in YYYY-MM-DD format, 2021-13-28 is a format error (reason: MM should be in 1..12) but 2021-02-30 is an invalid input (reason: February doesn't have 30 days). However, issuing a 'Invalid date or incorrect format' error message for such a case (i.e., covering both bases) is acceptable if differentiating between the two qualifies for NotInScope.
Unnecessarily complicated (or hard-to-type) command formats can be considered a type.FeatureFlaw as it is expected that the input formats will be optimized to get things done fast. Some examples: using very long keywords when shorter ones do, or making keywords case-sensitive when there is no need for it, using hard to type special characters in the format when it is possible to avoid them.
Case sensitivity: In general, case sensitivity of something should follow the case sensitivity of the real world entity it represents e.g., as person names are not case sensitive in the real world, they shouldn't be case sensitive in the app either. The same applies for search key words. Incorrect case sensitivity can be considered a FeatureFlaw.

Documentation bugs

Broken links: Severity can be Low or Medium depending on how much inconvenience they cause to the reader.
Extra white space introduced by the PDF conversion: Not counted as bugs unless it hinders the reader. Cases such as a diagram being split between pages are considered bugs, because they hinder the reader.
'Hinder' the reader? Don't interpret 'hinder' as 'impossible to read'. Even formatting issues such as too much/little padding, font size, alignment, inconsistencies, etc. can 'hinder' the reader in the sense they can slow down the reader or require the reader to put more effort than necessary. Those things that 'need to be fixed' are still bugs but of lower severities (depending on how much they hinder the reader -- most likely severity.VeryLow if the issue is purely cosmetic).
UML notation variations caused by the diagramming tool: Can be rejected if not contradicting the standard notation (as given by the textbook) i.e., extra decorations that are not misleading.
Omitting optional notations is not a bug as long it doesn't hinder understanding.
Details missing from a diagram: In a similar vein to the above, omitting details from a diagram is OK if it does not mislead/hinder the reader.
Forgetting to include something is not the same as a deliberate decision to omit something in order to simplify the diagram e.g., the latter could accompany a note to the reader to mention which/some parts have been omitted, if it is worthwhile for the reader to know the omission.
While many UML notations are optional, haphazard omissions without a good reason can affects consistency which affects readability e.g., it can be considered a minor bug if a sequence diagram omits an activation bars in some places but not in other places and yet the omission doesn't make the diagram any easier to read.
Nitty-gritty details missing from the UG is not a bug long as the user is informed of those details using other means such as error messages or in-app help.
Minor typos and grammar errors: These are still considered as severity.VeryLow type.DocumentationBug bugs (even if it is in the actual UI) which carry a very tiny penalty.
Severity of bugs related to missing requirements (e.g., missing user stories)? Depends on the potential damage the omission can cause. Keep in mind that not documenting a requirement increases the risk of it not getting implemented in a timely manner (i.e., future developers will not know that feature needs to be implemented).
Unfulfilled NFRs: If the DG mentions non-functional requirements that are not met by the product, it can be a severity.Low DocumentationBug if the the NFR was unreasonable in the first place. Otherwise, it can be type.FeatureFlaw bug.
Details in the diagram too small: This is usually a symptom of having too much info in the diagram. A common example is sequence diagrams showing low-level details of multiple components (recommended: A sequence diagram should show internal interactions of at most one component i.e., treat other components as back boxes).
While the reader can zoom to see smaller details, this can still be considered a cosmetic issue (i.e., severity.VeryLow).

Decide who should take responsibility for the bug. Use the Assignees field to assign the issue to that person(s). There is no need to actually fix the bug though. It's simply an indication/acceptance of responsibility. If there is no assignee, we will distribute the penalty for that bug (if any) equally among all team members e.g., if the penalty is -0.4 and there are 4 members, each member will be penalized -0.1.
- If it is not easy to decide the assignee(s), we recommend (but not enforce) that the feature owner should be assigned bugs related to the feature, Reason: The feature owner should have defended the feature against bugs using automated tests and defensive coding techniques.
- It is also fine to not assign a bug to anyone, in which case the penalty will be divided equally among team members.

As far as possible, choose the correct type.*, severity.*, response.*, assignees, and duplicate status even for bugs you are not accepting. Reason: your non-acceptance may be rejected in a later phase, in which case we need to grade it as an accepted bug.
If a bug's 'duplicate' status was rejected later (i.e., the tester says it is not really a duplicate and the teaching team agrees with the tester), it will inherit the type/severity/assignees from the 'original' bug that it was claimed to be a duplicate of.
Justify your response. For all of the following cases, you must add a comment justifying your stance. Testers will get to respond to all those cases and will be considered by the teaching team in later phases (when resolving disputed bug reports).
If you don't provide a justification and the tester disagrees with your decision, the teaching team will have no choice but to rule in favor of the tester.
- downgrading severity
- non-acceptance of a bug
- changing the bug type
- non-obvious duplicate
FAQs:
1. Q: What if the bug is real but the tester used the wrong label (e.g., used the wrong type.*). Can we reject that bug?
  A: A bug is a bug irrespective of the label used. Instead of rejecting, rectify the label.
2. Q: What if the bug the tester reported is legit but the expected behavior tester suggested is not correct?
  A: You should accept the bug but state that you disagree with the expected/suggested behavior. Reason: the main job of the tester is to detect bugs; suggesting a solution is optional.

You can also refer to the below guidelines:

PE Phase 3: Tester Response

Start: Within 1 day after Phase 2 ends.

While you are waiting for Phase 3 to start, comments will be added to the bug reports in your /pe repo, to indicate the response each received from the receiving team. Please do not edit any of those comments or reply to them via the GitHub interface. Doing so can invalidate them, in which case the grading script will assume that you agree with the dev team's response. Instead, wait till the start of the Phase 3 is announced, after which you should use CATcher to respond.

Deadline: Thu, Nov 17th 2359

In this phase you will get to state whether you agree or disagree with the dev team's response to the bugs you reported. If a bug reported has been subjected to any of the below by the receiving dev team, you can record your objections and the reason for the objection.
- not accepted
- severity downgraded
- bug type changed
- bug flagged as duplicate (Note that you still get credit for bugs flagged as duplicates, unless you reported both bugs yourself. Nevertheless, it is in your interest to object to incorrect duplicate flags because when a bug is reported by more testers, it will be considered an 'obvious' bug and will earn slightly less credit than otherwise)

Don't feel upset if the dev team did not totally agree with most of the bugs you reported. That is to be expected, given you had very short time to make those bug decisions while the dev team had a lot more time to deliberate about them. Some may have given unreasonable (in your opinion) arguments against your bug reports; not to worry, just give your counter-arguments and leave it to the teaching team to decide (in the next phase) which position is more reasonable.

However, if the dev team's argument is not too far from 'reasonable', it may be better to agree than disagree.
Reason: an incorrect counter argument at this phase will lower your accuracy more than an incorrect decision made during the testing phase (because you now have more time to think about the bug) i.e., changing your position after you had more time to think of it and after having seen more information is encouraged, compared to sticking to your initial position 'no matter what'.

If you would like to revise your own initial type/severity in response to the team's inputs, you can state that in your explanation e.g., you rated the bug severity.High and the team changed it to severity.Low but now you think it should be severity.Medium (do not change the original labels yourself though).
You can also refer to the below guidelines:

If you do not respond to a dev response, we'll assume that you agree with it.
Procedure:

Using CATcher

When the phase has been announced as open, login to CATcher as usual (profile: CS2103/T PE).
For each issues listed in the Issues Pending Responses section:
- Click on it to go to the details, and read the dev team's response.
- If you disagree with any of the items listed, tick on the I disagree tick box and enter your justification for the disagreement, and click Save.
- If you are fine with the team's changes, click Save without any other changes upon which the issue will move to the Issue Responded section.
No action is required for the bugs the team accepted exactly as you reported them (i.e., no change to type or severity). They are shown in CATcher for your reference only.

Not using CATcher

FAQs:
1. Q. What if the team rejected my bug report without giving a reason?
  A. You can disagree with the rejection, and the teaching team will rule in your favor in the next phase.

PE Phase 4: Tutor Moderation

In this phase tutors will look through all dev responses you objected to in the previous phase and decide on a final outcome.
In the unlikely case we need your inputs, a tutor will contact you.