The Product Side

Product manager interview assignment — LearnWorlds, case study

This post is part of a series with answers to a real interview assignments. This one is for a product manager position at LearnWorlds. The assignment sums up to:

Step 1
A. List a few disadvantages of video as a learning tool.
B. Propose KPIs to evaluate the performance of your offering.

Step 2
Propose a few improvements regarding making videos more social and describe some user stories.

Step 3
Create a prototype.

Step 1

Cons of video learning

A. Some of the shortcomings of video as a learning tool are:

1. It is not interactive. Students see a pre-baked version of the class and cannot ask questions in real time. There is no interaction with other students too, as would be in a typical class environment.

2. It is difficult to delve into detail. Compared to a book, a video cannot display much information (without becoming too long to watch), so it is more suitable for introducing a subject and not for analyzing it[1].

3. It is difficult to locate specific context. A book has a table of contents, a detailed index and text search capability[2]. Even with chapter indexes, you may still have to watch a few minutes of content before locating the part you are interested in (it gets even worse, if you start at the wrong chapter).

4. Requires high bandwidth, both on the sender (i.e. streaming platform) side and on the receiver (i.e. student) side.

5. It is difficult to produce. For starters, the instructor would better learn about photography, lighting and video capturing. Then she should prepare and follow a script and do a lot of editing in order to create capturing content.

Candidate KPIs

B. We assume that the case is:

• Our company (say “LW”) is employing instructors (say “Ian” the instructor) to create videos.
• Another company (say “Acme”, the customer), is purchasing the video training services, in order to train their employees (say “Mary” the manager, i.e. the end user).

In order to evaluate the performance of LW’s offering, can use the following KPIs:

Supply side (i.e. instructors)
1. Number of available courses.
2. Number of available instructors.
3. Number and duration of available videos.
4. Cost of producing the videos.
5. Cost of instructors (salary, benefits, etc.)

Demand side (i.e. customers / users)
6. Customer subscriptions (companies / customers).
7. Percentage of cancellations.
8. Total rofit (total / per video / per instructor / per customer).
9. Average profit (total / per video / per instructor / per customer).

Ian as an instructor, may employ a different set of KPIs to evaluate the performance of his video courses
10. Hours of content consumed (total / per user).
11. Views (total / per user / per video)
12. Enrolments (total / per user / per video)
13. Watch time per video, vs nominal video duration, i.e. if a user keeps on skipping parts of the video then the actual watch time will be less than the nominal video duration.
14. Video ratings (assuming we allow the users to rate videos[3]): avg. rating and number of total ratings.
15. Comments / questions (assuming we allow the users to post comments / questions): total / avg. per user / avg. per video / likes — dislikes.

ACME has a different point of view. The main purpose of ACME doing any training is to instil new knowledge / skills to their employees (i.e. students) regardless of the method used for training. Therefore, the KPIs ACME will use to test if learning via video is a good idea, may be similar to the KPIs they use to test training in general. If video courses manage to either a) get better results or b) get the same results in shorter time, compared to other methods of training then they will have demonstrated an advantage.

With all this in mind some of the KPIs ACME may use to evaluate the performance of video courses/trainings are:

16. Certification pass rate and score (for example employees are expected to sit a certification exam after completing the training).
17. Training completion percentage rate.
18. Average time to completion.
19. Average cost to get a certification.

Step 2

Before coming up with candidate ideas, we may need to:
a. Discuss with the team in order to understand better the problem we are facing and which are the most important challenges.
b. Listen to customer feedback, either from support tickets, from mentions in forums or other places on the Internet (reddit, etc.).
c. See if the is any relevant research available.
d. See what solutions, our competitors have implemented so far and what has worked (or not) in real life.

Ideas

Let’s begin by adding some context around what it means to make a video course more “social”. This:

a. May involve creating more interactions with:
i. The instructor.
ii. The group of people watching the same video (let’s call them “classmates”)
iii. Other: the student’s friends / acquaintances in general (excluding the classmates).

b. May help with:
i. Facilitating the learning process of the student.
ii. Recognizing the student’s achievement of attending / completing the course.
iii. Letting others suggest new/similar courses to the student.

In this respect, we may examine the following ideas.

Note: I intentionally did not include ideas such as having a user forum, badges, sharing the certificate of completion on social media etc., as they are fairly common.

User stories

A. Following are some user stories on the first idea: “raise hand”.

(Note: instead of the “as a <role>, I <action>, in order to <justification>” format, we could use the “Given / when / then” format which can be mapped directly to behaviour driven tests. In this case, however, the user stories would be much longer.)

1. As a student, I can raise my hand as I am watching the video and send the instructor a question, in order to better understand the course.
Acceptance criteria:
• My question will be plain text (i.e. no formatting such as bold, italics, bullets).
• My question can contain attachments and external links.

2. As a student, I can mark my question as “private”, in order not to allow any other students to see it (only the instructor can see it). By default, a question is considered public.

3. As a student, I can see any unanswered questions I have sent to the instructor and delete them, so I do not ask questions I have found the answer for.
Conditions:
• I cannot delete a question the instructor has provided an answer for (or is starting to write one).

4. As a student, I can see the unanswered questions I have sent to the instructor and edit them, in order to better phrase my question.
Conditions:
• I cannot edit a question the instructor has provided an answer for (or is starting to write one).

5. As an instructor, I see a list of all the questions students asked when raising their hand, in order to answer them.
Acceptance criteria:
• The instructor should be able to sort the list on question text, time of video, student name, date posted. Sorting should be done on ascending or descending order.
• The instructor should be able to filter (search) the list on: question text, time of video from — to, student name, dated posted from — to.
• The instructor sees results in pages of 20 items (pagination). The instructor can change the page size to 10 / 20 / 50 records.

6. As an instructor, I can write an answer to a student question who has raised her hand.
Acceptance criteria:
• My response will be plain text (i.e. no formatting such as bold, italics, bullets).
• My response can contain attachments and external links.

7. As an instructor, I can provide the same answer to many student questions simultaneously, in order to save me time.

8. As a student, I receive an email and a push notification whenever the instructor answers one of my questions, in order not to miss the instructor’s reply.

9. As a student, as I am watching the video, I can see the points where other students have raised their hands, in order to gain insights by their questions.

10. As a student, I can see & search the public questions that other users have posted.

Step 3

We will create a prototype for user story #9 above, using Balsamiq. Our prototype is for the web app on a desktop screen.

As the user is watching the video, a yellow hand appears on top of the video, when a question was raised (similar to the way that likes appear inside a live Facebook video). The question text (and answer, if available), automatically appears (fade-in effect) in the “Questions” tab on the right.

1. We can have the video pause when the user clicks on the yellow raise hand, so that the student can read the question and answer (scrolling is supported). This approach may also be used on mobile.

If we want to avoid the click, we could pause the video If the student moves the mouse pointer inside the “Questions” tab. The video resumes automatically when the student moves the mouse pointer back inside the video window. (see https://stackoverflow.com/questions/47381793/html-js-how-to-play-stop-video-on-hover for a way to do so). However, this is not really intuitive and cannot be replicated on mobile.

2. In any other case, the question stays visible for 4 seconds and then fades out, so as not to distract the student attention from the video.

Apart from the happy path described so far, we have to think of edge cases, to see how our solution would perform.

Edge case: too many questions?

Note: there might be the case when you have many questions by various students, done in the same point of the video. Then the yellow hand and the respective question would appear on and off, and it would seem like they were blinking, which is not good. In this respect we could try to group questions:

Option 1 — grouping
If there are more than one questions asked at the same point of the video (i.e. with less than 10 seconds of each other) then we treat them as a group and show them one below the other. The total number of questions appears on top and the student can scroll down to see the other questions. The first and the last question of the group cannot be more than 30 seconds apart, i.e. in case a second group will be created.

Example A: Q1 is asked at 01:20, Q2 is asked at 01:24 and Q3 is asked at 01:50. Q1 and Q2 form a group and Q2 is standalone.

Example B: Q1 is asked at 01:20, Q2 is asked at 01:29, Q3 is asked at 01:39, Q4 is asked at 01:49, Q5 is asked at 01:59. Q1, Q2, Q3, Q4 form a group and Q5 does not belong in this group, even though it is asked just 9 seconds after Q4.

However, this is just a temporary solution, as questions may still become too many; anything more than 5 questions at the same point would make this solution impractical.

Option 2 — limiting
Another approach would be to always show only up to the two latest questions. That is, for every 30 seconds of video, we find the questions asked, we order them by date posted descending and show the first two. For the rest of the questions, we could have a link (like “see more questions”) that takes the student to a separate page where he can see them all, at his own pace.

Endnotes:

[1] The optimal video length around 10 mins and preferably no more than 30 mins, source: http://site.kaltura.com/rs/984-SDM-859/images/The_State_of_Video_in_Education_2015_a_Kaltura_Report.pdf?aliId=144122175

[2]This may be improved by letting people do a text search inside the video transcript and showing them the respective point in the video.

[3] Alternatively we may allow users to like/dislike a video, such as YouTube and Netflix do.