What metrics will you use to measure the success of YouTube Search? [Google PM Question]

Interviewer: How would you measure the success of YouTube search?

Shailesh: So let’s first understand YouTube search and its functionality so that we are on the same page.
Basically, YouTube Search is an inbuilt search engine inside YouTube right? And it helps the user to find videos and channels related to their intent.
We can do text-based search, we can do voice-based search etc. YouTube Search can understand our intent and based on that, it gives the most relevant and personalised results to us. This is what a YouTube Search does.
Interviewer: This is well-defined.
Shailesh: I will also briefly touch upon the goal of YouTube Search for different stakeholders and for overall YouTube, why it is important because it will help us to pinpoint the responsibilities of YouTube Search for the different stakeholders involved.
  • For Users: The goal of YouTube search is to clearly understand their intent and give the most relevant results in a very quick and easy manner.
  • For Creators: YouTube aims to make their content discoverable to the correct set of audiences. Then only there would be better engagement on the platform.
  • For YouTube: The goal is to make users and creators happy so that users should be able to discover content that’s most relevant to them. This will enable users to keep coming and watching the videos and will enable YouTube to achieve its goals.
Interviewer: How?

Shailesh: YouTube's goal is to make the platform very easy to use to watch videos online. That is the discovery of the videos is the first step right? If users can find the videos which they want to watch easily via search, they will keep coming and watching videos, which will increase the watch time on the platform and in turn the ad revenue. Now we also know how the goal of YouTube search is aiding the overall goal of the platform.

To achieve these goals, these are the actions which need to be done on the platform. 
  1. Users need to initiate the search easily
  2. Not only do Users need to initiate the search but also complete their search
  3. After completing the search, users need to click on the search results, if they are not clicking then something is not right, either may be search query understanding or ranking of the results
  4. After clicking, users need to watch for a considerable time, if the watch time is low via search it means, something is wrong, and we are not able to give the right set of videos to users. This is because the search is for intent-heavy users, they know what they are going to watch. If the view duration is low, that means something is wrong.

Now to achieve the goal, these are the actions that are needed, let’s map it to North Star Metrics and some L0 and L1 metrics.

So the Northstar metric would be the Watch Time contribution from Search on the Youtube Platform. That will be our North Star Metric. If that’s going down, then something or the other is going wrong and this North Star metric is also in line with Youtube's North Star metric. More Watch Time means more engagement which implies more revenue.

Now that we have found out the North Star Metric, it’s not enough, we have to break down the North Star Metric into Subsequent L0 metrics, which I will be tracking if I am a YouTube Search Product Manager. This is a super critical metric to understand and these L0 metrics should contribute to our North Star Metric.

  1. The first metric would be % Search Session, which means out of all the sessions which are happening on Youtube, in how many sessions at least 1 search has been made. This will indicate the adoption of the YouTube search.
  2. The second metric would be the average number of Searches per Search session. This means if this is the search session, then on average how many searches the user is doing? To move the North Star metric, increasing the Number of Sessions is not enough, users should be doing more searches.
  3. Another super critical metric is the YouTube Search abandonment Rate, if a user has done the search but is not clicking on any video. This should be low. If that is high that means in more searches, users are not clicking the search results. This indicates that the search results are not good, either YouTube is not able to understand the user's intent or YouTube is not putting the best search results on the top. So Increasing the Search session or the number of searches per search session is not enough, the YouTube search abandonment Rate should also be low.
  4. Finally, AVD i.e. Average View Duration of Video via Search. This is also one of the metrics we should measure.
All these 4 metrics are super important and will directly affect our North Star Metric.

Apart from the L0 Metrics, I will also measure some L1 metrics like

  1. Zero Search Results, which means for how many search queries are we giving 0 results, why it is important because tons of languages are there in the world, and it may happen that YouTube is not able to understand some languages which leads to 0 results.
  2. Another metric which I will track is Auto Suggest usage. So in 100 Search, how many searches are fired via autosuggest? It is important because we have defined the goal of YouTube search to enable users to find the relevant results easily and quickly. By using autosuggest, we are saving the user's typing effort and we are assisting users in their search.
  3. Finally, I will also measure the Median Scroll Depth, which means when users fire the search query and get the results, how much scroll do they have to do before doing the first click. That will tell how good our ranking of search results is.
Let’s see some of the instances of Metrics evaluation.
Metrics for YouTube


Suppose that Searches per search session are going up. What does it mean, is it good or bad?

So it will depend, for example, it can happen that the searches per session are going up because our search is not giving the results in the first go, users have to do multiple tries before getting the relevant results.
In that case, we will not blindly check if Searches per search session are going up so it will be good, we will see another supporting metric like abandonment rate, is this also going up?
If the abandonment rate is also going up, then there is some problem which requires immediate attention.

Important Links
-- Technomanagers

Comments

Popular posts from this blog

Guesstimates for Product Interviews, Tips to ask clarification Questions

Design Uber for Kids. [Product Management Interview Question]

System Design Concepts for Product Managers — Part 1