What metrics will you use to measure the success of YouTube Search? [Google PM Question]

By -
Interviewer: How would you measure the success of YouTube search?

Shailesh: So let’s first understand YouTube search and its functionality so that we are on the same page.
Basically, YouTube Search is an inbuilt search engine inside YouTube right? And it helps the user to find videos and channels related to their intent.
We can do text-based search, we can do voice-based search etc. YouTube Search can understand our intent and based on that, it gives the most relevant and personalised results to us. This is what a YouTube Search does.
Interviewer: This is well-defined.
Shailesh: I will also briefly touch upon the goal of YouTube Search for different stakeholders and for overall YouTube, why it is important because it will help us to pinpoint the responsibilities of YouTube Search for the different stakeholders involved.
  • For Users: The goal of YouTube search is to clearly understand their intent and give the most relevant results in a very quick and easy manner.
  • For Creators: YouTube aims to make their content discoverable to the correct set of audiences. Then only there would be better engagement on the platform.
  • For YouTube: The goal is to make users and creators happy so that users should be able to discover content that’s most relevant to them. This will enable users to keep coming and watching the videos and will enable YouTube to achieve its goals.
Interviewer: How?

Shailesh: YouTube's goal is to make the platform very easy to use to watch videos online. That is the discovery of the videos is the first step right? If users can find the videos which they want to watch easily via search, they will keep coming and watching videos, which will increase the watch time on the platform and in turn the ad revenue. Now we also know how the goal of YouTube search is aiding the overall goal of the platform.

To achieve these goals, these are the actions which need to be done on the platform. 
  1. Users need to initiate the search easily
  2. Not only do Users need to initiate the search but also complete their search
  3. After completing the search, users need to click on the search results, if they are not clicking then something is not right, either may be search query understanding or ranking of the results
  4. After clicking, users need to watch for a considerable time, if the watch time is low via search it means, something is wrong, and we are not able to give the right set of videos to users. This is because the search is for intent-heavy users, they know what they are going to watch. If the view duration is low, that means something is wrong.

Now to achieve the goal, these are the actions that are needed, let’s map it to North Star Metrics and some L0 and L1 metrics.

So the Northstar metric would be the Watch Time contribution from Search on the Youtube Platform. That will be our North Star Metric. If that’s going down, then something or the other is going wrong and this North Star metric is also in line with Youtube's North Star metric. More Watch Time means more engagement which implies more revenue.

Now that we have found out the North Star Metric, it’s not enough, we have to break down the North Star Metric into Subsequent L0 metrics, which I will be tracking if I am a YouTube Search Product Manager. This is a super critical metric to understand and these L0 metrics should contribute to our North Star Metric.

  1. The first metric would be % Search Session, which means out of all the sessions which are happening on Youtube, in how many sessions at least 1 search has been made. This will indicate the adoption of the YouTube search.
  2. The second metric would be the average number of Searches per Search session. This means if this is the search session, then on average how many searches the user is doing? To move the North Star metric, increasing the Number of Sessions is not enough, users should be doing more searches.
  3. Another super critical metric is the YouTube Search abandonment Rate, if a user has done the search but is not clicking on any video. This should be low. If that is high that means in more searches, users are not clicking the search results. This indicates that the search results are not good, either YouTube is not able to understand the user's intent or YouTube is not putting the best search results on the top. So Increasing the Search session or the number of searches per search session is not enough, the YouTube search abandonment Rate should also be low.
  4. Finally, AVD i.e. Average View Duration of Video via Search. This is also one of the metrics we should measure.
All these 4 metrics are super important and will directly affect our North Star Metric.

Apart from the L0 Metrics, I will also measure some L1 metrics like

  1. Zero Search Results, which means for how many search queries are we giving 0 results, why it is important because tons of languages are there in the world, and it may happen that YouTube is not able to understand some languages which leads to 0 results.
  2. Another metric which I will track is Auto Suggest usage. So in 100 Search, how many searches are fired via autosuggest? It is important because we have defined the goal of YouTube search to enable users to find the relevant results easily and quickly. By using autosuggest, we are saving the user's typing effort and we are assisting users in their search.
  3. Finally, I will also measure the Median Scroll Depth, which means when users fire the search query and get the results, how much scroll do they have to do before doing the first click. That will tell how good our ranking of search results is.
Let’s see some of the instances of Metrics evaluation.
Metrics for YouTube


Suppose that Searches per search session are going up. What does it mean, is it good or bad?

So it will depend, for example, it can happen that the searches per session are going up because our search is not giving the results in the first go, users have to do multiple tries before getting the relevant results.
In that case, we will not blindly check if Searches per search session are going up so it will be good, we will see another supporting metric like abandonment rate, is this also going up?
If the abandonment rate is also going up, then there is some problem which requires immediate attention.

Important Links
-- Technomanagers

Comments

Post a Comment

Popular posts from this blog

Guesstimates for Product Interviews, Tips to ask clarification Questions

By -
Image
  Photo by  Antoine Dautry  on  Unsplash Guesstimates for Product Interviews, Tips to ask clarification Questions Estimate the number of Hotels | Estimate the number of users of Google Drive Estimate the number of hotels? Clarification Questions Understanding of Hotel Are we also including Airbnb, lodges or only hotels? [Only Hotels] Which geography are we talking about? [Bangalore] Overall Approach Define the area of Bangalore that will be around 600Km². Since there is a national park, a lot of lakes, forests, and wetlands so assuming that around ~50% of the area is there where a potential hotel can reside. So we are looking around an area of 300Km², we can divide this area into 3 parts according to the density, assuming a normal distribution of 25% highly dense for the hotel, 50% mid dense and the rest 25% is low dense. For highly dense that is closely around 75km², for a Km² we can have 5 hotels, that is around 375 hotels. For mid dense(150Km²), on an average, we have around 2.5Hote
Click here to Read »

Design Uber for Kids. [Product Management Interview Question]

By -
Image
Image credits - https://gbksoft.com/blog Interviewer: Design Uber for kids Priyanka: Before we proceed, I would like to clarify my understanding of Uber. So, Uber is a digital ride-hailing platform that connects riders with drivers. The passengers can request a ride on the platform, and the driver can fulfil the demand by dropping them at the desired location. Uber acts as a facilitator between the two and takes a cut out of each ride that happens through its platform. Besides this, Uber also has business verticals like Uber Eats, Uber Freight, etc. Is my understanding correct? Interviewer: Yes. Priyanka: Alright, to understand the problem better, I would like to ask some clarifying questions. When we talk about kids, I am assuming we are referring to an age group of 10–16 years who can travel alone. Also, I am assuming that parents will be booking the rides for their kids. Interviewer: That’s a fair assumption Priyanka: Also, we are catering to the age group of 10–16, so there w
Click here to Read »

System Design Concepts for Product Managers — Part 1

By -
Image
This is a system design series for Product Managers who should have a high-level understanding of System Design. We will first cover the basic concepts of System Design. This is part 1, and stay tuned for future updates. 1. Load Balancer What is a Load Balancer? Load Balancer:  Load Balancer is a common term used widely when you are working on an application tha t  serves enormous traffic. Let’s understand with an example: Assume that Netflix has deployed several servers to handle the number of client requests. When we hit “Netflix.com”, how does Netflix know which server it has routed my request to? This is where the load balancer comes into the picture; load Balancers distribute the incoming traffic from the client efficiently across different servers and make sure that no server is getting overheated or overloaded. Reliability and availability are maintained by redirecting the incoming request to the available servers. Credits:  https://codeburst.io/ Load Balancing Algorithms The lo
Click here to Read »