What should be the frequency of Ads in YouTube Shorts? [Google PM Interview Question]
Interviewer: Suppose you are a YouTube Product Manager, How would you decide the number of Ads shown in between YouTube Shorts?
Should we show Ads alternatively one after the other or Show we show Ads after every 2 shorts or How would you decide what is that number? Is it after every 3 videos, 4 videos or 5 videos?
I hope you have understood the question, if not, I will clarify your doubts
Shailesh: I will start by asking a few clarifying questions. The first question that comes to my mind - Is this the first time we are showing Ads in the YouTube shorts, what’s the current behavior? Why this question is important is because whatever we do, we need to measure and assess according to the default behavior of the platform.
Interviewer: Let’s assume this is the first time we are trying to show Ads in between YouTube shorts.
Connect on LinkedIn: https://www.linkedin.com/in/priyanka-dalmia/
Connect on LinkedIn: https://www.linkedin.com/in/shailesh-sharma/
Shailesh: The second question which I want to ask is, what kind of Ads we are planning to show to the users? Are these Ads Video Ads, Display Ads or what mix of both? Why I am asking this question is because knowing this, I will be setting up the hypothesis and the launch criteria, since the engagement level of both types of Ads are different. Video Ads are more engaging than display ads.
Interviewer: Let’s assume that these are the video ads only which we are thinking about here.
Shailesh: The third question which I would like to ask is whether the Shorts capability has been rolled out to all the YouTube users or a set of users or limited Geography? This information is important for setting up the experiment.
Interviewer: Let’s assume that it is rolled out to 100% of the users.
Shailesh: Ok I have the information now. The way I would like to structure this problem is, first I will define the broader goal of the YouTube shorts, what are the different stakeholders who are there , what value they are getting from YouTube Shorts right?
This will help us to narrow our focus on the key metrics and with the help of which we will be able to decide what to do or what not to do.
Then I will define a certain hypothesis which I will be testing in the A/B test and set the Roll Out criteria, Roll Back criteria or criteria where we want to test it again. This is how I will structure this.
So first let’s understand the YouTube Shorts:
So YouTube shorts are the short form of the content, by short I mean less than 60 seconds. YouTube launched this product with the gain popularity of short form of content like Instagram Reels, TikTok. Also users behavior is slightly different for YouTube shorts, you generally don’t search for YouTube Shorts like you do for the normal video on YouTube right? Although you can, but you just do an infinite scroll right?
So now let’s see who are the different stakeholders that are involved in the YouTube Shorts and the value they are seeking from the YouTube Shorts.
Content Creator - Content Creators wants users to discover their shorts and make sure that they watch as much as possible
Content Consumer - Content Consumers want relevant shorts to be in their feed so that they can get maximum value out of their time.
YouTube Shorts - YouTube want to surface the right content form the content creator to the content consumer, so that they spend maximum time on the platform. If they spend more time on the platform, they can surface more ads and more money to them.
Interviewer: What’s the intersection of the value? Be it Content Creator and Content consumer and YouTube.
Shailesh: It is more time spent on the platform.
Interviewer: So what do you think of the North Star metric for the product?
Shailesh: It should be - Time spent on YouTube Shorts per Day right?
Now let’s see some of the metrics which will become important to check what Ad frequency is good for the platform right?
For that first Let’s break down the North Star metric into Daily Active Shorts Users that means users who watch at least one YouTube Shorts per day. Why it is important to check because it can happen that if your ads frequency is too much then people will become so frustrated and they are not coming very often as they were coming earlier, that’s where it is important.
So we have covered the daily Active Short users, now multiple by Number of Sessions they are doing , that is also important because it may take a hit from the Ad intervention we might be doing.
Another thing is Number of minutes watched per session.
Some of the other critical metrics to check are number of YouTube shorts Videos watch per Session, Ads impression that mean how many ads users are seeing and Ads CTR , what’s the click through rate on Ads.
Finally one more important metric which we need to check is the creator side metric that is Number of Shorts posted per day, why it is important? because Ads can have some downstream impact, suppose with some Ads frequency, people are not seeing that many shorts, engagement on the shorts has gone down, with that creators will also not post much shorts because they don’t find any value in it and it’s a vicious cycle for the platform. That’s where we will check the Number of Shorts posted per day also.
Interviewer: I am happy that you have given a reasoning for everything. A good product manager always gives a proper reasoning about the importance of the metric.
Shailesh: Thank you. Now let me form some of the hypotheses we should test.
First Hypothesis (Showing Ads alternatively) that means after every Youtube Short, there will be a video Ad.
Second Hypothesis ( Showing Ads after K position , here I am assuming K to be 4 - That means after 4 Shorts, the Ad will come.
In the real world when you would have to test with multiple such buckets but for the interest of the time, I am doing it with only 2 buckets.
In the first hypothesis, I am expecting my Ads impression to go up drastically and increase in our absolute Ads clicks as well but not in the proportion of the increase in the impression. However because the Ads are too frequent, we are expecting users to bounce back and decrease in the watch time. It can also happen that users like the Ads and click on the Ads, this will also take users away from the platform and decrease the watch time too. It may also happen that creators don’t find value in the Shorts and start posting less Shorts which can also take a hit on the Number of Shorts posted per day.
In the second hypothesis, I am expecting the Ads impression to go up but not drastically like in the first, but since the Ads are not too frequent I am expecting the watch time to go down here as well but not that much.
Interviewer: Now since we have the Hypothesis ready, tell me how you would set up the A/B test and Launch Criteria.
Shailesh: Ads Impression should go up by at least 20% , if let’s say the Ads impression up by only 2%, then there is no point of launching Ads which will hit the customer experience right. So in this case we will relook how we are launching Ads.
Ads Clicks should go up.
Daily Active Shorts Users should not go Down, if it is neutral that is also fine.
Shorts Session Per User should not go Down either
Number of Shorts watch per session shouldn’t go down less than K% ( 1%)
WatchTime shouldn’t go down less than 1% either, if it is going down let’s say 10%, I would certainly not launch this feature
Number of Shorts posted per Day shouldn’t go down
This should be our launch criteria. All the metrics should be statistically significant. If the results are not significant, then we need to restart the experiment. I will always check the required Sample size and power of the test before starting any experiment.
So in real life, we will test with multiple buckets, the buckets which will give maximum Ad revenue and click with minimum impact in the watchtime, will be rolled out to 100% of the users.
If in each bucket the watch time is dropped by 10%, then we will roll back this feature and will see some of the other interventions for Ads.
To recap, so what we have done is, First we understood about the product and it’s values. We define certain metrics to look into and form an hypothesis. After than we form the A/B test and decide the launch criteria.
Important Links:
-- Technomanagers
Comments
Post a Comment