How would you measure the success of Instagram Feed?

By -

Interviewer: How would you measure the success of Instagram Feed?

Shailesh: I will start by defining what Instagram is. Basically, Instagram is a social media app where you can share photos and videos to the world.
Interviewer: Sounds right

Shailesh: But what is Instagram Feed? So, the Instagram feed is the Newsfeed which you see when you open Instagram. To be precise, it’s a personalized stream of photos and videos that is based on your interests and the people you follow right? Here it’s important to understand that it’s personalized.
Interviewer: And what do you think the goal of the Instagram News Feed is?

Shailesh: So the goal of Instagram newsfeed is to increase engagement on the platform so that users can enjoy and discover what they are interested in. Now I would like to talk about the different stakeholders for Instagram Newsfeed.
I am thinking of Content consumers, Content creators and Instagram itself, then

Interviewer: You are absolutely correct.So what are the Content Consumers are looking for?
Shailesh: They are looking for personalized and relevant content on Instagram so that they can discover content of their interest and browse more and more right?

Interviewer: What do you think the content creators are looking for?
Shailesh: Content creators want their content should get visibility via newsfeed to more and more users so that they can build their target audience.

Interviewer: What do you think Instagram is looking for?
Shailesh: Instagram wants more and more content consumers should find relevant discovery via Newsfeed so that they spend most of their time browsing it.
Why because if they browse more, then more ads on Instagram can surface to the users, which means more money to Instagram.
I think we are all clear about the value that NewsFeed is providing to all the stakeholders.

Interviewer: What do you think is the intersection of the values?
Shailesh: Intersection of the value for all the stakeholders, Content creators, Content consumers and Instagram. It's a relevant discovery of the content which results in endless browsing.

Intersection of Value

Interviewer: Now that we know what’s the intersection of the values? What do you think the Northstar Metric should be?
Shailesh: The Northstar Metric should be the Total Number of minutes/Hours spent on Instagram Feed Per Day or Per Week.

Let’s understand certain nuances of this.

Don’t confuse the Homepage with Feed. Newsfeed is just part of the Homepage. The homepage has different elements like Stories, Reels, NewsFeed etc.
So let’s understand with the help of an example,  If people are coming on Instagram and just viewing all the stories and exit the app, that will not be counted as the time spent on the Newsfeed.

You will better understand when we will break down the Northstar Metric into different L0 metrics.

The time spent on the News feed can be broken down into:

Total Number of sessions Happening on Instagram per day * % session that is News Feed session. ( How are we defining a Newsfeed session ?)
If you have interacted with any post be it liking, sharing, saving, commenting or at least viewing 4-5 posts, that is a Newsfeed session right?

Then Multiply by the average session length.

Here the first term tells you the Top of the funnel traffic, the second term tells you the adoption of Newsfeed and the 3rd term tells you the engagement.

Now let’s see some of the L1 metrics for this.
Some of the key metrics I will track to check the Daily Number of sessions are DAU and sessions per user. If that is going up, that means the daily number of sessions on Instagram will also go up. Apart from these, I will also see metrics like what is the New User Growth on the platform.

I will also see DAU/WAU which will tell me how many weekly active users are also my daily active users. Another important metric I will check is the Weekly User churn or Monthly User Churn. If that is going up that will not be a healthy sign for Instagram and for News Feed also the top of the funnel will decrease.

Now I hope you are clear about the Top of the Funnel metric.

Now let’s see the metric related to the Adoption of Newsfeed, that would be

% Newsfeed Session in which there are Likes, Comments, Share is happening.
What is the average number of Newsfeed sessions per user per day
We will also track the number of posts created per day. Why? If people are not finding the Newsfeed relevant or helpful they will not browse through the newsfeed and creators will also not put effort on creating posts right?
We will also track the number of connections made via Newsfeed.
This will also be one of the key metrics, I think I should have mentioned that in the PPT as well.

Let’s move forward, one other metric I would see is the number of posts viewed per user or median scroll depth that will directly impact the session length.

Now let’s see some of the counter metrics for this.

For counter metrics, I would track the Newsfeed bounce rate which means all those sessions in which you opened Instagram and do nothing just quit the app. Another one I will track is the unfollow rate via Newsfeed, which will indicate our Newsfeed is not that relevant and great. It can also be many posts that are reported via newsfeed.


Metrics Evaluation

Now let’s see one tradeoff-related question for this Newsfeed, this will give us a better clarity.
So let’s assume that, recently Instagram has launched a reaction feature in the Newsfeed. That means apart from just liking the post, you can depict different emotions. in the Newsfeed.

You did an A/B test on this and found out that the Reactions are up by 20% and comments are down by 10%, what would you do?

So let’s understand this in greater detail.

So when we were building this feature, we would have certain hypotheses in mind right.
We would have set the goal of the Reaction feature.
The goal of the reaction feature is to make it easy for users to engage and depict their emotions right?
Reactions feature will increase the overall engagement on the Newsfeed and motivate the Content creators to create more posts right?

This is the goal and hypothesis we would have set for the Reaction feature. So we would check these metrics apart from the Reactions and Comment Metrics, right?

The Number of comments can go down, right? Because now users find it a bit easier to engage and depict their emotions without using their comments. It is saving their time.

We need to see some of the other metrics to check whether to ship or not Ship it.

First I will check some of the L1 Metric.

I will check the Average number of engagements per day, ideally it should go up. The engagement can be a reaction or Comment. On an aggregate level, it should go up, right?

Then I will check Average number of engagement per post, which should also increase in this A/B testing. Both these metrics will tell people are finding it useful.

We will also check has the user level engagement increased or not. That means after the launch of the reaction feature, more people are engaging with Newsfeed or not? 

We will also check if the Number of posts created per day has increased or not. We would expect the number of posts should increase or stay neutral, it should not go down. If the number of posts created per day after this feature goes down, we will not launch this feature.

Now let’s see some of the L0 metrics,
I would track the % feed session if that is going up. Ideally, it should go up, if that’s going down we will not launch this feature.

Another metric I would track is the session length. The session length can go down because people are now not commenting, it is saving time, so on the aggregate level it can go down but not much. We are okay if it goes down a bit or stays neutral.


Finally, the Northstar metric, average time spent on the Newsfeed should go up. Because now more and more people are engaging with Newsfeed, % Newsfeed sessions on the platform are going up, and creators are motivated to create more posts, so ideally this should go up. If that’s going down we will not launch this feature.

Interviewer: Good, we can wrap now.

A/B Testing - Launch & No Launch Criteria

Important Links:

-- Technomanagers

Connect on LinkedIn: https://www.linkedin.com/in/priyanka-dalmia/
Connect on LinkedIn: https://www.linkedin.com/in/shailesh-sharma/

Comments

Post a Comment

Popular posts from this blog

Guesstimates for Product Interviews, Tips to ask clarification Questions

By -
Image
  Photo by  Antoine Dautry  on  Unsplash Guesstimates for Product Interviews, Tips to ask clarification Questions Estimate the number of Hotels | Estimate the number of users of Google Drive Estimate the number of hotels? Clarification Questions Understanding of Hotel Are we also including Airbnb, lodges or only hotels? [Only Hotels] Which geography are we talking about? [Bangalore] Overall Approach Define the area of Bangalore that will be around 600Km². Since there is a national park, a lot of lakes, forests, and wetlands so assuming that around ~50% of the area is there where a potential hotel can reside. So we are looking around an area of 300Km², we can divide this area into 3 parts according to the density, assuming a normal distribution of 25% highly dense for the hotel, 50% mid dense and the rest 25% is low dense. For highly dense that is closely around 75km², for a Km² we can have 5 hotels, that is around 375 hotels. For mid dense(150Km²), on an average, we ...
Click here to Read »

Design Uber for Kids. [Product Management Interview Question]

By -
Image
Image credits - https://gbksoft.com/blog Interviewer: Design Uber for kids Priyanka: Before we proceed, I would like to clarify my understanding of Uber. So, Uber is a digital ride-hailing platform that connects riders with drivers. The passengers can request a ride on the platform, and the driver can fulfil the demand by dropping them at the desired location. Uber acts as a facilitator between the two and takes a cut out of each ride that happens through its platform. Besides this, Uber also has business verticals like Uber Eats, Uber Freight, etc. Is my understanding correct? Interviewer: Yes. Priyanka: Alright, to understand the problem better, I would like to ask some clarifying questions. When we talk about kids, I am assuming we are referring to an age group of 10–16 years who can travel alone. Also, I am assuming that parents will be booking the rides for their kids. Interviewer: That’s a fair assumption Priyanka: Also, we are catering to the age group of 10–16, so the...
Click here to Read »

System Design Concepts for Product Managers — Part 1

By -
Image
This is a system design series for Product Managers who should have a high-level understanding of System Design. We will first cover the basic concepts of System Design. This is part 1, and stay tuned for future updates. 1. Load Balancer What is a Load Balancer? Load Balancer:  Load Balancer is a common term used widely when you are working on an application tha t  serves enormous traffic. Let’s understand with an example: Assume that Netflix has deployed several servers to handle the number of client requests. When we hit “Netflix.com”, how does Netflix know which server it has routed my request to? This is where the load balancer comes into the picture; load Balancers distribute the incoming traffic from the client efficiently across different servers and make sure that no server is getting overheated or overloaded. Reliability and availability are maintained by redirecting the incoming request to the available servers. Credits:  https://codeburst.io/ Load Balancing Alg...
Click here to Read »