Lessons from the Frontline: 5 Takeaways on Keeping Instagram Safe

Fostering a Safe and Positive Community Experience on Instagram

At Instagram, maintaining a platform that is safe, positive, and free from harmful or inappropriate content is of paramount importance. However, achieving this goal is no simple feat, especially when dealing with the vast amount of unconnected content surfaced through features like Explore and hashtag pages. These sections showcase posts from accounts users don’t follow, making it crucial to have robust systems in place to identify and address policy violations.

To tackle this challenge head-on, Instagram assembled a dedicated team last year, solely focused on detecting and removing violating or inappropriate content from these unconnected surfaces. This specialized effort yielded invaluable insights and lessons that could potentially benefit other teams grappling with similar content quality challenges.

Lesson 1: Measuring Quality Is a Nuanced Endeavor One of the most significant hurdles faced was accurately measuring content quality in a standardized and scalable manner. With no industry-wide benchmark for objectively determining what constitutes “high-quality” content, the team initially resorted to manually reviewing and rating test content. However, this approach proved unsustainable, as hand-labeling every piece of content for A/B tests yielded noisy and inconclusive results.

The solution emerged in the form of a hybrid approach, combining human labeling for calibration with machine learning classifiers. This “best of both worlds” strategy enabled the team to scale labeling across experiments, ultimately leading to reliable quality measurements that could be operationalized and incorporated into their workflows.

Lesson 2: Embracing “Read-Path” Models for Precise Enforcement Traditionally, content moderation systems have relied on “write-path” classifiers, which assess whether a piece of content violates policies based solely on the content itself at the time of upload. However, this approach fails to consider crucial contextual signals, such as comments or engagement data, which can provide valuable insights into the content’s reception and potential for harm.

To address this limitation, Instagram developed “read-path” models that run in real-time as users engage with content on Explore. These models evaluate whether an impression is violating or not based on both the content itself and real-time engagement signals. This approach proved to be significantly more accurate at detecting bad behavior, as it considers the full context surrounding the content.

Lesson 3: The Importance of Sourcing Filters in Tandem with Read-Path Models While read-path models improved ranking quality, the team quickly realized that basic sourcing filters were still necessary to provide broad protection. Instagram’s system follows a two-step process: first, sourcing retrieves eligible content based on a user’s interests; then, ranking orders that sourced content for display.

Through this process, the team learned two key lessons:

For low-prevalence violations that are sparsely represented in the data, sourcing upload filters are essential, as read-path models may consistently miss these rare cases. High-precision sourcing filters are required to avoid ranking irrelevant or violating content from the outset. If the ranking stage receives only problematic inputs, the effectiveness of the ranking algorithms becomes moot. Ultimately, both sourcing filters and the fine-grained read-path model work in tandem to ensure optimal content quality.

Lesson 4: Rigorous Model Performance Tracking Enhances User Experience Continuously monitoring the precision and recall of production models is not merely a best practice but a necessity for ensuring a positive user experience. This diligent approach serves two crucial purposes:

It enables the team to quickly identify any model decay, data shifts, or underlying feature issues that could impact performance. It provides insights into the types of violations that the models are allowing or missing, illuminating areas for improvement. To facilitate this process, Instagram’s team implemented comprehensive dashboards that visualize the content their models incorrectly allowed or blocked. This transparency accelerated their iteration cycles by exposing the specific areas that required focused attention and optimization efforts.

Lesson 5: Embracing Flexibility with Percentile-Based Filtering Initially, Instagram employed rigid model score thresholds as quality filters, blocking content below a certain score and allowing content above a specified threshold. However, this approach proved brittle, as even minor changes to model inputs could drastically alter score distributions and render the predetermined thresholds ineffective.

To address this issue, the team transitioned to a more flexible approach, leveraging percentile filtering or regular model score re-calibration. This percentile-based method provided a more stable and reliable quality enforcement system that could gracefully adapt to the ever-evolving ecosystem of content on Instagram.

Conclusion:

Tailoring Novel Techniques for Complex Content Ecosystems Maintaining a safe and positive user experience on Instagram requires innovative techniques tailored specifically for the complexities of content ranking systems. By focusing on operationalizing quality metrics, utilizing multi-stage modeling approaches, rigorously monitoring user impacts, and embracing flexible, percentile-based filtering methods, Instagram’s team has made significant strides in upholding their mission of providing a positive community experience.

These lessons underscore the importance of continuous adaptation and the willingness to challenge traditional approaches in order to stay ahead of the curve. As content ecosystems become increasingly complex, embracing novel strategies and leveraging the power of advanced technologies, such as machine learning, will be crucial for ensuring the safety and well-being of online communities.