How MediaRuntime’s Layered Moderation Pipeline Works

Content moderation is not one binary question asked of one black box. A production media pipeline has to move quickly through obvious cases, spend more attention on uncertainty, preserve evidence, and leave the final product policy in the hands of the application using it. MediaRuntime approaches that problem as a layered moderation cascade. Think of the first layer as the bouncer at the door: fast, specialized, and designed to recognize when a requested safety check is clearly below the escalation band. When the bouncer is not confident enough to clear a category, that category moves to a higher-context check instead of being guessed through.

Why moderation is layered

Running the most expensive analysis on every image and every sampled video frame would add latency and cost even when the material is clearly routine. Depending only on a lightweight screen, however, would ask a fast classifier to make decisions outside the cases it handles best. The cascade separates those responsibilities. The first layer efficiently clears low-risk results. A second layer is reserved for categories that cross an internal uncertainty gate. Policy logic then converts the available evidence into allow, review, or block decisions. This design spends deeper analysis where it can change the answer while keeping the common path responsive.

The pipeline from request to report

Image or sampled video frame
        │
        ▼
Layer 1: fast category screeners
        │
        ├── clearly below escalation band ──► allow evidence
        │
        └── uncertain category ─────────────► Layer 2: higher-context check
                                                   │
                                                   ├── resolved ─► policy decision
                                                   └── unavailable or unresolved ─► review fail-safe

Frame/category decisions ─► aggregate verdict + evidence ─► job report

Layer 1: the fast bouncers

When moderation is enabled, MediaRuntime evaluates only the checks requested by the job: sexual content, violence or gore, dangerous content, or all three. Lightweight specialist screeners produce an initial signal for those categories. A result that is clearly below the service’s escalation band can be cleared at this layer. The bouncer is not being treated as a universal final authority; its job is to identify the straightforward low-risk path and recognize when a category deserves a closer look. The internal escalation bands are service-controlled and can evolve as the pipeline is evaluated and tuned.

Layer 2: selective higher-context checking

If a Layer 1 signal enters the uncertainty band, MediaRuntime escalates the affected category to a higher-context check. Escalation is selective: a frame may need deeper examination for violence while another requested category was already cleared. That avoids repeating more expensive work for questions that did not trigger the gate. The higher layer returns evidence for the escalated category, and MediaRuntime’s policy layer maps it into an effective decision. Importantly, an escalation failure is not converted into an allow. If the higher check cannot resolve a category, the pipeline fails safely to review so the calling application can hold the asset for a person or another policy step.

Video moderation is a cascade across time

An image can enter the cascade directly. A video first becomes a bounded set of frames sampled at a service-controlled interval. Each sampled frame passes through the same Layer 1 and selective-escalation flow. MediaRuntime then aggregates the strongest category evidence across the sampled frames. The report records how many frames were sampled and identifies flagged frame indexes, approximate timestamps, verdicts, and categories. The job-level verdict follows the most serious effective decision observed in the sample. Because sampling is bounded, developers should read the returned interval and frame count from each report rather than assume that every frame was inspected.

{
  "verdict": "review",
  "flagged_checks": ["violence"],
  "decisions": {
    "violence": {
      "decision": "review",
      "raw_decision": "review"
    }
  },
  "evidence": {
    "frames_sampled": 8,
    "frames_flagged": [
      {
        "frame_index": 3,
        "timestamp_sec": 20,
        "verdict": "review",
        "flagged_checks": ["violence"]
      }
    ]
  },
  "video": {
    "frame_interval_sec": 10,
    "max_frames": 24
  }
}

The current API is report-only by design

MediaRuntime currently runs moderation before transcoding, but the public API accepts mode: report only. A moderation result does not automatically cancel or block the media job. The transcode continues, and the completed job plus terminal webhook carry the moderation report. This keeps infrastructure analysis separate from product enforcement. A marketplace might hold review results before listing. A social application might route them to a trust-and-safety queue. An internal archive might attach the report for search and audit without changing access. MediaRuntime provides the structured evidence; your application decides what allow, review, and block mean for its users and policies.

Request moderation with the transcode job

Moderation is a Premium feature for a single image or video input. Audio-only jobs, multi-input batches, and Sandbox jobs do not accept it. Omit checks to request all supported categories, or send only the categories relevant to your application.

{
  "file_url": "gs://value-returned-by-upload-url",
  "metadata": {
    "asset_id": "ugc-video-0426",
    "media_type": "video"
  },
  "moderation": {
    "enabled": true,
    "mode": "report",
    "checks": ["sexual", "violence", "dangerous"]
  },
  "outputs": [
    {
      "type": "mp4",
      "preset": "mp4_720p_h264_aac",
      "path_suffix": "playback"
    }
  ]
}

What your application receives

The completed job exposes the requested configuration and result under moderation, including the overall verdict, per-category decisions, flagged categories, scores, and video evidence when applicable. A complete moderation_result.json is also included in the job bundle and may be exposed through meta.moderation_result.url. Moderation work is estimated, reserved, and settled separately in usage.breakdown.moderation_units. Treat scores and labels as decision support, not facts. Sampling can miss brief events, visual context can be ambiguous, and every product has different rules. Build a human-review and appeal path where consequences matter, and do not interpret an allow result as a guarantee that content is safe, lawful, or compliant with your own policy.

Escalate uncertainty, not every request

The central idea behind MediaRuntime moderation is simple: clear straightforward low-risk signals quickly, escalate uncertainty selectively, preserve a fail-safe review outcome, and return evidence that an application can act on. That layered approach makes moderation a useful part of an asynchronous media pipeline rather than an unexplained yes-or-no gate. Explore the full request and response contract at https://mediaruntime.com/docs, or contact the MediaRuntime team at https://mediaruntime.com/contact to discuss a production workflow.