Skip to content

label-view.md

Latest commit

 

History

History
356 lines (275 loc) · 13.9 KB

label-view.md

File metadata and controls

356 lines (275 loc) · 13.9 KB
Raw

Label View

What is a Label View?

A label view is a Feast primitive for storing judgments about entities — reward signals, safety scores, human reviews, ground-truth answers — separately from the immutable observations stored in feature views.

Feature View Label View
Stores What was observed What was judged
Example Agent prompt and response response_quality: "poor", is_safe: 0
Writers Usually one source Multiple labelers (human, LLM judge, code)
Changes over time Append-only Updated by new label writes

Use a label view when you need governed, training-ready labels that multiple sources can write, disagree on, and resolve — not when you need append-only feature data.

Prerequisites

Before using label views, you need:

  • Feast with the LabelView primitive
  • A feature repo with at least one Entity
  • A PushSource (or batch DataSource) for label ingestion
  • feast apply run after defining label views

Why use Label Views?

Separate features from judgments. Mixing reward labels into feature views blurs two different lifecycles — observations are append-only; labels are overwritten, corrected, and debated.

Support multiple labelers. Human reviewers, safety scanners, and LLM judges can all label the same entity. Feast tracks who wrote what via labeler_field and resolves conflicts via ConflictPolicy.

Generate training datasets. Compose label views with feature views in a FeatureService and retrieve features + labels together with point-in-time correctness.

Data label in the UI. Configure data labeling profiles (annotation profiles) so data scientists can label data directly in the Feast UI — entity forms, document spans, bulk review, or active learning.

Feedback vs Expectations

Not all labels serve the same purpose. Feast distinguishes two common types — both stored in a label view, modeled with field names and tags:

Feedback Expectation
Question How good was the actual output? What is the correct answer?
Example response_quality: "poor", relevance: "irrelevant" ground_truth: "relevant", is_default: 1
Typical writers Human, LLM judge, automated code Human experts (gold standard)
Training use Reward signal, quality filter, active-learning queue Supervised target column
Conflict handling Common — use ConflictPolicy Rare — usually one authoritative source

Feast does not require separate primitives for feedback and expectations. Model them with field names and tags:

tags={
    "feast.io/field-role:response_quality": "feedback",    # judgment about output
    "feast.io/field-role:ground_truth": "expectation",     # correct answer
}

Practical pattern:

  • Feedback label view — multi-labeler, ConflictPolicy, fields like response_quality, safety_score, relevance
  • Expectation fields or view — stable ground truth, fields like ground_truth, expected_answer, is_default
  • Mixed view — one label view with both (e.g. RAG: relevance = feedback, ground_truth = expectation)

Sources of Labels

Label views accept labels from any source. Track the writer in labeler_field:

Source labeler example Typical role
Human reviewer human-reviewer@company.com Feedback or expectation
LLM judge gpt-4-evaluator Feedback (quality scores)
Automated scanner nemo-guardrails Feedback (safety signals)
Batch import risk-ops-team Expectation (historical outcomes) 
labels_df = pd.DataFrame({
    "user_id": ["user-001"],
    "response_quality": ["poor"],
    "is_safe": [0],
    "labeler": ["human-reviewer@company.com"],
    "event_timestamp": [pd.Timestamp.now()],
})
store.push("agent_feedback_labels_push_source", labels_df)

When to use Label Views

Use a FeatureView when… Use a LabelView when…
Data is observational and append-only Data is a judgment or data label (annotation)
One source writes the data Multiple labelers may disagree
No conflict resolution needed You need governed conflict resolution
No labeling UI needed You want structured data labeling workflows (annotation)

How Label Views Work

Step 1: Define a label view

from datetime import timedelta

from feast import Entity, FeatureService, Field, FileSource, PushSource
from feast.labeling import ConflictPolicy, LabelView
from feast.types import Float32, Int64, String

user = Entity(name="user_id", join_keys=["user_id"])

agent_feedback_labels = LabelView(
    name="agent_feedback_labels",
    entities=[user],
    schema=[
        Field(name="response_quality", dtype=String),
        Field(name="is_safe", dtype=Int64),
        Field(name="reviewer_notes", dtype=String),
        Field(name="labeler", dtype=String),
    ],
    source=PushSource(
        name="agent_feedback_labels_push_source",
        batch_source=FileSource(
            name="agent_feedback_labels_batch",
            path="data/agent_feedback.parquet",
            timestamp_field="event_timestamp",
        ),
    ),
    labeler_field="labeler",
    conflict_policy=ConflictPolicy.LAST_WRITE_WINS,
    reference_feature_view="user_profile",
    description="Human and automated feedback on agent responses.",
    tags={
        "feast.io/labeling-method": "entity-form",
        "feast.io/field-role:response_quality": "feedback",
        "feast.io/field-role:is_safe": "feedback",
        "feast.io/field-role:reviewer_notes": "metadata",
        "feast.io/label-values:response_quality": "excellent,good,acceptable,poor,harmful",
        "feast.io/label-values:is_safe": "1,0",
        "feast.io/label-widget:response_quality": "enum",
        "feast.io/label-widget:is_safe": "binary",
        "feast.io/label-widget:reviewer_notes": "text",
    },
)

Run feast apply to register the label view.

Step 2: Push labels

Labels are written with FeatureStore.push():

import pandas as pd
from feast import FeatureStore

store = FeatureStore(repo_path="feature_repo/")

labels_df = pd.DataFrame({
    "user_id": ["user-001", "user-002"],
    "response_quality": ["good", "harmful"],
    "is_safe": [1, 0],
    "reviewer_notes": ["Accurate summary", "Unsafe medical advice"],
    "labeler": ["human-reviewer", "nemo-guardrails"],
    "event_timestamp": pd.to_datetime(["2025-01-15", "2025-01-15"]),
})

store.push("agent_feedback_labels_push_source", labels_df)

Each push appends to the offline store (full history retained) and updates the online store (latest value per key).

Step 3: Data label in the Feast UI

Open the label view in the Feast UI Data Labeling tab (Annotate tab). The UI reads data labeling tags (annotation tags) and shows the right workflow:

  1. Open Label Views in the sidebar
  2. Select a label view (check the Data Labeling badge on the list page)
  3. Go to the Data Labeling tab (Annotate tab)
  4. Choose a data labeling method (annotation method): Entity Form, Document Span, Review & Edit, or Active Learning
  5. Submit labels — they are pushed to the label view's PushSource

Step 4: Join labels with features for training

training_service = FeatureService(
    name="agent_training_service",
    features=[
        user_profile_fv,          # immutable features
        agent_feedback_labels,    # mutable labels
    ],
)

training_df = store.get_historical_features(
    entity_df=entities_df,
    features=training_service,
).to_df()

Training pipelines get features and resolved labels in one retrieval call.

Data Labeling Profiles (Annotation Profiles)

Data labeling profiles (annotation profiles) configure how labels are created in the UI. Set them via tags — no schema changes required.

Supported profiles

Profile Best for UI experience
entity-form RLHF, safety review, per-entity feedback Form — one entity at a time
document-span RAG chunk labeling, span labeling (annotation) Load document, label chunks
table Bulk review, correcting existing labels Editable table with dropdowns
active-learning Label high-value unlabeled entities Queue from a reference feature view

Choosing a profile

Answer one question:

  1. "I need to review agent responses one at a time" → entity-form
  2. "I need to label document chunks for RAG" → document-span
  3. "I need to correct labels in bulk" → table
  4. "I want to label only the most valuable unlabeled items" → active-learning (requires reference_feature_view)

The Data Labeling tab (Annotate tab) shows only relevant data labeling methods (annotation methods) per profile:

Profile Methods shown
document-span Document Span, Review & Edit
entity-form Entity Form, Review & Edit, Active Learning
table Review & Edit, Active Learning, Entity Form
active-learning Active Learning, Entity Form, Review & Edit

Tag reference

Tag Purpose Example values
feast.io/labeling-method Primary data labeling method (annotation method) entity-form, document-span, table
feast.io/field-role:<field> Semantic role of a field feedback, expectation, label, metadata, content, span_start, span_end
feast.io/label-values:<field> Allowed label values relevant,irrelevant
feast.io/label-widget:<field> Input widget type enum, binary, text, number

Examples by use case

Agent feedback (RLHF / safety)

Feedback from human reviewers and automated safety layers on agent responses.

agent_feedback_labels = LabelView(
    name="agent_feedback_labels",
    entities=[user],
    schema=[
        Field(name="response_quality", dtype=String),
        Field(name="is_safe", dtype=Int64),
        Field(name="reviewer_notes", dtype=String),
        Field(name="labeler", dtype=String),
    ],
    source=feedback_push_source,
    labeler_field="labeler",
    conflict_policy=ConflictPolicy.LAST_WRITE_WINS,
    reference_feature_view="user_profile",
    tags={"feast.io/labeling-method": "entity-form", ...},
)

RAG chunk labeling (feedback + expectation)

One view can hold both a retrieval judgment and ground truth:

rag_chunk_labels = LabelView(
    name="rag_chunk_labels",
    entities=[chunk],
    schema=[
        Field(name="source_document", dtype=String),
        Field(name="chunk_text", dtype=String),
        Field(name="relevance", dtype=String),       # feedback
        Field(name="ground_truth", dtype=String),    # expectation
        Field(name="chunk_start", dtype=Int64),
        Field(name="chunk_end", dtype=Int64),
        Field(name="labeler", dtype=String),
    ],
    source=rag_push_source,
    labeler_field="labeler",
    conflict_policy=ConflictPolicy.MAJORITY_VOTE,
    tags={
        "feast.io/labeling-method": "document-span",
        "feast.io/field-role:relevance": "feedback",
        "feast.io/field-role:ground_truth": "expectation",
        "feast.io/field-role:chunk_text": "content",
        "feast.io/label-values:relevance": "relevant,irrelevant",
    },
)

Historical ground truth (batch labels)

Pre-existing label tables loaded via feast materialize:

loan_default_labels = LabelView(
    name="loan_default_labels",
    entities=[loan],
    schema=[
        Field(name="is_default", dtype=Int64),
        Field(name="delinquency_days", dtype=Int64),
        Field(name="loss_severity", dtype=Float32),
        Field(name="labeler", dtype=String),
    ],
    source=PushSource(
        name="loan_default_labels_push_source",
        batch_source=FileSource(
            path="data/loan_defaults.parquet",
            timestamp_field="event_timestamp",
        ),
    ),
    labeler_field="labeler",
    conflict_policy=ConflictPolicy.LABELER_PRIORITY,
    tags={
        "feast.io/labeling-method": "table",
        "feast.io/field-role:is_default": "expectation",
    },
)

Conflict policies

When multiple labelers write different values for the same entity, ConflictPolicy picks one value for offline store reads (training, UI browse):

Policy When to use
LAST_WRITE_WINS Default. Most recent write wins.
LABELER_PRIORITY Trusted labelers override others (e.g. human over LLM judge).
MAJORITY_VOTE Consensus labeling (e.g. multiple labelers on RAG chunks).

{% hint style="info" %} Conflict policies apply to the offline store (training). The online store always uses last-write-wins. Full label history is always retained in the offline store. {% endhint %}

Best practices

Name fields by intent. Use response_quality (feedback) and ground_truth (expectation) — not generic score or label.

Tag field roles. Set feast.io/field-role:<field> to feedback or expectation so your team and UI know what each field means.

Match conflict policy to label type. Use LABELER_PRIORITY when humans correct automated judges. Use MAJORITY_VOTE for multi-labeler consensus. Use LAST_WRITE_WINS for simple feedback streams.

Link to features. Set reference_feature_view so the UI and documentation show which feature view the labels apply to.

Separate noisy feedback from stable ground truth. When possible, put expectations in dedicated fields or views with stricter writer conventions (human-only).

Limitations

  • Conflict policies are enforced on offline reads only; online store is always last-write-wins.
  • LABELER_PRIORITY requires explicit labeler ordering configuration.
  • Data labeling profiles (annotation profiles) are UI configuration via tags — not enforced at the SDK write path.

Next steps