The MIVAA Metadata Management System provides comprehensive, AI-powered extraction of 200+ metadata attributes from PDF catalogs. The system uses DynamicMetadataExtractor integrated into the Product Discovery pipeline to extract metadata across 9 functional categories during PDF processing.
DynamicMetadataExtractor - AI-powered extraction of comprehensive metadata (200+ fields)
Product Discovery Service - Discovers products and enriches with metadata
Metadata Scope Detector - Classifies metadata as product-specific vs catalog-general
Metadata Application Service - Applies metadata to products with override logic
Metadata API - RESTful endpoints for metadata management
The DynamicMetadataExtractor organizes metadata into 9 comprehensive functional categories:
Metadata that mentions a specific product name and applies only to that product.
Example: Text: "NOVA tile has R11 slip resistance" Scope: product_specific Applies To: ["NOVA"] Relevance: 0.95 (HIGH)
Metadata that explicitly states it applies to all products.
Example: Text: "All tiles in this catalog are made in Spain" Scope: catalog_general_explicit Applies To: "all" Relevance: 0.5 (MEDIUM)
Metadata mentioned once without product context, implicitly applying to all products.
Example: Text: "Available in 15×38" Scope: catalog_general_implicit Applies To: "all" Relevance: 0.5 (MEDIUM)
Detection Patterns:
Metadata that applies to a specific product category.
Example: Text: "All matte tiles have R11 slip resistance" Scope: category_specific Applies To: ["matte_tiles"] Relevance: 0.7 (MEDIUM-HIGH)
Metadata extraction is now integrated directly into the Product Discovery pipeline (Stage 0):
Stage 0: Product Discovery ├── 0A: Discover Products (Claude/GPT) │ ├── Identify product names │ ├── Extract page ranges │ ├── Extract basic metadata (designer, dimensions, variants) │ └── Classify content by category │ └── 0B: Enrich Products with Comprehensive Metadata ├── For each discovered product: │ ├── Extract product-specific text │ ├── Call DynamicMetadataExtractor │ ├── Extract 200+ metadata fields across 9 categories │ └── Merge with discovery metadata │ └── Store enriched products in database
The enrichment process: Product Discovery identifies products, then for each product the system extracts product-specific text, calls DynamicMetadataExtractor, and merges metadata in priority order — original discovery metadata takes highest priority, followed by critical fields (material_category, factory_name, factory_group), followed by the 200+ dynamically discovered fields. All metadata is stored in the products.metadata JSONB field.
The legacy chunk-based metadata application is still available for post-processing:
PDF content is split into semantic chunks during document processing.
PDF → PyMuPDF4LLM → Markdown → Semantic Chunking → Document Chunks
Each chunk is analyzed to determine its metadata scope. The scope detector returns scope, confidence, applies_to, extracted_metadata, and is_override for each chunk.
Metadata is applied to products in a specific order to handle overrides correctly:
When product-specific metadata overrides catalog-general metadata, the system tracks it using an _overrides array in the product metadata JSON.
A catalog with general information on page 1 ("Available in 15×38", "Made in Spain", "Factory: Castellón Ceramics"), then individual product pages for NOVA (R11 slip resistance, matte finish, no dimensions), HARMONY (R12 slip resistance, glossy finish, 20×40 dimensions), and ESSENCE (R10 slip resistance, no dimensions).
_overrides: ["dimensions"]), and has its own R12 slip resistance and glossy finish.Three metadata fields are always required and extracted:
These fields are validated during PDF processing and must be present for successful processing.
Detect metadata scope for a text chunk.
The request takes chunk_content, product_names array, and document_context. The response includes a scope_result with scope (e.g., "catalog_general_implicit"), confidence, reasoning, applies_to, extracted_metadata, is_override, and processing_time.
Apply metadata to products with scope-aware override logic.
The request takes document_id and a chunks_with_scope array, each with chunk_id, content, scope, applies_to, extracted_metadata, and is_override. The response includes products_updated, overrides_detected, metadata_applied (per product), and processing_time.
List metadata with filtering and pagination.
Query Parameters:
document_id - Filter by document IDproduct_id - Filter by product IDscope - Filter by scope typemetadata_key - Filter by metadata keylimit - Maximum results (default: 100)offset - Offset for pagination (default: 0)The response includes total_count, items (each with product_id, product_name, metadata_key, metadata_value, scope, and is_override), limit, and offset.
Get metadata statistics and analytics. Returns total_products, total_metadata_fields, catalog_general_count, product_specific_count, override_count, and most_common_fields.
The metadata system is integrated into the PDF processing pipeline at Stage 4 (after product creation):
Stage 0: Product Discovery (Claude/GPT) Stage 1: PDF Extraction (PyMuPDF4LLM) Stage 2: Chunk Creation (Semantic Chunking) Stage 3: Product Creation (Vision Analysis) Stage 4: Metadata Application ← NEW Stage 5: Image Processing Stage 6: Embedding Generation
The products table stores all metadata in a metadata JSONB field, allowing flexible storage of the 200+ dynamic fields extracted by DynamicMetadataExtractor.
The metadata JSONB object contains named fields (e.g., dimensions, slip_resistance, finish, country_of_origin, factory_name), an _overrides array listing which fields were overridden from catalog-general values, and a _scope object mapping each field name to its detected scope type (e.g., catalog_general_implicit or product_specific).
_overrides array to know which fields were overridden