Consolidated Entity Propagation

flowchart TD 
   
    Identify[Identify Database Files] --> Select[Select Database]
    Select --> Load[Load Database in SQL and Report Size]
    Load --> Decision{Start Project?}
    Decision --> |Yes| Sample[Sample Data]
    Decision --> |No| Delete(((Delete Database))) 
    Sample --> Explore[Explore with Samples]
   

    Explore --> mappable_sg
    Explore --> freeform_sg
    Explore --> json_sg
    Explore --> docs_sg

    subgraph mappable_sg [Identify Mappable Fields]
        Relate[Identify Mappable Fields] --> Link[Link Fields]
        Link --> View[Create View]
        View --> Export[Export Full Data View]
        View --> |Revise| Relate
    end

    subgraph freeform_sg [Identify Free Form Fields]
    Freeform[Identify Free Form Text Fields] --> FullFree[Export Freeform Field As CSV ] 
    FullFree --> ProcessFree[Process and detect on CSV]
    end

    subgraph json_sg [Identify JSON Fields]
    Json[Identify JSON Fields] --> |Predictable| LinkJson[Link Json Fields]
    Json --> |Unpredictable| FullJson[Extract Field Entries as seperate docs]
    FullJson --> ProcessJson[Process JSON Documents]
    ProcessJson --> DetectJson[Run Detection on JSON Documents]
    end

    subgraph docs_sg [Identify Doc Link Fields]
    Document[Identify Fields Linked to Docs] --> Fetch[Fetch and Process Documents]
    Fetch --> Key[Select Fields to Search and Add As Entities]
   
    end
 
    Key --> Map
    LinkJson --> Map
    Export --> |Approved| Map[Map Fields to Entity List]
    ProcessFree --> Datamine[Data Mine]
    DetectJson --> Datamine
    Key --> Datamine
    Map --> Consolidation 
    Datamine --> Review[Review Documents]
    Review --> Consolidation(((Run Consolidation)))