What it does
This utility maps a company website, selects high-signal pages, and produces a structured knowledge base.
The output can include company basics, product descriptions, pricing tiers, brand colors, screenshots, source URLs, field-level provenance, and a completeness score.
Why split the pipeline
Not every field needs an LLM. The extraction combines targeted model passes with deterministic parsing for things such as colors, logos, prices, calls to action, and change hashes.
There is also a small evaluation path for comparing generated output with labeled examples.
Implementation details
- URLs are classified into page types such as home, about, products, pricing, resources, careers, legal, and contact before extraction.
- The typed output model covers company basics, writing guidance, design assets, competition, positioning, culture, development, legal pages, products, and pricing.
- Deterministic helpers normalize colors, discover likely logo assets, extract pricing signals, and calculate a completeness score.
- The final JSON records source URLs and can attach field-level provenance for later review.
Extract vs scrape
The useful lesson was that "extract the company" is too vague. The repo ended up using two different modes:
Full extractor -> map the site, run richer Firecrawl extraction,
merge deterministic signals, return CompanyKnowledgeBase
Lite extractor -> scrape high-signal pages, gather heuristics,
let a lightweight LLM polish a stakeholder-friendly summary
The distinction matters because some fields are semantic and some are mechanical:
# AI extraction is useful for mission, product descriptions, positioning.
extracted = client.extract(
urls=["https://example.com/", "https://example.com/pricing"],
schema=FULL_COMPANY_SCHEMA,
prompt="Extract company info, products, pricing, culture...",
)
# Raw scraping is better for deterministic details.
colors = re.findall(r"#[0-9a-fA-F]{6}", html_doc.html)
What I learned
Extraction systems get better when the output admits where each fact came from. A pretty summary is not enough. The reviewer needs source URLs, screenshots, field-level provenance, and completeness checks so they can decide what to trust.