AUTO
Competitor Data Scraping Automation
Powered by OptiFlowProject overview
An automation pipeline that continuously scrapes product data from competitor sites and other web sources using n8n, then routes it into a custom Flask API that cleans and structures the data for downstream use. Built while working at OptiFlow, where I contributed to the pipeline design and API implementation.
My role
Automation Engineer
Key responsibilities
- Designed the end-to-end scraping and ingestion pipeline
- Built scheduled n8n workflows against multiple web sources
- Developed the Flask API that receives and processes scraped payloads
- Implemented error handling, retries, and run monitoring
- Structured the processed output for downstream consumption
Technical contributions
- Built n8n workflows handling pagination, throttling, and per-source extraction rules
- Developed a Flask REST API for normalization, deduplication, and validation
- Implemented scheduled triggers to keep data continuously refreshed
- Added retry and failure-handling logic so partial source outages do not break runs
Business impact
Replaced slow, error-prone manual research with an automated pipeline that keeps product and competitor data continuously up to date.
Completed while working at OptiFlow. Presented as professional experience — not a MaveriKode project, and all intellectual property remains with OptiFlow.