Portfolio under active development.

Back to Projects

Market Flow Intelligence

Automated financial data collection and analysis pipeline for ETF flows and insider trading (now a standalone application)

Market Flow Intelligence
Published:

Market Flow Intelligence

Project Overview

Market Flow Intelligence is an automated data pipeline that collects, analyzes, and visualizes financial market flow data. The system focuses on two key areas:

  1. ETF Flow Analysis: Tracking money movements in and out of Exchange-Traded Funds across different sectors
  2. Insider Trading Activity: Monitoring and analyzing corporate insider buying and selling patterns

This project combines data engineering, financial analysis, and interactive visualization to provide actionable market insights.

Technical Implementation

Architecture

The project uses a hybrid architecture combining Python data processing with a Next.js frontend:

┌─────────────────────┐     ┌─────────────────────┐     ┌─────────────────────┐
│                     │     │                     │     │                     │
│  Data Collection    │────▶│  Data Processing    │────▶│  Data Visualization │
│  (Python)           │     │  (Python)           │     │  (Next.js/React)    │
│                     │     │                     │     │                     │
└─────────────────────┘     └─────────────────────┘     └─────────────────────┘

Key Components

  • Data Collectors: Python modules that fetch data from financial APIs and web sources
  • Data Processors: Analysis modules that transform raw data into meaningful insights
  • Chart Generators: Plotly-based visualization modules that create interactive charts
  • API Layer: Python serverless functions deployed on Vercel
  • Frontend Dashboard: Next.js application with interactive components

Python-JavaScript Integration

One of the key technical challenges was integrating a Python-based data science project with a Next.js web application. This was solved by:

  1. Deploying Python code as Vercel serverless functions
  2. Creating a consistent API interface between Python and JavaScript
  3. Implementing proper error handling and data validation
  4. Using base64 encoding for transferring chart images

Features

ETF Flow Analysis

The system tracks fund flows across major ETFs and sectors, providing insights into where institutional and retail money is moving:

  • Daily flow tracking for 100+ major ETFs
  • Sector-based aggregation and analysis
  • Historical trend visualization
  • Anomaly detection for unusual money movements

Insider Trading Intelligence

Corporate insider transactions can provide valuable signals about company prospects. The system:

  • Monitors Form 4 filings from the SEC
  • Identifies significant insider buying and selling patterns
  • Calculates insider buy/sell ratios
  • Highlights clusters of insider activity

Interactive Dashboard

The web dashboard provides:

  • Interactive charts with filtering capabilities
  • Data tables with sorting and searching
  • Automated insights and commentary
  • Historical data comparison

Development Process

The project was developed using an incremental approach:

  1. Data Collection: Built Python scrapers and API integrators
  2. Analysis Pipeline: Developed data processing and statistical analysis
  3. Visualization: Created chart generation modules with Plotly
  4. API Layer: Deployed Python code as serverless functions
  5. Frontend: Built interactive dashboard with Next.js

Technical Challenges & Solutions

Challenge: Data Freshness

Solution: Implemented incremental data collection with scheduled jobs to update the database, ensuring data is always current while minimizing API calls.

Challenge: Performance Optimization

Solution: Used Supabase for data storage to reduce computation needs and implemented client-side caching with appropriate TTL.

Challenge: Deployment Constraints

Solution: Optimized Python dependencies and split complex operations into smaller functions to work within Vercel's serverless function limitations.

Future Enhancements

  1. Machine Learning Models: Add predictive analytics for market flows
  2. Real-time Alerts: Implement notification system for significant market movements
  3. Custom Watchlists: Allow users to create personalized monitoring dashboards
  4. API Access: Provide API endpoints for external applications

Conclusion

Market Flow Intelligence demonstrates how data science techniques can be applied to financial markets to extract actionable insights. By combining Python's data processing capabilities with Next.js's interactive frontend, the project delivers a powerful tool for financial analysis.