Challenge

Aggregated and normalized over 5TB of multi-depth order book data from 30+ crypto exchanges over 7 years, requiring optimized ETL pipelines, compression, and efficient query performance for analytics and visualization. Still collecting data today.

Developed a robust analytics engine capable of detecting large trades, regional money flow shifts, and liquidity changes in real-time while providing historical depth for 30,000 users.

Originally hosted on Google Cloud Platform with managed MySQL and Redis for caching, the infrastructure was later migrated to a co-location facility and containerized with Kubernetes to drastically reduce operational costs and latency.

Designed and implemented a RESTful API allowing external trading bots, quantitative research tools, and analytics dashboards to consume CMF’s aggregated datasets programmatically.

Added support for sentiment analytics by integrating Twitter and Reddit data feeds, enabling correlation between social buzz and market movements across assets and sectors.

Built highly customizable screeners allowing users to define their own data fields, metrics, and statistical windows, offering unparalleled flexibility in analyzing and comparing digital assets.