Tools & Technologies

• Languages: R (tidyverse, data.table, quantmod, lubridate)
• Platform: Databricks (R Notebooks, Delta Lake)
• Data Sources: NASDAQ feeds, SEC filings
• Storage: Delta Lake
• Orchestration: Databricks Jobs, DBFS mount
• Visualization: ggplot2, plotly, Databricks Dashboards

Objective

Design and automate a suite of customized indices based on NDX-listed stocks, including rebalancing, performance benchmarking, and full audit trail maintenance.

Key Components

1. Index Design & Construction

• Developed sectoral and thematic indices (e.g., NDX Index).
• Rule-based screening: liquidity, float-adjusted market cap, listing type.

2. Data Pipeline (Databricks)

• Ingested price, market cap, and sector data via APIs into Delta Lake.
• Implemented full ETL using R Notebooks on Databricks platform.

3. Rebalancing Logic

• Quarterly/monthly rebalancing using optimization via quadprog.
• Scheduled with Databricks Jobs and stored as Delta tables with history.

4. Backtesting & Analytics

• Calculated returns, volatility, Sharpe, drawdown, and tracking error.
• Benchmarked against NDX Composite, QQQ, and thematic ETFs.

5. Maintenance Automation

• Handled corporate actions (splits, delistings) automatically.
• Alert system to flag violations in eligibility criteria.

6. Compliance & Audit Trail

• Used Delta Lake versioning and Change Data Feed for traceability.
• Each rebalance includes justification and rule-based logging.

Outcomes

• 90% reduction in manual effort for index maintenance and rebalance.
• Performance outperformance demonstrated in backtests (e.g., GreenTech index).
• Comprehensive dashboard for index attribution and sector exposure.

Future Enhancements

• Incorporate ESG and alternative datasets.
• Enable near-real-time streaming updates via Auto Loader.