How I Migrated Absolute Currency Rate Calculations to Kaggle and Automated Reviews with the Gemini API

Introduction: The Challenge of Real-Time FX Data in Financial Modeling

In the fast-paced world of foreign exchange (FX) trading and financial analytics, the calculation of absolute currency rates—where each currency pair is expressed in a common base (e.g., USD)—is a foundational but computationally intensive task. Financial analysts often rely on relative cross rates from sources like Yahoo Finance or the European Central Bank, but converting these into absolute matrices requires normalization and interpolation. Traditionally, this was done on local machines or dedicated servers, leading to bottlenecks in reproducibility and scalability.

A recent detailed technical article on Habr (published July 2026) chronicles one developer's journey to migrate this entire pipeline to Kaggle, the popular data science platform, and then layer on top an automated review generation system using Google's Gemini API. The author faced three core problems: (1) downloading and cleaning raw FX quotes from multiple sources, (2) computing absolute rates via linear programming to ensure triangular arbitrage consistency, and (3) generating human-readable market summaries without manual effort. The solution described is a fully automated, cloud-native workflow that runs on Kaggle's free tier and outputs structured reviews ready for publication.

This article breaks down the technical steps—from data ingestion on Kaggle to prompt engineering with Gemini 1.5 Pro—and highlights the practical trade-offs involved. Whether you are a quantitative analyst, a data engineer, or a fintech blogger, the migration strategy offers a replicable template for moving financial computations into reproducible, serverless environments.

Why Move to Kaggle? The Limitations of Local Pipelines

Before the migration, the author ran the FX calculation scripts on a local workstation. The process had several pain points:
- Manual execution: Every time new data was needed (daily or hourly), the analyst had to manually trigger the Python scripts.
- Dependency hell: Libraries like pandas, cvxpy (for convex optimization), and yfinance required specific versions, leading to conflicts.
- No versioning: The code and output files were stored locally without proper version control, making audits difficult.
- Limited compute: For a full matrix of 30+ major currencies, the optimization step took 5–10 minutes on a laptop, blocking other work.

Kaggle offered a compelling alternative: free GPU/TPU access, pre-installed data science libraries, a persistent /kaggle/working directory, and built-in scheduling via Notebooks. The author specifically chose Kaggle over alternatives like Google Colab because of its dataset hosting and kernel versioning features. As the article notes, "Kaggle allows you to attach datasets directly to notebooks, and you can schedule runs daily using the 'Schedule' button—something Colab doesn't natively support without workarounds."

Step 1: Data Ingestion and Cleaning on Kaggle

The first technical step was to pull live FX rates. The author used the yfinance library to fetch bid/ask quotes for major currency pairs (EUR/USD, GBP/USD, USD/JPY, etc.) from Yahoo Finance. However, the raw data contained missing timestamps and outliers. The cleaning process involved:
- Dropping rows where the spread (ask - bid) exceeded 5% of the midpoint—a sign of stale quotes.
- Forward-filling missing values for a maximum of 2 consecutive periods.
- Converting all quotes to midpoints: (bid + ask) / 2.

The author created a Kaggle dataset (fx-raw-quotes) that was updated daily via a scheduled notebook. The notebook used the kagglehub library to upload the cleaned CSV back to the dataset. This ensured that the raw data was versioned and reproducible.

import yfinance as yf
import pandas as pd
from datetime import datetime, timedelta

pairs = ['EURUSD=X', 'GBPUSD=X', 'USDJPY=X', 'USDCHF=X', 'AUDUSD=X']
data = {}
for pair in pairs:
    ticker = yf.Ticker(pair)
    hist = ticker.history(period='1d', interval='1m')
    data[pair] = hist['Close'].iloc[-1]

df = pd.DataFrame([data])
df.to_csv('/kaggle/working/latest_rates.csv', index=False)

Step 2: Computing Absolute Currency Rates with Linear Programming

This is the core algorithmic challenge. Given relative quotes (e.g., 1 EUR = 1.12 USD, 1 USD = 0.85 GBP), the goal is to find a set of absolute rates (e.g., USD = 1.0, EUR = 1.12, GBP = 1.32) that minimize the sum of squared deviations from the observed quotes, while ensuring consistency (no triangular arbitrage). The author formulated this as a convex optimization problem using cvxpy.

Let r_i be the absolute rate of currency i relative to USD (so r_USD = 1). For each observed pair (i, j) with quote q_ij, the model minimizes sum((r_i / r_j - q_ij)^2). The constraints ensure that all rates are positive. This is a simple quadratic program (QP) that can be solved in under 2 seconds for 30 currencies on Kaggle's CPU.

import cvxpy as cp

# Observed quotes matrix (n x n) with NaN for missing
Q = ... # from cleaned data
n = Q.shape[0]
r = cp.Variable(n, nonneg=True)
objective = cp.Minimize(cp.sum_squares(cp.multiply(Q, r.reshape(-1,1)) - r))
prob = cp.Problem(objective, [r[0] == 1])  # assume first currency is USD
prob.solve()
absolute_rates = r.value

The author noted that the QP solution ensures that if you multiply rates along a cycle, you get 1.0, eliminating arbitrage opportunities. The resulting absolute rates were stored in a new dataset (fx-absolute-rates).

Step 3: Automating Reviews with the Gemini API

Once the absolute rates were computed, the next task was to generate a human-readable market review. The author used Google's Gemini 1.5 Pro API (accessed via the google-generativeai Python package). The prompt was carefully engineered to produce a structured summary with:
- Top 3 strongest and weakest currencies (in terms of absolute rate change vs. previous day).
- Notable movements (e.g., "USD strengthened 0.5% against EUR due to Fed hawkish comments").
- A volatility index based on the standard deviation of daily changes across all pairs.

The prompt included the previous day's absolute rates as context, plus the current rates, formatted as JSON. The Gemini API returned a markdown-formatted review that was saved to a text file and uploaded to the dataset.

import google.generativeai as genai
genai.configure(api_key='YOUR_API_KEY')
model = genai.GenerativeModel('gemini-1.5-pro')

prompt = f"""You are a financial analyst. Given the absolute FX rates for today and yesterday, write a concise market review in markdown.

Today's rates: {today_rates_json}
Yesterday's rates: {yesterday_rates_json}

Include:
- Top 3 gainers and losers
- One notable trend
- Volatility score (low/medium/high)
"""
response = model.generate_content(prompt)
with open('/kaggle/working/review.md', 'w') as f:
    f.write(response.text)

The author emphasized that the Gemini API was chosen over OpenAI's GPT-4 because of its native support for long-context windows (up to 2M tokens), which could theoretically ingest entire month-long rate histories. In practice, they used only two days of data to keep costs low—each request cost about $0.001.

Step 4: Orchestration and Scheduling

To make the pipeline fully autonomous, the author combined all steps into a single Kaggle notebook that:
1. Fetches raw data from Yahoo Finance.
2. Cleans and uploads to a raw dataset.
3. Runs the QP solver and uploads absolute rates.
4. Calls Gemini API and saves the review.

The notebook was scheduled to run every weekday at 8:00 AM UTC using Kaggle's built-in scheduler (available under the Notebook's "Schedule" tab). The author also set up a simple webhook to post the review to a Slack channel via requests.post—though that part is optional.

A major technical hurdle was API key management. Kaggle notebooks can access secrets via environment variables set in the notebook's "Settings" panel. The author stored the Gemini API key as a secret named GEMINI_API_KEY and retrieved it with import os; os.environ['GEMINI_API_KEY'].

Results and Key Metrics

After two months of operation, the system produced the following statistics:
- Data completeness: 98% of scheduled runs succeeded (failures were due to Yahoo Finance downtime).
- Computation time: Average 4.3 minutes per run (including API calls).
- Review quality: The author manually reviewed a random sample of 20 generated reviews and found 18 to be factually accurate and stylistically acceptable. The two failures were due to Gemini hallucinating a currency pair that didn't exist (e.g., "XAG/USD" when silver is not in the matrix). This was fixed by adding a system instruction to restrict output to the 30 specified currencies.
- Cost: Kaggle's free tier covered all computation. Gemini API costs averaged $0.25 per month (for ~20 runs).

Comparative Analysis: Kaggle vs. Alternatives

For the author's use case, Kaggle struck the best balance between free resources and integrated scheduling. The main limitation was the 9-hour session limit, but since the notebook ran for under 5 minutes, it was irrelevant.

Potential Improvements and Pitfalls

The article also discussed future enhancements:
- Adding more sources: The author plans to incorporate data from the European Central Bank (ECB) for official fixing rates, which could be combined with Yahoo Finance market rates. ASI Biont supports connecting to multiple financial data sources via API—more details can be found on the ASI Biont platform.
- Error handling: The current system fails silently if Yahoo Finance returns no data. Adding a retry mechanism with exponential backoff is a priority.
- Model fine-tuning: Instead of using a generic Gemini prompt, the author is experimenting with a few-shot approach that includes two example reviews in the prompt to improve consistency.

One important pitfall noted: Kaggle's free tier limits internet access to port 443 (HTTPS). The Gemini API call worked fine, but if the author had tried to use a non-standard port or a custom server, it would have failed. Always test connectivity first.

Conclusion: A Blueprint for Automated Financial Pipelines

The migration of absolute currency rate calculations to Kaggle, coupled with Gemini API-powered review generation, demonstrates a modern approach to financial data pipeline automation. The author proved that even complex optimization tasks (QP with 30 variables) can run cost-effectively in a serverless environment, and that large language models can produce usable market summaries with careful prompt engineering.

For data scientists and fintech developers, this case study offers a practical template: use Kaggle for scheduled data processing, version your datasets, and integrate LLMs for natural language output. The entire codebase (except API keys) is public on Kaggle, making it easy to fork and adapt. As the article concludes, "The hardest part wasn't the math or the code—it was realizing that the bottleneck was deployment, not computation."

Source