Shayan Koohi
Shayan Koohi
CS & AI · Sapienza
Live Open to AI engineer roles — Rome or remote EU Drag · use ← → keys

I build AI systems and the evals that tell me when they're wrong.

Final-year Applied CS & AI at Sapienza University of Rome — GPA 29.2/30, graduating March 2027. Multi-year iOS engineer before that, so the models I train end up inside real apps. Three years teaching IGCSE maths and science: I can explain the hard parts.

What I actually do

Anyone can get a demo working. I measure whether it works.

Every number on this page comes from a harness I can hand you. The most recent run of one: a ground-truth audit that found 3 of 4 apparent retrieval failures were bugs in my eval dataset, not the retriever — killing a planned week of retrieval work before I wrote a line of it.

Evals before features

Two-phase harness for course-notes-agent: retrieval scored with zero generation calls, generation scored separately — so "wrong answer" resolves to a cause instead of a shrug. Ground truth anchored to verbatim source text, not chunk ids, so re-ingestion can't silently invalidate it.

Baselines that embarrass me

I hand-rolled BM25 in ~60 lines to check whether the embedding model earned its keep. It ties text-embedding-3-small at top-1 — 0.76 vs 0.77 — and wins outright on the one query dense retrieval genuinely fails. Deliberately untuned: a baseline adjusted until it loses isn't a baseline.

RAG that refuses

PDF → paragraph-aware chunks → Postgres + pgvector → cosine top-k → grounded answer from gpt-5-mini behind a FastAPI /ask. 9 of 9 out-of-scope questions correctly refused, 0 answered with fabricated content.

Models behind APIs

LSTM, 1D-CNN and Transformer trained and compared on NASA C-MAPSS; the winner serves live inference at 13.74 test RMSE through FastAPI with per-engine 30-cycle sliding-window buffering. A system, not a notebook.

Data at ~10⁸ rows

Streaming ETL on AWS EC2: ~100M rated Lichess games, one pass per compressed dump, headers only, resumable. Aggregates into SQLite; analysis in pandas, scikit-learn and statsmodels — validated against 20 + 20 placebo tests, not just p-values.

And it ships

Docker Compose for multi-service stacks, FastAPI inference services, Vercel for the TypeScript tools — plus native iOS in Swift, shipped. Rare enough in AI engineering that it's worth saying out loud.

Eval-first RAG

course-notes-agent — the eval is the project.

A RAG system over my Calculus 2 notes, built eval-first: the retrieval pipeline is deliberately simple, and the most developed part of the repo is the harness that measures it. Two findings came out of it — both from distrusting a number instead of building a feature.

ishayankoo001/course-notes-agent ↗
Retrieval eval · 62 rows · two-phase harness

Two findings, both from distrusting a number

Reproducible

Two charts. The first compares overall retrieval recall at k=1, 3 and 5 before and after a ground-truth audit: recall@1 rises from 0.69 to 0.73, recall@3 from 0.90 to 0.93, and recall@5 from 0.94 to 0.98 — with the retriever unchanged, because every correction was made to the eval dataset rather than the system. The second compares recall@5 over the same 53 in-scope rows: the dense retriever using text-embedding-3-small reaches 0.98 and misses 1 row, an untuned BM25 baseline reaches 0.92 and misses 6, and the union of both reaches 1.00 and misses none.

The ground-truth auditSame retriever · corrected answer key
Retrieval recall before and after the ground-truth audit Recall@1 rises from 0.69 to 0.73, recall@3 from 0.90 to 0.93, recall@5 from 0.94 to 0.98, with the retriever unchanged.
Does the embedding model earn its keep?recall@5 · 53 rows · full 0–1 axis untuned baseline
Recall at k=5 for dense retrieval, BM25 and their union Dense embeddings reach 0.98 and miss one row, untuned BM25 reaches 0.92 and misses six, and the union of both reaches 1.00 and misses none.
  • 46both retrievers hitof 53 in-scope rows
  • 6only dense hitsparaphrase, not wording
  • 1only BM25 hitsrare exact token · sin(xy)
  • 0neither hitsunion reaches recall@5 = 1.00
  1. PDF ingestpymupdf
  2. Chunksparagraph-aware · ~500 tok
  3. pgvectorcosine top-k
  4. Grounded answergpt-5-mini · /ask
FastAPIPostgres + pgvector text-embedding-3-smallgpt-5-mini BM25 baselineTwo-phase eval harness Blinded LLM judgeDocker
app/bm25.pyPython
# Deliberately NOT tuned. Textbook k1=1.5, b=0.75, plain \w+.
# A baseline adjusted until it loses is not a baseline —
# and one adjusted until it wins isn't either.
K1 = 1.5    # term-frequency saturation
B  = 0.75   # length normalisation

def score(self, query: str) -> list[tuple[int, float]]:
    for term in tokenize(query):
        f = self.tf[i].get(term, 0)
        if not f: continue
        norm = 1 - self.b + self.b*self.doc_len[i]/self.avgdl
        s += self.idf[term] * (f*(self.k1+1)) / (f + self.k1*norm)
Listing 01The comparison point — ~60 lines, no model, no API call, published in 1994. It ties text-embedding-3-small at top-1
evals/run_eval.pyPython
# WHY GROUND TRUTH IS ANCHORED TO SOURCE TEXT, NOT CHUNK IDS:
# chunk ids come from a SERIAL column — re-ingestion renumbers
# everything and silently invalidates every row in the dataset.
for anchor in row["expected_anchors"]:
    matching = {cid for cid, content in chunks if anchor in content}
    if not matching:
        # A silent zero here looks exactly like a retrieval
        # failure — and sends you debugging the wrong component.
        raise RuntimeError(f"{row['id']}: anchor not found")
    requirements.append(matching)
Listing 02Ground truth is software too — an anchor that resolves to zero chunks raises loudly instead of scoring as a miss

Phase 1 and Phase 2 are separate on purpose: a wrong answer has two causes with opposite fixes, and end-to-end measurement reports “wrong” and leaves you guessing. Refusals are split by retrieval coverage, so the generator isn't blamed for retrieval's misses. The refusal judge is blinded — it sees the answer text only, never the question or the expected label, because a judge that can infer the hoped-for answer confirms a hypothesis instead of measuring one. And the headline numbers are optimistic by construction: 28 chunks, questions generated from those chunks, k=5 returning 18% of the corpus. The repo says so, in the README, above the fold.

Real-time ML

Sentinel-IoT — a full ML system, not a notebook.

Three architectures trained and compared on NASA C-MAPSS turbofan data; the winner predicts Remaining Useful Life at 13.74 test RMSE behind a FastAPI service, with a live React dashboard turning those predictions into operational alerts. The model is the small part — the buffering, the alert thresholds and the deployment are the project.

Fig. 01Live demo — fleet cards sorted by urgency, RUL trend charts, alert feed polling every 2s
React 18RechartsVite + Tailwind FastAPIPyTorch · LSTM Docker ComposeREST + OpenAPI docs
Sentinel-IoT dashboard — engine in warning state with RUL trend chart
Fig. 02Warning state — RUL below 50 cycles, trend flagging degradation
Sentinel-IoT dashboard — critical alert with immediate action required
Fig. 03Critical alert — immediate action required, surfaced to the top
  1. Simulatorstreams NASA C-MAPSS sensor cycles, one reading at a timePOST /api/sensor-data
  2. FastAPI backendbuffers a 30-cycle sliding window per engine, runs LSTM inference, owns alert logicGET /api/engines · /api/alerts
  3. React dashboardfleet cards sorted by urgency, RUL trend charts, live alert feedpoll every 2s
dashboard/src/Dashboard.jsxReact
// Fire all three GETs in parallel — concurrent, not sequential.
const fetchData = useCallback(async () => {
  const [enginesRes, alertsRes, healthRes] = await Promise.all([
    fetch(`${API_URL}/api/engines`),
    fetch(`${API_URL}/api/alerts`),
    fetch(`${API_URL}/api/health`),
  ]);
  /* ... */
}, []);

useEffect(() => {
  fetchData();
  const interval = setInterval(fetchData, POLL_INTERVAL);
  return () => clearInterval(interval);  // no leaks
}, [fetchData]);
Listing 03Pull-based by design — simpler than a websocket, proxy-friendly, 2s latency is right for human monitoring
src/api.pyFastAPI
# The API is PASSIVE — it receives; it never fetches.
# Simulator, real sensor or unit test all talk to it identically.
class SensorReading(BaseModel):
    engine_id: int
    cycle: int
    sensor_values: list[float]

# The API owns the buffer AND the alert thresholds —
# "what counts as critical" is a business rule, not a UI concern.
WINDOW_SIZE = 30
THRESHOLD_WARNING = 50
THRESHOLD_CRITICAL = 20
Listing 04Typed contracts via Pydantic — validation + auto-generated Swagger docs at /docs for free
Native iOS

Dibs — I also ship native apps.

A shared task manager for iOS, designed and built end-to-end in SwiftUI: post a task to a group, someone calls dibs, everyone sees the plan. On top of multi-year freelance iOS work. An AI engineer who can put a model behind a native app instead of handing it over as an endpoint is rare enough to be worth a section.

Fig. 04Live walkthrough — 2-minute screen recording
SwiftUINative dark-first design system AI schedule assistantWorkload estimates Multi-year freelance iOS
Dibs — home screen
HomeFig. 05today's workload at a glance
Dibs — Ask Today: AI assistant summarising the week's workload
Ask TodayFig. 06AI answers about your schedule
Dibs — calendar with deadline density per day
CalendarFig. 07deadline density at a glance
Dibs — task detail with Claim this task action
Claim itFig. 08one clear primary action
i.

The model lives inside the app

"Ask Today" answers questions about your own schedule — the assistant reads real task data and workload estimates, not a generic chat box bolted onto a sidebar. Shipping that means owning the whole path: prompt, latency budget, and what the UI does while it waits.

ii.

Hierarchy before decoration

Every screen has one job: "Today" leads with the only number that matters — ≈ 2h 15m of work — before any list. Overdue state is colour + copy + position, never colour alone. A full custom colour system underneath, and motion used to explain state changes rather than to show off.

Data engineering

~10⁸ games, one streaming pass.

The data-engineering half of the job: do trap openings spread like viral misinformation? Built on AWS EC2, streaming ~100M rated Lichess games into SQLite one compressed dump at a time, analysed with pandas and scikit-learn — and checked against placebo distributions rather than p-values alone.

AWS EC2Python streaming ETL SQLite + SQL viewspandas · scikit-learn statsmodels · PCA
chess-openings-as-memes ↗
Behavioral signal lab · raw volume + PCA

From chess boom to meme axis

Study complete

About 100 million rated Lichess games were processed across 30 observed months using one streaming pass per file. The monthly all-opening volume chart rises from 15.8 million games in January 2019 to 35.3 million at the July 2020 pandemic and Queen's Gambit shock. A second chart shows exact PCA loadings for trickiness, soundness, punish gap, and surprise; PC1 explains 38.4 percent and PC2 explains 27.1 percent of variance.

Monthly blitz volumeKept Lichess games · all openings raw counts · millions
Monthly kept Lichess blitz game volume Thirty observed months. Volume starts at 15.8 million games in January 2019, jumps to 35.3 million in July 2020, and reaches the mid-forties through 2021. 35.3m · JUL 2020Pandemic + Queen's Gambit:volume nearly doubles
Feature loadingsPCA · four z-scored features
PCA feature loadings PC1 loadings are 0.508 for trickiness, 0.726 for soundness, negative 0.463 for punish gap, and negative 0.010 for surprise. PC2 loadings are negative 0.621, 0.029, negative 0.646, and 0.442 respectively.
  • ~108rated games parsedLichess monthly dumps
  • 30months analysedresumable EC2 ingestion
  • 1streaming passper file · headers only
  • 20 + 20placebo testsrandom months · control openings
  1. EC2 ingest.pgn.zst dumps
  2. Single passfilter + aggregate
  3. SQLiteviews + rating bins
  4. Analysisregression + PCA

Pandemic play and The Queen's Gambit nearly doubled monthly blitz volume; PCA then compresses trickiness, soundness, punish gap and surprise into interpretable axes. The placebo tests are the point — 20 random months and 20 control openings, so an effect has to beat the null distribution it was measured against, not just a threshold.

More builds

Smaller tools, same care.

01.

WASAText — Vue 3 front to Go back

A WhatsApp-style messenger built end-to-end for Sapienza's Web & Software Architecture course — groups, replies, reactions, read receipts. Designed OpenAPI-first (897-line spec before a line of Go), shipped in Docker. Closed the exam with 30 e lode.

Vue 3 · Go · OpenAPI · SQLite · Docker View repo ↗
02.

Science Duel — classroom tool

Next.js + TypeScript internal web tool with password-gated flows, attendance, timed duels, score tracking and leaderboards. Deployed on a Vercel-style CD workflow and used weekly in my real classes.

Next.js · React · TypeScript · Vercel View repo ↗
03.

Phonebook — CRUD, end to end

Express REST API with MongoDB/Mongoose persistence, CORS, JSON middleware and request logging, consumed by a React/Vite client with Axios. The bread-and-butter stack of client work, done cleanly.

Node.js · Express · MongoDB · React View repo ↗
04.

Teaching-ops automation

Internal Python tools that generate, fill, digitally sign and track lesson-report PDFs for my teaching job — replacing a fully manual paperwork workflow for good.

Python · ReportLab · automation
05.

Full Stack Open

University of Helsinki's deep-dive into modern web development — React, Redux, Node.js, Express, MongoDB, GraphQL and TypeScript, exercise by exercise.

React · Node · GraphQL · TS View repo ↗
06.

Production experience

Software engineering intern at TazarvAfzar Co. — REST API integrations and relational schema design in MS SQL Server, inside a real production environment.

REST · MS SQL Server · production
Method

When a test fails, two things could be broken.

Most people only consider the first one. On my own project the test was wrong three times as often as the system — which is the entire reason I write the harness before the feature.

01.

The test is software too

Four retrieval failures shared a persuasive root cause and justified a week of chunking work. I checked the answer key instead: three of the four were bookkeeping errors in the dataset. recall@5 went 0.94 → 0.98 with the retriever untouched. One real failure is an anecdote, not a pattern.

02.

A number needs a ceiling and an interval

recall@1 can't exceed 0.91 on my dataset — rows needing two chunks max out at 0.5 when k=1 — so quoting 0.73 against a mental baseline of 1.00 measures a structural artifact. Two decimals, deliberately: at n=53 the third would assert precision the sample size doesn't support.

03.

Credentials

30 e lode in Deep Learning and in AI Lab: Computer Vision & NLP at Sapienza. Stanford Machine Learning Specialization (Coursera). 3rd place, Sharif University CS Summer School national competition.

Education

The grades behind the projects.

Final year of Applied CS & AI at Sapienza, graduating March 2027.

Sapienza University of Rome

BSc — Applied Computer Science & Artificial Intelligence

29.2/30
GPA · 2023 — graduating March 2027
ENEnglish — fluent, IELTS 8.0
FAPersian — native
ITBased in Rome since 2023
CourseYearResult
Web & Software Architecture202630 e lode
Programming I202430 e lode
Programming II202430 e lode
Deep Learning202630 e lode
AI Lab: Computer Vision & NLP202530 e lode
Human–Computer Interaction202630 / 30
Probability202530 / 30

"30 e lode" = 30 with distinction, the highest possible grade in the Italian system. Full transcript available on request.

How I work

Hire the trajectory, not just the snapshot.

I'm early in my career and honest about it — what I offer is speed of learning, ownership, and work whose numbers you can check.

i.

I learn whatever the product needs

PyTorch for one project, pgvector for another, SwiftUI, Go, EC2 — none of these were handed to me in a lecture. This field reshapes itself every few months; if your stack moves to a different model, framework or serving path next quarter, I'll meet it there.

ii.

Responsible & independent

I've balanced a full-time degree with a real teaching job since 2023 — deadlines, classrooms, paperwork. When I say something will be done, it gets done, without needing a push.

iii.

Decisions written down, with revisit conditions

Open any file in my repos — the reasoning is next to the code, including what I rejected and what would make me change my mind. decisions.md in course-notes-agent logs every call and the condition that reopens it. Nobody should have to reverse-engineer why a threshold is 30.

iv.

I can explain the hard parts

Three years teaching IGCSE maths and science — if you can get a room of teenagers through a proof, you can get a product manager through why recall@1 has a ceiling. I take feedback without ego and give it with care. Persian, English, and enough Italian to laugh at the right moments.