The Discovery Asymmetry in Global Hackathons
Most builders discover hackathons only after they appear on legacy directory front pages. By then, half the registration window has evaporated. Inside **OXAKALABS // Skunkworks**, four years of 48-hour builds taught us a basic structural rule: **frontier AI sprints—from Cerebral Valley and Google DeepMind to Anthropic, Vercel, and protocol syndicates—are announced first as raw posts on X (Twitter) or Luma calendar links.**
When we architected the **G-Gravitator Live Hackathon Tracker (`/hackathons`)**, we refused to build another static link directory. We needed an aerospace-grade radar that unifies 10 distinct providers while staying resilient against API rate limits, noisy social chatter, and cross-posted duplicates.
1. The Multi-Tier X (Twitter) Adapter Pattern
Relying on a single social scraping endpoint is a guaranteed single point of failure. We structured `lib/ingestion/hackathons/twitter-adapter.ts` around a three-stage fallback cascade:
// Canonical X Query Operator used by G-Gravitator
const X_HACKATHON_QUERY =
'("hackathon" OR "buildathon") ("AI" OR "agents" OR "LLM" OR "prize" OR "$") ' +
'(apply OR register OR lu.ma OR devpost OR dorahacks) -is:retweet lang:en';- **Primary Mode (`x_api`):** Queries X API v2 Recent Search every 6 hours across our operator grammar and high-signal monitored accounts (`@devpost`, `@DoraHacks`, `@lablabai`, `@cerebral_valley`, `@GoogleAI`, `@AnthropicAI`, `@OpenAIDevs`, `@vercel`, `@supabase`).
- **Search Bridge Mode (`search_api` / `apify`):** Falls back to indexed `site:x.com` discovery when direct bearer quotas are constrained.
- **Zero-Key Public Syndication & oEmbed Resolver:** Whenever an OXAKALABS scout or community builder pastes an `x.com` status URL into `POST /api/hackathons/submit`, our resolver extracts the tweet payload directly via X's public Syndication and oEmbed endpoints—expanding shortened `t.co` links before passing the text to normalization.
2. Two-Stage Noise Gate: Heuristics First, Gemini Flash Second
To prevent runaway LLM bills, raw feeds never touch the model unfiltered. Every candidate first passes a deterministic pre-filter:
- **Batch Ceiling:** Hard cap of 30 raw tweets and 40 news items per cron execution.
- **Temporal & Lexical Guard:** Rejects posts containing retrospective markers (*"congrats to the winners"*, *"recap of last weekend"*, *"demo day highlights"*) unless paired with an explicit upcoming registration deadline.
Survivors enter our structured normalization gate powered by **Gemini 2.5 Flash** (`@google/genai`) with a strict JSON Schema contract—backed by a deterministic heuristic parser when running offline:
export interface NormalizedHackathonCandidate {
is_valid_upcoming_hackathon: boolean;
title: string;
organizer: string;
summary: string; // Strictly 2 crisp sentences
prize_pool_usd: number | null;
prize_display: string;
start_at: string | null;
deadline_at: string;
format: "ONLINE" | "IN_PERSON" | "HYBRID";
location: string;
themes: string[];
registration_url: string;
confidence_score: number; // 0.00 - 1.00
}Candidates scoring `>= 0.80` auto-publish to the live radar. Candidates between `0.50` and `0.79` route directly into the **Curator Admin Console (`/admin`)** moderation queue. Anything below `0.50` is dropped at the perimeter.
3. Cross-Source Fingerprint Deduplication
A major hackathon announced on X by `@GoogleAI` on Tuesday will appear on Devpost by Thursday. Naive aggregators render two separate cards.
Our deduplicator computes a dual key: - **Lexical Fingerprint:** `normalize(titleTokens) + "::" + normalize(organizer) + "::" + YYYY_MM` - **Canonical URL Fingerprint:** Stripping protocol, `www.`, query parameters, and trailing slashes from the resolved registration URL.
When an incoming X signal matches an existing Devpost record, we merge their `sources` JSONB arrays into a single database row—preserving the highest-confidence deadline and prize pool while rendering **both `DEVPOST` and `X` badges** on a single card.
One sharp digest of AI breakthroughs, winning hackathon builds, and Lab Dispatches. Zero fluff.