Why I Built
These Four Things
Personal Itch 個人需求
- Wanted to actually try computer vision · 影像辨識 hands-on — not just read about it in slides
- Tired of static, sit-still gaming — what if I could play games standing up, with friends, in the same room?
- Wanted a private way to find IG non-followers without third-party logins
Bigger Question 真正想問的問題
- How far can AI coding assistants take a real project from zero?
- Can the same hand-tracking foundation serve both productivity (Mac control) and play (Fruit Ninja)?
- Where does AI break down and human judgment become essential?
- If AI writes the code, can it also present it?
The thesis: four real artifacts at different points on the AI-autonomy spectrum — and watching where the line lives between what AI can do alone and what still needs a human.
Four Projects, One Theme:
AI as a Development Partner
VisionTrigger
Control macOS with hand gestures and voice — fully vibe-coded with Claude Code & GitHub Copilot.
Fruit Ninja (Webcam)
Hand-tracked Fruit Ninja clone — solo + 2v2 multiplayer. Extension of VisionTrigger's hand-tracking pipeline.
iTrace
Instagram followback checker — required manual reverse engineering of Instagram's private API.
This Presentation
Entire slide deck designed by AI — given only the codebases as input. Delivered in under 3 minutes.
Every Tool I Used —
Cost, Purpose & How
| Tool | Cost · 是否付費 | Purpose · 用途 | How I Used It · 如何使用 |
|---|---|---|---|
| Claude Code (Anthropic) | PAID · Claude Max ~$200/mo | Full-stack code generation, architecture, refactoring | CLI agent in terminal — prompted in plain English; AI read files, wrote code iteratively |
| GitHub Copilot | FREE · Student license | Inline code autocomplete during typing | VS Code extension — tab to accept inline suggestions while editing |
| MediaPipe HandLandmarker (Google) | FREE · Open-source | 21-joint hand tracking — shared by VisionTrigger & Fruit Ninja | Python mediapipe package; CPU inference; up to 4 hands tracked for multiplayer |
| Pygame | FREE · Open-source | Fruit Ninja game loop, fruit/bomb rendering, juice splat layer | pip install pygame; 60 FPS draw loop + SRCALPHA surface caching |
| Faster-Whisper | FREE · Open-source | AI model — real-time speech-to-text for VisionTrigger | Model auto-downloaded, runs 100% offline; no cloud calls |
| Postman | FREE · Personal plan | API testing & reverse engineering Instagram's private endpoints | Copied browser DevTools requests, replayed & mutated to discover endpoints |
| OpenCV · React+Vite · Python | FREE · Open-source | Camera capture / web frontend / runtime | Standard package managers (pip, npm); used as documented |
Disclosure: only Claude Code is paid. Copilot is free under my student account; everything else is open-source. No tool's output was hidden or attributed to me.
VisionTrigger
— What is it?
- Control macOS using hand gestures detected through your MacBook camera
- Triggers Shortcuts, AppleScript, or any shell command
- Also supports voice commands via Faster-Whisper (local, no internet)
- Fully customizable gestures and voice-to-action mappings via
config.yaml - Register new gestures in-app — no restart needed
VisionTrigger
— AI at the Core, Vibe-Coded End-to-End
AI MODELS · 核心模型
- MediaPipe HandLandmarker — 21 joints per frame; finger state encoded as
(thumb, index, middle, ring, pinky)bool tuple - Faster-Whisper — real-time speech-to-text, all local; no cloud or latency
- RMS-based VAD: mic only activates on sound, not constant streaming
- THUMBS_UP vs DOWN disambiguated by thumb-tip y-coordinate
VIBE-CODED · 用 AI 寫的
- Single-threaded OpenCV loop + 2 daemon threads (action + voice)
- Gesture state machine: IDLE → CAPTURE → NAME → ACTION → CONFIRM
- Voice template matching:
{placeholder}regex capture + substitution - Hot-patching: new gestures register without restart
Claude Code — handled full-stack: algorithms, UI drawing, config I/O, threading, refactoring.
GitHub Copilot — inline suggestions during active coding sessions.
Fruit Ninja
— Same Hand-Tracking, Now for Play
- Reused VisionTrigger's hand-tracking pipeline — same MediaPipe model, same camera abstraction, different output
- Built with Python + Pygame — 60 FPS game loop, sprite cache, persistent juice splat layer
- 5 fruits (watermelon, orange, apple, lemon, kiwi) + bombs with animated fire
- 3 solo difficulties — Easy/Medium/Hard tune lives, spawn rate, and required slice speed (700 → 1100 px/s)
- Optional Kinect v2 via the same tracker abstraction
2v2 Multiplayer · 多人對戰
- 4 players, one camera — MediaPipe tracks up to 4 hands at once
- Hand IDs 0&1 → Team A (red), 2&3 → Team B (blue)
- 90-second timed battle; higher score wins; bombs subtract 1 from your team
The motivation: VisionTrigger proved hands can replace the mouse. Fruit Ninja asked — can they also replace the controller, and bring friends into the same physical space instead of static, solo gameplay?
iTrace
— Instagram Followback Checker
- Find who you follow but who doesn't follow back — live at itrace.stanleyowen.com
- Online mode: script injected into Instagram DevTools, fetches data via Instagram's internal API, renders floating overlay
- Offline mode: upload Instagram data export JSON — compared locally, zero network calls
- All data client-side — no backend, no accounts, full privacy
- Bulk unfollow with rate-limit-safe throttling + 5-cycle cooldown
- Frontend: React + Vite; injected overlay: vanilla JS so it runs anywhere
Core Algorithm
const notFollowBack =
following.filter(u =>
!followersSet.has(u));
Simple set difference — the hard part was getting the data, not computing the result. That's the next slide.
The Difficulty:
Instagram's Undocumented Private API
- Instagram has no public API for reading your own followers/following list programmatically
- The app calls an internal API not documented anywhere:
/api/v1/friendships/followers/ - Required identifying the right headers (e.g.
x-ig-app-id: 936619743392459), auth cookies, and pagination parameters - The unfollow endpoint needed CSRF token extraction from cookies + correct POST format
- AI alone could not figure this out — the API behavior had to be observed and verified manually
This is where human judgment became essential — not just writing code, but understanding a system that wasn't designed to be understood from the outside.
How I Cracked It:
Postman + Manual Analysis
friendships to find the real API calls being made by the Instagram web app.
next_max_id), and design retry + rate-limit logic around the discovered endpoints.
This Presentation
— Also Made by AI in < 3 Minutes
- Input: only the codebases — no slides, no outline, just “make a presentation”
- Output: full HTML/CSS slide deck, dark sci-fi theme, 17-slide narrative
- Time: < 3 minutes from prompt to finished deck
- Human work: zero HTML written — only feedback & small corrections
What AI did
- Read CLAUDE.md + 1000+ lines of source
- Derived narrative (vibe-coded vs reverse-eng) unprompted
- Designed theme, wrote all HTML/CSS/JS in one pass
What human did
- Provided the codebases as context
- Added name & student ID after the fact
- Small text adjustments via chat
The same AI tool that built the projects also presented them — this deck is itself a demonstration of the tool's capability.
AI Can Do A Lot —
But Humans Still Matter
| Task | VisionTrigger | Fruit Ninja | iTrace | This PPT |
|---|---|---|---|---|
| Code / markup written | AI ✓ | AI ✓ | AI ✓ | AI ✓ |
| Architecture / structure | AI ✓ | AI ✓ | AI ✓ | AI ✓ |
| Content sourcing | Human intent | Human intent | Human intent | Read codebase ✓ |
| Undocumented API / research | Not needed | Not needed | Manual reverse-eng. ✗ | Not needed |
| Design & visual layout | AI ✓ | AI ✓ | AI ✓ | AI ✓ |
| Time to deliver | Days | Days | Days | < 3 minutes |
What I Learned
AI accelerates, not replaces · AI 是加速器
VisionTrigger + Fruit Ninja were both vibe-coded — and Fruit Ninja reused VisionTrigger's pipeline, so AI handled the “extend an existing system” loop too. Complex software shipped in days.
Some problems need human eyes · 有些問題只能靠人
iTrace required observing live network traffic and validating an undocumented API — AI alone couldn't get there. The line lives at “things AI has never seen.”
Even “full AI” needs verification · 全 AI 也要驗證
This deck was AI-generated in < 3 minutes — but a human still reviewed every slide, corrected details, and approved the final output.
Human judgment is the constant · 判斷力才是核心
Whether AI does 20% or 100%, a human must verify the result is correct, honest, and actually says what you meant to say.
AI can be the whole crew — but humans must still verify the flight.