Frizzle's Rubric: Distributed AI Evaluation System
Click any component to explore technical details • Hover to see connections
Distributed AI System at Scale
Building Frizzle's Rubric required coordinating 7 specialized evaluation components across a secure, scalable infrastructure. The system needed to handle concurrent prompt evaluations while maintaining low latency and high availability.
ML Pipeline Design
Fine-tuned 5 DistilBERT models on custom datasets, achieving 94% accuracy on prompt clarity evaluation.
Infrastructure Security
Implemented VPC with bastion host architecture, enabling secure access to private subnet resources.
Async Orchestration
Designed timeout-resilient API coordination across microservices with graceful error handling.
Technical Problem Solving
SSH Agent Forwarding
Problem: Complex SSH access through bastion host to private instances
Solution: Configured ProxyJump with agent forwarding for seamless multi-hop SSH
Host orchestrator
HostName 172.31.48.224
ProxyJump frizzlesrubric-bastion
ForwardAgent yes
Service Discovery
Problem: Coordinating API calls across 7 distributed services
Solution: Configuration-driven service discovery with hardcoded IPs for hackathon speed
EXPERTS = {
'clarity': '172.31.48.225:8003',
'grammar': '172.31.48.226:8004',
# ... 5 more experts
}
Cost Optimization
Problem: $3,905/month AWS costs unsustainable for demo
Solution: Analyzed cost breakdown and identified GPU instances as primary driver
- GPU Instances: $3,033.60/month (77.7%)
- Storage: $535.00/month (13.7%)
- Networking: $222.20/month (5.7%)