Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Strategies
- Trade-offs between hosted APIs and self-hosted solutions
- Patterns for on-premises, edge, and air-gapped environments
- Single model versus model portfolio approaches
- Cost-per-decision analysis
Hardware Sizing and Capacity
- Memory usage based on parameter count
- CPU-only versus GPU-accelerated performance
- Planning for concurrency and throughput
- Allowing for growth in task length
Serving Stacks
- Options such as vLLM and SGLang
- Execution via llama.cpp and GGUF
- Using Ollama for local development
- Selecting stacks based on decision workload characteristics
Quantization While Preserving Calibration
- Precision formats and memory efficiency
- Assessing quality and calibration impact
- Per-tensor and importance-aware quantization methods
- Validation testing post-quantization
Latency Engineering
- Distinguishing median from tail latency
- Latency implications of context length
- Warm starts and model caching techniques
- Batching multiple decision queries
Edge and Constrained Environments
- Running small decision models on CPU
- Deployment on Apple Silicon and embedded devices
- Offline operation and update protocols
- Distributing weights in air-gapped settings
Containerization and Scaling
- Designing images for model serving
- Managing volumes and caches
- Horizontal scaling and routing strategies
- Resource constraints and GPU sharing
Observability
- Tracking request, latency, and error metrics
- Monitoring confidence distributions
- Recording decision outcomes and escalations
- Alerting on calibration drift
Versioning and Rollout
- Version control for models and adapters
- Shadow and canary deployment strategies
- Maintaining pinned model files for audit trails
- Rollback procedures
Security and Multi-Tenancy
- Authentication and rate limiting implementation
- Tenant isolation for prompts and logs
- Network segmentation and egress controls
- Auditing decision logs
Requirements
- Experience with Linux administration and containerization.
- Knowledge of model inference servers and GPU management.
- Fundamental understanding of REST APIs, load balancing, and monitoring.
Audience
- MLOps and platform engineers managing inference infrastructure.
- SRE teams supporting AI services.
- AI engineers deploying decision models on-premises.
- Architects designing air-gapped or regulated AI deployments.
