A/B testing AI features requires extra rigor because AI outputs are noisy and user responses vary widely. Fundamentals: 1) Statistical significance (p < 0.05) is necessary but not sufficient - need practical significance (effect size). A 0.1% conversion lift may be statistically significant with large sample but not worth the operational complexity. 2) Power analysis before starting: determine required sample size to detect your minimum detectable effect (MDE) at desired power (80%). Use scipy.stats or statsmodels. Typical AI feature: MDE 1%, alpha 0.05, power 0.8 -> need ~15,000 users per variant. 3) Sequential testing: traditional fixed-horizon tests require committing to sample size upfront. Sequential testing (Wald, mSPRT) allows peeking and stopping early when significant. Use for faster iteration. 4) Multi-metric evaluation: define primary metric (conversion), guardrail metrics (latency, cost, error rate), secondary metrics (engagement). Test fails if guardrails deteriorate even if primary improves. 5) Bayesian A/B testing: compute P(B > A) directly, easier to interpret than p-values, naturally handles continuous monitoring. 6) Long-term effects: novelty bias means AI features often show initial boost that normalizes. Run tests for 2-4 weeks minimum. 7) Segment analysis: check if effect is consistent across user segments - AI features often help some segments but hurt others.