Saki's First Learning Phase: What We Found

TL;DR: We let Saki (our AI assistant) try to improve herself over 11 days. She experimented a lot — 96 different rule variations — but ultimately performed slightly worse than when she started. The system correctly caught this and stopped. We've stabilized everything and are preparing for Phase 2.

What Is This About?

Saketek has an autonomous learning system built into Saki, our AI shopping assistant. Instead of us manually tweaking how Saki responds, she can propose her own improvements, test them, and keep what works.

Phase 1 was the first real test of this system. We wanted to answer: Can Saki reliably improve herself without human intervention?

The short answer: not yet — but we learned a lot about why.

What Happened (11 Days, Feb 17–28)

The Experiment

• Saki started with a baseline set of rules (how to greet customers, handle requests, etc.)
• The learning system observed her conversations and proposed rule changes
• Each change was tested: if it helped, it stayed; if it hurt, it was rolled back

The Problem: Too Many Changes, Too Fast

Saki was changing her own rules 44x faster than the target rate. More than half of her proposed changes made things worse and had to be undone. She was essentially trying too many things at once without giving each change enough time to prove itself.

Performance: Slight Decline

The overall decline was small (-2.2%), but the 7-day trend showed acceleration downward (-14.5%). The system correctly flagged this as a statistically significant regression — meaning it wasn't just random noise, it was a real decline (confidence: 95.4%).

Other Issues Found

• 12.4% of conversations hit token limits (Saki's responses were too long), which we've now started tracking
• 22 experiments were running simultaneously when the limit should have been 1 — they were interfering with each other

What We Fixed

During the audit, we applied three fixes:

1. Token overflow tracking — Saki now properly logs when her responses are too long, so the learning system can learn from these failures
2. Historical data recovery — We backfilled 14 past overflow events that weren't being tracked
3. Experiment cleanup — Cancelled all 22 overlapping experiments and enforced a strict limit of 1 experiment at a time

What's Next (Phase 2)

The core learning system works — it can propose changes, test them, and detect when things go wrong. What it needs is better guardrails:

• Slower rate of change (fewer proposals per day)
• Longer evaluation windows before deciding if a change helped
• Stricter limits on simultaneous experiments
• Better handling of token limits in conversations

Technical Appendix

The sections below contain the raw statistical data for technical review.

Statistical Test Results

Learning System Stats

Immutable Snapshot

This data snapshot is cryptographically hashed to ensure it can't be altered after the fact.

snapshot_hash: 5b317a57a3fea294e706ac85fc819298267a6e71a7504d1be192d089744e0eb9
timestamp:     2026-02-28T03:30:30Z
event_type:    REGRESSION_DETECTED
confidence:    0.954

{
  "auto_proposals": 44,
  "auto_rollbacks": 24,
  "baseline_mean": 0.6737,
  "baseline_policy_version": 1,
  "baseline_sample_size": 70,
  "baseline_std_dev": 0.2452,
  "belief_count": 72,
  "belief_creation_rate": 6.5455,
  "convergence_status": "NOT_CONVERGED",
  "data_completeness_score": 1.0,
  "effect_size": 0.3869,
  "experiment_count": 44,
  "experiment_window": "2026-02-17T07:54:31Z / 2026-02-28T02:06:17Z",
  "final_mean": 0.6587,
  "final_sample_size": 81,
  "final_std_dev": 0.2322,
  "overflow_logged_in_failure_patterns": true,
  "policy_versions_created_per_day": 8.7273,
  "regression_detected": true,
  "rolling_30d_mean_trend": "no_previous_window",
  "rolling_30d_std_trend": "no_previous_window",
  "rolling_7d_mean_trend": "declining (0.6909 -> 0.5905)",
  "rolling_7d_std_trend": "declining (0.2699 -> 0.0895)",
  "statistical_power_score": 0.954,
  "t_test_p_value": 0.014852,
  "token_overflow_rate": 0.1239,
  "total_policy_versions": 96
}