My web app for Spanish learning, EspaLuz, failed. It had a clean UI, a custom-built chat interface, and a free tier that attracted 100 sign-ups in its first month. Zero conversions to paid. My WhatsApp agent, built as a weekend hack, had three paying users within two weeks. The difference wasn't the AI; it was the friction. Users didn't want another app to open. They wanted to learn where they already were. This forced me to rethink conversation memory, especially with WhatsApp's stateless API and my zero-VC budget.
The Web App's Fatal Flaw: Forcing a New Habit
The web app's architecture was standard: React frontend, FastAPI backend, PostgreSQL for user data, and Pinecone for vector search. Each user interaction triggered an LLM call (initially OpenAI, later a mix of Groq and Claude via a custom router). Memory was handled by embedding recent conversation turns and querying Pinecone to retrieve relevant past interactions. This worked technically. The conversations were coherent. The problem was adoption.
Users would sign up, chat for 10-15 minutes, and then disappear. The feedback, when I could get it, was always the same: "It's good, but I already have Duolingo," or "I forget to open it." The cognitive load of remembering to visit espaluz.com was too high. I was asking them to change their daily routine for a product that wasn't orders of magnitude better than existing solutions.
The WhatsApp Pivot: Embracing Constraint
WhatsApp's API is brutally simple and stateless. Every incoming message is a new event. No built-in session management. No rich UI. Just text. This meant my existing Pinecone-based memory system was overkill and too expensive for a product I was trying to validate with zero budget. I needed an AI language learning WhatsApp conversation memory system that was cheap, robust, and could maintain context across days, not just minutes.
I decided on a two-layer memory approach, entirely in-memory and on-disk, without any paid vector store.
Layer 1: Short-Term, In-Memory Context Window
For immediate conversational flow, I implemented a simple, rolling context window. Each user's active conversation history (the last 10-15 turns, depending on token count) is stored in a Python dictionary in the FastAPI application's memory. This dictionary is keyed by the WhatsApp user ID.
When a message arrives:
1. Retrieve the user's current context list from the dictionary.
2. Append the new user message.
3. Pass the entire context list to the LLM.
4. Append the LLM's response to the context list.
5. Trim the context list to the maximum allowed turns/tokens, removing the oldest entries.
This short-term memory is volatile. If the FastAPI process restarts, or if a user is inactive for too long (e.g., 30 minutes), this memory is lost. This is where Layer 2 comes in.
Layer 2: Long-Term, Summarized Memory (PostgreSQL)
To provide true conversation continuity, even across days or weeks, I implemented a long-term memory system using PostgreSQL. This is where the "no vector store" constraint became a design driver.
Instead of embedding every message, I summarize entire conversation segments. Every 10-15 turns (or after a period of inactivity, like 30 minutes), the current short-term conversation history is passed to an LLM with a specific prompt: "Summarize the key learning points, vocabulary introduced, and grammatical concepts discussed in the following conversation segment. Focus on what the user struggled with or learned. Output in bullet points."
Example prompt:
You are an AI assistant tasked with summarizing a Spanish language learning conversation.
Summarize the key learning points, vocabulary introduced, and grammatical concepts discussed in the following conversation segment.
Focus on what the user struggled with or learned. Output in bullet points.
Conversation:
User: Hola, ¿cómo estás?
AI: ¡Hola! Estoy muy bien, gracias. ¿Y tú?
User: Bien. Quiero practicar los verbos "ser" y "estar". Me confunden.
AI: Entendido. "Ser" es para cosas permanentes, como "Soy estudiante". "Estar" es para estados temporales, como "Estoy cansado". ¿Puedes darme un ejemplo con "ser"?
User: Yo soy en la casa.
AI: Casi. "Estoy en la casa" es correcto porque la ubicación es temporal. "Soy" es para identidad. Intenta de nuevo.
User: Ah, entiendo. Yo estoy feliz.
AI: ¡Perfecto! "Estoy feliz" es correcto. La felicidad es un estado temporal.
Summary:
The LLM's summary (typically 100-300 tokens) is then stored in a conversation_summaries table in PostgreSQL, linked to the user ID and a timestamp.
When a user returns after a long absence, or if their short-term memory is lost:
1. The system queries PostgreSQL for the most recent 3-5 summaries for that user.
2. These summaries are prepended to the prompt sent to the LLM, effectively "reminding" the AI of past learning.
3. The LLM then generates a response, incorporating this long-term context.
This approach is significantly cheaper than a vector store:
- No embedding costs for every turn: Only for summaries, which happen much less frequently.
- No vector database hosting costs: Standard PostgreSQL is sufficient.
- Reduced LLM calls for retrieval: Summaries are retrieved via simple SQL queries, not complex vector searches.
The trade-off is precision. The AI doesn't recall every specific phrase from weeks ago, but it remembers the gist of what was covered, which is often sufficient for language learning.
Oracle Cloud Infrastructure: The Free Tier Advantage
My entire backend runs on Oracle Cloud Infrastructure (OCI) Free Tier. This is critical for a bootstrapped project.
- Always Free Compute: An Ampere A1 instance (4 OCPU, 24 GB RAM) runs my FastAPI application, Groq/Claude router, and WhatsApp webhook.
- Always Free Autonomous Database: A PostgreSQL-compatible instance handles all user data and conversation summaries.
- Network Egress: The free tier includes 10 TB of egress per month, more than enough for my current traffic.
This setup costs me $0 per month for infrastructure, allowing me to focus my limited funds on LLM API calls.
The 3 Paying Users vs. 100 Free Sign-ups
The three paying users of the WhatsApp agent taught me more than 100 free sign-ups on the web app ever could.
1. "I don't want to open another app." This was the most common feedback. WhatsApp is where they already spend their time. The friction of context switching was a deal-breaker for the web app.
2. "It remembers what I learned last week." The two-layer memory, especially the PostgreSQL summaries, made a huge difference. Users felt a continuous learning journey, even if they only chatted for 10 minutes a day, a few times a week. This was the core value proposition that converted them.
3. "It's like having a tutor in my pocket." The conversational nature, combined with the memory, created a sense of personalized instruction. They weren't just drilling vocabulary; they were having a conversation that adapted to their progress.
These users were willing to pay $9/month because the product integrated seamlessly into their lives and provided tangible value: consistent, personalized Spanish practice without extra effort. The web app, despite its technical sophistication, failed because it ignored user behavior.
My current agent routing uses a simple Python dictionary to map user IDs to preferred LLM providers (Groq for speed, Claude for complex explanations). This allows me to dynamically switch based on cost and performance, and even A/B test different models for specific user segments.
Frequently Asked Questions
Q: Why not use a simple database table for raw message history instead of summaries?
A: Storing raw message history would quickly exceed context window limits for LLMs, requiring expensive embedding and vector search for retrieval. Summaries are compact, cheaper to store, and provide higher-level context relevant to learning, reducing token costs for LLM prompts.
Q: How do you handle concurrent messages from the same user on WhatsApp?
A: WhatsApp's API sends messages sequentially. My FastAPI application processes them in order. For very high concurrency, a message queue (like Kafka or RabbitMQ) would be necessary, but for a single user, the in-memory dictionary and database updates are sufficient and fast enough.
Q: What if the LLM summary is poor or misses key details?
A: This is a risk. I continuously refine the summary prompt and monitor summary quality. For critical information, I might add specific tags or keywords to the summary prompt to ensure certain elements (e.g., "user struggled with 'ser' vs 'estar'") are always included.
Q: How do you manage the cost of LLM API calls with zero VC funding?
A: I aggressively use Groq for most interactions due to its low cost and high speed. Claude 3 Haiku is used for more complex explanations or when Groq struggles. I also implement token limits on responses and use efficient prompt engineering to minimize input tokens. My current LLM spend is under $50/month.
Q: What's the biggest technical challenge you faced with this setup?
A: Ensuring robust error handling and retry mechanisms for WhatsApp API calls and LLM interactions. WhatsApp can have transient issues, and LLMs can occasionally return malformed JSON or rate limit. Implementing exponential backoff and clear logging was crucial for stability.