Storing No Conversation Logs, and Keeping Costs Down

Introduction

When people roll out an AI assistant, the instinct is usually "let's analyze the conversation logs and improve it." That's genuinely a good approach. For this build, though, we deliberately chose to store no conversations at all.

This article covers why we made that call, what we gave up in exchange, and the work we did to keep costs down.

Choosing not to store conversations

Guest questions get personal

What a guest asks during a stay isn't limited to how the appliances work. Health issues, situations with the people they're travelling with, personal worries — any of it can surface.

Having that readable by the operator didn't feel right as part of the guest experience. So we decided that questions, answers, and even summaries of them are never written to a database, a file, or a log. Anywhere.

When an error occurs, the only thing recorded is the type of error. The content of the question isn't kept.

What we gave up

There's a clear price for this decision.

All of that would be genuinely useful for improving operations. We chose not to store it anyway, because guest privacy came first. Improvement instead comes from direct guest feedback and from the operator testing the assistant themselves.

Whether that's the right call depends on the situation, but what matters is making the choice with the trade-off fully understood.

The one thing we do keep

There's a single exception: the browser stores a count of how many questions have been asked. Its only purpose is to avoid recommending the same handful of local spots every time.

Not the content of the questions — just the count. And it never goes to the server. That felt like an acceptable line.

Answering without calling the AI

Most of the cost disappears by not calling

An AI API charges you every time you call it. Which also means: if you can avoid the call, there's no charge.

In real use, most questions are resolved before step three. The AI only runs for questions we didn't see coming.

Don't fetch what you won't use

The feature that searches for nearby places can also return photo data. But the AI's answers never use photos.

Even so, we hadn't turned photo retrieval off at first, which produced more than 2,000 wasted calls a month. In money, a bit over 1,000 yen monthly. A small amount, but not one you can ignore if it runs unnoticed forever.

The fix was making the caller specify that photos aren't needed. It's just honoring the basic rule: don't go fetch what you won't use.

Cap how much text you pass

The volume of source material passed to the AI translates directly into cost. So we set a character limit and truncate anything beyond it.

The clever bit is the truncation order. Our own documents go first and externally fetched information goes last, so when the limit is hit, it's the external material that gets cut. The ordering ensures the important information is never the first thing to disappear.

Originally we limited by number of chunks, but chunk lengths vary, which caused a bug where area information was trimmed every single time. Switching the limit to a character count fixed it.

Building the knowledge is a one-time cost

Converting documents into searchable form happens once, at ingestion. It doesn't run again on every question.

The actual bill came to a few hundred yen for ingesting over 100 pages of material. You pay once, and after that you're only searching.

Preventing abuse and runaway usage

Rate limits per booking, not per connection

A flood of questions in a short window sends costs through the roof. So we capped it at 12 questions per minute.

The refinement was applying the limit per guest booking rather than per connection source. A family staying at the same property shares one Wi-Fi connection, so counting by connection means one heavy user locks everybody out.

Limit the input length too

Questions are capped at 300 characters. A pasted wall of text costs proportionally more API spend, and 300 characters is plenty for a question about the property's facilities.

Make every failure look identical

An invalid URL, an access attempt outside the usable window, a request for an appliance that doesn't exist — all of these return the same "not found" response.

Distinguishing error types lets someone infer things like "this URL exists but has expired." Giving every failure the same face is the safer choice.

Wrapping up

Three things mattered most in the operational design of this assistant.

1. If you decide not to store it, go all the way — return no diagnostic data, and structurally remove any chance of a record
2. Cut costs by not calling — narrow with menus, and never fetch information you won't use
3. Pick the right unit for your limits — rate limits apply per booking so one guest can't lock out their companions