I Decrypted My Own Notes to Feed an AI. Privacy and Memory Can't Coexist Yet.

The first decrypted note appeared in my terminal at 11:47 PM on a Sunday. It was a half-finished paragraph about a project I'd abandoned three weeks ago. Something I'd typed into Obsidian on my phone while waiting for coffee, then forgotten about completely.

My AI agent hadn't forgotten. It had pulled the note from my Obsidian LiveSync vault, decrypted the E2EE layer, and was reading it in real-time. 400 milliseconds after I'd finished typing.

I sat there for a minute. Not because the decryption was impressive (it was, but that's not the point). Because I'd spent six months building Engram, a memory engine that gives AI agents persistent, graph-based memory. And the single biggest limitation was always the same: the agent only knew what I explicitly told it. It didn't know what I was thinking. Now it does.

The Problem With AI That Can't Read Your Notes

I run my entire knowledge system through Obsidian. Project plans, architecture decisions, half-baked ideas, journal entries, meeting notes. Everything lives in a vault synced across my phone, laptop, and the $7 VPS that runs my agent infrastructure.

The sync layer is Self-hosted LiveSync, a community plugin that uses CouchDB as the replication backend. It's the only free option that gives you real-time sync, chunk-level conflict resolution, and end-to-end encryption. I enabled E2EE the day I set it up. The passphrase encrypts every note before it leaves my device. CouchDB stores ciphertext. Nobody can read my notes on the server.

That was the whole point.

But I kept hitting the same wall with my agent loop. Rex (my AI agent) would ask me about a project, and I'd realize the context it needed was sitting in an Obsidian note I'd written two days ago. I'd copy-paste the note into the conversation. Or summarize it from memory. Or just tell Rex to figure it out without the context and watch it produce something mediocre.

The friction was constant. I had a memory engine. I had a vault full of knowledge. They couldn't talk to each other because I'd encrypted the bridge.

Breaking My Own Encryption

I spent an evening reading the LiveSync source code. The encryption scheme is solid. PBKDF2-SHA256 with 310,000 iterations derives a master key from my passphrase. HKDF-SHA256 derives a unique key per chunk. AES-256-GCM handles the symmetric encryption with a 12-byte IV and a 16-byte auth tag.

The binary layout for each encrypted chunk looks like this:

%=<base64( IV[12 bytes] + hkdfSalt[32 bytes] + ciphertext + authTag[16 bytes] )>

That %= prefix is important. It signals the newer HKDF-scheme encryption. Older LiveSync versions used a different format. If your decryption code doesn't check the prefix, you'll silently fail on older notes.

I implemented a LiveSyncCrypto class in TypeScript. The master key derivation is one call:

crypto.pbkdf2Sync(passphrase, pbkdf2Salt, 310_000, 32, 'sha256')

Per-chunk key derivation:

crypto.hkdfSync('sha256', masterKey, hkdfSalt, Buffer.alloc(0), 32)

Then AES-256-GCM decrypt. Straightforward, once you know the layout.

The part I didn't expect: the PBKDF2 salt doesn't live in the encrypted payload. LiveSync stores it in a CouchDB local document at _local/obsidian_livesync_sync_parameters. Local documents don't replicate. If you rebuild CouchDB from a backup, that salt is gone, and every decryption attempt fails with no useful error message. I lost an hour to this before I added auto-fetch at startup.

The Changes Feed and the Chunk Problem

CouchDB has a _changes endpoint that streams every document modification as newline-delimited JSON. Open a persistent HTTP connection with feed=continuous, and you get a real-time firehose of every edit in the database.

But LiveSync doesn't store notes as single documents. It uses content-defined chunking. Edit one paragraph in a long note, and LiveSync only syncs that chunk. Each chunk is a separate CouchDB document with type: 'leaf' and e_: true. The parent document holds a children[] array of chunk IDs, and sometimes (not always) embeds chunk data in an eden field.

Reassembly means fetching every chunk in order, decrypting each one, and concatenating the plaintext. If one chunk fetch fails, you get garbled output. And here's the thing nobody tells you: a single edit fires multiple _changes events (the parent doc update plus each chunk update). Without debouncing, you'll decrypt and analyze the same note four times in three seconds.

I settled on a 30-second debounce window. It catches the rapid-fire events from LiveSync without adding noticeable delay.

The Feedback Loop Bug

This one was fun.

The pipeline works like this: CouchDB streams a change, I decrypt the note, fork the result into Engram for memory ingest and a BullMQ job queue for AI analysis. Rex picks up the job, analyzes the note, and writes its analysis back to the vault as VPS Agent/Analysis - notePath.

Writes go through the same encryption pipeline in reverse. Generate random IV and HKDF salt, encrypt with AES-256-GCM, create chunk doc, create parent doc, push to CouchDB. LiveSync syncs the encrypted note to all my devices.

The bug: that write triggers another _changes event. Rex sees a new note change. Analyzes it. Writes another analysis. Sees another change. My phone started buzzing with Telegram notifications at two per second.

The fix was trivial. Track every doc ID you write to an ownWrites set. Skip any change event matching a tracked ID for 60 seconds. But finding the bug required watching the recursion happen in real time, which was both horrifying and a little funny.

What the System Actually Does Now

The full architecture, running on the same VPS that hosts six MCP servers:

1. CouchDB _changes feed streams every vault edit in real-time
2. Filter out config files (.obsidian/), own writes, deleted notes, and empty docs
3. Debounce rapid edits (30s window)
4. Fetch and decrypt all chunks. SHA256-hash the content, skip if unchanged
5. Fork: ingest into Engram (memory graph) and queue for Rex analysis (BullMQ)
6. Rex reads the note, cross-references with existing memory, writes an analysis note back
7. Telegram notification with an obsidian:// deep link

Twelve tests pass against real CouchDB. The watcher runs as a persistent service alongside everything else on the box. It's been stable for a week.

And it's genuinely useful. Rex now surfaces connections between notes I'd forgotten about. It flags action items I buried in journal entries. It cross-references project plans with actual progress. The gap between "what I know" and "what my AI knows" has collapsed.

The Privacy Paradox I Can't Resolve

Here's where I stop feeling clever and start feeling uneasy.

I set up E2EE specifically so that nobody could read my notes on the server. The passphrase was my guarantee. Even if someone compromised the VPS, they'd get ciphertext.

Then I wrote code that stores the passphrase in a .bashrc env var on that same VPS and decrypts every note in real-time. I am the threat model I was protecting against.

Some things that bother me:

The agent has no ACL. It reads every note in the vault. Journal entries, half-formed complaints about work, financial planning notes, gift ideas for my wife. There's no "skip this folder" option. I could add one, but I haven't, because the value comes from the agent knowing everything.

There's no forget mechanism. Once Engram ingests a note, that memory lives in the graph permanently. If I write something I regret and delete it from Obsidian, the memory persists. The note is gone, but the knowledge isn't. And there's no reliable way to audit what an AI system actually retains.

The 60-second ownWrites window is fragile. If Rex's analysis takes longer than 60 seconds to complete and write back, it'll re-trigger analysis of its own output. I haven't stress-tested this. I probably should.

Content hashes sit in a JSON state file on disk. Even without the passphrase, someone with access to that file could see which notes changed and when. It's a metadata leak, not a content leak. But metadata alone can be weaponized in connected systems.

What This Means for Everyone Building AI Memory

The industry is publishing a lot of thoughtful work on agent memory and privacy right now. MIT Tech Review called it "privacy's next frontier." Stanford published the Opal paper on private AI memory with trusted execution environments. ClawSouls shipped encrypted memory sync. Mem0 added a whole section on privacy to their 2026 State of Agent Memory report.

All of this assumes a clean separation: memory on one side, encryption on the other. The agent works within the encrypted boundary, or the data never enters the agent at all.

I don't think that separation holds. Not for personal AI. The entire value of a memory system like Engram is that it knows what you know. And what you know lives behind the encryption you set up to protect it. You can have an agent that respects your encryption boundaries and knows nothing useful about you. Or you can have an agent that reads everything and is genuinely helpful. I haven't found the middle ground. And every layer you add to bridge the gap creates new verification debt you didn't plan for.

Maybe the answer is something like Opal's TEE approach, where the decryption happens inside trusted hardware and the plaintext never hits disk. Maybe it's per-folder encryption with selective access. Maybe the answer is that personal AI memory requires a fundamentally different trust model than the one we inherited from cloud storage.

I'm not sure yet. What I know is that the system I built works, my agent is sharper than it's ever been, and I'm still not totally comfortable with what I gave up to get here.