Resolving IndexedDB Migrations

[magiccodingman]

Table-First Migration System - (Snapshot-Based Versioning)

I'm starting this new discussion thread as I'd like to move out of the alpha stage when both of the following is confirmed:
1.) General confidence in stability.
2.) The migration system is complete.

So, I'd like to propose the architecture I want to introduce to bring this project to its final form.

---

The Core Philosophy: Table-First Thinking

Unlike relational databases, IndexedDB stores are isolated and non-relational. Yet most migration tooling still thinks in terms of full database versioning, which doesn’t match the reality of how IndexedDB actually behaves.

So here’s the foundational shift:

We stop versioning the database. We start versioning the tables.

In IndexedDB, updating a “database version” is just a means to an end—it’s only there to update object stores (tables). The goal is not to upgrade the DB, but to bring each table to its correct state. That’s where this new system begins.

---

How It Works (High-Level Overview)

1. Each Table Has Its Own Version
   We treat tables as isolated, trackable units. Their version history is controlled independently, stored in an internal tracking table.

2. Snapshots Represent Each Version
   A CLI tool (powered by the Magic Scaffolding Protocol) creates table snapshots, which capture the schema state of a table at a given version.

3. Diffs Drive Migrations
   When a new snapshot is generated, the system compares it to the previous version and auto-generates migration steps. For common operations (like added fields or indexes), it scaffolds automatically.

4. Manual Conflict Handling with Assistive Guidance
   When renames or ambiguous changes are detected, the CLI pauses and provides conflict resolution prompts. Developers write small mapping functions to resolve these manually—but only when needed.

5. Stored Functions Bridge Versions
   When going from version 3 to 6, the system walks through and executes each required migration function in order. It never skips intermediate migrations unless explicitly marked as "collapsible."

6. Stored in Source Control
   Snapshots and migrations live in your repo (e.g., wwwroot/__migrations__) so they’re versioned with your code and deployed with your app.

---

Automation with Blazor / C#

This system is language-agnostic at its core, but full automation is supported for Blazor and C# projects via a NuGet + CLI tool combo.

• Just run magic-indexdb snapshot (or similar)
• It scans your project
• Creates table snapshots
• Detects diffs
• Applies validations
• Outputs any manual mappings needed

Non-C# devs can still follow the same protocol manually—the format is documented, and the enforcement system still provides immense value. But C# devs get a fully automated path.

---

Resilience: Failure Is Not an Option

Client-side IndexedDB corruption is unacceptable—especially for offline-first, decentralized apps. This protocol is designed to self-heal:

1. Detection: Schema drift or tampering is detected via internal metadata + snapshot diffs
2. Self-Recovery: If possible, the system autocorrects by rolling forward or back
3. Controlled Rebuild: If unrecoverable, the table is wiped and rebuilt (other tables remain untouched)
4. Internal Tracking Table: Used to store deployed versions per table

Your data stays safe, and your app doesn’t break—period.

---

Acknowledging IndexedDB's Nature

IndexedDB is not a relational system. No joins. No foreign keys. No built-in constraints.

So instead of fighting that, we embrace it:

• Tables are independent containers—like little Linux VMs.
• The database is just a bucket—its version is an implementation detail.
• This system enables drag-and-drop table reuse across apps and databases, since tables are fully isolated and schema-versioned.

---

What It Looks Like (Simplified)

// v1
{
  "Name": "string",
  "Age": "number"
}

// v2
{
  "FullName": "string", // renamed
  "Age": "number",
  "Email": "string"
}

If you ran a snapshot:

> magic-indexdb snapshot
[!] Conflict detected:
- 'Name' removed
- 'FullName' added

Do you want to:
[1] Rename 'Name' to 'FullName'
[2] Treat 'Name' as deleted
[3] Manual mapping required

Once resolved, the CLI generates a versioned function:

function migrate_1_to_2(record) {
  record.FullName = record.Name;
  delete record.Name;
  record.Email = "";
  return record;
}

---

Already Prototyped in Production Environments

This isn’t theory. I’ve built and run versions of this system between both production at scale and via prototyping. The scaffolding CLI has been deployed by my company for national use for years. It's merely a small addition. The dynamic scaffolding and migration system has been prototyped manually already. The current proposal is a formalization and automation of what’s already worked.

---

Call for Feedback

This is a radical shift, I know. But I believe it’s the right way forward for anyone trying to build serious apps with IndexedDB. I’m opening this up to hear:

• What do you like?
• What questions do you have?
• What would you want to see in the CLI?
• Are there edge cases or concerns you want addressed?

Let’s shape this together. The protocol, snapshot structure, CLI commands (which is actually both commands and a CLI GUI), and fallback options are still evolving—and I’d love to build this with the community’s input.
d I’d love to build this with the community’s input.

Thanks for reading!

[yueyinqiu]

Looks like it could be implemented, though maybe a little complicated. Anyway this sounds great!

[c-emin]

Each customer is assigned a Guid. I would like to be able to make a separate DB of that customer identified by their Guid.
I have it caching large amounts of data into the IndexedDb. In order to test/view a customer's data I would just "point" to the correct Guid for that customer. Otherwise, with a static one name Db, I would have to delete the Db and re-cache all the data for the different customer.

1.) Are you deploying new databases with pre-made tables? Yes, each DB will have all the same tables/fields
2.) When you deploy tables dynamically are you adding a different mix of tables? No, each DB will use the same Table schema.

Let me know if you need more clarification.

[yueyinqiu]

There is actually a hidden method QueryOverride behind. But the method is not included in the interface and is attributed with [Obsolete("Not decided if this will be built in further or removed")].

So if I'm not mistaken, the follow code should work:

IMagicQuery<Person> personQuery = await ((dynamic)_MagicDb).QueryOverride<Person>("databaseName", "objectStoreName");

I know this is not a good idea, but you can temporarily use it, and check whether exposing such a QueryOverride meets your need.

[c-emin]

I removed the Obsolete Attribute, uncommented the interface and I tried it a couple of ways... But still could not get it to create the Database with the name "MyDbName".

// Targets the default database automatically if called without any parameters.

IMagicQuery<Person> personQuery = await _MagicDb.Query<Person>();
await personQuery.ClearTable();                                             
//this works of course

IMagicQuery<Person> personQuery = await ((dynamic)_MagicDb).QueryOverride<Person>("MyDbName", "Person");      
await personQuery.ClearTable();
//'object' does not contain a definition for 'QueryOverride'

IMagicQuery<Person> personQuery = await _MagicDb.QueryOverride<Person>("MyDbName", "Person");                   
await personQuery.ClearTable();
//Database MyDbName doesnt exist

[magiccodingman]

Hmm, I see what you're saying. I'm hoping to have time to work on this again this weekend or the following week. But the current migration system is quite strict on it being required to know what databases it's working with.

But are you using a string database name purposely? Will you not know the database name until runtime because it's dynamic?

Is the deployment of the hardcoded repository system not working?

Just curious but in the documentation here:
https://sayou.biz/Magic-IndexedDB-Blazor-Installation

on the "Defining Your IndexedDB Schema" section with how you can deploy databases and tables separately. Is that not a possibility for you and you require real time runtime based table drops into tables that can only be created via runtime?

Never mind, after fully digesting your comment. I see exactly what you're saying. Okay I have a plan to resolve this

[magiccodingman]

@yueyinqiu Yea it is hidden and I don't even think I added it to the documentation. Because I was still scoping out how that would work. I think I disabled that somewhere because the system wasn't ready for dynamic plug and play table drops.

I should be able to carve out some time this weekend though to enable it.

Just got to finish doing dumb adult stuff before I can put my hands back on this project.