Getting started¶

Get DTDE running in five minutes. By the end you'll have a sharded DbContext, three logical shards backed by real per-shard tables, and a working LINQ query that fans out across them.

Prerequisites¶

• .NET 8 SDK or newer (the package multi-targets net8.0, net9.0, net10.0).
• An EF Core provider — any will do. This guide uses SQLite because it has zero setup.
• About five minutes.

How DTDE actually shards your data¶

Two layers, one contract:

1. Per entity (in OnModelCreating) you declare how a row maps to a shard key — entity.ShardBy(c => c.Region) for property-value sharding, ShardByDate(...), ShardByHash(...), etc.
2. In Program.cs you declare the shards themselves — dtde.AddShards("EU", "US", "APAC") for table-mode (one DB, many tables) or dtde.AddShard("EU", "Server=eu-db;...") for database-mode (a separate DB per shard).

The contract that ties them together: the shard's id (the string you pass to AddShard("EU") or AddShards(...)) must equal the value the entity's shard-key property carries at runtime. So a Customer row with Region = "EU" lands in the shard registered with id "EU". If the property value doesn't match any registered shard, the row has nowhere to go and DTDE will throw at write time.

That's the whole mental model. The rest is just choosing where shards live.

1. Install the package¶

dotnet add package Dtde.EntityFramework
dotnet add package Microsoft.EntityFrameworkCore.Sqlite

Dtde.EntityFramework transitively pulls in Dtde.Core and Dtde.Abstractions.

2. Define an entity¶

public class Customer
{
    public int Id { get; set; }
    public string Name { get; set; } = string.Empty;
    public string Region { get; set; } = string.Empty; // shard-key property
    public DateTime CreatedAt { get; set; }
}

Any string-ish property works — region codes, tenant ids, whatever your domain calls it. DTDE makes no assumption about the name or content.

3. Inherit DtdeDbContext, declare the shard key in OnModelCreating¶

using Dtde.EntityFramework;
using Dtde.EntityFramework.Extensions;
using Microsoft.EntityFrameworkCore;

public class AppDbContext : DtdeDbContext
{
    public DbSet<Customer> Customers => Set<Customer>();

    public AppDbContext(DbContextOptions<AppDbContext> options)
        : base(options) { }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);

        modelBuilder.Entity<Customer>(entity =>
        {
            entity.HasKey(c => c.Id);
            entity.ShardBy(c => c.Region);   // <-- the only DTDE-specific line
        });
    }
}

ShardBy(c => c.Region) says "use the value of Region as the shard key". That's all the model needs to know.

4. Wire it up in Program.cs¶

One call, two callbacks:

using Dtde.EntityFramework.Extensions;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddDtdeDbContext<AppDbContext>(
    (db, conn) => db.UseSqlite(conn ?? "Data Source=app.db"),
    dtde => dtde.AddShards("EU", "US", "APAC"));

var app = builder.Build();

What's happening:

• The first lambda — (db, conn) => db.UseSqlite(conn ?? "...") — configures the EF Core provider. DTDE invokes it twice: once for the parent context (with conn = null → fall through to your default) and once per shard (with conn set to that shard's connection string).
• The second lambda — dtde => ... — declares your shards.

Above is table-mode: one SQLite file, three logical shards, three tables (Customers_EU, Customers_US, Customers_APAC) created automatically by DTDE's per-shard model.

Want database-mode (one database per shard)? Same call, you just supply each shard's connection string:

builder.Services.AddDtdeDbContext<AppDbContext>(
    (db, conn) => db.UseSqlite(conn ?? "Data Source=base.db"),
    dtde => dtde
        .AddShard("EU", "Data Source=eu.db")
        .AddShard("US", "Data Source=us.db")
        .AddShard("APAC", "Data Source=apac.db"));

Each per-shard context now connects to its own database; the table is named Customers everywhere because there's no name conflict — different DB.

Want mixed mode — per-shard tables spread across multiple databases (e.g. EU and US tables in one regional DB, APAC tables in another)? Use the AddTableShardInDatabase helper:

builder.Services.AddDtdeDbContext<AppDbContext>(
    (db, conn) => db.UseSqlite(conn ?? "Data Source=base.db"),
    dtde => dtde
        .AddTableShardInDatabase("EU",   "Data Source=primary.db")
        .AddTableShardInDatabase("US",   "Data Source=primary.db")
        .AddTableShardInDatabase("APAC", "Data Source=secondary.db"));

primary.db ends up with Customers_EU + Customers_US; secondary.db ends up with Customers_APAC. Each shard gets its own per-shard table (table-mode rewriting still applies), but the tables are spread across the databases you choose.

5. Provision the shards (one-time)¶

For samples, integration tests, or a fresh dev environment, call:

using (var scope = app.Services.CreateScope())
{
    var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
    await db.EnsureAllShardsCreatedAsync();
}

This walks every registered shard and creates its tables (table-mode) or its database + tables (database-mode). In production you'd usually run EF Core migrations per shard instead.

6. Use it¶

Standard EF Core LINQ:

public class CustomersController(AppDbContext db) : ControllerBase
{
    [HttpPost]
    public async Task<IActionResult> Create(Customer customer)
    {
        db.Customers.Add(customer);     // routed by Customer.Region
        await db.SaveChangesAsync();
        return Ok();
    }

    [HttpGet]
    public Task<List<Customer>> All() => db.Customers.ToListAsync();
    // No filter on shard key → fans out across all shards, merges results.

    [HttpGet("{region}")]
    public Task<List<Customer>> ByRegion(string region) =>
        db.Customers.Where(c => c.Region == region).ToListAsync();
    // Where on shard key → DTDE prunes to the matching shard only.
}

Three rules, summarised:

• Inserts: routed by the shard-key value on the entity.
• Queries with Where on the shard key: pruned to one shard.
• Queries without: fanned out across all shards, results merged.

What if I need...¶

...time-bucketed sharding (transactions, audit logs)¶

// Entity
modelBuilder.Entity<Order>()
    .ShardByDate(o => o.OrderDate, DateShardInterval.Year);

// Program.cs — one shard per year
dtde => dtde.AddShards("2023", "2024", "2025")

The order's year (extracted from o.OrderDate) becomes the shard key.

...hash-based even distribution¶

modelBuilder.Entity<UserProfile>()
    .ShardByHash(u => u.UserId, shardCount: 8);

dtde => dtde.AddShards("0", "1", "2", "3", "4", "5", "6", "7")

DTDE computes the user-id hash modulo 8 to pick the shard.

...different shard topologies for different entities — shard groups¶

The example above works only because every entity in the DbContext shares the same shard topology. The moment you have two entities with different shard counts — say, eight hash buckets for users and three yearly buckets for orders — you need to tell DTDE which shards belong to which entity. That's what shard groups are for.

A shard group is a named set of shards. Entities pick the group they live on; shard ids are unique within a group, so "0" in a hash8 group is a different physical shard from "0" in a hash3 group.

// Program.cs
builder.Services.AddDtdeDbContext<AppDbContext>(
    (db, conn) => db.UseSqlite(conn ?? "Data Source=app.db"),
    dtde => dtde
        .AddShardGroup("hash8",  g => g.AddShards("0","1","2","3","4","5","6","7"))
        .AddShardGroup("years",  g => g.AddShards("2023","2024","2025")));

// AppDbContext.OnModelCreating
modelBuilder.Entity<UserProfile>()
    .ShardByHash(u => u.UserId, 8)
    .UseShardGroup("hash8");

modelBuilder.Entity<Order>()
    .ShardByDate(o => o.OrderDate, DateShardInterval.Year)
    .UseShardGroup("years");

Per group, all the rules from the rest of the guide apply: AddShards puts the shards in table-mode (one DB, per-shard tables); AddShard(id, conn) puts them in database-mode (one DB per shard); AddTableShardInDatabase gives you mixed-mode (per-shard tables across multiple databases). Each group has its own table-mode / database-mode / mixed-mode choice — they don't have to agree.

Default-group rule keeps the simple case simple: if you don't call AddShardGroup and don't call UseShardGroup, every shard goes into the implicit default group and every entity binds to it. The original AddShards("EU","US","APAC") + entity.ShardBy(c => c.Region) example above still works without any changes.

...point-in-time queries (temporal entities)¶

Add ValidFrom/ValidTo properties on the entity, declare them in OnModelCreating, then query with db.ValidAt<T>(date):

public class Contract
{
    public int Id { get; set; }
    public string ContractNumber { get; set; } = string.Empty;
    public DateTime ValidFrom { get; set; }
    public DateTime? ValidTo { get; set; }
}

// AppDbContext.OnModelCreating
modelBuilder.Entity<Contract>()
    .HasTemporalValidity(c => c.ValidFrom, c => c.ValidTo);

// Querying
var asOfLastMonth = await db
    .ValidAt<Contract>(DateTime.UtcNow.AddMonths(-1))
    .ToListAsync();

...customising the per-shard table-name pattern¶

Default is {Table}_{ShardId} (so Customers_EU). Override per entity:

modelBuilder.Entity<Customer>()
    .ShardBy(c => c.Region)
    .WithTablePattern("{Table}__shard_{ShardId}");   // → Customers__shard_EU

Tokens: {Table} (entity table name), {Schema} (default dbo), {ShardId}.

...shard configs from JSON instead of code¶

dtde => dtde.AddShardsFromConfig("shards.json");

JSON schema in Configuration reference.

...atomic writes across multiple shards (cross-shard transactions)¶

SaveChangesAsync on the parent context auto-promotes to a cross-shard transaction when changes span multiple shards — that's transparent and nothing extra is required. For explicit control, open one yourself:

await using var tx = await db.BeginCrossShardTransactionAsync();

var euCtx = (await ((CrossShardTransaction)tx).GetOrCreateParticipantAsync(euShard)).Context;
var usCtx = (await ((CrossShardTransaction)tx).GetOrCreateParticipantAsync(usShard)).Context;

euCtx.Set<Customer>().Add(new Customer { Region = "EU", ... });
usCtx.Set<Customer>().Add(new Customer { Region = "US", ... });

await tx.CommitAsync();

DTDE runs a two-phase-commit (2PC) across the enlisted shards: prepare everywhere first, only commit if every shard votes yes. Disposing the transaction without committing rolls every participant back. With exactly one shard enlisted at commit time, DTDE skips the prepare phase (single-shard fast path) — same atomicity, less overhead.

Tweak isolation, timeout, and retry behaviour via CrossShardTransactionOptions:

var options = new CrossShardTransactionOptions
{
    IsolationLevel = CrossShardIsolationLevel.Serializable,
    Timeout = TimeSpan.FromSeconds(15),
};
await using var tx = await db.BeginCrossShardTransactionAsync(options);

...high-throughput inserts and deletes (bulk operations)¶

BulkInsertAsync routes each entity to its target shard, batches per shard, and commits all shards together (2PC) when more than one is touched:

var newCustomers = new List<Customer>
{
    new() { Region = "EU", ... },
    new() { Region = "US", ... },
    new() { Region = "EU", ... },
    // ...
};

var inserted = await db.BulkInsertAsync(newCustomers);

BulkDeleteAsync fans an ExecuteDelete out across every shard in the entity's shard group — set-based, no SELECT round-trip, no change tracker:

var deleted = await db.BulkDeleteAsync<Customer>(c => c.LastSeen < cutoff);

For cross-shard ExecuteUpdate (the EF Core 7+ bulk-update API), open a cross-shard transaction yourself and call ExecuteUpdateAsync on each participant's context — the public extension method here is intentionally small because EF Core 8/9 use SetPropertyCalls<T> while EF Core 10 moved to UpdateSettersBuilder<T>.

The single canonical map¶

Next steps¶

• Sharding guide — when each strategy is appropriate.
• Temporal guide — bi-temporal versioning, point-in-time queries.
• Cross-shard transactions — 2PC across shards.
• Sample projects — six runnable apps, one per strategy.