Skip to content

Sync Engine v2 — Database Design (built to absorb every future case)

Companion to sync-engine-v2-plan.md. This is the schema of record. Design goal: the DB structure is the durable part — new activity types, new sport groups, new ranking metrics, new leaderboard dimensions (team / company / region / anything), and periodic boards must all be config or query changes, never migrations.

Three design principles (why the schema survives the future)

  1. Immutable log + materialized aggregate + derived periods. sync_activities is the append-only source of truth (one row per Strava activity, full raw + promoted columns). Every total is derived from it. sync_entity_totals is a materialized all-time cache for fast reads. Any weekly/daily/custom-window board is a query over sync_activities — no new storage needed. ⇒ Consequence: totals can always be rebuilt from the log; a config change can never corrupt history, and re-grouping is just a recompute.

  2. Entity polymorphism — ONE totals table for every dimension. Legacy uses a separate table per dimension (individual / team / company). We use a single sync_entity_totals keyed by (entity_type, entity_ref). Adding "region", "age band", "gender", or any future leaderboard dimension is a new enum value + tally fan-out — zero new tables.

  3. Metric-agnostic ranking. Totals carry the handful of real metrics as columns (distance, moving_time, elevation, calories, count); each group declares which metric it ranks by (event_activities.metric). Hours-mode, elevation challenges, activity-count challenges all fall out of one column choice — no schema fork per tracking mode.


CONFIG LAYER — per-event definition (source of truth for grouping)

strava_activity — global master catalogue (wire up: seed from 6 → full list)

coltypenote / future case
idbigint PK
slugvarchar(64) UNIQUEStrava sport_type string (Ride, VirtualRun, EBikeRide)
namevarchar(128)friendly label ("Outdoor Ride")
default_contextenum('outdoor','indoor')default indoor/outdoor hint
is_activetinyint(1)selectable in the editor
timestamps

Future: a brand-new Strava type Strava invents = one row here. Nothing else changes.

event_activities — named groups per event (= ranking dimension = frontend chip)

coltypenote / future case
idbigint PKreferenced by totals as group_id
event_idbigint
event_leaderboard_tab_idbigint NULLwhich tab the group shows under
namevarchar(128)"Cycling", "Run/Walk"
slugvarchar(64)stable chip key the frontend/URL uses
metricenum('distance','moving_time','elevation','calories','activities') default 'distance'what this group ranks by
is_overalltinyint(1)marks the whole-event "Overall" group (union of all counted)
enabletinyint(1)
sort_orderintchip order
timestamps
UNIQUE(event_id, slug)

Future: any number of groups; hours-mode = metric='moving_time'; an elevation challenge = metric='elevation'. Overall is just a group with is_overall=1.

event_activity_map_strava — group membership (which Strava types feed a group)

coltypenote / future case
idbigint PK
event_activity_idbigint FK
strava_activity_idbigint FK → strava_activity
contextenum('outdoor','indoor','both') default 'both'count only when the activity matches
manual_allowtinyint(1)allow manual entries of this type into this group
timestamps
UNIQUE(event_activity_id, strava_activity_id, context)

Future / overlap: the SAME strava type maps into many groups (many rows) — one activity then contributes to every group it belongs to. E-bike case: put EBikeRide in the Overall group but NOT in Cycling → counts to total, not to cycle, no code. Swim pool/open-water: two rows, context indoor vs outdoor. Allow-list = union of all enabled groups' memberships.

(Reused config, unchanged: event_leaderboard_tab (tabs + table_column JSON), sync_event_settings (engine/shadow/rules/connection), leaderboard_setting.)


RUNTIME LAYER — engine output

sync_activities — immutable per-activity log (SOURCE OF TRUTH) (exists; add one index)

Keep as-is (event_id, activity_id, tgp_userid, sport_type, distance, moving_time, total_elevation_gain, calories, start_date_local, status, raw JSON, evaluation cols…). Add:

INDEX sync_act_period (event_id, tgp_userid, start_date_local)   -- powers derived periodic boards

Future: every all-time total AND every weekly/daily/custom board derives from this table. Change a group definition mid-event → sync:rebuild recomputes everything from here. No data loss.

sync_entity_totals — THE unified materialized aggregate (NEW — replaces sync_totals/team/group)

coltypenote / future case
idbigint PK
event_idbigint
group_idbigint NULLFK event_activities; NULL = overall whole-event total
entity_typeenum('user','team','group','region')the leaderboard dimension
entity_refvarchar(191)natural key: tgp_userid | tga_team_id | company name | region name
distance_kmdouble
moving_time_sint
elevation_mdouble
caloriesdouble
activities_countint
members_countint NULLteams/companies/regions (null for user)
last_activity_atdatetime NULL
rankintcomputed per the group's metric within (event, group, entity_type)
dirtytinyint(1)RankStep recompute flag
timestamps
UNIQUE(event_id, group_id, entity_type, entity_ref)
INDEX(event_id, group_id, entity_type, rank) ← the leaderboard read

Future — this one table covers ALL of: {overall, per-group} × {user, team, company, region}. A new dimension (age band, gender, corporate tier) = a new entity_type enum value + a tally fan-out line. No new table, ever. Any metric shown per row is already a column.

sync_period_totals — periodic cache (DEFERRED — build only if derivation is too slow)

Same shape + period_type enum('all','weekly','daily','custom'), period_key varchar(24) (2026-W32). Not built now; weekly boards derive from sync_activities on demand first.

(Ops tables, unchanged: sync_runs, sync_quota_samples, sync_alerts, sync_dead_letters, sync_activity_log, sync_settings_audit, sync_engine_state.)


Algorithms

Tally (fan-out). For each accepted activity:

  1. Resolve entities of the actor: user=tgp_userid; team=team_users membership(s); group= event_users.group; region=event_users.subdistrict (each only if that dimension's tab is on).
  2. Resolve groups: all event_activities whose membership contains the activity's sport_type honoring context/manual_allow, plus the is_overall group (NULL group_id).
  3. For each (group × entity) upsert the sync_entity_totals row: add distance/time/elevation/calories, bump count, advance last_activity_at, set dirty=1. (metres→km once, here.)

Rank. For each (event, group_id, entity_type) with dirty rows: ROW_NUMBER() OVER (ORDER BY <metric-column> DESC) → write rank, clear dirty. Metric column chosen from the group's metric. Debounced (skip if ranked <60s ago).

Rebuild (sync:rebuild --event=). Truncate sync_entity_totals for the event, replay all counted rows in sync_activities through Tally, then Rank. Used after any group-definition change. This is the safety net that makes config edits fearless.

Periodic (derived). SELECT entity, SUM(distance) FROM sync_activities WHERE event_id=? AND status='counted' AND start_date_local BETWEEN ? AND ? GROUP BY entity ORDER BY 2 DESC. Cache in sync_period_totals only if measured cost demands it.


Every-future-case matrix

Future caseHandled byMigration?
Strava adds a new activity typestrava_activity rowno
New sport group for an eventevent_activities row + membershipsno
A type in several groups (overlap)multiple event_activity_map_strava rowsno
Same type counted diff indoor/outdoorcontext on membershipno
Manual entries per type/groupmanual_allow + sync_activities.manualno
E-bike counts to total but not cyclein Overall group, not in Cycling groupno
Rank by time (hours events)event_activities.metric='moving_time'no
Rank by elevation / calories / activity countmetric valueno
Team leaderboardentity_type='team'no
Company / Impact-Partner boardentity_type='group'no
Region / district boardentity_type='region'no
A NEW dimension (age band, gender, tier)new entity_type enum value + 1 tally lineno (enum alter only)
Weekly / daily / custom-window boardderive from sync_activities (opt. sync_period_totals)no
Change group definition mid-eventsync:rebuild from the logno
Multiple metrics displayed per rowalready columns on sync_entity_totalsno
Qualification gate (donation)donation-side qualified, merged at readno
Audit / history / config change trailsync_activity_log, sync_settings_auditno

If a requested future case is NOT in this table, that is the signal to revisit the schema — the intent is that everything above is config/query, and only a genuinely new kind of data is a migration.

Migration from the current shadow schema

The shadow engine currently writes sync_totals + sync_team_totals with a binary cycling/running split. Because the engine is off / shadow and event 52 has no data, now is the zero-cost moment to switch:

  1. Add sync_entity_totals (guarded migration). 2. Repoint TallyStep/RankStep to it (fan-out + metric rank). 3. Retire sync_totals/sync_team_totals writes (no live readers — sync-ops admin repoints to the new table). 4. Optional: expose a sync_totals-shaped VIEW over sync_entity_totals WHERE group_id IS NULL AND entity_type='user' if any transitional reader wants it.

Build order (schema-first)

  • A1 seed strava_activity; A2 create event_activities + event_activity_map_strava + models; A3 admin group editor; A4 seed event 52 groups.
  • B1 sync_entity_totals migration + index on sync_activities; B2 fan-out TallyStep; B3 metric-aware RankStep; B4 sync:rebuild; B5 shadow on 52.
  • C WL read adapter reads sync_entity_totals (group_id + entity_type) → merge with donation.
  • D go live on 52.

Organiser guide and developer documentation for the TogoActive platform.