Core is stateless: everything it knows lives in the store. Two backends implement the same
store.Store interface and the same key layout — etcd (store: etcd, the default) and redis
(store: redis).
etcd keys are namespaced by etcd.prefix (default /eru); redis uses the same paths as plain
keys. {} marks a substituted value.
| Key | Holds |
|---|---|
/pod/info/{podname} |
Pod metadata |
/node/{nodename} |
Node metadata: endpoint, pod, labels, bypass, test |
/node/{podname}:pod/{nodename} |
Pod-to-node index; listing a pod’s nodes is a prefix scan here |
/node/{nodename}:workloads/{workloadID} |
Node-to-workload index |
/status:node/{nodename} |
Node liveness, written with a lease. Its presence is the node being up |
/workloads/{workloadID} |
Workload metadata: pod, node, name, labels, image, env, resources |
/deploy/{appname}/{entrypoint}/{nodename}/{workloadID} |
Deploy index; counting a prefix gives instances per node |
/status/{appname}/{entrypoint}/{nodename}/{workloadID} |
Workload status, written by the agent with a lease |
/processing/{appname}/{entrypoint}/{nodename}/{ident} |
In-flight deploy count for one deploy round |
/services/{address} |
One live core instance, written with a lease |
/selfmon/active |
The node-status-watcher election key, written with a lease |
/wal/{address}/{seq} |
One journal entry of the core instance registered at {address}; see Operations |
/wal-replay/{address} |
Held by the instance replaying that journal after its owner disappeared |
/resource/cpumem/{nodename} |
The built-in cpumem plugin’s own bookkeeping |
The deploy count a strategy sees is /deploy/... plus /processing/..., which is how two
concurrent deploys of the same app avoid stacking on the same node.
store/etcdv3 (Mercury) wraps the v3 client. Everything below it goes through meta.KV, a thin
layer over etcd’s KV, Lease and Watch APIs that adds create/update semantics, batch transactions,
BindStatus (put a status key leased against an entity key, failing if the entity is gone) and
ephemeral keys.
namespace.NewKV/NewWatcher/NewLease on
etcd.prefix, so no code has to prepend it.etcd.ca, etcd.cert and etcd.key are all set.etcd.auth.username / password.ServiceStatusStream, NodeStatusStream and WorkloadStatusStream are
etcd watches on a prefix. ServiceStatusStream opens its watch before its initial Get, so
nothing is missed between the snapshot and the stream; the node and workload streams carry no
snapshot and report changes only.store/redis (Rediaron) implements the same interface with go-redis. Batch writes become
transactional pipelines, while multi-key reads and deletes are single MGET / DEL commands;
Watch-style streams become keyspace notifications on
__keyspace@{db}__:{pattern}, so the redis server must be configured to emit them — core cannot
see status changes otherwise. Ephemeral keys are ordinary keys with a TTL, refreshed on a timer.
One behaviour differs from etcd and is marked as such in the code: Update checks key
existence before writing rather than doing it in one transaction. Create and CreateAndDecr
run as one Lua script, so a conflicting key writes nothing and the processing counter must exist,
as on etcd. A status heartbeat that carries an unchanged value only refreshes the key’s TTL, and
TTL refreshes are not watch events, so streams see status changes rather than heartbeats. A status
without a TTL whose entity core has not recorded is kept for an hour, as on etcd. Tests
run against an embedded miniredis.
The built-in cpumem resource plugin keeps its node records in the same store — see
Resource plugins. A store: redis deployment that predates this kept those
records in etcd under <etcd.prefix>/resource/cpumem/: copy them into redis under the same keys
before starting the new cores, and move every core together, since an old core would keep writing
the etcd copy.
store.CreateLock(key, ttl) returns a DistributedLock with Lock and Unlock, and
Lock returns a context that is cancelled if the lock is lost. TTL is lock_timeout.
concurrency.Mutex under etcd.lock_prefix (default __lock__/eru)redislock under redis.lock_prefix (default /lock)Core takes three kinds of lock, always in sorted order and always released in reverse:
| Key | Taken by |
|---|---|
plock_{podname} |
RemovePod, and a deploy whose plan spans more than one node |
clock_{workloadID} |
anything that touches one workload |
cnode_op_{podname}_{nodename} |
node operations that must not interleave, notably remap |
A deploy that plans on more than one node holds the pod lock for the whole allocation phase; a single-candidate deploy holds only that node’s operation lock.
--embedded-storage starts a single-member etcd inside the process, storing its data under
$TMPDIR/eru-core-etcd, and hands core an in-process client namespaced by etcd.prefix. No
listener is published, so nothing outside the process can reach it — including a second core
instance, the node status watcher of another host, or etcdctl.
It is meant for development and tests. For anything else, point etcd.machines at a real cluster.
Three things in core are leases that must be refreshed or they vanish, and each vanishing is meaningful:
| Key | Refreshed by | If it expires |
|---|---|---|
/services/{addr} |
this core instance, every grpc.service_heartbeat_interval |
clients stop routing to this instance |
/status:node/{nodename} |
the node’s agent, via SetNodeStatus |
the node is considered down and its workloads are marked down |
/selfmon/active |
the elected watcher, every ha_keepalive_interval |
another instance takes over the watcher role |