Initial YAML config with parity to legacy .conf format by caitlinross · Pull Request #259 · codes-org/codes

caitlinross · 2026-07-09T22:13:55Z

This PR adds a YAML front-end for CODES configuration. A .yaml/.yml config
compiles into exactly the same in-memory configuration (ConfigVTable) that
the legacy .conf parser produces, so models and the mapping layer are
completely unaware of which format was used — no model code changes, and
.conf files continue to work unchanged. The format you pick is dispatched on
file extension.

Why

The legacy .conf format is a custom grammar with a flex/bison parser: opaque
to newcomers, unfriendly to tooling, and easy to get subtly wrong. YAML gives
us a standard, readable format that editors, linters, and scripts already
understand, and it lets the config say what you mean (a dragonfly of a given
shape) instead of how the simulator wants it spelled (LP counts per repetition).

The format

Every config declares schema_version: 1 and expresses its topology one of
three ways, from most to least abstract:

Parametric fabric blocks for known network families — dragonfly
(including dally, plus, and custom variants), fat-tree, slim fly, torus, and
express mesh. You state the fabric parameters; the compiler derives the LP
groups and PARAMS the models expect.
Flat networks as just a component + node count (simplenet, simplep2p,
loggp).
Explicit format: groups — a general escape hatch that can express
anything the legacy format can, including custom group names, extra LP
types, and annotations.

Beyond topology: a pass-through sections: block carries model-specific
sections verbatim, annotated keys (key@cluster) are supported throughout,
and include: lets configs be split and reused across files (included files
are read with collective MPI I/O, same as the main config).

Internally the compiler is a pure C++ core with no ROSS or MPI dependency —
errors are exceptions translated to tw_error at a single C boundary — which
is what makes it unit-testable in isolation.

Parity

Pretty much everything expressible in .conf today is expressible in YAML,
and the tree proves it: every legacy config file now has a YAML twin sitting
next to it, and each twin was mechanically verified to compile to the same
parsed configuration as its .conf. The one deliberate exception is
tests/workload/codes-workload-test.conf, a PARAMS-only file with no
LPGROUPS block — a degenerate shape the YAML format intentionally does not
express (documented in the tests README).

Testing

Unit tests cover the compiler core directly (GoogleTest, no simulator, no
MPI): fabric derivations, validation and error paths, includes, and
malformed input.
CI runs conf-vs-yaml equivalence tests that execute a model under both
formats and diff the per-LP lp-io output byte for byte wherever the
model's output is reproducible, falling back to event-count markers where
it isn't (this required un-hardcoding lp-io paths in a few test binaries,
which strengthened the pre-existing equivalence tests too).
The tutorial ping-pong examples — the usual starting point for new
developers — are covered in both formats.

doc/dev/yaml-config.md is a complete standalone reference for the format,
and the new public API carries doxygen comments. Creating the yaml twins also
flushed out some long-standing config rot (hardcoded paths to
contributors' machines, configs inconsistent with their committed topology
files), which is cleaned up here.

Introduce a config compiler that reads the friendly YAML/JSON topology and component format and lowers it to the same ConfigVTable the .conf text parser produces, so codes_mapping and the models consume it unchanged through the configuration_get_* accessors. The front-end is three layers so the parsing/validation logic is unit-testable and never leaks ROSS: - config_compiler: a pure C++ core, compile(text) -> compiled_config, an ordered plain-data image of the LPGROUPS/PARAMS tree. It does all parsing and validation with no dependency on ROSS, MPI, or abort: on any invalid input it throws config_error, and ryml's own parse errors are routed to throw as well. - config_emitter: a dumb pass that turns the compiled_config into a ConfigVTable; no validation, its coverage comes from the equivalence tests. - yaml_configfile: a thin extern "C" shim, the only place ROSS lives -- it runs compile() + emit() and translates a config_error into tw_error. Real-run behavior (abort + diagnostic) is unchanged from the .conf path; "abort" is now a boundary policy rather than being baked into every validation site. Every rank reads identical bytes and runs the identical deterministic compile, so malformed input still aborts everywhere at once. configuration_load dispatches by file extension: a .yaml/.yml/.json path goes through the front-end, everything else through the existing text parser. The global config stays a ConfigVTable* either way, so no call site or model changes. ryml is vendored and always built, so there is no feature guard. Validation the throwing core enforces now: schema_version is required, integer, and any value this build doesn't know is a hard error (a newer config can't be read safely); unknown top-level keys, unknown topology keys, and nested blocks a component/fabric doesn't consume are errors rather than silent drops. This first cut compiles the parametric-fabric source for the regular dragonfly: the group/repetition and per-router LP counts are derived from the fabric shape (the same num_routers-driven math the model does internally), per-link-class bandwidth/vc_size and routing map to the model's PARAMS, and modelnet_order is derived from the fabric model. Values are carried through as their raw scalar text so a compiled config is byte-comparable to the .conf it replaces. Verified against modelnet-synthetic-dragonfly.conf: model-net-synthetic produces byte-identical per-LP lp-io output from the .yaml twin.

Point the synthetic-dragonfly lp-io equivalence test at the YAML config instead of running the .conf twice, so it now asserts the YAML front-end and the legacy text parser drive the model to byte-identical per-LP output.

Extend the YAML front-end to the flat all-to-all network models, simplenet and simplep2p. A flat topology is expressed as one component -- the workload model (nw-lp) bundled with the NIC model it runs over (network: simplenet|simplep2p) plus that model's params -- and a scalar node count: topology: format: flat component: compute_node nodes: 16 The compiler lays out one repetition per node (the workload LP and its NIC LP), derives modelnet_order from the network model, and passes the component params (message_size, packet_size, and simplep2p's matrix-file references) straight through to PARAMS. simplep2p's link table stays referenced by path to its existing matrix files; the friendly form supplies only the node count. This is deliberately not a node/edge graph: the Cytoscape element form (per-node data, per-edge bandwidth/latency) is a new representation with no consuming model yet, so it waits for the WAN model. Adds a marker-based config-equivalence helper (the flat-model test binaries write lp-io to a fixed path, so committed-event counts are compared instead of lp-io dirs) and registers both twins. Verified: modelnet-simplep2p-test and modelnet-test produce identical event counts from the .yaml and the .conf.

The fabric model descriptor already varies the per-repetition router/switch LP count and whether the router is a distinct model-net method, so add the two families as registry entries plus their shape->counts derivations: - fattree (internally generated): one repetition per edge switch, switch_radix/2 terminals each, one switch LP per level, and only the terminal in modelnet_order. - dragonfly-dally (file enumerated): repetitions = num_groups * num_routers with the shape counts passed through as genuine inputs, and connections.{intra,inter} mapped to the model's connection-file keys and referenced by path. Verified: model-net-synthetic-dragonfly-all produces byte-identical per-LP lp-io from the dally YAML and its .conf; model-net-synthetic-fattree produces matching committed-event counts from the fattree YAML and its .conf (fattree's lp-io switch stats are not reproducible run to run, so event counts are the signal). The dally test configures both configs with an absolute path to the shared connection files so the runner's per-run subdirs resolve them.

Add YAML config support for five more model-net networks, each with a .conf-equivalence test proving the compiled config drives the model to identical results: - loggp (flat/p2p; one-line registry entry) - torus (internally-generated; no separate router LP) - express-mesh (internally-generated) - slimfly (internally-generated; list-valued generator sets) - dragonfly-plus (file-enumerated; per-LP lp-io equivalence) Two small additions to the compiler core support these: - null router LP: torus folds routing into the terminal node, so the LPGROUPS router line is skipped when a fabric has no router method. - list-valued fabric params: slimfly's generator_set_X / _X_prime are written as YAML sequences and emitted as multi-value PARAMS (the emitter already handled multi-value keys). This brings YAML coverage to 10 of 11 network models. dragonfly-custom remains.

Complete model-net YAML parity: dragonfly-custom is the last topology to land. Like dragonfly-dally/plus it is file-enumerated, so the shape counts (num_router_rows x num_router_cols routers per group, num_groups groups, num_cns_per_router terminals per router) are genuine inputs the compiler derives the LP layout from and that must match the binary connection files. The golden fixture is the 8-group Theta-style custom dragonfly from scripts/dragonfly-custom/example (6x16 router mesh -> 768 routers). The older in-tree custom configs were unusable as a baseline (dead hard-coded paths, and a transposed rows/cols shape that segfaulted the model against the available connection files); the equivalence test uses the corrected shape and passes at per-LP lp-io. Every model-net topology can now be configured from YAML.

…ames Add a top-level `sections:` block to the YAML config front-end for config a model reads directly by name (DIRECTOR, NETWORK_SURROGATE, resource, storage, ...). Each entry is emitted verbatim as a top-level section: scalars become single-value keys, lists multi-value keys, nested maps subsections. This restores the .conf ergonomics for such config -- a new feature adds its section with no compiler change -- while topology stays friendly and strictly validated. A section may optionally register an *open* schema (section_schemas[]): required keys are enforced while any other key still passes through, which suits a feature whose config is still in flux. `resource` is registered (its `available` key is mandatory -- resource-lp.c aborts without it), so a missing key is now a clear compile-time error. LPGROUPS/PARAMS are reserved (emitted from the topology). Section names are case-insensitive: the all-caps convention (PARAMS, DIRECTOR, ...) is a historical carryover, so mcs_findsubsection matches case-insensitively. Keys stay case-sensitive -- models read them by exact name. Verified end to end: a model reads `params` written lowercase. Also lands the first real unit test of the ROSS-free compiler core (tests/codes-config-compiler-test.cxx), asserting on compiled_config for the pass-through, open-schema, reserved-name, and case-preservation behavior -- with no codes/ROSS/MPI link, as the layered design intends. Documents the feature and a two-tier "adding a config section" process in doc/dev/yaml-config.md.

Add an explicit LP-layout form to the YAML front-end for configs that are not a single friendly network -- storage clusters, resource/buffer tests, multi- partition and mapping layouts -- where the compiler derives nothing and the user lays out the groups directly: topology: format: groups params: { message_size: 512 } groups: TRITON_GRP: repetitions: 1 lps: { nw-lp: 1, lsm: 1 } Each group's `repetitions` and `lps` (lp-type -> count) transcribe a .conf LPGROUPS directly, validated (positive counts, known keys). An LP-type key may carry an annotation as `type@annotation` -- the spelling codes_mapping splits on '@' -- so the same type can appear more than once per group. `params` becomes PARAMS (scalars/lists/nested pass through). Composes with the `sections:` pass-through for a model that also reads its own section. Verified end to end: lsm-test (TRITON_GRP + lsm section) and resource-test (BUF + resource section) run byte-identically from their .conf and their new explicit-groups .yaml twins. The config-equivalence harness gains a CONFIG_FLAG option so binaries that take the config as a ROSS option (--conf=, --codes-config=) rather than a trailing positional can be equivalence-tested. Unit tests (codes-config-compiler-test) cover the groups/params structure, annotations, composition with sections, and the validation errors. Documented in doc/dev/yaml-config.md.

…iles A YAML config can now pull in other files with a top-level `include:` (a filename or a list), so a shared fragment -- a network, a set of components, a model section -- can be factored out and reused: define it once, vary the rest (e.g. the same section across different layouts). Included files are the base; the including file overrides them. `components` and `sections` merge by name (local wins on a clash); `topology`/`schema_version` are singular (local replaces). Multiple includes apply in list order, local last. One level deep for now -- an included file may not itself `include:`. The compiler core stays I/O-free: include resolution is a loader-boundary concern. compile() now takes the main document plus already-read base documents and merges them; a new parse_includes() extracts a document's include list. The extern "C" shim (yaml_configfile_load) gains the config path, reads the referenced files relative to its directory, and hands their contents to the core. configuration_load threads the path through. Included files must be readable by every rank (documented). Tests: unit tests for the multi-document merge (component/section merge and override, topology-from-base, nested-include rejection, parse_includes) and an end-to-end equivalence test -- lsm-test-include.yaml includes a shared lsm-workload fragment and runs byte-identical to lsm-test.conf. Documented in doc/dev/yaml-config.md.

Close the validation gaps found in review of the config compiler core: - shape values parse strictly: a non-integer (num_groups: abc) or trailing garbage (9x) is a diagnostic naming the key, not a silent 0/9; a backstop after every fabric derivation rejects degenerate layouts (repetitions/terminals <= 0) before they reach codes_mapping - dim_length: reject empty segments, trailing commas, and space-separated lists; guard product overflow; cross-check the entry count against n_dims (torus, express-mesh) - reject an empty component model: (was an empty LP-type key in LPGROUPS), unexpected keys under hosts:, the component type: key (reserved; would otherwise leak into PARAMS), network: on a parametric host component, and an odd fattree switch_radix - replace the deprecated 4-arg ryml::Callbacks constructor with the default constructor + setters (silences the only warning in new code) - catch std::exception at the extern "C" boundary so bad_alloc & co. reach tw_error with a message instead of std::terminate - match the config extension case-insensitively (.YAML, .Yml, .JSON) Adds 13 unit tests (first parametric-path coverage plus each new error path) and the matching yaml-config.md updates.

The YAML shim used to read include: fragments itself with std::ifstream -- one POSIX open per rank per include (a metadata storm on parallel filesystems at scale) and a hang risk: a file readable on some nodes but not others makes those ranks tw_error while the rest proceed into the next collective. Includes are now read in configuration_load with the same collective MPI_File_read_all pattern as the main config -- one collective read per file across the job. The shim gains yaml_configfile_list_includes() (resolves include paths from the main document's bytes, no file I/O) plus a matching free helper, and yaml_configfile_load() now consumes the include bytes directly, making it the pure bytes -> ConfigVTable function its header always described. A missing or unreadable include aborts the job with a diagnostic naming the resolved path.

The simplep2p / simplenet / loggp / torus / express-mesh / slimfly and both lsm config-equivalence tests only compared a marker line ("Net Events Processed") -- not because their models cannot be diffed per-LP, but because their test binaries wrote lp-io to a fixed, non-per-run path, so two runs' output could not coexist. Give modelnet-test, modelnet-simplep2p-test and lsm-test the same configurable lp-io options already used by model-net-synthetic (--lp-io-dir / --lp-io-use-suffix; no output when the flag is unset), then upgrade those eight registrations to codes_add_lpio_equivalence_test, which diffs the per-LP lp-io output (send/recv counts, timings, torus link stats, ...) on top of the marker -- a strictly stronger check. lsm-test takes its config via --conf=, so add a CONFIG_FLAG option to codes_add_lpio_equivalence_test mirroring the one already on codes_add_config_equivalence_test. Nothing consumed these binaries' previous fixed-path lp-io output (checked tests/*.sh, tests/expected/ and CI), so emitting no lp-io when the flag is unset is safe; the plain run-tests for the touched binaries still pass. Left marker-based, with corrected comments: - resource-test (buffer_test) emits no lp-io at all. - fattree: its network switch/msg stats ARE reproducible run to run, but model-net-synthetic-fattree also dumps a PARAMS_LOG sim_log.txt into the lp-io dir with wall-clock-derived ROSS columns (running_time, event_rate, ...) that vary every run, and the binary hardcodes its lp-io path, so a whole-dir lp-io diff would spuriously fail. The old comment blaming the switch stats was inaccurate and is fixed. modelnet-p2p-bw likewise hardcodes lp_io_prepare but only backs a single- run smoke test, so no existing comparison would benefit; left unchanged.

Add yaml configs to ping-pong tutorial and tests proving the YAML front-end drives the model identically to the legacy .conf. To cover the surrogate, teach codes_add_config_equivalence_test the SETUP and REQUIRE options codes_add_equivalence_test already has: with SETUP, CONFIG_A/B are the bare names a per-run setup script generates, so one script generating both the .conf and its .yaml twin lets the two formats be compared on the run- time-assembled config; REQUIRE asserts the surrogate actually switched. The new surrogate-config-equivalence-setup.sh generates both from their templates with the same env, so they differ only in format.

Every LPGROUPS-format .conf (and .conf.in template) now has a YAML twin stored beside it with the same basename, so each config is usable and readable in either format regardless of whether a test runs it. Fabric form is used where a supported family's derivation reproduces the LPGROUPS/PARAMS exactly; the explicit groups form transcribes the rest (extra LP types like dir-nw-lp, custom group names, per-cluster annotations, and shapes the parametric families don't derive). Each twin was verified mechanically: the YAML compiled through the front-end and the .conf parsed by the legacy parser produce structurally identical configurations (sections matched case-insensitively, group order preserved, key values compared exactly; templates compared with identical placeholder substitutions on both sides). The one exclusion is tests/workload/codes-workload-test.conf: it is PARAMS-only (no LPGROUPS), and the YAML front-end requires a topology block, so it has no expressible twin.

also removes the placeholder unit smoke test that isn't a real codes test.

…rrors Cover the three untested validation areas of the YAML config compiler core: - schema_version: missing, unsupported (2), and non-integer values are rejected on the main document, and an included base document that restates a disagreeing schema_version is rejected too - flat-topology validation: every error path now has the coverage its parametric mirror image already had -- an unexpected topology key (the documented edges/graph rejection), non-integer / zero / negative / missing nodes, missing component, a reference to an undefined component, an empty component model:, a missing network:, and an unknown network model - malformed / non-map input: a YAML syntax error throws config_error through the installed ryml parse callback (never print-and-abort, ryml's default), and a sequence or scalar top-level document is rejected with the top-level-mapping diagnostic

workloads.conf and workloads-synthetic.conf were near-identical example workload lists (line 1 "216 synthetic" identical; line 2 a placeholder dumpi trace path). workloads.conf still carried an absolute trace path on a specific machine, while the synthetic copy already used a /path/to/ placeholder.

codecov · 2026-07-10T04:21:49Z

Codecov Report

❌ Patch coverage is 78.41270% with 136 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/modelconfig/config_compiler.cxx	76.75%	119 Missing ⚠️
src/modelconfig/yaml_configfile.cxx	81.25%	9 Missing ⚠️
src/modelconfig/configuration.c	82.97%	8 Missing ⚠️

📢 Thoughts on this report? Let us know!

caitlinross added 21 commits July 9, 2026 23:18

tests: check dragonfly YAML against its .conf twin

b92bcab

Point the synthetic-dragonfly lp-io equivalence test at the YAML config instead of running the .conf twice, so it now asserts the YAML front-end and the legacy text parser drive the model to byte-identical per-LP output.

docs: document running with a YAML config

678f6fb

docs: make yaml-config.md a complete standalone reference

7298e47

docs: doxygen-comment the YAML front-end C boundary and IR fields

242f33e

docs: add a README for the tests directory

7a41457

also removes the placeholder unit smoke test that isn't a real codes test.

tests: rename modelnet-test.conf to modelnet-test-simplenet.conf

85f887e

caitlinross force-pushed the initial-yaml-config branch from 1413197 to 1cddeec Compare July 10, 2026 04:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Initial YAML config with parity to legacy .conf format#259

Initial YAML config with parity to legacy .conf format#259
caitlinross wants to merge 21 commits into
codes-org:masterfrom
caitlinross:initial-yaml-config

caitlinross commented Jul 9, 2026

Uh oh!

codecov Bot commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

caitlinross commented Jul 9, 2026

Why

The format

Parity

Testing

Uh oh!

codecov Bot commented Jul 10, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant