Skip to content

rewrite of sei-load generator and sender.#56

Open
pompon0 wants to merge 39 commits into
mainfrom
gprusak-nonce-reset
Open

rewrite of sei-load generator and sender.#56
pompon0 wants to merge 39 commits into
mainfrom
gprusak-nonce-reset

Conversation

@pompon0

@pompon0 pompon0 commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Primary reason for the rewrite is support for non-happy paths for autobahn loadtesting - rejected transactions/not successfully executed/etc. The idea is that if sending a tx fails, we need to check the account nonce before sending the next transaction. Note that this affects only accounts from the long-lived pools - accounts generated for the sake of sending just 1 transaction (aka "new accounts") do not benefit from this extra logic.

The nonces, as well as the txs waiting to be sent are managed by TxsQueue, which ensures that:

  • inserted (generated) transactions have expected nonces
  • transactions per account are sent sequentially (otherwise they will be rejected by autobahn)

TxsQueue is managed by ShardedSender.

Additionally

  • removed unnecessary abstractions in both sender and generator logic
  • made Generator.Run accept the sender which exposes Nonce(), Flush() and Send() calls
  • added Generator <-> Sender integration tests
  • removed open/closed loop logic (might need to be reintroduced later if needed)
  • removed utils/rng in favor of just using a single Rand synchronously (we don't have other use cases). If concurrent Rand usage is needed, a SplitRng method was introduced (slower but robust alternative to SplitMix).

@pompon0 pompon0 requested a review from bdchatham June 26, 2026 16:14
@pompon0 pompon0 marked this pull request as ready for review June 29, 2026 16:59
@cursor

cursor Bot commented Jun 29, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Large refactor of transaction generation, nonce sequencing, and send failure handling on the critical load-test path; open-loop arrival scheduling was removed, which may change benchmark behavior.

Overview
Rewrites the generator and sender so load tests can recover when sends fail (e.g. Autobahn rejects txs). Tracked pool accounts get nonce sync from chain on failure; untracked one-shot accounts skip that.

Sender: Replaces per-shard worker queues with a TxsQueue (bounded by MaxInFlight) that enforces per-address nonce order and sequential sends. ShardedSender pushes into the queue, pops ready txs, and on RPC failure for tracked senders calls Reset(addr, chain nonce) after eth_getTransactionCount. ethClient dials all endpoints and shards by sender address; sends are synchronous from the sender loop (no per-endpoint request channels).

Generator: Drops the dispatcher, weighted generator, utils/rng sub-streams, and open/closed-loop scheduling. NewGenerator(rng, cfg) builds scenarios; Run(ctx, rng, txSender) loops scenarios, asks txSender.Nonce(sender), and Send. Prewarm is Generator.Prewarm. Gas/distribution pickers take an explicit *rand.Rand at draw time.

main: Wires generator directly to the sender background task; run summary no longer reports open-loop drops.

Config: Settings.Validate now only requires MaxInFlight > 0 (arrival-model checks removed).

Adds generator↔sender integration tests (nonce rewinds, mixed tracked/untracked pools).

Reviewed by Cursor Bugbot for commit 628acaf. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread sender/sharded_sender.go
Comment thread sender/txs_queue.go
Comment thread main.go
Comment thread sender/txs_queue.go
if nonce != state.nextNonce {
// It is expected in case of send failure.
log.Printf("bad nonce for %v: got %v, want %v", addr, nonce, state.nextNonce)
return nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped txs look successful

Medium Severity

When TxsQueue.Push rejects a transaction for a nonce mismatch, it logs and returns nil. Generator.Run treats that as a successful send and immediately builds another transaction, which can spin at full speed without making progress until context cancel if the queue and generator stay desynchronized.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 91db965. Configure here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WAI

Comment thread sender/sharded_sender.go
}, nil
cfg: cfg,
queue: NewTxsQueue(cfg.Settings.MaxInFlight),
limiter: limiter,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Buffer size no longer bounds queue

Medium Severity

NewShardedSender sizes TxsQueue only from Settings.MaxInFlight. Settings.BufferSize and LoadConfig.TotalQueueSize() are no longer used, so existing configs that tune bufferSize/--buffer-size do not limit queue depth or memory as before.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 34ba3fe. Configure here.

Comment thread sender/metrics.go
attribute.String("chain_id", stats.ChainID),
))
}
}*/

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worker queue metric disabled

Low Severity

The worker_queue_length observable gauge callback no longer records observations: the ShardStats loop is commented out and only assigns _ = ss. Scrapers always see an empty series after the sender rewrite removed per-shard queues.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 34ba3fe. Configure here.

Comment thread generator/generator.go Outdated
Comment thread generator/generator.go
Comment thread generator/generator.go
}
ltx := &types.LoadTx{EthTx: tx, IntendedSendTime: time.Now(), Scenario: scenario}
if err := txSender.Send(ctx, ltx); err != nil {
return err

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale nonce before async send

Medium Severity

Run reads txSender.Nonce, builds and signs the tx, then calls Send without holding sender/queue state. A concurrent send failure and Reset can advance the queue nonce before Push, producing signed txs the queue rejects.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b83e882. Configure here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rejection is silent and it is expected, since we send the txs asynchronously.

Comment thread main.go
Comment thread main.go
// Start the sender (starts all workers)
s.SpawnBgNamed("sender", func() error { return sharedSender.Run(ctx) })
log.Printf("✅ Connected to %d endpoints", len(cfg.Endpoints))
snd = sharedSender

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Workers setting no longer applied

Medium Severity

The send path now runs a single ShardedSender background task and no longer uses Settings.TasksPerEndpoint (workers) or LoadConfig.GetNumShards, even though those settings remain in config, CLI, and tests. Raising --workers does not increase RPC parallelism as documented.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit a28e7c0. Configure here.

Comment thread generator/generator.go
instance.Deployed = true

if address.Cmp(common.Address{}) != 0 {
address := instance.Scenario.Deploy(g.config, deployer, uint64(i))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy nonces skip unused slots

High Severity

Sequential scenario deployment passes the scenario list index as the deployer nonce. Instances that do not broadcast a deployment transaction still advance that index, so the next contract deployment can be signed with a nonce above the account’s on-chain nonce and be rejected.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit cce23ae. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 7 total unresolved issues (including 6 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 628acaf. Configure here.

Comment thread sender/sharded_sender.go
inclusion.Register(tx)
}
return ctx.Err()
return nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Workers setting no longer applied

Medium Severity

Settings.TasksPerEndpoint / --workers no longer scale send parallelism: ShardedSender.Run uses one rate-limited pop/send loop and ethClient dials endpoints without per-worker tasks, while main.go still logs total tasks from workers × endpoints.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 628acaf. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants