Loss Functions¶

TorchTrade provides specialized loss functions for training RL trading agents, built on TorchRL's LossModule interface.

Available Loss Functions¶

Loss Function	Type	Use Case
DGLoss	Policy Gradient	Delight-gated updates — no importance sampling needed
GroupRelativePGLoss	Policy Gradient	One-step RL with SLTP environments
SAOLoss	Policy Gradient (LLM)	Single-rollout LLM RL with a critic baseline — no K-sample group needed
CTRLLoss	Representation Learning	Self-supervised encoder training
CTRLPPOLoss	Combined	Joint policy + representation learning

For standard multi-step RL (PPO, DQN, SAC, IQL), use TorchRL's built-in loss modules directly.

DGLoss¶

Delightful Policy Gradient — gates each update by \(\sigma(\text{delight} / \eta)\), where \(\text{delight} = \text{advantage} \times \text{surprisal}\). This suppresses rare failures and amplifies rare successes without requiring importance sampling (no old log-probs needed).

\[ \begin{aligned} \text{surprisal} &= -\log \pi(a|s) \\ \text{advantage} &= r - b \\ \text{delight} &= \text{advantage} \times \text{surprisal} \\ \text{gate} &= \sigma(\text{delight} / \eta) \\ \mathcal{L} &= -\mathbb{E}\left[\log \pi(a|s) \cdot \text{sg}(\text{gate} \cdot \text{advantage})\right] \end{aligned} \]

Key difference from PPO/GRPO: DG uses only the current policy's log-probabilities. No importance ratios, no behavior probabilities. This makes it simpler and naturally robust to stale or off-policy data.

Why DG for trading?¶

Trading rewards are heavy-tailed: most trades produce small P&L, but a few outliers (flash crashes, breakouts) dominate the gradient. Standard PG and GRPO weight updates purely by advantage magnitude, so one catastrophic trade can overwhelm training even when the policy assigned near-zero probability to that action.

The delight gate fixes this. A large loss on an unlikely action produces negative delight, pushing the gate toward 0 and shrinking its gradient contribution. Conversely, an unexpected win produces positive delight, pushing the gate toward 1 so the policy learns faster from it. The net effect: black-swan losses don't destabilize training, and rare profitable signals get amplified.

DG also drops the importance sampling machinery that GRPO needs. GRPO requires old log-probs (sample_log_prob) to compute policy ratios, which means storing behavior policy state and dealing with ratio clipping when data gets stale. DG only needs the current policy's log-probs, so it works cleanly with replay buffers, asynchronous collection, and distributed setups where the collecting policy drifts from the learner.

When to use DG vs GRPO:

Scenario	Recommendation
One-step SLTP with fresh on-policy batches	Either works; GRPO is well-tested
Sequential environments (multi-step episodes)	DG, no need to track old log-probs across steps
Replay buffer / offline data	DG, no importance sampling artifacts
Distributed collection with stale actors	DG, designed for this (see arXiv:2603.20521)
Heavy-tailed reward distributions	DG, gate suppresses outlier-driven gradient noise

Parameter	Default	Description
`actor_network`	Required	Policy network (ProbabilisticTensorDictSequential)
`eta`	1.0	Sigmoid gate temperature (lower = sharper gating)
`baseline`	`"mean"`	Baseline type: `"mean"` or `"none"`
`entropy_bonus`	True	Whether to add entropy regularization
`entropy_coeff`	0.01	Entropy regularization coefficient

from torchtrade.losses import DGLoss

loss_module = DGLoss(actor_network=actor, eta=1.0, baseline="mean")

for batch in collector:
    loss_td = loss_module(batch)
    loss = loss_td["loss_objective"] + loss_td["loss_entropy"]
    loss.backward()
    optimizer.step()

    # DG-specific diagnostics
    print(f"gate: {loss_td['gate'].item():.3f}, advantage: {loss_td['advantage'].item():.3f}")

Baseline modes:

"mean" — batch mean of rewards (default, simple and effective)
"none" — no baseline (raw rewards as advantage)

Papers:

Delightful Policy Gradient (arXiv:2603.14608) — Introduces DG and shows it enhances directional accuracy within single contexts and shifts expected gradient closer to supervised cross-entropy across multiple contexts.
Delightful Distributed Policy Gradient (arXiv:2603.20521) — Extends DG to distributed RL with stale, buggy, or mismatched actors. DG without off-policy correction outperforms importance-weighted PG with exact behavior probabilities.

Example: See examples/losses/dg_mnist.py for a comparison of CE, REINFORCE, and DG on MNIST framed as a bandit problem.

GroupRelativePGLoss¶

Group Relative Policy Optimization for one-step RL with numeric discrete-action agents (e.g. a CNN+MLP categorical policy over action_levels) — not for LLM/text-generating actors. Designed for OneStepTradingEnv where episodes are single decisions with SL/TP bracket orders.

Normalizes advantage within each batch: advantage = (reward - mean) / std, computed across dim 0 of the incoming tensordict. This is only correct group-relative normalization because dim 0 is engineered to be a genuine "K samples of the same state" axis. In examples/online_rl/grpo/utils.py's make_environment, every parallel copy of OneStepTradingEnv is constructed with the same config seed, so each builds an identical internal RNG and samples the same episode-start state; because the env is one-step (every step returns done=True), all copies reset in lockstep. So at each collected time-step, all parallel copies share the same underlying market state, differing only in the policy's sampled action. Breaking that setup — a single env (whose size-1 group axis makes the advantage NaN), differing config seeds across copies, or flattening the batch before the loss runs — silently degrades training to a zero-signal or NaN baseline with no error raised.

Not related to torchrl.objectives.llm.GRPOLoss. TorchRL's own LLM-specific GRPOLoss requires an LLMWrapperBase actor (it dispatches through LLM-only methods and reads a token-level dist.mask) and expects advantage precomputed via MCAdvantage (which itself requires a string prompt). Neither applies here — passing torchtrade's numeric actor to torchrl.objectives.llm.GRPOLoss raises RuntimeError: TensorDictSequential does not support keyword arguments other than 'tensordict_out' or in_keys.... Use GroupRelativePGLoss for numeric trading agents; only reach for torchrl.objectives.llm.GRPOLoss if training an LLM's own weights via RL.

Parameter	Default	Description
`actor_network`	Required	Policy network (ProbabilisticTensorDictSequential)
`entropy_coeff`	0.01	Entropy regularization coefficient
`epsilon_low` / `epsilon_high`	0.2	Clipping bounds for policy ratio

from torchtrade.losses import GroupRelativePGLoss

loss_module = GroupRelativePGLoss(actor_network=actor, entropy_coeff=0.01)

for batch in collector:
    loss_td = loss_module(batch)
    loss = loss_td["loss_objective"] + loss_td["loss_entropy"]
    loss.backward()
    optimizer.step()

Paper: DeepSeekMath (arXiv:2402.03300) — Section 2.2

SAOLoss¶

Single-Rollout Asynchronous Optimization for LLM trading actors. Where the LLM GRPO path builds its baseline from a group of K completions of the same bar, SAO trains on one rollout per bar and recovers the baseline from a learned critic \(V(s)\) instead. This removes the K-fold generation cost of the group baseline — the dominant expense when the actor is an LLM — at the price of a small critic.

SAOLoss is a thin subclass of TorchRL's LLM GRPOLoss that reuses the entire pipeline (assistant-token masking, per-token log-weights, aggregation, ESS) and overrides only the policy objective:

\[ \begin{aligned} r_t &= \exp\!\left(\log \pi_\theta(a_t|s) - \log \pi_{\text{rollout}}(a_t|s)\right) \\ f(r_t) &= \begin{cases} r_t & \text{if } 1-\varepsilon_l < r_t < 1+\varepsilon_h \\ 0 & \text{otherwise} \end{cases} \\ \mathcal{L} &= -\mathbb{E}_t\left[ f(r_t)\cdot \hat{A} \right], \qquad \hat{A} = R - V(s) \end{aligned} \]

Two swaps vs GRPO. (1) Clip → DIS mask (Eq. 3): in-band, \(f\) is the identity (the ratio is retained as an off-policy weight); out-of-band tokens are masked to zero, not clamped to the boundary — permitting the aggressive asymmetric "clip-higher" the paper relies on. (2) Group baseline → critic advantage: \(\hat{A} = R - V(s)\), where \(V(s)\) is supplied by the trainer per bar. There is no entropy or KL term by default, matching the paper's objective.

Run it via LLMTrainer(loss="sao") — the trainer wires the single-rollout collection, the ObservationCritic on the numeric bar state (a valid action-independent baseline; the LLM is only the policy), and the critic's own optimizer on the paper's faster-value schedule (critic_updates=2):

from torchtrade.llm.train import LLMTrainer

# defaults are the paper's math/TIR values (ε_l=0.3, ε_h=5.0, entropy off);
# override any via loss_kwargs, e.g. loss_kwargs={"epsilon_low": 0.8, "epsilon_high": 3.0} for the SWE-Bench setting.
LLMTrainer(df=df, config=config, loss="sao", num_generations=4).train()  # 4 = distinct bars/step

Parameter	Default	Description
`epsilon_low`	0.3	Lower trust-region half-width \(\varepsilon_l\) (paper's math/TIR value)
`epsilon_high`	5.0	Upper trust-region half-width \(\varepsilon_h\) — the asymmetric "clip-higher" is the point of DIS (paper: 5.0 math/TIR, 3.0 SWE-Bench)
`entropy_bonus`	False	Add an entropy bonus (paper's objective has none)
`masking_strategy`	`"rlhf"`	Score assistant/answer tokens only

Note

SAOLoss pulls the LLM/vllm stack, so it is not exported from torchtrade.losses; import it from torchtrade.losses.sao_loss or use LLMTrainer(loss="sao").

Paper: Single-Rollout Asynchronous Optimization (arXiv:2607.07508)

CTRLLoss¶

Cross-Trajectory Representation Learning for self-supervised encoder training. Trains encoders to recognize behavioral similarity across trajectories without rewards, improving zero-shot generalization.

Parameter	Default	Description
`encoder_network`	Required	Encoder that produces embeddings
`embedding_dim`	Required	Dimension of encoder output
`num_prototypes`	512	Learnable prototype vectors
`sinkhorn_iters`	3	Sinkhorn-Knopp iterations
`temperature`	0.1	Softmax temperature
`myow_coeff`	1.0	MYOW loss coefficient

from torchtrade.losses import CTRLLoss

ctrl_loss = CTRLLoss(
    encoder_network=encoder,
    embedding_dim=128,
    num_prototypes=512,
)

for batch in collector:
    loss_td = ctrl_loss(batch)
    loss_td["loss_ctrl"].backward()
    optimizer.step()

Paper: Cross-Trajectory Representation Learning (arXiv:2106.02193)

CTRLPPOLoss¶

Combines ClipPPOLoss with CTRLLoss for joint policy and encoder training. The encoder learns useful representations while the policy learns to act.

from torchtrade.losses import CTRLLoss, CTRLPPOLoss
from torchrl.objectives import ClipPPOLoss

combined_loss = CTRLPPOLoss(
    ppo_loss=ClipPPOLoss(actor, critic),
    ctrl_loss=CTRLLoss(encoder, embedding_dim=128),
    ctrl_coeff=0.5,
)

for batch in collector:
    loss_td = combined_loss(batch)
    total_loss = loss_td["loss_objective"] + loss_td["loss_critic"] + loss_td["loss_ctrl"]
    total_loss.backward()
    optimizer.step()