Skip to main content

PreparedDecodePlan

Trait PreparedDecodePlan 

Source
pub trait PreparedDecodePlan {
    // Required methods
    fn backend(&self) -> PreparedBackend;
    fn state(&self) -> PreparedPlanState;
    fn counters(&self) -> ExecutionCounters;
    fn run_decode_step(
        &mut self,
        input: DecodeStepInput<'_>,
        output: &mut DecodeStepOutput,
    ) -> Result<(), DecodeStepError>;
}
Expand description

Tier-1 execution contract for an already prepared model/backend pair.

Required Methods§

Source

fn backend(&self) -> PreparedBackend

Prepared backend.

Source

fn state(&self) -> PreparedPlanState

Plan state.

Source

fn counters(&self) -> ExecutionCounters

Execution counters.

Source

fn run_decode_step( &mut self, input: DecodeStepInput<'_>, output: &mut DecodeStepOutput, ) -> Result<(), DecodeStepError>

Execute one autoregressive step.

Implementations must not allocate, compile, discover tensors, upload immutable data, or select another backend. A failure is returned to the owner, which records any explicit fallback outside this call.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§