Struct CudaMegaPassExecutor
pub struct CudaMegaPassExecutor {
pub last_execution: Option<MegaPassExecutionPath>,
pub graph_instantiated: bool,
pub graph_key: u64,
pub node_count: u64,
pub h2d_bytes_per_token: u64,
pub resident_weights: HashSet<u64>,
pub k_cache: Vec<f32>,
pub v_cache: Vec<f32>,
}Expand description
Scalar transformer reference executor with a resident KV cache.
Fields§
§last_execution: Option<MegaPassExecutionPath>§graph_instantiated: bool§graph_key: u64§node_count: u64§h2d_bytes_per_token: u64§resident_weights: HashSet<u64>§k_cache: Vec<f32>§v_cache: Vec<f32>Implementations§
§impl CudaMegaPassExecutor
impl CudaMegaPassExecutor
pub fn new() -> CudaMegaPassExecutor
pub fn cuda_available() -> bool
pub fn register_resident_weight(&mut self, key: u64)
pub fn prepare_step( &self, desc: &MegaPassStepDescriptor, plan: &impl MegaPassPlanView, ) -> Result<MegaPassParameterPacks, MegaPassError>
pub fn execute_mega_pass(
&mut self,
desc: &MegaPassStepDescriptor,
plan: &impl MegaPassPlanView,
hidden: &mut [f32],
output_norm: Option<&[f32]>,
lm_head_raw: Option<&[u8]>,
) -> Result<u32, MegaPassError>
pub fn execute_mega_pass( &mut self, desc: &MegaPassStepDescriptor, plan: &impl MegaPassPlanView, hidden: &mut [f32], output_norm: Option<&[f32]>, lm_head_raw: Option<&[u8]>, ) -> Result<u32, MegaPassError>
Execute the full multi-layer transformer mega-pass (RMSNorm -> QKV -> RoPE -> KV write -> online-softmax GQA SDPA -> O proj -> SwiGLU FFN -> Output RMSNorm -> LM Head -> ArgMax) and return the selected token ID.
Trait Implementations§
§impl Debug for CudaMegaPassExecutor
impl Debug for CudaMegaPassExecutor
§impl Default for CudaMegaPassExecutor
impl Default for CudaMegaPassExecutor
§fn default() -> CudaMegaPassExecutor
fn default() -> CudaMegaPassExecutor
Returns the “default value” for a type. Read more
Auto Trait Implementations§
impl Freeze for CudaMegaPassExecutor
impl RefUnwindSafe for CudaMegaPassExecutor
impl Send for CudaMegaPassExecutor
impl Sync for CudaMegaPassExecutor
impl Unpin for CudaMegaPassExecutor
impl UnsafeUnpin for CudaMegaPassExecutor
impl UnwindSafe for CudaMegaPassExecutor
Blanket Implementations§
§impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
Aggregate shares in an MPC protocol.
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more§impl<T> NoneValue for Twhere
T: Default,
impl<T> NoneValue for Twhere
T: Default,
type NoneType = T
§fn null_value() -> T
fn null_value() -> T
The none-equivalent value.