Struct FusedAttentionPipeline
pub struct FusedAttentionPipeline {
pub pipeline: ComputePipeline,
pub layout: BindGroupLayout,
}Expand description
GPU compute pipeline for fused FlashAttention dispatch.
Fields§
§pipeline: ComputePipeline§layout: BindGroupLayoutImplementations§
§impl FusedAttentionPipeline
impl FusedAttentionPipeline
pub fn new(device: &Device) -> FusedAttentionPipeline
pub fn try_new( device: &Device, ) -> Result<FusedAttentionPipeline, TensorPipelineError>
pub fn try_new_packed(
device: &Device,
) -> Result<FusedAttentionPipeline, TensorPipelineError>
pub fn try_new_packed( device: &Device, ) -> Result<FusedAttentionPipeline, TensorPipelineError>
Four storage bindings; binding 1 contains weights followed by packed mask pages.
pub fn new_packed(device: &Device) -> FusedAttentionPipeline
pub fn dispatch<'a>( &'a self, cpass: &mut ComputePass<'a>, bind_group: &'a BindGroup, num_heads: u32, )
pub fn dispatch_batched<'a>(
&'a self,
cpass: &mut ComputePass<'a>,
bind_group: &'a BindGroup,
num_heads: u32,
batch_len: u32,
)
pub fn dispatch_batched<'a>( &'a self, cpass: &mut ComputePass<'a>, bind_group: &'a BindGroup, num_heads: u32, batch_len: u32, )
Dispatch Q+attention (proj_kind == 0) over per-token mask pages in a batch.
Auto Trait Implementations§
impl !RefUnwindSafe for FusedAttentionPipeline
impl !UnwindSafe for FusedAttentionPipeline
impl Freeze for FusedAttentionPipeline
impl Send for FusedAttentionPipeline
impl Sync for FusedAttentionPipeline
impl Unpin for FusedAttentionPipeline
impl UnsafeUnpin for FusedAttentionPipeline
Blanket Implementations§
§impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
Aggregate shares in an MPC protocol.
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more