TanStack
Media

Motion Generation

You need an attack clip for a character, and keyframing it by hand takes hours. generateMotion() turns a prompt into a skeletal animation file. You download the FBX and import it into your engine.

Warning

Motion generation is experimental. The API can change. The provider bills for each generation.

Generate an FBX clip

This script makes an orc axe attack and saves the FBX next to the script. It takes about 5 minutes. The clip itself takes from seconds to a few minutes.

  1. Install the packages:
shell
npm i @tanstack/ai @tanstack/ai-fal
  1. Put your fal key in .env. Get one at fal.ai/dashboard/keys.
shell
FAL_KEY=your-fal-key
  1. Add this script. It submits the job, checks it every 2 seconds, then downloads each file:
ts
import { writeFile } from 'node:fs/promises'
import { generateMotion, getMotionJobStatus } from '@tanstack/ai'
import { falMotion } from '@tanstack/ai-fal'

const adapter = falMotion('fal-ai/hunyuan-motion')

const { jobId } = await generateMotion({
  adapter,
  threadId: 'orc-attacks',
  prompt:
    'A heavy orc warrior raises a two-handed axe overhead and slams it down',
  duration: 3,
})

let status = await getMotionJobStatus({ adapter, jobId })
while (status.status === 'pending' || status.status === 'processing') {
  await new Promise((resolve) => setTimeout(resolve, 2000))
  status = await getMotionJobStatus({ adapter, jobId })
}

if (!status.motion) {
  throw new Error(status.error ?? 'Motion generation failed')
}

for (const asset of status.motion.assets) {
  const response = await fetch(asset.url)
  if (!response.ok) throw new Error(`Download failed: ${asset.url}`)
  const fileName = asset.fileName ?? `orc-attack.${asset.format}`
  await writeFile(fileName, Buffer.from(await response.arrayBuffer()))
  console.log(`Saved ${fileName}`)
}
  1. Run the script. Then import the FBX into your engine.

You now have an FBX attack clip on disk, ready for your engine.

Import the clip into your engine

The FBX holds a humanoid skeleton and its animation. Clip length and frame rate are inside the file.

  • Retarget the clip. The fal docs do not name the skeleton. The files hold an SMPL-H skeleton (52 bones, root bone Pelvis) at 30 fps, with root motion in the Pelvis bone. Retarget the clip to your rig in your engine, for example with the Humanoid rig type in Unity or the IK Retargeter in Unreal.
  • You cannot send your own rig. fal takes no character or skeleton input.
  • The clip has no weapon. Your engine attaches the axe to the hand bone. The prompt can describe the hands, for example "a two-handed grip".
  • Keep the files. fal file URLs expire after the media expiry time in your fal account. Download each file and store it with your game assets.

Use your own character

A fal clip is on fal's skeleton, so you retarget it in your engine. Uthana can make the clip on your own character:

  1. Upload a humanoid FBX or GLB with createUthanaCharacter() from @tanstack/ai-uthana.
  2. Pass the character id as modelOptions.character_id to uthanaMotion().

See Uthana: use your own character.

What you get back

The result has only what the provider returns. fal returns the files, the seed, and the billed units.

MotionGenerationResult:

FieldWhat it is
jobIdThe provider job id
modelThe model that made the clip
assetsEvery file the provider returned
seedThe seed the provider used. Send it again in modelOptions to repeat a take.
usageBilling usage, when the adapter can read it

Each MotionAsset:

FieldWhat it is
urlDownload URL. It can expire.
format'fbx', 'bvh', 'glb', 'gltf', 'json', or the name the provider uses
contentType, fileName, fileSizeSet when the provider reports them
expiresAtWhen url stops working, if the provider reports it

fal does not return clip length, frame rate, skeleton names, root motion, or contact timing. If your game needs an impact time, you author it.

Build an ability from two clips

A "Leap Slam" needs two clips: the orc attack, and the enemy reaction. Make each one with its own generateMotion() call. Then link them in a manifest that your game reads:

json
{
  "ability": "orc-leap-slam",
  "attacker": { "clip": "orc-leap-slam.fbx", "seed": 1234 },
  "reactions": [{ "clip": "enemy-knockdown.fbx", "trigger": "impact" }],
  "events": [{ "name": "impact", "timeSeconds": 1.6, "source": "authored" }],
  "effects": ["shockwave", "dust", "ground-cracks", "flying-rocks"]
}

A designer sets timeSeconds after a preview of the attack clip. The provider does not return it.

Your engine does the rest:

  • It retargets both clips to your rigs and attaches the axe.
  • It lines up the two-handed grip, blends the clips, and places the characters.
  • When the impact event fires, it applies damage and plays the reaction.
  • It spawns the shockwave, dust, cracks, and rocks, and plays the audio and camera shake.

The Generate Motion tutorial builds this page in React, with the manifest.

Show the job in a browser

Pass stream: true. The server then submits the job, checks it, and streams each status to the browser.

Server route at /api/generate/motion:

ts
import {
  generateMotion,
  generationParamsFromRequest,
  toServerSentEventsResponse,
} from '@tanstack/ai'
import { falMotion } from '@tanstack/ai-fal'

export async function POST(request: Request) {
  const { input, threadId, runId } = await generationParamsFromRequest(
    'motion',
    request,
  )
  if (typeof input.prompt !== 'string') {
    return new Response('Send a text prompt.', { status: 400 })
  }

  const stream = generateMotion({
    adapter: falMotion('fal-ai/hunyuan-motion'),
    prompt: input.prompt,
    duration: typeof input.duration === 'number' ? input.duration : undefined,
    stream: true,
    threadId,
    runId,
  })
  return toServerSentEventsResponse(stream)
}

React component. There is no motion hook, so use the generic useGeneration hook with MotionGenerationResult:

tsx
import { useState } from 'react'
import { fetchServerSentEvents, useGeneration } from '@tanstack/ai-react'
import type { MotionGenerationResult } from '@tanstack/ai'

type MotionInput = { prompt: string; duration: number }

export function MotionClip() {
  const [jobStatus, setJobStatus] = useState('Not started')
  const { generate, result, isLoading, error } = useGeneration<
    MotionInput,
    MotionGenerationResult
  >({
    connection: fetchServerSentEvents('/api/generate/motion'),
    threadId: 'orc-leap-slam',
    onChunk: (chunk) => {
      if (chunk.type !== 'CUSTOM' || chunk.name !== 'motion:status') return
      const value: unknown = chunk.value
      if (
        typeof value === 'object' &&
        value !== null &&
        'status' in value &&
        typeof value.status === 'string'
      ) {
        setJobStatus(value.status)
      }
    },
  })
  const fbx = result?.assets.find((asset) => asset.format === 'fbx')

  return (
    <div>
      <button
        onClick={() =>
          generate({ prompt: 'An orc swings a two-handed axe', duration: 4 })
        }
        disabled={isLoading}
      >
        Generate
      </button>
      <p>Status: {jobStatus}</p>
      {error ? <p role="alert">{error.message}</p> : null}
      {fbx ? <a href={fbx.url}>Download FBX</a> : null}
    </div>
  )
}

The stream sends these events:

EventWhenValue
RUN_STARTEDThe run starts
CUSTOM motion:job:createdThe provider accepted the job{ jobId, model }
CUSTOM motion:statusAfter each status check{ jobId, status, progress?, queuePosition?, error? }
CUSTOM generation:resultThe files are readyMotionGenerationResult
RUN_FINISHEDThe run ends

status is 'pending', 'processing', 'completed', or 'failed'. If the job fails, the stream sends RUN_ERROR. generation:progress comes only when the provider reports a real number. fal does not.

Stream options:

  • pollingInterval: the time between status checks. The default is 2000 ms.
  • maxDuration: the longest time the stream waits. After that, the stream sends RUN_ERROR and the job keeps running. The default is 600000 ms (10 minutes).

Stop and cancel

Stop and cancel are two different things:

  • Stop watching. abortSignal, timeout, or a closed stream stops the local work only. The provider job keeps running, and you still pay for it.
  • Cancel on the provider. cancelMotionJob() asks the provider to stop the job.
ts
import { cancelMotionJob } from '@tanstack/ai'
import { falMotion } from '@tanstack/ai-fal'

await cancelMotionJob({
  adapter: falMotion('fal-ai/hunyuan-motion'),
  jobId: 'the-job-id',
})

On fal:

  • fal removes a job that waits in the queue.
  • A job that already runs gets the cancel request, but it can still finish.
  • fal refuses a cancel for a job that is complete. Then cancelMotionJob() throws.

If an adapter has no cancel API, cancelMotionJob() throws.

Let an agent make clips

An agent can make clips with two tools. One tool starts a job, and one tool checks it. A clip takes from seconds to minutes, so the agent starts the job, keeps working, and checks later.

ts
// tools.ts
import {
  generateMotion,
  getMotionJobStatus,
  toolDefinition,
} from '@tanstack/ai'
import { falMotion } from '@tanstack/ai-fal'
import { z } from 'zod'

const motionAdapter = falMotion('fal-ai/hunyuan-motion')

export const generateMotionClip = toolDefinition({
  name: 'generate_motion_clip',
  description:
    'Start a skeletal animation clip for a humanoid character. Returns a job id.',
  inputSchema: z.object({
    prompt: z.string().meta({
      description: 'The body motion, for example a two-handed axe swing',
    }),
    duration: z
      .number()
      .min(0.5)
      .max(12)
      .optional()
      .meta({ description: 'Clip length in seconds' }),
  }),
}).server(async ({ prompt, duration }) => {
  const { jobId } = await generateMotion({
    adapter: motionAdapter,
    threadId: 'motion-agent',
    prompt,
    duration,
  })
  return { jobId }
})

export const checkMotionClip = toolDefinition({
  name: 'check_motion_clip',
  description:
    'Check a motion clip job. Returns the status, and the file URLs when the clip is done.',
  inputSchema: z.object({ jobId: z.string() }),
}).server(async ({ jobId }) => {
  const status = await getMotionJobStatus({ adapter: motionAdapter, jobId })
  return {
    status: status.status,
    error: status.error,
    files:
      status.motion?.assets.map((asset) => ({
        format: asset.format,
        url: asset.url,
      })) ?? [],
  }
})

Pass both tools to chat():

ts
// api/chat.ts
import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
} from '@tanstack/ai'
import { openaiText } from '@tanstack/ai-openai'
import { checkMotionClip, generateMotionClip } from './tools'

export async function POST(request: Request) {
  const { messages, threadId, runId } = await chatParamsFromRequest(request)

  const stream = chat({
    adapter: openaiText('gpt-6.1-sol'),
    messages,
    threadId,
    runId,
    tools: [generateMotionClip, checkMotionClip],
  })

  return toServerSentEventsResponse(stream)
}

Ask the agent for "an orc attack clip". It calls generate_motion_clip, then check_motion_clip until the files are ready. For more about tools, see Server Tools.

Pick a provider

ProviderFilesNotesIn TanStack AI
fal Hunyuan MotionFBX, JSONSeed control. Cancel for queued jobs. No input for your own character.falMotion(). See fal.ai.
Meshy text-to-motionFBX (prime mode), BVH (swift mode)2 to 10 second clips. A running task cannot be cancelled. Needs a paid plan.Not built in yet
UthanaFBX, GLBUpload your own character, and Uthana makes the clip on it. 3.0 takes the motion from a generated video. No seed. No cancel.uthanaMotion(). See Uthana.

To use a provider that is not built in, write an adapter.

Write a motion adapter

An adapter connects generateMotion() to a provider. Extend BaseMotionAdapter and add three methods:

  • createMotionJob: submit the prompt and return the job id.
  • getMotionStatus: return 'pending', 'processing', 'completed', or 'failed'.
  • getMotion: return the finished files.
  • cancelMotionJob (optional): add it only when the provider has a cancel API.
ts
import { BaseMotionAdapter } from '@tanstack/ai/adapters'
import type {
  MotionGenerationOptions,
  MotionGenerationResult,
  MotionJobResult,
  MotionStatusResult,
} from '@tanstack/ai'

// Your provider client. Replace it with real HTTP or SDK calls.
declare const studio: {
  submit: (
    input: { prompt: string; seconds?: number; seed?: number },
    signal?: AbortSignal,
  ) => Promise<{ id: string }>
  status: (
    id: string,
    signal?: AbortSignal,
  ) => Promise<{ state: 'queued' | 'running' | 'done' | 'error'; message?: string }>
  files: (
    id: string,
    signal?: AbortSignal,
  ) => Promise<Array<{ url: string; extension: string }>>
}

interface StudioMotionOptions {
  seed?: number
}

export class StudioMotionAdapter extends BaseMotionAdapter<
  'studio-motion-1',
  StudioMotionOptions
> {
  readonly name = 'studio'

  async createMotionJob(
    options: MotionGenerationOptions<StudioMotionOptions>,
  ): Promise<MotionJobResult> {
    options.logger.request(`activity=generateMotion provider=studio`, {
      provider: 'studio',
      model: this.model,
    })
    const job = await studio.submit(
      {
        prompt: options.prompt,
        seconds: options.duration,
        seed: options.modelOptions?.seed,
      },
      options.abortSignal,
    )
    return { jobId: job.id, model: this.model }
  }

  async getMotionStatus(
    jobId: string,
    options?: { abortSignal?: AbortSignal },
  ): Promise<MotionStatusResult> {
    const job = await studio.status(jobId, options?.abortSignal)
    switch (job.state) {
      case 'queued':
        return { jobId, status: 'pending' }
      case 'running':
        return { jobId, status: 'processing' }
      case 'done':
        return { jobId, status: 'completed' }
      case 'error':
        return { jobId, status: 'failed', error: job.message }
    }
  }

  async getMotion(
    jobId: string,
    options?: { abortSignal?: AbortSignal },
  ): Promise<MotionGenerationResult> {
    const files = await studio.files(jobId, options?.abortSignal)
    return {
      jobId,
      model: this.model,
      assets: files.map((file) => ({ url: file.url, format: file.extension })),
    }
  }
}

Use it like any other adapter: generateMotion({ adapter: new StudioMotionAdapter('studio-motion-1'), ... }).

Rules for an adapter:

  • Return only the metadata that the provider sends. Do not guess a clip length or a frame rate.
  • Set progress only when the provider reports a real number.
  • If the provider answers at once, do the work in createMotionJob. Then return 'completed' on the first getMotionStatus call.
  • Pass abortSignal to the HTTP call. It stops the request, not the provider job.

Advanced

Check a job from another request

getMotionJobStatus() needs only the adapter and the job id. You can submit in one request and check in another request, or in another process. When the job is complete, status.motion holds the files.

Middleware, tracing, and devtools

A job that you submit and check later is one middleware run:

  • The generateMotion() call without stream opens the run.
  • The getMotionJobStatus() call that sees 'completed' or 'failed' closes it.
  • Pass the same middleware array and the same threadId to both calls.

With stream: true, one call is one run from submit to result.

otelMiddleware() names the span operation motion_generation. See OpenTelemetry. The devtools show motion:request:started, motion:request:completed, motion:request:error, and motion:usage events.