When creating or editing a Runway workflow, use the fewest nodes that clearly express the requested pipeline. Lay out every node left-to-right with non-overlapping `nodeProps.position` coordinates: inputs at x≈0; processing at x≈600-1100; outputs at x≈1600. Keep each column vertically spaced instead of making one long vertical chain. For structured LLM steps, group the system-prompt constant, LLM, and JSON Parse node in the processing area; make the system prompt require JSON matching the parser paths, then wire LLM `text` directly into JSON Parse `text`. Read `runway://docs/workflows/authoring`, copy valid node shapes from an existing workflow, and call `validate_workflow_graph` before saving.
Protocol
Pass100/10016 / 16 pts
Connection, authentication, and protocol compatibility.
How to fix: For tools that render a UI resource, add openai/toolInvocation with string fields invoking and invoked (present-tense labels shown while the tool runs and after it finishes, e.g. "Searching…" / "Searched").
How to fix: Use a ui:// URI and text/html;profile=mcp-app. For mcp-use v1, use widgetMetadata.metadata.widgetDescription; for v2, use the bound tool's view.description. The current description check reads _meta.ui.widgetDescription or _meta["openai/widgetDescription"] on resource contents or linked tools, not resource.description. Consequently, v2 view.description alone may leave this warning: report the checker compatibility gap instead of adding unsupported v1 fields or claiming the warning is fixed.
How to fix: For ChatGPT submission, set one exact HTTPS origin such as "https://app.example.com" on every view via _meta.ui.domain (or openai/widgetDomain). Paths are not valid and all views in one app must share the origin.
The title clearly communicates the tool’s action and subject. Keep it concise.
Description98/100
Confirms the Runway connection is authenticated and returns the workspace this MCP connection is pinned to (chosen at sign-in) plus the list of image/video models available in that workspace. It does not return personal account details. Call this first to confirm auth and to pick a model the user can actually run. Check `modelDeprecations`: briefly warn about an upcoming sunset when using that model; it remains usable until that date. Never offer a retired or sunset task variant. Ask before switching models. If `deprecationsKnown` is false, deprecation status could not be checked; do not claim a model is current or retired from that result. Also call it after a paid_plan_required result — `availableImageModels` and `availableVideoModels` are the only models you may offer as a fallback. Do not guess from a generate tool schema. Runway account availability is checked against the currently connected workspace. If `upgradeUrl` is present, the connected workspace is on a free plan. When the user asked for something that plan cannot run (every video tool, or an image model missing from `availableImageModels`), call `show_plans_and_credits` with `gatedCapability` set to the blocked action (e.g. "Generating video") to render the inline upgrade card — do not stop at describing the limitation. Paste `upgradeUrl` only if that tool is unavailable. If `multipleWorkspacesAvailable` is true, the user belongs to other Runway workspaces — call `list_workspaces` to see them. To switch the active workspace, the user must disconnect and reconnect the MCP from their client.
The description explains the capability, when to use it, and what to expect.
Inputs97/100
Input documentation gives an agent enough context to populate the arguments.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
{
"type": "object",
"required": [
"authenticated",
"multipleWorkspacesAvailable",
"availableImageModels",
"availableVideoModels"
],
"properties": {
"team": {
"description": "The workspace this MCP connection is pinned to, or null if none: {id, name, organization, isPersonal, wasExplicitlyPicked}."
},
"credits": {
"type": "object",
"description": "Credit balance of the connected workspace: {unlimited: false, total, plan, purchased} — plan credits reset monthly, purchased ones do not — or {unlimited: true}. Omitted when the balance could not be resolved. Mention the balance to the user only when they ask or when it is low."
},
"nextTool": {
"type": "string",
"description": "Present on free plans: 'show_plans_and_credits'. Call it (with `gatedCapability`) when the user asked for a paid-only capability such as video."
},
"proTrial": {
"type": "object",
"description": "Present only while a free Pro trial is running: {active: true, endsAt, renews, credits, cancelTool: 'cancel_pro_trial'}."
},
"upgradeUrl": {
"type": "string",
"description": "Present on free plans. Fallback link only — prefer calling `nextTool` so the user gets the inline upgrade card."
},
"authenticated": {
"type": "boolean",
"description": "True when the Runway connection is authenticated."
},
"deprecationsKnown": {
"type": "boolean",
"description": "Whether API task deprecation facts could be resolved for this workspace."
},
"gatedCapabilities": {
"type": "object",
"description": "Present on free plans. Ready-made `gatedCapability` phrases for show_plans_and_credits, keyed by what the user asked for: {video, editVideo, expandVideo, multishotVideo, upscaleImage}."
},
"modelDeprecations": {
"type": "array",
"items": {
"type": "object"
},
"description": "API facts for deprecated/retired task variants: {modelId, modelLabel, tool, taskType, sunsetAt, reason, retired, unavailable}. Upcoming sunsets remain usable. Models with an unavailable task variant are omitted from the broad available-model lists; never use an unavailable variant."
},
"disabledModelsKnown": {
"type": "boolean",
"description": "Whether the workspace admin-disabled model list could be resolved."
},
"availableImageModels": {
"type": "array",
"items": {
"type": "string"
},
"description": "Image model ids usable in the connected workspace."
},
"availableVideoModels": {
"type": "array",
"items": {
"type": "string"
},
"description": "Video model ids usable in the connected workspace."
},
"multipleWorkspacesAvailable": {
"type": "boolean",
"description": "True when the user belongs to other workspaces (see list_workspaces)."
}
},
"description": "Connection status, the pinned workspace, and the model ids available in it. Contains no personal account details.",
"additionalProperties": true
}
Original tool metadata
{
"name": "whoami",
"title": "Account & Models",
"description": "Confirms the Runway connection is authenticated and returns the workspace this MCP connection is pinned to (chosen at sign-in) plus the list of image/video models available in that workspace. It does not return personal account details. Call this first to confirm auth and to pick a model the user can actually run. Check `modelDeprecations`: briefly warn about an upcoming sunset when using that model; it remains usable until that date. Never offer a retired or sunset task variant. Ask before switching models. If `deprecationsKnown` is false, deprecation status could not be checked; do not claim a model is current or retired from that result. Also call it after a paid_plan_required result — `availableImageModels` and `availableVideoModels` are the only models you may offer as a fallback. Do not guess from a generate tool schema. Runway account availability is checked against the currently connected workspace. If `upgradeUrl` is present, the connected workspace is on a free plan. When the user asked for something that plan cannot run (every video tool, or an image model missing from `availableImageModels`), call `show_plans_and_credits` with `gatedCapability` set to the blocked action (e.g. \"Generating video\") to render the inline upgrade card — do not stop at describing the limitation. Paste `upgradeUrl` only if that tool is unavailable. If `multipleWorkspacesAvailable` is true, the user belongs to other Runway workspaces — call `list_workspaces` to see them. To switch the active workspace, the user must disconnect and reconnect the MCP from their client.",
"inputSchema": {
"type": "object",
"properties": {
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"authenticated",
"multipleWorkspacesAvailable",
"availableImageModels",
"availableVideoModels"
],
"properties": {
"team": {
"description": "The workspace this MCP connection is pinned to, or null if none: {id, name, organization, isPersonal, wasExplicitlyPicked}."
},
"credits": {
"type": "object",
"description": "Credit balance of the connected workspace: {unlimited: false, total, plan, purchased} — plan credits reset monthly, purchased ones do not — or {unlimited: true}. Omitted when the balance could not be resolved. Mention the balance to the user only when they ask or when it is low."
},
"nextTool": {
"type": "string",
"description": "Present on free plans: 'show_plans_and_credits'. Call it (with `gatedCapability`) when the user asked for a paid-only capability such as video."
},
"proTrial": {
"type": "object",
"description": "Present only while a free Pro trial is running: {active: true, endsAt, renews, credits, cancelTool: 'cancel_pro_trial'}."
},
"upgradeUrl": {
"type": "string",
"description": "Present on free plans. Fallback link only — prefer calling `nextTool` so the user gets the inline upgrade card."
},
"authenticated": {
"type": "boolean",
"description": "True when the Runway connection is authenticated."
},
"deprecationsKnown": {
"type": "boolean",
"description": "Whether API task deprecation facts could be resolved for this workspace."
},
"gatedCapabilities": {
"type": "object",
"description": "Present on free plans. Ready-made `gatedCapability` phrases for show_plans_and_credits, keyed by what the user asked for: {video, editVideo, expandVideo, multishotVideo, upscaleImage}."
},
"modelDeprecations": {
"type": "array",
"items": {
"type": "object"
},
"description": "API facts for deprecated/retired task variants: {modelId, modelLabel, tool, taskType, sunsetAt, reason, retired, unavailable}. Upcoming sunsets remain usable. Models with an unavailable task variant are omitted from the broad available-model lists; never use an unavailable variant."
},
"disabledModelsKnown": {
"type": "boolean",
"description": "Whether the workspace admin-disabled model list could be resolved."
},
"availableImageModels": {
"type": "array",
"items": {
"type": "string"
},
"description": "Image model ids usable in the connected workspace."
},
"availableVideoModels": {
"type": "array",
"items": {
"type": "string"
},
"description": "Video model ids usable in the connected workspace."
},
"multipleWorkspacesAvailable": {
"type": "boolean",
"description": "True when the user belongs to other workspaces (see list_workspaces)."
}
},
"description": "Connection status, the pinned workspace, and the model ids available in it. Contains no personal account details.",
"additionalProperties": true
},
"annotations": {
"title": "Account & Models",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description98/100
Lists every Runway workspace the authenticated user belongs to, with role and a compact disabled-model summary per workspace. The MCP connection is pinned to ONE workspace (chosen at sign-in); use this tool to confirm which workspace you are currently posting to and to compare options before suggesting a switch. By default this omits full disabled-model arrays to keep the response compact; pass includeDisabledModels=true only when you need the exact policy list. To switch workspaces, the user must disconnect and reconnect the Runway MCP from their client (this is intentional — it keeps account context and audit logs aligned to a single workspace per session).
The description explains the capability, when to use it, and what to expect.
Inputs98/100
Input documentation gives an agent enough context to populate the arguments.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
includeDisabledModels
Include full disabledModels arrays for each workspace. Default false to avoid large responses.
{
"type": "object",
"required": [
"workspaces"
],
"properties": {
"workspaces": {
"type": "array",
"items": {
"type": "object",
"required": [
"id",
"name",
"isCurrentlySelected"
],
"properties": {
"id": {
"type": "number"
},
"name": {
"type": "string"
},
"role": {
"description": "User's role in this workspace, or null."
},
"isPersonal": {
"type": "boolean"
},
"organization": {
"description": "{id, name} of the owning organization, or null."
},
"disabledModelCount": {
"description": "Number of admin-disabled models, or null if unknown."
},
"isCurrentlySelected": {
"type": "boolean"
}
},
"additionalProperties": true
},
"description": "Workspace summaries: {id, name, organization, isPersonal, role, disabledModelCount, disabledModels?, isCurrentlySelected}."
},
"currentTeamId": {
"description": "Workspace id this MCP connection is pinned to, or null."
},
"switchInstructions": {
"type": "string",
"description": "How the user switches the pinned workspace."
},
"includeDisabledModels": {
"type": "boolean",
"description": "Whether full disabledModels arrays were included."
}
},
"description": "All Runway workspaces the user belongs to, with role and disabled-model summaries.",
"additionalProperties": true
}
Original tool metadata
{
"name": "list_workspaces",
"title": "List Workspaces",
"description": "Lists every Runway workspace the authenticated user belongs to, with role and a compact disabled-model summary per workspace. The MCP connection is pinned to ONE workspace (chosen at sign-in); use this tool to confirm which workspace you are currently posting to and to compare options before suggesting a switch. By default this omits full disabled-model arrays to keep the response compact; pass includeDisabledModels=true only when you need the exact policy list. To switch workspaces, the user must disconnect and reconnect the Runway MCP from their client (this is intentional — it keeps account context and audit logs aligned to a single workspace per session).",
"inputSchema": {
"type": "object",
"properties": {
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"includeDisabledModels": {
"type": "boolean",
"description": "Include full disabledModels arrays for each workspace. Default false to avoid large responses."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"workspaces"
],
"properties": {
"workspaces": {
"type": "array",
"items": {
"type": "object",
"required": [
"id",
"name",
"isCurrentlySelected"
],
"properties": {
"id": {
"type": "number"
},
"name": {
"type": "string"
},
"role": {
"description": "User's role in this workspace, or null."
},
"isPersonal": {
"type": "boolean"
},
"organization": {
"description": "{id, name} of the owning organization, or null."
},
"disabledModelCount": {
"description": "Number of admin-disabled models, or null if unknown."
},
"isCurrentlySelected": {
"type": "boolean"
}
},
"additionalProperties": true
},
"description": "Workspace summaries: {id, name, organization, isPersonal, role, disabledModelCount, disabledModels?, isCurrentlySelected}."
},
"currentTeamId": {
"description": "Workspace id this MCP connection is pinned to, or null."
},
"switchInstructions": {
"type": "string",
"description": "How the user switches the pinned workspace."
},
"includeDisabledModels": {
"type": "boolean",
"description": "Whether full disabledModels arrays were included."
}
},
"description": "All Runway workspaces the user belongs to, with role and disabled-model summaries.",
"additionalProperties": true
},
"annotations": {
"title": "List Workspaces",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Lists the connected workspace's visible Brand Kits with their category names and descriptions, without loading assets. Use this for cheap discovery when the user asks for branded work and no Brand Kit is already in context. Call `get_brand_kit` only after a kit is selected or when there is a single clearly applicable kit. An empty list means this workspace has no visible Brand Kits.
The description explains the capability, when to use it, and what to expect.
Inputs96/100
Input documentation gives an agent enough context to populate the arguments.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
{
"name": "list_brand_kits",
"title": "List Brand Kits",
"description": "Lists the connected workspace's visible Brand Kits with their category names and descriptions, without loading assets. Use this for cheap discovery when the user asks for branded work and no Brand Kit is already in context. Call `get_brand_kit` only after a kit is selected or when there is a single clearly applicable kit. An empty list means this workspace has no visible Brand Kits.",
"inputSchema": {
"type": "object",
"properties": {
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind",
"brandKits"
],
"properties": {
"kind": {
"enum": [
"brand_kit_list"
],
"type": "string"
},
"brandKits": {
"type": "array",
"items": {
"type": "object",
"required": [
"id",
"name",
"categories"
],
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"updatedAt": {
"description": "Last update time, or null."
},
"categories": {
"type": "array",
"items": {
"type": "object",
"required": [
"id",
"name"
],
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"description": {
"description": "Category description, or null."
}
},
"additionalProperties": true
}
}
},
"additionalProperties": true
}
}
},
"description": "Visible Brand Kits in the connected workspace.",
"additionalProperties": true
},
"annotations": {
"title": "List Brand Kits",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description97/100
Gets one visible Brand Kit from the connected workspace, including its curated image, video, and audio assets grouped by category. Call this when a kit has been selected, when there is one clearly applicable kit, or when the user asks what is inside a kit. Image and video asset URLs can be passed unchanged into the existing generation tools as references. Reading Brand Kits is free and non-destructive.
The description explains the capability, when to use it, and what to expect.
Inputs95/100
Input documentation gives an agent enough context to populate the arguments.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
The title clearly communicates the tool’s action and subject. Keep it concise.
Description90/100
Lists Runway editor workflows in the authenticated workspace (name, id, updated time). Does not include the graph — use `get_workflow` with `include_graph: true` to inspect. If a similar workflow already exists, reuse it (`get_workflow` / `save_workflow_version` / `run_workflow`) instead of creating a duplicate. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
The description explains the capability, when to use it, and what to expect.
Inputs92/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
limit
Max results (10-100, default 12).
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
cursor
Pagination cursor from a previous list_workflows response.
{
"type": "object",
"required": [
"kind",
"workflows"
],
"properties": {
"kind": {
"enum": [
"workflow_list"
],
"type": "string"
},
"workflows": {
"type": "array",
"items": {
"type": "object",
"required": [
"id",
"name"
],
"properties": {
"id": {
"type": "string",
"description": "Workflow UUID for get_workflow / run_workflow."
},
"name": {
"type": "string"
},
"createdAt": {
"type": "string"
},
"updatedAt": {
"type": "string"
},
"description": {
"description": "Workflow description, or null."
}
},
"additionalProperties": true
}
},
"nextCursor": {
"description": "Pagination cursor for the next page, or null."
}
},
"description": "Paginated list of saved workflows in the connected workspace.",
"additionalProperties": true
}
Original tool metadata
{
"name": "list_workflows",
"title": "List Workflows",
"description": "Lists Runway editor workflows in the authenticated workspace (name, id, updated time). Does not include the graph — use `get_workflow` with `include_graph: true` to inspect. If a similar workflow already exists, reuse it (`get_workflow` / `save_workflow_version` / `run_workflow`) instead of creating a duplicate. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.",
"inputSchema": {
"type": "object",
"properties": {
"limit": {
"type": "number",
"maximum": 100,
"minimum": 10,
"description": "Max results (10-100, default 12)."
},
"cursor": {
"type": "string",
"description": "Pagination cursor from a previous list_workflows response."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind",
"workflows"
],
"properties": {
"kind": {
"enum": [
"workflow_list"
],
"type": "string"
},
"workflows": {
"type": "array",
"items": {
"type": "object",
"required": [
"id",
"name"
],
"properties": {
"id": {
"type": "string",
"description": "Workflow UUID for get_workflow / run_workflow."
},
"name": {
"type": "string"
},
"createdAt": {
"type": "string"
},
"updatedAt": {
"type": "string"
},
"description": {
"description": "Workflow description, or null."
}
},
"additionalProperties": true
}
},
"nextCursor": {
"description": "Pagination cursor for the next page, or null."
}
},
"description": "Paginated list of saved workflows in the connected workspace.",
"additionalProperties": true
},
"annotations": {
"title": "List Workflows",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description95/100
Gets a Runway workflow by id (name, description, updated time). Set `include_graph: true` to also load the saved graph. Optional `version` loads a specific version instead of latest. The graph JSON is for your next tool call — summarize the pipeline for the user; do not dump node IDs or ASCII diagrams unless they ask how it is wired. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
The description explains the capability, when to use it, and what to expect.
Inputs88/100
Clarify units, where to obtain this value, or conventions not already expressed in the schema.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
workflow_id
Workflow UUID.
Clarify units, where to obtain this value, or conventions not already expressed in the schema.
include_graph
If true, include the saved graph (latest, or `version` if set).
Clear enough to supply this argument.
version
Specific version number to load when include_graph is true (default: latest).
Align the output schema with the tool’s documented result.
View output schema
{
"type": "object",
"required": [
"kind",
"id",
"name"
],
"properties": {
"id": {
"type": "string",
"description": "Workflow UUID."
},
"kind": {
"enum": [
"workflow"
],
"type": "string"
},
"name": {
"type": "string"
},
"version": {
"type": "number",
"description": "Loaded graph version (present when include_graph is true)."
}
},
"description": "Workflow metadata. When include_graph is true, the full graph JSON is returned in the text content (second block), not here.",
"additionalProperties": true
}
Original tool metadata
{
"name": "get_workflow",
"title": "Get Workflow",
"description": "Gets a Runway workflow by id (name, description, updated time). Set `include_graph: true` to also load the saved graph. Optional `version` loads a specific version instead of latest. The graph JSON is for your next tool call — summarize the pipeline for the user; do not dump node IDs or ASCII diagrams unless they ask how it is wired. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.",
"inputSchema": {
"type": "object",
"required": [
"workflow_id"
],
"properties": {
"version": {
"type": "number",
"description": "Specific version number to load when include_graph is true (default: latest)."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"workflow_id": {
"type": "string",
"description": "Workflow UUID."
},
"include_graph": {
"type": "boolean",
"description": "If true, include the saved graph (latest, or `version` if set)."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind",
"id",
"name"
],
"properties": {
"id": {
"type": "string",
"description": "Workflow UUID."
},
"kind": {
"enum": [
"workflow"
],
"type": "string"
},
"name": {
"type": "string"
},
"version": {
"type": "number",
"description": "Loaded graph version (present when include_graph is true)."
}
},
"description": "Workflow metadata. When include_graph is true, the full graph JSON is returned in the text content (second block), not here.",
"additionalProperties": true
},
"annotations": {
"title": "Get Workflow",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Runs a saved Runway workflow (latest version) in the authenticated workspace. Consumes credits. Returns immediately with a task id — the inline MCP App viewer polls progress when available. Only call `get_task` to poll when no inline viewer is present or the user asks for status. Optional `node_ids` runs only those workflow nodes (upstream must already have succeeded). Optional `node_outputs` overrides constant/asset values for this run without saving a new version. Use `get_workflow` with `include_graph: true` first if you need node ids. Runway account availability is checked against the currently connected workspace. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
The description explains the capability, when to use it, and what to expect.
Inputs88/100
Clarify units, where to obtain this value, or conventions not already expressed in the schema.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
workflow_id
Workflow UUID to run.
Clarify units, where to obtain this value, or conventions not already expressed in the schema.
node_ids
Optional workflow-node UUIDs for a partial run. Omit to run all configured nodes.
Clear enough to supply this argument.
node_outputs
Optional per-node overrides (nodeId → port → value) for this run only. Asset nodes accept a dataset asset id or a hosted media URL.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "run_workflow",
"title": "Run Workflow",
"description": "Runs a saved Runway workflow (latest version) in the authenticated workspace. Consumes credits. Returns immediately with a task id — the inline MCP App viewer polls progress when available. Only call `get_task` to poll when no inline viewer is present or the user asks for status. Optional `node_ids` runs only those workflow nodes (upstream must already have succeeded). Optional `node_outputs` overrides constant/asset values for this run without saving a new version. Use `get_workflow` with `include_graph: true` first if you need node ids. Runway account availability is checked against the currently connected workspace. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.",
"inputSchema": {
"type": "object",
"required": [
"workflow_id"
],
"properties": {
"node_ids": {
"type": "array",
"items": {
"type": "string"
},
"description": "Optional workflow-node UUIDs for a partial run. Omit to run all configured nodes."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"workflow_id": {
"type": "string",
"description": "Workflow UUID to run."
},
"node_outputs": {
"type": "object",
"description": "Optional per-node overrides (nodeId → port → value) for this run only. Asset nodes accept a dataset asset id or a hosted media URL."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Run Workflow",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description87/100
Creates a new Runway workflow with an initial graph (version 1). Do not invent graph JSON — `list_workflows` then `get_workflow` include_graph: true and copy nodes/edges. If a similar workflow already exists in the workspace, update that one instead of creating a duplicate. Use the fewest necessary nodes and set non-overlapping `nodeProps.position` coordinates: inputs left, processing (including system prompt + LLM + JSON Parse) center, outputs right. Never tell the user to implement the graph themselves. Call `validate_workflow_graph` first. Response includes an editor URL and validation errors/warnings even when saved. To run it, call `run_workflow` next. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
The description explains the capability, when to use it, and what to expect.
Inputs75/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
name
Workflow display name (max 80 characters).
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
description
Optional short description.
Explain the meaning of this value, not just its name.
graph
API graph JSON `{ version, nodes[], edges[] }` with UUID node ids and `nodeType`. Every node must have a non-overlapping left-to-right `nodeProps.position`; minimize redundant constants and transforms. Copy from `get_workflow` include_graph. See `runway://docs/workflows/examples`. Not `{ type, config, inputs }` or ids like `input_photo`.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind",
"id",
"name",
"version"
],
"properties": {
"id": {
"type": "string",
"description": "Workflow UUID. Pass to `run_workflow`."
},
"kind": {
"enum": [
"workflow_created"
],
"type": "string"
},
"name": {
"type": "string"
},
"errors": {
"type": "array",
"items": {
"type": "object",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
},
"additionalProperties": true
}
},
"version": {
"type": "number",
"description": "Always 1 for a newly created workflow."
},
"warnings": {
"type": "array",
"items": {
"type": "object",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
},
"additionalProperties": true
}
},
"editorUrl": {
"type": "string",
"description": "Runway editor URL for the workflow, safe to show the user."
},
"errorCount": {
"type": "number"
},
"warningCount": {
"type": "number"
}
},
"description": "The created workflow. The graph is saved even when `errorCount` is above zero, so check it before calling `run_workflow`.",
"additionalProperties": true
}
Original tool metadata
{
"name": "create_workflow",
"title": "Create Workflow",
"description": "Creates a new Runway workflow with an initial graph (version 1). Do not invent graph JSON — `list_workflows` then `get_workflow` include_graph: true and copy nodes/edges. If a similar workflow already exists in the workspace, update that one instead of creating a duplicate. Use the fewest necessary nodes and set non-overlapping `nodeProps.position` coordinates: inputs left, processing (including system prompt + LLM + JSON Parse) center, outputs right. Never tell the user to implement the graph themselves. Call `validate_workflow_graph` first. Response includes an editor URL and validation errors/warnings even when saved. To run it, call `run_workflow` next. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.",
"inputSchema": {
"type": "object",
"required": [
"name",
"graph"
],
"properties": {
"name": {
"type": "string",
"description": "Workflow display name (max 80 characters)."
},
"graph": {
"type": "object",
"description": "API graph JSON `{ version, nodes[], edges[] }` with UUID node ids and `nodeType`. Every node must have a non-overlapping left-to-right `nodeProps.position`; minimize redundant constants and transforms. Copy from `get_workflow` include_graph. See `runway://docs/workflows/examples`. Not `{ type, config, inputs }` or ids like `input_photo`."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"description": {
"type": "string",
"description": "Optional short description."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind",
"id",
"name",
"version"
],
"properties": {
"id": {
"type": "string",
"description": "Workflow UUID. Pass to `run_workflow`."
},
"kind": {
"enum": [
"workflow_created"
],
"type": "string"
},
"name": {
"type": "string"
},
"errors": {
"type": "array",
"items": {
"type": "object",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
},
"additionalProperties": true
}
},
"version": {
"type": "number",
"description": "Always 1 for a newly created workflow."
},
"warnings": {
"type": "array",
"items": {
"type": "object",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
},
"additionalProperties": true
}
},
"editorUrl": {
"type": "string",
"description": "Runway editor URL for the workflow, safe to show the user."
},
"errorCount": {
"type": "number"
},
"warningCount": {
"type": "number"
}
},
"description": "The created workflow. The graph is saved even when `errorCount` is above zero, so check it before calling `run_workflow`.",
"additionalProperties": true
},
"annotations": {
"title": "Create Workflow",
"readOnlyHint": false,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description93/100
Saves a new version of an existing workflow graph. Graph must be API JSON copied from `get_workflow` — not `{ type, config }`. Keep the graph minimal and its `nodeProps.position` layout readable: inputs left, processing center, outputs right, with no overlaps. Optional `expected_version` guards against stale edits (must be within 1 of latest). Response includes an editor URL. Validate first with `validate_workflow_graph`. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
The description explains the capability, when to use it, and what to expect.
Inputs88/100
Clarify units, where to obtain this value, or conventions not already expressed in the schema.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
workflow_id
Workflow UUID.
Clarify units, where to obtain this value, or conventions not already expressed in the schema.
graph
API graph JSON `{ version, nodes[], edges[] }` with UUID ids. Preserve or improve a non-overlapping left-to-right `nodeProps.position` layout. Copy from `get_workflow`.
Clear enough to supply this argument.
expected_version
Version number the client last read; API rejects if more than 1 behind latest.
{
"type": "object",
"required": [
"kind",
"id",
"name",
"version"
],
"properties": {
"id": {
"type": "string",
"description": "Workflow UUID. Pass to `run_workflow`."
},
"kind": {
"enum": [
"workflow_version_saved"
],
"type": "string"
},
"name": {
"type": "string"
},
"errors": {
"type": "array",
"items": {
"type": "object",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
},
"additionalProperties": true
}
},
"version": {
"type": "number",
"description": "Version number just created."
},
"warnings": {
"type": "array",
"items": {
"type": "object",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
},
"additionalProperties": true
}
},
"editorUrl": {
"type": "string",
"description": "Runway editor URL for the workflow, safe to show the user."
},
"errorCount": {
"type": "number"
},
"warningCount": {
"type": "number"
}
},
"description": "The newly appended workflow version. Earlier versions are preserved. The graph is saved even when `errorCount` is above zero.",
"additionalProperties": true
}
Original tool metadata
{
"name": "save_workflow_version",
"title": "Save Workflow Version",
"description": "Saves a new version of an existing workflow graph. Graph must be API JSON copied from `get_workflow` — not `{ type, config }`. Keep the graph minimal and its `nodeProps.position` layout readable: inputs left, processing center, outputs right, with no overlaps. Optional `expected_version` guards against stale edits (must be within 1 of latest). Response includes an editor URL. Validate first with `validate_workflow_graph`. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.",
"inputSchema": {
"type": "object",
"required": [
"workflow_id",
"graph"
],
"properties": {
"graph": {
"type": "object",
"description": "API graph JSON `{ version, nodes[], edges[] }` with UUID ids. Preserve or improve a non-overlapping left-to-right `nodeProps.position` layout. Copy from `get_workflow`."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"workflow_id": {
"type": "string",
"description": "Workflow UUID."
},
"expected_version": {
"type": "number",
"description": "Version number the client last read; API rejects if more than 1 behind latest."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind",
"id",
"name",
"version"
],
"properties": {
"id": {
"type": "string",
"description": "Workflow UUID. Pass to `run_workflow`."
},
"kind": {
"enum": [
"workflow_version_saved"
],
"type": "string"
},
"name": {
"type": "string"
},
"errors": {
"type": "array",
"items": {
"type": "object",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
},
"additionalProperties": true
}
},
"version": {
"type": "number",
"description": "Version number just created."
},
"warnings": {
"type": "array",
"items": {
"type": "object",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
},
"additionalProperties": true
}
},
"editorUrl": {
"type": "string",
"description": "Runway editor URL for the workflow, safe to show the user."
},
"errorCount": {
"type": "number"
},
"warningCount": {
"type": "number"
}
},
"description": "The newly appended workflow version. Earlier versions are preserved. The graph is saved even when `errorCount` is above zero.",
"additionalProperties": true
},
"annotations": {
"title": "Save Workflow Version",
"readOnlyHint": false,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description94/100
Validates a workflow graph without saving. Use before `create_workflow` or `save_workflow_version`. Returns task deprecations for model steps. Tell the user about upcoming sunsets and ask before switching a retired model. Graph must be API JSON (UUID node ids, `nodeType`, `nodeOutputs`) — copy from `get_workflow` include_graph, do not invent `{ type, config }`. Set `estimate_cost: true` to also fetch a credit estimate for the graph. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
The description explains the capability, when to use it, and what to expect.
Inputs97/100
Input documentation gives an agent enough context to populate the arguments.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
graph
API graph JSON `{ version, nodes[], edges[] }`. Copy from `get_workflow` include_graph. Not `{ type, config, inputs }`.
Clear enough to supply this argument.
estimate_cost
If true, also POST /v1/dynamic_workflows/cost-estimate.
The title clearly communicates the tool’s action and subject. Keep it concise.
Description91/100
Generate a polished creative product ad video from a product URL or product image plus a campaign idea.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
Use this for product marketing / creative ad requests like:
- "Make a jewelry ad featuring a chameleon in a jewelry store"
- "Turn this product page into a cinematic ad"
- "Use this product image and this character reference to make a funny product commercial"
v1 supports only format: `creative_ad_video`.
Workflow:
1. Get the exact product image from `productImages[0]` or extract the best high-resolution product image from `productUrl`.
2. Use optional `referenceImages` as character, scene, mood, or world references.
3. Internally generate a 3x3 storyboard image, preferring GPT Image 2 for paid workspaces and falling back to Nano Banana Pro otherwise.
4. Use that storyboard as sequential shot guidance for Seedance 2, along with the product image and references, to create the final video.
PRODUCT URL FALLBACK:
- `productUrl` extraction is best-effort. Some ecommerce sites block automated access or hide product images behind scripts.
- If extraction fails, do not retry the same URL or claim Runway cannot make the ad. Ask the user for a screenshot of the product page or a direct product image upload, then call this tool again with `productImages` / `productImageFile`.
- Do not call `generate_image` with a text description of the product to create the product reference. That produces a lookalike, not the real SKU.
- For branded/luxury products, the exact product image is required for an accurate ad. Only invent or generate a stand-in product if the user explicitly agrees that an approximate substitute is acceptable.
Inputs:
- productUrl: Product landing page URL. The tool will scrape metadata and rank image candidates, preferring high-resolution product/gallery images.
- productImages: Product image URLs from uploads, recent assets, or previous generations. Preferred over productUrl extraction.
- productImageFile: ChatGPT-only uploaded product image file. Use this when ChatGPT provides a user-uploaded product image.
- referenceImages: Optional character/scene/mood/world references.
- promptText: Required creative campaign idea. If the user requests a named artist/director/photographer/studio style, describe the visual qualities instead and do not include the name.
- duration: 10 by default; use 15 only when the user asks for a longer ad.
STYLE SAFETY: Do not include names of artists, directors, photographers, studios, or other creators anywhere in prompts submitted to Runway tasks. When style direction is needed, translate it into camera language, palette, lighting, genre, era, composition, and material texture.
USER-FACING REPLIES: DO NOT mention internal model names unless asked. Do not mention the intermediate storyboard unless useful. Tell the user the creative ad video is generating and the viewer will update when ready.
The description explains the capability, when to use it, and what to expect.
Inputs35/100
9 of 23 inputs have descriptions. Add descriptions to the missing inputs, including where to obtain IDs and how to supply values.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
productUrl
Product landing page URL. Used only if productImages is empty.
Clear enough to supply this argument.
productImages › items › url
Not provided
Missing description. Explain what this value means and how to supply it.
productImages › items › tag
Not provided
Missing description. Explain what this value means and how to supply it.
productImageFile › download_url
Temporary ChatGPT file download URL.
Clear enough to supply this argument.
productImageFile › file_id
ChatGPT file ID.
Clear enough to supply this argument.
productImageFile › mime_type
MIME type, e.g. image/png.
Clear enough to supply this argument.
productImageFile › file_name
Original file name.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
referenceImages › items › url
Not provided
Missing description. Explain what this value means and how to supply it.
referenceImages › items › tag
Not provided
Missing description. Explain what this value means and how to supply it.
promptText
Required creative ad idea. If the user requests a named artist/director/photographer/studio style, describe the visual qualities instead and do not include the name.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
format
Only supported value in v1: creative_ad_video.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
duration
Final video duration. Default 10. Use 15 only when the user asks for longer.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
storyboardImage › url
Not provided
Missing description. Explain what this value means and how to supply it.
productMetadata › url
Not provided
Missing description. Explain what this value means and how to supply it.
productMetadata › title
Not provided
Missing description. Explain what this value means and how to supply it.
productMetadata › description
Not provided
Missing description. Explain what this value means and how to supply it.
productMetadata › image › url
Not provided
Missing description. Explain what this value means and how to supply it.
productMetadata › image › width
Not provided
Missing description. Explain what this value means and how to supply it.
productMetadata › image › height
Not provided
Missing description. Explain what this value means and how to supply it.
productMetadata › image › alt
Not provided
Missing description. Explain what this value means and how to supply it.
productMetadata › image › source
Not provided
Missing description. Explain what this value means and how to supply it.
productMetadata › image › score
Not provided
Missing description. Explain what this value means and how to supply it.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "generate_product_marketing_video",
"title": "Generate Product Marketing Video",
"description": "Generate a polished creative product ad video from a product URL or product image plus a campaign idea.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nUse this for product marketing / creative ad requests like:\n- \"Make a jewelry ad featuring a chameleon in a jewelry store\"\n- \"Turn this product page into a cinematic ad\"\n- \"Use this product image and this character reference to make a funny product commercial\"\n\nv1 supports only format: `creative_ad_video`.\n\nWorkflow:\n1. Get the exact product image from `productImages[0]` or extract the best high-resolution product image from `productUrl`.\n2. Use optional `referenceImages` as character, scene, mood, or world references.\n3. Internally generate a 3x3 storyboard image, preferring GPT Image 2 for paid workspaces and falling back to Nano Banana Pro otherwise.\n4. Use that storyboard as sequential shot guidance for Seedance 2, along with the product image and references, to create the final video.\n\nPRODUCT URL FALLBACK:\n- `productUrl` extraction is best-effort. Some ecommerce sites block automated access or hide product images behind scripts.\n- If extraction fails, do not retry the same URL or claim Runway cannot make the ad. Ask the user for a screenshot of the product page or a direct product image upload, then call this tool again with `productImages` / `productImageFile`.\n- Do not call `generate_image` with a text description of the product to create the product reference. That produces a lookalike, not the real SKU.\n- For branded/luxury products, the exact product image is required for an accurate ad. Only invent or generate a stand-in product if the user explicitly agrees that an approximate substitute is acceptable.\n\nInputs:\n- productUrl: Product landing page URL. The tool will scrape metadata and rank image candidates, preferring high-resolution product/gallery images.\n- productImages: Product image URLs from uploads, recent assets, or previous generations. Preferred over productUrl extraction.\n- productImageFile: ChatGPT-only uploaded product image file. Use this when ChatGPT provides a user-uploaded product image.\n- referenceImages: Optional character/scene/mood/world references.\n- promptText: Required creative campaign idea. If the user requests a named artist/director/photographer/studio style, describe the visual qualities instead and do not include the name.\n- duration: 10 by default; use 15 only when the user asks for a longer ad.\n\nSTYLE SAFETY: Do not include names of artists, directors, photographers, studios, or other creators anywhere in prompts submitted to Runway tasks. When style direction is needed, translate it into camera language, palette, lighting, genre, era, composition, and material texture.\n\nUSER-FACING REPLIES: DO NOT mention internal model names unless asked. Do not mention the intermediate storyboard unless useful. Tell the user the creative ad video is generating and the viewer will update when ready.",
"inputSchema": {
"type": "object",
"required": [
"promptText"
],
"properties": {
"format": {
"enum": [
"creative_ad_video"
],
"type": "string",
"description": "Only supported value in v1: creative_ad_video."
},
"duration": {
"enum": [
10,
15
],
"type": "number",
"description": "Final video duration. Default 10. Use 15 only when the user asks for longer."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"productUrl": {
"type": "string",
"description": "Product landing page URL. Used only if productImages is empty."
},
"promptText": {
"type": "string",
"description": "Required creative ad idea. If the user requests a named artist/director/photographer/studio style, describe the visual qualities instead and do not include the name."
},
"productImages": {
"type": "array",
"items": {
"type": "object",
"required": [
"url"
],
"properties": {
"tag": {
"type": "string"
},
"url": {
"type": "string"
}
},
"additionalProperties": false
},
"description": "Product image URLs [{url, tag}]. Preferred over productUrl extraction."
},
"productMetadata": {
"type": "object",
"properties": {
"url": {
"type": "string"
},
"image": {
"type": "object",
"properties": {
"alt": {
"type": "string"
},
"url": {
"type": "string"
},
"score": {
"type": "number"
},
"width": {
"type": "number"
},
"height": {
"type": "number"
},
"source": {
"type": "string"
}
},
"description": "Backward-compatible continuation metadata from older storyboard responses. Ignored by the handler; productImages[0] is the canonical product image.",
"additionalProperties": false
},
"title": {
"type": "string"
},
"description": {
"type": "string"
}
},
"description": "Internal continuation only: product URL metadata.",
"additionalProperties": false
},
"referenceImages": {
"type": "array",
"items": {
"type": "object",
"required": [
"url"
],
"properties": {
"tag": {
"type": "string"
},
"url": {
"type": "string"
}
},
"additionalProperties": false
},
"description": "Optional character, scene, mood, or world reference image URLs [{url, tag}]."
},
"storyboardImage": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string"
}
},
"description": "Internal continuation only: completed storyboard image from the first workflow stage.",
"additionalProperties": false
},
"productImageFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. image/png."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded product image. Prefer productImages when you already have a public or Runway-hosted URL.",
"additionalProperties": false
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Generate Product Marketing Video",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description95/100
Generate OR edit an image using a Runway-hosted image model. This is the only image tool — there is no separate "edit_image" tool. Pass the source image as `referenceImages[0]` to edit it.
MODEL STATUS: Call `whoami` before choosing a model. Check `modelDeprecations` for upcoming sunsets and unavailable task variants; warn briefly about a relevant upcoming sunset, and ask before switching models. Model enums describe supported inputs, not current availability.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
For creative product ad videos from a product URL or product image, use `generate_product_marketing_video` instead. It handles product-image extraction, storyboard generation, and final video creation. If product URL extraction fails, do NOT use this tool to synthesize the product from a text description unless the user explicitly agrees to a stand-in; ask for the real product image or screenshot so the ad uses the actual SKU as reference.
For video generation/editing, use `generate_video` (single shot) or `generate_multishot_video` (3-5 connected scenes).
To translate an existing ad's on-screen copy into another language while keeping the same creative, use `localize_ad` — it translates the text and preserves the layout, which prompting an edit here does not do reliably.
MODES:
1. Text-to-image — pass `promptText` only. The model creates a new image from scratch. Example: "a corgi astronaut floating in a nebula, cinematic lighting".
2. Image edit / transform — pass the source image as `referenceImages[0]` (give it a tag like "input" or "cat") and describe the change in `promptText`. Example: `referenceImages: [{url: ..., tag: "cat"}], promptText: "@cat on a beach at sunset"`. Use for: background swaps, style transfer, object additions/removals, restyling, photo retouching.
3. Composite from multiple references — pass 2+ `referenceImages` each with a distinct tag, then reference them by tag in the prompt. Example: `referenceImages: [{url: ..., tag: "person"}, {url: ..., tag: "room"}], promptText: "@person standing in @room"`.
HANDLING USER ATTACHMENTS:
ChatGPT: if it supplies an uploaded image as `referenceImageFile`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as `referenceImages[0].url`.
Claude/Cursor/local agents: if the user attached an image, run `init_upload` -> curl -> `complete_upload` first to get a Runway-hosted asset URL, then pass it as `referenceImages[].url`.
REUSING ASSETS: URLs returned by other Runway tools (image outputs from prior `generate_image` calls, `complete_upload` outputs) are stable, hosted asset URLs. Pass them directly into `referenceImages[].url` — never re-upload. COPY THESE URLS VERBATIM. They end in a long `_jwt=` authorization token that is meaningless if even one character changes. Copy the whole string exactly as it appeared in the earlier tool result; never retype it, truncate it, shorten it for readability, or reconstruct it from the parts you remember. If you cannot reproduce the URL exactly, call `init_upload` and use the URL it returns instead.
EDIT BUTTON (viewer → chat): when a user message says "Edit this image" and names a Runway task ID, the user clicked Edit on a finished image in the inline viewer. Follow this two-step flow:
1. Do NOT generate yet. Reply in one or two sentences: confirm you can edit that image (describe it briefly if you know its prompt) and ask what they want changed — background, style, objects, colors, text.
2. Once they answer, call this tool with `referenceImages: [{ taskId: "<that task ID>", tag: "input" }]` (never retype the image URL), `promptText` describing the change (e.g. "@input at sunset"), and the same ratio as the source image when you know it. Leave `model` unset (the default is the best editing model).
MULTIPLE IMAGES / COUNT:
When the user asks for multiple images, options, variations, or directions, use `count` for the number of separate images. Rewrite `promptText` so it describes ONE standalone image only. Do NOT include phrases like "3 images", "three options", "multiple versions", "grid", "collage", "contact sheet", "side-by-side", or "set of images" in `promptText` unless the user explicitly wants a single image containing a grid/collage. For product variations, prompt for one clean standalone product composition and let `count` create separate tasks, e.g. "single standalone product image of @product in a premium studio hero composition, not a collage or grid".
If the user wants deliberately different concepts (for example studio hero, ecommerce packshot, and lifestyle scene), make separate `generate_image` calls with `count: 1` and a tailored one-image prompt for each concept.
STYLE SAFETY: Do not include names of artists, directors, photographers, studios, or other living creators as style anchors. If the user asks for a named style, translate it into neutral visual descriptors such as camera language, palette, lighting, genre, era, composition, and material texture before calling the tool.
USER-FACING REPLIES: Pick the right model internally, but DO NOT mention the model name to the user (e.g. "using Nano Banana Pro", "with GPT Image 2") unless they explicitly ask which model was used. Talk about the image content — subject, style, composition — not the engine.
MODEL NAMES (a user may name a model by its product name; pass the id on the left):
- `nano-banana-pro` = Nano Banana Pro
- `nano-banana-2.1` = Nano Banana 2.1
- `nano-banana-2` = Nano Banana 2
- `nano-banana-2-lite` = Nano Banana 2 Lite
- `gpt-image-2` = GPT Image 2
- `gpt-image-2.5-flare` = GPT Image 2.5 Flare
- `gpt-image-2.5-sunburst` = GPT Image 2.5 Sunburst
- `seedream-5` = Seedream 5.0
- `seedream-5-pro` = Seedream 5.0 Pro
- `seedream-5-flash` = Seedream 5.0 Flash
- `ideogram-4` = Ideogram 4.0
- `ideogram-4.5` = Ideogram 4.5
- `ideogram-4.5-precise-edit` = Ideogram 4.5 Precise Edit
- `grok-imagine-image-2` = Grok Imagine Image 2
- `muse-image` = Muse Image
- `fibo-1.5` = FIBO 1.5
- `fibo-edit-1.5` = FIBO Edit 1.5
- `gen-4` = Gen-4
- `gen-4-image-turbo` = Gen-4 Turbo Image
MODELS:
| model | tier | best for |
|--------------------|--------|-----------------------------------------------------------------------------------------|
| nano-banana-pro | any | DEFAULT except ChatGPT. Photo-real images, character consistency, edits/composites with references. |
| nano-banana-2.1 | any | Newest flash Nano Banana. Close prompt adherence, precise edits, up to 14 references, 1K-4K. |
| nano-banana-2 | any | Faster, lower-cost Gemini flash model for photo-real images and edits. |
| nano-banana-2-lite | any | Cheapest/fastest Gemini model. Great for drafts, iterations, and high-volume 1K images. |
| gpt-image-2 | paid | Readable text in images, charts/infographics, strict adherence to complex instructions. |
| gpt-image-2.5-flare | paid | DEFAULT on ChatGPT. Faster GPT Image 2.5 for everyday iteration; in-image text, 1K-4K, up to 16 references. |
| gpt-image-2.5-sunburst | paid | Highest-fidelity GPT Image 2.5 for precision edits; slower, same controls as Flare. |
| seedream-5 | any | Many-reference composites (up to 14 reference images), low-cost 2K/3K output. |
| seedream-5-pro | any | Higher-quality Seedream for reference-guided product/style work. Max 10 references. |
| seedream-5-flash | paid | Fastest, lowest-cost Seedream (2 credits/image). 1K/2K, up to 10 references. |
| ideogram-4 | paid | Ideogram 4.0. Text-to-image, or remix from a single reference image (max 1). |
| ideogram-4.5 | paid | Ideogram 4.5. Text-to-image or reference-guided generation from up to 5 references, 1K/2K. |
| ideogram-4.5-precise-edit | paid | Targeted edits that keep the rest of the image intact. REQUIRES the source as referenceImages[0]. |
| grok-imagine-image-2 | paid | Grok Imagine Image 2. Text-to-image or edits/composites from up to 3 references. |
| muse-image | any | Meta Muse Image. Very-low-cost generation or edits/composites with up to 10 references. |
| fibo-1.5 | paid | Bria FIBO 1.5, trained on licensed data. Text-to-image or reinterpret 1 reference, 1MP/4MP. |
| fibo-edit-1.5 | paid | Bria FIBO Edit 1.5. Instruction edits. REQUIRES 1-4 reference images. |
| gen-4 | any | Fast, low-cost general-purpose Runway model. |
| gen-4-image-turbo | any | Cheapest tier for rapid iteration on edits/composites. REQUIRES 1-3 reference images. |
Picking heuristic:
- nano-banana-pro (DEFAULT except ChatGPT) — photo-realistic images, edits, multi-image composites; highest quality of the Gemini image models.
- nano-banana-2.1 — pick when speed/cost matter more than the extra polish of Pro and the brief must be followed closely: reference-guided edits, complex multi-part prompts, or subject consistency across up to 14 references. Prefer it over nano-banana-2; it costs less (4/6/12 credits per image at 1K/2K/4K). Pro still leads on final-grade polish.
- nano-banana-2 — the previous flash model, for photo-real generation and edits. Pick it only when the user asks for it by name.
- nano-banana-2-lite — pick for the cheapest/fastest option: quick drafts, rapid iteration, or high-volume generation where 1K output is fine.
- gpt-image-2 — pick when the image needs readable text, charts, infographics, or strict adherence to a long instruction list.
- gpt-image-2.5-flare (DEFAULT on ChatGPT) — pick for iterating on text-heavy or instruction-heavy images: it is the fast GPT Image 2.5 variant (~40s) with 1K-4K output and up to 16 references. On ChatGPT, omit `model` (or pass this id) unless the user names another model. Availability is account-gated: on an unknown-task-type error, fall back to gpt-image-2 or nano-banana-pro.
- gpt-image-2.5-sunburst — same controls as Flare, tuned for precision editing and the highest fidelity. Pick when final quality matters more than latency (~120s, same credit cost as Flare at a given tier). Same account gate and fallback as Flare.
- seedream-5 — pick when a composite needs MANY references (it accepts up to 14 reference images vs 2-3 elsewhere), or for cheap high-volume 2K generation.
- seedream-5-pro — pick over seedream-5 when quality matters more than cost on reference-heavy product shots, style/color/sketch references, or multilingual in-image text. Costs more than seedream-5 and caps at 10 references with no 3K tier. Less predictable on very specific fine-detail instructions; prefer gpt-image-2 when English in-image text fidelity is paramount.
- seedream-5-flash — pick for fast, cheap Seedream drafts or high-volume output (2 credits/image at 1K or 2K), including 21:9 at up to 2K, with up to 10 references. It has no grounding, so use seedream-5 or seedream-5-pro for prompts about current products or events. Availability is account-gated: on an unknown-task-type error, fall back to seedream-5 or nano-banana-pro.
- ideogram-4 — Ideogram 4.0. Text-to-image, or "remix" from a single reference image (max 1). Pick when the user asks for it by name.
- ideogram-4.5 — Ideogram 4.5. Text-to-image, or reference-guided generation from up to 5 references, at 1K or 2K. Pick when the user asks for Ideogram by name (prefer it over ideogram-4 unless they name 4.0). Costs about 25 credits/image for text-to-image and 8 with references. Availability is account-gated: on an unknown-task-type error, fall back to ideogram-4 or nano-banana-pro.
- ideogram-4.5-precise-edit — Ideogram 4.5's precise edit mode. Pick for surgical changes to an existing image where everything else must stay untouched: swap or fix in-image text, change one object's color or material, remove or add a single element, retouch a detail. Put the image being edited in referenceImages[0]; up to 4 more references can show what to add (e.g. a logo or product). Output keeps the source image's dimensions, so ratio and imageSize are ignored. Costs about 25 credits/image. Do NOT use it for restyles, new scenes, or prompt-only generation (use ideogram-4.5 or nano-banana-pro). Same account gate as ideogram-4.5: on an unknown-task-type error, fall back to nano-banana-pro.
- grok-imagine-image-2 — xAI's Grok Imagine Image 2. Text-to-image, single-image edits, or composites from up to 3 tagged references. Prompts cap at 2500 characters (shorter than every other model here). Pick when the user asks for Grok by name, or wants its extra-wide/tall ratios (1:2, 2:1, 9:20, 20:9, 9:19.5, 19.5:9). Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-pro.
- muse-image — Meta's very-low-cost option (1 credit/image) for drafts, batch exploration, or edits/composites using up to 10 references. It has one fixed resolution around 2.4MP and is weaker on dense photorealistic scenes and fine texture. Reference tags are not supported, so describe reference roles in the prompt by order instead of using @tags. Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-pro.
- fibo-1.5 — Bria FIBO 1.5, trained entirely on licensed data. Text-to-image, or a reinterpretation of a single reference image (max 1). imageSize 1K (1MP, 5 credits/image) or 4K (4MP, 10 credits/image). Pick when the user asks for FIBO or Bria by name, or wants a licensed-data model. Reference tags are ignored. Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-pro.
- fibo-edit-1.5 — Bria FIBO Edit 1.5. Edits 1-4 source images from an instruction in promptText (e.g. "change the bicycle to blue"); 5 credits/image. REQUIRES referenceImages: put the image being edited first. Reference tags are ignored, so refer to images by order. If the user asks for a FIBO edit without an image, ask for one instead of calling the tool. Use fibo-1.5 for prompt-only generation. Same account gate as fibo-1.5.
- gen-4 — pick for speed, or when the user explicitly wants to continue with the current account settings.
- gen-4-image-turbo — pick as the lowest-cost option for RAPID iteration/exploration on reference-based edits and composites (cheapest tier). It ONLY works with 1-3 reference images — it has no text-to-image mode, so never use it for prompt-only generation (use gen-4 or nano-banana-pro for that).
ASPECT RATIOS:
- nano-banana-pro: auto, 21:9, 16:9, 4:3, 1:1, 3:4, 9:16
- nano-banana-2.1: auto, 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, 9:21, 21:9, 1:4, 4:1, 1:8, 8:1
- nano-banana-2: auto, 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, 21:9, 1:4, 4:1, 1:8, 8:1
- nano-banana-2-lite: auto, 1:1, 3:2, 2:3, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, 21:9
- gpt-image-2: 21:9, 16:9, 3:2, 4:3, 5:4, 1:1, 4:5, 3:4, 2:3, 9:16
- gpt-image-2.5-flare, gpt-image-2.5-sunburst: 21:9, 16:9, 3:2, 4:3, 5:4, 1:1, 4:5, 3:4, 2:3, 9:16
- seedream-5: 1:1, 4:3, 3:4, 16:9, 9:16, 3:2, 2:3, 21:9
- seedream-5-pro: auto, 1:1, 4:3, 3:4, 16:9, 9:16, 3:2, 2:3
- seedream-5-flash: auto, 1:1, 4:3, 3:4, 16:9, 9:16, 3:2, 2:3, 21:9
- ideogram-4: 1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3, 21:9, 9:21
- ideogram-4.5: auto, 1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3, 5:4, 4:5, 2:1, 1:2
- ideogram-4.5-precise-edit: keeps the source image's aspect ratio; ratio is ignored.
- grok-imagine-image-2: auto, 1:1, 3:4, 4:3, 9:16, 16:9, 2:3, 3:2, 9:19.5, 19.5:9, 9:20, 20:9, 1:2, 2:1
- muse-image: auto, 21:9, 16:9, 3:2, 4:3, 1:1, 3:4, 2:3, 9:16
- fibo-1.5: 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9
- fibo-edit-1.5: auto (default, follows the source), or with 2-4 reference images only: 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9
- gen-4: 16:9, 9:16, 1:1, 4:3, 3:4, 21:9
- gen-4-image-turbo: 16:9, 9:16, 1:1, 4:3, 3:4, 21:9
IMAGE SIZE (resolution):
- nano-banana-pro, nano-banana-2.1, nano-banana-2: 1K (default), 2K, 4K. Higher tiers cost more credits and take longer.
- gpt-image-2.5-flare, gpt-image-2.5-sunburst: 1K (default), 2K, 4K. 2K is the same credit cost as 1K; only 4K costs more.
- seedream-5: 2K (default) or 3K. 3K is the same credit cost as 2K.
- seedream-5-pro: 1K or 2K (default). 2K costs roughly twice 1K.
- seedream-5-flash: 1K or 2K (default), at the same credit cost.
- grok-imagine-image-2: 1K (default) or 2K.
- ideogram-4.5: 1K or 2K (default). Ignored when ratio is auto.
- fibo-1.5: 1K (1MP, default) or 4K (4MP). 4K costs twice 1K.
- nano-banana-2-lite, gpt-image-2, muse-image, gen-4, gen-4-image-turbo, ideogram-4, ideogram-4.5-precise-edit, fibo-edit-1.5: not configurable — the imageSize parameter is ignored (nano-banana-2-lite outputs at 1K only; muse-image outputs around 2.4MP; gen-4-image-turbo outputs at 1080p; ideogram-4.5-precise-edit keeps the source size).
Parameters:
- model: One of `nano-banana-pro`, `nano-banana-2.1`, `nano-banana-2`, `nano-banana-2-lite`, `gpt-image-2`, `gpt-image-2.5-flare`, `gpt-image-2.5-sunburst`, `seedream-5`, `seedream-5-pro`, `seedream-5-flash`, `ideogram-4`, `ideogram-4.5`, `ideogram-4.5-precise-edit`, `grok-imagine-image-2`, `muse-image`, `fibo-1.5`, `fibo-edit-1.5`, `gen-4`, `gen-4-image-turbo`. Default: `nano-banana-pro`. On ChatGPT: `gpt-image-2.5-flare`.
- promptText: Required. Descriptive prompt for ONE standalone output image. For edits, describe the change ("make the background neon"). For composites, reference tagged images ("@cat sitting on @couch"). If `count` > 1, remove the requested quantity from this prompt and do not ask for a grid/collage/contact sheet.
- ratio: Aspect ratio. See per-model list above.
- imageSize: Output resolution tier. nano-banana-pro, nano-banana-2.1, and nano-banana-2: `1K` (default), `2K`, `4K` — higher tiers cost more credits and take longer. gpt-image-2.5-flare and gpt-image-2.5-sunburst: `1K` (default), `2K` (same cost as 1K), `4K` (costs more). seedream-5: `2K` (default) or `3K` at the same cost. seedream-5-pro: `1K` or `2K` (default), where 2K costs about twice 1K. seedream-5-flash: `1K` or `2K` (default) at the same cost. grok-imagine-image-2: `1K` (default) or `2K`. ideogram-4.5: `1K` or `2K` (default), ignored when ratio is auto. fibo-1.5: `1K` (1MP, default) or `4K` (4MP). Ignored by nano-banana-2-lite (1K only), gpt-image-2, muse-image (about 2.4MP), gen-4, and gen-4-image-turbo (1080p).
- grounding: Optional boolean, default false. When true the model runs a live web search while generating, so it can pull in current brand, product, trend, or event details it would otherwise have to guess at (e.g. "this season's packaging", "the current team kit"). ONLY `seedream-5` and `seedream-5-pro` support it (not `seedream-5-flash`) — on any other model the flag is dropped, so switch to a Seedream model when the user asks for something current. Leave it off for ordinary prompts: it adds latency and makes output less deterministic.
- transparentBackground: Optional boolean, default false. When true the output is a PNG with a transparent (alpha) background — use it for logos, stickers, icons, product cutouts, and other assets meant to be layered over something else. Describe only the subject in `promptText` (e.g. "a red fox mascot logo"), not a background. ONLY `seedream-5-pro`, `seedream-5-flash`, `gpt-image-2`, and `gpt-image-2.5-sunburst` support it. gpt-image-2.5-flare does NOT — pick gpt-image-2.5-sunburst or gpt-image-2 instead. If `model` is omitted, the server picks a supporting model (gpt-image-2 on paid plans, seedream-5-pro on free plans); explicitly naming any other model returns an error. seedream-5-pro and seedream-5-flash allow at most 1 reference image with this flag. To cut the background out of an EXISTING image, use `remove_image_background` instead.
- count: Number of separate image tasks to generate in one go. Options: 1, 2, 3, 4. Default: 1. This is NOT a request for one output image containing multiple panels.
- referenceImages: Optional array of `{url | taskId, tag}` — for edits and composites. `tag` is the alias used in the prompt (e.g. `tag: "cat"` lets you write "@cat" in promptText), except muse-image, fibo-1.5, and fibo-edit-1.5 ignore tags and use reference order. Prefer `taskId` (the Runway task ID of a finished image) whenever the image came from a Runway tool; the server resolves the URL. Otherwise URLs come from `complete_upload` or any public https URL. REQUIRED (1-3) for gen-4-image-turbo, which has no text-to-image mode. REQUIRED for ideogram-4.5-precise-edit, where referenceImages[0] is the image being edited and up to 4 more are references. REQUIRED (1-4) for fibo-edit-1.5. ideogram-4 and fibo-1.5 accept at most 1 reference image; grok-imagine-image-2 accepts at most 3; ideogram-4.5 at most 5; muse-image, seedream-5-pro, and seedream-5-flash accept at most 10; seedream-5 and nano-banana-2.1 at most 14; gpt-image-2.5-flare and gpt-image-2.5-sunburst accept at most 16.
The description explains the capability, when to use it, and what to expect.
Inputs52/100
9 of 15 inputs have descriptions. Add descriptions to the missing inputs, including where to obtain IDs and how to supply values.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
model
Image model id. Default: nano-banana-pro. On ChatGPT: gpt-image-2.5-flare.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
promptText
Text prompt for ONE standalone image. If count > 1, keep the prompt about a single output, not a grid/collage/contact sheet. If the user requests a named artist/director/photographer/studio style, describe the visual qualities instead and do not include the name.
Clear enough to supply this argument.
ratio
Aspect ratio. Supported values vary by model — see model description.
Explain the meaning of this value, not just its name.
imageSize
Output resolution tier. Gemini models (nano-banana-pro, nano-banana-2.1, nano-banana-2): 1K (default), 2K, 4K. gpt-image-2.5-flare and gpt-image-2.5-sunburst: 1K (default), 2K, 4K. seedream-5: 2K (default) or 3K. seedream-5-pro: 1K or 2K (default). grok-imagine-image-2: 1K (default) or 2K. ideogram-4.5: 1K or 2K (default). fibo-1.5: 1K (default) or 4K. Ignored by other models.
Clear enough to supply this argument.
grounding
Let the model run a live web search while generating, so it can use current brand, product, trend, or event details. Only seedream-5 and seedream-5-pro support this; ignored by other models. Default false.
Clear enough to supply this argument.
transparentBackground
Output a PNG with a transparent background (logos, stickers, icons, cutouts). Only seedream-5-pro, seedream-5-flash, gpt-image-2, and gpt-image-2.5-sunburst support this (not gpt-image-2.5-flare). With model omitted the server picks a supporting model; naming another model returns an error. Default false.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
count
Number of separate image tasks to generate in one tool call. Options: 1, 2, 3, 4. Default: 1.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
referenceImages › items › taskId
Runway task ID of a finished image generation. The server resolves the image URL itself — use this instead of copying a long signed URL.
Clear enough to supply this argument.
referenceImages › items › url
Not provided
Missing description. Explain what this value means and how to supply it.
referenceImages › items › tag
Not provided
Missing description. Explain what this value means and how to supply it.
referenceImageFile › download_url
Not provided
Missing description. Explain what this value means and how to supply it.
referenceImageFile › file_id
Not provided
Missing description. Explain what this value means and how to supply it.
referenceImageFile › mime_type
Not provided
Missing description. Explain what this value means and how to supply it.
referenceImageFile › file_name
Not provided
Missing description. Explain what this value means and how to supply it.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "generate_image",
"title": "Generate Image",
"description": "Generate OR edit an image using a Runway-hosted image model. This is the only image tool — there is no separate \"edit_image\" tool. Pass the source image as `referenceImages[0]` to edit it.\n\nMODEL STATUS: Call `whoami` before choosing a model. Check `modelDeprecations` for upcoming sunsets and unavailable task variants; warn briefly about a relevant upcoming sunset, and ask before switching models. Model enums describe supported inputs, not current availability.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nFor creative product ad videos from a product URL or product image, use `generate_product_marketing_video` instead. It handles product-image extraction, storyboard generation, and final video creation. If product URL extraction fails, do NOT use this tool to synthesize the product from a text description unless the user explicitly agrees to a stand-in; ask for the real product image or screenshot so the ad uses the actual SKU as reference.\n\nFor video generation/editing, use `generate_video` (single shot) or `generate_multishot_video` (3-5 connected scenes).\n\nTo translate an existing ad's on-screen copy into another language while keeping the same creative, use `localize_ad` — it translates the text and preserves the layout, which prompting an edit here does not do reliably.\n\nMODES:\n1. Text-to-image — pass `promptText` only. The model creates a new image from scratch. Example: \"a corgi astronaut floating in a nebula, cinematic lighting\".\n2. Image edit / transform — pass the source image as `referenceImages[0]` (give it a tag like \"input\" or \"cat\") and describe the change in `promptText`. Example: `referenceImages: [{url: ..., tag: \"cat\"}], promptText: \"@cat on a beach at sunset\"`. Use for: background swaps, style transfer, object additions/removals, restyling, photo retouching.\n3. Composite from multiple references — pass 2+ `referenceImages` each with a distinct tag, then reference them by tag in the prompt. Example: `referenceImages: [{url: ..., tag: \"person\"}, {url: ..., tag: \"room\"}], promptText: \"@person standing in @room\"`.\n\nHANDLING USER ATTACHMENTS:\nChatGPT: if it supplies an uploaded image as `referenceImageFile`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as `referenceImages[0].url`.\nClaude/Cursor/local agents: if the user attached an image, run `init_upload` -> curl -> `complete_upload` first to get a Runway-hosted asset URL, then pass it as `referenceImages[].url`.\n\nREUSING ASSETS: URLs returned by other Runway tools (image outputs from prior `generate_image` calls, `complete_upload` outputs) are stable, hosted asset URLs. Pass them directly into `referenceImages[].url` — never re-upload. COPY THESE URLS VERBATIM. They end in a long `_jwt=` authorization token that is meaningless if even one character changes. Copy the whole string exactly as it appeared in the earlier tool result; never retype it, truncate it, shorten it for readability, or reconstruct it from the parts you remember. If you cannot reproduce the URL exactly, call `init_upload` and use the URL it returns instead.\n\nEDIT BUTTON (viewer → chat): when a user message says \"Edit this image\" and names a Runway task ID, the user clicked Edit on a finished image in the inline viewer. Follow this two-step flow:\n1. Do NOT generate yet. Reply in one or two sentences: confirm you can edit that image (describe it briefly if you know its prompt) and ask what they want changed — background, style, objects, colors, text.\n2. Once they answer, call this tool with `referenceImages: [{ taskId: \"<that task ID>\", tag: \"input\" }]` (never retype the image URL), `promptText` describing the change (e.g. \"@input at sunset\"), and the same ratio as the source image when you know it. Leave `model` unset (the default is the best editing model).\n\nMULTIPLE IMAGES / COUNT:\nWhen the user asks for multiple images, options, variations, or directions, use `count` for the number of separate images. Rewrite `promptText` so it describes ONE standalone image only. Do NOT include phrases like \"3 images\", \"three options\", \"multiple versions\", \"grid\", \"collage\", \"contact sheet\", \"side-by-side\", or \"set of images\" in `promptText` unless the user explicitly wants a single image containing a grid/collage. For product variations, prompt for one clean standalone product composition and let `count` create separate tasks, e.g. \"single standalone product image of @product in a premium studio hero composition, not a collage or grid\".\nIf the user wants deliberately different concepts (for example studio hero, ecommerce packshot, and lifestyle scene), make separate `generate_image` calls with `count: 1` and a tailored one-image prompt for each concept.\n\nSTYLE SAFETY: Do not include names of artists, directors, photographers, studios, or other living creators as style anchors. If the user asks for a named style, translate it into neutral visual descriptors such as camera language, palette, lighting, genre, era, composition, and material texture before calling the tool.\n\nUSER-FACING REPLIES: Pick the right model internally, but DO NOT mention the model name to the user (e.g. \"using Nano Banana Pro\", \"with GPT Image 2\") unless they explicitly ask which model was used. Talk about the image content — subject, style, composition — not the engine.\n\nMODEL NAMES (a user may name a model by its product name; pass the id on the left):\n- `nano-banana-pro` = Nano Banana Pro\n- `nano-banana-2.1` = Nano Banana 2.1\n- `nano-banana-2` = Nano Banana 2\n- `nano-banana-2-lite` = Nano Banana 2 Lite\n- `gpt-image-2` = GPT Image 2\n- `gpt-image-2.5-flare` = GPT Image 2.5 Flare\n- `gpt-image-2.5-sunburst` = GPT Image 2.5 Sunburst\n- `seedream-5` = Seedream 5.0\n- `seedream-5-pro` = Seedream 5.0 Pro\n- `seedream-5-flash` = Seedream 5.0 Flash\n- `ideogram-4` = Ideogram 4.0\n- `ideogram-4.5` = Ideogram 4.5\n- `ideogram-4.5-precise-edit` = Ideogram 4.5 Precise Edit\n- `grok-imagine-image-2` = Grok Imagine Image 2\n- `muse-image` = Muse Image\n- `fibo-1.5` = FIBO 1.5\n- `fibo-edit-1.5` = FIBO Edit 1.5\n- `gen-4` = Gen-4\n- `gen-4-image-turbo` = Gen-4 Turbo Image\n\nMODELS:\n| model | tier | best for |\n|--------------------|--------|-----------------------------------------------------------------------------------------|\n| nano-banana-pro | any | DEFAULT except ChatGPT. Photo-real images, character consistency, edits/composites with references. |\n| nano-banana-2.1 | any | Newest flash Nano Banana. Close prompt adherence, precise edits, up to 14 references, 1K-4K. |\n| nano-banana-2 | any | Faster, lower-cost Gemini flash model for photo-real images and edits. |\n| nano-banana-2-lite | any | Cheapest/fastest Gemini model. Great for drafts, iterations, and high-volume 1K images. |\n| gpt-image-2 | paid | Readable text in images, charts/infographics, strict adherence to complex instructions. |\n| gpt-image-2.5-flare | paid | DEFAULT on ChatGPT. Faster GPT Image 2.5 for everyday iteration; in-image text, 1K-4K, up to 16 references. |\n| gpt-image-2.5-sunburst | paid | Highest-fidelity GPT Image 2.5 for precision edits; slower, same controls as Flare. |\n| seedream-5 | any | Many-reference composites (up to 14 reference images), low-cost 2K/3K output. |\n| seedream-5-pro | any | Higher-quality Seedream for reference-guided product/style work. Max 10 references. |\n| seedream-5-flash | paid | Fastest, lowest-cost Seedream (2 credits/image). 1K/2K, up to 10 references. |\n| ideogram-4 | paid | Ideogram 4.0. Text-to-image, or remix from a single reference image (max 1). |\n| ideogram-4.5 | paid | Ideogram 4.5. Text-to-image or reference-guided generation from up to 5 references, 1K/2K. |\n| ideogram-4.5-precise-edit | paid | Targeted edits that keep the rest of the image intact. REQUIRES the source as referenceImages[0]. |\n| grok-imagine-image-2 | paid | Grok Imagine Image 2. Text-to-image or edits/composites from up to 3 references. |\n| muse-image | any | Meta Muse Image. Very-low-cost generation or edits/composites with up to 10 references. |\n| fibo-1.5 | paid | Bria FIBO 1.5, trained on licensed data. Text-to-image or reinterpret 1 reference, 1MP/4MP. |\n| fibo-edit-1.5 | paid | Bria FIBO Edit 1.5. Instruction edits. REQUIRES 1-4 reference images. |\n| gen-4 | any | Fast, low-cost general-purpose Runway model. |\n| gen-4-image-turbo | any | Cheapest tier for rapid iteration on edits/composites. REQUIRES 1-3 reference images. |\n\nPicking heuristic:\n- nano-banana-pro (DEFAULT except ChatGPT) — photo-realistic images, edits, multi-image composites; highest quality of the Gemini image models.\n- nano-banana-2.1 — pick when speed/cost matter more than the extra polish of Pro and the brief must be followed closely: reference-guided edits, complex multi-part prompts, or subject consistency across up to 14 references. Prefer it over nano-banana-2; it costs less (4/6/12 credits per image at 1K/2K/4K). Pro still leads on final-grade polish.\n- nano-banana-2 — the previous flash model, for photo-real generation and edits. Pick it only when the user asks for it by name.\n- nano-banana-2-lite — pick for the cheapest/fastest option: quick drafts, rapid iteration, or high-volume generation where 1K output is fine.\n- gpt-image-2 — pick when the image needs readable text, charts, infographics, or strict adherence to a long instruction list.\n- gpt-image-2.5-flare (DEFAULT on ChatGPT) — pick for iterating on text-heavy or instruction-heavy images: it is the fast GPT Image 2.5 variant (~40s) with 1K-4K output and up to 16 references. On ChatGPT, omit `model` (or pass this id) unless the user names another model. Availability is account-gated: on an unknown-task-type error, fall back to gpt-image-2 or nano-banana-pro.\n- gpt-image-2.5-sunburst — same controls as Flare, tuned for precision editing and the highest fidelity. Pick when final quality matters more than latency (~120s, same credit cost as Flare at a given tier). Same account gate and fallback as Flare.\n- seedream-5 — pick when a composite needs MANY references (it accepts up to 14 reference images vs 2-3 elsewhere), or for cheap high-volume 2K generation.\n- seedream-5-pro — pick over seedream-5 when quality matters more than cost on reference-heavy product shots, style/color/sketch references, or multilingual in-image text. Costs more than seedream-5 and caps at 10 references with no 3K tier. Less predictable on very specific fine-detail instructions; prefer gpt-image-2 when English in-image text fidelity is paramount.\n- seedream-5-flash — pick for fast, cheap Seedream drafts or high-volume output (2 credits/image at 1K or 2K), including 21:9 at up to 2K, with up to 10 references. It has no grounding, so use seedream-5 or seedream-5-pro for prompts about current products or events. Availability is account-gated: on an unknown-task-type error, fall back to seedream-5 or nano-banana-pro.\n- ideogram-4 — Ideogram 4.0. Text-to-image, or \"remix\" from a single reference image (max 1). Pick when the user asks for it by name.\n- ideogram-4.5 — Ideogram 4.5. Text-to-image, or reference-guided generation from up to 5 references, at 1K or 2K. Pick when the user asks for Ideogram by name (prefer it over ideogram-4 unless they name 4.0). Costs about 25 credits/image for text-to-image and 8 with references. Availability is account-gated: on an unknown-task-type error, fall back to ideogram-4 or nano-banana-pro.\n- ideogram-4.5-precise-edit — Ideogram 4.5's precise edit mode. Pick for surgical changes to an existing image where everything else must stay untouched: swap or fix in-image text, change one object's color or material, remove or add a single element, retouch a detail. Put the image being edited in referenceImages[0]; up to 4 more references can show what to add (e.g. a logo or product). Output keeps the source image's dimensions, so ratio and imageSize are ignored. Costs about 25 credits/image. Do NOT use it for restyles, new scenes, or prompt-only generation (use ideogram-4.5 or nano-banana-pro). Same account gate as ideogram-4.5: on an unknown-task-type error, fall back to nano-banana-pro.\n- grok-imagine-image-2 — xAI's Grok Imagine Image 2. Text-to-image, single-image edits, or composites from up to 3 tagged references. Prompts cap at 2500 characters (shorter than every other model here). Pick when the user asks for Grok by name, or wants its extra-wide/tall ratios (1:2, 2:1, 9:20, 20:9, 9:19.5, 19.5:9). Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-pro.\n- muse-image — Meta's very-low-cost option (1 credit/image) for drafts, batch exploration, or edits/composites using up to 10 references. It has one fixed resolution around 2.4MP and is weaker on dense photorealistic scenes and fine texture. Reference tags are not supported, so describe reference roles in the prompt by order instead of using @tags. Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-pro.\n- fibo-1.5 — Bria FIBO 1.5, trained entirely on licensed data. Text-to-image, or a reinterpretation of a single reference image (max 1). imageSize 1K (1MP, 5 credits/image) or 4K (4MP, 10 credits/image). Pick when the user asks for FIBO or Bria by name, or wants a licensed-data model. Reference tags are ignored. Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-pro.\n- fibo-edit-1.5 — Bria FIBO Edit 1.5. Edits 1-4 source images from an instruction in promptText (e.g. \"change the bicycle to blue\"); 5 credits/image. REQUIRES referenceImages: put the image being edited first. Reference tags are ignored, so refer to images by order. If the user asks for a FIBO edit without an image, ask for one instead of calling the tool. Use fibo-1.5 for prompt-only generation. Same account gate as fibo-1.5.\n- gen-4 — pick for speed, or when the user explicitly wants to continue with the current account settings.\n- gen-4-image-turbo — pick as the lowest-cost option for RAPID iteration/exploration on reference-based edits and composites (cheapest tier). It ONLY works with 1-3 reference images — it has no text-to-image mode, so never use it for prompt-only generation (use gen-4 or nano-banana-pro for that).\n\nASPECT RATIOS:\n- nano-banana-pro: auto, 21:9, 16:9, 4:3, 1:1, 3:4, 9:16\n- nano-banana-2.1: auto, 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, 9:21, 21:9, 1:4, 4:1, 1:8, 8:1\n- nano-banana-2: auto, 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, 21:9, 1:4, 4:1, 1:8, 8:1\n- nano-banana-2-lite: auto, 1:1, 3:2, 2:3, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, 21:9\n- gpt-image-2: 21:9, 16:9, 3:2, 4:3, 5:4, 1:1, 4:5, 3:4, 2:3, 9:16\n- gpt-image-2.5-flare, gpt-image-2.5-sunburst: 21:9, 16:9, 3:2, 4:3, 5:4, 1:1, 4:5, 3:4, 2:3, 9:16\n- seedream-5: 1:1, 4:3, 3:4, 16:9, 9:16, 3:2, 2:3, 21:9\n- seedream-5-pro: auto, 1:1, 4:3, 3:4, 16:9, 9:16, 3:2, 2:3\n- seedream-5-flash: auto, 1:1, 4:3, 3:4, 16:9, 9:16, 3:2, 2:3, 21:9\n- ideogram-4: 1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3, 21:9, 9:21\n- ideogram-4.5: auto, 1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3, 5:4, 4:5, 2:1, 1:2\n- ideogram-4.5-precise-edit: keeps the source image's aspect ratio; ratio is ignored.\n- grok-imagine-image-2: auto, 1:1, 3:4, 4:3, 9:16, 16:9, 2:3, 3:2, 9:19.5, 19.5:9, 9:20, 20:9, 1:2, 2:1\n- muse-image: auto, 21:9, 16:9, 3:2, 4:3, 1:1, 3:4, 2:3, 9:16\n- fibo-1.5: 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9\n- fibo-edit-1.5: auto (default, follows the source), or with 2-4 reference images only: 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9\n- gen-4: 16:9, 9:16, 1:1, 4:3, 3:4, 21:9\n- gen-4-image-turbo: 16:9, 9:16, 1:1, 4:3, 3:4, 21:9\n\nIMAGE SIZE (resolution):\n- nano-banana-pro, nano-banana-2.1, nano-banana-2: 1K (default), 2K, 4K. Higher tiers cost more credits and take longer.\n- gpt-image-2.5-flare, gpt-image-2.5-sunburst: 1K (default), 2K, 4K. 2K is the same credit cost as 1K; only 4K costs more.\n- seedream-5: 2K (default) or 3K. 3K is the same credit cost as 2K.\n- seedream-5-pro: 1K or 2K (default). 2K costs roughly twice 1K.\n- seedream-5-flash: 1K or 2K (default), at the same credit cost.\n- grok-imagine-image-2: 1K (default) or 2K.\n- ideogram-4.5: 1K or 2K (default). Ignored when ratio is auto.\n- fibo-1.5: 1K (1MP, default) or 4K (4MP). 4K costs twice 1K.\n- nano-banana-2-lite, gpt-image-2, muse-image, gen-4, gen-4-image-turbo, ideogram-4, ideogram-4.5-precise-edit, fibo-edit-1.5: not configurable — the imageSize parameter is ignored (nano-banana-2-lite outputs at 1K only; muse-image outputs around 2.4MP; gen-4-image-turbo outputs at 1080p; ideogram-4.5-precise-edit keeps the source size).\n\nParameters:\n- model: One of `nano-banana-pro`, `nano-banana-2.1`, `nano-banana-2`, `nano-banana-2-lite`, `gpt-image-2`, `gpt-image-2.5-flare`, `gpt-image-2.5-sunburst`, `seedream-5`, `seedream-5-pro`, `seedream-5-flash`, `ideogram-4`, `ideogram-4.5`, `ideogram-4.5-precise-edit`, `grok-imagine-image-2`, `muse-image`, `fibo-1.5`, `fibo-edit-1.5`, `gen-4`, `gen-4-image-turbo`. Default: `nano-banana-pro`. On ChatGPT: `gpt-image-2.5-flare`.\n- promptText: Required. Descriptive prompt for ONE standalone output image. For edits, describe the change (\"make the background neon\"). For composites, reference tagged images (\"@cat sitting on @couch\"). If `count` > 1, remove the requested quantity from this prompt and do not ask for a grid/collage/contact sheet.\n- ratio: Aspect ratio. See per-model list above.\n- imageSize: Output resolution tier. nano-banana-pro, nano-banana-2.1, and nano-banana-2: `1K` (default), `2K`, `4K` — higher tiers cost more credits and take longer. gpt-image-2.5-flare and gpt-image-2.5-sunburst: `1K` (default), `2K` (same cost as 1K), `4K` (costs more). seedream-5: `2K` (default) or `3K` at the same cost. seedream-5-pro: `1K` or `2K` (default), where 2K costs about twice 1K. seedream-5-flash: `1K` or `2K` (default) at the same cost. grok-imagine-image-2: `1K` (default) or `2K`. ideogram-4.5: `1K` or `2K` (default), ignored when ratio is auto. fibo-1.5: `1K` (1MP, default) or `4K` (4MP). Ignored by nano-banana-2-lite (1K only), gpt-image-2, muse-image (about 2.4MP), gen-4, and gen-4-image-turbo (1080p).\n- grounding: Optional boolean, default false. When true the model runs a live web search while generating, so it can pull in current brand, product, trend, or event details it would otherwise have to guess at (e.g. \"this season's packaging\", \"the current team kit\"). ONLY `seedream-5` and `seedream-5-pro` support it (not `seedream-5-flash`) — on any other model the flag is dropped, so switch to a Seedream model when the user asks for something current. Leave it off for ordinary prompts: it adds latency and makes output less deterministic.\n- transparentBackground: Optional boolean, default false. When true the output is a PNG with a transparent (alpha) background — use it for logos, stickers, icons, product cutouts, and other assets meant to be layered over something else. Describe only the subject in `promptText` (e.g. \"a red fox mascot logo\"), not a background. ONLY `seedream-5-pro`, `seedream-5-flash`, `gpt-image-2`, and `gpt-image-2.5-sunburst` support it. gpt-image-2.5-flare does NOT — pick gpt-image-2.5-sunburst or gpt-image-2 instead. If `model` is omitted, the server picks a supporting model (gpt-image-2 on paid plans, seedream-5-pro on free plans); explicitly naming any other model returns an error. seedream-5-pro and seedream-5-flash allow at most 1 reference image with this flag. To cut the background out of an EXISTING image, use `remove_image_background` instead.\n- count: Number of separate image tasks to generate in one go. Options: 1, 2, 3, 4. Default: 1. This is NOT a request for one output image containing multiple panels.\n- referenceImages: Optional array of `{url | taskId, tag}` — for edits and composites. `tag` is the alias used in the prompt (e.g. `tag: \"cat\"` lets you write \"@cat\" in promptText), except muse-image, fibo-1.5, and fibo-edit-1.5 ignore tags and use reference order. Prefer `taskId` (the Runway task ID of a finished image) whenever the image came from a Runway tool; the server resolves the URL. Otherwise URLs come from `complete_upload` or any public https URL. REQUIRED (1-3) for gen-4-image-turbo, which has no text-to-image mode. REQUIRED for ideogram-4.5-precise-edit, where referenceImages[0] is the image being edited and up to 4 more are references. REQUIRED (1-4) for fibo-edit-1.5. ideogram-4 and fibo-1.5 accept at most 1 reference image; grok-imagine-image-2 accepts at most 3; ideogram-4.5 at most 5; muse-image, seedream-5-pro, and seedream-5-flash accept at most 10; seedream-5 and nano-banana-2.1 at most 14; gpt-image-2.5-flare and gpt-image-2.5-sunburst accept at most 16.",
"inputSchema": {
"type": "object",
"required": [
"promptText"
],
"properties": {
"count": {
"enum": [
1,
2,
3,
4
],
"type": "number",
"description": "Number of separate image tasks to generate in one tool call. Options: 1, 2, 3, 4. Default: 1."
},
"model": {
"enum": [
"nano-banana-pro",
"nano-banana-2.1",
"nano-banana-2",
"nano-banana-2-lite",
"gpt-image-2",
"gpt-image-2.5-flare",
"gpt-image-2.5-sunburst",
"seedream-5",
"seedream-5-pro",
"seedream-5-flash",
"ideogram-4",
"ideogram-4.5",
"ideogram-4.5-precise-edit",
"grok-imagine-image-2",
"muse-image",
"fibo-1.5",
"fibo-edit-1.5",
"gen-4",
"gen-4-image-turbo"
],
"type": "string",
"description": "Image model id. Default: nano-banana-pro. On ChatGPT: gpt-image-2.5-flare."
},
"ratio": {
"type": "string",
"description": "Aspect ratio. Supported values vary by model — see model description."
},
"grounding": {
"type": "boolean",
"description": "Let the model run a live web search while generating, so it can use current brand, product, trend, or event details. Only seedream-5 and seedream-5-pro support this; ignored by other models. Default false."
},
"imageSize": {
"enum": [
"1K",
"2K",
"3K",
"4K"
],
"type": "string",
"description": "Output resolution tier. Gemini models (nano-banana-pro, nano-banana-2.1, nano-banana-2): 1K (default), 2K, 4K. gpt-image-2.5-flare and gpt-image-2.5-sunburst: 1K (default), 2K, 4K. seedream-5: 2K (default) or 3K. seedream-5-pro: 1K or 2K (default). grok-imagine-image-2: 1K (default) or 2K. ideogram-4.5: 1K or 2K (default). fibo-1.5: 1K (default) or 4K. Ignored by other models."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"promptText": {
"type": "string",
"description": "Text prompt for ONE standalone image. If count > 1, keep the prompt about a single output, not a grid/collage/contact sheet. If the user requests a named artist/director/photographer/studio style, describe the visual qualities instead and do not include the name."
},
"referenceImages": {
"type": "array",
"items": {
"type": "object",
"properties": {
"tag": {
"type": "string"
},
"url": {
"type": "string"
},
"taskId": {
"type": "string",
"description": "Runway task ID of a finished image generation. The server resolves the image URL itself — use this instead of copying a long signed URL."
}
},
"additionalProperties": false
},
"description": "Optional reference images [{url | taskId, tag}]. Each entry needs EITHER `taskId` (preferred when the image came from a Runway tool — e.g. the user clicked Edit in the viewer) OR `url`."
},
"referenceImageFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string"
},
"file_name": {
"type": "string"
},
"mime_type": {
"type": "string"
},
"download_url": {
"type": "string"
}
},
"description": "ChatGPT-only file param for a user-uploaded reference/source image. Prefer referenceImages when you already have a public or Runway-hosted URL.",
"additionalProperties": false
},
"transparentBackground": {
"type": "boolean",
"description": "Output a PNG with a transparent background (logos, stickers, icons, cutouts). Only seedream-5-pro, seedream-5-flash, gpt-image-2, and gpt-image-2.5-sunburst support this (not gpt-image-2.5-flare). With model omitted the server picks a supporting model; naming another model returns an error. Default false."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Generate Image",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description95/100
Generate OR edit a video using a Runway-hosted video model. Pass the source video as `referenceVideo` to edit/restyle it.
MODEL STATUS: Call `whoami` before choosing a model. Check `modelDeprecations` for upcoming sunsets and unavailable task variants; warn briefly about a relevant upcoming sunset, and ask before switching models. Model enums describe supported inputs, not current availability.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
PAID PLAN: If this tool returns paid_plan_required, follow the NEXT line in that result — it asks for `show_plans_and_credits` with the blocked action as `gatedCapability`, which renders the upgrade card. Do not retry with a different video model from this schema; every video model needs a paid plan. Only offer models `whoami` lists in `availableVideoModels` / `availableImageModels`, and only after calling `whoami`.
CHOOSING BETWEEN THIS TOOL AND `edit_video` FOR FOOTAGE EDITS:
- Use `edit_video` (Aleph 2.0 in-context editing) when the user wants a targeted, surgical change to existing footage and everything else must stay exactly as shot: "make the sneakers red", "change the outfit", "remove the object on the wall", "change the season to winter", "relight the scene", element swaps, or restyles that must preserve the original motion/framing/timing — or whenever the user mentions Aleph or Edit Studio. `edit_video` also requires a paid workspace — do not send free-plan users there as a workaround.
- Use THIS tool's video-to-video mode (`referenceVideo`, seedance-2) when the source video is more of a loose reference: motion transfer, style/structure inspiration, heavy re-imagining, or when the source violates Aleph's constraints (must be 2-30s, <=1080p, <=30fps, <=10 cuts, conventional aspect ratio).
- The two are mutual fallbacks: if an `edit_video` result was unsatisfying (or the source is out of Aleph's constraints), try v2v here; if a v2v edit here changed more than the user asked for, suggest `edit_video` instead.
For creative product ad videos from a product URL or product image, use `generate_product_marketing_video` instead. It handles product-image extraction, cinematic storyboard generation, and the final ad video.
For multi-shot videos (one prompt drives 3-5 connected scenes), use `generate_multishot_video` instead.
MODES:
1. Text-to-video — pass `promptText` only. Most models support this directly. `gen-4-turbo` is the exception: for best reliability, generate a still image first with `generate_image`, then animate that image with `gen-4-turbo`. Example: "aerial drone shot, slow push forward over misty mountains at sunrise".
2. Image-to-video — pass `startFrame.url` (or ChatGPT's `startImageFile`) plus `promptText` to animate a still image. Use this when you want to bring a specific image to life. Example: a product photo + "slow 360 orbit around the bottle".
3. Video-to-video (edit/restyle) — pass `referenceVideo.url` plus a `promptText` describing the desired change. Use this to *modify* an existing video while preserving motion and composition: remove or replace backgrounds, remove unwanted objects/people, add new objects, swap backgrounds, change lighting, restyle scene, alter weather/time-of-day. Supported by the Seedance family (`seedance-2` is the DEFAULT for v2v; `seedance-2.5` handles sources longer than 15s), `kling-o3-pro`, and the WAN family (`wan3`, `wan3-prime`). Example: an existing dance video + "remove the background and place the dancer on a clean white studio backdrop".
VIDEO EDITING REQUESTS:
- If the user asks to remove a background, remove an object/person, clean up a scene, add something to an existing video, or otherwise edit footage, DO NOT say Runway MCP cannot do it. First consider `edit_video` (Aleph 2.0, paid workspace) per the routing guidance above; otherwise use video-to-video here: pass the source clip as `referenceVideo` / `referenceVideoFile`, default to `seedance-2`, and describe the edit in `promptText`.
- For background removal, ask for or use the input video as `referenceVideo` and prompt for the desired replacement/background treatment, e.g. "remove the background and isolate the subject on a plain neutral studio backdrop" or "replace the background with a clean white seamless studio".
DRAFT MODE (seedance-2.5 only):
- `draft: true` renders a fast 480p preview at the 480p rate (20 credits/s). If the user likes it, call `enhance_video_draft` with its task ID to re-render the same shot at 1080p (68 credits/s) within 7 days.
- Draft plus enhance costs MORE than a direct 1080p render, so only draft when the user wants to iterate on a shot (try several prompts, pick one) or explicitly asks for a draft/preview. For a single clip the user already wants in high quality, render at 1080p directly.
DURATION DEFAULTS (the tool sets these automatically when you OMIT `duration`):
- Text-to-video & image-to-video → 10s (or the model's largest supported value <= 10s; veo-3.1 → 8s).
- Video-to-video WITH `referenceVideo.durationSeconds` → largest model-supported value <= source. (e.g. 20s source with kling-o3-pro {5,10,15} → 15s; 8s source with seedance-2 {4..15} → 8s.)
- Video-to-video WITHOUT `referenceVideo.durationSeconds` → model's MAX supported duration (15s for seedance/kling) so the edit doesn't get accidentally truncated.
DURATION FOR V2V — IMPORTANT:
Always pass `referenceVideo.durationSeconds` when you know it (read it from the file metadata you uploaded). It's the only way the tool can know how much of the source to edit.
- DO NOT pass `duration: 5` for a 20-second source video — that will silently clip the edit to the first 5 seconds. Just OMIT `duration` and pass `referenceVideo.durationSeconds` instead, and the tool will pick the right output length.
- Only set `duration` explicitly when the user asks for a shorter clip ("just make me a 5-second teaser of this 20-second video").
DO NOT use this tool to edit a still **image** → use `generate_image` and pass the image as `referenceImages[0]`.
ANIMATE BUTTON (viewer → chat): when a user message says "Animate this image" and names a Runway task ID, the user clicked Animate on a finished image in the inline viewer. Follow this two-step flow:
1. Do NOT generate yet. Reply in one or two sentences: confirm you can turn that image into a short video (default 10 seconds), describe the image briefly if you know its prompt, and ask what should happen in it — motion, camera, mood — and whether they want a different length.
2. Once they answer, call this tool with `startFrame.taskId` set to that task ID (never retype the image URL), `promptText` built from their answer, and `duration` only if they asked for a specific length. Keep the ratio matching the source image when you know it.
HANDLING USER ATTACHMENTS:
- ChatGPT: if it supplies an uploaded image as `startImageFile`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as `startFrame.url`.
- ChatGPT: if it supplies an uploaded video as `referenceVideoFile`, pass it directly for video-to-video edits. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as `referenceVideo.url`.
- ChatGPT: NEVER pass local paths such as `/mnt/data/file.png` or `/mnt/data/file.mp4` as `startFrame.url` or `referenceVideo.url`; the remote MCP server cannot read them. Use `startImageFile` / `referenceVideoFile` instead.
- Claude/Cursor/local agents: if the user attached a local image or video, run `init_upload` -> curl -> `complete_upload` first to get a Runway-hosted asset URL, then pass it as `startFrame.url`, `endFrame.url`, `referenceImages[].url`, or `referenceVideo.url`.
REUSING ASSETS: URLs returned by other Runway tools (image outputs from `generate_image`, the asset URL from `complete_upload`) are stable, hosted asset URLs. Pass them directly into the relevant field — never re-upload. COPY THESE URLS VERBATIM. They end in a long `_jwt=` authorization token that is meaningless if even one character changes. Copy the whole string exactly as it appeared in the earlier tool result; never retype it, truncate it, shorten it for readability, or reconstruct it from the parts you remember. If you cannot reproduce the URL exactly, call `init_upload` and use the URL it returns instead. Note: outputs of `generate_video` itself are NOT yet usable as `referenceVideo` (re-upload via `init_upload` if needed).
USER-FACING REPLIES: Pick the right model internally, but DO NOT mention the model name to the user (e.g. "using Veo 3.1", "with Seedance 2") unless they explicitly ask which model was used. Talk about the video content — subject, motion, mood — not the engine.
MODEL NAMES (a user may name a model by its product name; pass the id on the left):
- `seedance-2` = Seedance 2.0
- `seedance-2.5` = Seedance 2.5
- `seedance-2-fast` = Seedance 2.0 Fast
- `seedance-2-mini` = Seedance 2.0 Mini
- `kling-o3-pro` = Kling O3 Pro
- `kling-o3-standard` = Kling 3.0 Omni Standard
- `kling-3-pro` = Kling 3.0 Pro
- `kling-3-standard` = Kling 3.0 Standard
- `kling-3-turbo` = Kling 3.0 Turbo
- `kling-3-motion-control` = Kling 3.0 Motion Control
- `kling-3-4k` = Kling 3.0 4K
- `kling-o3-4k` = Kling 3.0 Omni 4K
- `gen-4.5` = Gen-4.5
- `veo-3.1` = Veo 3.1
- `grok-imagine-1.5` = Grok Imagine 1.5
- `gemini-omni-flash` = Gemini Omni Flash
- `h3-max` = MiniMax H3 Max
- `hailuo-3` = MiniMax H3
- `wan3` = WAN 3.0
- `wan3-prime` = WAN 3.0 Prime
- `gen-4-turbo` = Gen-4 Turbo
MODELS (`*` = image-first workflow):
| model | tier | t2v | i2v | v2v | end | refs | audio | durations | max res |
|-------------------|------|-----|-----|-----|-----|------|-------|------------|---------|
| seedance-2 | paid | Y | Y | Y | Y | Y | Y | 4-15 | 4k |
| seedance-2.5 | paid | Y | Y | Y | Y | Y | Y | 4-30 | 1080p |
| seedance-2-fast | paid | Y | Y | Y | Y | Y | Y | 4-15 | 720p |
| seedance-2-mini | paid | Y | Y | Y | Y | Y | Y | 4-15 | 720p |
| kling-o3-pro | paid | Y | Y | Y | Y | Y | Y | 5, 10, 15 | 1080p |
| kling-o3-standard | paid | Y | Y | Y | Y | Y | Y | 3-15 | 720p |
| kling-3-pro | paid | Y | Y | - | Y | - | Y | 5, 10, 15 | 1080p |
| kling-3-standard | paid | Y | Y | - | Y | - | Y | 3-15 | 720p |
| kling-3-turbo | paid | Y | Y | - | - | - | Y | 3-15 | 1080p |
| kling-3-4k | paid | Y | Y | - | Y | - | Y | 3-15 | 4k |
| kling-o3-4k | paid | Y | Y | - | Y | Y | Y | 3-15 | 4k |
| gen-4.5 | paid | Y | Y | - | - | - | - | 2-10 | 1080p |
| veo-3.1 | paid | Y | Y | - | Y | - | Y | 4, 6, 8 | 1080p |
| grok-imagine-1.5 | paid | Y | Y | - | - | Y | Y | 1-15 | 1080p |
| gemini-omni-flash | paid | Y | Y | - | - | Y | - | 3-10 | 720p |
| h3-max | paid | Y | Y | - | Y | - | - | 5-15 | 768p |
| hailuo-3 | paid | Y | Y | Y | Y | Y | Y | 5-15 | 1080p |
| wan3 | paid | Y | Y | Y | Y | Y | Y | 2-30 | 1080p |
| wan3-prime | paid | Y | Y | Y | Y | Y | Y | 2-30 | 1080p |
| gen-4-turbo | paid | * | Y | - | - | - | - | 5, 10 | 720p |
Legend: t2v = text-to-video, i2v = image-to-video (startFrame), v2v = video-to-video (referenceVideo, edit/restyle existing video), end = end-frame target, refs = reference images. `*` for gen-4-turbo means it is not native text-to-video: it needs a starting image. For text-only requests, create a still image first, then call gen-4-turbo with that image as `startFrame`.
Picking heuristic:
- seedance-2 (DEFAULT for t2v/i2v/v2v) — best general-purpose, including existing-footage edits like removing/replacing backgrounds, removing objects/people, and adding new elements. Pick this unless you have a reason not to. The only Seedance tier that does 4k (reserve 4k for explicit 4K requests — it costs ~4x 1080p).
- seedance-2.5 — newest Seedance generation. Same capabilities as seedance-2 plus durations up to 30s and prompts up to 15000 characters, and goes up to 1080p (no 4k). The only model with draft mode (see DRAFT MODE below). Pick when the user asks for Seedance 2.5 by name, wants a clip longer than 15s, or wants to preview cheaply before committing to 1080p. Availability is account-gated: if the API rejects it with an unknown-task-type error, fall back to seedance-2.
- seedance-2-fast — same capabilities as seedance-2 but faster and cheaper; caps at 720p. Pick when latency/cost matter more than resolution and 1080p isn't needed.
- seedance-2-mini — cheapest Seedance tier; identical features, 720p max. Pick for budget-sensitive or high-volume generation.
- kling-o3-pro — pick when consistency matters: character/product must look identical across shots, brand or serialized content, or when v2v needs to preserve identity. Best for ads, episodic content, and product storytelling. Optimized for 1-2 main subjects.
- kling-o3-standard — lower-cost 720p Omni tier with the same identity/style references and v2v editing strengths. Pick for consistency-focused iteration when 1080p is unnecessary; supports whole-second durations from 3-15s, with video-reference edits capped at 10s.
- kling-3-pro — pick when prompt fidelity matters more than reference-driven consistency: complex multi-character scenes (3+ people), crowded environments, experimental scripts, rapid ideation without reference assets. (No v2v support.)
- kling-3-standard — lower-cost Kling 3.0 tier for text or first/end-frame generation at 720p, with optional native audio and whole-second durations from 3-15s. Pick it for high-volume iteration when 1080p is unnecessary. (No prompt reference images or v2v support.)
- kling-3-turbo — pick for fast prompt iteration or quick first-frame animation at 720p/1080p. It accepts one start frame, has always-on native audio, and supports whole-second durations from 3-15s. (No end frame, reference images, or v2v support.)
- kling-3-4k — native 4K Kling generation with optional audio and start/end-frame control. Use only when the user explicitly asks for Kling 3.0 4K or true 4K output: it costs 43 credits/second (129 credits at the 3s minimum). Availability is account-gated; on an unknown-task-type or unavailable-model error, fall back to seedance-2 at 4k. (No reference images or v2v support.)
- kling-o3-4k — native 4K Kling Omni generation with optional audio, start/end frames, and up to 7 identity/style references. Use when the user explicitly asks for Kling O3/Omni 4K and consistency matters. It costs 43 credits/second (129 credits at the 3s minimum). (No v2v support.)
- veo-3.1 — pick when photorealism and natively-generated audio matter (Veo invents dialogue spoken by characters in-frame, plus ambient sound matched to the scene). (No v2v support.)
- gen-4.5 — pick for keyframe-driven storytelling (provide a startFrame and the model anchors at timestamp 0). (No v2v support.)
- grok-imagine-1.5 — fast generation with native audio that is ALWAYS ON (no toggle — do not pass generateAudio: false) and audio-driven performances. Pick for quick turnaround, short clips (durations go down to 1s in 1s steps), or reference images addressable from the prompt (tag a reference "hero" and write "@hero" in promptText). Constraints: startFrame and referenceImages are mutually exclusive; up to 7 reference images but they cap output at 720p (1080p is t2v/i2v only); extra ratios 3:2 and 2:3. Availability is account-gated: on an unknown-task-type error, fall back to seedance-2. (No v2v support.)
- gemini-omni-flash — paid, real t2v/i2v. Pick for physics-realistic motion (gravity, fluids, collisions), knowledge-grounded scenes, or fusing image + text references in one request. Fixed 720p, no audio, ratios 16:9/9:16/auto only, up to 5 reference images. (No v2v support.)
- h3-max — MiniMax H3 Max, fal's post-trained H3 variant. Pick only when the user asks for H3 Max by name. It supports text-to-video or first/end-frame keyframes, whole-second durations from 5-15s, and 480p/768p output (`720p` is accepted as an alias for 768p). It has no aspect-ratio control, arbitrary reference images, reference video, or native audio. Prompts up to 6000 characters. Availability is account-gated: on a 403 "This model is not available for your account." fall back to hailuo-3 or seedance-2.
- hailuo-3 — MiniMax H3. Native audio is ALWAYS ON (no toggle — do not pass generateAudio: false). Pick for keyframe-driven or reference-driven clips up to 15s with sound, or when the user asks for H3/MiniMax by name; it also does v2v. Ratio note: with reference images or a reference video it defaults to `adaptive` (the reference's own framing); `adaptive` is rejected for text-only prompts. Prompts up to 6000 characters. Availability is account-gated: on a 403 "This model is not available for your account." fall back to seedance-2.
- wan3 — Alibaba WAN 3.0 (standard). Pick when the user asks for WAN/Alibaba by name without saying Prime, or for long clips with sound: durations run 2-30s and native audio is on by default (pass `generateAudio: false` to turn it off). Takes up to 10 reference images, a reference video of at most 15s, and prompts up to 20000 characters; it also does v2v. Ratio note: with reference images or a reference video it defaults to `adaptive` (the reference's own framing). Start/end frames cannot be combined with reference images or a reference video. Availability is account-gated: on an unknown-task-type error, fall back to seedance-2.
- wan3-prime — high-speed WAN 3.0. Same controls as wan3 (t2v/i2v/v2v, 2-30s, audio, refs, prompt limit) but faster and ~40% more credits (14/s at 720p vs 10/s). Pick when the user asks for WAN Prime / WAN 3.0 Prime / wan3_prime by name, or wants faster WAN generations. Same flag gating as wan3: on an unknown-task-type error, fall back to wan3 then seedance-2.
- gen-4-turbo — image-to-video model. If the user supplied an image, pass it as `startFrame`. For text-only requests, first call `generate_image`, then call `generate_video` with `model: "gen-4-turbo"` and `startFrame.url` set to the generated image URL. Do not keep retrying if the starter-frame step is still pending or fails. (No v2v support.)
Parameters:
- model: One of `seedance-2`, `seedance-2.5`, `seedance-2-fast`, `seedance-2-mini`, `kling-o3-pro`, `kling-o3-standard`, `kling-3-pro`, `kling-3-standard`, `kling-3-turbo`, `kling-3-motion-control`, `kling-3-4k`, `kling-o3-4k`, `gen-4.5`, `veo-3.1`, `grok-imagine-1.5`, `gemini-omni-flash`, `h3-max`, `hailuo-3`, `wan3`, `wan3-prime`, `gen-4-turbo`. Default for t2v/i2v: `seedance-2`. Default for v2v (when `referenceVideo` is set): `seedance-2`.
- promptText: Required. For t2v/i2v, describe the motion/style. For v2v, describe the edit ("remove the background", "remove the person in the left background", "add a floating logo above the table", "make it nighttime", "snow falling", "cyberpunk neon"). Prompts longer than 3500 characters are truncated. Model-specific limits: `kling-3-turbo`, `kling-3-4k`, and `kling-o3-4k` 2500; `seedance-2.5` 15000; `h3-max` 6000; `hailuo-3` 6000; WAN family (`wan3`, `wan3-prime`) 20000.
- ratio: Aspect ratio (16:9, 9:16, 1:1, plus 4:3, 3:4, 21:9, 3:2, 2:3 on some models). Ignored for v2v — the model preserves the source video's ratio.
- duration: Seconds (model-dependent — see table). USUALLY OMIT — defaults to 10s for t2v/i2v, and to the largest model-supported value <= source for v2v. See DURATION DEFAULTS above. Only set explicitly when the user asks for a non-default length.
- startFrame: `{taskId}` or `{url}` — i2v only. Animate this image. Prefer `taskId` (the Runway task ID of a finished image) whenever the image came from a Runway tool; the server resolves the URL. Otherwise `url` must be public HTTPS or a Runway-hosted asset URL. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `startImageFile` for ChatGPT uploads.
- startImageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`. Use this instead of `startFrame` only when ChatGPT provides a user-uploaded file.
- endFrame: `{url}` — i2v only. Final-frame target. Requires startFrame. (Seedance family, kling-o3-pro, kling-o3-4k, kling-3-pro, kling-3-standard, veo-3.1, h3-max, hailuo-3, wan3, wan3-prime.)
- referenceImages: Array of `{url, tag}` — i2v style/subject anchors. (Seedance family, kling-o3-pro, kling-o3-4k, kling-o3-standard, grok-imagine-1.5, hailuo-3, wan3, wan3-prime. Kling O3 and grok take up to 7; hailuo-3 takes up to 9 and WAN takes up to 10. MiniMax H3 and WAN ignore tags and cannot combine references with startFrame/endFrame. For grok-imagine-1.5 the tag is prompt-addressable: tag "hero" -> write "@hero" in promptText.)
- referenceVideo: `{url, durationSeconds?}` — v2v source video for editing/restyling existing footage, including removing/replacing backgrounds, removing objects/people, and adding elements. URL must be a Runway-hosted `/datasets/<UUID>.<ext>` URL from `complete_upload`. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `referenceVideoFile` for ChatGPT uploads. Pass `durationSeconds` from your file metadata when known so the model picks an appropriate output length.
- referenceVideoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. Use this instead of `referenceVideo` only when ChatGPT provides a user-uploaded video.
- referenceVideos: Array of `{url, durationSeconds?}` — extra reference clips to borrow from (a dance's choreography, a camera move, a character, a look), separate from `referenceVideo`, the footage to edit. Seedance family only: seedance-2.5 takes up to 10 videos per request and 30s combined; seedance-2, -fast and -mini take 3 and 15s. A `referenceVideo` source counts toward both caps. Same URL rules as `referenceVideo`; pass `durationSeconds` when known. Not combinable with startFrame/endFrame. Say in promptText what each clip is for, in order (e.g. "the dancer from video 1 does the choreography from video 2").
- generateAudio: Whether to generate native audio (model-dependent; default true for audio-capable models). For v2v with kling-o3-pro, controls whether to keep the original audio. kling-3-turbo, grok-imagine-1.5, and hailuo-3 have no toggle (audio is always on) — omit this parameter for them. h3-max is silent and rejects `generateAudio: true`.
- resolution: Override default resolution (kling-3-4k and kling-o3-4k: fixed 4k; kling-3-standard and kling-o3-standard: fixed 720p; kling-3-motion-control: 720p/1080p, default 720p; kling-3-turbo: 720p/1080p, default 1080p; seedance-2: 480p/720p/1080p/4k; seedance-2.5: 480p/720p/1080p; seedance-2-fast & seedance-2-mini: 480p/720p; veo: 720p/1080p — 1080p requires 8s; grok-imagine-1.5: 480p/720p/1080p — 1080p not with referenceImages; h3-max: 480p/768p (720p alias accepted); hailuo-3: 720p/1080p; wan3 & wan3-prime: 480p/720p/1080p; gemini-omni-flash: fixed 720p). COST GUARDRAIL: 4K is expensive (Kling 4K costs 43 credits/second; seedance-2 `4k` costs roughly 4x `1080p`). ONLY use a 4K model or set `4k` when the user explicitly asks for 4K (or "2160p"/"four K"). Never infer `4k` from vague quality requests like "high quality", "best", "high-res", or "crisp" — use the default or `1080p` for those.
- characterOrientation: Kling Motion Control only. `image` (default) follows the character-image framing and accepts a 3-10s driving video; `video` follows the driving-video framing and accepts 3-30s.
The description explains the capability, when to use it, and what to expect.
Inputs83/100
26 of 27 inputs have descriptions. Add descriptions to the missing inputs, including where to obtain IDs and how to supply values.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
model
Video model id. Default for t2v/i2v: seedance-2. Default for v2v: seedance-2.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
promptText
Required. For text-to-video: describe the scene and camera motion. For image-to-video: describe what should happen to the image. For video-to-video: describe the edit ("remove the background", "remove the person in the left background", "add a floating logo above the table", "make it nighttime", "swap background to a beach").
Clear enough to supply this argument.
ratio
Aspect ratio. Supported values vary by model. Ignored when referenceVideo is provided (the model preserves the source ratio).
Clear enough to supply this argument.
duration
Duration in seconds. Supported values vary by model (see DURATION DEFAULTS in description). USUALLY OMIT THIS — when unset, the tool defaults to 10s for t2v/i2v and to the largest supported value <= source for v2v. For v2v specifically: do NOT pass `duration: 5` for a longer source video; pass `referenceVideo.durationSeconds` and leave `duration` unset.
Clear enough to supply this argument.
startFrame › taskId
Runway task ID of a finished image generation. The server resolves the image URL itself — use this instead of copying a long signed URL.
Clear enough to supply this argument.
startFrame › url
Public HTTPS or Runway-hosted image URL. Never use /mnt/data or other local file paths here.
Clear enough to supply this argument.
startImageFile › download_url
Temporary ChatGPT file download URL.
Clear enough to supply this argument.
startImageFile › file_id
ChatGPT file ID.
Clear enough to supply this argument.
startImageFile › mime_type
MIME type, e.g. image/png.
Clear enough to supply this argument.
startImageFile › file_name
Original file name.
Clear enough to supply this argument.
endFrame › url
Image URL.
Explain the meaning of this value, not just its name.
referenceImages › items › url
Not provided
Missing description. Explain what this value means and how to supply it.
referenceImages › items › tag
Optional alias used in the prompt (e.g. tag "cat" lets you write "@cat" in promptText).
Clear enough to supply this argument.
referenceVideo › url
Runway-hosted video URL from complete_upload (format: https://<cdn>/datasets/<UUID>.<ext>?_jwt=...). Never use /mnt/data or other local file paths here.
Clear enough to supply this argument.
referenceVideo › durationSeconds
Source video duration in seconds. ALWAYS pass when known — the tool uses it to auto-select the best supported output duration (largest <= source) so a 20s source produces a 15s edit instead of being clipped to the model default of 5s.
Clear enough to supply this argument.
referenceVideoFile › download_url
Temporary ChatGPT file download URL.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
referenceVideoFile › file_id
ChatGPT file ID.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
referenceVideoFile › mime_type
MIME type, e.g. video/mp4.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
referenceVideoFile › file_name
Original file name.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
referenceVideos › items › url
Runway-hosted video URL from complete_upload (format: https://<cdn>/datasets/<UUID>.<ext>?_jwt=...) or a public HTTPS video URL.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
referenceVideos › items › durationSeconds
Clip length in seconds, when known.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
referenceVideoDurationSeconds
Source duration for referenceVideoFile. Required by Kling Motion Control and recommended whenever video file metadata is known.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
generateAudio
Generate native audio (model-dependent). For kling-o3-pro v2v: controls whether to keep the original audio (default true). Omit for kling-3-turbo, grok-imagine-1.5, and hailuo-3 — their audio is always on and generateAudio: false is rejected. h3-max is silent and rejects generateAudio: true.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
resolution
Output resolution (model-dependent). kling-3-4k and kling-o3-4k: fixed 4k (omit or pass 4k). kling-3-standard and kling-o3-standard: fixed 720p. kling-3-motion-control: 720p/1080p (default 720p). kling-3-turbo: 720p/1080p (default 1080p). seedance-2: 480p/720p/1080p/4k; seedance-2.5: 480p/720p/1080p; seedance-2-fast & seedance-2-mini: 480p/720p. Veo: 720p/1080p (1080p requires 8s). grok-imagine-1.5: 480p/720p/1080p (1080p not with referenceImages). h3-max: 480p/768p (720p alias accepted). hailuo-3: 720p/1080p. wan3 & wan3-prime: 480p/720p/1080p. gemini-omni-flash: fixed 720p (omit this parameter). Default for other models when omitted: 720p. 4K is expensive, so ONLY use it when the user explicitly requests 4K — never for vague "high quality"/"best"/"high-res" phrasing.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
characterOrientation
Kling Motion Control only: image follows the character image framing (3-10s driving video); video follows the driving video framing (3-30s). Default: image.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
draft
seedance-2.5 only. Render a cheap 480p preview that `enhance_video_draft` can later re-render at 1080p. See DRAFT MODE in the description for when it is worth it.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "generate_video",
"title": "Generate Video",
"description": "Generate OR edit a video using a Runway-hosted video model. Pass the source video as `referenceVideo` to edit/restyle it.\n\nMODEL STATUS: Call `whoami` before choosing a model. Check `modelDeprecations` for upcoming sunsets and unavailable task variants; warn briefly about a relevant upcoming sunset, and ask before switching models. Model enums describe supported inputs, not current availability.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nPAID PLAN: If this tool returns paid_plan_required, follow the NEXT line in that result — it asks for `show_plans_and_credits` with the blocked action as `gatedCapability`, which renders the upgrade card. Do not retry with a different video model from this schema; every video model needs a paid plan. Only offer models `whoami` lists in `availableVideoModels` / `availableImageModels`, and only after calling `whoami`.\n\nCHOOSING BETWEEN THIS TOOL AND `edit_video` FOR FOOTAGE EDITS:\n- Use `edit_video` (Aleph 2.0 in-context editing) when the user wants a targeted, surgical change to existing footage and everything else must stay exactly as shot: \"make the sneakers red\", \"change the outfit\", \"remove the object on the wall\", \"change the season to winter\", \"relight the scene\", element swaps, or restyles that must preserve the original motion/framing/timing — or whenever the user mentions Aleph or Edit Studio. `edit_video` also requires a paid workspace — do not send free-plan users there as a workaround.\n- Use THIS tool's video-to-video mode (`referenceVideo`, seedance-2) when the source video is more of a loose reference: motion transfer, style/structure inspiration, heavy re-imagining, or when the source violates Aleph's constraints (must be 2-30s, <=1080p, <=30fps, <=10 cuts, conventional aspect ratio).\n- The two are mutual fallbacks: if an `edit_video` result was unsatisfying (or the source is out of Aleph's constraints), try v2v here; if a v2v edit here changed more than the user asked for, suggest `edit_video` instead.\n\nFor creative product ad videos from a product URL or product image, use `generate_product_marketing_video` instead. It handles product-image extraction, cinematic storyboard generation, and the final ad video.\n\nFor multi-shot videos (one prompt drives 3-5 connected scenes), use `generate_multishot_video` instead.\n\nMODES:\n1. Text-to-video — pass `promptText` only. Most models support this directly. `gen-4-turbo` is the exception: for best reliability, generate a still image first with `generate_image`, then animate that image with `gen-4-turbo`. Example: \"aerial drone shot, slow push forward over misty mountains at sunrise\".\n2. Image-to-video — pass `startFrame.url` (or ChatGPT's `startImageFile`) plus `promptText` to animate a still image. Use this when you want to bring a specific image to life. Example: a product photo + \"slow 360 orbit around the bottle\".\n3. Video-to-video (edit/restyle) — pass `referenceVideo.url` plus a `promptText` describing the desired change. Use this to *modify* an existing video while preserving motion and composition: remove or replace backgrounds, remove unwanted objects/people, add new objects, swap backgrounds, change lighting, restyle scene, alter weather/time-of-day. Supported by the Seedance family (`seedance-2` is the DEFAULT for v2v; `seedance-2.5` handles sources longer than 15s), `kling-o3-pro`, and the WAN family (`wan3`, `wan3-prime`). Example: an existing dance video + \"remove the background and place the dancer on a clean white studio backdrop\".\n\nVIDEO EDITING REQUESTS:\n- If the user asks to remove a background, remove an object/person, clean up a scene, add something to an existing video, or otherwise edit footage, DO NOT say Runway MCP cannot do it. First consider `edit_video` (Aleph 2.0, paid workspace) per the routing guidance above; otherwise use video-to-video here: pass the source clip as `referenceVideo` / `referenceVideoFile`, default to `seedance-2`, and describe the edit in `promptText`.\n- For background removal, ask for or use the input video as `referenceVideo` and prompt for the desired replacement/background treatment, e.g. \"remove the background and isolate the subject on a plain neutral studio backdrop\" or \"replace the background with a clean white seamless studio\".\n\nDRAFT MODE (seedance-2.5 only):\n- `draft: true` renders a fast 480p preview at the 480p rate (20 credits/s). If the user likes it, call `enhance_video_draft` with its task ID to re-render the same shot at 1080p (68 credits/s) within 7 days.\n- Draft plus enhance costs MORE than a direct 1080p render, so only draft when the user wants to iterate on a shot (try several prompts, pick one) or explicitly asks for a draft/preview. For a single clip the user already wants in high quality, render at 1080p directly.\n\nDURATION DEFAULTS (the tool sets these automatically when you OMIT `duration`):\n- Text-to-video & image-to-video → 10s (or the model's largest supported value <= 10s; veo-3.1 → 8s).\n- Video-to-video WITH `referenceVideo.durationSeconds` → largest model-supported value <= source. (e.g. 20s source with kling-o3-pro {5,10,15} → 15s; 8s source with seedance-2 {4..15} → 8s.)\n- Video-to-video WITHOUT `referenceVideo.durationSeconds` → model's MAX supported duration (15s for seedance/kling) so the edit doesn't get accidentally truncated.\n\nDURATION FOR V2V — IMPORTANT:\nAlways pass `referenceVideo.durationSeconds` when you know it (read it from the file metadata you uploaded). It's the only way the tool can know how much of the source to edit.\n- DO NOT pass `duration: 5` for a 20-second source video — that will silently clip the edit to the first 5 seconds. Just OMIT `duration` and pass `referenceVideo.durationSeconds` instead, and the tool will pick the right output length.\n- Only set `duration` explicitly when the user asks for a shorter clip (\"just make me a 5-second teaser of this 20-second video\").\n\nDO NOT use this tool to edit a still **image** → use `generate_image` and pass the image as `referenceImages[0]`.\n\nANIMATE BUTTON (viewer → chat): when a user message says \"Animate this image\" and names a Runway task ID, the user clicked Animate on a finished image in the inline viewer. Follow this two-step flow:\n1. Do NOT generate yet. Reply in one or two sentences: confirm you can turn that image into a short video (default 10 seconds), describe the image briefly if you know its prompt, and ask what should happen in it — motion, camera, mood — and whether they want a different length.\n2. Once they answer, call this tool with `startFrame.taskId` set to that task ID (never retype the image URL), `promptText` built from their answer, and `duration` only if they asked for a specific length. Keep the ratio matching the source image when you know it.\n\nHANDLING USER ATTACHMENTS:\n- ChatGPT: if it supplies an uploaded image as `startImageFile`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as `startFrame.url`.\n- ChatGPT: if it supplies an uploaded video as `referenceVideoFile`, pass it directly for video-to-video edits. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as `referenceVideo.url`.\n- ChatGPT: NEVER pass local paths such as `/mnt/data/file.png` or `/mnt/data/file.mp4` as `startFrame.url` or `referenceVideo.url`; the remote MCP server cannot read them. Use `startImageFile` / `referenceVideoFile` instead.\n- Claude/Cursor/local agents: if the user attached a local image or video, run `init_upload` -> curl -> `complete_upload` first to get a Runway-hosted asset URL, then pass it as `startFrame.url`, `endFrame.url`, `referenceImages[].url`, or `referenceVideo.url`.\n\nREUSING ASSETS: URLs returned by other Runway tools (image outputs from `generate_image`, the asset URL from `complete_upload`) are stable, hosted asset URLs. Pass them directly into the relevant field — never re-upload. COPY THESE URLS VERBATIM. They end in a long `_jwt=` authorization token that is meaningless if even one character changes. Copy the whole string exactly as it appeared in the earlier tool result; never retype it, truncate it, shorten it for readability, or reconstruct it from the parts you remember. If you cannot reproduce the URL exactly, call `init_upload` and use the URL it returns instead. Note: outputs of `generate_video` itself are NOT yet usable as `referenceVideo` (re-upload via `init_upload` if needed).\n\nUSER-FACING REPLIES: Pick the right model internally, but DO NOT mention the model name to the user (e.g. \"using Veo 3.1\", \"with Seedance 2\") unless they explicitly ask which model was used. Talk about the video content — subject, motion, mood — not the engine.\n\nMODEL NAMES (a user may name a model by its product name; pass the id on the left):\n- `seedance-2` = Seedance 2.0\n- `seedance-2.5` = Seedance 2.5\n- `seedance-2-fast` = Seedance 2.0 Fast\n- `seedance-2-mini` = Seedance 2.0 Mini\n- `kling-o3-pro` = Kling O3 Pro\n- `kling-o3-standard` = Kling 3.0 Omni Standard\n- `kling-3-pro` = Kling 3.0 Pro\n- `kling-3-standard` = Kling 3.0 Standard\n- `kling-3-turbo` = Kling 3.0 Turbo\n- `kling-3-motion-control` = Kling 3.0 Motion Control\n- `kling-3-4k` = Kling 3.0 4K\n- `kling-o3-4k` = Kling 3.0 Omni 4K\n- `gen-4.5` = Gen-4.5\n- `veo-3.1` = Veo 3.1\n- `grok-imagine-1.5` = Grok Imagine 1.5\n- `gemini-omni-flash` = Gemini Omni Flash\n- `h3-max` = MiniMax H3 Max\n- `hailuo-3` = MiniMax H3\n- `wan3` = WAN 3.0\n- `wan3-prime` = WAN 3.0 Prime\n- `gen-4-turbo` = Gen-4 Turbo\n\nMODELS (`*` = image-first workflow):\n| model | tier | t2v | i2v | v2v | end | refs | audio | durations | max res |\n|-------------------|------|-----|-----|-----|-----|------|-------|------------|---------|\n| seedance-2 | paid | Y | Y | Y | Y | Y | Y | 4-15 | 4k |\n| seedance-2.5 | paid | Y | Y | Y | Y | Y | Y | 4-30 | 1080p |\n| seedance-2-fast | paid | Y | Y | Y | Y | Y | Y | 4-15 | 720p |\n| seedance-2-mini | paid | Y | Y | Y | Y | Y | Y | 4-15 | 720p |\n| kling-o3-pro | paid | Y | Y | Y | Y | Y | Y | 5, 10, 15 | 1080p |\n| kling-o3-standard | paid | Y | Y | Y | Y | Y | Y | 3-15 | 720p |\n| kling-3-pro | paid | Y | Y | - | Y | - | Y | 5, 10, 15 | 1080p |\n| kling-3-standard | paid | Y | Y | - | Y | - | Y | 3-15 | 720p |\n| kling-3-turbo | paid | Y | Y | - | - | - | Y | 3-15 | 1080p |\n| kling-3-4k | paid | Y | Y | - | Y | - | Y | 3-15 | 4k |\n| kling-o3-4k | paid | Y | Y | - | Y | Y | Y | 3-15 | 4k |\n| gen-4.5 | paid | Y | Y | - | - | - | - | 2-10 | 1080p |\n| veo-3.1 | paid | Y | Y | - | Y | - | Y | 4, 6, 8 | 1080p |\n| grok-imagine-1.5 | paid | Y | Y | - | - | Y | Y | 1-15 | 1080p |\n| gemini-omni-flash | paid | Y | Y | - | - | Y | - | 3-10 | 720p |\n| h3-max | paid | Y | Y | - | Y | - | - | 5-15 | 768p |\n| hailuo-3 | paid | Y | Y | Y | Y | Y | Y | 5-15 | 1080p |\n| wan3 | paid | Y | Y | Y | Y | Y | Y | 2-30 | 1080p |\n| wan3-prime | paid | Y | Y | Y | Y | Y | Y | 2-30 | 1080p |\n| gen-4-turbo | paid | * | Y | - | - | - | - | 5, 10 | 720p |\n\nLegend: t2v = text-to-video, i2v = image-to-video (startFrame), v2v = video-to-video (referenceVideo, edit/restyle existing video), end = end-frame target, refs = reference images. `*` for gen-4-turbo means it is not native text-to-video: it needs a starting image. For text-only requests, create a still image first, then call gen-4-turbo with that image as `startFrame`.\n\nPicking heuristic:\n- seedance-2 (DEFAULT for t2v/i2v/v2v) — best general-purpose, including existing-footage edits like removing/replacing backgrounds, removing objects/people, and adding new elements. Pick this unless you have a reason not to. The only Seedance tier that does 4k (reserve 4k for explicit 4K requests — it costs ~4x 1080p).\n- seedance-2.5 — newest Seedance generation. Same capabilities as seedance-2 plus durations up to 30s and prompts up to 15000 characters, and goes up to 1080p (no 4k). The only model with draft mode (see DRAFT MODE below). Pick when the user asks for Seedance 2.5 by name, wants a clip longer than 15s, or wants to preview cheaply before committing to 1080p. Availability is account-gated: if the API rejects it with an unknown-task-type error, fall back to seedance-2.\n- seedance-2-fast — same capabilities as seedance-2 but faster and cheaper; caps at 720p. Pick when latency/cost matter more than resolution and 1080p isn't needed.\n- seedance-2-mini — cheapest Seedance tier; identical features, 720p max. Pick for budget-sensitive or high-volume generation.\n- kling-o3-pro — pick when consistency matters: character/product must look identical across shots, brand or serialized content, or when v2v needs to preserve identity. Best for ads, episodic content, and product storytelling. Optimized for 1-2 main subjects.\n- kling-o3-standard — lower-cost 720p Omni tier with the same identity/style references and v2v editing strengths. Pick for consistency-focused iteration when 1080p is unnecessary; supports whole-second durations from 3-15s, with video-reference edits capped at 10s.\n- kling-3-pro — pick when prompt fidelity matters more than reference-driven consistency: complex multi-character scenes (3+ people), crowded environments, experimental scripts, rapid ideation without reference assets. (No v2v support.)\n- kling-3-standard — lower-cost Kling 3.0 tier for text or first/end-frame generation at 720p, with optional native audio and whole-second durations from 3-15s. Pick it for high-volume iteration when 1080p is unnecessary. (No prompt reference images or v2v support.)\n- kling-3-turbo — pick for fast prompt iteration or quick first-frame animation at 720p/1080p. It accepts one start frame, has always-on native audio, and supports whole-second durations from 3-15s. (No end frame, reference images, or v2v support.)\n- kling-3-4k — native 4K Kling generation with optional audio and start/end-frame control. Use only when the user explicitly asks for Kling 3.0 4K or true 4K output: it costs 43 credits/second (129 credits at the 3s minimum). Availability is account-gated; on an unknown-task-type or unavailable-model error, fall back to seedance-2 at 4k. (No reference images or v2v support.)\n- kling-o3-4k — native 4K Kling Omni generation with optional audio, start/end frames, and up to 7 identity/style references. Use when the user explicitly asks for Kling O3/Omni 4K and consistency matters. It costs 43 credits/second (129 credits at the 3s minimum). (No v2v support.)\n- veo-3.1 — pick when photorealism and natively-generated audio matter (Veo invents dialogue spoken by characters in-frame, plus ambient sound matched to the scene). (No v2v support.)\n- gen-4.5 — pick for keyframe-driven storytelling (provide a startFrame and the model anchors at timestamp 0). (No v2v support.)\n- grok-imagine-1.5 — fast generation with native audio that is ALWAYS ON (no toggle — do not pass generateAudio: false) and audio-driven performances. Pick for quick turnaround, short clips (durations go down to 1s in 1s steps), or reference images addressable from the prompt (tag a reference \"hero\" and write \"@hero\" in promptText). Constraints: startFrame and referenceImages are mutually exclusive; up to 7 reference images but they cap output at 720p (1080p is t2v/i2v only); extra ratios 3:2 and 2:3. Availability is account-gated: on an unknown-task-type error, fall back to seedance-2. (No v2v support.)\n- gemini-omni-flash — paid, real t2v/i2v. Pick for physics-realistic motion (gravity, fluids, collisions), knowledge-grounded scenes, or fusing image + text references in one request. Fixed 720p, no audio, ratios 16:9/9:16/auto only, up to 5 reference images. (No v2v support.)\n- h3-max — MiniMax H3 Max, fal's post-trained H3 variant. Pick only when the user asks for H3 Max by name. It supports text-to-video or first/end-frame keyframes, whole-second durations from 5-15s, and 480p/768p output (`720p` is accepted as an alias for 768p). It has no aspect-ratio control, arbitrary reference images, reference video, or native audio. Prompts up to 6000 characters. Availability is account-gated: on a 403 \"This model is not available for your account.\" fall back to hailuo-3 or seedance-2.\n- hailuo-3 — MiniMax H3. Native audio is ALWAYS ON (no toggle — do not pass generateAudio: false). Pick for keyframe-driven or reference-driven clips up to 15s with sound, or when the user asks for H3/MiniMax by name; it also does v2v. Ratio note: with reference images or a reference video it defaults to `adaptive` (the reference's own framing); `adaptive` is rejected for text-only prompts. Prompts up to 6000 characters. Availability is account-gated: on a 403 \"This model is not available for your account.\" fall back to seedance-2.\n- wan3 — Alibaba WAN 3.0 (standard). Pick when the user asks for WAN/Alibaba by name without saying Prime, or for long clips with sound: durations run 2-30s and native audio is on by default (pass `generateAudio: false` to turn it off). Takes up to 10 reference images, a reference video of at most 15s, and prompts up to 20000 characters; it also does v2v. Ratio note: with reference images or a reference video it defaults to `adaptive` (the reference's own framing). Start/end frames cannot be combined with reference images or a reference video. Availability is account-gated: on an unknown-task-type error, fall back to seedance-2.\n- wan3-prime — high-speed WAN 3.0. Same controls as wan3 (t2v/i2v/v2v, 2-30s, audio, refs, prompt limit) but faster and ~40% more credits (14/s at 720p vs 10/s). Pick when the user asks for WAN Prime / WAN 3.0 Prime / wan3_prime by name, or wants faster WAN generations. Same flag gating as wan3: on an unknown-task-type error, fall back to wan3 then seedance-2.\n- gen-4-turbo — image-to-video model. If the user supplied an image, pass it as `startFrame`. For text-only requests, first call `generate_image`, then call `generate_video` with `model: \"gen-4-turbo\"` and `startFrame.url` set to the generated image URL. Do not keep retrying if the starter-frame step is still pending or fails. (No v2v support.)\n\nParameters:\n- model: One of `seedance-2`, `seedance-2.5`, `seedance-2-fast`, `seedance-2-mini`, `kling-o3-pro`, `kling-o3-standard`, `kling-3-pro`, `kling-3-standard`, `kling-3-turbo`, `kling-3-motion-control`, `kling-3-4k`, `kling-o3-4k`, `gen-4.5`, `veo-3.1`, `grok-imagine-1.5`, `gemini-omni-flash`, `h3-max`, `hailuo-3`, `wan3`, `wan3-prime`, `gen-4-turbo`. Default for t2v/i2v: `seedance-2`. Default for v2v (when `referenceVideo` is set): `seedance-2`.\n- promptText: Required. For t2v/i2v, describe the motion/style. For v2v, describe the edit (\"remove the background\", \"remove the person in the left background\", \"add a floating logo above the table\", \"make it nighttime\", \"snow falling\", \"cyberpunk neon\"). Prompts longer than 3500 characters are truncated. Model-specific limits: `kling-3-turbo`, `kling-3-4k`, and `kling-o3-4k` 2500; `seedance-2.5` 15000; `h3-max` 6000; `hailuo-3` 6000; WAN family (`wan3`, `wan3-prime`) 20000.\n- ratio: Aspect ratio (16:9, 9:16, 1:1, plus 4:3, 3:4, 21:9, 3:2, 2:3 on some models). Ignored for v2v — the model preserves the source video's ratio.\n- duration: Seconds (model-dependent — see table). USUALLY OMIT — defaults to 10s for t2v/i2v, and to the largest model-supported value <= source for v2v. See DURATION DEFAULTS above. Only set explicitly when the user asks for a non-default length.\n- startFrame: `{taskId}` or `{url}` — i2v only. Animate this image. Prefer `taskId` (the Runway task ID of a finished image) whenever the image came from a Runway tool; the server resolves the URL. Otherwise `url` must be public HTTPS or a Runway-hosted asset URL. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `startImageFile` for ChatGPT uploads.\n- startImageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`. Use this instead of `startFrame` only when ChatGPT provides a user-uploaded file.\n- endFrame: `{url}` — i2v only. Final-frame target. Requires startFrame. (Seedance family, kling-o3-pro, kling-o3-4k, kling-3-pro, kling-3-standard, veo-3.1, h3-max, hailuo-3, wan3, wan3-prime.)\n- referenceImages: Array of `{url, tag}` — i2v style/subject anchors. (Seedance family, kling-o3-pro, kling-o3-4k, kling-o3-standard, grok-imagine-1.5, hailuo-3, wan3, wan3-prime. Kling O3 and grok take up to 7; hailuo-3 takes up to 9 and WAN takes up to 10. MiniMax H3 and WAN ignore tags and cannot combine references with startFrame/endFrame. For grok-imagine-1.5 the tag is prompt-addressable: tag \"hero\" -> write \"@hero\" in promptText.)\n- referenceVideo: `{url, durationSeconds?}` — v2v source video for editing/restyling existing footage, including removing/replacing backgrounds, removing objects/people, and adding elements. URL must be a Runway-hosted `/datasets/<UUID>.<ext>` URL from `complete_upload`. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `referenceVideoFile` for ChatGPT uploads. Pass `durationSeconds` from your file metadata when known so the model picks an appropriate output length.\n- referenceVideoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. Use this instead of `referenceVideo` only when ChatGPT provides a user-uploaded video.\n- referenceVideos: Array of `{url, durationSeconds?}` — extra reference clips to borrow from (a dance's choreography, a camera move, a character, a look), separate from `referenceVideo`, the footage to edit. Seedance family only: seedance-2.5 takes up to 10 videos per request and 30s combined; seedance-2, -fast and -mini take 3 and 15s. A `referenceVideo` source counts toward both caps. Same URL rules as `referenceVideo`; pass `durationSeconds` when known. Not combinable with startFrame/endFrame. Say in promptText what each clip is for, in order (e.g. \"the dancer from video 1 does the choreography from video 2\").\n- generateAudio: Whether to generate native audio (model-dependent; default true for audio-capable models). For v2v with kling-o3-pro, controls whether to keep the original audio. kling-3-turbo, grok-imagine-1.5, and hailuo-3 have no toggle (audio is always on) — omit this parameter for them. h3-max is silent and rejects `generateAudio: true`.\n- resolution: Override default resolution (kling-3-4k and kling-o3-4k: fixed 4k; kling-3-standard and kling-o3-standard: fixed 720p; kling-3-motion-control: 720p/1080p, default 720p; kling-3-turbo: 720p/1080p, default 1080p; seedance-2: 480p/720p/1080p/4k; seedance-2.5: 480p/720p/1080p; seedance-2-fast & seedance-2-mini: 480p/720p; veo: 720p/1080p — 1080p requires 8s; grok-imagine-1.5: 480p/720p/1080p — 1080p not with referenceImages; h3-max: 480p/768p (720p alias accepted); hailuo-3: 720p/1080p; wan3 & wan3-prime: 480p/720p/1080p; gemini-omni-flash: fixed 720p). COST GUARDRAIL: 4K is expensive (Kling 4K costs 43 credits/second; seedance-2 `4k` costs roughly 4x `1080p`). ONLY use a 4K model or set `4k` when the user explicitly asks for 4K (or \"2160p\"/\"four K\"). Never infer `4k` from vague quality requests like \"high quality\", \"best\", \"high-res\", or \"crisp\" — use the default or `1080p` for those.\n- characterOrientation: Kling Motion Control only. `image` (default) follows the character-image framing and accepts a 3-10s driving video; `video` follows the driving-video framing and accepts 3-30s.",
"inputSchema": {
"type": "object",
"required": [
"promptText"
],
"properties": {
"draft": {
"type": "boolean",
"description": "seedance-2.5 only. Render a cheap 480p preview that `enhance_video_draft` can later re-render at 1080p. See DRAFT MODE in the description for when it is worth it."
},
"model": {
"enum": [
"seedance-2",
"seedance-2.5",
"seedance-2-fast",
"seedance-2-mini",
"kling-o3-pro",
"kling-o3-standard",
"kling-3-pro",
"kling-3-standard",
"kling-3-turbo",
"kling-3-motion-control",
"kling-3-4k",
"kling-o3-4k",
"gen-4.5",
"veo-3.1",
"grok-imagine-1.5",
"gemini-omni-flash",
"h3-max",
"hailuo-3",
"wan3",
"wan3-prime",
"gen-4-turbo"
],
"type": "string",
"description": "Video model id. Default for t2v/i2v: seedance-2. Default for v2v: seedance-2."
},
"ratio": {
"type": "string",
"description": "Aspect ratio. Supported values vary by model. Ignored when referenceVideo is provided (the model preserves the source ratio)."
},
"duration": {
"type": "number",
"description": "Duration in seconds. Supported values vary by model (see DURATION DEFAULTS in description). USUALLY OMIT THIS — when unset, the tool defaults to 10s for t2v/i2v and to the largest supported value <= source for v2v. For v2v specifically: do NOT pass `duration: 5` for a longer source video; pass `referenceVideo.durationSeconds` and leave `duration` unset."
},
"endFrame": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"description": "Image URL."
}
},
"description": "Image-to-video: end-frame target image. Requires startFrame. Supported by the Seedance family, kling-o3-pro, kling-o3-4k, kling-3-pro, kling-3-standard, veo-3.1, h3-max, hailuo-3, wan3, wan3-prime.",
"additionalProperties": false
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"promptText": {
"type": "string",
"description": "Required. For text-to-video: describe the scene and camera motion. For image-to-video: describe what should happen to the image. For video-to-video: describe the edit (\"remove the background\", \"remove the person in the left background\", \"add a floating logo above the table\", \"make it nighttime\", \"swap background to a beach\")."
},
"resolution": {
"type": "string",
"description": "Output resolution (model-dependent). kling-3-4k and kling-o3-4k: fixed 4k (omit or pass 4k). kling-3-standard and kling-o3-standard: fixed 720p. kling-3-motion-control: 720p/1080p (default 720p). kling-3-turbo: 720p/1080p (default 1080p). seedance-2: 480p/720p/1080p/4k; seedance-2.5: 480p/720p/1080p; seedance-2-fast & seedance-2-mini: 480p/720p. Veo: 720p/1080p (1080p requires 8s). grok-imagine-1.5: 480p/720p/1080p (1080p not with referenceImages). h3-max: 480p/768p (720p alias accepted). hailuo-3: 720p/1080p. wan3 & wan3-prime: 480p/720p/1080p. gemini-omni-flash: fixed 720p (omit this parameter). Default for other models when omitted: 720p. 4K is expensive, so ONLY use it when the user explicitly requests 4K — never for vague \"high quality\"/\"best\"/\"high-res\" phrasing."
},
"startFrame": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Public HTTPS or Runway-hosted image URL. Never use /mnt/data or other local file paths here."
},
"taskId": {
"type": "string",
"description": "Runway task ID of a finished image generation. The server resolves the image URL itself — use this instead of copying a long signed URL."
}
},
"description": "Image-to-video: source image to animate. Pass EITHER `taskId` (preferred when the image came from a Runway tool — e.g. the user clicked Animate in the viewer or you have the generate_image task ID) OR `url` (public HTTPS or Runway-hosted). Do not pass ChatGPT local paths like /mnt/data/...; use startImageFile instead.",
"additionalProperties": false
},
"generateAudio": {
"type": "boolean",
"description": "Generate native audio (model-dependent). For kling-o3-pro v2v: controls whether to keep the original audio (default true). Omit for kling-3-turbo, grok-imagine-1.5, and hailuo-3 — their audio is always on and generateAudio: false is rejected. h3-max is silent and rejects generateAudio: true."
},
"referenceVideo": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"description": "Runway-hosted video URL from complete_upload (format: https://<cdn>/datasets/<UUID>.<ext>?_jwt=...). Never use /mnt/data or other local file paths here."
},
"durationSeconds": {
"type": "number",
"description": "Source video duration in seconds. ALWAYS pass when known — the tool uses it to auto-select the best supported output duration (largest <= source) so a 20s source produces a 15s edit instead of being clipped to the model default of 5s."
}
},
"description": "Video-to-video: source video to edit/restyle. Use this for footage edits such as removing/replacing backgrounds, removing unwanted objects or people, adding objects, and changing scene style. URL must be a Runway-hosted /datasets/<UUID>.<ext> URL returned by complete_upload. Do not pass ChatGPT local paths like /mnt/data/...; use referenceVideoFile instead. Supported by the Seedance family, kling-o3-pro, kling-o3-standard, hailuo-3, wan3, and wan3-prime; the tool will default model to seedance-2 if you set referenceVideo without specifying model.",
"additionalProperties": false
},
"startImageFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. image/png."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source image. Prefer startFrame.url when you already have a public or Runway-hosted URL.",
"additionalProperties": false
},
"referenceImages": {
"type": "array",
"items": {
"type": "object",
"required": [
"url"
],
"properties": {
"tag": {
"type": "string",
"description": "Optional alias used in the prompt (e.g. tag \"cat\" lets you write \"@cat\" in promptText)."
},
"url": {
"type": "string"
}
},
"additionalProperties": false
},
"description": "Image-to-video: optional style/subject reference images. Supported by the Seedance family, kling-o3-pro, kling-o3-4k, kling-o3-standard, grok-imagine-1.5, hailuo-3, wan3, and wan3-prime (Kling O3: max 7; grok: max 7, caps output at 720p, cannot combine with startFrame; hailuo-3: max 9, tags ignored, cannot combine with startFrame/endFrame; WAN: max 10, tags ignored, cannot combine with startFrame/endFrame)."
},
"referenceVideos": {
"type": "array",
"items": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"description": "Runway-hosted video URL from complete_upload (format: https://<cdn>/datasets/<UUID>.<ext>?_jwt=...) or a public HTTPS video URL."
},
"durationSeconds": {
"type": "number",
"minimum": 0,
"description": "Clip length in seconds, when known."
}
},
"additionalProperties": false
},
"maxItems": 10,
"description": "Seedance family only: extra reference clips (choreography, camera move, character, look), separate from referenceVideo (the footage to edit). seedance-2.5: up to 10 videos per request and 30s combined; seedance-2/-fast/-mini: 3 and 15s. A referenceVideo source counts toward both caps. URLs follow the referenceVideo rules. Not combinable with startFrame/endFrame."
},
"referenceVideoFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. video/mp4."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source video. Prefer referenceVideo.url when you already have a public or Runway-hosted URL.",
"additionalProperties": false
},
"characterOrientation": {
"enum": [
"image",
"video"
],
"type": "string",
"description": "Kling Motion Control only: image follows the character image framing (3-10s driving video); video follows the driving video framing (3-30s). Default: image."
},
"referenceVideoDurationSeconds": {
"type": "number",
"minimum": 0,
"description": "Source duration for referenceVideoFile. Required by Kling Motion Control and recommended whenever video file metadata is known."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Generate Video",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description98/100
Re-render a Seedance 2.5 draft video at 1080p. A draft is a 480p preview made with `generate_video` using `model: "seedance-2.5"` and `draft: true`. The enhanced video keeps the draft's prompt, references, aspect ratio, duration and audio.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
COST: billed as a full 1080p Seedance 2.5 render (68 credits/s, plus reference-video seconds). The draft's cost is not credited back.
WHEN TO USE:
- The user liked a draft and wants the full-quality version ("make this one in HD", "enhance it", "render that at 1080p").
WHEN NOT TO USE:
- To change anything about the shot (prompt, framing, length): make a new draft with `generate_video` instead. Enhancement cannot change the content.
- For videos that were not generated as drafts, including normal seedance-2.5 renders: use `upscale_video` to sharpen an existing video.
RULES:
- Pass the draft's task ID as `taskId`. The draft must have finished and be less than 7 days old.
- Enhance each draft once. Calling this again for the same draft creates, and bills, a second 1080p render.
USER-FACING REPLIES: Talk about the result ("rendering your draft in full quality"), not the underlying engine.
The description explains the capability, when to use it, and what to expect.
Inputs97/100
Input documentation gives an agent enough context to populate the arguments.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
taskId
Runway task ID of a finished Seedance 2.5 draft (a generate_video call with draft: true).
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "enhance_video_draft",
"title": "Enhance Video Draft",
"description": "Re-render a Seedance 2.5 draft video at 1080p. A draft is a 480p preview made with `generate_video` using `model: \"seedance-2.5\"` and `draft: true`. The enhanced video keeps the draft's prompt, references, aspect ratio, duration and audio.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nCOST: billed as a full 1080p Seedance 2.5 render (68 credits/s, plus reference-video seconds). The draft's cost is not credited back.\n\nWHEN TO USE:\n- The user liked a draft and wants the full-quality version (\"make this one in HD\", \"enhance it\", \"render that at 1080p\").\n\nWHEN NOT TO USE:\n- To change anything about the shot (prompt, framing, length): make a new draft with `generate_video` instead. Enhancement cannot change the content.\n- For videos that were not generated as drafts, including normal seedance-2.5 renders: use `upscale_video` to sharpen an existing video.\n\nRULES:\n- Pass the draft's task ID as `taskId`. The draft must have finished and be less than 7 days old.\n- Enhance each draft once. Calling this again for the same draft creates, and bills, a second 1080p render.\n\nUSER-FACING REPLIES: Talk about the result (\"rendering your draft in full quality\"), not the underlying engine.",
"inputSchema": {
"type": "object",
"required": [
"taskId"
],
"properties": {
"taskId": {
"type": "string",
"description": "Runway task ID of a finished Seedance 2.5 draft (a generate_video call with draft: true)."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Enhance Video Draft",
"readOnlyHint": false,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Edit an existing video with Runway's Aleph 2.0 in-context video editor: it changes ONLY what you ask for and preserves everything else — subjects, motion, framing, timing, background, lighting.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
PAID PLAN REQUIRED: Aleph 2.0 is not available on free Runway workspaces. On a free workspace this tool returns paid_plan_required without creating a task — follow the NEXT line in that result (`show_plans_and_credits` with `gatedCapability`) and do not retry. Tell the user that editing video needs a paid plan rather than naming the model. Do not guess a fallback from this schema.
WHEN TO USE THIS vs generate_video:
- USE edit_video (this tool) for targeted, surgical edits of existing footage where preservation matters: "make the sneakers red", "change the outfit to a yellow fur coat", "remove the items on the wall behind her", "add graffiti on the wall", "change the season to winter", "relight the scene", "make it dark anime style" — or whenever the user mentions Aleph or Edit Studio.
- USE generate_video with referenceVideo (seedance-2) when the source video is more of a loose reference: motion transfer, using the clip as style/structure inspiration, or heavy re-imagining where preserving the original isn't the point. Also fall back to it when the source violates Aleph's constraints (see below), or when an Aleph result was unsatisfying — and suggest edit_video when a seedance v2v edit changed more than the user wanted.
- USE expand_video to change the video's ASPECT RATIO by outpainting beyond the frame edges ("make it vertical", "uncrop", "convert to 21:9") — this tool edits content within the frame, expand_video grows the frame.
SOURCE VIDEO CONSTRAINTS (Aleph 2.0): 2-30 seconds, up to 1080p, up to 30fps, at most 10 cuts/shot changes, conventional aspect ratio. Cost: 28 credits/second of source (56 credit minimum), so a wasted submission is expensive — that's why the keyframe preview checkpoint below is the default.
HOW THE WORKFLOW WORKS (a keyframe-driven edit is much higher quality than a text-only one):
1. CANDIDATES — call with `promptText` + `video` only (video.durationSeconds required for this stage). The tool extracts ~4 evenly spaced frames from the video and returns them inline with their timestamps. LOOK at the frames and pick the one that most clearly shows the edit target (e.g. the widest shot for a background change; a close-up of the subject for a detail change). You may also pick your own timestamp if none fit.
2. KEYFRAME PREVIEW — call again with the SAME arguments plus `keyframeTimestampSeconds`. The tool extracts that exact frame, edits it with an image model to apply your change, and returns the edited keyframe inline. SHOW it to the user and ask if they're happy before continuing (this is the default checkpoint). If they want a different look, re-call with a refined `promptText` or a different timestamp.
3. SUBMIT — once approved, call again with `keyframeImage.url` set to the edited keyframe URL and the SAME `keyframeTimestampSeconds`. The tool submits the Aleph 2.0 video edit task; the result arrives via the inline viewer / `get_task`.
SHORTCUTS:
- If the user has said they want to skip the preview/approval step, pass `skipPreview: true` at step 2 — the tool generates the keyframe AND submits the video edit in one call.
- If the user supplies their own already-edited frame, pass it directly as `keyframeImage.url` with the timestamp it came from.
- `textOnly: true` submits a text-prompt-only edit with no keyframe — lower quality, but a valid fallback when frame extraction or the keyframe edit fails.
PROMPTING (Aleph 2.0 guide): simple, targeted prompts work best — an action verb (add, remove, change, replace, re-light, re-style) plus the desired transformation, e.g. "change the outfit and bag to soft yellow". Because Aleph only changes what you ask for, precise language beats long descriptions. Use `extraMotionPrompt` ONLY for motion that isn't implied by the keyframe or the original video (e.g. "fire begins to spread up the trees") — most edits don't need it.
REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.
HANDLING USER ATTACHMENTS:
- ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `video.url`.
- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`.
- ALWAYS pass `video.durationSeconds` when you know it (from file metadata) — the candidates stage requires it, and it makes credit accounting accurate.
USER-FACING REPLIES: Talk about the edit (what's changing in the video), not the model or task machinery. Do not mention model names unless the user asks.
Parameters:
- promptText: Required. The edit instruction — what to change; everything else is preserved.
- video: `{url, durationSeconds}` — source video. URL must be a Runway-hosted /datasets/ asset URL (from complete_upload) or a public HTTPS video URL.
- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`.
- keyframeTimestampSeconds: Timestamp (seconds) of the frame that anchors the edit. Must be within the video duration. Setting this triggers the keyframe edit + preview stage.
- keyframeImage: `{url}` — an ALREADY-EDITED keyframe image (from step 2's preview, or user-supplied). Requires `keyframeTimestampSeconds`. Setting this submits the video edit.
- keyframeModel: Image model for the keyframe edit. One of `nano-banana-pro`, `nano-banana-2.1`, `nano-banana-2`, `nano-banana-2-lite`, `gpt-image-2`, `gpt-image-2.5-flare`, `gpt-image-2.5-sunburst`, `seedream-5`, `seedream-5-pro`, `seedream-5-flash`, `ideogram-4`, `ideogram-4.5`, `ideogram-4.5-precise-edit`, `grok-imagine-image-2`, `muse-image`, `fibo-1.5`, `fibo-edit-1.5`, `gen-4`, `gen-4-image-turbo`. Default: `nano-banana-pro`.
- extraMotionPrompt: Optional motion instruction for motion that isn't implied by the keyframe or original video.
- skipPreview: When true at the keyframe stage, submit the video edit immediately after generating the keyframe instead of pausing for approval. Only use when the user has explicitly opted to skip the check.
- textOnly: Submit a text-prompt-only edit with no keyframe (lower quality fallback).
The description explains the capability, when to use it, and what to expect.
Inputs95/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
promptText
Required. The edit instruction — action verb + desired change, e.g. "make the sneakers red", "remove the items on the wall behind her", "change the season to winter". Everything not described is preserved.
Clear enough to supply this argument.
video › taskId
Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated.
Clear enough to supply this argument.
video › url
Runway-hosted dataset URL (preferred) or public HTTPS video URL. Never use /mnt/data or other local file paths here.
Clear enough to supply this argument.
video › durationSeconds
Source video duration in seconds (2-30 supported). ALWAYS pass when known — required for the candidate-frames stage and used for accurate credit accounting.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
video › assetId
Runway dataset asset id for the source video. Do NOT invent this. The candidate/preview stages return a resolved `video` object that includes it — pass that object back verbatim on the follow-up calls so the tool reuses the registered dataset instead of re-uploading the video.
Clear enough to supply this argument.
videoFile › download_url
Temporary ChatGPT file download URL.
Clear enough to supply this argument.
videoFile › file_id
ChatGPT file ID.
Clear enough to supply this argument.
videoFile › mime_type
MIME type, e.g. video/mp4.
Clear enough to supply this argument.
videoFile › file_name
Original file name.
Clear enough to supply this argument.
keyframeTimestampSeconds
Timestamp (seconds) in the source video of the frame that anchors the edit. Pick it from the candidate frames this tool returns (or choose your own). Must be less than the video duration. Reuse the SAME value when later passing keyframeImage.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
keyframeImage › url
Edited keyframe image URL (Runway-hosted or public HTTPS).
Clear enough to supply this argument.
keyframeModel
Image model used to edit the keyframe. Default: nano-banana-pro.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
extraMotionPrompt
Optional motion instruction for motion that is NOT implied by the keyframe or the original video (e.g. "fire begins to spread up the trees"). Most edits do not need this.
Clear enough to supply this argument.
skipPreview
When true at the keyframe stage, submit the video edit immediately after generating the edited keyframe (no approval pause). Only set when the user explicitly opted to skip the preview.
Clear enough to supply this argument.
textOnly
Submit a text-prompt-only edit with no keyframe. Lower quality — use as a fallback when the keyframe workflow is not possible.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "edit_video",
"title": "Edit Video (Aleph 2.0)",
"description": "Edit an existing video with Runway's Aleph 2.0 in-context video editor: it changes ONLY what you ask for and preserves everything else — subjects, motion, framing, timing, background, lighting.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nPAID PLAN REQUIRED: Aleph 2.0 is not available on free Runway workspaces. On a free workspace this tool returns paid_plan_required without creating a task — follow the NEXT line in that result (`show_plans_and_credits` with `gatedCapability`) and do not retry. Tell the user that editing video needs a paid plan rather than naming the model. Do not guess a fallback from this schema.\n\nWHEN TO USE THIS vs generate_video:\n- USE edit_video (this tool) for targeted, surgical edits of existing footage where preservation matters: \"make the sneakers red\", \"change the outfit to a yellow fur coat\", \"remove the items on the wall behind her\", \"add graffiti on the wall\", \"change the season to winter\", \"relight the scene\", \"make it dark anime style\" — or whenever the user mentions Aleph or Edit Studio.\n- USE generate_video with referenceVideo (seedance-2) when the source video is more of a loose reference: motion transfer, using the clip as style/structure inspiration, or heavy re-imagining where preserving the original isn't the point. Also fall back to it when the source violates Aleph's constraints (see below), or when an Aleph result was unsatisfying — and suggest edit_video when a seedance v2v edit changed more than the user wanted.\n- USE expand_video to change the video's ASPECT RATIO by outpainting beyond the frame edges (\"make it vertical\", \"uncrop\", \"convert to 21:9\") — this tool edits content within the frame, expand_video grows the frame.\n\nSOURCE VIDEO CONSTRAINTS (Aleph 2.0): 2-30 seconds, up to 1080p, up to 30fps, at most 10 cuts/shot changes, conventional aspect ratio. Cost: 28 credits/second of source (56 credit minimum), so a wasted submission is expensive — that's why the keyframe preview checkpoint below is the default.\n\nHOW THE WORKFLOW WORKS (a keyframe-driven edit is much higher quality than a text-only one):\n1. CANDIDATES — call with `promptText` + `video` only (video.durationSeconds required for this stage). The tool extracts ~4 evenly spaced frames from the video and returns them inline with their timestamps. LOOK at the frames and pick the one that most clearly shows the edit target (e.g. the widest shot for a background change; a close-up of the subject for a detail change). You may also pick your own timestamp if none fit.\n2. KEYFRAME PREVIEW — call again with the SAME arguments plus `keyframeTimestampSeconds`. The tool extracts that exact frame, edits it with an image model to apply your change, and returns the edited keyframe inline. SHOW it to the user and ask if they're happy before continuing (this is the default checkpoint). If they want a different look, re-call with a refined `promptText` or a different timestamp.\n3. SUBMIT — once approved, call again with `keyframeImage.url` set to the edited keyframe URL and the SAME `keyframeTimestampSeconds`. The tool submits the Aleph 2.0 video edit task; the result arrives via the inline viewer / `get_task`.\n\nSHORTCUTS:\n- If the user has said they want to skip the preview/approval step, pass `skipPreview: true` at step 2 — the tool generates the keyframe AND submits the video edit in one call.\n- If the user supplies their own already-edited frame, pass it directly as `keyframeImage.url` with the timestamp it came from.\n- `textOnly: true` submits a text-prompt-only edit with no keyframe — lower quality, but a valid fallback when frame extraction or the keyframe edit fails.\n\nPROMPTING (Aleph 2.0 guide): simple, targeted prompts work best — an action verb (add, remove, change, replace, re-light, re-style) plus the desired transformation, e.g. \"change the outfit and bag to soft yellow\". Because Aleph only changes what you ask for, precise language beats long descriptions. Use `extraMotionPrompt` ONLY for motion that isn't implied by the keyframe or the original video (e.g. \"fire begins to spread up the trees\") — most edits don't need it.\n\nREUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.\n\nHANDLING USER ATTACHMENTS:\n- ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `video.url`.\n- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`.\n- ALWAYS pass `video.durationSeconds` when you know it (from file metadata) — the candidates stage requires it, and it makes credit accounting accurate.\n\nUSER-FACING REPLIES: Talk about the edit (what's changing in the video), not the model or task machinery. Do not mention model names unless the user asks.\n\nParameters:\n- promptText: Required. The edit instruction — what to change; everything else is preserved.\n- video: `{url, durationSeconds}` — source video. URL must be a Runway-hosted /datasets/ asset URL (from complete_upload) or a public HTTPS video URL.\n- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`.\n- keyframeTimestampSeconds: Timestamp (seconds) of the frame that anchors the edit. Must be within the video duration. Setting this triggers the keyframe edit + preview stage.\n- keyframeImage: `{url}` — an ALREADY-EDITED keyframe image (from step 2's preview, or user-supplied). Requires `keyframeTimestampSeconds`. Setting this submits the video edit.\n- keyframeModel: Image model for the keyframe edit. One of `nano-banana-pro`, `nano-banana-2.1`, `nano-banana-2`, `nano-banana-2-lite`, `gpt-image-2`, `gpt-image-2.5-flare`, `gpt-image-2.5-sunburst`, `seedream-5`, `seedream-5-pro`, `seedream-5-flash`, `ideogram-4`, `ideogram-4.5`, `ideogram-4.5-precise-edit`, `grok-imagine-image-2`, `muse-image`, `fibo-1.5`, `fibo-edit-1.5`, `gen-4`, `gen-4-image-turbo`. Default: `nano-banana-pro`.\n- extraMotionPrompt: Optional motion instruction for motion that isn't implied by the keyframe or original video.\n- skipPreview: When true at the keyframe stage, submit the video edit immediately after generating the keyframe instead of pausing for approval. Only use when the user has explicitly opted to skip the check.\n- textOnly: Submit a text-prompt-only edit with no keyframe (lower quality fallback).",
"inputSchema": {
"type": "object",
"required": [
"promptText"
],
"properties": {
"video": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Runway-hosted dataset URL (preferred) or public HTTPS video URL. Never use /mnt/data or other local file paths here."
},
"taskId": {
"type": "string",
"description": "Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated."
},
"assetId": {
"type": "string",
"description": "Runway dataset asset id for the source video. Do NOT invent this. The candidate/preview stages return a resolved `video` object that includes it — pass that object back verbatim on the follow-up calls so the tool reuses the registered dataset instead of re-uploading the video."
},
"durationSeconds": {
"type": "number",
"description": "Source video duration in seconds (2-30 supported). ALWAYS pass when known — required for the candidate-frames stage and used for accurate credit accounting."
}
},
"description": "Source video to edit. URL must be a Runway-hosted /datasets/<UUID>.<ext> URL from complete_upload or a public HTTPS URL. Do not pass ChatGPT local paths like /mnt/data/...; use videoFile instead.",
"additionalProperties": false
},
"textOnly": {
"type": "boolean",
"description": "Submit a text-prompt-only edit with no keyframe. Lower quality — use as a fallback when the keyframe workflow is not possible."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"videoFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. video/mp4."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source video. Prefer video.url when you already have a Runway-hosted or public URL.",
"additionalProperties": false
},
"promptText": {
"type": "string",
"description": "Required. The edit instruction — action verb + desired change, e.g. \"make the sneakers red\", \"remove the items on the wall behind her\", \"change the season to winter\". Everything not described is preserved."
},
"skipPreview": {
"type": "boolean",
"description": "When true at the keyframe stage, submit the video edit immediately after generating the edited keyframe (no approval pause). Only set when the user explicitly opted to skip the preview."
},
"keyframeImage": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"description": "Edited keyframe image URL (Runway-hosted or public HTTPS)."
}
},
"description": "An already-edited keyframe image — typically the preview URL returned by the keyframe stage, after user approval. Requires keyframeTimestampSeconds (the timestamp the frame was extracted at). Setting this submits the Aleph video edit.",
"additionalProperties": false
},
"keyframeModel": {
"enum": [
"nano-banana-pro",
"nano-banana-2.1",
"nano-banana-2",
"nano-banana-2-lite",
"gpt-image-2",
"gpt-image-2.5-flare",
"gpt-image-2.5-sunburst",
"seedream-5",
"seedream-5-pro",
"seedream-5-flash",
"ideogram-4",
"ideogram-4.5",
"ideogram-4.5-precise-edit",
"grok-imagine-image-2",
"muse-image",
"fibo-1.5",
"fibo-edit-1.5",
"gen-4",
"gen-4-image-turbo"
],
"type": "string",
"description": "Image model used to edit the keyframe. Default: nano-banana-pro."
},
"extraMotionPrompt": {
"type": "string",
"description": "Optional motion instruction for motion that is NOT implied by the keyframe or the original video (e.g. \"fire begins to spread up the trees\"). Most edits do not need this."
},
"keyframeTimestampSeconds": {
"type": "number",
"description": "Timestamp (seconds) in the source video of the frame that anchors the edit. Pick it from the candidate frames this tool returns (or choose your own). Must be less than the video duration. Reuse the SAME value when later passing keyframeImage."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Edit Video (Aleph 2.0)",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
Expand internal abbreviations and remove technical prefixes from the display title.
Description99/100
Expand a video to a new aspect ratio by GENERATING new content beyond the original frame edges (video outpainting / uncrop / reframe) with Runway's Aleph 2.0. The original pixels stay at their scale and position; the model fills the new regions so lighting, motion, and subjects continue seamlessly past the old edges.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
PAID PLAN REQUIRED: Aleph 2.0 is not available on free Runway workspaces. On a free workspace this tool returns paid_plan_required without creating a task — follow the NEXT line in that result (`show_plans_and_credits` with `gatedCapability`) and do not retry. Tell the user that expanding video needs a paid plan rather than naming the model. Do not guess a fallback from this schema.
WHEN TO USE THIS:
- The user wants to change a video's aspect ratio WITHOUT cropping: "make this vertical for TikTok/Reels", "convert 16:9 to 9:16", "make it cinematic 21:9", "uncrop this video", "expand the frame".
- NOT for content edits (use `edit_video`) and NOT for generating new footage (use `generate_video`). This tool only grows the canvas.
SOURCE VIDEO CONSTRAINTS (Aleph 2.0): 2-30 seconds, up to 1080p, up to 30fps, at most 10 cuts/shot changes. Cost: 28 credits/second of source (56 credit minimum).
REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.
HANDLING USER ATTACHMENTS:
- ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `video.url`.
- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`.
- ALWAYS pass `video.durationSeconds` when you know it (from file metadata) — it makes credit accounting accurate.
USER-FACING REPLIES: Talk about the reframe (what ratio the video is becoming and what will fill the new space), not the model or task machinery. Do not mention model names unless the user asks.
Parameters:
- targetRatio: Required. The aspect ratio to expand to. One of `16:9`, `9:16`, `1:1`, `4:3`, `3:4`, `21:9`, `2:3`, `3:2`, `4:5`, `5:4`, `5:3`, `3:5`. Pick a ratio DIFFERENT from the source video's — expanding to the same ratio is a no-op.
- video: `{url, durationSeconds}` — source video. URL must be a Runway-hosted /datasets/ asset URL (from complete_upload) or a public HTTPS video URL.
- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`.
- promptText: Optional. What should appear in the newly generated regions (e.g. "a sandy beach extends to both sides"). Omit to let the model continue the existing scene — that is the right default for most requests.
The description explains the capability, when to use it, and what to expect.
Inputs91/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
targetRatio
Aspect ratio to expand the video to. Must differ from the source video's ratio.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
video › taskId
Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated.
Clear enough to supply this argument.
video › url
Runway-hosted dataset URL (preferred) or public HTTPS video URL. Never use /mnt/data or other local file paths here.
Clear enough to supply this argument.
video › durationSeconds
Source video duration in seconds (2-30 supported). ALWAYS pass when known — used for accurate credit accounting.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
video › assetId
Runway dataset asset id for the source video. Do NOT invent this — only pass a value a previous expand_video or edit_video call returned for the SAME video, so the tool reuses the registered dataset instead of re-uploading.
Clear enough to supply this argument.
videoFile › download_url
Temporary ChatGPT file download URL.
Clear enough to supply this argument.
videoFile › file_id
ChatGPT file ID.
Explain the meaning of this value, not just its name.
videoFile › mime_type
MIME type, e.g. video/mp4.
Clear enough to supply this argument.
videoFile › file_name
Original file name.
Clear enough to supply this argument.
promptText
Optional. Describe what the newly generated regions should contain. Omit to let the model continue the existing scene (recommended default).
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "expand_video",
"title": "Expand Video (Aleph 2.0)",
"description": "Expand a video to a new aspect ratio by GENERATING new content beyond the original frame edges (video outpainting / uncrop / reframe) with Runway's Aleph 2.0. The original pixels stay at their scale and position; the model fills the new regions so lighting, motion, and subjects continue seamlessly past the old edges.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nPAID PLAN REQUIRED: Aleph 2.0 is not available on free Runway workspaces. On a free workspace this tool returns paid_plan_required without creating a task — follow the NEXT line in that result (`show_plans_and_credits` with `gatedCapability`) and do not retry. Tell the user that expanding video needs a paid plan rather than naming the model. Do not guess a fallback from this schema.\n\nWHEN TO USE THIS:\n- The user wants to change a video's aspect ratio WITHOUT cropping: \"make this vertical for TikTok/Reels\", \"convert 16:9 to 9:16\", \"make it cinematic 21:9\", \"uncrop this video\", \"expand the frame\".\n- NOT for content edits (use `edit_video`) and NOT for generating new footage (use `generate_video`). This tool only grows the canvas.\n\nSOURCE VIDEO CONSTRAINTS (Aleph 2.0): 2-30 seconds, up to 1080p, up to 30fps, at most 10 cuts/shot changes. Cost: 28 credits/second of source (56 credit minimum).\n\nREUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.\n\nHANDLING USER ATTACHMENTS:\n- ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `video.url`.\n- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`.\n- ALWAYS pass `video.durationSeconds` when you know it (from file metadata) — it makes credit accounting accurate.\n\nUSER-FACING REPLIES: Talk about the reframe (what ratio the video is becoming and what will fill the new space), not the model or task machinery. Do not mention model names unless the user asks.\n\nParameters:\n- targetRatio: Required. The aspect ratio to expand to. One of `16:9`, `9:16`, `1:1`, `4:3`, `3:4`, `21:9`, `2:3`, `3:2`, `4:5`, `5:4`, `5:3`, `3:5`. Pick a ratio DIFFERENT from the source video's — expanding to the same ratio is a no-op.\n- video: `{url, durationSeconds}` — source video. URL must be a Runway-hosted /datasets/ asset URL (from complete_upload) or a public HTTPS video URL.\n- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`.\n- promptText: Optional. What should appear in the newly generated regions (e.g. \"a sandy beach extends to both sides\"). Omit to let the model continue the existing scene — that is the right default for most requests.",
"inputSchema": {
"type": "object",
"required": [
"targetRatio"
],
"properties": {
"video": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Runway-hosted dataset URL (preferred) or public HTTPS video URL. Never use /mnt/data or other local file paths here."
},
"taskId": {
"type": "string",
"description": "Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated."
},
"assetId": {
"type": "string",
"description": "Runway dataset asset id for the source video. Do NOT invent this — only pass a value a previous expand_video or edit_video call returned for the SAME video, so the tool reuses the registered dataset instead of re-uploading."
},
"durationSeconds": {
"type": "number",
"description": "Source video duration in seconds (2-30 supported). ALWAYS pass when known — used for accurate credit accounting."
}
},
"description": "Source video to expand. URL must be a Runway-hosted /datasets/<UUID>.<ext> URL from complete_upload or a public HTTPS URL. Do not pass ChatGPT local paths like /mnt/data/...; use videoFile instead.",
"additionalProperties": false
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"videoFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. video/mp4."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source video. Prefer video.url when you already have a Runway-hosted or public URL.",
"additionalProperties": false
},
"promptText": {
"type": "string",
"description": "Optional. Describe what the newly generated regions should contain. Omit to let the model continue the existing scene (recommended default)."
},
"targetRatio": {
"enum": [
"16:9",
"9:16",
"1:1",
"4:3",
"3:4",
"21:9",
"2:3",
"3:2",
"4:5",
"5:4",
"5:3",
"3:5"
],
"type": "string",
"description": "Aspect ratio to expand the video to. Must differ from the source video's ratio."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Expand Video (Aleph 2.0)",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description97/100
Generate a multi-shot video — 3 to 5 connected scenes from a single story or per-shot prompts. Powered by Kling 3.0 (standard at 720p, pro at 1080p). No `model` parameter — resolution selects the engine.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
Use this when the user wants a short narrative or sequence:
- "a 10-second mini-story about a cat finding a friend"
- "a multi-scene product ad: closeup, lifestyle, hero shot"
- "transition through 3 environments: forest -> desert -> beach"
For a single continuous shot, use `generate_video` instead. For editing/restyling an existing video, also use `generate_video` (with `referenceVideo`). To join clips that already exist into one video, use `stitch_videos`.
REQUIRES A PAID RUNWAY WORKSPACE. On free workspaces this tool returns an account-limitation error without creating a task; do not retry.
MODES:
1. `auto` (DEFAULT) — Pass a single `storyPrompt` (or `promptText`, accepted as an alias) and the workflow plans the shots for you. Best when the user describes the *story* not the individual scenes. Example: `mode: "auto", storyPrompt: "a barista discovers their cafe has been frozen in time"`.
2. `custom` — Pass per-shot prompts as `shots: [{prompt}, ...]` (3-5 shots). Total `duration` is divided evenly across them. Best when the user has specific ideas for each scene. Example: `mode: "custom", shots: [{prompt: "wide shot of cafe"}, {prompt: "barista looks up, surprised"}, {prompt: "close on a frozen droplet of coffee"}]`.
Optional `firstSceneImage` anchors the opening frame for either mode.
Availability is checked against the current Runway workspace. Multi-shot tasks routinely take 5-10 minutes.
USER-FACING REPLIES: DO NOT mention the underlying engine (Kling 3.0 / Pro) to the user unless they explicitly ask which model was used. Talk about the multi-shot story — the scenes, transitions, and overall narrative — not the engine.
Parameters:
- mode: `'auto'` (default) or `'custom'`.
- storyPrompt: Required for `mode='auto'`. Full story / overall scene description.
- shots: Required for `mode='custom'`. Array of `{prompt}`, 3-5 entries.
- duration: Total seconds — `5` | `10` (default) | `15`.
- aspectRatio: `'16:9'` (default) | `'1:1'` | `'9:16'`. `ratio` is accepted as an alias.
- resolution: `'720p'` (default, Kling 3.0 Standard) | `'1080p'` (Kling 3.0 Pro).
- sound: Boolean, generate audio (default true).
- firstSceneImage: Optional `{url, assetId?}` to anchor the first frame. If `assetId` is omitted, the URL is imported to a Runway dataset asset before submission because the multi-shot workflow requires a resolvable asset reference.
The description explains the capability, when to use it, and what to expect.
Inputs67/100
10 of 12 inputs have descriptions. Add descriptions to the missing inputs, including where to obtain IDs and how to supply values.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
mode
'auto' (default) or 'custom'.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
storyPrompt
Required for mode='auto'. Full story or scene description. promptText is also accepted as an alias.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
promptText
Alias for storyPrompt in mode='auto' for consistency with other generation tools.
Clear enough to supply this argument.
shots › items › prompt
Not provided
Missing description. Explain what this value means and how to supply it.
duration
Total duration in seconds: 5, 10 (default), or 15.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
aspectRatio
'16:9' (default), '1:1', or '9:16'. ratio is also accepted as an alias.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
ratio
Alias for aspectRatio for consistency with other generation tools.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
resolution
'720p' (default, Kling 3.0 Standard) or '1080p' (Kling 3.0 Pro).
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
sound
Generate audio (default true).
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
firstSceneImage › url
Not provided
Missing description. Explain what this value means and how to supply it.
firstSceneImage › assetId
Optional Runway dataset asset ID. If omitted, the MCP server imports the URL to create one before submitting.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "generate_multishot_video",
"title": "Generate Multi-Shot Video",
"description": "Generate a multi-shot video — 3 to 5 connected scenes from a single story or per-shot prompts. Powered by Kling 3.0 (standard at 720p, pro at 1080p). No `model` parameter — resolution selects the engine.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nUse this when the user wants a short narrative or sequence:\n- \"a 10-second mini-story about a cat finding a friend\"\n- \"a multi-scene product ad: closeup, lifestyle, hero shot\"\n- \"transition through 3 environments: forest -> desert -> beach\"\n\nFor a single continuous shot, use `generate_video` instead. For editing/restyling an existing video, also use `generate_video` (with `referenceVideo`). To join clips that already exist into one video, use `stitch_videos`.\n\nREQUIRES A PAID RUNWAY WORKSPACE. On free workspaces this tool returns an account-limitation error without creating a task; do not retry.\n\nMODES:\n1. `auto` (DEFAULT) — Pass a single `storyPrompt` (or `promptText`, accepted as an alias) and the workflow plans the shots for you. Best when the user describes the *story* not the individual scenes. Example: `mode: \"auto\", storyPrompt: \"a barista discovers their cafe has been frozen in time\"`.\n2. `custom` — Pass per-shot prompts as `shots: [{prompt}, ...]` (3-5 shots). Total `duration` is divided evenly across them. Best when the user has specific ideas for each scene. Example: `mode: \"custom\", shots: [{prompt: \"wide shot of cafe\"}, {prompt: \"barista looks up, surprised\"}, {prompt: \"close on a frozen droplet of coffee\"}]`.\n\nOptional `firstSceneImage` anchors the opening frame for either mode.\n\nAvailability is checked against the current Runway workspace. Multi-shot tasks routinely take 5-10 minutes.\n\nUSER-FACING REPLIES: DO NOT mention the underlying engine (Kling 3.0 / Pro) to the user unless they explicitly ask which model was used. Talk about the multi-shot story — the scenes, transitions, and overall narrative — not the engine.\n\nParameters:\n- mode: `'auto'` (default) or `'custom'`.\n- storyPrompt: Required for `mode='auto'`. Full story / overall scene description.\n- shots: Required for `mode='custom'`. Array of `{prompt}`, 3-5 entries.\n- duration: Total seconds — `5` | `10` (default) | `15`.\n- aspectRatio: `'16:9'` (default) | `'1:1'` | `'9:16'`. `ratio` is accepted as an alias.\n- resolution: `'720p'` (default, Kling 3.0 Standard) | `'1080p'` (Kling 3.0 Pro).\n- sound: Boolean, generate audio (default true).\n- firstSceneImage: Optional `{url, assetId?}` to anchor the first frame. If `assetId` is omitted, the URL is imported to a Runway dataset asset before submission because the multi-shot workflow requires a resolvable asset reference.",
"inputSchema": {
"type": "object",
"properties": {
"mode": {
"enum": [
"auto",
"custom"
],
"type": "string",
"description": "'auto' (default) or 'custom'."
},
"ratio": {
"enum": [
"16:9",
"1:1",
"9:16"
],
"type": "string",
"description": "Alias for aspectRatio for consistency with other generation tools."
},
"shots": {
"type": "array",
"items": {
"type": "object",
"required": [
"prompt"
],
"properties": {
"prompt": {
"type": "string"
}
},
"additionalProperties": false
},
"maxItems": 5,
"minItems": 3,
"description": "Required for mode='custom'. Per-shot prompts (3-5 shots)."
},
"sound": {
"type": "boolean",
"description": "Generate audio (default true)."
},
"duration": {
"enum": [
5,
10,
15
],
"type": "number",
"description": "Total duration in seconds: 5, 10 (default), or 15."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"promptText": {
"type": "string",
"description": "Alias for storyPrompt in mode='auto' for consistency with other generation tools."
},
"resolution": {
"enum": [
"720p",
"1080p"
],
"type": "string",
"description": "'720p' (default, Kling 3.0 Standard) or '1080p' (Kling 3.0 Pro)."
},
"aspectRatio": {
"enum": [
"16:9",
"1:1",
"9:16"
],
"type": "string",
"description": "'16:9' (default), '1:1', or '9:16'. ratio is also accepted as an alias."
},
"storyPrompt": {
"type": "string",
"description": "Required for mode='auto'. Full story or scene description. promptText is also accepted as an alias."
},
"firstSceneImage": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string"
},
"assetId": {
"type": "string",
"description": "Optional Runway dataset asset ID. If omitted, the MCP server imports the URL to create one before submitting."
}
},
"description": "Optional first-scene anchor image.",
"additionalProperties": false
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Generate Multi-Shot Video",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Generate music from a text description — songs, instrumentals, scores, loops, stings — and return an MP3. Describe genre, mood, instrumentation, tempo, structure, and (for songs) lyrics or vocal style in `promptText`.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
MODELS (pick with `model`; default `lyria-3-pro`):
- `lyria-3-pro` (8 credits): Full-length music — songs, scores, instrumentals. Use for a complete track or soundtrack.
- `lyria-3-clip` (4 credits): Short music clips — stings, loops, background beds, jingles. Use for intros, transitions, or a bed under a short video.
Output length is fixed per model — there is no duration parameter. Use `lyria-3-clip` for short beds/stings and `lyria-3-pro` for a full track.
WHEN TO USE:
- "make me a song about…", "write a lo-fi beat", "background music for my video", "an orchestral score that builds", "a 10-second jingle".
- NOT for speech/voiceover, NOT for sound effects, NOT for adding a soundtrack to an existing video (generate the music here, then combine externally).
PAID PLAN REQUIRED: music generation is not available on free Runway workspaces. On a free workspace this tool returns paid_plan_required without creating a task — follow the NEXT line in that result (`show_plans_and_credits` with `gatedCapability`) and do not retry. Tell the user that generating music needs a paid plan rather than naming the model. `whoami` does not list music models; do not treat an empty `availableVideoModels` as a music signal.
AVAILABILITY: some workspaces have this model disabled by an admin. If the result is an account_limitation with reason admin_disabled_model, tell the user and stop — do not retry with the other model.
PROMPTING TIPS: lead with genre and mood, then instrumentation and tempo, then structure ("intro, verse, chorus, outro"). For songs with vocals, include the lyrics or the theme and vocal style. For instrumental beds say "instrumental, no vocals". Max 5000 characters.
USER-FACING REPLIES: Talk about the music (genre, mood, what it's for), not the model or task machinery. Do not mention Lyria or Google unless the user asks. The output is an MP3 the viewer plays inline; headless agents get the URL from `get_task`.
Parameters:
- promptText: Required. What the music should sound like. Max 5000 characters.
- model: Optional. `lyria-3-pro` (default, full track) or `lyria-3-clip` (short clip).
- name: Optional. Display name for the track in the user's Runway library. Defaults to the prompt.
The description explains the capability, when to use it, and what to expect.
Inputs81/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
promptText
Description of the music: genre, mood, instrumentation, tempo, structure, lyrics/vocal style. Max 5000 characters.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
model
Music model. 'lyria-3-pro' (default) for a full-length track; 'lyria-3-clip' for a short sting/loop/bed.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
name
Optional display name for the track in the user's Runway library. Defaults to the prompt.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "generate_music",
"title": "Generate Music",
"description": "Generate music from a text description — songs, instrumentals, scores, loops, stings — and return an MP3. Describe genre, mood, instrumentation, tempo, structure, and (for songs) lyrics or vocal style in `promptText`.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nMODELS (pick with `model`; default `lyria-3-pro`):\n- `lyria-3-pro` (8 credits): Full-length music — songs, scores, instrumentals. Use for a complete track or soundtrack.\n- `lyria-3-clip` (4 credits): Short music clips — stings, loops, background beds, jingles. Use for intros, transitions, or a bed under a short video.\nOutput length is fixed per model — there is no duration parameter. Use `lyria-3-clip` for short beds/stings and `lyria-3-pro` for a full track.\n\nWHEN TO USE:\n- \"make me a song about…\", \"write a lo-fi beat\", \"background music for my video\", \"an orchestral score that builds\", \"a 10-second jingle\".\n- NOT for speech/voiceover, NOT for sound effects, NOT for adding a soundtrack to an existing video (generate the music here, then combine externally).\n\nPAID PLAN REQUIRED: music generation is not available on free Runway workspaces. On a free workspace this tool returns paid_plan_required without creating a task — follow the NEXT line in that result (`show_plans_and_credits` with `gatedCapability`) and do not retry. Tell the user that generating music needs a paid plan rather than naming the model. `whoami` does not list music models; do not treat an empty `availableVideoModels` as a music signal.\n\nAVAILABILITY: some workspaces have this model disabled by an admin. If the result is an account_limitation with reason admin_disabled_model, tell the user and stop — do not retry with the other model.\n\nPROMPTING TIPS: lead with genre and mood, then instrumentation and tempo, then structure (\"intro, verse, chorus, outro\"). For songs with vocals, include the lyrics or the theme and vocal style. For instrumental beds say \"instrumental, no vocals\". Max 5000 characters.\n\nUSER-FACING REPLIES: Talk about the music (genre, mood, what it's for), not the model or task machinery. Do not mention Lyria or Google unless the user asks. The output is an MP3 the viewer plays inline; headless agents get the URL from `get_task`.\n\nParameters:\n- promptText: Required. What the music should sound like. Max 5000 characters.\n- model: Optional. `lyria-3-pro` (default, full track) or `lyria-3-clip` (short clip).\n- name: Optional. Display name for the track in the user's Runway library. Defaults to the prompt.",
"inputSchema": {
"type": "object",
"required": [
"promptText"
],
"properties": {
"name": {
"type": "string",
"maxLength": 500,
"description": "Optional display name for the track in the user's Runway library. Defaults to the prompt."
},
"model": {
"enum": [
"lyria-3-pro",
"lyria-3-clip"
],
"type": "string",
"description": "Music model. 'lyria-3-pro' (default) for a full-length track; 'lyria-3-clip' for a short sting/loop/bed."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"promptText": {
"type": "string",
"maxLength": 5000,
"description": "Description of the music: genre, mood, instrumentation, tempo, structure, lyrics/vocal style. Max 5000 characters."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Generate Music",
"readOnlyHint": false,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Generate a sound effect from a text description — foley, ambience, impacts, UI sounds, creature noises, whooshes, loops — and return an MP3. Describe the sound source, texture, environment, and how it evolves in `promptText`.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
COST: 1 credit per second of requested `duration` (rounded up); 2 credits when `duration` is omitted and the model picks the length. Durations run 0.5–30 seconds.
WHEN TO USE:
- "a door creaking open", "rain on a tin roof, 10 seconds", "sci-fi laser blast", "a looping crackling campfire", "footsteps on gravel", "a notification chime".
- NOT for music (use `generate_music`), NOT for speech or voiceover (use `generate_speech`), NOT for adding sound to an existing video (generate the effect here, then combine externally).
AVAILABILITY: sound effects are behind a workspace feature flag. If the API rejects the task type, tell the user the feature is not enabled on their workspace and stop — do not retry.
PROMPTING TIPS: name the source first ("heavy wooden door"), then the action ("slams shut"), then the space ("in a stone hallway, long reverb tail"). Mention pacing for longer effects ("starts quiet, builds to a roar"). Set `loop: true` for ambience or beds meant to repeat seamlessly. Max 450 characters.
USER-FACING REPLIES: Talk about the sound (what it is, how long, what it's for), not the model or task machinery. Do not mention ElevenLabs unless the user asks. The output is an MP3 the viewer plays inline; headless agents get the URL from `get_task`.
Parameters:
- promptText: Required. What the sound effect should be. Max 450 characters.
- duration: Optional. Length in seconds, 0.5–30. Omit to let the model choose.
- loop: Optional. true to design the effect to loop seamlessly (ambience, beds). Default false.
- name: Optional. Display name for the asset in the user's Runway library. Defaults to the prompt.
The description explains the capability, when to use it, and what to expect.
Inputs85/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
promptText
Description of the sound effect: source, action, environment, texture, how it evolves. Max 450 characters.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
duration
Length in seconds (0.5–30). Costs 1 credit per second, rounded up. Omit to let the model pick (2 credits).
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
loop
Design the effect to loop seamlessly (ambience, beds, engines). Default false.
Clear enough to supply this argument.
name
Optional display name for the asset in the user's Runway library. Defaults to the prompt.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "generate_sound_effect",
"title": "Generate Sound Effect",
"description": "Generate a sound effect from a text description — foley, ambience, impacts, UI sounds, creature noises, whooshes, loops — and return an MP3. Describe the sound source, texture, environment, and how it evolves in `promptText`.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nCOST: 1 credit per second of requested `duration` (rounded up); 2 credits when `duration` is omitted and the model picks the length. Durations run 0.5–30 seconds.\n\nWHEN TO USE:\n- \"a door creaking open\", \"rain on a tin roof, 10 seconds\", \"sci-fi laser blast\", \"a looping crackling campfire\", \"footsteps on gravel\", \"a notification chime\".\n- NOT for music (use `generate_music`), NOT for speech or voiceover (use `generate_speech`), NOT for adding sound to an existing video (generate the effect here, then combine externally).\n\nAVAILABILITY: sound effects are behind a workspace feature flag. If the API rejects the task type, tell the user the feature is not enabled on their workspace and stop — do not retry.\n\nPROMPTING TIPS: name the source first (\"heavy wooden door\"), then the action (\"slams shut\"), then the space (\"in a stone hallway, long reverb tail\"). Mention pacing for longer effects (\"starts quiet, builds to a roar\"). Set `loop: true` for ambience or beds meant to repeat seamlessly. Max 450 characters.\n\nUSER-FACING REPLIES: Talk about the sound (what it is, how long, what it's for), not the model or task machinery. Do not mention ElevenLabs unless the user asks. The output is an MP3 the viewer plays inline; headless agents get the URL from `get_task`.\n\nParameters:\n- promptText: Required. What the sound effect should be. Max 450 characters.\n- duration: Optional. Length in seconds, 0.5–30. Omit to let the model choose.\n- loop: Optional. true to design the effect to loop seamlessly (ambience, beds). Default false.\n- name: Optional. Display name for the asset in the user's Runway library. Defaults to the prompt.",
"inputSchema": {
"type": "object",
"required": [
"promptText"
],
"properties": {
"loop": {
"type": "boolean",
"description": "Design the effect to loop seamlessly (ambience, beds, engines). Default false."
},
"name": {
"type": "string",
"maxLength": 500,
"description": "Optional display name for the asset in the user's Runway library. Defaults to the prompt."
},
"duration": {
"type": "number",
"maximum": 30,
"minimum": 0.5,
"description": "Length in seconds (0.5–30). Costs 1 credit per second, rounded up. Omit to let the model pick (2 credits)."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"promptText": {
"type": "string",
"maxLength": 450,
"description": "Description of the sound effect: source, action, environment, texture, how it evolves. Max 450 characters."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Generate Sound Effect",
"readOnlyHint": false,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description96/100
Turn text into spoken audio (text-to-speech) with one of Runway's preset voices and return an MP3. Pass the exact words to speak in `text` and pick a `voice` by name.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
COST: `eleven_v3` and `eleven_multilingual_v2`: 1 credit per 50 characters of `text`, rounded up (minimum 1), max 5000 characters per call. `eleven_v4`: 2.2 credits per 1,000 characters until October 12, 2026 PT, then 5 credits per 1,000 characters, rounded up (minimum 1), max 2500 characters per call. `fish-drama-3`: 1 credit per 250 characters, rounded up (minimum 1) (provisional), max 5000 characters per call. Split longer scripts into several calls.
AVAILABILITY: `eleven_v4` and `fish-drama-3` need a paid plan and are not enabled on every workspace yet. If one is not enabled, tell the user and offer `eleven_v3` instead — do not retry that model.
MODELS (pick with `model`; default `eleven_v3`):
- `eleven_v3`: Most expressive. Understands inline delivery tags in the text such as [whispers], [laughs], [sighs], [excited]. Best for dialogue, characters, and emotive narration.
- `eleven_multilingual_v2`: Steadier and more predictable; ignores delivery tags. Best for long neutral narration, product voiceover, and non-English text.
- `eleven_v4`: Newest expressive model with inline delivery tags ([laughs], [whispers], [pause]) and a much lower per-character price. Max 2500 characters per call. The only model with multi-speaker `dialogue` and library voices from `search_voices`. Paid plans only, and not yet enabled on every workspace.
- `fish-drama-3`: Fish Audio's expressive drama model. Direct delivery with bracketed cues such as [barely holding it together] or [small laugh], or wrap words in <whisper>...</whisper>. Uses its own seven English voices (see FISH AUDIO VOICES); no voice controls, speed, languageCode, voiceId or dialogue. Max 5000 characters per call. Paid plans only, and not yet enabled on every workspace.
VOICES for the Eleven models (pick with `voice`; default `Leslie`). Choose by the user's description ("deep male narrator", "cute cartoon voice", "British woman, calm"); do not ask the user to pick from the full list unless they want to browse:
- Maya: feminine, young, african american accent, chill (conversational)
- Arjun: masculine, middle aged, indian accent, intense (narrator)
- Serene: feminine, young, american accent, calm (narrator)
- Bernard: masculine, middle aged, american accent, confident (animated)
- Billy: masculine, middle aged, american accent, hyped (animated)
- Mark: masculine, middle aged, american accent, excited (animated)
- Clint: masculine, middle aged, american accent, husky (animated)
- Mabel: feminine, middle aged, us southern accent, relaxed (narrator)
- Chad: masculine, young, american accent, excited (animated)
- Leslie: feminine, middle aged, american accent, professional (narrator)
- Eleanor: feminine, middle aged, canadian accent, classy (narrator)
- Elias: masculine, old, american accent, calm (narrator)
- Elliot: masculine, middle aged, american accent, anxious (animated)
- Grungle: masculine, middle aged, american accent, raspy (animated)
- Brodie: masculine, young, american accent, chill (animated)
- Sandra: feminine, middle aged, american accent, sassy (conversational)
- Kirk: masculine, middle aged, british accent, rough (animated)
- Kylie: feminine, young, australian accent, confident (influencer)
- Lara: feminine, young, american accent, confident (influencer)
- Lisa: feminine, young, german accent, casual (conversational)
- Malachi: masculine, middle aged, american accent, confident (animated)
- Marlene: feminine, middle aged, british accent, classy (narrator)
- Martin: masculine, middle aged, american accent, rough (animated)
- Miriam: feminine, middle aged, british accent, calm (narrator)
- Monster: masculine, old, american accent, deep (animated)
- Paula: feminine, middle aged, british accent, calm (conversational)
- Pip: feminine, young, american accent, cute (animated)
- Rusty: masculine, old, british accent, raspy (animated)
- Ragnar: masculine, middle aged, american accent, deep (animated)
- Xylar: neutral, middle aged, american accent, modulated (animated)
- Maggie: feminine, young, american accent, calm (narration)
- Jack: masculine, old, irish accent, sailor (video games)
- Katie: feminine, young, american accent, soft (news)
- Noah: masculine, young, american accent, calm (meditation)
- James: masculine, middle aged, australian accent, casual (conversational)
- Rina: feminine, young, american accent, calm (meditation)
- Ella: feminine, young, american accent, emotional (narration)
- Frank: masculine, young, american accent, deep (narration)
- Claudia: feminine, young, british-swedish accent, seductive (characters)
- Niki: feminine, middle aged, american accent, friendly (narration)
- Vincent: masculine, old, australian accent, calm (news)
- Tom: masculine, middle aged, british accent, authoritative (news)
- Wanda: feminine, middle aged, american accent, pleasant (interactive)
- Benjamin: masculine, middle aged, american accent, deep (narration)
- Kiana: feminine, young, american accent, whisper (audiobook)
- Rachel: feminine, young, english-swedish accent, childish (animation)
FISH AUDIO VOICES: `fish-drama-3` only accepts these English voices (default `Sarah`); the voices above do not work with it:
- Sarah: an engaged speaker, female, narration
- Adrian: a steady and reliable narrator, male
- Selene: a meditative female voice, meditation, ASMR
- Ethan: a curious explainer
- Hannah: a conversation specialist
- Jordan: a motivational speaker
- Laura: a confident female narrator
WHEN TO USE:
- "read this aloud", "voiceover for my video: …", "narrate this script in a British accent", "say 'welcome back' in a robotic voice".
- "a conversation between a detective and a suspect", "a two-host podcast intro": `eleven_v4` with `dialogue`.
- NOT for music (use `generate_music`), NOT for sound effects (use `generate_sound_effect`), NOT for cloning a specific person's voice, NOT for translating or re-voicing an existing audio/video file.
PROMPTING TIPS: `text` is spoken verbatim — do not include stage directions as plain prose. With `eleven_v3` or `eleven_v4` you may add bracketed delivery tags inline, e.g. "[whispers] Don't move. [pause] Did you hear that?"; `eleven_multilingual_v2` ignores tags. `eleven_v4` rejects SSML `<break>` tags — use [pause]. Use punctuation and line breaks to control pacing. Set `languageCode` when the text is not English and pronunciation matters. Use a 2-letter ISO 639-1 code (`es`, `de`, `pt`) or a 3-letter code for eleven_v3 (`eng`, `fil`) — never a region tag like `en-US` or `pt-BR`.
USER-FACING REPLIES: Talk about the voiceover (which voice, what it says, how long), not the model or task machinery. Do not mention ElevenLabs unless the user asks. The output is an MP3 the viewer plays inline; headless agents get the URL from `get_task`.
Parameters:
- text: Required unless `dialogue` is given. The words to speak, verbatim. Max 5000 characters (2500 for `eleven_v4`).
- voice: Optional. Preset voice name from the lists above. Default `Leslie`, or `Sarah` for `fish-drama-3`.
- voiceId: Optional, `eleven_v4` only. A voice from `search_voices` when no preset fits the user's description.
- dialogue: Optional, `eleven_v4` only. Ordered lines `[{ voice | voiceId, text }]` for a conversation between up to 10 voices in one MP3 (max 2,000 characters in total). Use instead of `text`; lines with the same voice are the same speaker.
- model: Optional. `eleven_v3` (default, expressive, supports tags), `eleven_multilingual_v2` (steady), `eleven_v4` (expressive, cheaper, paid plans, max 2500 characters), or `fish-drama-3` (Fish Audio drama voices, paid plans, max 5000 characters).
- speed: Optional. 0.7–1.2; 1.0 is normal.
- languageCode: Optional. 2-letter ISO 639-1 code (e.g. `es`, `de`, `pt`) or 3-letter code for eleven_v3 (e.g. `eng`, `fil`). Do not include a region.
- stability, similarityBoost, style, useSpeakerBoost, seed, previousText, nextText: Optional voice controls (see each parameter); not for `fish-drama-3`. Leave them unset unless the user asks for the effect.
- name: Optional. Display name for the asset in the user's Runway library. Defaults to the text.
The description explains the capability, when to use it, and what to expect.
Inputs83/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
text
The exact words to speak. Max 5000 characters, or 2500 for eleven_v4.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
voice
Preset voice name. Default 'Leslie', or 'Sarah' for fish-drama-3, which only takes Sarah, Adrian, Selene, Ethan, Hannah, Jordan, Laura. Pick from the descriptions in the tool description.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
model
Speech model. 'eleven_v3' (default) is most expressive and supports inline [tags]; 'eleven_multilingual_v2' is steadier; 'eleven_v4' supports [tags], costs less per character, needs a paid plan, and takes at most 2500 characters; 'fish-drama-3' uses Fish Audio voices, needs a paid plan, and takes no voice controls.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
speed
Speech speed multiplier, 0.7–1.2. 1.0 is normal.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
languageCode
2-letter ISO 639-1 code (e.g. 'en', 'es', 'pt') or 3-letter code for eleven_v3 (e.g. 'eng', 'fil'). Do not send a region/locale such as 'en-US' or 'pt-BR'.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
stability
0–1: lower allows a broader emotional range, higher is steadier.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
similarityBoost
0–1: how closely the output tracks the original voice.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
style
0–1 style exaggeration: higher amplifies the voice's style.
Clear enough to supply this argument.
useSpeakerBoost
Boost similarity to the original voice at a small latency cost.
Clear enough to supply this argument.
seed
Whole number. Same seed and settings tend to sound alike; not guaranteed.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
previousText
Script text just before this call, so a split script joins smoothly. Not spoken. Ignored by eleven_v3.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
nextText
Script text just after this call, so a split script joins smoothly. Not spoken. Ignored by eleven_v3.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
voiceId
eleven_v4 only. A voiceId from search_voices, instead of a preset `voice`.
Clear enough to supply this argument.
dialogue › items › text
The words this speaker says, verbatim. Inline [tags] work.
Clear enough to supply this argument.
dialogue › items › voice
Preset voice name for this line.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
dialogue › items › voiceId
A voiceId from search_voices, instead of `voice`.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
name
Optional display name for the asset in the user's Runway library. Defaults to the text.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "generate_speech",
"title": "Generate Speech",
"description": "Turn text into spoken audio (text-to-speech) with one of Runway's preset voices and return an MP3. Pass the exact words to speak in `text` and pick a `voice` by name.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nCOST: `eleven_v3` and `eleven_multilingual_v2`: 1 credit per 50 characters of `text`, rounded up (minimum 1), max 5000 characters per call. `eleven_v4`: 2.2 credits per 1,000 characters until October 12, 2026 PT, then 5 credits per 1,000 characters, rounded up (minimum 1), max 2500 characters per call. `fish-drama-3`: 1 credit per 250 characters, rounded up (minimum 1) (provisional), max 5000 characters per call. Split longer scripts into several calls.\n\nAVAILABILITY: `eleven_v4` and `fish-drama-3` need a paid plan and are not enabled on every workspace yet. If one is not enabled, tell the user and offer `eleven_v3` instead — do not retry that model.\n\nMODELS (pick with `model`; default `eleven_v3`):\n- `eleven_v3`: Most expressive. Understands inline delivery tags in the text such as [whispers], [laughs], [sighs], [excited]. Best for dialogue, characters, and emotive narration.\n- `eleven_multilingual_v2`: Steadier and more predictable; ignores delivery tags. Best for long neutral narration, product voiceover, and non-English text.\n- `eleven_v4`: Newest expressive model with inline delivery tags ([laughs], [whispers], [pause]) and a much lower per-character price. Max 2500 characters per call. The only model with multi-speaker `dialogue` and library voices from `search_voices`. Paid plans only, and not yet enabled on every workspace.\n- `fish-drama-3`: Fish Audio's expressive drama model. Direct delivery with bracketed cues such as [barely holding it together] or [small laugh], or wrap words in <whisper>...</whisper>. Uses its own seven English voices (see FISH AUDIO VOICES); no voice controls, speed, languageCode, voiceId or dialogue. Max 5000 characters per call. Paid plans only, and not yet enabled on every workspace.\n\nVOICES for the Eleven models (pick with `voice`; default `Leslie`). Choose by the user's description (\"deep male narrator\", \"cute cartoon voice\", \"British woman, calm\"); do not ask the user to pick from the full list unless they want to browse:\n- Maya: feminine, young, african american accent, chill (conversational)\n- Arjun: masculine, middle aged, indian accent, intense (narrator)\n- Serene: feminine, young, american accent, calm (narrator)\n- Bernard: masculine, middle aged, american accent, confident (animated)\n- Billy: masculine, middle aged, american accent, hyped (animated)\n- Mark: masculine, middle aged, american accent, excited (animated)\n- Clint: masculine, middle aged, american accent, husky (animated)\n- Mabel: feminine, middle aged, us southern accent, relaxed (narrator)\n- Chad: masculine, young, american accent, excited (animated)\n- Leslie: feminine, middle aged, american accent, professional (narrator)\n- Eleanor: feminine, middle aged, canadian accent, classy (narrator)\n- Elias: masculine, old, american accent, calm (narrator)\n- Elliot: masculine, middle aged, american accent, anxious (animated)\n- Grungle: masculine, middle aged, american accent, raspy (animated)\n- Brodie: masculine, young, american accent, chill (animated)\n- Sandra: feminine, middle aged, american accent, sassy (conversational)\n- Kirk: masculine, middle aged, british accent, rough (animated)\n- Kylie: feminine, young, australian accent, confident (influencer)\n- Lara: feminine, young, american accent, confident (influencer)\n- Lisa: feminine, young, german accent, casual (conversational)\n- Malachi: masculine, middle aged, american accent, confident (animated)\n- Marlene: feminine, middle aged, british accent, classy (narrator)\n- Martin: masculine, middle aged, american accent, rough (animated)\n- Miriam: feminine, middle aged, british accent, calm (narrator)\n- Monster: masculine, old, american accent, deep (animated)\n- Paula: feminine, middle aged, british accent, calm (conversational)\n- Pip: feminine, young, american accent, cute (animated)\n- Rusty: masculine, old, british accent, raspy (animated)\n- Ragnar: masculine, middle aged, american accent, deep (animated)\n- Xylar: neutral, middle aged, american accent, modulated (animated)\n- Maggie: feminine, young, american accent, calm (narration)\n- Jack: masculine, old, irish accent, sailor (video games)\n- Katie: feminine, young, american accent, soft (news)\n- Noah: masculine, young, american accent, calm (meditation)\n- James: masculine, middle aged, australian accent, casual (conversational)\n- Rina: feminine, young, american accent, calm (meditation)\n- Ella: feminine, young, american accent, emotional (narration)\n- Frank: masculine, young, american accent, deep (narration)\n- Claudia: feminine, young, british-swedish accent, seductive (characters)\n- Niki: feminine, middle aged, american accent, friendly (narration)\n- Vincent: masculine, old, australian accent, calm (news)\n- Tom: masculine, middle aged, british accent, authoritative (news)\n- Wanda: feminine, middle aged, american accent, pleasant (interactive)\n- Benjamin: masculine, middle aged, american accent, deep (narration)\n- Kiana: feminine, young, american accent, whisper (audiobook)\n- Rachel: feminine, young, english-swedish accent, childish (animation)\n\nFISH AUDIO VOICES: `fish-drama-3` only accepts these English voices (default `Sarah`); the voices above do not work with it:\n- Sarah: an engaged speaker, female, narration\n- Adrian: a steady and reliable narrator, male\n- Selene: a meditative female voice, meditation, ASMR\n- Ethan: a curious explainer\n- Hannah: a conversation specialist\n- Jordan: a motivational speaker\n- Laura: a confident female narrator\n\nWHEN TO USE:\n- \"read this aloud\", \"voiceover for my video: …\", \"narrate this script in a British accent\", \"say 'welcome back' in a robotic voice\".\n- \"a conversation between a detective and a suspect\", \"a two-host podcast intro\": `eleven_v4` with `dialogue`.\n- NOT for music (use `generate_music`), NOT for sound effects (use `generate_sound_effect`), NOT for cloning a specific person's voice, NOT for translating or re-voicing an existing audio/video file.\n\nPROMPTING TIPS: `text` is spoken verbatim — do not include stage directions as plain prose. With `eleven_v3` or `eleven_v4` you may add bracketed delivery tags inline, e.g. \"[whispers] Don't move. [pause] Did you hear that?\"; `eleven_multilingual_v2` ignores tags. `eleven_v4` rejects SSML `<break>` tags — use [pause]. Use punctuation and line breaks to control pacing. Set `languageCode` when the text is not English and pronunciation matters. Use a 2-letter ISO 639-1 code (`es`, `de`, `pt`) or a 3-letter code for eleven_v3 (`eng`, `fil`) — never a region tag like `en-US` or `pt-BR`.\n\nUSER-FACING REPLIES: Talk about the voiceover (which voice, what it says, how long), not the model or task machinery. Do not mention ElevenLabs unless the user asks. The output is an MP3 the viewer plays inline; headless agents get the URL from `get_task`.\n\nParameters:\n- text: Required unless `dialogue` is given. The words to speak, verbatim. Max 5000 characters (2500 for `eleven_v4`).\n- voice: Optional. Preset voice name from the lists above. Default `Leslie`, or `Sarah` for `fish-drama-3`.\n- voiceId: Optional, `eleven_v4` only. A voice from `search_voices` when no preset fits the user's description.\n- dialogue: Optional, `eleven_v4` only. Ordered lines `[{ voice | voiceId, text }]` for a conversation between up to 10 voices in one MP3 (max 2,000 characters in total). Use instead of `text`; lines with the same voice are the same speaker.\n- model: Optional. `eleven_v3` (default, expressive, supports tags), `eleven_multilingual_v2` (steady), `eleven_v4` (expressive, cheaper, paid plans, max 2500 characters), or `fish-drama-3` (Fish Audio drama voices, paid plans, max 5000 characters).\n- speed: Optional. 0.7–1.2; 1.0 is normal.\n- languageCode: Optional. 2-letter ISO 639-1 code (e.g. `es`, `de`, `pt`) or 3-letter code for eleven_v3 (e.g. `eng`, `fil`). Do not include a region.\n- stability, similarityBoost, style, useSpeakerBoost, seed, previousText, nextText: Optional voice controls (see each parameter); not for `fish-drama-3`. Leave them unset unless the user asks for the effect.\n- name: Optional. Display name for the asset in the user's Runway library. Defaults to the text.",
"inputSchema": {
"type": "object",
"properties": {
"name": {
"type": "string",
"maxLength": 300,
"description": "Optional display name for the asset in the user's Runway library. Defaults to the text."
},
"seed": {
"type": "number",
"maximum": 4294967295,
"minimum": 0,
"description": "Whole number. Same seed and settings tend to sound alike; not guaranteed."
},
"text": {
"type": "string",
"maxLength": 5000,
"description": "The exact words to speak. Max 5000 characters, or 2500 for eleven_v4."
},
"model": {
"enum": [
"eleven_v3",
"eleven_multilingual_v2",
"eleven_v4",
"fish-drama-3"
],
"type": "string",
"description": "Speech model. 'eleven_v3' (default) is most expressive and supports inline [tags]; 'eleven_multilingual_v2' is steadier; 'eleven_v4' supports [tags], costs less per character, needs a paid plan, and takes at most 2500 characters; 'fish-drama-3' uses Fish Audio voices, needs a paid plan, and takes no voice controls."
},
"speed": {
"type": "number",
"maximum": 1.2,
"minimum": 0.7,
"description": "Speech speed multiplier, 0.7–1.2. 1.0 is normal."
},
"style": {
"type": "number",
"maximum": 1,
"minimum": 0,
"description": "0–1 style exaggeration: higher amplifies the voice's style."
},
"voice": {
"enum": [
"Maya",
"Arjun",
"Serene",
"Bernard",
"Billy",
"Mark",
"Clint",
"Mabel",
"Chad",
"Leslie",
"Eleanor",
"Elias",
"Elliot",
"Grungle",
"Brodie",
"Sandra",
"Kirk",
"Kylie",
"Lara",
"Lisa",
"Malachi",
"Marlene",
"Martin",
"Miriam",
"Monster",
"Paula",
"Pip",
"Rusty",
"Ragnar",
"Xylar",
"Maggie",
"Jack",
"Katie",
"Noah",
"James",
"Rina",
"Ella",
"Frank",
"Claudia",
"Niki",
"Vincent",
"Tom",
"Wanda",
"Benjamin",
"Kiana",
"Rachel",
"Sarah",
"Adrian",
"Selene",
"Ethan",
"Hannah",
"Jordan",
"Laura"
],
"type": "string",
"description": "Preset voice name. Default 'Leslie', or 'Sarah' for fish-drama-3, which only takes Sarah, Adrian, Selene, Ethan, Hannah, Jordan, Laura. Pick from the descriptions in the tool description."
},
"voiceId": {
"type": "string",
"description": "eleven_v4 only. A voiceId from search_voices, instead of a preset `voice`."
},
"dialogue": {
"type": "array",
"items": {
"type": "object",
"required": [
"text"
],
"properties": {
"text": {
"type": "string",
"description": "The words this speaker says, verbatim. Inline [tags] work."
},
"voice": {
"enum": [
"Maya",
"Arjun",
"Serene",
"Bernard",
"Billy",
"Mark",
"Clint",
"Mabel",
"Chad",
"Leslie",
"Eleanor",
"Elias",
"Elliot",
"Grungle",
"Brodie",
"Sandra",
"Kirk",
"Kylie",
"Lara",
"Lisa",
"Malachi",
"Marlene",
"Martin",
"Miriam",
"Monster",
"Paula",
"Pip",
"Rusty",
"Ragnar",
"Xylar",
"Maggie",
"Jack",
"Katie",
"Noah",
"James",
"Rina",
"Ella",
"Frank",
"Claudia",
"Niki",
"Vincent",
"Tom",
"Wanda",
"Benjamin",
"Kiana",
"Rachel"
],
"type": "string",
"description": "Preset voice name for this line."
},
"voiceId": {
"type": "string",
"description": "A voiceId from search_voices, instead of `voice`."
}
},
"additionalProperties": false
},
"maxItems": 200,
"minItems": 1,
"description": "eleven_v4 only. Multi-speaker dialogue as ordered lines, instead of `text`. Up to 10 distinct voices and 2,000 characters in total. With dialogue, only `stability` (0, 0.5 or 1), `languageCode` and `seed` apply."
},
"nextText": {
"type": "string",
"maxLength": 5000,
"description": "Script text just after this call, so a split script joins smoothly. Not spoken. Ignored by eleven_v3."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"stability": {
"type": "number",
"maximum": 1,
"minimum": 0,
"description": "0–1: lower allows a broader emotional range, higher is steadier."
},
"languageCode": {
"type": "string",
"pattern": "^[a-zA-Z]{2,3}$",
"maxLength": 3,
"minLength": 2,
"description": "2-letter ISO 639-1 code (e.g. 'en', 'es', 'pt') or 3-letter code for eleven_v3 (e.g. 'eng', 'fil'). Do not send a region/locale such as 'en-US' or 'pt-BR'."
},
"previousText": {
"type": "string",
"maxLength": 5000,
"description": "Script text just before this call, so a split script joins smoothly. Not spoken. Ignored by eleven_v3."
},
"similarityBoost": {
"type": "number",
"maximum": 1,
"minimum": 0,
"description": "0–1: how closely the output tracks the original voice."
},
"useSpeakerBoost": {
"type": "boolean",
"description": "Boost similarity to the original voice at a small latency cost."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Generate Speech",
"readOnlyHint": false,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Search the Eleven v4 voice library (thousands of voices beyond the preset list in `generate_speech`). Returns voices with a `voiceId` to pass to `generate_speech` with `model: "eleven_v4"`, either as `voiceId` or as a dialogue line's `voiceId`. Free: does not spend credits.
WHEN TO USE: the user wants a voice the preset list does not cover ("an old Scottish fisherman", "a Brazilian Portuguese news anchor"), or asks to browse voices. Otherwise use the preset `voice` names.
Filters match the library's lowercase labels exactly; if a filtered search returns nothing, retry with fewer filters or with `query` alone.
AVAILABILITY: not enabled on every workspace yet. If it is not enabled, use preset voices instead.
The description explains the capability, when to use it, and what to expect.
Inputs92/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
query
Free text matched against voice names and descriptions, e.g. 'raspy', 'news anchor', 'scottish'.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
gender
e.g. 'male', 'female', 'neutral'.
Clear enough to supply this argument.
age
e.g. 'young', 'middle_aged', 'old'.
Clear enough to supply this argument.
accent
e.g. 'american', 'british', 'australian'.
Clear enough to supply this argument.
language
ISO 639-1 code, e.g. 'en', 'es', 'pt'.
Clear enough to supply this argument.
useCase
e.g. 'narrative_story', 'conversational', 'characters_animation', 'social_media', 'advertisement', 'informative_educational'.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
limit
Results per page. Default 20.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
cursor
`nextCursor` from a previous call, for the next page.
{
"name": "search_voices",
"title": "Search Voices",
"description": "Search the Eleven v4 voice library (thousands of voices beyond the preset list in `generate_speech`). Returns voices with a `voiceId` to pass to `generate_speech` with `model: \"eleven_v4\"`, either as `voiceId` or as a dialogue line's `voiceId`. Free: does not spend credits.\n\nWHEN TO USE: the user wants a voice the preset list does not cover (\"an old Scottish fisherman\", \"a Brazilian Portuguese news anchor\"), or asks to browse voices. Otherwise use the preset `voice` names.\n\nFilters match the library's lowercase labels exactly; if a filtered search returns nothing, retry with fewer filters or with `query` alone.\n\nAVAILABILITY: not enabled on every workspace yet. If it is not enabled, use preset voices instead.",
"inputSchema": {
"type": "object",
"properties": {
"age": {
"type": "string",
"description": "e.g. 'young', 'middle_aged', 'old'."
},
"limit": {
"type": "number",
"maximum": 50,
"minimum": 1,
"description": "Results per page. Default 20."
},
"query": {
"type": "string",
"maxLength": 200,
"description": "Free text matched against voice names and descriptions, e.g. 'raspy', 'news anchor', 'scottish'."
},
"accent": {
"type": "string",
"description": "e.g. 'american', 'british', 'australian'."
},
"cursor": {
"type": "string",
"description": "`nextCursor` from a previous call, for the next page."
},
"gender": {
"type": "string",
"description": "e.g. 'male', 'female', 'neutral'."
},
"useCase": {
"type": "string",
"description": "e.g. 'narrative_story', 'conversational', 'characters_animation', 'social_media', 'advertisement', 'informative_educational'."
},
"language": {
"type": "string",
"description": "ISO 639-1 code, e.g. 'en', 'es', 'pt'."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind",
"voices"
],
"properties": {
"kind": {
"enum": [
"voice_list"
],
"type": "string"
},
"voices": {
"type": "array",
"items": {
"type": "object",
"required": [
"voiceId",
"name"
],
"properties": {
"name": {
"type": "string"
},
"voiceId": {
"type": "string"
},
"previewUrl": {
"type": "string"
},
"description": {
"type": "string"
}
},
"additionalProperties": true
}
},
"nextCursor": {
"description": "Cursor for the next page, or null."
}
},
"additionalProperties": true
},
"annotations": {
"title": "Search Voices",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description100/100
Localize an existing ad image for another market: the on-screen copy is translated into the target language and the ad is regenerated with the same layout, typography feel, product, and visual creative. Use this instead of re-generating an ad from scratch when the user wants the SAME creative in another language.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
WHEN TO USE THIS:
- "localize this ad for Japan", "translate this banner into Spanish", "give me German/French/Korean versions of this creative", "adapt this ad for the LATAM market".
- One call = one language. For several markets, call this once per language (each call is a separate task and a separate charge).
- NOT for translating a *document* or plain text (no image involved), NOT for editing anything other than the ad's language (use `generate_image` with referenceImages), NOT for upscaling (`upscale_image`).
IMAGES ONLY: there is no video ad localization. If the user asks to localize a video ad, say that isn't supported yet rather than trying `edit_video` — Aleph rewrites frames and will not reliably retype on-screen copy.
SOURCE AD REQUIREMENTS: the image must contain readable on-screen text. If it has none, the task fails with "No translatable text found in the reference image" — when that happens, tell the user the ad has no detectable copy to translate instead of retrying. High-resolution sources with clear typography localize far more reliably than heavily stylized layouts.
COST: flat 21 credits per call. OUTPUT: a single localized image.
HANDLING USER ATTACHMENTS:
- ChatGPT: pass an uploaded image as `imageFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `image.url`.
- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `image.url`. Outputs of `generate_image` can be passed directly as `image.url`.
USER-FACING REPLIES: Talk about the localized ad and the language, not the model or task machinery.
Parameters:
- image: `{url}` — the source ad to localize. Must be a Runway-hosted asset URL (from `complete_upload` or `generate_image`) or a public HTTPS image URL.
- imageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`.
- targetLanguage: Required. ISO-style code or English name. Supported: ar (Arabic), zh (Chinese (Simplified)), zh-Hant (Chinese (Traditional)), nl (Dutch), en (English), fr (French), de (German), el (Greek), hi (Hindi), id (Indonesian), it (Italian), ja (Japanese), ko (Korean), pl (Polish), pt (Portuguese), ru (Russian), es (Spanish), sv (Swedish), th (Thai), tr (Turkish), uk (Ukrainian), vi (Vietnamese). Regional variants collapse to the base language (e.g. `pt-BR` -> Portuguese); `zh` is Simplified and `zh-Hant` is Traditional Chinese.
The description explains the capability, when to use it, and what to expect.
Inputs89/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
image › url
Runway-hosted asset URL (preferred) or public HTTPS image URL. Never use /mnt/data or other local file paths here.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
imageFile › download_url
Temporary ChatGPT file download URL.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
imageFile › file_id
ChatGPT file ID.
Clear enough to supply this argument.
imageFile › mime_type
MIME type, e.g. image/png.
Clear enough to supply this argument.
imageFile › file_name
Original file name.
Clear enough to supply this argument.
targetLanguage
Target language for the localized ad, as an ISO-style code or English name (e.g. "ja" or "Japanese"). Supported: ar (Arabic), zh (Chinese (Simplified)), zh-Hant (Chinese (Traditional)), nl (Dutch), en (English), fr (French), de (German), el (Greek), hi (Hindi), id (Indonesian), it (Italian), ja (Japanese), ko (Korean), pl (Polish), pt (Portuguese), ru (Russian), es (Spanish), sv (Swedish), th (Thai), tr (Turkish), uk (Ukrainian), vi (Vietnamese).
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "localize_ad",
"title": "Localize Ad",
"description": "Localize an existing ad image for another market: the on-screen copy is translated into the target language and the ad is regenerated with the same layout, typography feel, product, and visual creative. Use this instead of re-generating an ad from scratch when the user wants the SAME creative in another language.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nWHEN TO USE THIS:\n- \"localize this ad for Japan\", \"translate this banner into Spanish\", \"give me German/French/Korean versions of this creative\", \"adapt this ad for the LATAM market\".\n- One call = one language. For several markets, call this once per language (each call is a separate task and a separate charge).\n- NOT for translating a *document* or plain text (no image involved), NOT for editing anything other than the ad's language (use `generate_image` with referenceImages), NOT for upscaling (`upscale_image`).\n\nIMAGES ONLY: there is no video ad localization. If the user asks to localize a video ad, say that isn't supported yet rather than trying `edit_video` — Aleph rewrites frames and will not reliably retype on-screen copy.\n\nSOURCE AD REQUIREMENTS: the image must contain readable on-screen text. If it has none, the task fails with \"No translatable text found in the reference image\" — when that happens, tell the user the ad has no detectable copy to translate instead of retrying. High-resolution sources with clear typography localize far more reliably than heavily stylized layouts.\n\nCOST: flat 21 credits per call. OUTPUT: a single localized image.\n\nHANDLING USER ATTACHMENTS:\n- ChatGPT: pass an uploaded image as `imageFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `image.url`.\n- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `image.url`. Outputs of `generate_image` can be passed directly as `image.url`.\n\nUSER-FACING REPLIES: Talk about the localized ad and the language, not the model or task machinery.\n\nParameters:\n- image: `{url}` — the source ad to localize. Must be a Runway-hosted asset URL (from `complete_upload` or `generate_image`) or a public HTTPS image URL.\n- imageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`.\n- targetLanguage: Required. ISO-style code or English name. Supported: ar (Arabic), zh (Chinese (Simplified)), zh-Hant (Chinese (Traditional)), nl (Dutch), en (English), fr (French), de (German), el (Greek), hi (Hindi), id (Indonesian), it (Italian), ja (Japanese), ko (Korean), pl (Polish), pt (Portuguese), ru (Russian), es (Spanish), sv (Swedish), th (Thai), tr (Turkish), uk (Ukrainian), vi (Vietnamese). Regional variants collapse to the base language (e.g. `pt-BR` -> Portuguese); `zh` is Simplified and `zh-Hant` is Traditional Chinese.",
"inputSchema": {
"type": "object",
"required": [
"targetLanguage"
],
"properties": {
"image": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"description": "Runway-hosted asset URL (preferred) or public HTTPS image URL. Never use /mnt/data or other local file paths here."
}
},
"description": "Source ad image to localize. URL must be a Runway-hosted asset URL (complete_upload or generate_image output) or a public HTTPS image URL. Do not pass ChatGPT local paths like /mnt/data/...; use imageFile instead.",
"additionalProperties": false
},
"imageFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. image/png."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source ad. Prefer image.url when you already have a Runway-hosted or public URL.",
"additionalProperties": false
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"targetLanguage": {
"type": "string",
"description": "Target language for the localized ad, as an ISO-style code or English name (e.g. \"ja\" or \"Japanese\"). Supported: ar (Arabic), zh (Chinese (Simplified)), zh-Hant (Chinese (Traditional)), nl (Dutch), en (English), fr (French), de (German), el (Greek), hi (Hindi), id (Indonesian), it (Italian), ja (Japanese), ko (Korean), pl (Polish), pt (Portuguese), ru (Russian), es (Spanish), sv (Swedish), th (Thai), tr (Turkish), uk (Ukrainian), vi (Vietnamese)."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Localize Ad",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
Expand internal abbreviations and remove technical prefixes from the display title.
Description99/100
Remove the background from an image: SAM3 segmentation isolates the subject you describe and outputs a PNG with a TRANSPARENT background (alpha channel) — the subject is kept, everything else is removed.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
WHEN TO USE THIS:
- "remove the background", "cut out the person/product", "make the background transparent", "isolate the subject", "give me a sticker/cutout of X".
- NOT for replacing the background with new content or other edits (use `generate_image` with referenceImages), NOT for upscaling (`upscale_image`). For videos use `remove_video_background`.
SOURCE IMAGE CONSTRAINTS: at most 4096px per side. Cost: 1 credit per image. If the subject can't be found in the image the task fails with "No detections found" — rephrase the subject and retry once, then tell the user what couldn't be found.
OUTPUT: a PNG with an alpha channel (true transparency).
HANDLING USER ATTACHMENTS:
- ChatGPT: pass an uploaded image as `imageFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `image.url`.
- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `image.url`. Outputs of `generate_image` can be passed directly as `image.url`.
USER-FACING REPLIES: Talk about the cutout (what's being isolated and that the background becomes transparent), not the model or task machinery. Do not mention SAM3 unless the user asks.
Parameters:
- subject: Required. What to KEEP, described in a few words (e.g. "the person", "the sneaker", "the dog on the left"). Max 200 characters.
- image: `{url}` — source image. URL must be a Runway-hosted asset URL (from complete_upload or generate_image) or a public HTTPS image URL.
- imageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`.
The description explains the capability, when to use it, and what to expect.
Inputs86/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
subject
What to KEEP in the image, in a few words (e.g. "the person"). Max 200 characters.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
image › url
Runway-hosted asset URL (preferred) or public HTTPS image URL. Never use /mnt/data or other local file paths here.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "remove_image_background",
"title": "Remove Image Background (SAM3)",
"description": "Remove the background from an image: SAM3 segmentation isolates the subject you describe and outputs a PNG with a TRANSPARENT background (alpha channel) — the subject is kept, everything else is removed.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nWHEN TO USE THIS:\n- \"remove the background\", \"cut out the person/product\", \"make the background transparent\", \"isolate the subject\", \"give me a sticker/cutout of X\".\n- NOT for replacing the background with new content or other edits (use `generate_image` with referenceImages), NOT for upscaling (`upscale_image`). For videos use `remove_video_background`.\n\nSOURCE IMAGE CONSTRAINTS: at most 4096px per side. Cost: 1 credit per image. If the subject can't be found in the image the task fails with \"No detections found\" — rephrase the subject and retry once, then tell the user what couldn't be found.\n\nOUTPUT: a PNG with an alpha channel (true transparency).\n\nHANDLING USER ATTACHMENTS:\n- ChatGPT: pass an uploaded image as `imageFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `image.url`.\n- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `image.url`. Outputs of `generate_image` can be passed directly as `image.url`.\n\nUSER-FACING REPLIES: Talk about the cutout (what's being isolated and that the background becomes transparent), not the model or task machinery. Do not mention SAM3 unless the user asks.\n\nParameters:\n- subject: Required. What to KEEP, described in a few words (e.g. \"the person\", \"the sneaker\", \"the dog on the left\"). Max 200 characters.\n- image: `{url}` — source image. URL must be a Runway-hosted asset URL (from complete_upload or generate_image) or a public HTTPS image URL.\n- imageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`.",
"inputSchema": {
"type": "object",
"required": [
"subject"
],
"properties": {
"image": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"description": "Runway-hosted asset URL (preferred) or public HTTPS image URL. Never use /mnt/data or other local file paths here."
}
},
"description": "Source image. URL must be a Runway-hosted asset URL (complete_upload or generate_image output) or a public HTTPS image URL. Do not pass ChatGPT local paths like /mnt/data/...; use imageFile instead.",
"additionalProperties": false
},
"subject": {
"type": "string",
"maxLength": 200,
"description": "What to KEEP in the image, in a few words (e.g. \"the person\"). Max 200 characters."
},
"imageFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. image/png."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source image. Prefer image.url when you already have a Runway-hosted or public URL.",
"additionalProperties": false
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Remove Image Background (SAM3)",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
Expand internal abbreviations and remove technical prefixes from the display title.
Description100/100
Remove the background from a video: SAM3 segmentation tracks the subject you describe across every frame and outputs a .webm video with a TRANSPARENT background (alpha channel) — the subject is kept, everything else is removed. Set `invert: true` to do the opposite: cut the subject OUT and keep the background.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
WHEN TO USE THIS:
- "remove the background", "cut out the person", "green-screen this video", "isolate the product", "make the background transparent".
- NOT for replacing the background with new content (use `edit_video` — it can restyle or swap backgrounds in place), NOT for content generation (`generate_video`), NOT for sharpening (`upscale_video`).
SOURCE VIDEO CONSTRAINTS: at most 60 seconds (~1800 frames at 30fps). Cost: 1 credit/second of input video. `video.durationSeconds` is REQUIRED — the task is billed by duration and the backend rejects sources without one.
OUTPUT: a .webm file with an alpha channel. Warn the user that some players show transparent regions as black/gray; the transparency is real and works in editors and compositing tools.
AVAILABILITY: account-gated. If the task is rejected with an "Unknown task type" error, this workspace does not have access yet — tell the user the feature isn't enabled for their account instead of retrying.
REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.
HANDLING USER ATTACHMENTS:
- ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `video.url`. Still pass `video.durationSeconds` (from file metadata).
- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`.
USER-FACING REPLIES: Talk about the cutout (what's being isolated and that the background becomes transparent), not the model or task machinery. Do not mention SAM3 unless the user asks.
Parameters:
- subject: Required. What to KEEP, described in a few words (e.g. "the person", "the red car", "the dog on the left"). Max 200 characters.
- video: `{url?, durationSeconds, assetId?}` — source video. URL must be a Runway-hosted /datasets/ asset URL (from complete_upload) or a public HTTPS video URL; omit it only when passing videoFile. durationSeconds is ALWAYS required.
- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. Pass the duration alongside it as `video: {durationSeconds}`.
- keepSound: Optional, default true. Keep the source audio track in the output.
- invert: Optional, default false. Remove the SUBJECT and keep the background instead.
The description explains the capability, when to use it, and what to expect.
Inputs96/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
subject
What to KEEP in the video, in a few words (e.g. "the person"). Max 200 characters. With invert: true this is what gets removed instead.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
video › taskId
Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated.
Clear enough to supply this argument.
video › url
Runway-hosted dataset URL from complete_upload (preferred) or public HTTPS video URL. Never use /mnt/data or other local file paths here. Omit when passing videoFile.
Clear enough to supply this argument.
video › durationSeconds
Source video duration in seconds (max 60). REQUIRED — the task is billed by duration and rejected without one.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
video › assetId
Runway dataset asset id for the source video. Do NOT invent this — only pass a value a previous call returned for the SAME video, so the tool reuses the registered dataset instead of re-uploading.
Clear enough to supply this argument.
videoFile › download_url
Temporary ChatGPT file download URL.
Clear enough to supply this argument.
videoFile › file_id
ChatGPT file ID.
Clear enough to supply this argument.
videoFile › mime_type
MIME type, e.g. video/mp4.
Clear enough to supply this argument.
videoFile › file_name
Original file name.
Clear enough to supply this argument.
keepSound
Keep the source audio track in the output (default true).
Clear enough to supply this argument.
invert
Invert the selection: remove the subject and keep the background (default false).
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "remove_video_background",
"title": "Remove Video Background (SAM3)",
"description": "Remove the background from a video: SAM3 segmentation tracks the subject you describe across every frame and outputs a .webm video with a TRANSPARENT background (alpha channel) — the subject is kept, everything else is removed. Set `invert: true` to do the opposite: cut the subject OUT and keep the background.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nWHEN TO USE THIS:\n- \"remove the background\", \"cut out the person\", \"green-screen this video\", \"isolate the product\", \"make the background transparent\".\n- NOT for replacing the background with new content (use `edit_video` — it can restyle or swap backgrounds in place), NOT for content generation (`generate_video`), NOT for sharpening (`upscale_video`).\n\nSOURCE VIDEO CONSTRAINTS: at most 60 seconds (~1800 frames at 30fps). Cost: 1 credit/second of input video. `video.durationSeconds` is REQUIRED — the task is billed by duration and the backend rejects sources without one.\n\nOUTPUT: a .webm file with an alpha channel. Warn the user that some players show transparent regions as black/gray; the transparency is real and works in editors and compositing tools.\n\nAVAILABILITY: account-gated. If the task is rejected with an \"Unknown task type\" error, this workspace does not have access yet — tell the user the feature isn't enabled for their account instead of retrying.\n\nREUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.\n\nHANDLING USER ATTACHMENTS:\n- ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `video.url`. Still pass `video.durationSeconds` (from file metadata).\n- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`.\n\nUSER-FACING REPLIES: Talk about the cutout (what's being isolated and that the background becomes transparent), not the model or task machinery. Do not mention SAM3 unless the user asks.\n\nParameters:\n- subject: Required. What to KEEP, described in a few words (e.g. \"the person\", \"the red car\", \"the dog on the left\"). Max 200 characters.\n- video: `{url?, durationSeconds, assetId?}` — source video. URL must be a Runway-hosted /datasets/ asset URL (from complete_upload) or a public HTTPS video URL; omit it only when passing videoFile. durationSeconds is ALWAYS required.\n- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. Pass the duration alongside it as `video: {durationSeconds}`.\n- keepSound: Optional, default true. Keep the source audio track in the output.\n- invert: Optional, default false. Remove the SUBJECT and keep the background instead.",
"inputSchema": {
"type": "object",
"required": [
"subject"
],
"properties": {
"video": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Runway-hosted dataset URL from complete_upload (preferred) or public HTTPS video URL. Never use /mnt/data or other local file paths here. Omit when passing videoFile."
},
"taskId": {
"type": "string",
"description": "Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated."
},
"assetId": {
"type": "string",
"description": "Runway dataset asset id for the source video. Do NOT invent this — only pass a value a previous call returned for the SAME video, so the tool reuses the registered dataset instead of re-uploading."
},
"durationSeconds": {
"type": "number",
"description": "Source video duration in seconds (max 60). REQUIRED — the task is billed by duration and rejected without one."
}
},
"description": "Source video. url is required unless videoFile is provided; durationSeconds is ALWAYS required (pass video: {durationSeconds} alongside videoFile). Do not pass ChatGPT local paths like /mnt/data/... as url; use videoFile instead.",
"additionalProperties": false
},
"invert": {
"type": "boolean",
"description": "Invert the selection: remove the subject and keep the background (default false)."
},
"subject": {
"type": "string",
"maxLength": 200,
"description": "What to KEEP in the video, in a few words (e.g. \"the person\"). Max 200 characters. With invert: true this is what gets removed instead."
},
"keepSound": {
"type": "boolean",
"description": "Keep the source audio track in the output (default true)."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"videoFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. video/mp4."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source video. Prefer video.url when you already have a Runway-hosted or public URL. Pass the duration via video.durationSeconds alongside this.",
"additionalProperties": false
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Remove Video Background (SAM3)",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Upscale an existing image to a higher resolution (2x, 4x, 8x, or 16x) using Runway's AI image upscaler. Use this to sharpen, denoise, and increase the resolution of an image — it does NOT change the content, composition, or style; for those edits use `generate_image` with `referenceImages`.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
PAID PLAN REQUIRED: This upscaler is not available on free Runway workspaces. If the connected workspace is on a free plan the task will be rejected with an account-limitation message — tell the user image upscaling isn't available in their current workspace and do not retry. There is no free upscaling alternative; do NOT try to fake an upscale with `generate_image`.
WHEN TO USE:
- The user wants a low-res / soft / blurry / grainy image made crisper or larger (e.g. "upscale this to 4x", "make this image higher resolution", "enhance / denoise this photo").
- The user has a finished image (their own upload, or an output from `generate_image`) and wants a higher-fidelity version of the exact same image.
WHEN NOT TO USE:
- To restyle, edit, remove/replace backgrounds or objects, or otherwise change the content → use `generate_image` with `referenceImages[0]`.
- To generate a brand-new image → use `generate_image`.
- To upscale a video → use `upscale_video`.
INPUT:
- Provide the source image as `image.url`: a Runway-hosted URL (from `complete_upload` or a `generate_image` output) or a public HTTPS image URL. The tool imports it into Runway automatically.
- Pass `image.width`/`image.height` when known so scale-factor limits can be checked before submitting. The input must be at least 300x300px, and the output (width * scaleFactor x height * scaleFactor) must stay under ~25 megapixels — e.g. a 4000x3000 image only supports 1 step of 2x.
FLAVOR / SCALE RULES:
- flavor `sublime` (DEFAULT — artistic/illustrated images, smooth gradients, vibrant colors): scaleFactor 2, 4, 8, or 16.
- flavor `photo` (photographic images, natural colors, realistic detail): scaleFactor 2 only.
- flavor `photo_denoiser` (noise reduction for low-light/grainy photos): scaleFactor 2 only.
HANDLING USER ATTACHMENTS:
- Claude/Cursor/local agents: if the user attached a local image, run `init_upload` -> curl -> `complete_upload` first, then pass the returned URL as `image.url`.
- ChatGPT: if it supplies an uploaded image as `imageFile`, pass it directly. NEVER pass local paths such as `/mnt/data/file.png` as `image.url`; use `imageFile` instead.
USER-FACING REPLIES: Talk about the result ("upscaling your image 4x"), not the underlying engine. Do not mention the model/provider unless the user explicitly asks.
Parameters:
- image: `{ url, width?, height? }` — the source image to upscale. URL must be public HTTPS or Runway-hosted. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `imageFile` for ChatGPT uploads.
- imageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`.
- scaleFactor: 2, 4, 8, or 16. Default: 2. Factors above 2 require flavor `sublime`.
- flavor: `sublime` | `photo` | `photo_denoiser`. Default: `sublime`. Pick `photo` for photographs, `photo_denoiser` for grainy/low-light photos.
- sharpen: 0-100. Default: 10. Higher = crisper edges.
- smartGrain: 0-100. Default: 10. Adds natural film-like grain to avoid a plasticky look.
- ultraDetail: 0-100. Default: 30. Higher = more invented micro-detail (can drift from the source).
The description explains the capability, when to use it, and what to expect.
Inputs88/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
image › url
Public HTTPS or Runway-hosted image URL. Never use /mnt/data or other local file paths here.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
image › width
Source image width in pixels. Pass when known so scale-factor limits are validated up front.
Clear enough to supply this argument.
image › height
Source image height in pixels. Pass when known so scale-factor limits are validated up front.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
flavor
Upscaling flavor. Default: 'sublime'. 'sublime' for artistic/illustrated images, 'photo' for photographs, 'photo_denoiser' for grainy/low-light photos.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
sharpen
Sharpening strength 0-100. Default: 10.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
smartGrain
Film-grain strength 0-100. Default: 10.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "upscale_image",
"title": "Upscale Image",
"description": "Upscale an existing image to a higher resolution (2x, 4x, 8x, or 16x) using Runway's AI image upscaler. Use this to sharpen, denoise, and increase the resolution of an image — it does NOT change the content, composition, or style; for those edits use `generate_image` with `referenceImages`.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nPAID PLAN REQUIRED: This upscaler is not available on free Runway workspaces. If the connected workspace is on a free plan the task will be rejected with an account-limitation message — tell the user image upscaling isn't available in their current workspace and do not retry. There is no free upscaling alternative; do NOT try to fake an upscale with `generate_image`.\n\nWHEN TO USE:\n- The user wants a low-res / soft / blurry / grainy image made crisper or larger (e.g. \"upscale this to 4x\", \"make this image higher resolution\", \"enhance / denoise this photo\").\n- The user has a finished image (their own upload, or an output from `generate_image`) and wants a higher-fidelity version of the exact same image.\n\nWHEN NOT TO USE:\n- To restyle, edit, remove/replace backgrounds or objects, or otherwise change the content → use `generate_image` with `referenceImages[0]`.\n- To generate a brand-new image → use `generate_image`.\n- To upscale a video → use `upscale_video`.\n\nINPUT:\n- Provide the source image as `image.url`: a Runway-hosted URL (from `complete_upload` or a `generate_image` output) or a public HTTPS image URL. The tool imports it into Runway automatically.\n- Pass `image.width`/`image.height` when known so scale-factor limits can be checked before submitting. The input must be at least 300x300px, and the output (width * scaleFactor x height * scaleFactor) must stay under ~25 megapixels — e.g. a 4000x3000 image only supports 1 step of 2x.\n\nFLAVOR / SCALE RULES:\n- flavor `sublime` (DEFAULT — artistic/illustrated images, smooth gradients, vibrant colors): scaleFactor 2, 4, 8, or 16.\n- flavor `photo` (photographic images, natural colors, realistic detail): scaleFactor 2 only.\n- flavor `photo_denoiser` (noise reduction for low-light/grainy photos): scaleFactor 2 only.\n\nHANDLING USER ATTACHMENTS:\n- Claude/Cursor/local agents: if the user attached a local image, run `init_upload` -> curl -> `complete_upload` first, then pass the returned URL as `image.url`.\n- ChatGPT: if it supplies an uploaded image as `imageFile`, pass it directly. NEVER pass local paths such as `/mnt/data/file.png` as `image.url`; use `imageFile` instead.\n\nUSER-FACING REPLIES: Talk about the result (\"upscaling your image 4x\"), not the underlying engine. Do not mention the model/provider unless the user explicitly asks.\n\nParameters:\n- image: `{ url, width?, height? }` — the source image to upscale. URL must be public HTTPS or Runway-hosted. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `imageFile` for ChatGPT uploads.\n- imageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`.\n- scaleFactor: 2, 4, 8, or 16. Default: 2. Factors above 2 require flavor `sublime`.\n- flavor: `sublime` | `photo` | `photo_denoiser`. Default: `sublime`. Pick `photo` for photographs, `photo_denoiser` for grainy/low-light photos.\n- sharpen: 0-100. Default: 10. Higher = crisper edges.\n- smartGrain: 0-100. Default: 10. Adds natural film-like grain to avoid a plasticky look.\n- ultraDetail: 0-100. Default: 30. Higher = more invented micro-detail (can drift from the source).",
"inputSchema": {
"type": "object",
"properties": {
"image": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"description": "Public HTTPS or Runway-hosted image URL. Never use /mnt/data or other local file paths here."
},
"width": {
"type": "number",
"description": "Source image width in pixels. Pass when known so scale-factor limits are validated up front."
},
"height": {
"type": "number",
"description": "Source image height in pixels. Pass when known so scale-factor limits are validated up front."
}
},
"description": "Source image to upscale. URL must be a public HTTPS image URL or a Runway-hosted URL (complete_upload or generate_image output). Do not pass ChatGPT local paths like /mnt/data/...; use imageFile instead.",
"additionalProperties": false
},
"flavor": {
"enum": [
"sublime",
"photo",
"photo_denoiser"
],
"type": "string",
"description": "Upscaling flavor. Default: 'sublime'. 'sublime' for artistic/illustrated images, 'photo' for photographs, 'photo_denoiser' for grainy/low-light photos."
},
"sharpen": {
"type": "number",
"description": "Sharpening strength 0-100. Default: 10."
},
"imageFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. image/png."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source image. Prefer image.url when you already have a public or Runway-hosted URL.",
"additionalProperties": false
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"smartGrain": {
"type": "number",
"description": "Film-grain strength 0-100. Default: 10."
},
"scaleFactor": {
"enum": [
2,
4,
8,
16
],
"type": "number",
"description": "Upscale factor. Default: 2. Factors above 2 require flavor 'sublime'."
},
"ultraDetail": {
"type": "number",
"description": "Invented micro-detail strength 0-100. Default: 30."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Upscale Image",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description97/100
Upscale an existing video to a higher resolution (up to 4K) using Runway's AI video upscaler. Use this to sharpen, clean up, and increase the resolution of footage — it does NOT change the content, motion, style, or duration; for those edits use `generate_video` (video-to-video).
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
WHEN TO USE:
- The user wants a low-res / soft / blurry video made crisper or larger (e.g. "upscale this to 4K", "make this video higher resolution", "enhance the quality of this clip").
- The user has a finished clip (their own upload, or an output from another tool that they re-uploaded) and wants a higher-fidelity version of the exact same video.
WHEN NOT TO USE:
- To restyle, edit, remove/replace backgrounds, remove objects, or otherwise change the content → use `generate_video` with `referenceVideo`.
- To generate a brand-new video → use `generate_video`.
INPUT:
- Provide the source video as `video.url`. This must be a Runway-hosted `/datasets/<UUID>.<ext>` URL returned by `complete_upload`, OR a public HTTPS video URL (the tool imports it into Runway automatically).
- Pass `video.durationSeconds` when known. Videos must be under 40 seconds.
REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.
HANDLING USER ATTACHMENTS:
- Claude/Cursor/local agents: if the user attached a local video, run `init_upload` -> curl -> `complete_upload` first to get a Runway-hosted asset URL, then pass it as `video.url`.
- ChatGPT: if it supplies an uploaded video as `videoFile`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as the source. NEVER pass local paths such as `/mnt/data/file.mp4` as `video.url`; use `videoFile` instead.
REUSING ASSETS: A URL from `complete_upload` is a stable, hosted asset URL — pass it directly as `video.url`, never re-upload. Note: outputs of `generate_video` are NOT yet usable as `video.url` here (re-upload via `init_upload` if needed).
USER-FACING REPLIES: Talk about the result ("upscaling your video to 4K"), not the underlying engine. Do not mention the model/provider unless the user explicitly asks.
Parameters:
- video: `{ url, durationSeconds? }` — the source video to upscale. URL must be public HTTPS or a Runway-hosted `/datasets` URL. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `videoFile` for ChatGPT uploads.
- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. Use this instead of `video` only when ChatGPT provides a user-uploaded file.
The description explains the capability, when to use it, and what to expect.
Inputs90/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
video › taskId
Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated.
Clear enough to supply this argument.
video › url
Public HTTPS or Runway-hosted video URL. Never use /mnt/data or other local file paths here.
Clear enough to supply this argument.
video › durationSeconds
Source video duration in seconds. Pass when known — videos must be under 40s.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
videoFile › download_url
Temporary ChatGPT file download URL.
Clear enough to supply this argument.
videoFile › file_id
ChatGPT file ID.
Explain the meaning of this value, not just its name.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "upscale_video",
"title": "Upscale Video",
"description": "Upscale an existing video to a higher resolution (up to 4K) using Runway's AI video upscaler. Use this to sharpen, clean up, and increase the resolution of footage — it does NOT change the content, motion, style, or duration; for those edits use `generate_video` (video-to-video).\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nWHEN TO USE:\n- The user wants a low-res / soft / blurry video made crisper or larger (e.g. \"upscale this to 4K\", \"make this video higher resolution\", \"enhance the quality of this clip\").\n- The user has a finished clip (their own upload, or an output from another tool that they re-uploaded) and wants a higher-fidelity version of the exact same video.\n\nWHEN NOT TO USE:\n- To restyle, edit, remove/replace backgrounds, remove objects, or otherwise change the content → use `generate_video` with `referenceVideo`.\n- To generate a brand-new video → use `generate_video`.\n\nINPUT:\n- Provide the source video as `video.url`. This must be a Runway-hosted `/datasets/<UUID>.<ext>` URL returned by `complete_upload`, OR a public HTTPS video URL (the tool imports it into Runway automatically).\n- Pass `video.durationSeconds` when known. Videos must be under 40 seconds.\n\nREUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.\n\nHANDLING USER ATTACHMENTS:\n- Claude/Cursor/local agents: if the user attached a local video, run `init_upload` -> curl -> `complete_upload` first to get a Runway-hosted asset URL, then pass it as `video.url`.\n- ChatGPT: if it supplies an uploaded video as `videoFile`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as the source. NEVER pass local paths such as `/mnt/data/file.mp4` as `video.url`; use `videoFile` instead.\n\nREUSING ASSETS: A URL from `complete_upload` is a stable, hosted asset URL — pass it directly as `video.url`, never re-upload. Note: outputs of `generate_video` are NOT yet usable as `video.url` here (re-upload via `init_upload` if needed).\n\nUSER-FACING REPLIES: Talk about the result (\"upscaling your video to 4K\"), not the underlying engine. Do not mention the model/provider unless the user explicitly asks.\n\nParameters:\n- video: `{ url, durationSeconds? }` — the source video to upscale. URL must be public HTTPS or a Runway-hosted `/datasets` URL. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `videoFile` for ChatGPT uploads.\n- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. Use this instead of `video` only when ChatGPT provides a user-uploaded file.",
"inputSchema": {
"type": "object",
"properties": {
"video": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Public HTTPS or Runway-hosted video URL. Never use /mnt/data or other local file paths here."
},
"taskId": {
"type": "string",
"description": "Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated."
},
"durationSeconds": {
"type": "number",
"description": "Source video duration in seconds. Pass when known — videos must be under 40s."
}
},
"description": "Source video to upscale. URL must be a public HTTPS video URL or a Runway-hosted /datasets/<UUID>.<ext> URL returned by complete_upload. Do not pass ChatGPT local paths like /mnt/data/...; use videoFile instead.",
"additionalProperties": false
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"videoFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. video/mp4."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source video. Prefer video.url when you already have a public or Runway-hosted URL.",
"additionalProperties": false
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Upscale Video",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Convert an existing standard (SDR) video into a true HDR delivery using Runway Ruby. This regrades the footage — expanding highlights, contrast and colour into an HDR container — it does NOT change the content, motion, style, or duration; for those edits use `generate_video` or `edit_video`.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
WHEN TO USE:
- The user wants an HDR master or HDR delivery of a clip they already have ("make this HDR", "convert to HDR10", "give me an HLG version", "HDR ProRes for my edit").
WHEN NOT TO USE:
- To restyle, edit or regenerate the footage → `generate_video` / `edit_video`.
- To raise resolution or sharpen → `upscale_video`.
SOURCE VIDEO CONSTRAINTS: at most 30 seconds. Cost: 20 credits per second of source video, or 40 credits per second above 4 megapixels (roughly above 4K). Pass `video.durationSeconds` when known so an over-long clip is rejected before any upload.
OUTPUT FORMATS:
- `hdr10` (default) — HDR10 master, the safest general-purpose delivery.
- `hlg` — Hybrid Log-Gamma, for broadcast-style delivery.
- `hdr_prores` — HDR ProRes `.mov` for editorial. Most browsers CANNOT play this file: tell the user to download it and open it in an editor or player, and do not describe it as viewable inline. `proresProfile` (422, 422 HQ, 4444) applies only to this format.
AVAILABILITY: account-gated. If the call comes back saying HDR conversion is not enabled, or that the plan does not include HDR output, this workspace does not have access — tell the user instead of retrying or switching format.
REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.
HANDLING USER ATTACHMENTS:
- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`.
- ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths such as `/mnt/data/file.mp4` as `video.url`. Pass the duration alongside it as `video: {durationSeconds}`.
USER-FACING REPLIES: Talk about the result ("converting your clip to HDR10"), not the underlying engine. Do not mention the model unless the user explicitly asks.
Parameters:
- video: `{url?, durationSeconds?, assetId?}` — the source video. URL must be a Runway-hosted `/datasets` URL (from `complete_upload`) or a public HTTPS video URL; omit it only when passing `videoFile`.
- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`.
- outputFormat: Optional, default `hdr10`. One of hdr10, hlg, hdr_prores.
- proresProfile: Optional, only valid with `outputFormat: 'hdr_prores'`. One of 422, 422 HQ, 4444.
The description explains the capability, when to use it, and what to expect.
Inputs94/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
video › taskId
Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated.
Clear enough to supply this argument.
video › url
Runway-hosted dataset URL from complete_upload (preferred) or public HTTPS video URL. Never use /mnt/data or other local file paths here. Omit when passing videoFile.
Clear enough to supply this argument.
video › durationSeconds
Source video duration in seconds. Pass when known — HDR conversion supports clips up to 30s.
Clear enough to supply this argument.
video › assetId
Runway dataset asset id for the source video. Do NOT invent this — only pass a value a previous call returned for the SAME video, so the tool reuses the registered dataset instead of re-uploading.
Clear enough to supply this argument.
videoFile › download_url
Temporary ChatGPT file download URL.
Clear enough to supply this argument.
videoFile › file_id
ChatGPT file ID.
Clear enough to supply this argument.
videoFile › mime_type
MIME type, e.g. video/mp4.
Clear enough to supply this argument.
videoFile › file_name
Original file name.
Clear enough to supply this argument.
outputFormat
HDR delivery format (default hdr10). hdr_prores returns a .mov most browsers cannot play — the user must download it.
Clear enough to supply this argument.
proresProfile
ProRes tier, only valid with outputFormat: 'hdr_prores' (default '422 HQ').
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
Annotations100/100
All four behavior hints are declared.
{
"title": "Convert Video to HDR",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
Output schema87/100
The output schema explains the structured result.
View output schema
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "convert_video_to_hdr",
"title": "Convert Video to HDR",
"description": "Convert an existing standard (SDR) video into a true HDR delivery using Runway Ruby. This regrades the footage — expanding highlights, contrast and colour into an HDR container — it does NOT change the content, motion, style, or duration; for those edits use `generate_video` or `edit_video`.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nWHEN TO USE:\n- The user wants an HDR master or HDR delivery of a clip they already have (\"make this HDR\", \"convert to HDR10\", \"give me an HLG version\", \"HDR ProRes for my edit\").\n\nWHEN NOT TO USE:\n- To restyle, edit or regenerate the footage → `generate_video` / `edit_video`.\n- To raise resolution or sharpen → `upscale_video`.\n\nSOURCE VIDEO CONSTRAINTS: at most 30 seconds. Cost: 20 credits per second of source video, or 40 credits per second above 4 megapixels (roughly above 4K). Pass `video.durationSeconds` when known so an over-long clip is rejected before any upload.\n\nOUTPUT FORMATS:\n- `hdr10` (default) — HDR10 master, the safest general-purpose delivery.\n- `hlg` — Hybrid Log-Gamma, for broadcast-style delivery.\n- `hdr_prores` — HDR ProRes `.mov` for editorial. Most browsers CANNOT play this file: tell the user to download it and open it in an editor or player, and do not describe it as viewable inline. `proresProfile` (422, 422 HQ, 4444) applies only to this format.\n\nAVAILABILITY: account-gated. If the call comes back saying HDR conversion is not enabled, or that the plan does not include HDR output, this workspace does not have access — tell the user instead of retrying or switching format.\n\nREUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.\n\nHANDLING USER ATTACHMENTS:\n- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`.\n- ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths such as `/mnt/data/file.mp4` as `video.url`. Pass the duration alongside it as `video: {durationSeconds}`.\n\nUSER-FACING REPLIES: Talk about the result (\"converting your clip to HDR10\"), not the underlying engine. Do not mention the model unless the user explicitly asks.\n\nParameters:\n- video: `{url?, durationSeconds?, assetId?}` — the source video. URL must be a Runway-hosted `/datasets` URL (from `complete_upload`) or a public HTTPS video URL; omit it only when passing `videoFile`.\n- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`.\n- outputFormat: Optional, default `hdr10`. One of hdr10, hlg, hdr_prores.\n- proresProfile: Optional, only valid with `outputFormat: 'hdr_prores'`. One of 422, 422 HQ, 4444.",
"inputSchema": {
"type": "object",
"properties": {
"video": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Runway-hosted dataset URL from complete_upload (preferred) or public HTTPS video URL. Never use /mnt/data or other local file paths here. Omit when passing videoFile."
},
"taskId": {
"type": "string",
"description": "Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated."
},
"assetId": {
"type": "string",
"description": "Runway dataset asset id for the source video. Do NOT invent this — only pass a value a previous call returned for the SAME video, so the tool reuses the registered dataset instead of re-uploading."
},
"durationSeconds": {
"type": "number",
"description": "Source video duration in seconds. Pass when known — HDR conversion supports clips up to 30s."
}
},
"description": "Source video to convert. url is required unless videoFile is provided. Do not pass ChatGPT local paths like /mnt/data/...; use videoFile instead.",
"additionalProperties": false
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"videoFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. video/mp4."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source video. Prefer video.url when you already have a public or Runway-hosted URL.",
"additionalProperties": false
},
"outputFormat": {
"enum": [
"hdr10",
"hlg",
"hdr_prores"
],
"type": "string",
"description": "HDR delivery format (default hdr10). hdr_prores returns a .mov most browsers cannot play — the user must download it."
},
"proresProfile": {
"enum": [
"422",
"422 HQ",
"4444"
],
"type": "string",
"description": "ProRes tier, only valid with outputFormat: 'hdr_prores' (default '422 HQ')."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Convert Video to HDR",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Convert an existing video — Runway-made or not — to a different frame rate (24, 25, 30, 48, 50, 60, 120, 23.98, 29.97, 59.94 fps) using Runway's Enhance Frame Rate. This synthesises the in-between frames for smooth motion at the new rate; it does NOT change the content, resolution, style, or duration of the footage. For those edits use `generate_video`, `edit_video` or `upscale_video`.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
WHEN TO USE:
- The user needs a clip at a specific delivery frame rate: broadcast (25 or 59.94 fps), an editing timeline (25 or 30 fps), a client spec, or smoother high-frame-rate playback (48, 60, 120 fps). Runway generations are 24 fps, so "make this 30fps", "I need 59.94 for broadcast", "convert to 60fps" all belong here.
WHEN NOT TO USE:
- To raise resolution or sharpen → `upscale_video`.
- To restyle, edit, extend or regenerate the footage → `generate_video` / `edit_video` / `expand_video`.
- To grade into HDR → `convert_video_to_hdr`.
- For a frame rate that is not in the list above: there is no custom fps. Offer the nearest preset instead.
SOURCE VIDEO CONSTRAINTS: at most 300 seconds (5 minutes), up to 4K (3840x2160 in either orientation), aspect ratio at most 4:1. Standard (SDR) sources only — HDR inputs are not supported yet. Cost: 1 credit per 2 seconds of source video (minimum 1 credit), so a 60s clip costs 30 credits. Pass `video.durationSeconds` when known so an over-long clip is rejected before any upload.
TARGET FRAME RATE: `targetFrameRate` is required. Common picks — `25` (PAL broadcast / EU timelines), `29.97` or `30` (NTSC / US timelines, social), `59.94` (US broadcast HFR), `48`, `50`, `60`, `120` (smooth / slow-motion sources). `23.98` and `24` are also available for conforming footage that is not already 24 fps.
AVAILABILITY: paid plans only, and account-gated. If the call comes back saying the feature is not enabled or requires a paid plan, tell the user instead of retrying or switching frame rate.
REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.
HANDLING USER ATTACHMENTS:
- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`.
- ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths such as `/mnt/data/file.mp4` as `video.url`. Pass the duration alongside it as `video: {durationSeconds}`.
USER-FACING REPLIES: Talk about the result ("converting your clip to 59.94 fps"), not the underlying engine. Do not mention the model unless the user explicitly asks.
Parameters:
- video: `{url?, durationSeconds?, assetId?, taskId?}` — the source video. URL must be a Runway-hosted `/datasets` URL (from `complete_upload`) or a public HTTPS video URL; omit it only when passing `videoFile` or `video.taskId`.
- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`.
- targetFrameRate: Required. One of 24, 25, 30, 48, 50, 60, 120, 23.98, 29.97, 59.94 (fps, as a string).
The description explains the capability, when to use it, and what to expect.
Inputs95/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
video › taskId
Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated.
Clear enough to supply this argument.
video › url
Runway-hosted dataset URL from complete_upload (preferred) or public HTTPS video URL. Never use /mnt/data or other local file paths here. Omit when passing videoFile.
Clear enough to supply this argument.
video › durationSeconds
Source video duration in seconds. Pass when known — frame rate conversion supports clips up to 300s and is billed at 1 credit per 2s.
Clear enough to supply this argument.
video › assetId
Runway dataset asset id for the source video. Do NOT invent this — only pass a value a previous call returned for the SAME video, so the tool reuses the registered dataset instead of re-uploading.
Clear enough to supply this argument.
videoFile › download_url
Temporary ChatGPT file download URL.
Clear enough to supply this argument.
videoFile › file_id
ChatGPT file ID.
Clear enough to supply this argument.
videoFile › mime_type
MIME type, e.g. video/mp4.
Clear enough to supply this argument.
videoFile › file_name
Original file name.
Clear enough to supply this argument.
targetFrameRate
Output frame rate in fps. 25 = PAL broadcast, 29.97/30 = NTSC timelines, 59.94 = US broadcast HFR, 48/50/60/120 = smooth or slow-motion delivery. No custom values.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "enhance_frame_rate",
"title": "Enhance Frame Rate",
"description": "Convert an existing video — Runway-made or not — to a different frame rate (24, 25, 30, 48, 50, 60, 120, 23.98, 29.97, 59.94 fps) using Runway's Enhance Frame Rate. This synthesises the in-between frames for smooth motion at the new rate; it does NOT change the content, resolution, style, or duration of the footage. For those edits use `generate_video`, `edit_video` or `upscale_video`.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nWHEN TO USE:\n- The user needs a clip at a specific delivery frame rate: broadcast (25 or 59.94 fps), an editing timeline (25 or 30 fps), a client spec, or smoother high-frame-rate playback (48, 60, 120 fps). Runway generations are 24 fps, so \"make this 30fps\", \"I need 59.94 for broadcast\", \"convert to 60fps\" all belong here.\n\nWHEN NOT TO USE:\n- To raise resolution or sharpen → `upscale_video`.\n- To restyle, edit, extend or regenerate the footage → `generate_video` / `edit_video` / `expand_video`.\n- To grade into HDR → `convert_video_to_hdr`.\n- For a frame rate that is not in the list above: there is no custom fps. Offer the nearest preset instead.\n\nSOURCE VIDEO CONSTRAINTS: at most 300 seconds (5 minutes), up to 4K (3840x2160 in either orientation), aspect ratio at most 4:1. Standard (SDR) sources only — HDR inputs are not supported yet. Cost: 1 credit per 2 seconds of source video (minimum 1 credit), so a 60s clip costs 30 credits. Pass `video.durationSeconds` when known so an over-long clip is rejected before any upload.\n\nTARGET FRAME RATE: `targetFrameRate` is required. Common picks — `25` (PAL broadcast / EU timelines), `29.97` or `30` (NTSC / US timelines, social), `59.94` (US broadcast HFR), `48`, `50`, `60`, `120` (smooth / slow-motion sources). `23.98` and `24` are also available for conforming footage that is not already 24 fps.\n\nAVAILABILITY: paid plans only, and account-gated. If the call comes back saying the feature is not enabled or requires a paid plan, tell the user instead of retrying or switching frame rate.\n\nREUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it.\n\nHANDLING USER ATTACHMENTS:\n- Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`.\n- ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths such as `/mnt/data/file.mp4` as `video.url`. Pass the duration alongside it as `video: {durationSeconds}`.\n\nUSER-FACING REPLIES: Talk about the result (\"converting your clip to 59.94 fps\"), not the underlying engine. Do not mention the model unless the user explicitly asks.\n\nParameters:\n- video: `{url?, durationSeconds?, assetId?, taskId?}` — the source video. URL must be a Runway-hosted `/datasets` URL (from `complete_upload`) or a public HTTPS video URL; omit it only when passing `videoFile` or `video.taskId`.\n- videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`.\n- targetFrameRate: Required. One of 24, 25, 30, 48, 50, 60, 120, 23.98, 29.97, 59.94 (fps, as a string).",
"inputSchema": {
"type": "object",
"required": [
"targetFrameRate"
],
"properties": {
"video": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Runway-hosted dataset URL from complete_upload (preferred) or public HTTPS video URL. Never use /mnt/data or other local file paths here. Omit when passing videoFile."
},
"taskId": {
"type": "string",
"description": "Runway task ID of a finished video generation from earlier in this conversation (e.g. a generate_video result). The server resolves the video URL and duration itself — pass this instead of url, and never ask the user to upload a video Runway just generated."
},
"assetId": {
"type": "string",
"description": "Runway dataset asset id for the source video. Do NOT invent this — only pass a value a previous call returned for the SAME video, so the tool reuses the registered dataset instead of re-uploading."
},
"durationSeconds": {
"type": "number",
"description": "Source video duration in seconds. Pass when known — frame rate conversion supports clips up to 300s and is billed at 1 credit per 2s."
}
},
"description": "Source video to convert. url is required unless videoFile or taskId is provided. Do not pass ChatGPT local paths like /mnt/data/...; use videoFile instead.",
"additionalProperties": false
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"videoFile": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. video/mp4."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded source video. Prefer video.url when you already have a public or Runway-hosted URL.",
"additionalProperties": false
},
"targetFrameRate": {
"enum": [
"24",
"25",
"30",
"48",
"50",
"60",
"120",
"23.98",
"29.97",
"59.94"
],
"type": "string",
"description": "Output frame rate in fps. 25 = PAL broadcast, 29.97/30 = NTSC timelines, 59.94 = US broadcast HFR, 48/50/60/120 = smooth or slow-motion delivery. No custom values."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Enhance Frame Rate",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description96/100
Join 2-20 video clips end to end into one video file, in the order given, with hard cuts or crossfades.
CREDIT MODE: Runway account availability is checked against the currently connected workspace.
WHEN TO USE:
- The user generated several clips (e.g. multiple `generate_video` shots of a story) and wants them as one video: "put these together", "combine the clips", "make it one video", "join them into a 30s ad".
- The user has their own clips and wants them assembled in a specific order.
WHEN NOT TO USE:
- To generate a new multi-scene video from a story → `generate_multishot_video`.
- To make one clip longer by generating more footage → `expand_video` / `generate_video`.
- Side-by-side / split-screen layouts, trimming, text overlays or adding music are not supported by this tool.
CLIPS: `clips` is the ordered list of sources. For each clip:
- A video a Runway tool generated earlier in this conversation → pass `{taskId}` only. The server looks up the video and its duration. Do NOT ask the user to download or re-upload it.
- Any other video → pass `{url, durationSeconds}`. `durationSeconds` is REQUIRED for URL clips (the stitch checks the total length against the files). Local agents: read it with `ffprobe`; upload local files first via `init_upload` -> curl -> `complete_upload`.
- Clips should share an aspect ratio; mixed ratios are stitched but will not fill the frame consistently.
TRANSITIONS: `transition` is `cut` (default, hard cuts) or `crossfade`. `crossfadeSeconds` (0.1-4.9, default 0.5) sets the overlap; each crossfade shortens the result by that amount.
LIMITS & COST: total length at most 60 minutes. Cost: 1 credit per 20 seconds of total input (minimum 1 credit), so three 10s clips cost 2 credits. Paid Runway plans only — on a free workspace this returns an account-limitation result without creating a task; do not retry.
USER-FACING REPLIES: Talk about the result ("joining your 3 clips into one 30s video"), not the underlying processing.
Parameters:
- clips: Required. 2-20 items in play order, each `{taskId}` or `{url, durationSeconds, assetId?}`.
- transition: `'cut'` (default) | `'crossfade'`.
- crossfadeSeconds: Crossfade length in seconds when `transition` is `'crossfade'`.
The description explains the capability, when to use it, and what to expect.
Inputs93/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
clips › items › taskId
Runway task ID of a finished video generation from earlier in this conversation. The server resolves the video and its duration — pass this instead of url.
Clear enough to supply this argument.
clips › items › url
Runway-hosted dataset URL from complete_upload or a public HTTPS video URL. Never a local path like /mnt/data/....
Clear enough to supply this argument.
clips › items › durationSeconds
Clip length in seconds. Required with url; optional with taskId.
Clear enough to supply this argument.
clips › items › assetId
Runway dataset asset id for this url. Do NOT invent this — only pass a value a previous call returned for the SAME video.
Clear enough to supply this argument.
transition
How clips meet: 'cut' (default, hard cut) or 'crossfade'.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
crossfadeSeconds
Crossfade length in seconds (default 0.5). Only used when transition is 'crossfade'.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
}
Original tool metadata
{
"name": "stitch_videos",
"title": "Stitch Videos",
"description": "Join 2-20 video clips end to end into one video file, in the order given, with hard cuts or crossfades.\n\nCREDIT MODE: Runway account availability is checked against the currently connected workspace.\n\nWHEN TO USE:\n- The user generated several clips (e.g. multiple `generate_video` shots of a story) and wants them as one video: \"put these together\", \"combine the clips\", \"make it one video\", \"join them into a 30s ad\".\n- The user has their own clips and wants them assembled in a specific order.\n\nWHEN NOT TO USE:\n- To generate a new multi-scene video from a story → `generate_multishot_video`.\n- To make one clip longer by generating more footage → `expand_video` / `generate_video`.\n- Side-by-side / split-screen layouts, trimming, text overlays or adding music are not supported by this tool.\n\nCLIPS: `clips` is the ordered list of sources. For each clip:\n- A video a Runway tool generated earlier in this conversation → pass `{taskId}` only. The server looks up the video and its duration. Do NOT ask the user to download or re-upload it.\n- Any other video → pass `{url, durationSeconds}`. `durationSeconds` is REQUIRED for URL clips (the stitch checks the total length against the files). Local agents: read it with `ffprobe`; upload local files first via `init_upload` -> curl -> `complete_upload`.\n- Clips should share an aspect ratio; mixed ratios are stitched but will not fill the frame consistently.\n\nTRANSITIONS: `transition` is `cut` (default, hard cuts) or `crossfade`. `crossfadeSeconds` (0.1-4.9, default 0.5) sets the overlap; each crossfade shortens the result by that amount.\n\nLIMITS & COST: total length at most 60 minutes. Cost: 1 credit per 20 seconds of total input (minimum 1 credit), so three 10s clips cost 2 credits. Paid Runway plans only — on a free workspace this returns an account-limitation result without creating a task; do not retry.\n\nUSER-FACING REPLIES: Talk about the result (\"joining your 3 clips into one 30s video\"), not the underlying processing.\n\nParameters:\n- clips: Required. 2-20 items in play order, each `{taskId}` or `{url, durationSeconds, assetId?}`.\n- transition: `'cut'` (default) | `'crossfade'`.\n- crossfadeSeconds: Crossfade length in seconds when `transition` is `'crossfade'`.",
"inputSchema": {
"type": "object",
"required": [
"clips"
],
"properties": {
"clips": {
"type": "array",
"items": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Runway-hosted dataset URL from complete_upload or a public HTTPS video URL. Never a local path like /mnt/data/...."
},
"taskId": {
"type": "string",
"description": "Runway task ID of a finished video generation from earlier in this conversation. The server resolves the video and its duration — pass this instead of url."
},
"assetId": {
"type": "string",
"description": "Runway dataset asset id for this url. Do NOT invent this — only pass a value a previous call returned for the SAME video."
},
"durationSeconds": {
"type": "number",
"description": "Clip length in seconds. Required with url; optional with taskId."
}
},
"additionalProperties": false
},
"maxItems": 20,
"minItems": 2,
"description": "Clips to join, in play order. Use {taskId} for videos Runway generated in this conversation; use {url, durationSeconds} for anything else."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"transition": {
"enum": [
"cut",
"crossfade"
],
"type": "string",
"description": "How clips meet: 'cut' (default, hard cut) or 'crossfade'."
},
"crossfadeSeconds": {
"type": "number",
"maximum": 4.9,
"minimum": 0.1,
"description": "Crossfade length in seconds (default 0.5). Only used when transition is 'crossfade'."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"type": "string",
"description": "Result discriminator, e.g. 'task_pending', 'task_batch_pending', 'account_limitation', 'model_deprecated'."
},
"meta": {
"type": "object",
"description": "Viewer metadata: model id/label, aspect ratio, resolution, dimensions, duration, and audio flags."
},
"prompt": {
"type": "string",
"description": "Prompt the task was created with."
},
"reason": {
"type": "string",
"description": "Reason for account_limitation or model_deprecated results."
},
"status": {
"type": "string",
"description": "Last observed task status for still-running tasks."
},
"taskId": {
"type": "string",
"description": "Runway task ID for single-task results. Use with get_task."
},
"message": {
"type": "string",
"description": "Human-readable API detail when kind is model_deprecated."
},
"taskIds": {
"type": "array",
"items": {
"type": "string"
},
"description": "Runway task IDs for batch results (count > 1)."
},
"nextTool": {
"type": "string",
"description": "Tool to call next when kind is account_limitation or model_deprecated."
},
"taskType": {
"type": "string",
"description": "Internal Runway task type."
},
"topUpUrl": {
"type": "string",
"description": "Runway billing page to buy more credits when lowCredits is true on a paid plan. Free plans cannot top up; they get upgradeUrl instead."
},
"capability": {
"type": "string",
"description": "What the plan blocked, phrased as the user's own action (e.g. 'Generating video'). Pass it to `show_plans_and_credits` as `gatedCapability` and use it when telling the user what happened."
},
"isFreePlan": {
"type": "boolean",
"description": "True when the connected workspace is on the Free plan. The inline viewer shows a one-line upgrade nudge under the finished media; you do not need to add anything."
},
"lowCredits": {
"type": "boolean",
"description": "True when the remaining balance may not cover another generation — mention it to the user in one short sentence with topUpUrl."
},
"modelLabel": {
"type": "string",
"description": "Human-readable model label for account-limitation results."
},
"outputKind": {
"enum": [
"image",
"video",
"audio"
],
"type": "string",
"description": "Whether the finished asset will be an image, a video, or audio."
},
"upgradeUrl": {
"type": "string",
"description": "Runway pricing/billing page. Present when kind is account_limitation, and on every accepted result from a free-plan workspace (see isFreePlan). The inline viewer renders it; do not repeat it unless lowCredits is true or the user asks about plans."
},
"creditsRemaining": {
"type": "number",
"description": "Approximate workspace credit balance after this submission, when it could be resolved."
}
},
"description": "Result envelope discriminated by `kind`. `task_pending` / `task_batch_pending` mean the generation task was accepted by Runway and is running in the background (poll get_task or let the inline viewer update); `account_limitation` means the connected workspace cannot run the request; `model_deprecated` means the model is retired and whoami should be called before offering alternatives; other kinds carry staged results or corrective guidance.",
"additionalProperties": true
},
"annotations": {
"title": "Stitch Videos",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description98/100
Initialize a file upload to Runway. Returns temporary upload URLs for direct upload.
USER-FACING REPLIES: Do not mention provider-specific storage services. Say "Runway upload", "Runway-hosted asset", or "temporary upload URL" instead.
USE THIS ONLY WHEN: The user has attached a NEW local file (image or video) that isn't already on Runway.
CHATGPT: If ChatGPT supplies the uploaded file as `file`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, complete the upload, and return the final asset URL. Do not ask the user to run curl.
CHATGPT: Do NOT call this tool with only `filename`, `fileSize`, and `mimeType`; that only returns curl instructions and cannot upload bytes from ChatGPT. For image-to-video, call `generate_video` with `startImageFile`. For video-to-video, call `generate_video` with `referenceVideoFile`.
DO NOT USE THIS FOR:
- URLs returned by `generate_image`, `generate_video`, or `generate_multishot_video`. Those URLs are already stable Runway-hosted assets — pass them directly as `startFrame.url` / `referenceImages[].url` / `referenceVideo.url`. Re-uploading wastes a turn and creates a duplicate asset.
- Public HTTPS image URLs: pass them directly to the generation/editing tool.
CLAUDE/CURSOR/LOCAL WORKFLOW:
1. Get the file size and MIME type using bash:
```bash
wc -c < "/absolute/path/to/your/file" | tr -d ' '
file --mime-type -b "/absolute/path/to/your/file"
```
2. Call this tool with the filename, fileSize, and mimeType.
3. Use bash/curl to upload the file to the temporary upload URL(s). The curl commands this tool returns include `-D -` so the response headers (including ETag) are written to stdout.
4. Capture the ETag header from each PUT response (it looks like `etag: "abc123..."`; strip the surrounding quotes).
5. Call `complete_upload` with the uploadId AND the parts array — `parts` is required even for single-part uploads.
6. Use the returned assetUrl as startFrame.url / referenceImages[].url / referenceVideo.url in the generation tools.
Parameters:
- file: ChatGPT-only uploaded file param `{download_url, file_id, mime_type, file_name}`. If present, the server completes the upload and returns an asset URL directly.
- filename: Name for the uploaded file (e.g., "video.mp4")
- fileSize: Size in bytes (from wc -c)
- mimeType: MIME type (from file --mime-type). Supported: image/jpeg, image/png, image/webp, image/gif, video/mp4, video/quicktime, video/webm
The description explains the capability, when to use it, and what to expect.
Inputs82/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
file › download_url
Temporary ChatGPT file download URL.
Clear enough to supply this argument.
file › file_id
ChatGPT file ID.
Clear enough to supply this argument.
file › mime_type
MIME type, e.g. image/png or video/mp4.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
file › file_name
Original file name.
Clear enough to supply this argument.
filename
Filename for the upload.
Explain the meaning of this value, not just its name.
fileSize
File size in bytes. Must be greater than 0.
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"enum": [
"upload_complete",
"upload_initialized",
"chatgpt_missing_file_param"
],
"type": "string"
},
"assetUrl": {
"type": "string",
"description": "Runway-hosted asset URL, present when kind is 'upload_complete'."
},
"mimeType": {
"type": "string",
"description": "Content-Type to send with each PUT."
},
"uploadId": {
"type": "string",
"description": "Upload id for complete_upload, present when kind is 'upload_initialized'."
},
"uploadUrls": {
"type": "array",
"items": {
"type": "string"
},
"description": "Temporary presigned PUT URLs, one per part in order, present when kind is 'upload_initialized'. PUT the file bytes to these URLs and capture the ETag response header from each."
},
"instructions": {
"type": "string",
"description": "Step-by-step upload instructions including ready-to-run curl commands and the complete_upload call template."
},
"numberOfParts": {
"type": "number",
"description": "How many parts the file must be uploaded in."
},
"partSizeBytes": {
"type": "number",
"description": "Size of each part except the last. Part N (1-based) covers bytes [(N-1)*partSizeBytes, min(N*partSizeBytes, fileSize))."
},
"uploadHeaders": {
"type": "object",
"description": "Extra headers that must be sent with each PUT (in addition to Content-Type).",
"additionalProperties": {
"type": "string"
}
}
},
"description": "Upload state discriminated by `kind`: 'upload_complete' (ChatGPT file path — `assetUrl` is ready to use) or 'upload_initialized' (curl path — upload the bytes, then call complete_upload with `uploadId`).",
"additionalProperties": true
}
Original tool metadata
{
"name": "init_upload",
"title": "Start Upload",
"description": "Initialize a file upload to Runway. Returns temporary upload URLs for direct upload.\n\nUSER-FACING REPLIES: Do not mention provider-specific storage services. Say \"Runway upload\", \"Runway-hosted asset\", or \"temporary upload URL\" instead.\n\nUSE THIS ONLY WHEN: The user has attached a NEW local file (image or video) that isn't already on Runway.\n\nCHATGPT: If ChatGPT supplies the uploaded file as `file`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, complete the upload, and return the final asset URL. Do not ask the user to run curl.\nCHATGPT: Do NOT call this tool with only `filename`, `fileSize`, and `mimeType`; that only returns curl instructions and cannot upload bytes from ChatGPT. For image-to-video, call `generate_video` with `startImageFile`. For video-to-video, call `generate_video` with `referenceVideoFile`.\n\nDO NOT USE THIS FOR:\n- URLs returned by `generate_image`, `generate_video`, or `generate_multishot_video`. Those URLs are already stable Runway-hosted assets — pass them directly as `startFrame.url` / `referenceImages[].url` / `referenceVideo.url`. Re-uploading wastes a turn and creates a duplicate asset.\n- Public HTTPS image URLs: pass them directly to the generation/editing tool.\n\nCLAUDE/CURSOR/LOCAL WORKFLOW:\n1. Get the file size and MIME type using bash:\n ```bash\n wc -c < \"/absolute/path/to/your/file\" | tr -d ' '\n file --mime-type -b \"/absolute/path/to/your/file\"\n ```\n\n2. Call this tool with the filename, fileSize, and mimeType.\n\n3. Use bash/curl to upload the file to the temporary upload URL(s). The curl commands this tool returns include `-D -` so the response headers (including ETag) are written to stdout.\n\n4. Capture the ETag header from each PUT response (it looks like `etag: \"abc123...\"`; strip the surrounding quotes).\n\n5. Call `complete_upload` with the uploadId AND the parts array — `parts` is required even for single-part uploads.\n\n6. Use the returned assetUrl as startFrame.url / referenceImages[].url / referenceVideo.url in the generation tools.\n\nParameters:\n- file: ChatGPT-only uploaded file param `{download_url, file_id, mime_type, file_name}`. If present, the server completes the upload and returns an asset URL directly.\n- filename: Name for the uploaded file (e.g., \"video.mp4\")\n- fileSize: Size in bytes (from wc -c)\n- mimeType: MIME type (from file --mime-type). Supported: image/jpeg, image/png, image/webp, image/gif, video/mp4, video/quicktime, video/webm",
"inputSchema": {
"type": "object",
"properties": {
"file": {
"type": "object",
"required": [
"download_url",
"file_id"
],
"properties": {
"file_id": {
"type": "string",
"description": "ChatGPT file ID."
},
"file_name": {
"type": "string",
"description": "Original file name."
},
"mime_type": {
"type": "string",
"description": "MIME type, e.g. image/png or video/mp4."
},
"download_url": {
"type": "string",
"description": "Temporary ChatGPT file download URL."
}
},
"description": "ChatGPT-only file param for a user-uploaded image or video. Prefer this over filename/fileSize/mimeType when ChatGPT provides it.",
"additionalProperties": false
},
"fileSize": {
"type": "number",
"minimum": 1,
"description": "File size in bytes. Must be greater than 0."
},
"filename": {
"type": "string",
"description": "Filename for the upload."
},
"mimeType": {
"enum": [
"image/jpeg",
"image/png",
"image/webp",
"image/gif",
"video/mp4",
"video/quicktime",
"video/webm"
],
"type": "string",
"description": "MIME type. Supported: image/jpeg, image/png, image/webp, image/gif, video/mp4, video/quicktime, video/webm."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"enum": [
"upload_complete",
"upload_initialized",
"chatgpt_missing_file_param"
],
"type": "string"
},
"assetUrl": {
"type": "string",
"description": "Runway-hosted asset URL, present when kind is 'upload_complete'."
},
"mimeType": {
"type": "string",
"description": "Content-Type to send with each PUT."
},
"uploadId": {
"type": "string",
"description": "Upload id for complete_upload, present when kind is 'upload_initialized'."
},
"uploadUrls": {
"type": "array",
"items": {
"type": "string"
},
"description": "Temporary presigned PUT URLs, one per part in order, present when kind is 'upload_initialized'. PUT the file bytes to these URLs and capture the ETag response header from each."
},
"instructions": {
"type": "string",
"description": "Step-by-step upload instructions including ready-to-run curl commands and the complete_upload call template."
},
"numberOfParts": {
"type": "number",
"description": "How many parts the file must be uploaded in."
},
"partSizeBytes": {
"type": "number",
"description": "Size of each part except the last. Part N (1-based) covers bytes [(N-1)*partSizeBytes, min(N*partSizeBytes, fileSize))."
},
"uploadHeaders": {
"type": "object",
"description": "Extra headers that must be sent with each PUT (in addition to Content-Type).",
"additionalProperties": {
"type": "string"
}
}
},
"description": "Upload state discriminated by `kind`: 'upload_complete' (ChatGPT file path — `assetUrl` is ready to use) or 'upload_initialized' (curl path — upload the bytes, then call complete_upload with `uploadId`).",
"additionalProperties": true
},
"annotations": {
"title": "Start Upload",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description98/100
Finalize a file upload and get the asset URL.
Call this AFTER successfully uploading the file bytes to the temporary upload URL(s) from `init_upload`.
USER-FACING REPLIES: Do not mention provider-specific storage services. Say "Runway upload", "Runway-hosted asset", or "temporary upload URL" instead.
`parts` is REQUIRED — even for single-part uploads. Each entry corresponds to one PUT response and carries the ETag from that response's `etag` header (strip surrounding quotes if present).
Returns the asset URL to use as `startFrame` / `endFrame` / `referenceImages[].url` in the generation tools, or `referenceVideo.url` in `generate_video` for video-to-video edits.
The description explains the capability, when to use it, and what to expect.
Inputs50/100
2 of 4 inputs have descriptions. Add descriptions to the missing inputs, including where to obtain IDs and how to supply values.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
uploadId
The uploadId from init_upload.
Clear enough to supply this argument.
parts › items › partNumber
Not provided
Missing description. Explain what this value means and how to supply it.
parts › items › etag
Not provided
Missing description. Explain what this value means and how to supply it.
{
"name": "complete_upload",
"title": "Complete Upload",
"description": "Finalize a file upload and get the asset URL.\n\nCall this AFTER successfully uploading the file bytes to the temporary upload URL(s) from `init_upload`.\n\nUSER-FACING REPLIES: Do not mention provider-specific storage services. Say \"Runway upload\", \"Runway-hosted asset\", or \"temporary upload URL\" instead.\n\n`parts` is REQUIRED — even for single-part uploads. Each entry corresponds to one PUT response and carries the ETag from that response's `etag` header (strip surrounding quotes if present).\n\nReturns the asset URL to use as `startFrame` / `endFrame` / `referenceImages[].url` in the generation tools, or `referenceVideo.url` in `generate_video` for video-to-video edits.",
"inputSchema": {
"type": "object",
"required": [
"uploadId",
"parts"
],
"properties": {
"parts": {
"type": "array",
"items": {
"type": "object",
"required": [
"partNumber",
"etag"
],
"properties": {
"etag": {
"type": "string"
},
"partNumber": {
"type": "number"
}
},
"additionalProperties": false
},
"minItems": 1,
"description": "Array of {partNumber, etag} from each PUT response. Required even for single-part uploads."
},
"uploadId": {
"type": "string",
"description": "The uploadId from init_upload."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind",
"assetUrl"
],
"properties": {
"kind": {
"enum": [
"upload_complete"
],
"type": "string"
},
"assetUrl": {
"type": "string",
"description": "Runway-hosted asset URL to pass into generation tools (startFrame.url, referenceImages[].url, referenceVideo.url, ...)."
}
},
"description": "Finalized upload with the reusable Runway-hosted asset URL.",
"additionalProperties": true
},
"annotations": {
"title": "Complete Upload",
"readOnlyHint": false,
"openWorldHint": false,
"idempotentHint": false,
"destructiveHint": false
}
}
No output schema is declared. Add one for structured results so clients can interpret returned fields.
Not declared
Original tool metadata
{
"name": "import-claude-motion-from-url",
"title": "Import Claude Motion Video",
"description": "Import an MP4 video into Runway from a URL.",
"inputSchema": {
"type": "object",
"required": [
"url",
"download_token"
],
"properties": {
"url": {
"type": "string",
"description": "MP4 URL. Expires within 10 minutes. Fetch it server-side."
},
"title": {
"type": "string",
"description": "Suggested title for the imported video."
},
"download_token": {
"type": "string",
"description": "Access token for the video file, given separately from url. Pass it through unchanged; never extract one from url or alter url."
},
"claude_design_project_id": {
"type": "string",
"description": "Stable identifier of the video; same on every re-send. Use it to update an existing import instead of creating a new one."
}
}
},
"annotations": {
"title": "Import Claude Motion Video",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description98/100
Lists recent uploaded and generated assets for the authenticated workspace. Returns asset IDs, media types, task IDs when available, and reusable asset URLs. Use this to help the user pick recent images or videos as references.
The description explains the capability, when to use it, and what to expect.
Inputs88/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
limit
Max results (10-50, default 12).
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Gets details for a Runway task by ID — used to check status and retrieve the result of a generation/edit task once it completes. Generation tools (`generate_image`, `generate_video`, `generate_multishot_video`, `run_workflow`) return immediately with a task ID. If an inline viewer rendered, do not poll; the viewer updates itself. If no viewer rendered, poll this tool until SUCCEEDED (wait 60-120s between calls for video/workflow). For `dynamic_workflow` tasks, the text includes overall %, per-step name/status, and output URLs when nodes finish. Parent `dynamic_workflow` SUCCEEDED can have empty artifacts — use the step Outputs, not artifacts[0]. Set `include_workflow_execution: true` only when you need node UUIDs. When status is SUCCEEDED the response includes the asset URL. USER-FACING REPLIES: describe the asset content; the `taskType` in the response encodes the model name and is for debugging only — keep it out of your reply unless the user explicitly asks which model was used.
The description explains the capability, when to use it, and what to expect.
Inputs83/100
Clarify units, where to obtain this value, or conventions not already expressed in the schema.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
id
Task ID.
Clarify units, where to obtain this value, or conventions not already expressed in the schema.
include_workflow_execution
Workflow runs only (`run_workflow`). If true, also include node UUIDs. Ignored for other tasks. Default is named steps + output URLs.
{
"name": "get_task",
"title": "Check Task Status",
"description": "Gets details for a Runway task by ID — used to check status and retrieve the result of a generation/edit task once it completes. Generation tools (`generate_image`, `generate_video`, `generate_multishot_video`, `run_workflow`) return immediately with a task ID. If an inline viewer rendered, do not poll; the viewer updates itself. If no viewer rendered, poll this tool until SUCCEEDED (wait 60-120s between calls for video/workflow). For `dynamic_workflow` tasks, the text includes overall %, per-step name/status, and output URLs when nodes finish. Parent `dynamic_workflow` SUCCEEDED can have empty artifacts — use the step Outputs, not artifacts[0]. Set `include_workflow_execution: true` only when you need node UUIDs. When status is SUCCEEDED the response includes the asset URL. USER-FACING REPLIES: describe the asset content; the `taskType` in the response encodes the model name and is for debugging only — keep it out of your reply unless the user explicitly asks which model was used.",
"inputSchema": {
"type": "object",
"required": [
"id"
],
"properties": {
"id": {
"type": "string",
"description": "Task ID."
},
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"include_workflow_execution": {
"type": "boolean",
"description": "Workflow runs only (`run_workflow`). If true, also include node UUIDs. Ignored for other tasks. Default is named steps + output URLs."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind",
"taskId"
],
"properties": {
"url": {
"type": "string",
"description": "Finished asset URL (kinds 'image', 'video', and 'audio')."
},
"kind": {
"enum": [
"task_pending",
"task_failed",
"image",
"video",
"audio",
"task_succeeded_no_artifact"
],
"type": "string"
},
"meta": {
"type": "object",
"description": "Media metadata: model id/label, width, height, duration, audio, ratio, and resolution."
},
"error": {
"type": "string",
"description": "Error message for failed tasks."
},
"prompt": {
"type": "string"
},
"status": {
"type": "string",
"description": "Raw Runway task status."
},
"taskId": {
"type": "string"
},
"taskType": {
"type": "string"
},
"errorInfo": {
"type": "object",
"description": "Categorized failure: {category, title, hint, reason?, moderationCategory?}."
},
"progressText": {
"type": "string"
},
"queuedSeconds": {
"type": "number"
},
"progressPercent": {
"type": "number"
},
"workflowExecution": {
"type": "object",
"description": "Per-node status summary for dynamic_workflow tasks."
}
},
"description": "Task state discriminated by `kind`: 'task_pending' (still running), 'task_failed', 'image'/'video'/'audio' (succeeded, `url` holds the asset), or 'task_succeeded_no_artifact'.",
"additionalProperties": true
},
"annotations": {
"title": "Check Task Status",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description99/100
Shows the connected workspace's remaining Runway credits plus available plans and top-up options as an inline card. Call it when the user asks about their credits, balance, plans, pricing, or how to buy more credits — and after a generation fails with an out-of-credits or low-credits result. After an out-of-credits failure, pass `creditsNeeded` (the approximate credit cost of that generation, from the failed tool's cost notes) to render a shortfall card. After a generation fails because the plan does not allow it, call it with `gatedCapability` set to the phrase from that error to render an upgrade card. Purchases are completed on runway.com; the card only links out. When the inline card renders, keep your reply to one short sentence — the card already shows the numbers and buttons.
The description explains the capability, when to use it, and what to expect.
Inputs99/100
Input documentation gives an agent enough context to populate the arguments.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Clear enough to supply this argument.
homeRefresh
Internal Runway home balance refresh. Omit for an inline plans card.
Clear enough to supply this argument.
creditsNeeded
Approximate credit cost of the generation that just failed with out-of-credits, estimated from the failed tool's stated cost (e.g. 28 credits/second × duration). Renders the card in shortfall mode: a "needs ~N credits — you have M" header with top-up framing. Only pass it when a generation actually failed for credits and the cost is stated or clearly estimable.
Clear enough to supply this argument.
gatedCapability
What the current plan blocked, phrased as the action the user was trying to take, copied verbatim from a paid-plan-required error (e.g. 'Generating video'). Renders the card in upgrade-gate mode: a "<capability> requires a paid plan" header, that phrase bolded in each plan's features, and "Unlock" buttons. Never pass a bare model name — most users do not know what those mean. Omit for plain balance/plans questions.
{
"type": "object",
"required": [
"kind",
"planName",
"isFreePlan",
"billingUrl",
"pricingUrl"
],
"properties": {
"kind": {
"type": "string",
"description": "Always 'plans_and_credits'."
},
"plans": {
"type": "array",
"items": {
"type": "object"
},
"description": "Self-serve plans shown as cards. A plan carries `offer: {monthlyPrice?, annualMonthlyPrice?, percentOff, durationInMonths?}` when this workspace qualifies for a discount. `durationInMonths` is only set for monthly intro coupons (first invoice); annual pricing is unchanged."
},
"topUp": {
"type": "object",
"description": "One-time credit packs: {available, minCredits, url, packs}. `url` opens the billing page with the buy-credits modal already open; only paid plans can top up."
},
"credits": {
"type": "object",
"description": "Remaining balance: {unlimited: false, total, plan, purchased} or {unlimited: true}. Omitted when it could not be resolved."
},
"planName": {
"type": "string",
"description": "Current plan display name, e.g. 'Free', 'Standard', 'Pro'."
},
"allowance": {
"type": "number",
"description": "Full-cycle credit allowance (for the usage bar)."
},
"billingUrl": {
"type": "string",
"description": "Runway billing page (change plan)."
},
"isFreePlan": {
"type": "boolean"
},
"pricingUrl": {
"type": "string",
"description": "Public pricing page with the full plan comparison."
},
"gateContext": {
"type": "object",
"description": "Present in upgrade-gate mode: {capability} for what the current plan cannot do. Any paid plan unlocks it — runwayml/api model gating is free-vs-paid only."
},
"workspaceId": {
"type": "number",
"description": "Workspace whose balance and plan are shown."
},
"shortfallContext": {
"type": "object",
"description": "Present in shortfall mode: {creditsNeeded} — the approximate cost of the generation that failed for credits."
},
"creditPackPickerVariant": {
"type": "string",
"description": "Resolved credit picker presentation. Either \"control\" or \"stacked\"."
}
},
"description": "Credit balance and plan catalog for the connected workspace, rendered by the inline plans & credits card.",
"additionalProperties": true
}
Original tool metadata
{
"name": "show_plans_and_credits",
"title": "Plans & Credits",
"description": "Shows the connected workspace's remaining Runway credits plus available plans and top-up options as an inline card. Call it when the user asks about their credits, balance, plans, pricing, or how to buy more credits — and after a generation fails with an out-of-credits or low-credits result. After an out-of-credits failure, pass `creditsNeeded` (the approximate credit cost of that generation, from the failed tool's cost notes) to render a shortfall card. After a generation fails because the plan does not allow it, call it with `gatedCapability` set to the phrase from that error to render an upgrade card. Purchases are completed on runway.com; the card only links out. When the inline card renders, keep your reply to one short sentence — the card already shows the numbers and buttons.",
"inputSchema": {
"type": "object",
"properties": {
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
},
"homeRefresh": {
"type": "boolean",
"description": "Internal Runway home balance refresh. Omit for an inline plans card."
},
"creditsNeeded": {
"type": "number",
"description": "Approximate credit cost of the generation that just failed with out-of-credits, estimated from the failed tool's stated cost (e.g. 28 credits/second × duration). Renders the card in shortfall mode: a \"needs ~N credits — you have M\" header with top-up framing. Only pass it when a generation actually failed for credits and the cost is stated or clearly estimable."
},
"gatedCapability": {
"type": "string",
"description": "What the current plan blocked, phrased as the action the user was trying to take, copied verbatim from a paid-plan-required error (e.g. 'Generating video'). Renders the card in upgrade-gate mode: a \"<capability> requires a paid plan\" header, that phrase bolded in each plan's features, and \"Unlock\" buttons. Never pass a bare model name — most users do not know what those mean. Omit for plain balance/plans questions."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind",
"planName",
"isFreePlan",
"billingUrl",
"pricingUrl"
],
"properties": {
"kind": {
"type": "string",
"description": "Always 'plans_and_credits'."
},
"plans": {
"type": "array",
"items": {
"type": "object"
},
"description": "Self-serve plans shown as cards. A plan carries `offer: {monthlyPrice?, annualMonthlyPrice?, percentOff, durationInMonths?}` when this workspace qualifies for a discount. `durationInMonths` is only set for monthly intro coupons (first invoice); annual pricing is unchanged."
},
"topUp": {
"type": "object",
"description": "One-time credit packs: {available, minCredits, url, packs}. `url` opens the billing page with the buy-credits modal already open; only paid plans can top up."
},
"credits": {
"type": "object",
"description": "Remaining balance: {unlimited: false, total, plan, purchased} or {unlimited: true}. Omitted when it could not be resolved."
},
"planName": {
"type": "string",
"description": "Current plan display name, e.g. 'Free', 'Standard', 'Pro'."
},
"allowance": {
"type": "number",
"description": "Full-cycle credit allowance (for the usage bar)."
},
"billingUrl": {
"type": "string",
"description": "Runway billing page (change plan)."
},
"isFreePlan": {
"type": "boolean"
},
"pricingUrl": {
"type": "string",
"description": "Public pricing page with the full plan comparison."
},
"gateContext": {
"type": "object",
"description": "Present in upgrade-gate mode: {capability} for what the current plan cannot do. Any paid plan unlocks it — runwayml/api model gating is free-vs-paid only."
},
"workspaceId": {
"type": "number",
"description": "Workspace whose balance and plan are shown."
},
"shortfallContext": {
"type": "object",
"description": "Present in shortfall mode: {creditsNeeded} — the approximate cost of the generation that failed for credits."
},
"creditPackPickerVariant": {
"type": "string",
"description": "Resolved credit picker presentation. Either \"control\" or \"stacked\"."
}
},
"description": "Credit balance and plan catalog for the connected workspace, rendered by the inline plans & credits card.",
"additionalProperties": true
},
"annotations": {
"title": "Plans & Credits",
"readOnlyHint": true,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": false
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description92/100
Cancels the free Runway Pro trial so it is not charged. The workspace keeps Pro until the trial end date. Only call this when the user explicitly asks to cancel their trial.
The description explains the capability, when to use it, and what to expect.
Inputs75/100
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
rationale
Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., "user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram".
Remove validation rules already expressed in this field’s schema; describe its meaning and usage instead.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"enum": [
"pro_trial_canceled",
"pro_trial_unavailable"
],
"type": "string"
},
"endsAt": {
"type": "string",
"description": "ISO time the trial and Pro access end."
},
"reason": {
"type": "string"
}
},
"description": "`kind: 'pro_trial_canceled'` with the date Pro access ends, or `kind: 'pro_trial_unavailable'` when there is no trial to cancel.",
"additionalProperties": true
}
Original tool metadata
{
"name": "cancel_pro_trial",
"title": "Cancel Pro Trial",
"description": "Cancels the free Runway Pro trial so it is not charged. The workspace keeps Pro until the trial end date. Only call this when the user explicitly asks to cancel their trial.",
"inputSchema": {
"type": "object",
"properties": {
"rationale": {
"type": "string",
"description": "Why are you calling this tool? What is the user actually trying to accomplish? Be specific — e.g., \"user uploaded a vacation photo and asked to animate it as a 5-second clip for Instagram\"."
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"enum": [
"pro_trial_canceled",
"pro_trial_unavailable"
],
"type": "string"
},
"endsAt": {
"type": "string",
"description": "ISO time the trial and Pro access end."
},
"reason": {
"type": "string"
}
},
"description": "`kind: 'pro_trial_canceled'` with the date Pro access ends, or `kind: 'pro_trial_unavailable'` when there is no trial to cancel.",
"additionalProperties": true
},
"annotations": {
"title": "Cancel Pro Trial",
"readOnlyHint": false,
"openWorldHint": false,
"idempotentHint": true,
"destructiveHint": true
}
}
The title clearly communicates the tool’s action and subject. Keep it concise.
Description80/100
Record a diagnostic report about a Runway tool problem in this MCP server's logs.
Examples of when to call:
- A generation failed or produced unexpected results
- You couldn't figure out which tool to use
- The user said the result wasn't what they wanted
- The prompting guide didn't have the info you needed
The Runway team can inspect these logs to investigate tool problems. Be specific about what went wrong.
The description explains the capability, when to use it, and what to expect.
Inputs88/100
Input documentation gives an agent enough context to populate the arguments.
{
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"enum": [
"feedback_received"
],
"type": "string"
}
},
"description": "Acknowledgment that the feedback was recorded.",
"additionalProperties": true
}
Original tool metadata
{
"name": "feedback",
"title": "Record Tool Feedback",
"description": "Record a diagnostic report about a Runway tool problem in this MCP server's logs.\n\nExamples of when to call:\n- A generation failed or produced unexpected results\n- You couldn't figure out which tool to use\n- The user said the result wasn't what they wanted\n- The prompting guide didn't have the info you needed\n\nThe Runway team can inspect these logs to investigate tool problems. Be specific about what went wrong.",
"inputSchema": {
"type": "object",
"required": [
"summary",
"attempted",
"userIntent"
],
"properties": {
"summary": {
"type": "string",
"description": "One sentence: what went wrong?"
},
"attempted": {
"type": "string",
"description": "What did you try?"
},
"userIntent": {
"type": "string",
"description": "What was the user actually trying to accomplish?"
}
},
"additionalProperties": false
},
"outputSchema": {
"type": "object",
"required": [
"kind"
],
"properties": {
"kind": {
"enum": [
"feedback_received"
],
"type": "string"
}
},
"description": "Acknowledgment that the feedback was recorded.",
"additionalProperties": true
},
"annotations": {
"title": "Record Tool Feedback",
"readOnlyHint": false,
"openWorldHint": true,
"idempotentHint": false,
"destructiveHint": false
}
}
Loading score analytics…
Put your MCP to the test
A few tasks worth trying with this server, chosen from its reviewed tools. These have not been run.
01Runway AssetsLists recent uploaded and generated assets for the authenticated workspace.
02Search VoicesSearch the Eleven v4 voice library (thousands of voices beyond the preset list in `generate_speech`).
03List WorkspacesLists every Runway workspace the authenticated user belongs to, with role and a compact disabled-model summary per workspace.
04Account & ModelsConfirms the Runway connection is authenticated and returns the workspace this MCP connection is pinned to (chosen at sign-in) plus the list of image/video m…
05List WorkflowsLists Runway editor workflows in the authenticated workspace (name, id, updated time).
Want to run these tasks against your MCP server with us?
mcp.runwayml.com setup, tools, quality score, publishing readiness, and alternatives.
How do I add mcp.runwayml.com to Claude, ChatGPT, or Cursor?
Add mcp.runwayml.com to Claude, ChatGPT, or Cursor by connecting its remote MCP URL in each client's MCP settings. In Claude, add a custom connector. In ChatGPT, add an MCP app. In Cursor, add an MCP server under Settings. The MCP URL is https://mcp.runwayml.com/mcp.
Is mcp.runwayml.com MCP ready for ChatGPT and Claude?
As of October 14, 2026, mcp.runwayml.com scores 88/100 on Manufact's MCP publishing checklist. The scan does not yet meet ChatGPT publishing checks and meets Claude publishing checks. Passing these checks does not guarantee marketplace approval.
How clear is mcp.runwayml.com's MCP tool documentation?
As of October 14, 2026, mcp.runwayml.com scores 92/100 in Manufact's review of tool titles, descriptions, input and output schemas, and behavior annotations. The review covers 38 of 38 discovered tools. The quality score measures documentation clarity; runtime behavior and marketplace readiness require separate checks.
What are the best alternatives to mcp.runwayml.com?
As of October 14, 2026, alternatives to mcp.runwayml.com in design-creative on the Manufact MCP catalog include mcp.motion.so, Image Generation, PosterMyWall, SlidesGPT, and Mermaid Chart.
How can mcp.runwayml.com improve its MCP documentation?
import-claude-motion-from-url is the lowest-scoring reviewed tool at 76/100. No output schema is declared. Add one for structured results so clients can interpret returned fields. Open its review for feedback and a copyable fix prompt, then rerun the documentation review after updating your tool metadata.
Not your MCP?
Review any public or OAuth MCP server for protocol compatibility, tool documentation, and ChatGPT and Claude publishing readiness. No account required.
