How to write prompts for AI with image and file inputs (multimodal)

Multimodal prompts combine text instructions with images, PDFs, or screenshots. Learn how to direct AI attention to what matters and extract structured output reliably.

← Back to guides

What multimodal AI models can and cannot do

They can: describe image content, read text in images (OCR-like), extract data from tables and forms, analyze charts and diagrams, compare multiple images, and answer questions grounded in what they see.
They cannot: reliably read very small text in low-resolution images, count objects precisely in dense scenes, or guarantee pixel-level accuracy on complex diagrams.
Quality of output depends heavily on image resolution and clarity. Blurry or low-contrast images produce vague descriptions.
For PDFs: some APIs accept PDFs directly; others require converting pages to images first. Check your model's documentation.

How to direct AI attention in images

Name what you want analyzed: 'Focus on the bar chart in the top-right corner' is clearer than 'analyze this image'.
Describe what the image contains if the model might not recognize the context: 'This is a screenshot of a warehouse management dashboard showing inventory levels.'
For documents and forms, specify which fields matter: 'Extract only the vendor name, invoice number, date, and total amount. Ignore line items.'
Use spatial anchors: 'the table in the second column', 'the highlighted row', 'the legend at the bottom'.
If the image has multiple elements, tell the model what to prioritize first and what to skip.

Common multimodal prompt mistakes

No image description: asking 'what does this show?' without context forces the model to guess the purpose. Tell it what kind of document or image you're sending.
Expecting precision on messy inputs: a photo of a handwritten note taken at an angle in dim lighting will produce uncertain results — scan or crop before prompting.
No output format specified: without one, the model might describe the image in prose when you needed JSON, or give you a table when you needed a summary.
Assuming it reads all text: models process images as visual patterns. Very small text, rotated text, or text on complex backgrounds may be missed or misread.

Templates

Visual data extraction from an image or screenshot

I'm sending you [a screenshot of a dashboard / a photo of a form / a chart image].

Context: [describe what the image contains and where it comes from — e.g. "This is a weekly sales report screenshot from our internal reporting tool."]

Extract the following fields:
- [Field 1]: [what it looks like or where it appears]
- [Field 2]: [what it looks like or where it appears]
- [Field 3]: [what it looks like or where it appears]

Return ONLY JSON:
{
  "field_1": "value or null",
  "field_2": "value or null",
  "field_3": "value or null",
  "extraction_notes": "any uncertainty or ambiguity noted"
}

Rules:
- Use null for any field you cannot read clearly.
- Do not guess or infer values not visible in the image.
- Add a note in extraction_notes if any field was difficult to read.

Opens on home with the prompt prefilled.

Open in Promptea

Document Q&A with source citation

I'm sending you [a PDF / a set of document images / a scanned contract].

Context: [briefly describe what the document is — e.g. "This is a vendor contract for software licensing, approximately 12 pages."]

Answer the following questions based ONLY on the content visible in the document:
1. [Your question 1]
2. [Your question 2]
3. [Your question 3]

For each answer:
- Quote the exact sentence or section that supports your answer.
- If the document does not contain the answer, say "Not found in the document" — do not infer.
- If the answer is ambiguous, explain why and quote the relevant passage.

Format:
**Q1:** [restate the question]
**Answer:** [your answer]
**Source:** "[exact quote from document]" (Page X / Section Y if identifiable)

Opens on home with the prompt prefilled.

Open in Promptea

FAQ

Which AI models support image inputs?

As of mid-2026, image inputs are supported by GPT-4o and later OpenAI models, Claude 3 and later Anthropic models, Gemini 1.5 and later Google models, and Grok 2 Vision. DeepSeek and Kimi also have multimodal variants. Always verify image support in the model's current documentation, as capabilities vary by API tier and may change.

How do I extract a table from a PDF using AI?

The most reliable approach: convert the PDF page containing the table to a high-resolution image (300 DPI or higher), then send it with a prompt that specifies the table location, the columns you need, and the output format (JSON or Markdown table). Ask the model to flag cells it cannot read clearly. For native-digital PDFs where text is selectable, copy-paste the table text directly into your prompt instead — text extraction is more reliable than visual OCR.