Module 10 of 11 · Advanced · 9 min read

Prompting with images, audio and video

Get accurate answers from screenshots, photos, charts, recordings and clips, and know where multimodal models still slip.

Key takeaways

1Tell the model what the input is and what to look at.
2Ask it to describe or transcribe before it analyses, so you can catch misreadings.
3Counting, small text, precise positions and exact numbers from charts are common weak spots.
4Images and recordings often contain private information. Check before you upload.

01What multimodal input unlocks

Many current models accept images, and some accept audio, video and documents such as PDFs directly. That opens whole workflows without typing: debug from a screenshot, extract a table from a photo of a receipt, critique a design mock-up, summarise a meeting recording, or review a product video.

Generating images and video is a different skill, covered in the Creative AI path. This module is about understanding inputs.

02Prompting with images

Treat the model like a colleague looking over your shoulder: tell it what you are showing and why.

Say what the image is: 'This is a screenshot of our checkout page on mobile.'
Direct attention: 'Look at the error banner at the top and the form below it.'
Ask for a description first, then the analysis, so misreadings are visible.
For text in images, ask for an exact transcription and to mark anything unreadable rather than guessing.
With several images, label them (Image 1, Image 2) and refer to them by label.

Example

Image 1 is our current pricing page; Image 2 is a competitor's. First, describe the layout and the plan names and prices you can read on each, marking anything unreadable as [unclear]. Then list three specific differences in how they present value to a first-time visitor, and suggest one change we could test on ours.

03Charts, documents and screenshots

Models are good at explaining what a chart shows and at reading clear printed text, but reading exact values off a chart is error-prone. If precision matters, provide the underlying data or ask the model to label values as approximate. For scanned documents, compare a sample of the extracted figures with the original before trusting a full extraction.

04Audio and video

For recordings, ask for a transcript with speaker labels and timestamps first, then the summary or analysis citing timestamps. That makes claims easy to check against the source. For video, models may analyse sampled frames rather than every moment, so brief events can be missed; ask the model to say what it could and could not observe.

Example

This is a 30-minute customer interview recording. 1. Transcribe it with speaker labels (Interviewer, Customer) and a timestamp every minute. 2. List the customer's top five pain points, each with the timestamp and a short direct quote. 3. Flag anything you could not hear clearly.

05Common failure modes

When a detail matters, crop or zoom the image to that region and ask again.

Counting many small objects, and precise spatial relationships ('the third icon from the left').
Small, blurry or stylised text, which may be read as plausible words that aren't there.
Exact numbers from charts and diagrams.
Confident descriptions of details that are not in the image at all.

06Privacy

Screenshots and photos often include more than you meant to share: names, email addresses, faces, account numbers, and location data in file metadata. Crop or redact before uploading, check your provider's data settings, and get consent before uploading recordings of other people.

Watch out

Don't upload images of IDs, payment cards or other people's private documents unless you are sure how the data is stored and used.

Finished reading?

Mark it complete to track your progress, then test yourself with the Advanced Prompting quiz.

Next: Safety: prompt injection, hallucinations and verification →