MarineInst · ECCV 2024 Oral · now with 2026 VLMs

Ask the ocean
anything.

MarineChat finds every creature in your underwater photo, describes each one, and answers your questions about it. It's built for students learning marine science and for researchers who want to check and correct what the AI sees.

Start exploring I'm a researcher
0images explored
0expert checks
0images contributed
Underwater example scene analysed by MarineInst
#1
instance maskselectedMarineInst ViT-H

How it works

From one photo to a scene full of descriptions

The pipeline is two-stage, as in the MarineInst paper: instance segmentation with binary instance filtering, then instance captioning with a VLM you choose.

  1. 1 · Dive in

    Drop in your own underwater photo or pick one from the gallery. Images you don't choose to share are deleted after 24 hours.

  2. 2 · Segment

    MarineInst (SAM ViT-H fine-tuned on MarineInst20M) proposes masks and keeps the true instances, which cuts down on over- and partial segmentation.

  3. 3 · Describe & ask

    Click any creature to get its caption from MarineGPT or a 2026-generation VLM, then keep chatting about biology, behaviour and habitat.

  4. 4 · Check & improve

    Researchers can mark masks and captions as right or wrong, fix the species and add missed instances. Every correction helps train the next MarineInst.

For students

Learn the ocean, one creature at a time

  • Explanations pitched for students, with key terms highlighted
  • One-tap questions: What is it? Where does it live? What does it eat?
  • Quiz me makes a multiple-choice question from the photo itself
  • Honest about uncertainty: when the AI isn't sure, it says so
Open learn mode →
For researchers

Audit, correct and contribute

  • Inspect instance and non-instance masks, with IoU and stability scores
  • Add missed instances by clicking positive and negative points
  • Rate captions, give the correct taxon and leave notes
  • Export COCO-RLE annotations, or share them with the MarineInst team
Open research mode →

Plug-in VLMs

Pick your expert

Choose the domain-specific MarineGPT or a recent general-purpose model. All of them run on a single 24 GB GPU, and only one is loaded at a time.

Citizen science

Help us light up the deep

Marine data is scarce. Your photos from dives, field surveys or lab work, along with your corrections, go straight into improving MarineInst20M and the models trained on it.

Contribute images Data & privacy

Drop an underwater photo

or · JPG / PNG / WebP · up to 15 MB

Not shared unless you say so. Uploads are deleted after 24 h.

Talking about the whole image

Load an image, tap a creature, then ask away.

AI-generated content can be wrong, especially species IDs. Check important facts with a field guide or WoRMS.

Run segmentation to see instances.

Mark each mask or caption as ✓ or ✗ in the Instances tab (or on the chips), add missed instances, then send your review to the MarineInst team.

Credit me (optional)

Contribute

Share your underwater images

Photos from dives, ROV/BRUV surveys, aquaria or lab work all help. Metadata such as location, depth and species makes them far more useful to researchers.

Drop up to 20 imagesor click to browse

About

MarineInst & MarineChat

The model

MarineInst is a foundation model for marine image analysis with instance visual description. It outputs instance masks for marine objects and captions for each of them. It was trained on MarineInst20M, the largest marine image dataset to date, whose masks come from human annotation plus automatic binary instance filtering.

MarineChat is the public, interactive front door to MarineInst. Captions come from plug-in VLMs: the domain-specific MarineGPT or recent general models (Qwen3.5, Gemma 4, Qwen3.8).

Paper MarineInst code MarineInst20M MarineGPT

Data & privacy

  • Images you upload to Explore are kept only so the app can work, and are deleted after 24 hours, unless you tick “allow the team to keep this image”.
  • Images sent through Contribute are kept under the license you choose.
  • We store a hash of your IP address for rate-limiting, never the address itself. Name, affiliation and email are optional.
  • Feedback and corrections (✓ / ✗, comments, edited masks) are used to evaluate and improve MarineInst models.
  • The Species tab uses public resources: iNaturalist, WoRMS, Wikipedia and Wikimedia Commons, plus BioCLIP-2 for visual matching. Only openly licensed photos and clips are shown, each with its author and license. They are fetched through our server, so your browser does not contact those sites. The one exception is a YouTube clip you choose to play.

Limitations

MarineInst struggles with crowded scenes (e.g. a school of tiny fish), objects in shadow or low visibility, and self-occlusion. VLM captions can hallucinate. Please treat species IDs as suggestions, and tell us when they're wrong.

Cite

@inproceedings{ziqiang2024marineinst,
  title={MarineInst: A Foundation Model for Marine Image Analysis with Instance Visual Description},
  author={Zheng, Ziqiang and Chen, Yiwei and Zeng, Huimin and Vu, Tuan-Anh and Hua, Binh-Son and Yeung, Sai-Kit},
  booktitle={ECCV},
  year={2024}
}