Object Detection in Satellite Imagery Using Deep Learning: A Practical Guide

A comprehensive guide for GIS professionals, remote sensing analysts, and ML engineers working at the intersection of deep learning and geospatial data.

Introduction

Satellite imagery has undergone a quiet revolution. What was once the exclusive domain of government agencies and large defense contractors is now commercially available at sub-meter resolution, refreshed daily, and accessible through straightforward APIs. At the same time, deep learning has transformed computer vision to a degree that would have seemed implausible a decade ago.

The convergence of these two trends has opened up a new class of geospatial intelligence applications: automatically detecting, counting, and monitoring objects across vast swaths of the Earth’s surface — from ships in a harbor to solar panels on rooftops, from informal settlements to illegal mining sites.

This guide is a practical walkthrough of how object detection in satellite imagery works, what makes it hard, and how to build systems that actually perform well in production.

Why Satellite Imagery Is Different

Before reaching for a standard computer vision pipeline, it is worth understanding how satellite imagery diverges from conventional RGB images. Treating satellite data like a photo from a smartphone camera is one of the most common and costly mistakes in this domain.

Scale and Resolution Heterogeneity

Objects in satellite imagery span wildly different spatial extents depending on the sensor and orbit. A vehicle detected in 30 cm resolution imagery from a commercial constellation (e.g., Maxar WorldView-3) might occupy 5–10 pixels. The same vehicle in Sentinel-2 imagery at 10 m/pixel is invisible. Your model must be explicitly designed for the resolution regime of your target data.

Spectral Bands Beyond RGB

Most satellite sensors capture more than three bands:

Multispectral: 4–8 bands, including Near-Infrared (NIR), Red Edge, and Short-Wave Infrared (SWIR)
Hyperspectral: Hundreds of narrow spectral bands
SAR (Synthetic Aperture Radar): Active microwave sensing; penetrates clouds and works at night

NIR alone dramatically improves vegetation discrimination and can help distinguish natural from man-made surfaces. SWIR is valuable for geological mapping and fire detection. If your pipeline discards non-RGB bands, it is leaving significant signal on the table.

Top-Down Viewing Angle (Nadir and Off-Nadir)

Unlike street-level photography, satellite imagery is captured from above. Objects appear as footprints, not profiles. A car looks like a rectangle; a building looks like a rectangle with a potential shadow offset. Off-nadir imagery introduces geometric distortions and shadow patterns that shift with look angle — something that causes models trained on nadir images to fail when tested on off-nadir collections.

Arbitrary Orientation

Objects on the ground have no preferred orientation relative to the image frame. A vessel at sea can point in any direction. This means standard axis-aligned bounding box detectors are often suboptimal; oriented bounding box (OBB) detectors are frequently more appropriate.

Class Imbalance and Small Object Density

Positive examples (e.g., aircraft on an airfield) are rare relative to background tiles (ocean, forest, urban fabric). A naive training split will produce a model that learns to predict “background” everywhere and still achieves high accuracy. Proper sampling strategies are essential.

Core Deep Learning Architectures for Object Detection

The object detection landscape has matured considerably. The following architectures are the most practically relevant for satellite imagery today.

Two-Stage Detectors

Two-stage detectors, pioneered by the R-CNN family, first generate region proposals and then classify each proposal.

Faster R-CNN remains a strong baseline. Its Region Proposal Network (RPN) generates candidate boxes, which are then refined and classified. On satellite imagery, it performs well when objects are reasonably large and class boundaries are clear.

Mask R-CNN extends Faster R-CNN with an instance segmentation head, useful when you need object outlines rather than just bounding boxes — for example, delineating individual building footprints.

Two-stage detectors tend to offer higher precision but are slower at inference, which matters when processing imagery at continental scale.

Single-Stage Detectors

Single-stage detectors trade some precision for speed by directly predicting bounding boxes and class scores in one forward pass.

YOLO (You Only Look Once) — in its v5, v8, and v9 iterations — has become a workhorse in satellite object detection due to its speed and reasonable accuracy. YOLOv8 in particular offers a mature ecosystem with built-in support for oriented bounding boxes (OBB), making it a natural fit for ship, aircraft, and vehicle detection tasks.

RetinaNet introduced Focal Loss to address the class imbalance problem endemic to dense detection. It remains relevant when false positive rates from background confusion are a primary concern.

Feature Pyramid Networks (FPN)

FPNs are not standalone detectors but a widely adopted backbone enhancement. They construct a multi-scale feature hierarchy, enabling a single model to detect both small and large objects effectively. Nearly every state-of-the-art satellite detection pipeline uses some form of FPN.

Oriented Object Detection

For objects whose orientation matters — aircraft, ships, vehicles in parking lots — standard horizontal bounding box detectors lose precision and introduce spurious overlaps.

Key architectures for oriented detection include:

ReDet: Uses rotation-equivariant networks to make feature extraction orientation-aware
Oriented R-CNN: Adds an oriented RPN and RoI head to the Faster R-CNN framework
S2A-Net: Alignment-based single-stage detector tuned for aerial imagery

Benchmarking on DOTA (see below) is the standard way to compare these approaches.

Transformers and Attention-Based Approaches

DETR (DEtection TRansformer) replaces the traditional RPN + NMS pipeline with a set-prediction formulation using transformer encoders and decoders. While computationally heavy, it eliminates post-processing heuristics and has spawned a family of improved variants (Deformable DETR, DINO).

SAM (Segment Anything Model) and its geospatial adaptations (e.g., GeoSAM) show promise as promptable segmentation backends that can be wrapped with a detection head for satellite use cases.

Key Datasets for Training and Benchmarking

No model is better than the data it was trained on. The following are the most widely used public datasets in satellite object detection.

DOTA (Dataset for Object Detection in Aerial Images)

The most comprehensive benchmark for aerial object detection. DOTA v2.0 contains over 1.8 million object instances across 18 categories — including large vehicles, small vehicles, ships, aircraft, harbors, and storage tanks — annotated with oriented bounding boxes. Resolution ranges from 0.25 m to 3 m. DOTA is the de facto standard for comparing detection architectures.

DIOR (Dataset for Object Detection in Optical Remote Sensing Images)

23 object categories, roughly 190,000 instances across 23,463 images. A widely used benchmark for horizontal bounding box detection on medium-resolution optical imagery.

xView

Commercial satellite imagery from DigitalGlobe (now Maxar) at 0.3 m resolution. Contains 60 object classes and over 1 million labeled instances. Particularly useful for fine-grained vehicle and infrastructure detection.

HRSC2016

Focused on ship detection in high-resolution optical imagery, with oriented bounding box annotations. A common benchmark for maritime surveillance models.

SpaceNet

A series of challenges (SpaceNet 1–8) covering building footprint extraction, road network extraction, and disaster response mapping. Built on commercial satellite imagery from Maxar and Planet.

Inria Aerial Image Labeling Dataset

Building segmentation dataset with 0.3 m resolution imagery over cities in Europe and North America. Widely used for urban mapping and building footprint extraction.

Practical Note on Data Acquisition

Public datasets rarely match your target domain exactly. When building production systems, plan for:

Domain adaptation: Models trained on one region or sensor often degrade on another
Custom annotation: For niche object classes, you will need to annotate your own data
Semi-supervised and weakly supervised techniques: To reduce labeling cost

Data Preprocessing Pipeline

Raw satellite imagery requires substantial preprocessing before it can enter a detection pipeline. Skipping or inadequately handling these steps is a leading cause of poor model performance.

1. Orthorectification and Geometric Correction

Raw satellite scenes are distorted by terrain relief and sensor geometry. Orthorectification corrects these distortions using a Digital Elevation Model (DEM), producing a georeferenced image aligned to a standard map projection. Most analysis-ready data products (e.g., Sentinel-2 L2A, Planet Surface Reflectance) include this step, but custom acquisitions may not.

2. Atmospheric Correction

Atmospheric scattering and absorption alter the spectral response recorded at the sensor. Surface reflectance products apply radiometric corrections to yield values representative of the actual surface. For change detection and cross-date comparisons, using surface reflectance rather than top-of-atmosphere radiance is critical.

3. Cloud and Shadow Masking

Cloud cover is the primary data quality issue in optical satellite analysis. Incorporate cloud masks (from QA bands or purpose-built models like s2cloudless) into your pipeline to exclude affected pixels before training and inference.

4. Tiling

Detection models operate on fixed-size input patches (commonly 512×512 or 1024×1024 pixels). Large satellite scenes must be tiled, with sufficient overlap between tiles (typically 10–20%) to avoid missing objects that straddle tile boundaries. During inference, predictions from overlapping tiles are merged using NMS.

# Pseudocode: tiling with overlap
def tile_image(image, tile_size=1024, overlap=0.2):
    stride = int(tile_size * (1 - overlap))
    tiles = []
    for y in range(0, image.height - tile_size + 1, stride):
        for x in range(0, image.width - tile_size + 1, stride):
            tile = image.crop(x, y, x + tile_size, y + tile_size)
            tiles.append((tile, x, y))
    return tiles

5. Normalization

Satellite imagery normalization is not as straightforward as subtracting ImageNet mean/std. Per-band statistics vary significantly by sensor, date, and region. Common approaches include:

Per-image percentile stretching (e.g., 2nd–98th percentile normalization)
Dataset-level statistics computed on your specific training corpus
Sensor-specific norms published for foundation models (e.g., SatMAE, Scale-MAE)

6. Data Augmentation

Standard augmentations (random crop, horizontal/vertical flip, color jitter) apply. Additionally, for satellite imagery:

Rotation augmentation is especially important given arbitrary object orientations
Multi-scale training helps with objects that vary in size across resolutions
Copy-paste augmentation (pasting object instances onto new backgrounds) is highly effective for rare classes

Training Strategies and Best Practices

Transfer Learning from ImageNet vs. Satellite-Specific Pretraining

ImageNet-pretrained weights remain a useful starting point, particularly for RGB-only pipelines. However, satellite-specific pretrained backbones increasingly outperform ImageNet initialization on remote sensing tasks.

Notable pretrained models:

SatMAE: Masked autoencoder pretrained on Sentinel-2 multispectral data
Scale-MAE: Scale-aware MAE pretraining for multi-resolution satellite imagery
RemoteCLIP: CLIP-style vision-language pretraining on satellite image-caption pairs
Prithvi (IBM/NASA): Foundation model pretrained on Harmonized Landsat Sentinel-2 data

When your target task involves multispectral bands (beyond RGB), these specialized backbones are strongly preferred over ImageNet weights.

Handling Class Imbalance

Several techniques help address the positive/negative imbalance in satellite detection:

Focal Loss: Down-weights easy negatives, focusing training on hard examples
Oversampling: Sample training tiles to ensure all tiles contain at least one positive instance
Hard Negative Mining: Identify background tiles that confuse the model and include them in training batches
Balanced batch sampling: Enforce class frequency constraints within each mini-batch

Sliding Window Inference with NMS

At inference time, tile the input image, run detection on each tile, project predictions back to image coordinates, and apply NMS to suppress duplicate detections across overlapping tiles.

# Pseudocode: merge predictions across tiles
def merge_predictions(tile_predictions, tile_offsets, iou_threshold=0.5):
    all_boxes = []
    for preds, (offset_x, offset_y) in zip(tile_predictions, tile_offsets):
        for box in preds:
            # Translate from tile coordinates to image coordinates
            global_box = translate_box(box, offset_x, offset_y)
            all_boxes.append(global_box)
    return non_max_suppression(all_boxes, iou_threshold)

Multi-Resolution and Multi-Scale Models

If your deployment must handle imagery at variable resolutions, consider:

Training on a mix of resolutions with resolution-tagged conditioning
Multi-scale inference (running detection at multiple zoom levels and merging results)
Models with resolution-invariant backbones (Scale-MAE, GFM)

Evaluation Metrics

Mean Average Precision (mAP)

The standard metric for detection benchmarks. AP is computed as the area under the precision-recall curve for a single class, at a given IoU threshold. mAP averages AP across all classes.

mAP@0.5: AP at 50% IoU overlap — the traditional PASCAL VOC metric
mAP@0.5:0.95: Average mAP across IoU thresholds from 0.5 to 0.95, used in COCO — a stricter measure of localization quality

Oriented Bounding Box IoU

For oriented detection, IoU must be computed between rotated rectangles. This requires polygon intersection routines (available in libraries such as shapely or via CUDA-accelerated implementations in mmrotate).

Additional Metrics for Geospatial Tasks

F1 score: Harmonic mean of precision and recall; useful when mAP is unintuitive for operators
Counting accuracy: For applications like vehicle or ship counting, absolute count error per scene
Geolocation error: Mean distance (in meters) between predicted and ground-truth object centroids
Change detection accuracy: For monitoring applications, the percentage of correctly detected change events

Real-World Applications

Maritime Domain Awareness

Detecting and tracking vessels in SAR and optical imagery for illegal fishing identification, sanctions compliance monitoring, and port traffic analysis. Companies like Spire, Windward, and Orbital Insight have built commercial products in this space.

Aviation and Defense

Aircraft detection and type classification on airfields, supporting fleet assessment and logistics monitoring. Oriented bounding box detection is essential here, as aircraft taxiing directions vary.

Infrastructure Monitoring

Solar panel mapping: Estimating renewable energy capacity at national scale
Oil and gas infrastructure: Detecting storage tanks, flare stacks, and well pads
Road and bridge change detection: Identifying new construction or damage after disasters

Agricultural Intelligence

Crop type mapping, farm boundary delineation, greenhouse detection, and irrigation infrastructure monitoring. Multispectral and SAR data are particularly valuable here.

Disaster Response and Humanitarian Applications

Mapping building damage after earthquakes, floods, or conflicts. The xBD dataset (from the xView2 challenge) and the Copernicus Emergency Management Service have driven significant progress in rapid damage assessment.

Urban Analytics

Counting cars in parking lots as a proxy for retail foot traffic, monitoring informal settlement growth, and tracking construction activity in rapidly urbanizing cities.

Tools, Libraries, and Frameworks

Deep Learning Frameworks

Library	Strengths
PyTorch	Dominant research framework; most detection codebases are PyTorch-based
TensorFlow / Keras	Strong for production deployment, particularly with TFLite and TF Serving

Detection Frameworks

Library	Use Case
mmdetection (OpenMMLab)	Comprehensive detection framework; extensive model zoo
mmrotate	Oriented object detection built on mmdetection
Ultralytics YOLOv8/v9	Fast iteration on YOLO family models; OBB support built-in
Detectron2 (Meta)	Mature two-stage detector framework with strong documentation

Geospatial Data Handling

Library	Use Case
GDAL / rasterio	Reading, writing, and transforming raster data
shapely	Geometric operations on vector features
geopandas	Spatial data analysis in a pandas-like interface
pyproj	Coordinate reference system transformations
torchgeo	PyTorch datasets and transforms for remote sensing data

Annotation Tools

QGIS: Full-featured GIS with raster annotation plugins
Label Studio: Flexible open-source annotation tool with bounding box and polygon support
Roboflow: Managed annotation and dataset versioning for object detection
CVAT: Open-source computer vision annotation tool with satellite imagery support

Challenges and Limitations

Domain Shift Across Sensors and Dates

A model trained on Maxar WorldView imagery will often perform poorly on PlanetScope data, even if both cover similar geographies. Sensor-to-sensor differences in resolution, spectral response, and viewing geometry create distribution shifts that can devastate model performance. Mitigation strategies include domain adaptation, fine-tuning on target domain data, and sensor-agnostic preprocessing.

Seasonal and Phenological Variation

Vegetation patterns change dramatically across seasons. A building detector trained only on summer imagery may fail when leaf cover is absent in winter, revealing previously hidden structures and altering the textural context around buildings.

Very Small Objects

Detecting objects that occupy only a handful of pixels is fundamentally hard. Super-resolution preprocessing or high-resolution input patches can help, but there are physical limits imposed by sensor resolution.

Lack of Labeled Data for Rare Classes

For unusual object classes — specific military equipment, rare industrial infrastructure, particular vessel types — the volume of labeled examples needed to train a robust detector may not be practically achievable. Few-shot and zero-shot detection methods are an active research area addressing this gap.

Computational Cost at Scale

Global-scale analysis at high resolution involves processing petabytes of imagery. Efficient tiling strategies, model quantization, hardware-aware deployment (GPU clusters, edge devices), and intelligent prioritization (only processing areas of interest) are all necessary for production deployments.

Ethical and Legal Considerations

Satellite-based object detection raises significant concerns around privacy, surveillance, and dual-use. Models that enable mass monitoring of individuals, tracking of refugees, or automated targeting require careful governance frameworks. Practitioners should be explicit about intended use cases and implement access controls accordingly.

The Road Ahead: GeoAI and Foundation Models

The next wave of progress in satellite object detection is being driven by geospatial foundation models — large-scale pretrained models that can be fine-tuned efficiently for diverse downstream tasks.

Vision-Language Models for Geospatial Queries: Models like RemoteCLIP and GeoChat enable text-driven object retrieval from satellite imagery, moving toward natural language interfaces for geographic analysis.

Segment Anything in Geospatial Contexts: GeoSAM and similar adaptations of Meta’s SAM bring interactive, promptable segmentation to satellite imagery — valuable for labeling efficiency and zero-shot generalization.

Temporal Modeling: Foundation models that incorporate the time dimension, allowing detection and change monitoring across image time series rather than single snapshots, are an emerging frontier.

Multimodal Fusion: Combining optical, SAR, and elevation data within unified architectures remains an open problem with significant practical payoff.

The trajectory is clear: object detection in satellite imagery is moving from task-specific, laboriously trained models toward general-purpose geospatial AI systems capable of answering open-ended questions about the Earth’s surface with minimal task-specific fine-tuning.

Conclusion

Object detection in satellite imagery is no longer a research curiosity. It is a production capability powering maritime surveillance, agricultural monitoring, disaster response, and urban analytics at global scale.

The practitioner’s path runs through several mandatory checkpoints: understanding what makes satellite data geometrically and spectrally distinct, selecting architectures matched to the object class and resolution regime, building robust preprocessing pipelines, training with appropriate strategies for class imbalance and scale variation, and evaluating with geospatially meaningful metrics.

The field is moving quickly. Geospatial foundation models are compressing the training data requirements that once made entry costly. Cloud-based platforms are democratizing access to both imagery and compute. And the range of addressable problems — from counting trees to monitoring international compliance — continues to expand.

For practitioners who invest the time to understand the domain-specific challenges, satellite object detection offers one of the most impactful applications of deep learning available today.

Table of Contents