Introduction & Locations

RarePlanes is a unique open-source machine learning dataset that incorporates both real and synthetically generated satellite imagery from Maxar and AI.Reverie. More specifically, RarePlanes focuses on the value of synthetic data to aid computer vision algorithms in their ability to automatically detect aircraft and their attributes in satellite imagery. Although other synthetic/real combination datasets exist, RarePlanes is the first dataset built to test the value of synthetic data from an overhead perspective. Previous research has shown that synthetic data can reduce the amount of real training data needed and potentially improve performance for many tasks in the computer vision domain. The dataset is expansive, and the largest openly available dataset to date focused on aircraft in satellite imagery. The real portion of the dataset consists of 253 Maxar WorldView-3 satellite images spanning 112 locations with ~14,700 hand annotated aircraft. The accompanying synthetic dataset is generated via AI.Reverie’s simulation platform and features 50,000 synthetic satellite images with over 600,000 aircraft annotations. Both the real and synthetically generated aircraft feature 10 fine grain attributes including: aircraft length, wingspan, wing-shape, wing-position, FAA wingspan class, propulsion, number of engines, number of vertical-stabilizers, if it has canards, and aircraft role.

Access from Amazon Web Services

The RarePlanes dataset is available through the AWS Open-Data Program for free download.

To download the real training and testing tiles (~3 GB) execute these commands:

aws s3 cp s3://rareplanes-public/real/tarballs/train/RarePlanes_train_geojson_aircraft_tiled.tar.gz . 
aws s3 cp s3://rareplanes-public/real/tarballs/train/RarePlanes_train_PS-RGB_tiled.tar.gz . 
aws s3 cp s3://rareplanes-public/real/tarballs/test/RarePlanes_test_geojson_aircraft_tiled.tar.gz . 
aws s3 cp s3://rareplanes-public/real/tarballs/test/RarePlanes_test_PS-RGB_tiled.tar.gz . 

All real data (~107 GB) including both tiled and untiled formats can be downloaded using this command:

aws s3 cp --recursive s3://rareplanes-public/real/tarballs/  . 

All synthetic data (~211 GB) including both tiled and untiled formats can be downloaded using this command:

aws s3 cp --recursive s3://rareplanes-public/synthetic/ . 

All model weights (~3.8 GB) used in the experiments can be downloaded here:

aws s3 cp --recursive s3://rareplanes-public/weights/ .

‍

Data Types and Structures

Imagery: .tif, .png

‍Labels: .geojson, .json

‍Metadata: .xml, .csv, .txt

‍

File Structure

RarePlanes-Public
├── LICENSE.txt
├── real
│   ├── metadata_annotations
│   │   ├── RarePlanes_Public_All_Annotations.geojson
│   │   ├── RarePlanes_Public_Metadata.csv
│   │   ├── RarePlanes_Public_Role_Counts.csv
│   │   ├── RarePlanes_Test_Coco_Annotations_tiled.json
│   │   └── RarePlanes_Train_Coco_Annotations_tiled.json
│   ├── tarballs
│   │   ├── test
│   │   └── train
│   ├── test
│   │   ├── MS
│   │   ├── PAN
│   │   ├── PS-RGB
│   │   ├── PS-RGB_cog
│   │   ├── PS-RGB_tiled
│   │   ├── geojson_aircraft
│   │   └── geojson_aircraft_tiled
│   └── train
│       ├── MS
│       ├── PAN
│       ├── PS-RGB
│       ├── PS-RGB_cog
│       ├── PS-RGB_tiled
│       ├── geojson_aircraft
│       └── geojson_aircraft_tiled
├── synthetic
│   ├── test
│   │   ├── images
│   │   ├── masks
│   │   └── xmls
│   └── train
│       ├── images
│       ├── masks
│       └── xmls
└── weights

### Real Full Location Products- Imagery and annotations cropped to the full areas annotated for each image strip.
PAN- 16-bit Panchromatic Imagery (.tif)
PS-RGB- 8-bit Pan-Sharpened RGB Imagery (.tif)
MS- 16-bit 8-channel Multi-Spectral Imagery (.tif)
PS-RGB_cog- 8-bit Pan-Sharpened RGB Imagery: Cloud Optimized GeoTiff (.tif)
geojson_aircraft- Geojson annotations containing aircraft for each unique location and image (.geojson)
RarePlanes_Public_All_Annotations.geojson- All annotations within the dataset. (.geojson)
RarePlanes_Public_Metadata.csv- Various metadata for each image and location within the dataset. (.csv) (see metadata section below)
RarePlanes_Public_Role_Counts.csv- Total number of aircraft by role (.csv)

### Real Tiled Products- pre-tiled to 512x512 and converted to PNGs for easy ingestion into common ML frameworks.
PS-RGB_tiled- 512x512 tiled Pan-Sharpened RGB Imagery in PNG format. Each tile contains at least 1 aircraft. Each image also has a corresponding XML file that contains georeferencing information for each tile. (.png & .xml).
geojson_aircraft_tiled- annotations for each 512x512 image tile.  Each tile contains at least 1 aircraft. (.geojson)
json files- COCO style annotations for 512x512 image tiles. Each tile contains at least 1 aircraft. (.json)

### Synthetic Data - Imagery and annotations
images- The synthetic images
masks - Pixel masks featuring instance segementations of each aircraft
xmls - The original AI.Reverie metadata and annotations for each image

### Weights - Pretrained weights for the experiments in the paper

‍

Real Data Naming Convention

All real data (geojson annotations and imagery) follow the same naming convention, with a corresponding annotation file for each imagery product:

'Location ID'_'CAT ID' .extension

The ‘Location ID’ is a unique integer value for each of the locations within RarePlanes. This value ranges between 1 and 130. Note that some locations have been removed with 112 unique values remaining.

The ‘CAT ID’ is a unique identifier for each Maxar WorldView 3 image strip. The dataset contains 253 unique image strips and CAT IDs.

The ‘extension’ will be one of the data types as listed above.

‍

Annotations and Attributes

The Annotation

Each aircraft is labeled in a diamond style with annotators instructed to label the nose, left-wing, tail, and right-wing in order. This annotation style has the advantage of being simplistic, easily reproducible, convertible to a bounding box, and ensures that aircraft are consistently annotated as other formats can often lead to imprecise labeling. Furthermore, this annotation style enables us to pull out two valuable features of aircraft: Their length and wingspan.

Aircraft Features and Geojson Attributes

For each annotation, individual features were labeled for each aircraft. Each feature is saved as an attribute in the geojson and coco annotations.

wing_type: (str) Wing type or shape is one of the most important features for distinguishing types of aircraft. We ultimately break wing shapes into four categories including: ‘straight’, ‘swept-back’, ‘delta’, and ‘variable-sweep’.

wing_postion: (str) We asked annotators to label if wings were ‘high mounted ’or ‘mid/low mounted’. Annotators ultimately struggled with this exercise, with a lack of consistent agreement across the dataset. This task is certainly challenging from an orbital perspective. These labels remain in the dataset, however their overall value may be partially limited.

num_engines: (int) We label the number of engines that a plane possesses. This is an integer value that ranges from 0 to 4.

propulsion: (str) We label the type of propulsion. For our dataset we label ‘propeller’ engines, ‘jet’ engines, and ‘unpowered’ aircraft.

canards: (str) These features are small forewings that are typical of a few military aircraft and even fewer civil aircraft. Canard’s are typically added to planes to increase maneuverability or reduce the load/airflow on the main wing. Canards are labeled in a binary fashion: ‘Yes’ or ‘No’.

num_tail_fins: (int) We label the number of vertical stabilizers that a plane possesses. This is an integer value that ranges from 1 to 2 in the dataset.

length: (float) As stated earlier, length estimates can be derived from the diamond annotation. We provide the length estimate in meters for each aircraft in floating point format.

wingspan: (float) We can also extract wingspan from our diamond annotation labels.We provide the wingspan estimate in meters for each aircraft in floating point format.

faa_wingspan_class: (int) Using this we can then classify aircraft by their FAA Aircraft Design Group Wingspan Class. These classes fall into six (1 to 6) categories and the wider a plane is, the longer and wider a runway it requires to land. This is important for identifying what planes could potentially land at certain airfields. We also use this to classify the size of certain aircraft (see role).

area: (float) The total area of each annotation (meters²).

• ‘Small Civil Transport/Utility’– Civil aircraft that fall into FAA Wingspan Class 1 (< 15m). role_id = 1
• ‘Medium Civil Transport/Utility’– Civil aircraft that fall into FAA Wingspan Classes 2 to 3 (< 36m but ≥15m). role_id = 2
• ‘Large Civil Transport/Utility’– Civil aircraft that fall into FAA Wingspan Classes 4 to 6 (≥ 36m). role_id = 3
• ‘Military Transport/Utility/AWAC’– Military aircraft used for non-combat operations. This includes transport detail or electronics warfare. role_id = 4
• ‘Military Bomber’– Military aircraft that are typically larger than their Military Fighter counterparts. These aircraft are capable of carrying bombs, torpedos, or cruise missiles. role_id = 5
• ‘Military Fighter/Interceptor/Attack’– Military aircraft that are focused on air-to-air or air-to-ground combat. role_id = 6
• ‘Military Trainer’– Military aircraft that are typically smaller propeller planes used to train newer pilots. role_id =7

See the Role_Counts.csv for types and numbers of each contained within the dataset.

Other Geojson Attributes

loc_id: (int) The unique identifier for each RarePlanes airfield location.

cat_id: (str) The unique image strip identifier for each WorldView-3 image.

location: (str) The string text of the geolocation automatically derived from geopy geocoders. Further georeferencing location information can be found within the metadata.csv.

Public_Train: (int) A binary integer value of 1 or 0. A value of 1 indicates the annotation is within the default training dataset. A value of 0 indicates that the annotation is not within the train set and is in the test set.

Public_Test: (int) A binary integer value of 1 or 0. A value of 1 indicates the annotation is within the default testing dataset. A value of 0 indicates that the annotation is not within the test set and is in the train set.

partialDec: (float) (Tiled Data Only) A floating point value between 0 and 1 indicating the proportion of a plane that was cropped. The lower the value, the more of the plane has been cropped. End users may want to use this to remove annotations that now display only a small percentage of an aircraft. We recommend removing any annotations with a value <0.25.

truncated: (int) (Tiled Data Only) A binary integer value of 1 or 0. A value of 1 indicates the annotation was cropped during the tiling process.

Real Imagery

By the numbers:

• 253 images
• 112 locations in 22 countries
• 2,142 km²
• 3.2 to 29.6° off-nadir

Training and Testing Splits

We provide an example training and testing split. End-users certainly can use this split but are encouraged to reorganize and create their own splits for their specific research or use-case(s).

How we split the data:

A “location” is the physical spot on earth, an “observation” is a single satellite image at any given location. Often, there are multiple observations per location. Approximately 75% of the dataset is included in the example training dataset with the remaining 25% for testing.

1.We first stratify airfields by country, a minimum of 1 image goes in train and 1 in test. If that is not possible for countries with a single location and single observation, that image goes into training.

2. For countries with 2 or more locations, a minimum of 1 full location (all observations) is held back for testing. For countries with >= 16 locations then up to a total of 12.5% of all locations (again including all observations) in that country are randomly sampled to be held back for testing.

3. Finally, of the data not yet included in the training and testing set, a final 12.5% of the observations are randomly selected to be included in the test set.

4. A few images are manually switched between the training and testing set to ensure that certain roles of aircraft are evenly represented.

Real Imagery Metadata

The RarePlanes_Public_Metadata.csv file contains other relevant metadata.

image_id: (str) The base naming convention for each image in the dataset (‘loc_id’ + ‘_’ + ‘cat_id’)

loc_id: (int) The unique identifier for each RarePlanes airfield location.

cat_id: (str) The unique image strip identifier for each WorldView-3 image.

Train_Set: (int) A binary integer value of 1 or 0. A value of 1 indicates the annotation is within the default training dataset. A value of 0 indicates that the annotation is not within the train set and is in the test set.

Test_Set: (int) A binary integer value of 1 or 0. A value of 1 indicates the annotation is within the default testing dataset. A value of 0 indicates that the annotation is not within the test set and is in the train set.

Country: (str) The country of the airfield in question.

State_Province: (str) The state, province, or other administrative district for each air field.

Air_Field: (str) The air field or airport name.

Weather: (str) The observed weather. This includes: ‘Clear Skies’, ‘Snow’, or ‘Cloud Cover or Haze’.

sensor: (str) The sensor, in this case all data is from WorldView 3, shortened to ‘WV03’.

collection_time_start: (datetime) The start time and date for imagery collection.

collection_time_end: (datetime)The end time and date for imagery collection.

cloud_cover_percentage: (int) The estimated percentage of cloud cover for the entire image strip. Note that this is for a much larger area beyond the extent of each airfield. These numbers will not be accurate for individual airfields.

off_nadir_max: (float) The maximum off-nadir angle for the full image strip.

target_azimuth_maximum: (float) The maximum target azimuth angle for the full image strip.

pan_resolution_minimum: (float) The minimum pan-sharpened resolution for the full image strip.

pan_resolution_maximum: (float) The maximum pan-sharpened resolution for the full image strip.

scan_direction: (str) The direction in which the sensor operated for collection. ‘Forward’ or ‘Reverse’.

avg_sun_elevation_angle: (float) The average sun elevation angle during collection.

Imagery Products

RarePlanes contains three types of imagery products. All imagery is derived from the Maxar WorldView-3 sensor. Note that no SWIR imagery is included. All data are ortho-rectified and atmospherically corrected to surface reflectance units (surface reflectance *10,000). PS-RGB imagery has undergone dynamic range adjustment (DRA) for conversion to 8-bit imagery.

PAN– 16-bit Panchromatic Imagery (0.31m best spatial resolution)
PS-RGB– 8-bit Pan-Sharpened RGB Imagery (0.31m best spatial resolution)
MS– 16-bit 8-channel VNIR Multi-Spectral Imagery (1.24m best spatial resolution)

Coordinate Reference System (CRS)

CRS: WGS 84 (EPSG:4326)— World Geodetic System 1984

Projection: None

Units: Decimal Degrees (Latitude/Longitude)

Time Series Component

Note that many locations have multiple images. When working with the dataset ensure you use both the loc_id and the image_id to select the appropriate annotations for the corresponding image.

Weather

Weather is visually interpreted for each image.

‘Clear Skies’- No observable clouds or haze over the airfield.

‘Snow’- At least ≥50% of the surface of the airfield is covered in snow.

‘Cloud Cover or Haze’- Any image that contains observable cloud cover or haze.

Synthetic Imagery

All synthetic data is created via the AI.Reverie simulator software. The synthetic datasets 50,000 images across 15 distinct locations, simulating a total area of 9331.2 km^2 containing 629,551 annotations of aircraft. The locations were selected based on available OpenStreetMap data. Each image features a simulated GSD of 0.3 meters and is collected from variable look-angles ranging between 5.0 to 30.0 degrees off-nadir. The imagery is evenly split across 5 distinct biomes including: `Alpine’, `Arctic’, `Temperate Evergreen Forests’, `Grasslands’, and `Tundra’. The biome parameter controls the type of vegetation, its density as well as the ground textures. Four unique weather conditions are also evenly distributed across the dataset including: `Overcast’, `Clear Sky’, `Snow’, and `Rain’. Other parameters includes the sunlight intensity, the time of the day or the off-nadir angle. Ultimately, this produces an expansive heterogeneous dataset with a wide variety of backgrounds. We believe that this dataset will be helpful in improving model generalizability to new areas and developing new algorithmic approaches that could move beyond aircraft detection.

References

[1] All aircraft sketches are attributable to Wikipedia user Steelpillow: https://commons.wikimedia.org/wiki/User:Steelpillow/Aircraft

‍