Generate anchor boxes

Methods to generate anchor boxes of different aspect ratios.

---

source

MatchResult

def MatchResult(
    box_ids:tuple, anchor_indices:dict, ious:dict, masks:dict, matched_boxes:dict
)->None:

Structured output from ground-truth/anchor matching.

box_ids preserves the identity and input order of every ground-truth box. The remaining mappings use those same IDs as keys. Tuple unpacking and integer indexing expose (matched_boxes, ious, masks) for compatibility.

To generate anchor boxes, we need three basic information:

• Input image size, image_sz: To position our anchor boxes within the maximum coordinates (width, height) of the image.
• Feature map size, feature_sz: Feature map is the size (width, height) of the output of a convolutional operation. A \(10\times10\) feature map would mean \(10\times10\) local receptive field locations can be traced back into the input image. These 100 receptive field locations (\(10\times10=100\)) in the input image would act as our initial anchor box candidates.

• Aspect ratio of anchor boxes, asp_ratio: To generate anchor boxes with different width to height ratio (default asp_ratio=1).

---

source

bx

def bx(
    image_sz:(<class 'int'>, <class 'tuple'>), # image size (width, height)
    feature_sz:(<class 'int'>, <class 'tuple'>), # feature map size (width, height)
    asp_ratio:float=None, # aspect ratio (width:height), by default None
    clip:bool=True, # whether to apply np.clip, by default True
    named:bool=True, # whether to return (coords, labels), by default True
    anchor_sfx:str='a', # suffix anchor label with anchor_sfx, by default "a"
    min_visibility:float=0.25, # minimum visibility dictates the condition for a box to be considered
    # valid. The value corresponds to the ratio of expected area of an anchor box
    # to the calculated area after clipping to image dimensions., by default 0.25
)->ArrayLike: # anchor box coordinates in `pascal_voc` format
if named=True, a list of anchor box labels are also returned.

Calculate anchor box coords given an image size and feature size for a single aspect ratio.

coords_1, labels_1 = bx(
    100,
    10,
    0.5,
)

coords_1

(#100) [[1, 0, 8, 12],[11, 0, 18, 12],[21, 0, 28, 12],[31, 0, 38, 12],[41, 0, 48, 12],[51, 0, 58, 12],[61, 0, 68, 12],[71, 0, 78, 12],[81, 0, 88, 12],[91, 0, 98, 12]...]

Usually multiple anchor boxes with different feature_sz and asp_ratio are needed. This requirement arises in the case of multiscale object detection.

For multiscale object detection, feature maps from different convolution operations of the network are used to trace back into the input image, to generate anchor boxes. The bxs method of pybx provides this possibility.

---

source

bxs

def bxs(
    image_sz:(<class 'int'>, <class 'tuple'>), # image size (width, height)
    feature_szs:list=None, # list of feature map sizes, each feature map size being an int or tuple, by default [(8, 8), (2, 2)]
    asp_ratios:list=None, # list of aspect ratios for anchor boxes, each aspect ratio being a float calculated by (width:height), by default [1 / 2.0, 1.0, 2.0]
    named:bool=True, # whether to return (coords, labels), by default True
    **kwargs
)->ArrayLike: # anchor box coordinates in pascal_voc format
if named=True, a list of anchor box labels are also returned.

Calculate anchor box coords given an image size and multiple feature sizes for mutiple aspect ratios.

coords, labels = bxs(100, [10, 8, 5, 2], [1, 0.5, 0.3])

coords.shape, len(labels)

((587, 4), 587)

All methods work with asymetric image_sz (and or feature_szs as well):

coords, labels = bxs((100, 200), [10, 8, 5, 2], [1, 0.5, 0.3])

coords.shape, len(labels)

((336, 4), 336)

Ground truth anchor boxes

Ground truth boxes are anchor boxes with maximum IOU with the true annotations. Matching functions return a MatchResult, which keeps box IDs, selected anchor indices, IoUs, masks, and matched boxes together. Supply stable application IDs such as UUIDs when identity must survive input reordering; otherwise IDs default to zero-based input positions.

For compatibility, a MatchResult can still be unpacked as (matched_boxes, ious, masks).

Load actual annotations.

true_annots = json.load(open("../data/annots.json"))
true_annots

[{'x_min': 130, 'y_min': 63, 'x_max': 225, 'y_max': 180, 'label': 'clock'},
 {'x_min': 13, 'y_min': 158, 'x_max': 90, 'y_max': 213, 'label': 'frame'}]

Convert to MultiBx for convenience:

true_annots_as_bx = get_bx(true_annots)
true_annots_as_bx

MultiBx(coords: 2, labels: 2)

Generate anchor boxes for object detection task, given that we know:

image_sz = (256, 256)  # to know the upper bounds of candidate bounding boxes
feature_sz = [20, 10, 8, 3]
asp_ratio = [1, 0.5, 0.3]

coords, labels = bxs((256, 256), [20, 10, 8, 3], [1, 0.5, 0.3])

coords.shape, len(labels)

((1741, 4), 1741)

n_boxes = coords.shape[0]
n_boxes

1741

Store coords as multibx for convenience.

coords_as_bx = get_bx(coords=coords, label=labels)

Can use the true annotations and anchor boxes to calulate the IOU.

true_annots_as_bx

MultiBx(coords: 2, labels: 2)

len(true_annots_as_bx), len(coords_as_bx)

(2, 1741)

The question: for each true label in the provided true annotations, what are the possible ground truth anchor boxes?

for annots in true_annots_as_bx:
    print(annots)

BaseBx(coords=[[130, 63, 225, 180]], label=['clock'])
BaseBx(coords=[[13, 158, 90, 213]], label=['frame'])

gt_anchors_per_class = defaultdict(lambda: L())
iou_per_box = defaultdict(lambda: L())
for annots in true_annots_as_bx:
    label = annots.label[0]  # is a list of len 1
    ious = [annots.iou(coords_as_bx[i]) for i in range(n_boxes)]
    iou_per_box[label].extend(ious)

iou_per_box.keys()

dict_keys(['clock', 'frame'])

max(iou_per_box["clock"])

0.3482

iou_per_box["clock"].argwhere(gt(0.3))  # indices of boxes with iou > 0.3

(#2) [1719,1728]

iou_per_box["frame"].argwhere(gt(0.3))  # indices of boxes with iou > 0.3

(#3) [1720,1729,1738]

(
    (max(iou_per_box["clock"])),
    max(iou_per_box["frame"]),
)  # add more anchor boxes to get better ground truth IOUs

(0.3482, 0.4488)

Improve the loop to return only those with good IOU:

iou_thresh = 0.3
gt_anchors_per_class = defaultdict(lambda: L())
iou_per_box = defaultdict(lambda: L())
for annots in true_annots_as_bx:
    label = annots.label[0]  # is a list of len 1
    ious = L([annots.iou(coords_as_bx[i]) for i in range(n_boxes)])
    ious_filter = ious.argwhere(gt(iou_thresh))
    # report filtered box IOUs
    iou_per_box[label].extend(ious[ious_filter])
    # report selected boxes
    gt_anchors_per_class[label] = stack_bxs_inplace(
        *[coords_as_bx[i] for i in ious_filter]
    )

iou_per_box["clock"], iou_per_box["frame"]

((#2) [0.34,0.3482], (#3) [0.3664,0.4488,0.3523])

gt_anchors_per_class["clock"]

MultiBx(coords: 2, labels: 2)

Not very far from the original annotations. These scales and aspect ratios can also be read from the label. The finer the anchor boxes, the better the starting positions (ground truth anchor boxes).

true_annots[0]

{'x_min': 130, 'y_min': 63, 'x_max': 225, 'y_max': 180, 'label': 'clock'}

tmp = L([1, 2, 3])
msk = tmp.map(lambda x: x > 1)

tmp[msk]

(#2) [2,3]

mask2idxs(msk)

[1, 2]

---

source

get_gt_thresh_iou

def get_gt_thresh_iou(
    true_annots, anchor_boxes, anchor_labels:NoneType=None, iou_thresh:float=0.3, return_ious:bool=False,
    return_masks:bool=False, update_labels:bool=True, box_ids:NoneType=None
):

Find positive anchors for each ground-truth box using an IoU threshold.

Can result in an uneven number of positive anchors per ground-truth box. Matching is independent for each ground-truth box, so an anchor may match more than one object. IoUs are calculated as one vectorized matrix and returned at full floating-point precision.

Args: true_annots (Any): True annotations, typically in pascal_voc format anchor_boxes (Any): Candidate anchor boxes, typically calculated with pybx.bxs anchor_labels (List, optional): Anchor box labels, will be overwritten with true labels if update_labels=True. Defaults to None. iou_thresh (float, optional): IOU threshold to filter out negative ground truth anchor boxes. Defaults to 0.3. return_ious (bool, optional): Return IOU values for selected positive ground truth anchor boxes. Defaults to False. return_masks (bool, optional): Return boolean masks for all anchor boxes indicating if a box is positive (True) or negative (False) ground truth box. Defaults to False. update_labels (bool, optional): Overwrite with true annotations. Defaults to True. box_ids (Iterable, optional): Unique identifiers for the ground-truth boxes. Defaults to their zero-based input positions.

Returns: MatchResult: Box IDs, selected anchor indices, optional IoUs and masks, and matched boxes. Every mapping is keyed by the IDs in box_ids.

coords

array([[  0,   0,  12,  12],
       [ 12,   0,  25,  12],
       [ 25,   0,  38,  12],
       ...,
       [ 19, 135,  66, 256],
       [104, 135, 151, 256],
       [189, 135, 236, 256]])

gt_anchors_per_box, ious_per_box, mask_per_box = get_gt_thresh_iou(
    true_annots, coords, iou_thresh=0.3, return_ious=True
)
gt_anchors_per_box

{0: MultiBx(coords: 2, labels: 2),
 1: MultiBx(coords: 3, labels: 3)}

mask_per_box

{}

true_annots

[{'x_min': 130, 'y_min': 63, 'x_max': 225, 'y_max': 180, 'label': 'clock'},
 {'x_min': 13, 'y_min': 158, 'x_max': 90, 'y_max': 213, 'label': 'frame'}]

ious_per_box[0]

(#2) [0.34,0.3482]

gt_anchors_per_box[0].coords

[[170, 85, 256, 170], [183, 67, 243, 188]]

If anchor box labels are passed, they can be preserved instead of overwriting with ground truth labels.

get_gt_thresh_iou(
    [100, 150, 180, 300, "hat"],
    coords,
    iou_thresh=0.3,
    anchor_labels=labels,
    update_labels=False,
    return_ious=True,
)

({0: MultiBx(coords: 3, labels: 3)},
 {0: (#3) [0.453,0.4899,0.3921]},
 {})

True annots can also be a list containing the label as the last item.

get_gt_thresh_iou([100, 150, 180, 300, "hat"], coords, iou_thresh=0.3, return_ious=True)

({0: MultiBx(coords: 3, labels: 3)},
 {0: (#3) [0.453,0.4899,0.3921]},
 {})

get_gt_thresh_iou(
    [[100, 150, 180, 300, "hat"], [100, 120, 280, 200, "shirt"]],
    coords,
    iou_thresh=0.3,
    return_ious=True,
)

/tmp/ipykernel_330543/1922175922.py:57: NoGroundTruthBxs: No good ground truth anchors found for label=shirt, try lowering threshold (iou_thresh=0.3 or increasing candidates.
  warnings.warn(

({0: MultiBx(coords: 3, labels: 3), 1: None},
 {0: (#3) [0.453,0.4899,0.3921], 1: (#0) []},
 {})

Method to also return just the max IOU ground truth boxes.

---

source

get_gt_max_iou

def get_gt_max_iou(
    true_annots, anchor_boxes, anchor_labels:NoneType=None, return_ious:bool=False, return_masks:bool=False,
    positive_boxes:int=1, update_labels:bool=True, box_ids:NoneType=None
):

Find the highest-IoU anchors for each ground-truth box.

Selects up to positive_boxes distinct anchors for each ground-truth box. Matching is independent for each ground-truth box, so an anchor may match more than one object. IoUs and rankings are calculated as vectorized NumPy operations, and IoUs are returned at full floating-point precision.

Args: true_annots (Any): True annotations, typically in pascal_voc format anchor_boxes (Any): Candidate anchor boxes, typically calculated with pybx.bxs anchor_labels (List, optional): Anchor box labels, will be overwritten with true labels if update_labels=True. Defaults to None. return_ious (bool, optional): Return IOU values for selected positive ground truth anchor boxes. Defaults to False. return_masks (bool, optional): Return boolean masks for all anchor boxes indicating if a box is positive (True) or negative (False) ground truth box. Defaults to False. update_labels (bool, optional): Overwrite with true annotations. Defaults to True. positive_boxes (int, optional): Number of extra/positive ground truth boxes to return. Defaults to 1. box_ids (Iterable, optional): Unique identifiers for the ground-truth boxes. Defaults to their zero-based input positions.

Returns: MatchResult: Box IDs, selected anchor indices, optional IoUs and masks, and matched boxes. Every mapping is keyed by the IDs in box_ids.

get_gt_max_iou(true_annots, coords)

({0: MultiBx(coords: 1, labels: 1),
  1: MultiBx(coords: 1, labels: 1)},
 {},
 {})

Can also use methods for the box to calculate properties or convert to different box formats.

tmp_bx = get_gt_max_iou(true_annots, coords)[0][0]
tmp_bx[0]

BaseBx(coords=[[183, 67, 243, 188]], label=['clock'])

tmp_bx[0].xywh()

[[183, 67, 60, 121, 'clock']]

tmp_bx[0].yolo(w=300, h=300, normalize=False)  # cx, cy, bw, bh

[[213.0, 127.5, 60.0, 121.0, 'clock']]

# here w h is the image w and h
tmp_bx[0].yolo(w=300, h=300, normalize=True)  # cx/w, cy/h, bw/w, bh/h

[[0.71, 0.425, 0.2, 0.4033333333333333, 'clock']]

np.round([0.4046875, 0.840625, 0.503125, 0.24375], 4)

array([0.4047, 0.8406, 0.5031, 0.2438])

true_annots

[{'x_min': 130, 'y_min': 63, 'x_max': 225, 'y_max': 180, 'label': 'clock'},
 {'x_min': 13, 'y_min': 158, 'x_max': 90, 'y_max': 213, 'label': 'frame'}]

gt_anchors_per_box, ious_per_box, mask_per_box = get_gt_max_iou(
    true_annots,
    coords,
    return_ious=True,
    return_masks=True,
    positive_boxes=1,  # number of positive bounding boxes to allow
)
gt_anchors_per_box, ious_per_box, mask_per_box

({0: MultiBx(coords: 1, labels: 1),
  1: MultiBx(coords: 1, labels: 1)},
 {0: (#1) [0.3482], 1: (#1) [0.4488]},
 {0: (#1741) [False,False,False,False,False,False,False,False,False,False...],
  1: (#1741) [False,False,False,False,False,False,False,False,False,False...]})

Vocabulary: - Ground truth bounding box - Ground truth anchor box or positive anchor box - Negative anchor box - Offset is calculated as ground truth bounding box minus positive anchor box coordinates (not for all anchor boxes, use mask) - Normalized offsets

Calculate offsets for ground truth anchor boxes

BaseBx also supports calculation of bounding box offset by calling the get_offset() method.

mask_per_box

{0: (#1741) [False,False,False,False,False,False,False,False,False,False...],
 1: (#1741) [False,False,False,False,False,False,False,False,False,False...]}

gt_anchors_per_box

{0: MultiBx(coords: 1, labels: 1),
 1: MultiBx(coords: 1, labels: 1)}

true_annots_bx = get_bx(true_annots)
true_annots_bx

MultiBx(coords: 2, labels: 2)

gt_anchors_per_box[0]

MultiBx(coords: 1, labels: 1)

gt_anchors_per_box[0]  # this is still a MultiBx

MultiBx(coords: 1, labels: 1)

len(gt_anchors_per_box[0])

1

gt_anchors_per_box[0][0]  # this is a BaseBx, which wont raise a warning

BaseBx(coords=[[183, 67, 243, 188]], label=['clock'])

true_annots_bx[0].get_offset(gt_anchors_per_box[0])

/run/media/data1/projects/pybx/pybx/basics.py:686: BxViolation: Other should be BaseBx, got MultiBx
  warnings.warn(BxViolation(f"Other should be BaseBx, got MultiBx"))

(#4) [-5.9167,-0.4959,2.2977,-0.1681]

true_annots_bx[0].get_offset(
    gt_anchors_per_box[0][0]
)  # by default normalize=True

(#4) [-5.9167,-0.4959,2.2977,-0.1681]

With normalize=False, it calculates simple difference between centers (dcx, dcy) and ratio of width and heights log(w'/w), log(h'/h) the two boxes.

true_annots_bx[0].get_offset(gt_anchors_per_box[0][0], normalize=False)  #

(#4) [-35.5,-6.0,95.0,117.0]

The following helper function repeats the same operation for multiple boxes, provided the masks (so that only ground truch anchor box offsets are calculated)

---

source

get_gt_offsets

def get_gt_offsets(
    true_annots:BaseBx, anchor_boxes, anchor_labels:NoneType=None, # do we need to pass this
    masks:NoneType=None, sigma:tuple=(0.1, 0.2), normalize:bool=True, log_func:ufunc=log, update_labels:bool=False
):

Calculates the offset of the true annotations from the anchor boxes using the get_offset method of BaseBx.

Args: true_annots (Any): True annotation for a single object, typically in pascal_voc format anchor_boxes (Any): Candidate anchor boxes, typically calculated with pybx.bxs anchor_labels (List, optional): Anchor box labels, will be overwritten with ground truth labels if update_labels=True. Defaults to None. masks (List, optional): Anchor box masks indicating if a box is positive/negative anchor box. If nothing is passed, offsets are calculated for all anchor boxes passed. Defaults to None. sigma (tuple, optional): Estimated of standard deviation for the distances and ratios. Defaults to (0.1, 0.2). normalize (bool, optional): Whether to normalize the offsets using the methods used in the SSD paper. Defaults to True. log_func (func, optional): Function for normalizing the ratio of widths and heights. Defaults to np.log. update_labels (bool, optional): Overwrite positive anchor boxes with object class and negative anchor boxes with background class. Defaults to False.

Returns: list: List of all anchor box offsets. list: List of corresponding anchor box labels.

The Ground truth bounding boxes should ideally be passed in as a BaseBx, or will attempt a conversion to BaseBx.

true_annots_bx[0], get_bx([true_annots[0]])[0]

(BaseBx(coords=[[130, 63, 225, 180]], label=['clock']),
 BaseBx(coords=[[130, 63, 225, 180]], label=['clock']))

# passing dict
gt_offsets_clock, labels = get_gt_offsets(
    true_annots[0], coords, masks=mask_per_box[0]
)
gt_offsets_clock[
    mask_per_box[0]
]  # using the mask to look at only the valid offset

array([[-5.9167, -0.4959,  2.2977, -0.1681]])

# passing list
gt_offsets_clock, labels = get_gt_offsets(
    [130, 63, 225, 180, "clock"], coords, masks=mask_per_box[0]
)
gt_offsets_clock[mask_per_box[0]]

array([[-5.9167, -0.4959,  2.2977, -0.1681]])

Updating the labels of the candidates to actual ground truth class labels might be a nice addition as well.

gt_offsets_clock, labels = get_gt_offsets(
    true_annots[0], coords, masks=mask_per_box[0], update_labels=False
)
labels.unique()

(#1) ['background']

gt_offsets_clock, labels = get_gt_offsets(
    true_annots[0], coords, masks=mask_per_box[0], update_labels=True
)
labels.unique()

(#2) ['background','clock']

If no ground truth label is passed with update_labels=True, then unknown is assigned. Dictionary annotations are supported.

gt_offsets_clock, labels = get_gt_offsets(
    [130, 63, 225, 180], coords, masks=mask_per_box[0], update_labels=True
)
labels.unique()

(#2) ['background','unknown']

gt_offsets_clock[:, 0]

array([0., 0., 0., ..., 0., 0., 0.])

gt_offsets_clock[
    mask_per_box[0]
]  # czan use the mask to index the only valid offset

array([[-5.9167, -0.4959,  2.2977, -0.1681]])

The log of widths and heights can be skipped or modified if a different log_func is passed log(w'/w), log(h'/h).

from fastcore.foundation import (
    noop,
)  # do no operation, basically an identity function f(x) = x

gt_offsets_clock_nolog, _ = get_gt_offsets(
    true_annots_bx[0], coords, masks=mask_per_box[0], log_func=noop
)
gt_offsets_clock_nolog[mask_per_box[0]]

array([[-5.9167, -0.4959,  7.9167,  4.8347]])

Or a custom funciton.

gt_offsets_clock_nolog, _ = get_gt_offsets(
    true_annots_bx[0],
    coords,
    masks=mask_per_box[0],
    log_func=lambda x: x * np.log2(x),
)
gt_offsets_clock_nolog[mask_per_box[0]]

array([[-5.9167, -0.4959,  5.2485, -0.2345]])

Repeating the same operations for all classes.

for box_id, true_bx in enumerate(true_annots_bx):
    get_gt_offsets(true_bx, coords, masks=mask_per_box[box_id])

true_annots_bx

MultiBx(coords: 2, labels: 2)

If a mask is not provided, offsets are calculated for all anchors, which makes the process slow.

%%timeit
get_gt_offsets(true_annots_bx[0], coords)  # without mask

135 ms ± 2.32 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
get_gt_offsets(true_annots_bx[0], coords, masks=mask_per_box[0])  # with mask

468 µs ± 90.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)