This notebook goes through how to build a Siamese dataset from scratch.
Common use cases:
- Person identification
- Small classification sample size
Why Siamese?
Let's think of an example problem:
I own 3 dogs and I want to differentiate between the three of them from a photo, but I only have 5 images of each animal. By our normal standards, we would say that is far too little of data for us to work with. But in this case, we have now 120 training samples (not including augmentation).
Our example will use the PETS dataset. We won't be training, but if you're dealing with this problem, you should have all the tools you need by now
from fastai.vision.all import *
src = untar_data(URLs.PETS)/'images'
We'll grab all the file names:
items = get_image_files(src)
And now we can start preparing our dataset. We will be doing everything at the lowest level possible today. First let's make a transform that will open some image from a filename and resize it.
def resized_image(fn:Path, sz=128):
"Opens an image from `fn` and resizes it to `sz`"
x = Image.open(fn).convert('RGB').resize((sz,sz))
return tensor(array(x)).permute(2,0,1).float()/255.
Now let's get two random images (that we know are different)
img1 = resized_image(items[0], 448)
img2 = resized_image(items[1], 448)
Now we need some way of viewing our image, along with a title. Let's make a TitledImage class:
class TitledImage(Tuple):
def show(self, ctx=None, **kwargs): show_titled_image(self, ctx=ctx, **kwargs)
TitledImage(img1, 'Test').show()
Now let's make something similar for a pair of images (our Siamese)
class SiameseImage(Tuple):
def show(self, ctx=None, **kwargs):
im1, im2, is_same = self
return show_image(torch.cat([im1,im2], dim=2), title=is_same, ctx=ctx, **kwargs)
Let's look at two examples (which look remarkably) similar to that image earlier:
SiameseImage(img1, img1, True).show(figsize=(7,7));
SiameseImage(img1, img2, False).show(figsize=(7,7));
class SiamesePair(Transform):
"A transform to generate Siamese data"
def __init__(self, items, labels):
self.items, self.labels, self.assoc = items,labels,self
sortlbl = sorted(enumerate(labels), key=itemgetter(1))
self.clsmap = {k:L(v).itemgot(0) for k,v in itertools.groupby(sortlbl, key=itemgetter(1))}
self.idxs = range_of(self.items)
def encodes(self,i):
"x: tuple of `i`th image and a random image from same or different class; y: True if same class"
othercls = self.clsmap[self.labels[i]] if random.random()>0.5 else self.idxs
otherit = random.choice(othercls)
same = tensor([self.labels[otherit]==self.labels[i]]).int()
return SiameseImage(self.items[i], self.items[otherit], same)
We are going to want some labels to be sued, so let's grab some:
labeller = RegexLabeller(pat = r'/([^/]+)_\d+.jpg$')
labels = items.map(labeller)
labels[:5], len(labels)
Now we can build our SiamesePair transform
sp = SiamesePair(items, labels)
Let's look at a few bits
sp.clsmap
sp.labels
Now finally, we can build our Pipeline
Bringing it to a DataLoader
First we'll want to make a Transform out of that resized_image function we had
OpenAndResize = Transform(resized_image)
And now that we have all the pieces together, let's build a Pipeline:
pipe = Pipeline([sp, OpenAndResize])
And take a look at it's first set:
x,y,z = pipe(0)
x.shape, y.shape, z
To turn anything into a DataLoader, we want it to first be a TfmdList. We can accomplish this by passing in a list of index's and a Pipeline to run through:
tls = TfmdLists(range_of(items), pipe)
And now make our Dataloaders
dls = tls.dataloaders(bs=16, after_batch=[Normalize.from_stats(*imagenet_stats)])
And we can look at a batch!
batch = dls.one_batch()
Now I did not get the show function working, so let's take a look the very "simple" way
a,b,c = batch[0][0], batch[1][0], batch[2][0]
a.shape, b.shape, c
from torchvision import transforms
im1 = transforms.ToPILImage()(batch[0][0]).convert("RGB")
im2 = transforms.ToPILImage()(batch[1][0]).convert("RGB")
display(im1, im2)
While we cannot display, you now have a DataLoader ready for Siamese networks!
For more information on training a model and predicting, see the official fastai Siamese tutorial here