Google ‘visually similar images’ search for beginners

As a visual counterpart to Google search engine, Google Images was introduced in 2001 to accommodate the growing need to find image content on the Internet. In the beginning, the search was only text-based and required a descriptive approach (search by keywords, filename, title, etc.), but a development of Google search algorithms brought a new feature in 2009, ‘Search by Image’. It allows searching by image as opposed to text by uploading an image from a computer or pasting the URL of an image from the web into the search box, or, alternatively, by using a ‘drag and drop’ option. The technology behind this tool analyses the image’s most distinctive points, lines, colours and textures and creates an algorithm that Google engine matches against billions of images in its index.1

The Google Images search by text or image offers an alternative for art historians to reach out for the content outside defined and limited collections, often functioning as digital silos on the web. Images outside those collections, that appear on various websites, blogs and digitally published resources are becoming more “searchable” this way.  Google Images has been already helping researchers in finding images in different sizes and resolutions and it is a favourite image resource among art historians.2 However, it is the search by visually similar images that can open new possibilities for art historians. 

The example here uses Google Images – Search by keywords: ‘frans hals portrait of a man holding a skull’. The search results for this well known portrait from the Barber Institute of Fine Arts, University of Birmingham give us many successful matches (Fig.1).

 

Google Images search by keywords - Frans Hals portrait of a man holding a skull

Fig. 1. Screenshot of Google Images Search by keywords: ‘frans hals portrait of a man holding a skull’.

 

To search ‘by image’ rather than text, we can drag and drop one of the images from the previous search results into the Google Images search bar (Fig. 2).

 

Google Images search by image - Frans Hals portrait of a man holding a skull

Fig. 2. Google Images – search by image

 

This time the results page gives us more detailed information about the image itself, its other ‘versions’ online and it allows us to filter the images by size (Fig. 3). The option to view visually similar images is on the bottom of the page.

 

Google Images search by image - Frans Hals portrait of a man holding a skull

Fig. 3. Screenshot of Google Images Search by image – results page.

 

The results of the first search show that this particular image appears in many sources online and in a similar, and good, ‘digital’ quality, which means that most of them were derived from the same digital reproduction (Fig. 4). The similarities picked by Google’s algorithm were almost perfect – see the image in the circle. It could easily make it to the top list of the ‘Find the Panda’ puzzles that triumphed on the Internet recently:-)

To see what I mean by ‘digital’ quality, search for images of Leonardo’s Mona Lisa or Jan van Eyck’s Arnolfini wedding to realize how many different ‘digital’ versions of the same masterpiece exist on the web. The colours, tones and hues are the main obvious differences, but other, like ‘trimmed’ or ‘stretched’ versions, also happen.

 

Google Images search by image - Frans Hals portrait of a man holding a skull

Fig. 4. Screenshot of Google Images Search by image – results page.

 

The second search results, visually similar images, give us a very interesting selection of images (Fig. 5).

 

Google Images search by image - visually similar images

Fig. 5. Screenshot of Google Images Search by image – visually similar images – results page.

 

This group of random, but visually similar portraits, would not be possible to view in a physical setting as it is. What’s even more interesting, this ‘selection’ is fluid as it changes due to the images being uploaded and removed from websites, new sites being indexed by Google, and also by the algorithm itself changing / improving over time. This screenshot was taken in September 2015, but a few more searches since then have proved that I would probably never get exactly the same ‘selection’ in my search results. However, even on the basis of these results, a further analysis could possible lead to new questions about paintings and artists that have not been linked before. This tool is far from being a perfect solution for art-historians searching for images online and exploring their ultimate connections, but it is a definitely a space to watch.


In my next post, I will write about one Unknown Man that was visually similar to George Frideric Handel and what I’ve found using digital tools… Watch this space!


  1. Johanna Wright, ‘Inside Search: Search by Text, Voice, or Image’, Google Search Blog, 2011 <http://insidesearch.blogspot.ie/2011/06/search-by-text-voice-or-image.html> [accessed 14 February 2016].
  2. Joan E. Beaudoin and Jessica Evans Brady, ‘Finding Visual Information: A Study of Image Resources Used by Archaeologists, Architects, Art Historians, and Artists’, Art Documentation: Journal of the Art Libraries Society of North America, 30.2 (2011), pp. 24–36 <http://www.jstor.org/stable/41244062>.