A scalable open-source framework for machine learning based image collection, annotation and classification: a case study for automatic fish species identification

Abstract

Citizen science platforms, social media and multiple smart phone applications enable collection of large amounts of georeferenced images. This provides a huge opportunity in biodiversity and ecological research, but also creates challenges for efficient data handling and processing. Recreational and small-scale fisheries is one of the fields that could be revolutionised by efficient, widely accessible and machine learning based processing of georeferenced images. The majority of non-commercial inland and coastal fisheries are considered data poor and are rarely assessed, yet they provide multiple societal benefits and can have large ecological impacts. Given that large quantities of fish observations and images are being collected by fishers every day, artificial intelligence (AI) and computer vision applications offer a great opportunity to improve data collection, automate analyses and inform management. Yet, to date, many AI image analysis applications in fisheries are focused on the commercial sector and are not publicly available for community use. In this study we present an open-source modular framework for large scale image storage, handling, annotation and automatic classification, using cost- and labour-efficient methodologies. The tool is based on TensorFlow Lite Model Maker library and includes data augmentation and transfer learning techniques, applied to different convolutional neural network models. We demonstrate the implementation of this framework in an example case study for automatic fish species identification from images taken through a recreational fishing smartphone application. The framework presented here is highly customisable for further advancement and community based image collection and annotation.