Introducing EuclidesDB, a multi-model machine learning feature database

Yesterday, EuclidesDB, a multi-model machine learning feature database released its v0.1. EuclidesDB 0.1 is tightly coupled with PyTorch and provides a backend for including and querying data on the model feature space.

EuclidesDB provides a simple standalone server that stores, builds indexes, and serves requests using efficient serialization and protocols with an easy API. It provides APIs for including new data into the database and querying it later. Since it uses gRPC (gRPC Remote Procedure Call) for communication, the API can be consumed in many different languages.

As mentioned earlier, it comes with a very tight integration with PyTorch, where libtorch is used as the backend to run traced models. And, thus provides a very easy pipeline to integrate new models into the Euclides DB C++ backend.

The concept behind EuclidesDB

EuclidesDB is based on two main concepts:

Module/Model are terms used interchangeably to represent every computation.

Model Space represents a space of features generated by a model.

When a user adds a new image or other kinds of data into the database, they need to specify which model should be used to index this data. This data is then forwarded into the specified models and their features are saved into a local key-value database.

Similarly, when a user queries for similar items on a model space, they need to make a request with a new image and specify on which model spaces they want to find similar items. Then similar items for each model space is returned together with their relevance.