Build your own Feature Store

Feature Store for ML

Welcome to the feature store newsletter brought to you by Logical Clocks, where we in a monthly cadence will highlight the latest news, events, and insights as we help make companies successful in their machine learning transformation journey and empower businesses to be applied-AI model-driven companies

How to Build your own Feature Store

As of May 2020, Logical Clocks are the only vendor of a Feature Store for machine learning (ML) and the only maker of a fully open-source and cloud-native Feature Store for ML. As such, we have many conversations with companies and organisations who are deciding between building their own feature store and buying one. 

If you are considering following down the same path, we suggest that you read this blog post to learn from our experience of building the Hopsworks Feature Store. We walk through the most important benefits of feature stores, explain the reasoning behind some of the decisions and choices we made (and did not make) and include a flowchart to help you make tough decisions.

Editor's Picks

Unifying Single-host and Distributed Machine Learning with Maggy

Moritz Meister from Logical Clocks introduces a new feature to an open-source framework, Maggy, that enables write-once training functions that can be reused in single-host Python programs and cluster-scale PySpark programs.

Read more

Why StreamSQL moved from Apache Kafka to Apache Pulsar

Simba Khadder wrote in April about why they moved to Apache Pulsar and how it is used to support their online feature store for ML.

Read more

Accelerate Your Machine Learning With The StreamSQL Feature Store

Simba Khadder had this interview at the Data Engineering Podcast on the feature store they are building at StreamSQL. It’s a good listen.

Read more

Hopsworks for AWS SageMaker

Fabio Buso from Logical Clocks wrote a step by step guide for AWS Sagemaker users on how to perform exploratory data analysis with the feature store, discovering available features, and join features together to create train/test datasets - all from the comfort of the Jupyter notebook.

Read more

Why we need DevOps for ML Data

Kevin Stumpf at wrote about the 5 challenges for infrastructure support for data for machine learning (building, combining, monitoring, and serving features, as well as accessing the raw data).

Read more

Feature Stores: a guide for CEOs

ScribbleData wrote a blog that gives a high-level description of the motivation for feature stores, their view of the industry, and their experience with feature stores.

Read more

ML Pipelines with a Feature Store

Altexsoft wrote a blog about ML pipelines and the role of the Feature Store in ML pipelines (“if an eCommerce store recommends products that other users with similar tastes and preferences purchased, the feature store will provide the model with features related to that.”).

Read more 

Machine Learning in Production: Lessons Learned from Deploying Our First ML Model

Alex Post from Clearcover wrote this article in about how the starting point for their ML production journey was fixing the data layer - that is, building a feature store to manage their features.

Read more

Manage your own Feature Store on Kubeflow with Hopsworks

Jim Dowling from Logical Clocks explains step by step on how to integrate Hopsworks with Kubeflow. As Kubernetes has limited support for Spark, this blog post shows how to Hopsworks is often used for both its feature store and Spark and scale-out deep learning capabilities.

Read more

Highlights of Spark AI Summit on Feature Stores

Spark+AI Summit ended last week and we could prove that feature stores have never been a hotter topic! There were six talks on feature stores at this year’s conference - that is a 600% increase from last year! Click below to watch the videos.

Building a Feature Store around Dataframes and Apache Spark

Fabio Buso and Jim Dowling from Logical Clocks talked about the Hopsworks Feature Store and how it integrates with Databricks. They explain how the Hopsworks Feature Store centralises features for ML for easier discovery and governance, enables the reuse of features in different ML projects and provides a single pipeline or engineering features for both training and inference.

Watch now

From Python to PySpark and Back Again – Unifying Single-host and Distributed Deep Learning with Maggy

Moritz Meister and Jim Dowling from Logical Clocks introduce an open-source framework, Maggy, that enables write-once training functions that can be reused in single-host Python programs and cluster-scale PySpark programs.

Watch now

A Declarative Feature Engineering Framework

Nikhil Simha from AirBnB will talk about Zipline, their declarative feature engineering framework and Feature Store.

Watch now

Orchestrating Spark ML Pipelines and MLflow for Production

Nathan Buesgens at Accenture is due to talk about orchestrating Spark ML Pipelines and MLflow for Production in a Feature Store - “ break the anti-pattern of ‘clone and own’ ML pipelines”.

Watch now

Assisted-AI Feature Selection for Big Data Modelling

Alvin Henrick from Clarify Health Solutions will talk about an AI assisted way to identify which features improve the accuracy of a model and by how much - using AI to help select the best features for your models.

Watch now

Accelerate Real-Time ML with a Feature Platform

Mike Del Balso of will talk about their Feature Store and its support for centralized management of features for serving and training along with version-control.

Watch now

Job Opportunities

Data Scientist – Machine Learning Platform

Spotify is hiring a ML engineer to work on their feature store (data platform for AI).
Apply here

Software Engineer - ML Serving Platform

Pinterest is hiring a Software Engineer to join their ML Serving Platform team and work on their ML feature store.

Apply here

Upcoming Webinars & Events

Webinar: Manage your own Feature Store on Databricks with Hopsworks

Attend this webinar to learn how to write Python or Scala programs to compute features and register them in Hopsworks, browse and inspect features, and create train/test datasets from within a notebook.

Latest Videos

Get Started with Hopsworks