Real-Time Predictions with a Feature Store

This video shows how you can leverage the Hopsworks online feature store to compute and ingest features and make them available to operational models making real-time predictions, with low latency and preventing skew between the training and serving features.

How to run an existing Spark Job in Hopsworks

End-to-end Anomaly Detection Model Using Hopsworks

Learn how to develop this state-of-the-art anomaly detection model and manage its entire lifecycle by applying modern MLOps and DataOps principles.

How to interact with the Hopsworks Feature Store using the Java or Scala API

This short video shows you how to interact with the Hopsworks Feature Store using the Java or Scala API.

How to create a Training Dataset in Hopsworks using the Java or Scala API

This short video shows you how to create training datasets in Hopsworks using the Java or Scala API.

Build ML models with fastai and Jupyter in Hopsworks

. Hopsworks provides Jupyter as a service in the platform, including kernels for writing PySpark/Spark and pure Python code. With an intuitive service to install Python libraries covered in a previous blog and access to a Jupyter notebook, getting started with your favourite ML library requires little effort in Hopsworks.

Manage Python libraries in Hopsworks

Hopsworks provides a Python environment per project that is shared among all the users in the project. All common installation alternatives are supported, , in addition to libraries packaged in a .whl or .egg file and those that reside on a git repository.

Real-Time Predictions with a Feature Store

This video shows how you can leverage the Hopsworks online feature store to compute and ingest features and make them available to operational models making real-time predictions, with low latency and preventing skew between the training and serving features.

Connect Hopsworks to AWS

Connecting Hopsworks to your organisation’s AWS account is the first step towards using the Feature Store: 1. Connect your AWS account, 2. Create an instance profile, 3. Create a S3 bucket, 4. Create a SSH key, 5. Enable permissions for Hopsworks to access

Connect Hopsworks to Azure

Connecting Hopsworks to your organisation’s Azure account is the first step towards using the Feature Store: 1. Connect your Azure account, 2. Create and configure a storage, 3. Add a ssh key to your resource group, 4. Enable permissions for Hopsworks to access

Simplifying Feature Engineering with a Feature Store

A feature store is a data platform that manages and governs your features for machine learning - both for training and serving. You have access to and can reuse previously engineered features available within the entire organisation, avoiding the need to write a feature engineering pipeline for every model put in production.

Hopsworks Feature Store with Microsoft Azure

The Hopsworks Feature Store is available today on Azure as both a managed platform (www.hopsworks.ai) and a custom Enterprise installation. It manages your features for training and serving models in a cluster under your control inside your organisation’s existing cloud account.

Provenance for Machine Learning Pipelines. Guest Lecture at Boston University.

This video goes into depth on the scale-out metadata architecture in Hopsworks and it is used to enable Impliciit Provenance for Machine Learning Pipelines. It was a Guest Lecture at Boston University in October 2020, on a course taught by John (Ioannis) Liagos.

Solve Fraud Challenges with Graph Network and Deep Learning in Hopsworks

Financial institutions invest huge amounts of resources in both identifying and preventing money laundering and fraud. Most existing systems for identifying money laundering are rules-based that are not capable of detecting ever changing schemes. Consequently, these systems generate too many false-positive alerts, taking time and money to run down.

Building a Feature Store around Dataframes and Apache Spark

A Feature Store enables machine learning (ML) features to be registered, discovered, and used as part of ML pipelines, thus making it easier to transform and validate the training data that is fed into machine learning systems. Feature stores can also enable consistent engineering of features between training and inference, but to do so, they need a common data processing platform. The first Feature Stores, developed at hyperscale AI companies such as Uber, Airbnb, and Facebook, enabled feature engineering using domain specific languages, providing abstractions tailored to the companies’ feature engineering domains. However, a general purpose Feature Store needs a general purpose feature engineering, feature selection, and feature transformation platform.

From Python to PySpark and Back Again Unifying Single host and Distributed Deep Learning with Maggy

Distributed deep learning offers many benefits – faster training of models using more GPUs, parallelizing hyperparameter tuning over many GPUs, and parallelizing ablation studies to help understand the behaviour and performance of deep neural networks. With Spark 3.0, GPUs are coming to executors in Spark, and distributed deep learning using PySpark is now possible. However, PySpark presents challenges for iterative model development – starting on development machines (laptops) and then re-writing them to run on cluster-based environments.

Feature Engineering with a Cloud-Native Feature Store with Hopsworks.ai

The Hopsworks.ai is the world's first managed feature store for the cloud that provides an open-source and general purpose feature store to design and operate AI applications at scale with Apache Spark.

Hopsworks Feature Store for Kubeflow and On-Premises Clusters (Hadoop)

The Hopsworks Feature Store is a platform for managing and serving features for machine learning that also integrates Kubeflow and On-Premises Clusters (Hadoop).

Hopsworks Feature Store for SageMaker

The Hopsworks Feature Store is a platform for managing and serving features for machine learning. It integrates seamlessly with popular platforms for Data Science and cloud-based data lakes based on S3.

Managed Feature Store

All hyperscale AI companies (LinkedIn, Facebook, AiBnB, Microsoft, Google, etc) build their machine learning platforms around a Feature Store.

Feature Store for Databricks

The Hopsworks Feature Store is a platform for managing and serving features for machine learning. It integrates seamlessly with popular platforms for Data Science and cloud-based data lakes based on S3.

The Feature Store for Machine Learning

The Hopsworks Feature Store is a storage and compute platform for managing, discovering, and sharing feature data for machine learning. It integrates seamlessly with popular platforms for Data Science, such as AWS Sagemaker, Databricks, on-premise Hadoop data lakes, and cloud-based data lakes based on S3.

Hopsworks.ai on AWS configure and start your cluster

Get started with hopsworks.ai

HopsFS (HDFS) on Hopsworks

On Hopsworks, learn how to: 1. upload/view/share data in Hopsworks 2. use HDFS (HopsFS) in python applications

Distributed training on Hopsworks with collective allreduce

On Hopsworks, learn how to: 1. train a TensorFlow model using many GPUs using Hopsworks 2. how to use CollectiveAllReduce distribution strategy.

Intro to TensorBoard on Hopsworks

On Hopsworks, learn how to: 1. Run TensorBoard from a Jupyter notebook 2. Inspect TensorBoards for historical experiments

Hyperparam optimization on TensorFlow with Hopsworks

On Hopsworks, learn how to: 1. run an asynchronous hyperparam opt job on Jupyter using PySpark and the Maggy framework 2. run a synchronous hyperparam opt job on Jupyter using PySpark and Hopsworks

dotAI 2018 - Jim Dowling - All AI Roads Lead to Distribution

Jim introduces the latest developments in distributed Deep Learning and how distribution can both massively reduce training time and enable parallel experimentation for both AutoML and hyperparameter optimization. We will introduce different distributed deep learning paradigms, including model-level parallelism and data-level parallelism, and show how data parallelism can be used for distributed training. We will also introduce the open-source Hops platfrom that supports GPUs as a resource, a more scalable HDFS filesystems and a secure multi-tenant environment. We will show how to program a machine learning pipeline, end-to-end with only Python code on Hops.

Berlin Buzzwords 2019: Jim Dowling – Hops in the Cloud

Hops is a European open-source, next-generation distribution of Apache Hadoop that is being repurposed for the cloud. In this talk, we will walk through some of recent technical developments in Hops, including solving the small files problem by stuffing them in metadata using NVMe disks, free-text search of file system with extended metadata (this is great for automated annotation of millions of images and then finding them in milliseconds with consistent), and most interestingly data-center level HA for HopsFS with millions of filesystem operations per second on real industrial workloads.

End-to-End ML pipelines with Beam, Flink, TensorFlow, and Hopsworks (Beam Summit Europe 2019)

Apache Beam is a key technology for building scalable End-to-End ML pipelines, as it is the data preparation and model analysis engine for TensorFlow Extended (TFX), a framework for horizontally scalable Machine Learning (ML) pipelines based on TensorFlow. In this talk, we present TFX on Hopsworks, a fully open-source platform for running TFX pipelines on any cloud or on-premise. Hopsworks is a project-based multi-tenant platform for both data parallel programming and horizontally scalable machine learning pipelines.

PyTorch on Hopsworks

On Hopsworks, learn how to run your first PyTorch application on a Jupyter notebook

Hopsworks' Feature Store, Petastorm and Tensorflow

On Hopsworks, learn how to: 1. Create Training/Test Datasets in Petastorm and register them with the Feature Store 2. Use Petastorm Training/Test datasets in the Feature Store to train and score a model in TensorFlow

End-to-End ML with Hopsworks' Feature Store, SKLearn, Model Serving, Inference

On Hopsworks, learn how to: 1. Register Features in the Hopsworks' Feature Store using the Python API 2. Create a Pandas DF from the Feature Store for Training Data

Running Jupyter notebooks in Airflow

On Hopsworks, learn how to: 1. run a Jupyter notebook as a Job, 2. run Jobs in an Airflow DAG

Model serving on Tensorflow in Hopsworks

On Hopsworks, learn how to: 1. Train and export a model from a TensorFlow application 2. Serve the named+versioned model on TensorFlow Serving Server

Create a Hopsworks Instance on Google Cloud

On Hopsworks, learn how to use GCE to create a Hopsworks instance using a Hopsworks image

Intro to MNIST on Hopsworks

On Hopsworks, learn how to: 1. run a TensorFlow application in a Jupyter notebook on Hopsworks 2. How to use a GPU to train the model

ROCm and Distributed Deep Learning on Spark and TensorFlow -Jim Dowling Logical Clocks AB, Ajit Mathews

ROCm, the Radeon Open Ecosystem, is an open-source software foundation for GPU computing on Linux. ROCm supports TensorFlow and PyTorch using MIOpen, a library of highly optimized GPU routines for deep learning. In this talk, we describe how Apache Spark is a key enabling platform for distributed deep learning on ROCm, as it enables different deep learning frameworks to be embedded in Spark workflows in a secure end-to-end machine learning pipeline. We will analyse the different frameworks for integrating Spark with Tensorflow on ROCm, from Horovod to HopsML to Databrick's Project Hydrogen.

Bay Area AI: Jim Downing, The Feature Store: the missing API between Data Engineering Science?

In this talk, we will introduce the world's first open-source Feature Store, built on Hopsworks, Apache Spark, and Apache Hive and targeting both TensorFlow/Keras and PyTorch. We will show how ML pipelines can be programmed, end-to-end, in Python, and the role of the Feature Store as a natural interface between Data Engineers and Data Scientists. In an end-to-end pipeline, we will show how the Feature Store works, and how you can write end-to-end ML pipelines in Python only

Distributed Deep Learning by Jim Dowling

Methods that scale with computation are the future of AI", Richard Sutton, father of reinforcement learning. Large labelled training datasets were only one of the key pillars of the deep learning revolution, the widespread availability of GPU compute was the other. The next phase of deep learning is the widespread availability of distributed GPU compute. As data volumes increase, GPU clusters will be needed for the new distributed methods that already produce the state-of-the-art results for ImageNet and Cifar-10, such as neural architecture search

Theofilos Kakantousis - Multi-tenant Deep Learning and Streaming as-a-Service with Hopsworks

Distributed Deep Learning with Apache Spark and TensorFlow

Enterprises and non-profit organizations often work with sensitive business or personal information, that must be stored in an encrypted form due to corporate confidentiality requirements, the new GDPR regulations, and other reasons. Unfortunately, a straightforward encryption doesn't work well for modern columnar data formats, such as Apache Parquet, that are leveraged by Spark for acceleration of data ingest and processing. When Parquet files are bulk-encrypted at the storage, their internal modules can't be extracted, leading to a loss of column / row filtering capabilities and a significant slowdown of Spark workloads. Existing solutions suffer from either performance or security drawbacks.

Distributed TensorFlow on Hops | Fabio Buso | OSCON 2018

Fabio Buso shares the latest developments in distributed TensorFlow and shows how distribution can both massively reduce training time and enable parallel experimentation for hyperparameter optimization. You’ll explore different distributed architectures for TensorFlow, including the parameter server and “ring allreduce” models, with a focus on open source TensorFlow frameworks that leverage Apache Spark to manage distributed training, such as Yahoo’s TensorFlowOnSpark, Uber’s Horovod, and the Hops model. Fabio also covers the different programming models supported and highlights the importance of cluster support for managing GPUs as a resource.

Multi-tenant Streaming-as-a-Service with Hops,

Hops FS is the first production-grade distributed hierarchical filesystem to store metadata normalized in an in-memory, shared nothing database. In this talk, we present how HopsFS reached a performance milestone by scaling the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. We discuss the challenges in building secure multi-tenant streaming applications on YARN that are metered and easy-to-debug. We also show how users in Hopsworks use the ELK stack for logging their running streaming applications as well as how they use Grafana and Graphite for monitoring.