Manage your own Feature Store on Databricks with Hopsworks
Wednesday July 15th @8:00 PM CEST
15
Hopsworks is an open-source platform for the development and operation of ML models, available as an on-premises platform (open-source or Enterprise version) and as a managed platform on AWS. Hopsworks' features include:
‣End-to-End ML Pipelines - the Feature Store to training to serving;
‣Manages ML Assets: Features, Experiments, Model Repository/Monitoring;
‣Project-based multi-tenancy - collaborate with sensitive data;
‣Enterprise integrations with Active Directory, LDAP, OAuth2, Kubernetes;
‣Full governance and provenance for ML assets, with GDPR compliance;
‣Open-source ML with Spark, TensorFlow, PyTorch, Scikit-Learn;
‣Solves the hardest scaling problems: training, hparam tuning, feature engineering.
The Hopsworks Feature Store is a platform for managing features for when both training and serving models, available as an on-premises platform (open-source or Enterprise version) and as a managed platform on AWS. Our Feature Store includes:
‣Ingest, validate, visualize features
‣Catalog, tag, and search for features;
‣Reuse features when creating train/test data;
‣Offline feature store for training and batch inference;
‣Online feature store for real-time features for online apps.
Hopsworks is an open-source platform for the development and operation of ML models, available as an on-premises platform (open-source or Enterprise version) and as a managed platform on AWS. Hopsworks' features include:
‣ End-to-End ML Pipelines - the Feature Store to training to serving;
‣ Manages ML Assets: Features, Experiments, Model Repository/Monitoring;
‣ Project-based multi-tenancy - collaborate with sensitive data in a shared cluster;
‣ Enterprise integrations with Active Directory, LDAP, OAuth2, Kubernetes;
‣ Full governance and provenance for ML assets, with GDPR compliance;
‣ Open-source ML with Spark, TensorFlow, PyTorch, Scikit-Learn;
‣ Solves the hardest scaling problems: training, hparam tuning, feature engineering.
The Hopsworks Feature Store is a platform for managing features for when both training and serving models, available as an on-premises platform (open-source or Enterprise version) and as a managed platform on AWS. Our Feature Store includes:
‣ Ingest, validate, visualize features;
‣ Catalog, tag, and search for features;
‣ Reuse features when creating train/test data;
‣ Offline feature store for training and batch inference;
‣ Online feature store for real-time features for online apps.
Our Feature Store is the foundation for managing Data for AI across all industries. It is enabling collaboration between data science, data engineering, and ML operations teams. The full Hopsworks platform builds on the Feature Store to provide a rich Data Science platform built on open-source ML frameworks for feature engineering, model training, and model serving - all at scale.
Managing genomic and medical data securely and at scale in on-premises clusters, while providing industry-leading analytics and AI capabilities.
Learn more >
Providing group-wide AI capabilities to solve prediction problems for fraud, markets, and customers.
Learn more >
Managing AI clusters on-premises, with industry-leading scale-out training of models and experiments on GPUs.
Learn more >
Providing group-wide AI capabilities to address challenges from regulators, cybercrime and customers.
Learn more >
Develop and Operate AI applications with Hopsworks.
Hopsworks manages and processes your data at scale for AI. The Hopsworks Feature Store manages features to be used in both training and serving models. It builds on HopsFS, the world's most scalable hierarchical HDFS- filesystem.
You can scale out training and hyperparameter optimization with as many GPUs as you can put in your cluster. And we provide framework support (Maggy and PySpark) to make distributed ML as Pythonic as possible.
ML pipelines have become the defacto way to productionize ML models. Hopsworks uses Airflow to orchestrate pipelines consisting of anything from (Py)Spark jobs, to Python programs on K8s, to Jupyter notebooks, to TensorFlow Extended (Beam/Flink).
JupyterLab is provided as a collaborative development environment, while jobs can also be deployed as programs: Python, PySpark, Scala/Java Spark, Beam/Flink.
Hopsworks can version all ML artifacts in ML pipelines: features in the feature store, train/test datasets, programs and pipelines in Github, and models in the model repository. Hopsworks also provides industry-leading support for provenance in ML pipelines: debug and explore lineage between processing steps and ML artifacts.
Unlike MLFlow and TFX, you do not need to re-write your piplines to add provenance: it is implicitly captured by our unique change-data-capture technology.
Hopsworks manages and processes your data at scale for AI. The Hopsworks Feature Store manages features to be used in both training and serving models. It builds on HopsFS, the world's most scalable hierarchical HDFS- filesystem.
You can scale out training and hyperparameter optimization with as many GPUs as you can put in your cluster. And we provide framework support (Maggy and PySpark) to make distributed ML as Pythonic as possible.
ML pipelines have become the defacto way to productionize ML models. Hopsworks uses Airflow to orchestrate pipelines consisting of anything from (Py)Spark jobs, to Python programs on K8s, to Jupyter notebooks, to TensorFlow Extended (Beam/Flink).
JupyterLab is provided as a collaborative development environment, while jobs can also be deployed as programs: Python, PySpark, Scala/Java Spark, Beam/Flink.
Hopsworks can version all ML artifacts in ML pipelines: features in the feature store, train/test datasets, programs and pipelines in Github, and models in the model repository. Hopsworks also provides industry-leading support for provenance in ML pipelines: debug and explore lineage between processing steps and ML artifacts.
Unlike MLFlow and TFX, you do not need to re-write your piplines to add provenance: it is implicitly captured by our unique change-data-capture technology.
Our Feature Store is enabling Data Management across all industries for AI. Improving the workflow, simplifying communication between teams and allowing ground breaking work to be produced by in-house teams and partners with unparalleled ease using one of the industry's first Feature Store.
“Hopsworks keeps big organisations on their toes for what is possible and offers the implementation speed and options of on-premise and cloud based on data scientists’ needs”
Marcus Smed
Agile Product Owner
@Swedbank
‘’Hopsworks is a project-oriented data platform with an extensive range of functionality such as the Feature Store that integrates well with other platforms such as Databricks and Sagemaker"
.jpg)
Xavier Mehaut
Senior Architect Big Data, Cloud & AI
@AI & DATA
Watch this Feature Store presentation to understand how it can improve your ML model development and operations.
Work with the industry's most scalable ML and Data platform, with technology built by a team of world-leading engineers and researchers.
The Feature Stores for ML Newsletter brings highlights, news, events, and insights related to Feature Stores and ML in industry.
