HopsFS
Open source, next generation scaleout metadata file system.

AI/ML needs a Key-Value store, and Redis is not up to it

RonDB shows higher availability and the ability to handle larger data sets in comparison with Redis, paving the way to be the fastest key-value store available.

read more
Hello Asynchronous Search for PySpark

Hopsworks supports easy hyperparameter optimization (both synchronous and asynchronous search), distributed training using PySpark, TensorFlow and GPUs.

read more
ExtremeEarth scales AI to the Earth Observation Community with Hopsworks

How ExtremeEarth Brings Large-scale AI to the Earth Observation Community with Hopsworks, the Data-intensive AI Platform

read more

Unified extended,
Searchable Metadata

HopsFS provides a principled architecture for easily extending metadata for files and directories.

In particular, this is useful in the domain of machine learning where we have both artifacts (feature data, training data, programs, models, log files) that are typically stored as files and metadata (experiments, hyperparameters, tags, metrics, etc ) that are stored in a metastore (often a relational database).

HopsFS unifies the artifact store and metastore.

100x Times Faster than AWS S3

What If you could build on top of S3 a distributed file system with a HDFS API that gives you POSIX goodness and improved performance?

That’s what we have done with a cloud-native release of HopsFS that is highly available across availability zones, has the same cost as S3, but has 100X the performance of S3 for file move/rename operations, and 3.4X the read throughput of S3 (EMRFS) for the DFSIO Benchmark (peer reviewed at ACM Middleware 2020).