Published by Theofilos Kakantousis on

Introducing Hopsworks 0.6.0

Hopsworks 0.6.0 brings the latest features, improvements and bug fixes. Highlights include:

  • HopsML- A Python-first framework for building machine learning pipelines.It enables Data Scientists to leverage the versatility of Python with the scale-out capabilities of Hopsworks running with GPUs. It includes features such as:
    • Experiments service – A new way for Data Scientists to create, organize, manage and reproduce their experiments. It provides a unified view of all the experiments ran using the experiment API in the hops python library.
    • hops-util-py – A helper library for Hops that facilitates development by hiding the complexity of running applications, discovering services and interacting with HopsFS and Hopsworks. A plethora of examples, from ingesting events stored in Kafka with PySpark, to doing Distributed Deep Learning on GPUs, is available in our github repository.
    • Serving -A scalable architecture for serving of TensorFlow and Keras models. It uses the TensorFlow Serving server running on Kubernetes to dynamically scale the number of serving instances and handle load balancing. Data Scientists can also do free-text search on the serving logs from within the Hopsworks UI.
  • TensorFlow 1.11 with CollectiveAllReduce
  • Spark 2.3.2.
  • Upgrade of Elastic Stack to major version 6, along with improvements in managing user’s dashboard in the Logs view.
  • Improved project conda environment management and stability.

Release notes – Hopsworks – Version 0.6.0

New Feature

Improvement

  • [HOPSWORKS-256] – Project name should be equal regardless of capitalization
  • [HOPSWORKS-357] – Move public dataset size to details
  • [HOPSWORKS-451] – karamel override attributes for node groups
  • [HOPSWORKS-452] – Add support for rebuilding Elasticsearch index in ePipe
  • [HOPSWORKS-564] – Refactor ePipe to easily enable or disable different watch units
  • [HOPSWORKS-565] – Update ePipe start command
  • [HOPSWORKS-578] – supply custom tensorlfow wheel to chef for installation
  • [HOPSWORKS-579] – Add to hopsworks-ca ability to sign csr using multiple CAs
  • [HOPSWORKS-591] – Admin UI (Conda): Show disk usage for conda, allow cleanup
  • [HOPSWORKS-595] – [kagent] Persistent state store for kagent
  • [HOPSWORKS-601] – [kagent] Remove Chef attributes from Python code
  • [HOPSWORKS-607] – Improve TensorFlow demo tour
  • [HOPSWORKS-611] – Unset MetaEnabled on the dataset before deleting to avoid redundant calls to Elasticsearch
  • [HOPSWORKS-613] – Add support for application certificate versioning
  • [HOPSWORKS-631] – Integrate Pydoop HDFS API into Hops API
  • [HOPSWORKS-643] – Refactor TfServing code to allow pluggable providers
  • [HOPSWORKS-644] – Kagent improvements: multiple params from single recipe & my_hostname function
  • [HOPSWORKS-645] – [hops-hadoop-chef] Enable encryption for the block transfer
  • [HOPSWORKS-648] – Conda envs Garbage Collection
  • [HOPSWORKS-667] – Unify Job logs and Jupyter Elasticsearch index and Kibana objects creation
  • [HOPSWORKS-674] – Parallelize materialization of certificates
  • [HOPSWORKS-676] – Make SparkUI logs intuitive
  • [HOPSWORKS-690] – Add button to stop particular Livy session in Jupyter dashboard
  • [HOPSWORKS-697] – Refine structure of keystores and truststores in Hops
  • [HOPSWORKS-705] – Extend TfLibMapping to support ongoing environment creation
  • [HOPSWORKS-709] – Link to hops-util-py docs in Python service
  • [HOPSWORKS-725] – [kagent-chef] Make Conda GC blacklist more generic
  • [HOPSWORKS-726] – [hops-hadoop-chef] Update Hopsworks CA endpoints in yarn-site.xml
  • [HOPSWORKS-736] – dela updates and fixes
  • [HOPSWORKS-757] – Chef improvements for upgrading ELK stack
  • [HOPSWORKS-761] – Run Expat at the end of the installation
  • [HOPSWORKS-771] – Allow memory quota configuration in yarn

Task

Sub-task

  • [HOPSWORKS-244] – Replace deprecated _ttl field in elasticsearch
  • [HOPSWORKS-251] – Replace log4j socketAppender for Logstash
  • [HOPSWORKS-299] – Add functionality to cancel job schedule
  • [HOPSWORKS-509] – REST API v2 – Refactor error handling
  • [HOPSWORKS-558] – Upgrade hopslog-chef and elastic-chef
  • [HOPSWORKS-559] – Upgrade Hopsworks to work with latest elasticsearch
  • [HOPSWORKS-560] – Upgrade ePipe to work with latest elasticsearch
  • [HOPSWORKS-584] – REST Api for Tensorflow Serving
  • [HOPSWORKS-694] – Notebooks for Kafka tour (Python/Scala)
  • [HOPSWORKS-698] – Add Kafka Python documentation to hops-doc
  • [HOPSWORKS-699] – Add kafka python dependencies to anaconda base environment
  • [HOPSWORKS-737] – Update dela config template according to updated dropwizard version
  • [HOPSWORKS-738] – Remove configurable attributes that can be retrieved from kagent attributes (ex. mysql ip/port)
  • [HOPSWORKS-741] – Upgrade livy to work with Spark 2.3.2
  • [HOPSWORKS-758] – Bump py4j version in Hopsworks and make it configurable at runtime
  • [HOPSWORKS-784] – Job status stuck in “Starting ApplicationMaster”
  • [HOPSWORKS-789] – hops-util-py should account for elastic index being lowercase
  • [HOPSWORKS-790] – KibanaProxyServlet and TensorBoardService lowercase projectname

Bug

  • [HOPSWORKS-224] – Vizops driver and executor tabs do not display data
  • [HOPSWORKS-369] – Restrict characters allowed in project names
  • [HOPSWORKS-429] – Fix configuration of TFSpark in JobService
  • [HOPSWORKS-526] – checkstyle rules out of date for Java 8
  • [HOPSWORKS-587] – Chef recipes bug fixes and common install version
  • [HOPSWORKS-594] – Fix compilation bug in tensorflow::default
  • [HOPSWORKS-599] – Kafka tour broken because of HOPSWORKS-587
  • [HOPSWORKS-602] – Update Hopsworks settings in cluster definition breaks flyway migrate
  • [HOPSWORKS-604] – Review OpenMPI and CUDA upgrade
  • [HOPSWORKS-605] – Chef: fix remote path for hops-examples and hops-util
  • [HOPSWORKS-608] – Put back the code for downloading hops-tensorflow.jar
  • [HOPSWORKS-610] – The project logging to ops_log should be done after the project is assigned an inode
  • [HOPSWORKS-614] – [kagent-chef] Be consistent on the separator in config.ini
  • [HOPSWORKS-615] – Kagent-chef upgrade broken
  • [HOPSWORKS-617] – Hopsworks-579 breaks build with -Pcluster profile
  • [HOPSWORKS-619] – nvidia-smi might not be available in hops::install to generate /tmp/num_gpus
  • [HOPSWORKS-620] – Hops cookbook broken for configuration update
  • [HOPSWORKS-624] – Proxy not handling capital letters in url
  • [HOPSWORKS-632] – Regex too strict for yarn quota updates in admin ui
  • [HOPSWORKS-635] – Enable logging for Livy
  • [HOPSWORKS-636] – Only install nccl2 if chef attribute cuda/download_terms/accept is ‘true’
  • [HOPSWORKS-646] – Inconsistent metadata due to parallel data readers
  • [HOPSWORKS-647] – Unshare dataset UI not working
  • [HOPSWORKS-650] – Job-monitor ui broken
  • [HOPSWORKS-652] – [hopslog-chef] Change primary group of yarn user to hadoop
  • [HOPSWORKS-653] – Driver and Executor links broken in Executors tab in Spark UI
  • [HOPSWORKS-660] – Zip/Unzip changes ownership for files in same directory as the unzipped folder or zipped file
  • [HOPSWORKS-663] – Admin UI is broken because of missing methods in Hosts
  • [HOPSWORKS-666] – MMLSpark breaks reproducible environments
  • [HOPSWORKS-668] – ProjectController ignores failures for adding Jupyter dataset and Hive service
  • [HOPSWORKS-670] – Conda search broken on master
  • [HOPSWORKS-675] – Handling of agent heartbeat could result in transactions timing-out
  • [HOPSWORKS-677] – [hops-hadoop-chef] Hopsworks host is not interpolated in yarn-site.xml
  • [HOPSWORKS-681] – Wrong filename for certificates (.cert.pem appended twice)
  • [HOPSWORKS-682] – Installed Python libraries do not change status
  • [HOPSWORKS-684] – TensorBoard broken for single Experiment in JobUI
  • [HOPSWORKS-691] – “Unhealthy HDFS blocks” display badly in admin pannel when running multiple NN
  • [HOPSWORKS-696] – /previewFile in DataSetService does not filter out control codes from JSON response
  • [HOPSWORKS-701] – JobService proxy should not set host config for client
  • [HOPSWORKS-704] – TensorBoard not showing events in Experiments service
  • [HOPSWORKS-707] – Lib status for anaconda envs still is broken
  • [HOPSWORKS-731] – Add additional Distributed Training configuration check to JupyterConfigFilesGenerator
  • [HOPSWORKS-746] – TfLibMapping missing TensorFlow 1.8
  • [HOPSWORKS-753] – Change default location of Unix MySQL socket for Rspec
  • [HOPSWORKS-756] – Jupyter NPE when starting it before Python service is enabled
  • [HOPSWORKS-760] – elastic-chef don’t use the elasticsearch lwrp to create the user
  • [HOPSWORKS-766] – If hopsworks.version is 0.6.0 replace flyway_schema_history table
  • [HOPSWORKS-768] – appservice tests fail randomly
  • [HOPSWORKS-772] – Chef replace provider uses local path
  • [HOPSWORKS-773] – Usernames should not contain any special char
  • [HOPSWORKS-777] – Hopsworks administrator should be able to delete Projects
  • [HOPSWORKS-779] – [ePipe] Fix a race condition while reading from ndb
  • [HOPSWORKS-783] – INFO log severity in UserStatusValidator is polluting the logs
  • [HOPSWORKS-786] – Fix file path for 0.6.0 flyway migration table
  • [HOPSWORKS-794] – flyway schema history should be replaced only if the last migration was 0.5.0
  • [HOPSWORKS-795] – Jupyter does not have a preselected mode set
  • [HOPSWORKS-797] – Swagger cannot generate json
  • [HOPSWORKS-799] – Lowercase project name for TfServing logs
Categories: Release