Home >The Catalogue>Services> SDIL, HPC and GPU Infrastructure for Machine Learning and Big Data Analysis
SERVICES

SDIL, HPC and GPU Infrastructure for Machine Learning and Big Data Analysis

TYPE
Big Data Platforms (PaaS), Infrastructure Access
REGION
Germany
LANGUAGE
English, German

Batch processing systems offer efficient means for highly scalable ML training. Together with the infrastructure, we provide the expertise needed to scale machine learning and data analytics on such systems for your experiment.

SERVICE DESCRIPTION

As one of the largest scientific institutions in Europe and the only German university of excellence with national large-scale research facilities, KIT combines a long university tradition with program-oriented cutting-edge research. KIT's cluster systems are the backbone for many international data and HPC operations.

Within EUHubs4Data, experiments can scale their data analysis and machine learning tasks on a modern research cluster optimized for model training, hyperparameter optimization, and simulation-based machine learning algorithms. The cluster consists of:

  • 520 "Thin" nodes (40 2.1 Ghz Intel cores / >=96 GB memory per node),
  • 352 "HPC Broadwell" nodes (28 2.0 GHz cores / 128 GB memory each),
  • 8 "Fat" nodes (80 cores / 3 TB memory each).

To support machine learning:

  • 4 nodes with 4 NVIDIA V100 and
  • 8 nodes with 8 NVIDIA V100 GPU accelerators are available (more accellerators are currently integrated)

The systems support deployment via SLURM schedulers supporting Docker, Podman, and Singularity containerization. For interactive use, a Jupyter hub environment can be used to access compute nodes. Tight integration with Gitlab-based continuous integration is also supported.

For machine learning use with Tensorflow, Pytorch, and Sklearn, Dask can be used as a parallelization layer. KIT supports porting ML architectures to this cluster system and will provide cluster time for experiments.

SPECIAL ACCESS CONDITIONS

Conditions and requirements for participation in an experiment within the Open Calls:

By participating in an EUHubs4Data Open Call, you are initially only applying for funding that originally comes from the European Commission and is awarded by the coordinator exclusively in its own name under the conclusion of a sub-grant agreement. This sub-grant agreement does not establish a contract with KIT, neither through your application nor through a possible positive funding decision.
KIT will therefore - also in your own interest - conclude a separate written agreement with you at the start of the experiment (based on our sample cooperation agreement). If you decide to propose the participation of KIT and SDIL infrastructure in your experiment, you must respect the following conditions. We provide this information in advance to ensure maximum transparency: please contact us if you have any questions. In the unlikely event that you are unable to conduct your experiment with our participation, we will attempt to assist you in selecting alternative services before the experiment begins.

Please note that contrary to the name "service", the above description is not a genuine commercial offer, but a listing of exclusive contributions as part of a genuine eye-to-eye collaboration.

For genuine commercial offerings related to the above topics, please feel free to contact us any time outside of the Open Calls.

PREREQUISITES

Existing PoC ML architecture with training data (e.g. from a previous task within an experiment)

CASE EXAMPLES

Currently we are working together with a local object detection startup ( https://www.kimoknow.de/ ) to set up an HPC driven data generation and image classification toolchain, that allows detection of realworld objects based on CAD-drawings: more info

LINKS

SERVICE OFFERED BY

MEMBER
SDIL Smart Data Innovation Lab
TYPE
DIH
COUNTRY
Germany

MORE INFORMATION ABOUT THIS SERVICE

Data Controller: INSTITUTO TECNOLÓGICO DE INFORMÁTICA (G96278734)
Purposes and legal basis: We will use your personal data to contact you back and answer your inquiries and provide you with information regarding our activity and in connection with our developments, research and services.
Data recipients: Your personal data will only be shared with the DIH your inquiry or information request may concern.
Rights: Regarding your personal data you have the right to access, rectify, erase, data portability, restrict processing , object, consent withdrawal and to file a complaint before the Supervisory Authority. More info
Exercise of rights: You can exercise the aforementioned rights by sending an e-mail to the e-mail address: dpo@iti.es or by sending a letter to the address Camino de Vera s/n, CPI Edif. 8, Acceso B, 46022 Valencia (Spain).