Is Systematic Data Sharding able to Stabilize Asynchronous Parameter Server Training?

Authors: N. Provatas, I. Konstantinou, N. Koziris

2021 IEEE International Conference on Big Data (Big Data)

Abstract: Over the last years, deep learning has gained an increase in popularity in various domains introducing complex models to handle the data explosion. However, while such model architectures can support the enormous amount of data, a single computing node cannot train the model using the whole data set in a timely fashion. Thus, specialized distributed architectures have been proposed, most of which follow data parallelism schemes, as the widely used parameter server approach. In this setup, each worker contributes to the training process in an asynchronous manner. While asynchronous training does not suffer from synchronization overheads, it introduces the problem of stale gradients which might cause the model to diverge during the training process. In this paper, we examine different data assignment schemes to workers, which facilitate the asynchronous learning approach. Specifically, w e propose two different algorithms to perform the data sharding. Our experimental evaluation indicated that when stratification is taken into account the validation results present up to 6X less variance compared to standard sharding creation. When further data exploration for hidden stratification is performed, validation metrics can be slightly optimized. This method also achieves to reduce the variance of training and validation metrics by up to 8X and 2X respectively.

research/2021_systematic.pdf

Exploiting data distribution in distributed learning of deep classification models under the parameter server architecture.

Authors:N. Provatas

2021 VLDB PhD Workshop

Abstract: Nowadays, deep learning models are used in a wide range of applications, including classification and recognition tasks. The constant growth on the data size has led to the use of more complex model architectures for creating neural network classifiers. Both the model complexity and the amount of data usually prohibit the training on a single machine, due to time and memory limitations. Thus, distributed learning setups have been proposed to train deep networks, when a vast amount of data is available. One such common setup follows the parameter server approach, where worker tasks compute gradients to update the network stored in the servers, often in a synchronous free manner. However, the lack of synchronization may harm the resulting model quality due to the effect of stale gradients, which are computed based on older model versions. In this PhD research, we aim to explore how asynchronous learning could benefit from data preprocessing tasks revealing hidden traits regarding the data distribution.

research/2021_vldb_workshop.pdf

A Performance Evaluation of Distributed Deep Learning Frameworks on CPU Clusters Using Image Classification Workloads

Authors: A. Krisilias, N. Provatas, N. Koziris, I. Konstantinou

2021 IEEE International Conference on Big Data (Big Data)

Abstract: Over the recent years, deep learning is widely being used in a variety of different fields and applications. The constant growth of data used to train complex models, has opened research in the distributed learning. In this domain, two main architectures are used to train models in a distribution fashion, allreduce and parameter server. Both support synchronous learning, while parameter server also supports asynchronous learning. These architectures are adopted by tech companies, which have developed multiple systems for this purpose. Among the most popular and widely used distributed deep learning systems are Google TensorFlow, Facebook PyTorch and Apache MXNet. In this paper, we quantify the performance gap between these systems and present a detailed analysis to discuss the parameters that affect their execution time. Overall, in synchronous learning setups, TensorFlow is slower compared to PyTorch by average 2.65X, while the latter lags MXNet by average 1.38X. Regarding asynchronous learning, MXNet is faster by average 3.22X in respect with TensorFlow.

research/2021_bench_dl.pdf

Selis bda: Big data analytics for the logistics domain

Author: N. Provatas, E. Kassela, N. Chalvantzis, A. Bakogiannis, I. Giannakopoulos, N. Koziris, I. Konstantinou

2020 IEEE International Conference on Big Data (Big Data)

research/2020_selis.pdf

Techniques for exploiting data distribution in neural network training.

Author: N. Provatas

M.Sc. Thesis, 2020

Abstract: Deep learning is particularly flourishing in a context of different applications, such as image classification. In order to train better predictive models a plethora of complex neural networks architectures has been developed. However, network’s complexity alone is not sufficient for the creation of high-precision models, which is directly linked to the quality and the data access pattern during training. A common practise to increase the accuracy of a model is to use a random batch of the data during the training instead of accessing them in a serial manner. In this diploma thesis, we examine the use of data distribution related techniques which will benefit the training process. Initially, we experiment with defining data access pattern according to their statistical distribution and produce validation metrics by up to 4.58%, without a significant time overhead. In addition, how data distribution can be utilized to perform the training process on a specific subset of the training set. Thus, we achieve the creation of equivalent predictive models in 1.2X times less time.

research/diploma_dsml.pdf

Towards faster distributed deep learning using data hashing techniques

Author: N. Provatas, I. Konstantinou, N. Koziris

2019 IEEE International Conference on Big Data (Big Data)

Abstract: Nowadays, deep learning is a crucial part of a variety of big data applications. Both the vast amount of data and the high complexity of the state-of-the-art neural networks have led to perform the network training in a distributed manner accross clusters. Since synchronization overheads are usually fatal for the training's performance, asynchronous training is usually preferred in such cases. However, this training mode is sensitive to conflicting updates. Such updates most commonly occur when the workers train on a totally different part of the data. To reduce this phenomenon, in this paper, we propose the use of hashing schemes when distributing training data across workers.

research/2019_towards_dl_hashing.pdf

General-purpose vs. specialized data analytics systems: A game of ml & sql thrones

Author: E. Kassela, N. Provatas, I. Konstantinou, N. Koziris

2019 IEEE International Conference on Big Data (Big Data)

Abstract: Over the past decade, a plethora of systems have emerged to support data analytics in various domains such as SQL and machine learning, among others. In each of the data analysis domains, there are now many different specialized systems that leverage domain-specific optimizations to efficiently execute their workloads. An alternative approach is to build a general-purpose data analytics system that uses a common execution engine and programming model to support workloads in different domains. In this work, we choose representative systems of each class (Spark, TensorFlow, Presto and Hive) and benchmark their performance on a wide variety of machine learning and SQL workloads. We perform an extensive comparative analysis on the strengths and limitations of each system and highlight major areas for improvement for all systems. We believe that the major insights gained from this study will be useful for developers to improve the performance of these systems

research/2019_gam.pdf

Bigoptibase: Big data analytics for base station energy consumption optimization

Author: E. Kassela, N. Provatas, I. Konstantinou, N. Koziris

2019 IEEE International Conference on Big Data (Big Data)

Abstract: Mobile Network Operators develop new technologies, as the 5G network, to handle the constantly increasing network traffic, while they put less effort on optimizing their operations. However, more energy efficient approaches are essential for cost reduction and compliance with energy footprint principles. Network usage and IoT data produced in the base stations can be used to develop such approaches. Considering the above, the BigOptiBase platform has been designed. Through this platform we will offer a big data analytics subsystem developed to provide elastic energy efficient solutions for the base stations using data analytics and machine learning technologies.

research/2019_bigoptibase.pdf

Deterministic and heuristic algorithms for designing and optimizing workflows on multiple machines on the cloud.

Author: N. Provatas

M.Eng. Thesis, 2016

Abstract: Modern workflows have become extremely large and complex. Each task in the workflow might be able to execute in more than one execution engines. Therefore, a workflow has many different execution plans. From all different execution plans available for the workflow, it is crucial to locate the best plans according to some objectives. In this diploma thesis, a meta-planner is built , which target is to locate the best pareto solutions according to the time and cost objectives. At first, the planner reads the description of the workflow from an XML file and then attempts to find the appropriate execution plans in two ways, using models that describe the time and cost of executing each operator in any engine. The first way uses and optimal algorithm that guarantees the best solutions. The second is the approximate genetic algorithm NSGA-II. The target is the evaluation of the two methods, which showed that the size of the workflow really affects the performance of the two algorithms. Instead, the number of the available execution engines affects mostly the optimal algorithm. Moreover, there are cases that NSGA-II is much faster than the optimal (up to 152 times), which indicates its usage given the small distance of the solution sets. In the end, it is found that using the meta-planner can locate better execution plans according to an objective, instead of executing the whole workflow in the same execution engine.

research/diploma_meng.pdf