Systems for processing large amounts of data

Higher education teachers: Kos Andrej

Subject description

Prerequisits:

• Knowledge if information and communication technologies and services.

Content (Syllabus outline):

Data collection: smart phones, sensors and internet-connected devices, web, cleaning and preparation of data, data anonymization and de-identification. Data retention; scalable relational databases, NoSQL databases, understanding the compromise between the consistency of data, performance and availability. Data processing: event-oriented processing, processing parallelization (map-reduce), extraction of structured data from unstructured. Analyses: efficient algorithms for processing and analysis of data, machine learning. Visualization, procedures and challenges of visualizing large amounts of data, other modalities of presentation of data (soundification, etc.). Applications of the presented techniques: systems for context detection, smart systems (applications of smart cities, smart transport, etc.), medical applications, social networks, financial systems

Objectives and competences:

• Is familiar with the concept of "big data". Able to evaluate the amount of data, the rate of events, their diversity, and the key challenges associated with large amounts of data.
• Knows the difference and can choose among relational or NoSQL database, and evaluate the appropriateness of use.
• Knows the strengths and weaknesses of map-reduce model and evaluates it in comparison with relational databases.
• Can apply basic analytical and visualization techniques for working with large amounts of data in a use-case.

Intended learning outcomes:

Understanding the concept of "big data": data volume, events and their diversity, and key challenges associated with large amounts of data. Understanding of relational databases, their capabilities and limitations. Understanding the capabilities, strengths and weaknesses of NoSQL databases. Understanding of map-reducer model, its strengths and weaknesses, as well as a comparison with relational databases. Understanding of basic analytical and visualization techniques for working with large amounts of data.

Learning and teaching methods:

• Lectures or mentoring
• Seminar

Study materials

1. European Commission: http://www.internet-of-things-research.eu/pdf/Converging_Technologies_for_Smart_Environments_and_Integrated_Ecosystems_IERC_Book_Open_Access_2013.pdf
2. Tom White: Hadoop: The Definitive Guide, 3rd Edition; Storage and Analysis at Internet Scale; O'Reilly Media
3. Jure Leskovec, Anand Rajaraman, Jeffrey D. Ullman: Mining of Massive Datasets, http://i.stanford.edu/~ullman/mmds/book.pdf
4. Jimmy Lin, Chris Dyer: Data-Intensive Text Processing with MapReduce, http://lintool.github.io/MapReduceAlgorithms/MapReduce-book-final.pdf
5. Tamara Munzner: Visualization Analysis and Design (2014 Draft) http://www.cs.ubc.ca/~tmm/courses/533/book/vispmp-draft.pdf
6. Scott Murray: Interactive Data Visualization for the Web: An Introduction to Designing with D3, O'Reilly Media