Technologies Big Data

---
name: layout-general
layout: true
class: left, middle

.remark-slide-number .progress-bar-container {
  position: absolute;
  bottom: 0;
  height: 4px;
  display: block;
  left: 0;
  right: 0;
}

.remark-slide-number .progress-bar {
  height: 100%;
  background-color: red;
}

/* custom.css */
.plot-callout {
  width: 300px;
  bottom: 5%;
  right: 5%;
  position: absolute;
  padding: 0px;
  z-index: 100;
}
.plot-callout img {
  width: 100%;
  border: 1px solid #23373B;
}
</style>

<div>
<style type="text/css">.xaringan-extra-logo {
width: 110px;
height: 128px;
z-index: 0;
background-image: url(./figs/UniversiteParisCite_logo_horizontal_couleur_RVB.jpeg);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:1em;
}
</style>
<script>(function () {
  let tries = 0
  function addLogo () {
    if (typeof slideshow === 'undefined') {
      tries += 1
      if (tries < 10) {
        setTimeout(addLogo, 100)
      }
    } else {
      document.querySelectorAll('.remark-slide-content:not(.hide_logo)')
        .forEach(function (slide) {
          const logo = document.createElement('a')
          logo.classList = 'xaringan-extra-logo'
          logo.href = 'http://master.math.univ-paris-diderot.fr/annee/m1-isifar/'
          slide.appendChild(logo)
        })
    }
  }
  document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

---

# Technologies Big Data : Introduction

### 2023-01-17

#### [Master I MIDS Master I Informatique]()

#### [Technologies Big Data](http://stephane-v-boucheron.fr/courses/isidata/)

#### [Amélie Gheerbrandt, Stéphane Gaïffas, Stéphane Boucheron](http://stephane-v-boucheron.fr)

---
exclude: true

# Big Data Technologies

## Master Mathematics and Informatics

.center[
    <img src="figs/lpsm.pnog" style="height: 160px;" />
    <img src="" style="width: 30px;" />
    <img src="figs/paris-diderot.png" style="height: 90px;" />
    <img src="" style="width: 30px;" />
    <img src="figs/uparis.png" style="height: 120px;" />
]

---

---
template: inter-slide

## Course logistics

---

### Who are we ?

.fl.w-50.pa2[

- Stéphane Boucheron
- LPSM
- Statistics <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M64 400C64 408.8 71.16 416 80 416H480C497.7 416 512 430.3 512 448C512 465.7 497.7 480 480 480H80C35.82 480 0 444.2 0 400V64C0 46.33 14.33 32 32 32C49.67 32 64 46.33 64 64V400zM128 320V236C128 228.3 130.8 220.8 135.9 214.1L215.3 124.2C228.3 109.4 251.4 109.7 263.1 124.8L303.2 171.8C312.2 182.7 328.6 183.4 338.6 173.4L359.6 152.4C372.7 139.3 394.4 140.1 406.5 154.2L472.3 231C477.3 236.8 480 244.2 480 251.8V320C480 337.7 465.7 352 448 352H159.1C142.3 352 127.1 337.7 127.1 320L128 320z"/></svg>
- [https://stephane-v-boucheron.fr](https://stephane-v-boucheron.fr)

]

.fl.w-50.pa2[

- Amélie Gheerbrant
- IRIF
- Data Science, Databases <svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M448 80V128C448 172.2 347.7 208 224 208C100.3 208 0 172.2 0 128V80C0 35.82 100.3 0 224 0C347.7 0 448 35.82 448 80zM393.2 214.7C413.1 207.3 433.1 197.8 448 186.1V288C448 332.2 347.7 368 224 368C100.3 368 0 332.2 0 288V186.1C14.93 197.8 34.02 207.3 54.85 214.7C99.66 230.7 159.5 240 224 240C288.5 240 348.3 230.7 393.2 214.7V214.7zM54.85 374.7C99.66 390.7 159.5 400 224 400C288.5 400 348.3 390.7 393.2 374.7C413.1 367.3 433.1 357.8 448 346.1V432C448 476.2 347.7 512 224 512C100.3 512 0 476.2 0 432V346.1C14.93 357.8 34.02 367.3 54.85 374.7z"/></svg>
- [https://www.irif.fr/~amelie/](https://www.irif.fr/~amelie/)

]

---

### Course logistics

- 24 hours = 2 hours `$\times$` .stress[12 weeks] : classes + hands-on

- Tuesdays, 10:30 - 12:30

#### About the hands-on

- Hands-on and homeworks using .stress[`Jupyter` notebooks]

- Using a `Docker` image  built for the course

- <svg aria-hidden="true" role="img" viewBox="0 0 640 512" style="height:1em;width:1.25em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M128 96h384v256h64V80C576 53.63 554.4 32 528 32h-416C85.63 32 64 53.63 64 80V352h64V96zM624 384h-608C7.25 384 0 391.3 0 400V416c0 35.25 28.75 64 64 64h512c35.25 0 64-28.75 64-64v-16C640 391.3 632.8 384 624 384zM365.9 286.2C369.8 290.1 374.9 292 380 292s10.23-1.938 14.14-5.844l48-48c7.812-7.813 7.812-20.5 0-28.31l-48-48c-7.812-7.813-20.47-7.813-28.28 0c-7.812 7.813-7.812 20.5 0 28.31l33.86 33.84l-33.86 33.84C358 265.7 358 278.4 365.9 286.2zM274.1 161.9c-7.812-7.813-20.47-7.813-28.28 0l-48 48c-7.812 7.813-7.812 20.5 0 28.31l48 48C249.8 290.1 254.9 292 260 292s10.23-1.938 14.14-5.844c7.812-7.813 7.812-20.5 0-28.31L240.3 224l33.86-33.84C281.1 182.4 281.1 169.7 274.1 161.9z"/></svg> Hands-on must be carried out using your .stress[own laptop]. Bring it at **all the courses**

---

### Course logistics

- The .stress[<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M352 256C352 278.2 350.8 299.6 348.7 320H163.3C161.2 299.6 159.1 278.2 159.1 256C159.1 233.8 161.2 212.4 163.3 192H348.7C350.8 212.4 352 233.8 352 256zM503.9 192C509.2 212.5 512 233.9 512 256C512 278.1 509.2 299.5 503.9 320H380.8C382.9 299.4 384 277.1 384 256C384 234 382.9 212.6 380.8 192H503.9zM493.4 160H376.7C366.7 96.14 346.9 42.62 321.4 8.442C399.8 29.09 463.4 85.94 493.4 160zM344.3 160H167.7C173.8 123.6 183.2 91.38 194.7 65.35C205.2 41.74 216.9 24.61 228.2 13.81C239.4 3.178 248.7 0 256 0C263.3 0 272.6 3.178 283.8 13.81C295.1 24.61 306.8 41.74 317.3 65.35C328.8 91.38 338.2 123.6 344.3 160H344.3zM18.61 160C48.59 85.94 112.2 29.09 190.6 8.442C165.1 42.62 145.3 96.14 135.3 160H18.61zM131.2 192C129.1 212.6 127.1 234 127.1 256C127.1 277.1 129.1 299.4 131.2 320H8.065C2.8 299.5 0 278.1 0 256C0 233.9 2.8 212.5 8.065 192H131.2zM194.7 446.6C183.2 420.6 173.8 388.4 167.7 352H344.3C338.2 388.4 328.8 420.6 317.3 446.6C306.8 470.3 295.1 487.4 283.8 498.2C272.6 508.8 263.3 512 255.1 512C248.7 512 239.4 508.8 228.2 498.2C216.9 487.4 205.2 470.3 194.7 446.6H194.7zM190.6 503.6C112.2 482.9 48.59 426.1 18.61 352H135.3C145.3 415.9 165.1 469.4 190.6 503.6V503.6zM321.4 503.6C346.9 469.4 366.7 415.9 376.7 352H493.4C463.4 426.1 399.8 482.9 321.4 503.6V503.6z"/></svg> webpage] of the course is:

.center[[https://stephane-v-boucheron.fr/courses/grosses-data/](https://stephane-v-boucheron.fr/courses/grosses-data/)]

- .stress[<svg aria-hidden="true" role="img" viewBox="0 0 384 512" style="height:1em;width:0.75em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M384 48V512l-192-112L0 512V48C0 21.5 21.5 0 48 0h288C362.5 0 384 21.5 384 48z"/></svg> Bookmark it] !

- Follow .stress[carefully] the steps described in the `tools` page:

.center[[https://stephanegaiffas.github.io/big_data_course/tools](https://stephanegaiffas.github.io/big_data_course/tools)]

- <svg aria-hidden="true" role="img" viewBox="0 0 640 512" style="height:1em;width:1.25em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M349.9 236.3h-66.1v-59.4h66.1v59.4zm0-204.3h-66.1v60.7h66.1V32zm78.2 144.8H362v59.4h66.1v-59.4zm-156.3-72.1h-66.1v60.1h66.1v-60.1zm78.1 0h-66.1v60.1h66.1v-60.1zm276.8 100c-14.4-9.7-47.6-13.2-73.1-8.4-3.3-24-16.7-44.9-41.1-63.7l-14-9.3-9.3 14c-18.4 27.8-23.4 73.6-3.7 103.8-8.7 4.7-25.8 11.1-48.4 10.7H2.4c-8.7 50.8 5.8 116.8 44 162.1 37.1 43.9 92.7 66.2 165.4 66.2 157.4 0 273.9-72.5 328.4-204.2 21.4 .4 67.6 .1 91.3-45.2 1.5-2.5 6.6-13.2 8.5-17.1l-13.3-8.9zm-511.1-27.9h-66v59.4h66.1v-59.4zm78.1 0h-66.1v59.4h66.1v-59.4zm78.1 0h-66.1v59.4h66.1v-59.4zm-78.1-72.1h-66.1v60.1h66.1v-60.1z"/></svg> Who knows about `docker` ?

---

### Course evaluation

- .stress[Evaluation] using **homeworks** and a **final project**

- Find a .stress[friend] : all work done by **pairs of students**

- **All your work** goes in your private repository and nowhere else: .stress[no emails] !

- All your homework will be using .stress[`jupyter` notebooks] or .stress[`quarto`] files

---
template: inter-slide

## <svg aria-hidden="true" role="img" viewBox="0 0 640 512" style="height:1em;width:1.25em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M349.9 236.3h-66.1v-59.4h66.1v59.4zm0-204.3h-66.1v60.7h66.1V32zm78.2 144.8H362v59.4h66.1v-59.4zm-156.3-72.1h-66.1v60.1h66.1v-60.1zm78.1 0h-66.1v60.1h66.1v-60.1zm276.8 100c-14.4-9.7-47.6-13.2-73.1-8.4-3.3-24-16.7-44.9-41.1-63.7l-14-9.3-9.3 14c-18.4 27.8-23.4 73.6-3.7 103.8-8.7 4.7-25.8 11.1-48.4 10.7H2.4c-8.7 50.8 5.8 116.8 44 162.1 37.1 43.9 92.7 66.2 165.4 66.2 157.4 0 273.9-72.5 328.4-204.2 21.4 .4 67.6 .1 91.3-45.2 1.5-2.5 6.6-13.2 8.5-17.1l-13.3-8.9zm-511.1-27.9h-66v59.4h66.1v-59.4zm78.1 0h-66.1v59.4h66.1v-59.4zm78.1 0h-66.1v59.4h66.1v-59.4zm-78.1-72.1h-66.1v60.1h66.1v-60.1z"/></svg> `Docker`

---

### Why `docker` ? What is it ?

- Don't mess with your `python` env. and configuration files
- Everything in embedded in a .stress[container] (better than a VM)
- A .stress[container] is an **instance** of an .stress[image]
- Same image = same environment for everybody 
- Same image = no {version, dependencies, install} problems
- It is an .stress[industrial standard] used everywhere now!
.pull-left[
<img src="figs/containers.png" style="width: 70%;" />
]
.pull-right[
<img src="figs/python_environment.png" style="width: 75%;" />
]

---

### `docker`

- Have a look at

.center[[https://stephanegaiffas.github.io/big_data_course/tools](https://stephanegaiffas.github.io/big_data_course/tools)]

- have a look at the `Dockerfile` to explain a little bit how the image is built

- perform a quick demo on how to use the `docker` image

<br>

#### And that's it for the logistics !

---

## Big data

---

### Big data

- .stress[Moore's Law]: computing power **doubled** every two years from 1975 to 2012

- Nowadays, **less** than two years and a half

- .stress[Rapid growth of datasets]: **internet activity**, social networks, genomics, physics, censor networks, etc.

- .stress[Data size trends]: **doubles every year** according to [IDC executive summary](https://www.emc.com/leadership/digital-universe/2014iview/executive-summary.htm)

- .stress[Now, data grows faster than Moore's law]

### Question

- How do we **scale** to **process it** and to **learn from it** ?

---

### Let's recall some units

#### bit

A bit is a value of either a 1 or 0 (on or off)

#### byte (B)

A byte is 8 bits

- 1 character, e.g., "a", is one byte

#### Kilobyte (KB)

A kilobyte is **1 024** B

- **2** or **3** paragraphs of text

---

### Let's recall some units

#### Megabyte (MB)

A megabyte is **1 048 576** B or **1 024** KB

- **873** pages of plain text
- **4** books (200 pages or 240 000 characters)

#### Gigabyte (GB)

A gigabyte is **1 073 741 824** B, **1 024** MB or **1 048 576** KB

- **894 784** pages of plain text (1 200 characters)
- **4 473** books (200 pages or 240 000 characters)
- **640** web pages (with 1.6 MB average file size)
- **341** digital pictures (with 3 MB average file size)
- **256** MP3 audio files (with 4 MB average file size)
- **1,5** 650 MB CD

---

### Let's recall some units

#### Terabyte (TB)

A terabyte is **1 099 511 627 776** B, **1 024** GB  or **1 048 576** MB.

- **916 259 689** pages of plain text (1 200 characters)
- **4 581 298** books (200 pages or 240 000 characters)
- **655 360** web pages (with 1.6 MB average file size)
- **349 525** digital pictures (with 3 MB average file size)
- **262 144** MP3 audio files (with 4 MB average file size)
- **1 613** 650 MB CD's
- **233** 4.38 GB DVDs
- **40** 25 GB Blu-ray discs

---

### Let's recall some units

#### Petabyte (PB)

A petabyte is **1 024** TB, **1 048 576** GB or **1 073 741 824** MB

- **938 249 922 368** pages of plain text (1 200 characters)
- **4 691 249 611** books (200 pages or 240 000 characters)
- **671 088 640** web pages (with 1.6 MB average file size)
- **357 913 941** digital pictures (with 3 MB average file size)
- **268 435 456** MP3 audio files (with 4 MB average file size)
- **1 651 910** 650 MB CD's
- **239 400** 4.38 GB DVDs
- **41 943** 25 GB Blu-ray discs

#### Exabyte, etc.

- 1 EB = 1 exabyte = 1 024 PB
- 1 ZB = 1 zettabyte = 1 024 EB

---

### Some figures

You have every .stress[single second]$^1$:

- At least **8,000 tweets** sent

- **900+ photos** posted on **Instagram**

- **Thousands of Skype calls** made

- Over **70,000 Google searches** performed

- Around **80,000 YouTube videos** viewed

- Over **2 million emails** sent

---

### Some figures

There are$^1$:

- .stress[5 billion web pages] as of mid-2019 (indexed web)

and we expect$^2$ :

- .stress[4.8 ZB] of annual IP traffic in 2022

Note that

- **1** ZB `$\approx$` **36 000** years of HD video
- Netflix's **entire catalog** is `$\approx$` **3.5 years** of HD video

.footnote[
`$^1$`[https://www.worldwidewebsize.com](https://www.worldwidewebsize.com) <br>
`$^2$`Cisco's Visual Networking Index
]

---

### Some figures

More figures :

- **facebook** daily logs: **60TB**

- **1000 genomes** project: **200TB**

- Google web index: **10+ PB**

- Cost of **1TB** of storage: **~$35**

- Time to read **1TB** from disk: **3 hours** if **100MB/s**

### Let's give some .stress[latencies] now

---

### Latency numbers

.f6[.pure-table.pure-table-striped[
| Memory type                        | Latency(ns)      | Latency(us) | (ms)   |                             |
| :--------------------------------- | ---------------: | ----------: | -----: | :-------------------------- |
| L1 cache reference                 |           0.5 ns |             |        |                             |  
| L2 cache reference                 |           7   ns |             |        | 14x L1 cache                |
| Main memory reference              |         100   ns |             |        | 20x L2, 200x L1             |
| Compress 1K bytes with Zippy       |       3,000   ns |        3 us |        |                             |
| Send 1K bytes over 1 Gbps network  |      10,000   ns |       10 us |        |                             |
| Read 4K randomly from SSD*         |     150,000   ns |      150 us |        | ~1GB/sec SSD                |
| Read 1 MB sequentially from memory |     250,000   ns |      250 us |        |                             |
| Round trip within same datacenter  |     500,000   ns |      500 us |        |                             |
| Read 1 MB sequentially from SSD*   |   1,000,000   ns |    1,000 us |   1 ms | ~1GB/sec SSD, 4X memory     |
| Disk seek                          |  10,000,000   ns |   10,000 us |  10 ms | 20x datacenter roundtrip    |
| Read 1 MB sequentially from disk   |  20,000,000   ns |   20,000 us |  20 ms | 80x memory, 20x SSD         |
| Send packet US -> Europe -> US     | 150,000,000   ns |  150,000 us | 150 ms | 600x memory                 |
]]

---

### Latency numbers

- Reading 1MB from **disk** = **100X** reading 1MB from **memory**

- Sending packet from **US to Europe to US** = **1 000 000X** main memory reference

#### General tendency

True in general, not always:

- memory operations : .stress[fastest]

- disk operations : .stress[slow]

- network operations : .stress[slowest]

---

### Latency numbers

.small[[https://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html](https://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html)]

---

### Humanized latency numbers

Lets multiply all these durations by a billion

.f6[.pure-table.pure-table-striped[
| Memory type                        | Latency      | Human duration                                        |
| :--------------------------------- | -----------: | ----------------------------------------------------: |
| L1 cache reference                 | 0.5 s        | One heart beat (0.5 s)                                |
| L2 cache reference                 | 7 s          | Long yawn                                             | 
| Main memory reference              | 100 s        | Brushing your teeth                                   |
| Send 2K bytes over 1 Gbps network  | 5.5 hr       | From lunch to end of work day                         | 
| SSD random read                    | 1.7 days     | A normal weekend                                      | 
| Read 1 MB sequentially from memory | 2.9 days     | A long weekend                                        |
| Round trip within same datacenter  | 5.8 days     | A medium vacation                                     | 
| Read 1 MB sequentially from SSD    | 11.6 days    | Waiting for almost 2 weeks for a delivery             |
| Disk seek                          | 16.5 weeks   | A semester in university                              |
| Read 1 MB sequentially from disk   | 7.8 months   | Almost producing a new human being                    | 
| Send packet US -> Europe -> US     | 4.8 years    | Average time it takes to complete a bachelor's degree | 
]]

---
template: inter-slide

## Challenges

---

### Challenges with big datasets

- Large data .stress[don't fit] on a **single** hard-drive

- **One** large machine .stress[can't process or store] **all** the data

- For **computations** how do we .stress[stream data] from the **disk to the different 
layers of memory** ?

- **Concurrent accesses** to the data: disks .stress[cannot] be **read in parallel**

---

### Solutions

- Combine .stress[several machines] containing **hard drives** and **processors** on a **network**

- Using .stress[commodity hardware]: cheap, common architecture i.e. **processor** + **RAM** + **disk**

- .stress[Scalability] = **more machines** on the network

- .stress[Partition] the data across the machines

---

### Challenges

Dealing with distributed computations adds **software complexity**

- .stress[Scheduling]: How to **split the work across machines**? Must exploit and optimize data locality since moving data is very expensive

- .stress[Reliability]: How to **deal with failure**? Commodity (cheap) hardware fails more often. @Google [1%, 5%] HD failure/year and 0.2% [DIMM](https://en.wikipedia.org/wiki/DIMM) failure/year

- .stress[Uneven performance] of the machines: some nodes are slower than others

---

### Solutions

- .stress[Schedule], **manage** and **coordinate** threads and resources using appropriate software

- .stress[Locks] to **limit** access to resources

- .stress[Replicate] data for **faster reading** and **reliability**

---

### Is it HPC ?

- **High Performance Computing** (HPC)

- **Parallel computing**

#### Comments

- For HPC, scaling-up means using a .stress[bigger machine]

- Huge performance increase for **medium** scale problems

- .stress[Very expensive], specialized machines, lots of processors and memory

#### Answer is no !

---

### The Big Data universe

Many technologies combining .stress[software] and .stress[cloud computing]

---

### The Big Data universe

Often used with/for with .stress[Machine Learning] (or AI)

---

### Tools

- Softwares such as .stress[`Spark`] or .stress[`HadoopMR`] (Hadoop Map Reduce) are in charge of these challenges
- They are .stress[distributed compute engines]: softwares that ease the development of distributed algorithms

They run on .stress[clusters] (several machine on a network), managed by a .stress[resource manager] such as :
- **`Yarn` :** 
[https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
- **`Mesos` :** [http://mesos.apache.org](http://mesos.apache.org)
- **`Kubernetes :` **[https://kubernetes.io](https://kubernetes.io/)

A resource manager ensures that the tasks running on the cluster do not try to use the same resources all at once

---

## `Apache Spark`

---

### Apache `Spark`

The course will focus mainly on .stress[`Spark`] for big data processing

[https://spark.apache.org](https://spark.apache.org)
]

- `Spark` is an .stress[industrial standard] <br>
(cf [https://spark.apache.org/powered-by.html](https://spark.apache.org/powered-by.html))
- One of the most used .stress[big data processing framework]
- .stress[Open source]

The predecessor of `Spark` is `Hadoop`

---

### `Hadoop`

- `Hadoop` has a simple API and good fault tolerance (tolerance to nodes failing midway through a processing job)

- The cost is lots of .stress[data shuffling] across the network

- With intermediate computations .stress[written to disk] **over the network** which we know is .stress[very time expensive]

It is made of three components:

- .stress[`HDFS`] (Highly Distributed File System) inspired from `GoogleFileSystem`, see 
.small[[https://ai.google/research/pubs/pub51](https://ai.google/research/pubs/pub51)]

- .stress[`YARN`] (Yet Another Ressource Negociator)

- .stress[`MapReduce`] inspired from Google <br> .small[[https://research.google.com/archive/mapreduce.html](https://research.google.com/archive/mapreduce.html)]

---

### MapReduce's wordcount example

---

### `Spark`

Advantages of `Spark` over `HadoopMR` ?

- .stress[In-memory storage]: use **RAM** for fast iterative computations
- .stress[Lower overhead] for starting jobs
- .stress[Simple and expressive] with `Scala`, `Python`, `R`, `Java` APIs
- .stress[Higher level libraries] with `SparkSQL`, `SparkStreaming`, etc.

Disadvantages of `Spark` over `HadoopMR` ?
 
- `Spark` requires servers with **more CPU** and **more memory**
- But still much cheaper than HPC

`Spark` is .stress[much faster] than `Hadoop`

- `Hadoop` uses **disk** and **network** 
- `Spark` tries to use **memory** as much as possible for operations while minimizing network use

---

### `Spark` and `Hadoop` comparison

<br>

.pure-table.pure-table-striped[
|                          | HadoopMR     | Spark                           |
| -----------------------: | -----------: | ------------------------------: |
| storage                  | Disk         | in-memory or disk               |
| operations               | Map, reduce  | Map, reduce, join, sample, among many others    |
| execution model          | Batch        | Batch, interactive, streaming   |
| Programming environments | Java         | Scala, Java, Python, R          |
]

---

### `Spark` and `Hadoop` comparison

For **logistic regression** training (a simple **classification** algorithm which requires **several passes** on a dataset)

.center[
  <img src="figs/spark-dev3.png" width=50%/>
]
<br>
.center[
  <img src="figs/logistic-regression.png" width=30%/>
]

---

### The `Spark` stack

---

### The `Spark` stack

---

### `Spark` can run "everywhere"

---
template: inter-slide

## Agenda, tools and references

---

### Very tentative agenda for the course

**Weeks 1, 2 and 3** <br> 
The .stress[`Python` data-science stack] for **medium-scale** problems

**Weeks 4 and 5** <br>
Introduction to .stress[`spark`] and its .stress[low-level API]

**Weeks 6, 7 and 8** <br>
`Spark`'s high level API: .stress[`spark.sql`]. Data from different formats and sources

**Week 9** <br>
Run a job on a cluster with .stress[`spark-submit`], monitoring, mistakes and debugging

**Weeks 10, 11, 12** <br>
Introduction to .stress[`spark-streaming`] and a glimpse on other big data technologies (Dask)

---

### Main tools for the course (tentative...)

#### Infrastructure

#### Python stack

.center[
<img src="figs/python.png" width=20%/>
<img src="" width=5%/>
<img src="figs/numpy.jpg" width=18%/>
<img src="" width=5%/>
<img src="figs/pandas.png" width=28%/>
<img src="" width=5%/>
<img src="figs/jupyter_logo.png" width=7%/>
]

#### Data Visualization

.center[
<img src="figs/matplotlib.png" width=20%/>
<img src="" width=5%/>
<img src="figs/seaborn.png" width=20%/>
<img src="" width=5%/>
<img src="figs/bokeh.png" width=20%/>
<img src="" width=5%/>
<img src="figs/plotly-logo.png" width=20%/>
]

---

### Main tools for the course (tentative...)

#### Big data processing

.center[
<img src="figs/spark.png" width=20%/>
<img src="" width=10%/>
<img src="figs/pyspark.jpg" width=20%/>
<img src="" width=10%/>
<img src="figs/dask.png" width=10%/>
]

#### Data storage / formats / querying

.center[
<img src="figs/sql.jpg" width=20%/>
<img src="" width=5%/>
<img src="figs/orc.png" width=20%/>
<img src="" width=5%/>
<img src="figs/parquet.png" width=30%/>

<img src="figs/json.png" width=20%/>
<img src="" width=15%/>
<img src="figs/hdfs.png" width=25%/>
]

---

### Learning resources

- .stress[Spark Documentation Website]  <br>
.small[[http://spark.apache.org/docs/latest/](http://spark.apache.org/docs/latest/)]

- .stress[API docs] <br>
.small[[http://spark.apache.org/docs/latest/api/scala/index.html](http://spark.apache.org/docs/latest/api/scala/index.html)] <br>
.small[[http://spark.apache.org/docs/latest/api/python/](http://spark.apache.org/docs/latest/api/python/)]

- .stress[`Databricks` learning notebooks] <br>
.small[[https://databricks.com/resources](https://databricks.com/resources)]

- .stress[StackOverflow] <br>
.small[[https://stackoverflow.com/tags/apache-spark](https://stackoverflow.com/tags/apache-spark)]  <br>
.small[[https://stackoverflow.com/tags/pyspark](https://stackoverflow.com/tags/pyspark)]

- .stress[More advanced] <br>
.small[[http://books.japila.pl/apache-spark-internals/](http://books.japila.pl/apache-spark-internals/)]

- .stress[Misc.] <br>
.small[[Next Generation Databases: NoSQLand Big Data by Guy Harrison](https://link.springer.com/book/10.1007/978-1-4842-1329-2)]<br>
.small[[Data Pipelines Pocket Reference by J. Densmore](https://www.oreilly.com/library/view/data-pipelines-pocket/9781492087823/)]

---

### Learning Resources

.pull-left-80[
- .stress[Book]: **"Spark The Definitive Guide"** 
  .small[[http://shop.oreilly.com/product/0636920034957.do](http://shop.oreilly.com/product/0636920034957.do)] <br>
  .tiny[[https://github.com/databricks/Spark-The-Definitive-Guide](https://github.com/databricks/Spark-The-Definitive-Guide)]
]

And the **most important thing is:**

.pull-left[
.stress[.large[Practice!]]
]
.pull-right[
  <img src="figs/wtf.jpg" style="height: 200px;" />
]

---

template:inter-slide

# Data centers

---

### Data centers

Wonder what a .stress[datacenter looks like] ?

- Have a look at [http://www.google.com/about/datacenters](http://www.google.com/about/datacenters)

---

### Data centers

Wonder what a .stress[datacenter looks like] ?

---

### Data centers

Wonder what a .stress[datacenter looks like] ?

<br>

.center[
<iframe width="672" height="378" src="https://www.youtube.com/embed/avP5d16wEp0" 
        frameborder="0" allowfullscreen>
</iframe>
]

---

# Thank you !