Commit: 99d97bde6db7c8945ce8b535b76b07d3a321c020 - myprojects-hg-reborn (hg)

Commit MetaInfo

• Revision: 99d97bde6db7c8945ce8b535b76b07d3a321c020 (tree)
• Zeit: 2018-04-26 05:38:19
• Autor: Lorenzo Isella <lorenzo.isella@gmai...>
• Commiter: Lorenzo Isella

Log Message

I did some extra work on the presentation.

Ändern Zusammenfassung

• modified: latex-documents/R-intro-code.Rnw

Diff

diff -r b781333e314d -r 99d97bde6db7 latex-documents/R-intro-code.Rnw
--- a/latex-documents/R-intro-code.Rnw	Wed Apr 25 22:37:39 2018 +0200
+++ b/latex-documents/R-intro-code.Rnw	Wed Apr 25 22:38:19 2018 +0200
@@ -132,7 +132,7 @@
 % Compile with  Rscript -e "library(knitr); knit('./R-intro-code.Rnw')"
   
   
-\title{Introduction to R}
+\title{Introduction to R  (with Hands on Applications!)}
 \framesubtitle{A researcher's perspective}
 \author{ {Lorenzo Isella}}
 \institute{DG TRADE, G2, Chief Economist Team} 
@@ -171,7 +171,7 @@
 
 
 \begin{frame}
-\frametitle{Dulcis in Fundo}
+\frametitle{What to Expect from this Training}
 % \framesubtitle{Test Frame}
 %   \subt{An optional subtitle}
 On the other hand, by the end of this training you will
@@ -191,14 +191,15 @@
   
   \frametitle{Overview of the Training}
 \begin{itemize}
-  
- \item We will go through the basics of R (data types, structures,
-   functions, etc...)
- \item we will also cover in some detail some fairly advanced topics
-   and recently added libraries (tidyverse)
-   \item R is evolving rapidly and an introductory training delivered
-     3 years ago would be out of sync with the reality of active users
-     in 2018.
+ \item Philosophy of the training: your goal is to get better,
+   faster and more productive at data analysis.
+   \item you are not interested in the 6 different kinds of atomic
+     vectors in R.
+\item So we will go head over heels on the basics and 
+  \item plunge into the tidyverse. Tidyverse is a collection of tools
+    for powerful and expressive data analysis and visualisation.
+    \item we will barely scratch the surface of many topics, but you
+      will have an idea of the state-of-the art R for data mining.
   
   \end{itemize}
 
@@ -299,7 +300,7 @@
 and computing the average distance (dist = mean(distance, na.rm =
 TRUE)) and arrival delay   (delay = mean(arr{\verb|_|}delay, na.rm = TRUE)). 
   
-<< highlight=T, eval=TRUE,message=F>>=
+<< highlight=T, eval=TRUE,message=F   >>=
 library(nycflights13)
 library(tidyverse)
 by_tailnum <- group_by(flights, tailnum)
@@ -322,7 +323,9 @@
 \item    logical: TRUE, FALSE
 \item     complex: 1+4i (complex numbers with real and imaginary parts)
 \end{itemize}
-You can also create your own, but we will not discuss this in these notes.
+You can also create your own data types and/or, but we will not discuss this
+in these notes. Later on, we will meet the tibbles -- the tidyverse
+reinterpretation of the basic R data frames.
 \end{frame}
 
 \begin{frame}[fragile]
@@ -390,6 +393,7 @@
 \frametitle{Lists 1/3}
 A list generalises the idea of a vector. It can hold items of
 different types. The name tag is optional
+\vspace*{-0.2cm}
 
 << eval=TRUE, highlight=F>>=
 Lst <- list(name="Fred", wife="Mary", 
@@ -496,23 +500,74 @@
 ``\verb|<-|'' to assign values.
 See for instance
 << eval=TRUE, highlight=T>>=
-x <- seq(5)
-x
+x <- c(1,2,3)
 x[2] <- -4
 x
+#and sometimes the puzzling
+y =2
+y= y +7 # new y = old y +7
+y
 @ 
 
 
 \end{frame}
 
+
 \begin{frame}[fragile]
-\frametitle{Functions in R 1/}
+\frametitle{Mutability of Data Structures -- Small Caveat}
+We saw that ``='' can be used to assign a value. Instead, ``==''
+is a \underline{logical} operator that checks if 
+two values/objects are identical. 
+See for instance
+<< eval=TRUE, highlight=T>>=
+x = 2
+x
+x == 2
+x == 3
+@ 
 
 
 \end{frame}
 
 
 \begin{frame}[fragile]
+\frametitle{Functions in R 1/2}
+A function is defined by an assignment of the form 
+<< eval=F, highlight=T  >>=
+name <- function(arg_1, arg_2, …) expression
+@ 
+The expression is an R expression that uses the arguments, arg\verb|_|i, to calculate a value. The value of the expression is the value returned for the function. 
+mean(), sum(), cumsum(), c(), are examples of R in-built functions we have
+already met.
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{Functions in R 2/2}
+Example functions of one and two variables.
+
+<< eval=T, highlight=T  >>=
+
+double <- function(x){ x*2}
+
+double_and_triple <- function(x,y) {c(x*2, y*3) }
+
+a <-7
+b <- 5
+double(a)
+double_and_triple(a,b)
+@ 
+
+\end{frame}
+
+
+% \begin{frame}[fragile]
+% \frametitle{Functions in R 3/2}
+% A technical remark: functions do \underline{not} modify their own arguments
+
+% \end{frame}
+
+
+\begin{frame}[fragile]
 \frametitle{Data Input and Output in R}
 \begin{itemize}
 \item R provides a number of facilities to import external data in different
@@ -521,21 +576,133 @@
   input/output by Excel. For importing and manipulating data, I recommend the
 tidyverse library.
 \end{itemize}
-If you just use base R, you can do
-<< eval=F, highlight=T>>=
-mydata<-read.csv(''filename.csv'')
-@ 
-
-or with the tidyverse library 
-
 << eval=F, highlight=T>>=
 library(tidyverse)
-mydata<-read_csv(''filename.csv'')
+# read data
+mydata<-read_csv("filename.csv")
+# write data
+write_csv(mydata, "my_output_data.csv")
 @ 
 
 \end{frame}
 
 
+
+\begin{frame}[fragile]
+\frametitle{Long Computations in R}
+
+R is a functional language, which means that your code often contains a lot of parenthesis, ( and ). When you have complex code, this often will mean that you will have to nest those parentheses together. This makes your R code hard to read and understand. 
+<< eval=T, highlight=T>>=
+## generate some arbitrary data
+x<-c(1e4, 1.1e4, 2.3e4, 1.8e4,7e4,4.1e4)
+# Compute the logarithm of `x`, return suitably 
+# lagged and iterated differences, 
+# compute the exponential function
+# and round the result
+round(exp(diff(log(x))), 1)
+@ 
+
+\end{frame}
+
+
+% \begin{frame}[fragile]
+% \frametitle{Long Computations in R}
+% Computations can often result in expressions which are hard to read. 
+% << eval=T, highlight=T>>=
+% ## generate some arbitrary data
+% x<-c(1e4, 1.1e4, 2.3e4, 1.8e4,7e4,4.1e4)
+% # Compute the logarithm of `x`, return suitably 
+% # lagged and iterated differences, 
+% # compute the exponential function
+% # and round the result
+% round(exp(diff(log(x))), 1)
+% @ 
+% Wouldn't it be nice to have a way to express these operations which is
+% easy to read and understand?
+
+
+% \end{frame}
+
+\begin{frame}[fragile]
+\frametitle{Enters the Pipe Operator}
+The pipe operator \verb|%>%|  has two fundamental properties
+\begin{enumerate}
+  
+\item  Function   $f(x)$ can be rewritten as $x$ \verb|%>%| $f$
+<< eval=T, highlight=F >>=
+x <- 10
+  # Compute the logarithm of `x` 
+log(x)
+x %>% log()
+@ 
+\item  Function $f(x, y)$ can be rewritten as $x$ \verb|%>%| $f(y)$
+
+<< eval=T, highlight=F >>=
+# Round pi
+round(pi, 6)
+pi %>% round(6)
+@
+  
+  \end{enumerate}
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{Why was This Invented at All?}
+
+The pipe operator \verb|%>%| provides you with a number of benefits
+\begin{enumerate}
+\item    You'll structure the sequence of your data operations from left to right, as apposed to from inside and out;
+\item    You'll avoid nested function calls;
+\item    You'll minimize the need for local variables and function definitions; And
+\item    You'll make it easy to add steps anywhere in the sequence of operations.
+\end{enumerate}
+<< eval=F, highlight=T >>=
+log(sin(sqrt(x))) # becomes
+x %>% sqrt() %>%
+      sin() %>%
+      log() #much easier to follow!
+@
+
+
+\end{frame}
+
+
+\begin{frame}[fragile]
+\frametitle{Application to the Previous Example}
+This sounds very abstract, but let us see \verb|%>%| in action
+<< eval=T, highlight=T>>=
+library(tidyverse)
+x<-c(1e4, 1.1e4, 2.3e4, 1.8e4,7e4,4.1e4)
+x %>% log() %>%
+    diff() %>%
+    exp() %>%
+    round(1)
+@ 
+Now you finally understand what is going on. Cleaner code is easier to
+share and extend.
+
+\end{frame}
+
+
+\begin{frame}[fragile]
+\frametitle{Modify a Sequence of Computations}
+Now that the operations are laid out as a sequence, it is much easier to modify them whenever we need to. For instance
+<< eval=T, highlight=T>>=
+# Compute the logarithm of `x`, return suitably 
+# lagged and iterated differences, 
+# compute the mean
+# and round the result with two digits
+library(tidyverse)
+x %>% log() %>%
+    diff() %>%
+    mean() %>%
+    round(2)
+@ 
+\end{frame}
+
+
+
 \begin{frame}[fragile]
 \frametitle{Tidyverse and R}
 \begin{itemize}
@@ -549,68 +716,254 @@
 
 \end{frame}
 
+
 \begin{frame}[fragile]
-\frametitle{Long Computations in R}
-Computations can often result in expressions which are hard to read. 
-<< eval=T, highlight=T>>=
-## generate some arbitrary data
-x<-c(1e4, 1.1e4, 2.3e4, 1.8e4,7e4,4.1e4)
-# Compute the logarithm of `x`, return suitably 
-# lagged and iterated differences, 
-# compute the exponential function
-# and round the result
-round(exp(diff(log(x))), 1)
+\frametitle{dplyr -- Data Manipulation 1/2}
+dplyr (part of the tidyverse family) is a \underline{grammar of data manipulation}.
+When working with data you must
+\begin{itemize}
+\item    Figure out what you want to do.
+\item    Describe those tasks in the form of a computer program.
+\item    Execute the program.
+\end{itemize}
+The dplyr package makes these steps fast and easy
+\begin{itemize}
+  \item By constraining your options, it helps you think about your data manipulation challenges.
+\item It provides simple “verbs”, functions that correspond to the most common data manipulation tasks, to help you translate your thoughts into code.
+\item It uses efficient backends, so you spend less time waiting for the computer.
+\end{itemize}
+
+    % filter() to select cases based on their values.
+    % arrange() to reorder the cases.
+    % select() and rename() to select variables based on their names.
+    % mutate() and transmute() to add new variables that are functions of existing variables.
+    % summarise() to condense multiple values to a single value.
+    % sample_n() and sample_frac() to take random samples.
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{dplyr -- Data Manipulation 2/2}
+dplyr is a grammar because it provides verbs that help you solve the most common data manipulation challenges:
+\begin{itemize}
+\item    mutate() adds new variables that are functions of existing variables
+\item    select() picks variables based on their names.
+\item    filter() picks cases based on their values.
+\item    summarise() reduces multiple values down to a single summary.
+\item    arrange() changes the ordering of the rows.
+\item  group\verb|_|by() which allows you to perform any operation ``by group''.
+\end{itemize}
+This works beautifully with the pipe operator.
+
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{Example with Balance of Payment Data}
+<< highlight=T, eval=TRUE,message=F, warning=F   >>=
+library(tidyverse)
+df<-read_csv("bop_flow2.csv") %>%
+    {.$Value=as.numeric(.$Value)
+                        .}
 @ 
-Wouldn't it be nice to have a way to express these operations which is
-easy to read and understand?
+
+Let us glimpse at the resulting table (only a few lines are shown)
+
+% df<-read_csv("bop_flow2.csv",col_types = cols(Value = "i"))
+
+
+
+\begin{table}[ht]
+\centering
+\scalebox{0.7}{
+\begin{tabular}{rlllll}
+  \hline
+TIME & GEO & CURRENCY & NACE\_R2 & STK\_FLOW & STK\_FLOW\_LABEL \\ 
+  \hline
+2016 & EU28 & Million euro & TOTAL & ASS & Assets \\ 
+  2016 & EU28 & Million euro & TOTAL & ASS & Assets \\ 
+  2016 & EU28 & Million euro & TOTAL & ASS & Assets \\ 
+   \hline
+\end{tabular}
+}
+\end{table}
+
+\begin{table}[ht]
+\centering
+\scalebox{0.7}{
+\begin{tabular}{rllllr}
+  \hline
+TIME & ENTITY & FDI\_ITEM & FDI\_ITEM\_LABEL & PARTNER & Value \\ 
+  \hline
+2016 & TOTAL & DO\_\_D\_\_F & Direct investment abroad (DIA) & CH & NA \\ 
+  2016 & TOTAL & DO\_\_D\_\_F & Direct investment abroad (DIA) & TR & NA \\ 
+  2016 & TOTAL & DO\_\_D\_\_F & Direct investment abroad (DIA) & RU & NA \\ 
+   \hline
+\end{tabular}
+}
+\end{table}
+
+
+
+
 
 
 \end{frame}
 
 \begin{frame}[fragile]
-\frametitle{Enters the Pipe Operator}
-The pipe operator \verb|%>%| is used to transfer the results of a
-computation in a pipeline.
-<< eval=T, highlight=T>>=
+\frametitle{dplyr Verbs in Action 1/4}
+In 2015, how many million euros did the EU28 (GEO) invest
+(FDI\verb|_|ITEM is  DO\verb|_|\verb|_|D\verb|_|\verb|_|F; ENTITY is TOTAL) in manufacture
+(NACE\verb|_|R2 is C) in Japan (PARTNER is JP) as outward net foreign
+direct investment (STK\verb|_|FLOW is NO)?
+<< highlight=T, eval=TRUE,message=F   >>=
 library(tidyverse)
-x %>% log() %>%
-    diff() %>%
-    exp() %>%
-    round(1)
+manu_JP <- df %>%filter(TIME==2015, GEO=="EU28",
+  STK_FLOW=="NO",FDI_ITEM=="DO__D__F",
+  ENTITY=="TOTAL",PARTNER=="JP", NACE_R2=="C") %>% 
+  select(TIME, GEO, PARTNER, NACE_R2, Value)
+manu_JP
 @ 
-This leads to cleaner code, easier to understand and debug.
+\end{frame}
+
+
+
+\begin{frame}[fragile]
+\frametitle{dplyr Verbs in Action 2/4}
+And the total FDI to the US for all years
+<< highlight=T, eval=TRUE,message=F   >>=
+library(tidyverse)
+FDI_US <- df %>%filter( GEO=="EU28",
+STK_FLOW=="NO",FDI_ITEM=="DO__D__F",
+ENTITY=="TOTAL",PARTNER =="US",NACE_R2=="FDI") %>% 
+select(TIME, GEO, PARTNER, NACE_R2, Value)
+FDI_US
+@ 
+\end{frame}
+
+
+\begin{frame}[fragile]
+\frametitle{dplyr Verbs in Action 3/4}
+And if you want the average FDI to the US along the years
+<< highlight=T, eval=TRUE,message=F   >>=
+library(tidyverse)
+FDI_US_mean <- df %>%filter( GEO=="EU28",
+STK_FLOW=="NO",FDI_ITEM=="DO__D__F",
+ENTITY=="TOTAL",PARTNER =="US", NACE_R2=="FDI")%>% 
+select(TIME, GEO, PARTNER, NACE_R2, Value) %>%
+summarise(mean_FDI_to_US=mean(Value))
+FDI_US_mean
+@ 
+\end{frame}
+
+
+\begin{frame}[fragile]
+\frametitle{dplyr Verbs in Action 4/4}
+Now you want to do the same for US and India in one go
+\vspace*{-0.2cm}
+<< highlight=T, eval=TRUE,message=F   >>=
+library(tidyverse)
+FDI_US_IN <- df %>%filter( GEO=="EU28",
+STK_FLOW=="NO",FDI_ITEM=="DO__D__F",
+ENTITY=="TOTAL",PARTNER %in% c("US", "IN"),
+NACE_R2=="FDI")%>% 
+select(TIME, GEO, PARTNER, NACE_R2, Value) %>%
+group_by(PARTNER) %>%
+summarise(mean_FDI=mean(Value))
+FDI_US_IN
+@ 
+\end{frame}
+
+
+\begin{frame}[fragile]
+\frametitle{dplyr -- Final Thoughts}
+\begin{itemize}
+  \item we barely scratched the surface of dplyr
+  \item but we have already seen filter, selection of columns and
+    computing statistics on groups of variables
+  \item thanks to the pipe operator, most of the code that you write
+    is reusable and readable
+  \item you do not worry about cells, indexes etc..., but you think
+    more about the questions you want to pose to your data.
+\end{itemize}  
+\end{frame}
+
+\begin{frame}[fragile]
+\frametitle{Tidy Data}
+The tidyverse is named after the tidy data format. In tidy data
+\begin{enumerate}
+\item    Each variable forms a column.
+\item    Each observation forms a row.
+\item    Each type of observational unit forms a table.
+\end{enumerate}
+
+Tidy data makes it easy for an analyst or a computer to extract needed
+variables because it provides a standard way of structuring a
+dataset. You do not need different strategies to extract different variables.
+The FDI flow data set was cast in a tidy format.
+
+Every time you have a data set with the year on the horizontal axis,
+you are sure that the data set is messy (not tidy).
 
 \end{frame}
 
 \begin{frame}[fragile]
-\frametitle{Reasons to use the pipe operator}
-
+\frametitle{Tidying Messy Datasets}
+Real data sets are often messy in every conceivable way, e.g.
+\begin{itemize}
+\item    Column headers are values, not variable names.
 
-\begin{enumerate}
-\item    You'll structure the sequence of your data operations from left to right, as apposed to from inside and out;
-\item    You'll avoid nested function calls;
-\item    You'll minimize the need for local variables and function definitions; And
-\item    You'll make it easy to add steps anywhere in the sequence of operations.
-\end{enumerate}
+\item    Multiple variables are stored in one column.
+
+\item    Variables are stored in both rows and columns.
+
+\item    Multiple types of observational units are stored in the same table.
+
+\item    A single observational unit is stored in multiple tables.
+\end{itemize}
+Tidying messy data sets is in itself a large topic; we'll focus only
+on one example in the following. 
 \end{frame}
 
 \begin{frame}[fragile]
-\frametitle{Enters the Pipe Operator Again}
-It is straightforward to modify the previous sequence of operations
-<< eval=T, highlight=T>>=
-# Compute the logarithm of `x`, return suitably 
-# lagged and iterated differences, 
-# compute the mean
-# and round the result with two digits
+\frametitle{Column headers are values, not variable names}
+This is one of the most common cases. See for instance some data about
+income and religion in the US
+<< highlight=T, eval=TRUE,message=F   >>=
 library(tidyverse)
-x %>% log() %>%
-    diff() %>%
-    mean() %>%
-    round(2)
+pew <-read_csv("income_religion.csv")
 @ 
+\begin{table}[ht]
+\centering
+\scalebox{0.7}{
+\begin{tabular}{lrrrrrr}
+  \hline
+religion & $<$\$10k & \$10-20k & \$20-30k & \$30-40k & \$40-50k & \$$>$50k \\ 
+  \hline
+Agnostic &  27 &  34 &  60 &  81 &  76 & 137 \\ 
+  Atheist &  12 &  27 &  37 &  52 &  35 &  70 \\ 
+  Buddhist &  27 &  21 &  30 &  34 &  33 &  58 \\ 
+  Catholic & 418 & 617 & 732 & 670 & 638 & 1116 \\ 
+  Don’t know &  15 &  14 &  15 &  11 &  10 &  35 \\ 
+  Evangelical & 575 & 869 & 1064 & 982 & 881 & 1486 \\ 
+  Hindu &   1 &   9 &   7 &   9 &  11 &  34 \\ 
+  Historically black & 228 & 244 & 236 & 238 & 197 & 223 \\ 
+  Jehovah's withnesses &  20 &  27 &  24 &  24 &  21 &  30 \\ 
+  Jewish &  19 &  19 &  25 &  25 &  30 &  95 \\ 
+   \hline
+\end{tabular}
+}
+\end{table}
+
+This dataset has three variables, religion, income class and frequency. 
+Religion and income class are non-variable columns, whereas the
+frequency is the only value.
 \end{frame}
 
+\begin{frame}[fragile]
+\frametitle{Tidying the Data Set 1/}
+To tidy the pew data set, we need to \underline{gather} the non-variable columns into a two-column key-value pair. 
 
+\end{frame}
 
 \begin{frame}[fragile]
 \frametitle{Linear Models in R}