1 Mixed Models

  • linear regressions only has fixed-effects [continuous] or fixed-factors [groups]: variables of interest (often your manipulation)
  • The only random factor assumed is your subject’s response
  • What if you have other effect or factors that might add random variation?
  • Random-effects/factors: Levels randomly sampled from a much larger population: Something you want to generalize across: Subjects, Words, Pictures, etc

1.1 Type of Mixed Models

  • We will explore crossed and designed designs
  • Today we will fully crossed, next class we will do crossed-nested designs

1.1.1 Crossed designs

  • Repeated measures: everyone does all the conditions
  • Longitudinal data: People are tracked over time
  • Growth Curve Models: Fitting functions (like polynomials) to behavior over time
  • Problem: Each person’s response is going to inter-correlated with their other responses

1.1.2 Nested designs

  • Multilevel/Hierarchical designs: Students nested in classrooms [Clustered data]
  • Problem: Clustering means each student’s scores within each classroom might be correlated because they all have the same teacher (for example)

1.1.3 Crossed & Nested designs

  • Longitudinal Multilevel Designs: Students nested in classrooms measured over many months
  • Partially Nested design: Different groups are measures over time
  • Now each person’s response is going to inter-correlated with their other responses, but that correlation might differ by which group they belong too

1.1.3.1 Inter-Correlations

  • All these correlations inflate your type I error!
  • Your standard errors will shrink because you are measuring people multiple

2 Practice data to make sense of random effects

  • Hypothetical Experiment, n = 6
  • DV: % of how happy they are
  • IV1: Time of measurement: 5 time windows
  • IV2: Social Setting: Alone vs With a group

2.1 As 2x5 RM ANOVA

  • Treats Time-Steps as a factor
  • Are there any differences between any time-steps?
  • Not slopes (predictable changes in time)
  • Treats Social as a factor
  • Is there a difference between Alone and With Group
  • Questions: Is there an effect of time and Social?
  • download files need lecture
library(lme4)     #mixed model package by Douglas Bates
library(afex)     #easy ANOVA package 
library(ggplot2)  #GGplot package for visualizing data
source("RegressionClass/L12_Mixed/HelperFunctions.R") # some custom functions
source("RegressionClass/L12_Mixed/RsquaredHelper.R") # some custom functions
###Read in Data file
HappyData <- read.table("RegressionClass/L12_Mixed/HappyStudy.csv", sep=",", header=T)
### Label variables
HappyData$Social <- factor(HappyData$SocialDummy,
                        levels = c(0,1),
                        labels = c("Alone", "Group"))

HappyData$Subject<-as.factor(HappyData$Subject)
##Plot raw data
hist(HappyData$HappyPercent)

  • We will transform using Fisher’s r-to-Z to normalize the data (but only because of how I created this data set for this analysis work in specific ways for this lecture)
#### Normalize DV!
HappyData$HappyPercentZ<-FisherRtoZ(HappyData$HappyPercent)
hist(HappyData$HappyPercentZ)

  • Let’s plot each person’s data (spaghetti plot)
theme_set(theme_bw(base_size = 12, base_family = "")) 

pab <-ggplot(data = HappyData, aes(x = TimeStep, y=HappyPercentZ, group=Subject))+
  facet_grid( ~ Social, labeller=label_both)+
  #  coord_cartesian(ylim = c(.03,.074))+ 
  geom_line(aes(color=Subject))+
  xlab("Time Step")+ylab("Happiness")
pab

  • Let’s plot each person’s data with the means plotted
##custom function to generate means and SE form any long format dataset. 
HappyDataPlot2<-summarySEwithin(HappyData,measurevar="HappyPercentZ", 
                               withinvars=c("Social","TimeStep"),
                               idvar="Subject", na.rm=FALSE, conf.interval=.95)
pab2 <-ggplot()+
  geom_line(data = HappyData, aes(x = as.factor(TimeStep), y =HappyPercentZ, group=Subject, color=Subject))+
  facet_grid(. ~ Social, labeller=label_both)+
  #  coord_cartesian(ylim = c(.03,.074))+ 
  geom_point(data = HappyDataPlot2, aes(x = as.factor(TimeStep), y =HappyPercentZ, group=Social))+
  geom_errorbar(data = HappyDataPlot2,aes(x = as.factor(TimeStep), y =HappyPercentZ, group=Social, ymin=HappyPercentZ-se, ymax=HappyPercentZ+se),colour="#6E6E6E",size=.25, width=.25)+
  xlab("Time Step")+ylab("Happiness")
pab2

  • Let’s examine an RM ANOVA results of this data
  • Remember time is treated like a factor (not a slope)
  • [note: I ran the ANOVA like a GLM just so we can extract an overall \(R^2\) and get a fitted results for each subject]
#######Anova (using type 3 sum of squares same as default in SPSS/SAS)
Anova.Model1 <-aov_car(HappyPercentZ ~ Social*TimeStep + Error(Subject/(Social*TimeStep)), 
                               data = HappyData, anova_table = list(correction = "none", MSE = FALSE))
Anova.Model1
## Anova Table (Type 3 tests)
## 
## Response: HappyPercentZ
##            Effect    df        F  ges p.value
## 1          Social  1, 5     2.21  .19     .20
## 2        TimeStep 4, 20 7.79 ***  .20   .0006
## 3 Social:TimeStep 4, 20     0.54 .007     .71
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '+' 0.1 ' ' 1
###### General Linear Model (same as ANOVA, but uses type 1 sum of squares) 
GLM.model<-glm(HappyPercentZ ~ Social*TimeStep, data=HappyData)
#fit data
HappyData$GLMfitted<-fitted(GLM.model)
#manually extract R2 to compare to mixed models
summary(lm(HappyPercentZ ~ GLMfitted, data=HappyData))$r.squared
## [1] 0.3234229
  • We can fit the GLM/ANOVA estimate for every single subject
  • How is our ANOVA doing in predicting each subject?
  • ANOVA is not predicting subject level; it is testing/predicting at the average level of the data
  • However, if we have longitudinal-like data (like in this study) why not make predictions at the subject-level?
FittedGlmPlot1 <-ggplot()+
  facet_grid(Subject ~ Social, labeller=label_both)+
  geom_line(data = HappyData, aes(x = TimeStep, y =GLMfitted))+
  geom_point(data = HappyData, aes(x = TimeStep, y =HappyPercentZ, group=Subject,colour = Subject),size=3)+
  xlab("Time Step")+ylab("Happiness")
FittedGlmPlot1

3 Random Effect Modeling: Fitting at different levels

  • Linear regression assumes one measurement per subject, but what is you have multiple measurements per subject?
  • If we have longitudinal data per subject we could estimate a slope per subject!
  • We could then extract the slopes estimates per subject [and do statistics on the slopes per subject]
  • But we lose the variances around those slopes! Such as below.
FittedlmPlot <-ggplot(data = HappyData, aes(x = TimeStep, y =HappyPercentZ, group=Subject))+
  facet_grid(Subject ~ Social, labeller=label_both)+
  geom_point(aes(colour = Subject), size=3)+
  geom_smooth(method = "lm",se=FALSE, color ="black", size=.5) +
  xlab("Time Step")+ylab("Happiness")
FittedlmPlot

3.1 Random intercepts

  • So we need to get a regression model to try to fit at the subject level
  • (1|Subject) means each subject can have their own intercept
  • REML=FALSE means fit the with maximum likelihood [like we did with GLM]
  • You can see the random, fixed terms, and correlations between the fixed terms
Model.1<-lmer(HappyPercentZ ~Social*TimeStep
                   +(1|Subject),  
                   data=HappyData, REML=FALSE)
summary(Model.1)
## Linear mixed model fit by maximum likelihood t-tests use Satterthwaite
##   approximations to degrees of freedom [lmerMod]
## Formula: HappyPercentZ ~ Social * TimeStep + (1 | Subject)
##    Data: HappyData
## 
##      AIC      BIC   logLik deviance df.resid 
##    -78.3    -65.7     45.2    -90.3       54 
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -1.96591 -0.77221 -0.02667  0.61174  1.96133 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  Subject  (Intercept) 0.00246  0.0496  
##  Residual             0.01160  0.1077  
## Number of obs: 60, groups:  Subject, 6
## 
## Fixed effects:
##                      Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)           0.29975    0.03962 31.61000   7.565 1.39e-08 ***
## SocialGroup          -0.08713    0.04817 54.00000  -1.809  0.07603 .  
## TimeStep              0.04789    0.01390 54.00000   3.444  0.00112 ** 
## SocialGroup:TimeStep -0.01326    0.01966 54.00000  -0.674  0.50293    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) SclGrp TimStp
## SocialGroup -0.608              
## TimeStep    -0.702  0.577       
## SclGrp:TmSt  0.496 -0.816 -0.707
  • Next we examine the intercepts for each person
  • The var and sd of these values are what was seen in the table above
ranef(Model.1)
## $Subject
##    (Intercept)
## 1 -0.002086499
## 2  0.059054796
## 3 -0.055446410
## 4  0.045854186
## 5 -0.034645965
## 6 -0.012730108
  • We can examine an \(R^2\) for the models, but we have a problem
  • Is the \(R^2\) coming from the fixed or random effect? We can parse that:
  • The marginal is the \(R^2\) for the fixed effects
  • The conditional \(R^2\) includes random + fixed effects
  • There is no standard as to which to report \(R^2\) and most people never report these at all as if the random effects are complex (nested, or crossed) the meaning of them changes
  • We will explore this more next week (but today I will use them to help make sense of what the random effects are doing)
rsquared.glmm(Model.1)
##            Class   Family     Link  Marginal Conditional       AIC
## 1 merModLmerTest gaussian identity 0.3271115     0.44485 -78.30432
  • Let’s plot the model fit
HappyData$Model.1.fitted<-predict(Model.1)
FittedlmPlot1 <-ggplot()+
  facet_grid(Subject ~ Social, labeller=label_both)+
  geom_line(data = HappyData, aes(x = TimeStep, y =Model.1.fitted))+
  geom_point(data = HappyData, aes(x = TimeStep, y =HappyPercentZ, group=Subject,colour = Subject), size=3)+
  xlab("Time Step")+ylab("Happiness")
FittedlmPlot1

3.2 Random intercept and slopes for social condition

  • Model 2 treats that Social condition as free to vary between Subjects, but accounts for the correlation within Subjects between the Social condition)
  • (1+Social|Subject) means each subject can have their intercept and social condition can vary as function of the subject
  • The correlations of the random terms are the correlations of each subjects intercept and slope of social condition
Model.2<-lmer(HappyPercentZ ~Social*TimeStep
                       +(1+Social|Subject),  
                       data=HappyData, REML=FALSE)
summary(Model.2)
## Linear mixed model fit by maximum likelihood t-tests use Satterthwaite
##   approximations to degrees of freedom [lmerMod]
## Formula: HappyPercentZ ~ Social * TimeStep + (1 + Social | Subject)
##    Data: HappyData
## 
##      AIC      BIC   logLik deviance df.resid 
##   -116.8   -100.1     66.4   -132.8       52 
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -1.78560 -0.64195 -0.07534  0.68067  1.83722 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev. Corr 
##  Subject  (Intercept) 0.014739 0.1214        
##           SocialGroup 0.027698 0.1664   -0.91
##  Residual             0.003906 0.0625        
## Number of obs: 60, groups:  Subject, 6
## 
## Fixed effects:
##                       Estimate Std. Error        df t value Pr(>|t|)    
## (Intercept)           0.299754   0.053358  7.260000   5.618 0.000705 ***
## SocialGroup          -0.087128   0.073468  7.340000  -1.186 0.272617    
## TimeStep              0.047886   0.008068 48.000000   5.935 3.15e-07 ***
## SocialGroup:TimeStep -0.013261   0.011411 48.000000  -1.162 0.250899    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) SclGrp TimStp
## SocialGroup -0.883              
## TimeStep    -0.302  0.220       
## SclGrp:TmSt  0.214 -0.311 -0.707
ranef(Model.2)
## $Subject
##    (Intercept) SocialGroup
## 1 -0.101458905  0.18907071
## 2  0.155472520 -0.15391566
## 3 -0.004602345 -0.12600429
## 4  0.148218108 -0.17212981
## 5 -0.162199143  0.22613420
## 6 -0.035430236  0.03684485
  • The \(R^2\) for fixed and random
rsquared.glmm(Model.2)
##            Class   Family     Link  Marginal Conditional       AIC
## 1 merModLmerTest gaussian identity 0.3271115    0.813065 -116.8292
  • Plot the model results
HappyData$Model.2.fitted<-predict(Model.2)
FittedlmPlot2 <-ggplot()+
  facet_grid(Subject ~ Social, labeller=label_both)+
  geom_line(data = HappyData, aes(x = TimeStep, y=Model.2.fitted))+
  geom_point(data = HappyData, aes(x = TimeStep, y =HappyPercentZ, group=Subject,colour = Subject), size=3)+
  #  coord_cartesian(ylim = c(.03,.074))+ 
  xlab("Time Step")+ylab("Happiness")
FittedlmPlot2

3.3 Random intercepts and slopes for time

  • Model 3 treats that timesteps as free to vary between Subjects, but accounts for the correlation within Subjects across time
  • (1+TimeStep|Subject) means each subject can have their intercept and time slope can vary as function of the subject
  • The correlations of the random terms are the correlations of each subjects intercept and slope of time
Model.3<-lmer(HappyPercentZ ~Social*TimeStep
                       +(1+TimeStep|Subject),  
                       data=HappyData, REML=FALSE)
summary(Model.3)
## Linear mixed model fit by maximum likelihood  ['lmerMod']
## Formula: HappyPercentZ ~ Social * TimeStep + (1 + TimeStep | Subject)
##    Data: HappyData
## 
##      AIC      BIC   logLik deviance df.resid 
##    -77.6    -60.9     46.8    -93.6       52 
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -1.91237 -0.79968  0.09564  0.76808  1.92161 
## 
## Random effects:
##  Groups   Name        Variance  Std.Dev. Corr
##  Subject  (Intercept) 0.0000000 0.00000      
##           TimeStep    0.0005673 0.02382   NaN
##  Residual             0.0106561 0.10323      
## Number of obs: 60, groups:  Subject, 6
## 
## Fixed effects:
##                      Estimate Std. Error t value
## (Intercept)           0.29975    0.03264   9.183
## SocialGroup          -0.08713    0.04617  -1.887
## TimeStep              0.04789    0.01650   2.903
## SocialGroup:TimeStep -0.01326    0.01885  -0.704
## 
## Correlation of Fixed Effects:
##             (Intr) SclGrp TimStp
## SocialGroup -0.707              
## TimeStep    -0.660  0.466       
## SclGrp:TmSt  0.577 -0.816 -0.571
## convergence code: 0
## unable to evaluate scaled gradient
## Model failed to converge: degenerate  Hessian with 1 negative eigenvalues
ranef(Model.3)
## $Subject
##   (Intercept)    TimeStep
## 1           0 -0.01125261
## 2           0  0.02847361
## 3           0 -0.02110006
## 4           0  0.02962424
## 5           0 -0.01224243
## 6           0 -0.01350274
  • The \(R^2\) for fixed and random
rsquared.glmm(Model.3)
##            Class   Family     Link  Marginal Conditional       AIC
## 1 merModLmerTest gaussian identity 0.3271115   0.4900174 -77.62223
  • Plot the model results
HappyData$Model.3.fitted<-predict(Model.3)
FittedlmPlot3 <-ggplot()+
  facet_grid(Subject ~ Social, labeller=label_both)+
  geom_line(data = HappyData, aes(x = TimeStep, y =Model.3.fitted))+
  geom_point(data = HappyData, aes(x = TimeStep, y =HappyPercentZ, group=Subject,colour = Subject), size=3)+
  #  coord_cartesian(ylim = c(.03,.074))+ 
  xlab("Time Step")+ylab("Happiness")
FittedlmPlot3

3.4 Random intercepts and slopes for time and social (but independent)

  • Model 4 treats that timesteps and Social as free to vary between Subjects but accounts for the correlation within Subjects across times and social
  • (1+Social|Subject)+(0+TimeStep|Subject) means each subject can have their intercept and can have their slope of social condition
  • Also, time slope can vary as function of the subject, but it treats the variance between social and time as independent
Model.4<-lmer(HappyPercentZ ~Social*TimeStep
                   +(1+Social|Subject)+(0+TimeStep|Subject),  
                   data=HappyData, REML=FALSE)
summary(Model.4)
## Linear mixed model fit by maximum likelihood t-tests use Satterthwaite
##   approximations to degrees of freedom [lmerMod]
## Formula: HappyPercentZ ~ Social * TimeStep + (1 + Social | Subject) +  
##     (0 + TimeStep | Subject)
##    Data: HappyData
## 
##      AIC      BIC   logLik deviance df.resid 
##   -141.7   -122.9     79.9   -159.7       51 
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -2.01289 -0.59864  0.07634  0.55073  2.04857 
## 
## Random effects:
##  Groups    Name        Variance  Std.Dev. Corr 
##  Subject   (Intercept) 0.0078407 0.08855       
##            SocialGroup 0.0286042 0.16913  -0.82
##  Subject.1 TimeStep    0.0008533 0.02921       
##  Residual              0.0016405 0.04050       
## Number of obs: 60, groups:  Subject, 6
## 
## Fixed effects:
##                       Estimate Std. Error        df t value Pr(>|t|)    
## (Intercept)           0.299754   0.038351  6.500000   7.816 0.000156 ***
## SocialGroup          -0.087128   0.071382  6.550000  -1.221 0.264367    
## TimeStep              0.047886   0.013022  7.150000   3.677 0.007595 ** 
## SocialGroup:TimeStep -0.013261   0.007395 41.280000  -1.793 0.080244 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) SclGrp TimStp
## SocialGroup -0.811              
## TimeStep    -0.109  0.059       
## SclGrp:TmSt  0.193 -0.207 -0.284
ranef(Model.4)
## $Subject
##    (Intercept) SocialGroup     TimeStep
## 1 -0.036220747  0.19861535 -0.034692674
## 2  0.109212381 -0.15276779  0.025623548
## 3  0.002062953 -0.14020009 -0.004870427
## 4  0.066518561 -0.17447021  0.044381696
## 5 -0.160470112  0.23127547 -0.001095735
## 6  0.018896963  0.03754727 -0.029346408
  • The \(R^2\) for fixed and random
rsquared.glmm(Model.4)
##            Class   Family     Link  Marginal Conditional       AIC
## 1 merModLmerTest gaussian identity 0.2921539   0.9298807 -141.7062
  • Plot the model results
HappyData$Model.4.fitted<-predict(Model.4)
FittedlmPlot4 <-ggplot()+
  facet_grid(Subject ~ Social, labeller=label_both)+
  geom_line(data = HappyData, aes(x = TimeStep, y =Model.4.fitted))+
  geom_point(data = HappyData, aes(x = TimeStep, y =HappyPercentZ, group=Subject,colour = Subject), size=3)+
  #  coord_cartesian(ylim = c(.03,.074))+ 
  xlab("Time Step")+ylab("Happiness")
FittedlmPlot4

3.5 Random intercepts and slopes for time and social (but not independent)

  • Model 5 treats that timesteps and Social as free to vary between Subjects but accounts for the correlation within Subjects across time and Social
  • (1+Social+ TimeStep|Subject) means each subject can have their intercept and can have their slope of social condition
  • Also, time slope can vary as function of the subject, but it treats the variance between social and time as correlated
Model.5<-lmer(HappyPercentZ ~Social*TimeStep
                       +(1+Social+TimeStep|Subject),  
                       data=HappyData, REML=FALSE)
summary(Model.5)
## Linear mixed model fit by maximum likelihood t-tests use Satterthwaite
##   approximations to degrees of freedom [lmerMod]
## Formula: HappyPercentZ ~ Social * TimeStep + (1 + Social + TimeStep |  
##     Subject)
##    Data: HappyData
## 
##      AIC      BIC   logLik deviance df.resid 
##   -142.3   -119.3     82.2   -164.3       49 
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -1.92765 -0.54367  0.03674  0.49684  2.22487 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev. Corr       
##  Subject  (Intercept) 0.007488 0.08654             
##           SocialGroup 0.028610 0.16914  -0.80      
##           TimeStep    0.000912 0.03020   0.39 -0.70
##  Residual             0.001626 0.04032             
## Number of obs: 60, groups:  Subject, 6
## 
## Fixed effects:
##                       Estimate Std. Error        df t value Pr(>|t|)    
## (Intercept)           0.299754   0.037559  6.490000   7.981 0.000138 ***
## SocialGroup          -0.087128   0.071369  6.540000  -1.221 0.264315    
## TimeStep              0.047886   0.013383  7.020000   3.578 0.008965 ** 
## SocialGroup:TimeStep -0.013261   0.007362 42.000000  -1.801 0.078830 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) SclGrp TimStp
## SocialGroup -0.789              
## TimeStep     0.229 -0.569       
## SclGrp:TmSt  0.196 -0.206 -0.275
ranef(Model.5)
## $Subject
##    (Intercept) SocialGroup     TimeStep
## 1 -0.030473887  0.19960051 -0.037608028
## 2  0.108716859 -0.15516418  0.026012347
## 3 -0.005714525 -0.13533773 -0.001240192
## 4  0.064396547 -0.17849550  0.045650168
## 5 -0.156192538  0.22833395 -0.003114846
## 6  0.019267545  0.04106295 -0.029699449
  • The \(R^2\) for fixed and random
rsquared.glmm(Model.5)
##            Class   Family     Link  Marginal Conditional     AIC
## 1 merModLmerTest gaussian identity 0.3271116    0.922186 -142.33
  • Plot the model results
HappyData$Model.5.fitted<-predict(Model.5)
FittedlmPlot5 <-ggplot()+
  facet_grid(Subject ~ Social, labeller=label_both)+
  geom_line(data = HappyData, aes(x = TimeStep, y =Model.5.fitted))+
  geom_point(data = HappyData, aes(x = TimeStep, y =HappyPercentZ, group=Subject,colour = Subject), size=3)+
  #  coord_cartesian(ylim = c(.03,.074))+ 
  xlab("Time Step")+ylab("Happiness")
FittedlmPlot5

3.6 Random intercepts and slopes for time x social

  • Model 6 treats that timesteps, Social and timesteps:Social (interaction) as free to vary between Subjects but accounts for the correlation within Subjects across time and Social
  • (1+Social*TimeStep|Subject) means (1+Social+TimeStep+Social:TimeStep|Subject) each subject can have their intercept, their slope of social condition, their time slope, and their interaction between social and time
  • Also, all random slopes and intercepts can be correlated
  • This is called the maximum model Barr et al., 2013 (these do not always fit best see Bates et al., 2015)
Model.6<-lmer(HappyPercentZ ~Social*TimeStep
                   +(1+Social*TimeStep|Subject),  
                   data=HappyData, REML=FALSE)
summary(Model.6)
## Linear mixed model fit by maximum likelihood t-tests use Satterthwaite
##   approximations to degrees of freedom [lmerMod]
## Formula: HappyPercentZ ~ Social * TimeStep + (1 + Social * TimeStep |  
##     Subject)
##    Data: HappyData
## 
##      AIC      BIC   logLik deviance df.resid 
##   -163.9   -132.5     97.0   -193.9       45 
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -2.13890 -0.49571  0.09645  0.58282  1.64482 
## 
## Random effects:
##  Groups   Name                 Variance  Std.Dev. Corr             
##  Subject  (Intercept)          0.0033178 0.05760                   
##           SocialGroup          0.0115506 0.10747  -0.49            
##           TimeStep             0.0017789 0.04218   0.51 -0.91      
##           SocialGroup:TimeStep 0.0011692 0.03419  -0.85  0.87 -0.77
##  Residual                      0.0007876 0.02806                   
## Number of obs: 60, groups:  Subject, 6
## 
## Fixed effects:
##                      Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)           0.29975    0.02513  6.04600  11.926 1.99e-05 ***
## SocialGroup          -0.08713    0.04564  6.02300  -1.909   0.1046    
## TimeStep              0.04789    0.01760  6.00500   2.721   0.0345 *  
## SocialGroup:TimeStep -0.01326    0.01487  6.05800  -0.892   0.4065    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) SclGrp TimStp
## SocialGroup -0.509              
## TimeStep     0.405 -0.821       
## SclGrp:TmSt -0.676  0.704 -0.762
ranef(Model.6)
## $Subject
##    (Intercept) SocialGroup     TimeStep SocialGroup:TimeStep
## 1  0.004890881  0.14596807 -0.053180086         2.501422e-02
## 2  0.077256851 -0.08225118  0.044139590        -3.865833e-02
## 3 -0.021269036 -0.10502666  0.007139793        -1.865968e-02
## 4  0.029691590 -0.11938190  0.062208657        -2.801749e-02
## 5 -0.107057025  0.11365137 -0.029970241         6.029698e-02
## 6  0.016486739  0.04704030 -0.030337713         2.430126e-05
  • The \(R^2\) for fixed and random
rsquared.glmm(Model.6)
##            Class   Family     Link  Marginal Conditional       AIC
## 1 merModLmerTest gaussian identity 0.3271114   0.9623085 -163.9052
  • Plot the model results
HappyData$Model.6.fitted<-predict(Model.6)
FittedlmPlot6 <-ggplot()+
  facet_grid(Subject ~ Social, labeller=label_both)+
  geom_line(data = HappyData, aes(x = TimeStep, y =Model.6.fitted))+
  geom_point(data = HappyData, aes(x = TimeStep, y =HappyPercentZ, group=Subject,colour = Subject), size=3)+
  #  coord_cartesian(ylim = c(.03,.074))+ 
  xlab("Time Step")+ylab("Happiness")
FittedlmPlot6

3.7 How to select a random structure

  • First, these models often fail to converge
  • Convergence mean that the optimizer (numerical method to find the parameters) has failed to reach an optimal or stable solution
  • There are many optimizers to select from in R if you need to change it (https://rstudio-pubs-static.s3.amazonaws.com/33653_57fc7b8e5d484c909b615d8633c01d51.html)
  • Failures might not just be a numerical optimization problem, but a problem with your specification of the random effects
  • You could over-specify your random effects in a way that is not parsimonious with the true random effects in the population
  • You could set them up in such a way that do not reflect the experiment

3.7.1 Practical Solution to random structure

  • This is a battlefield in LMM
  • The current best practice is to try to match your design as best as you can
  • If that fails, try different optimizer
  • If that fails, reduce the complexity of the random effect (remove random correlations one at a time)
  • If that fails, figure out which term is causing the problem and remove it
  • If you under-specify, you inflate your type I error, so it a balancing act

4 Fixed effects

  • Just like your linear regression you can run these models hierarchically, once the random effects are in place
  • you just test between model fits using a devience test (like we did with GLM) anova(model.1, model.2)

4.1 Plot the final model correctly

  • The method I used was to predict the data, but that works well if you have no covariates
  • If you have covariates, its best to use the effects package just like we did with linear regression
  • Also you have to decide to plot both fixed + random or just the fixed effects
  • Most people only plot the fixed effect, as they want to control for the random effects
  • However, if someone asks you to plot the data and the model fits together, you will need to tell them it complicated (as you are forced to plot fixed and random. Basically you use the code above I have been showing you)
  • Below is how to plot fixed only (and controling for covariates if you them)
  • I have used ggplot this time (unlike our past lectures to make it pretty)
library(effects)
Final.Fixed<-effect(c("Social*TimeStep"), Model.6,
                           xlevels=list(TimeStep=seq(0,4,1)))

# You have to convert the output to a dataframe
Final.Fixed<-as.data.frame(Final.Fixed)

Final.Fixed.Plot <-ggplot(data = Final.Fixed, aes(x = TimeStep, y =fit, group=Social))+
  coord_cartesian(xlim=c(0,4),ylim = c(0,.7))+ 
  geom_line(aes(color=Social), size=2)+
  geom_ribbon(aes(ymin=fit-se, ymax=fit+se,fill=Social),alpha=.2)+
  xlab("Time")+
  ylab("Happiness\nscore")+
  scale_color_manual(values=c("blue", "red"))+
  scale_fill_manual(values=c("blue", "red"))+
  theme_bw()+
  theme(text=element_text(face="bold", size=12),
        panel.grid.major = element_blank(), 
        panel.grid.minor = element_blank(), 
        panel.border = element_rect(fill = NA, colour = "NA"),
        axis.line = element_line(size = 1, colour = "grey80"),
        legend.title=element_blank(),
        legend.position = c(.2, .92))
Final.Fixed.Plot

5 P-values

  • Pvalues are hotly disputed in Mixed models. Why?
  • Null-hypothesis testing and mixed models are current major battlefield
  • GLM uses standard error to determine significance: Required degrees of freedom to correct z-distribution for “sampling error” to produce t/F tables.
  • Assumes normality
  • “Modern” methods [such as bootstrapping] for pvalue determination have focused on working toward distribution appropriate assumptions
  • Bayesian’s use more complex forms of bootstrapping (MCMC) to further refine this methods
  • for Mixed Models we have Degrees of freedom problem for the error term. Why?

5.1 Example of DF parsing

  • Sample 500, 11the graders take the SAT from 3 schools nested in 2 economic districts

  • Random sampling of 500 total from 3 schools [note, school 3 is twice as large school 1 and 2]
  • ANOVA Design: 1 factor: School OR district [they provide an unbalanced design so they can only analyzed one at time]
  • Assume homogeneity of variance of SAT score across schools and districts [bad assumption]
  • ANOVA can only ask 1 basic question: Are the means between the schools or districts different

5.1.1 ANOVA problem?

  • Why is ANOVA bad here?
  • School 1 is predominately poor vs School 2 which is rich, so school 1 has less funding, larger classrooms sizes, more social issues at home and in school, less teachers, etc.
  • This could impact both mean test scores and variance of test scores
  • Students from the poor district might have more diverse scores [high variance]
  • Student from rich district might have more similar scores, but higher average scores [low variance]
  • Poor student in the Rich school might also have changes in variance or mean test scores, which look very different from the poor or mixed school.

5.1.2 Mixed Solution

  • Mixed model solution to same exact data set?
  • Fixed factors: School and District [student belongs to]
  • SAT score MEANS could be different depending on the school a kid goes to and what district they are from.
  • Random Factors: District, School, Student
  • SAT score VARIANCES could be different depending on the school a kid goes to and what district they are from.
  • This allows each kid “group” to have its own mean and variance:
  • calculated about their classification based on school and district [so error term varies as a product of this custom classification]
  • So kids from the school 1, poor district are “allowed” to differ from kids from school 1, rich district from school 2 poor district, etc..
  • In the case when all the variables are categorical, the DF “seems” logical, in that the proper MSerror terms could be calculated. However the random factors have been nested.
  • So we cannot know how many of the people went into parsing the variance for each level of the random error term.
  • We could make assumptions, but some statisticians say over their dead bodies.
  • ANOVA is assumed to be robust against many violations, but that’s mostly only true in well controlled balanced designs.
  • When ANOVA is stretched into mixed model territory, it can result in unreasonable levels of type 1 error.
  • So what do we do?

5.2 Getting Pvalues

5.2.1 Path A: Don’t push your assumptions on me!

  1. Modeling Fitting testing
  • Likelihood ratio test [chi-square distribution]
  • D = 2 - ln (Likelihood of null model / Likelihood of alternative model]
  • Add/Remove fixed factors and test to see if model “fits the data better”
  • Suggested by Bates and Bolker [least assumptions]
  • FAST and well accepted [a bit anti-conservative some argue]
  • to do this you simply have to anova(model.1, model.2), just like we did with testing between \(R^2\) between model fits
  1. Bootstrapping of Confidence Intervals [can be parametric or semi-parametric]
  • Re-sampling method. Uses the distributions in your data to determine confidence intervals
  • Common method in modern stats. A much better method then assuming normality and t-distributions
  • VERY, VERY SLOW.
  • Par
# parametric
Model.Final.CI.boot <- confint(Model.6, method="boot",nsim=100,boot.type = c("perc")) 
Model.Final.CI.boot
##                            2.5 %      97.5 %
## .sig01                0.01618364  0.08900227
## .sig02               -0.94476411  0.75716991
## .sig03               -0.87405423  0.98038606
## .sig04               -0.99477191  0.37449140
## .sig05                0.04638626  0.17284850
## .sig06               -0.99923526 -0.14043440
## .sig07                0.13688918  0.99353025
## .sig08                0.01651919  0.06524354
## .sig09               -0.99497298  0.43327602
## .sig10                0.01255264  0.05510629
## .sigma                0.01868276  0.03203938
## (Intercept)           0.24990408  0.35342503
## SocialGroup          -0.18767370  0.01546711
## TimeStep              0.01077535  0.08099926
## SocialGroup:TimeStep -0.04301530  0.01548564

5.2.2 Path B: Let’s hold hands, put our fingers in our ears, and assume!

  1. Assume z distributions to get pvalues
  • Very, very common approach
  • If you number of observations is high, just assume DF -> inf [so t > 1.96, is significant]
  • This is possible because you assume your random effects have control for the random variations in the populations and your values good estimates of the populations
  • Simulation studies by Barr et al. show this to be a reasonable assumption IF and ONLY OF random structure is MAXIMAL, but This is HOTLY debated by Bates
  1. Estimate degrees of freedom
  • SAS/SPSS uses Satterthwaite approximations
  • However, there are also Kenward-Roger approximations [see Westfall, Kenny, & Judd, 2014]
  • Either are acceptable
  • Makes a number of assumptions, will mirror results of traditional ANOVA fairly closely [as you would expect when you make similar assumptions about DF]

[Note: You have to refit the model with the lmerTest package]

library(lmerTest) #mixed model add-on package to 
Model.Final<-lmer(HappyPercentZ ~Social*TimeStep
                  +(1+Social*TimeStep|Subject),  
                  data=HappyData, REML=FALSE)

summary(Model.Final,ddf = "lme4")
## Linear mixed model fit by maximum likelihood  ['lmerMod']
## Formula: HappyPercentZ ~ Social * TimeStep + (1 + Social * TimeStep |  
##     Subject)
##    Data: HappyData
## 
##      AIC      BIC   logLik deviance df.resid 
##   -163.9   -132.5     97.0   -193.9       45 
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -2.13890 -0.49571  0.09645  0.58282  1.64482 
## 
## Random effects:
##  Groups   Name                 Variance  Std.Dev. Corr             
##  Subject  (Intercept)          0.0033178 0.05760                   
##           SocialGroup          0.0115506 0.10747  -0.49            
##           TimeStep             0.0017789 0.04218   0.51 -0.91      
##           SocialGroup:TimeStep 0.0011692 0.03419  -0.85  0.87 -0.77
##  Residual                      0.0007876 0.02806                   
## Number of obs: 60, groups:  Subject, 6
## 
## Fixed effects:
##                      Estimate Std. Error t value
## (Intercept)           0.29975    0.02513  11.926
## SocialGroup          -0.08713    0.04564  -1.909
## TimeStep              0.04789    0.01760   2.721
## SocialGroup:TimeStep -0.01326    0.01487  -0.892
## 
## Correlation of Fixed Effects:
##             (Intr) SclGrp TimStp
## SocialGroup -0.509              
## TimeStep     0.405 -0.821       
## SclGrp:TmSt -0.676  0.704 -0.762
summary(Model.Final, ddf="Satterthwaite") ## SPSS/SAS method
## Linear mixed model fit by maximum likelihood t-tests use Satterthwaite
##   approximations to degrees of freedom [lmerMod]
## Formula: HappyPercentZ ~ Social * TimeStep + (1 + Social * TimeStep |  
##     Subject)
##    Data: HappyData
## 
##      AIC      BIC   logLik deviance df.resid 
##   -163.9   -132.5     97.0   -193.9       45 
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -2.13890 -0.49571  0.09645  0.58282  1.64482 
## 
## Random effects:
##  Groups   Name                 Variance  Std.Dev. Corr             
##  Subject  (Intercept)          0.0033178 0.05760                   
##           SocialGroup          0.0115506 0.10747  -0.49            
##           TimeStep             0.0017789 0.04218   0.51 -0.91      
##           SocialGroup:TimeStep 0.0011692 0.03419  -0.85  0.87 -0.77
##  Residual                      0.0007876 0.02806                   
## Number of obs: 60, groups:  Subject, 6
## 
## Fixed effects:
##                      Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)           0.29975    0.02513  6.04600  11.926 1.99e-05 ***
## SocialGroup          -0.08713    0.04564  6.02300  -1.909   0.1046    
## TimeStep              0.04789    0.01760  6.00500   2.721   0.0345 *  
## SocialGroup:TimeStep -0.01326    0.01487  6.05800  -0.892   0.4065    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) SclGrp TimStp
## SocialGroup -0.509              
## TimeStep     0.405 -0.821       
## SclGrp:TmSt -0.676  0.704 -0.762
summary(Model.Final,ddf = "Kenward-Roger") # Kenney et al suggested method
## Linear mixed model fit by maximum likelihood t-tests use Kenward-Roger
##   approximations to degrees of freedom [lmerMod]
## Formula: HappyPercentZ ~ Social * TimeStep + (1 + Social * TimeStep |  
##     Subject)
##    Data: HappyData
## 
##      AIC      BIC   logLik deviance df.resid 
##   -163.9   -132.5     97.0   -193.9       45 
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -2.13890 -0.49571  0.09645  0.58282  1.64482 
## 
## Random effects:
##  Groups   Name                 Variance  Std.Dev. Corr             
##  Subject  (Intercept)          0.0033178 0.05760                   
##           SocialGroup          0.0115506 0.10747  -0.49            
##           TimeStep             0.0017789 0.04218   0.51 -0.91      
##           SocialGroup:TimeStep 0.0011692 0.03419  -0.85  0.87 -0.77
##  Residual                      0.0007876 0.02806                   
## Number of obs: 60, groups:  Subject, 6
## 
## Fixed effects:
##                      Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)           0.29975    0.02513  6.99600  10.955 1.17e-05 ***
## SocialGroup          -0.08713    0.04564  7.04400  -1.751    0.123    
## TimeStep              0.04789    0.01760  7.13300   2.489    0.041 *  
## SocialGroup:TimeStep -0.01326    0.01487  6.95500  -0.820    0.439    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) SclGrp TimStp
## SocialGroup -0.509              
## TimeStep     0.405 -0.821       
## SclGrp:TmSt -0.676  0.704 -0.762
  • Best practice is to do model fitting (option 1) and verify it with options 2 OR 3.
  • Option 4 is useful if journal demands DF, but it can be very conservative when you have lots of variance issues
  • Bates says, Pvalues are waste of effort and DFs should not be calculated (but you will have to judge for yourself what makes sense)

6 Mixed to ANOVA

  • If you wanted to report main effects and interactions, instead of slopes you might convert your mixed model into an ANOVA
  • But remember, your slopes of time revert to being factors and not continuous variables
anova(Model.Final,ddf = "Kenward-Roger") # Kenney et al suggested method
## Analysis of Variance Table of type III  with  Kenward-Roger 
## approximation for degrees of freedom
##                    Sum Sq   Mean Sq NumDF DenDF F.value  Pr(>F)  
## Social          0.0024151 0.0024151     1     5  3.0665 0.14032  
## TimeStep        0.0067466 0.0067466     1     5  8.5663 0.03275 *
## Social:TimeStep 0.0005299 0.0005299     1     5  0.6728 0.44939  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
LS0tDQp0aXRsZTogJ0xpbmVhciBNaXhlZCBNb2RlbHMnDQpvdXRwdXQ6DQogIGh0bWxfZG9jdW1lbnQ6DQogICAgY29kZV9kb3dubG9hZDogeWVzDQogICAgZm9udHNpemU6IDhwdA0KICAgIGhpZ2hsaWdodDogdGV4dG1hdGUNCiAgICBudW1iZXJfc2VjdGlvbnM6IHllcw0KICAgIHRoZW1lOiBmbGF0bHkNCiAgICB0b2M6IHllcw0KICAgIHRvY19mbG9hdDoNCiAgICAgIGNvbGxhcHNlZDogbm8NCi0tLQ0KDQpgYGB7ciBzZXR1cCwgaW5jbHVkZT1GQUxTRX0NCmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSkNCmtuaXRyOjpvcHRzX2NodW5rJHNldChtZXNzYWdlID0gRkFMU0UpDQprbml0cjo6b3B0c19jaHVuayRzZXQod2FybmluZyA9ICBGQUxTRSkNCmtuaXRyOjpvcHRzX2NodW5rJHNldChmaWcud2lkdGg9NS4yNSkNCmtuaXRyOjpvcHRzX2NodW5rJHNldChmaWcuaGVpZ2h0PTUuMCkNCmtuaXRyOjpvcHRzX2NodW5rJHNldChmaWcuYWxpZ249J2NlbnRlcicpIA0KYGBgDQoNCg0KIyBNaXhlZCBNb2RlbHMNCi0gbGluZWFyIHJlZ3Jlc3Npb25zIG9ubHkgaGFzIGZpeGVkLWVmZmVjdHMgW2NvbnRpbnVvdXNdIG9yIGZpeGVkLWZhY3RvcnMgW2dyb3Vwc106IHZhcmlhYmxlcyBvZiBpbnRlcmVzdCAob2Z0ZW4geW91ciBtYW5pcHVsYXRpb24pDQotIFRoZSBvbmx5IHJhbmRvbSBmYWN0b3IgYXNzdW1lZCBpcyB5b3VyIHN1YmplY3QncyByZXNwb25zZQ0KLSBXaGF0IGlmIHlvdSBoYXZlIG90aGVyIGVmZmVjdCBvciBmYWN0b3JzIHRoYXQgbWlnaHQgYWRkIHJhbmRvbSB2YXJpYXRpb24/IA0KLSBSYW5kb20tZWZmZWN0cy9mYWN0b3JzOiBMZXZlbHMgcmFuZG9tbHkgc2FtcGxlZCBmcm9tIGEgbXVjaCBsYXJnZXIgcG9wdWxhdGlvbjogU29tZXRoaW5nIHlvdSB3YW50IHRvIGdlbmVyYWxpemUgYWNyb3NzOiBTdWJqZWN0cywgV29yZHMsIFBpY3R1cmVzLCBldGMNCg0KIyMgVHlwZSBvZiBNaXhlZCBNb2RlbHMNCi0gV2Ugd2lsbCBleHBsb3JlIGNyb3NzZWQgYW5kIGRlc2lnbmVkIGRlc2lnbnMNCi0gVG9kYXkgd2Ugd2lsbCBmdWxseSBjcm9zc2VkLCBuZXh0IGNsYXNzIHdlIHdpbGwgZG8gY3Jvc3NlZC1uZXN0ZWQgZGVzaWducyANCg0KIyMjIENyb3NzZWQgZGVzaWducw0KLSBSZXBlYXRlZCBtZWFzdXJlczogZXZlcnlvbmUgZG9lcyBhbGwgdGhlIGNvbmRpdGlvbnMNCi0gTG9uZ2l0dWRpbmFsIGRhdGE6IFBlb3BsZSBhcmUgdHJhY2tlZCBvdmVyIHRpbWUNCi0gR3Jvd3RoIEN1cnZlIE1vZGVsczogRml0dGluZyBmdW5jdGlvbnMgKGxpa2UgcG9seW5vbWlhbHMpIHRvIGJlaGF2aW9yIG92ZXIgdGltZSANCi0gUHJvYmxlbTogRWFjaCBwZXJzb24ncyByZXNwb25zZSBpcyBnb2luZyB0byBpbnRlci1jb3JyZWxhdGVkIHdpdGggdGhlaXIgb3RoZXIgcmVzcG9uc2VzIA0KDQojIyMgTmVzdGVkIGRlc2lnbnMNCi0gTXVsdGlsZXZlbC9IaWVyYXJjaGljYWwgZGVzaWduczogU3R1ZGVudHMgbmVzdGVkIGluIGNsYXNzcm9vbXMgW0NsdXN0ZXJlZCBkYXRhXQ0KLSBQcm9ibGVtOiBDbHVzdGVyaW5nIG1lYW5zIGVhY2ggc3R1ZGVudCdzIHNjb3JlcyB3aXRoaW4gZWFjaCBjbGFzc3Jvb20gbWlnaHQgYmUgY29ycmVsYXRlZCBiZWNhdXNlIHRoZXkgYWxsIGhhdmUgdGhlIHNhbWUgdGVhY2hlciAoZm9yIGV4YW1wbGUpDQoNCiMjIyBDcm9zc2VkICYgTmVzdGVkIGRlc2lnbnMNCi0gTG9uZ2l0dWRpbmFsIE11bHRpbGV2ZWwgRGVzaWduczogU3R1ZGVudHMgbmVzdGVkIGluIGNsYXNzcm9vbXMgbWVhc3VyZWQgb3ZlciBtYW55IG1vbnRocw0KLSBQYXJ0aWFsbHkgTmVzdGVkIGRlc2lnbjogRGlmZmVyZW50IGdyb3VwcyBhcmUgbWVhc3VyZXMgb3ZlciB0aW1lDQotIE5vdyBlYWNoIHBlcnNvbidzIHJlc3BvbnNlIGlzIGdvaW5nIHRvIGludGVyLWNvcnJlbGF0ZWQgd2l0aCB0aGVpciBvdGhlciByZXNwb25zZXMsIGJ1dCB0aGF0IGNvcnJlbGF0aW9uIG1pZ2h0IGRpZmZlciBieSB3aGljaCBncm91cCB0aGV5IGJlbG9uZyB0b28gDQoNCiMjIyMgSW50ZXItQ29ycmVsYXRpb25zIA0KLSBBbGwgdGhlc2UgY29ycmVsYXRpb25zIGluZmxhdGUgeW91ciB0eXBlIEkgZXJyb3IhDQotIFlvdXIgc3RhbmRhcmQgZXJyb3JzIHdpbGwgc2hyaW5rIGJlY2F1c2UgeW91IGFyZSBtZWFzdXJpbmcgcGVvcGxlIG11bHRpcGxlDQoNCiMgUHJhY3RpY2UgZGF0YSB0byBtYWtlIHNlbnNlIG9mIHJhbmRvbSBlZmZlY3RzIA0KLSBIeXBvdGhldGljYWwgRXhwZXJpbWVudCwgbiA9IDYNCi0gRFY6ICUgb2YgaG93IGhhcHB5IHRoZXkgYXJlIA0KLSBJVjE6IFRpbWUgb2YgbWVhc3VyZW1lbnQ6IDUgdGltZSB3aW5kb3dzDQotIElWMjogU29jaWFsIFNldHRpbmc6IEFsb25lIHZzIFdpdGggYSBncm91cA0KDQoNCiMjIEFzIDJ4NSBSTSBBTk9WQQ0KLSBUcmVhdHMgVGltZS1TdGVwcyBhcyBhIGZhY3RvciANCi0gQXJlIHRoZXJlIGFueSBkaWZmZXJlbmNlcyBiZXR3ZWVuIGFueSB0aW1lLXN0ZXBzPw0KLSBOb3Qgc2xvcGVzIChwcmVkaWN0YWJsZSBjaGFuZ2VzIGluIHRpbWUpDQotIFRyZWF0cyBTb2NpYWwgYXMgYSBmYWN0b3INCi0gSXMgdGhlcmUgYSBkaWZmZXJlbmNlIGJldHdlZW4gQWxvbmUgYW5kIFdpdGggR3JvdXANCi0gUXVlc3Rpb25zOiBJcyB0aGVyZSBhbiBlZmZlY3Qgb2YgdGltZSBhbmQgU29jaWFsPyANCi0gW2Rvd25sb2FkIGZpbGVzIG5lZWQgbGVjdHVyZV0oUmVncmVzc2lvbkNsYXNzL0wxMl9NaXhlZC9NaXhlZEZpbGVzLnppcCkNCg0KYGBge3J9DQpsaWJyYXJ5KGxtZTQpICAgICAjbWl4ZWQgbW9kZWwgcGFja2FnZSBieSBEb3VnbGFzIEJhdGVzDQpsaWJyYXJ5KGFmZXgpICAgICAjZWFzeSBBTk9WQSBwYWNrYWdlIA0KbGlicmFyeShnZ3Bsb3QyKSAgI0dHcGxvdCBwYWNrYWdlIGZvciB2aXN1YWxpemluZyBkYXRhDQpzb3VyY2UoIlJlZ3Jlc3Npb25DbGFzcy9MMTJfTWl4ZWQvSGVscGVyRnVuY3Rpb25zLlIiKSAjIHNvbWUgY3VzdG9tIGZ1bmN0aW9ucw0Kc291cmNlKCJSZWdyZXNzaW9uQ2xhc3MvTDEyX01peGVkL1JzcXVhcmVkSGVscGVyLlIiKSAjIHNvbWUgY3VzdG9tIGZ1bmN0aW9ucw0KIyMjUmVhZCBpbiBEYXRhIGZpbGUNCkhhcHB5RGF0YSA8LSByZWFkLnRhYmxlKCJSZWdyZXNzaW9uQ2xhc3MvTDEyX01peGVkL0hhcHB5U3R1ZHkuY3N2Iiwgc2VwPSIsIiwgaGVhZGVyPVQpDQojIyMgTGFiZWwgdmFyaWFibGVzDQpIYXBweURhdGEkU29jaWFsIDwtIGZhY3RvcihIYXBweURhdGEkU29jaWFsRHVtbXksDQogICAgICAgICAgICAgICAgICAgICAgICBsZXZlbHMgPSBjKDAsMSksDQogICAgICAgICAgICAgICAgICAgICAgICBsYWJlbHMgPSBjKCJBbG9uZSIsICJHcm91cCIpKQ0KDQpIYXBweURhdGEkU3ViamVjdDwtYXMuZmFjdG9yKEhhcHB5RGF0YSRTdWJqZWN0KQ0KIyNQbG90IHJhdyBkYXRhDQpoaXN0KEhhcHB5RGF0YSRIYXBweVBlcmNlbnQpDQpgYGANCg0KLSBXZSB3aWxsIHRyYW5zZm9ybSB1c2luZyBGaXNoZXIncyAqci10by1aKiB0byBub3JtYWxpemUgdGhlIGRhdGEgKGJ1dCBvbmx5IGJlY2F1c2Ugb2YgaG93IEkgY3JlYXRlZCB0aGlzIGRhdGEgc2V0IGZvciB0aGlzIGFuYWx5c2lzIHdvcmsgaW4gc3BlY2lmaWMgd2F5cyBmb3IgdGhpcyBsZWN0dXJlKQ0KDQpgYGB7cn0NCiMjIyMgTm9ybWFsaXplIERWIQ0KSGFwcHlEYXRhJEhhcHB5UGVyY2VudFo8LUZpc2hlclJ0b1ooSGFwcHlEYXRhJEhhcHB5UGVyY2VudCkNCmhpc3QoSGFwcHlEYXRhJEhhcHB5UGVyY2VudFopDQoNCmBgYA0KDQotIExldCdzIHBsb3QgZWFjaCBwZXJzb24ncyBkYXRhIChzcGFnaGV0dGkgcGxvdCkNCg0KYGBge3J9DQp0aGVtZV9zZXQodGhlbWVfYncoYmFzZV9zaXplID0gMTIsIGJhc2VfZmFtaWx5ID0gIiIpKSANCg0KcGFiIDwtZ2dwbG90KGRhdGEgPSBIYXBweURhdGEsIGFlcyh4ID0gVGltZVN0ZXAsIHk9SGFwcHlQZXJjZW50WiwgZ3JvdXA9U3ViamVjdCkpKw0KICBmYWNldF9ncmlkKCB+IFNvY2lhbCwgbGFiZWxsZXI9bGFiZWxfYm90aCkrDQogICMgIGNvb3JkX2NhcnRlc2lhbih5bGltID0gYyguMDMsLjA3NCkpKyANCiAgZ2VvbV9saW5lKGFlcyhjb2xvcj1TdWJqZWN0KSkrDQogIHhsYWIoIlRpbWUgU3RlcCIpK3lsYWIoIkhhcHBpbmVzcyIpDQpwYWINCmBgYA0KDQotIExldCdzIHBsb3QgZWFjaCBwZXJzb24ncyBkYXRhIHdpdGggdGhlIG1lYW5zIHBsb3R0ZWQNCg0KYGBge3J9DQojI2N1c3RvbSBmdW5jdGlvbiB0byBnZW5lcmF0ZSBtZWFucyBhbmQgU0UgZm9ybSBhbnkgbG9uZyBmb3JtYXQgZGF0YXNldC4gDQpIYXBweURhdGFQbG90Mjwtc3VtbWFyeVNFd2l0aGluKEhhcHB5RGF0YSxtZWFzdXJldmFyPSJIYXBweVBlcmNlbnRaIiwgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd2l0aGludmFycz1jKCJTb2NpYWwiLCJUaW1lU3RlcCIpLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGlkdmFyPSJTdWJqZWN0IiwgbmEucm09RkFMU0UsIGNvbmYuaW50ZXJ2YWw9Ljk1KQ0KcGFiMiA8LWdncGxvdCgpKw0KICBnZW9tX2xpbmUoZGF0YSA9IEhhcHB5RGF0YSwgYWVzKHggPSBhcy5mYWN0b3IoVGltZVN0ZXApLCB5ID1IYXBweVBlcmNlbnRaLCBncm91cD1TdWJqZWN0LCBjb2xvcj1TdWJqZWN0KSkrDQogIGZhY2V0X2dyaWQoLiB+IFNvY2lhbCwgbGFiZWxsZXI9bGFiZWxfYm90aCkrDQogICMgIGNvb3JkX2NhcnRlc2lhbih5bGltID0gYyguMDMsLjA3NCkpKyANCiAgZ2VvbV9wb2ludChkYXRhID0gSGFwcHlEYXRhUGxvdDIsIGFlcyh4ID0gYXMuZmFjdG9yKFRpbWVTdGVwKSwgeSA9SGFwcHlQZXJjZW50WiwgZ3JvdXA9U29jaWFsKSkrDQogIGdlb21fZXJyb3JiYXIoZGF0YSA9IEhhcHB5RGF0YVBsb3QyLGFlcyh4ID0gYXMuZmFjdG9yKFRpbWVTdGVwKSwgeSA9SGFwcHlQZXJjZW50WiwgZ3JvdXA9U29jaWFsLCB5bWluPUhhcHB5UGVyY2VudFotc2UsIHltYXg9SGFwcHlQZXJjZW50WitzZSksY29sb3VyPSIjNkU2RTZFIixzaXplPS4yNSwgd2lkdGg9LjI1KSsNCiAgeGxhYigiVGltZSBTdGVwIikreWxhYigiSGFwcGluZXNzIikNCnBhYjINCg0KYGBgDQoNCi0gTGV0J3MgZXhhbWluZSBhbiBSTSBBTk9WQSByZXN1bHRzIG9mIHRoaXMgZGF0YQ0KLSBSZW1lbWJlciB0aW1lIGlzIHRyZWF0ZWQgbGlrZSBhIGZhY3RvciAobm90IGEgc2xvcGUpDQotIFtub3RlOiBJIHJhbiB0aGUgQU5PVkEgbGlrZSBhIEdMTSBqdXN0IHNvIHdlIGNhbiBleHRyYWN0IGFuIG92ZXJhbGwgJFJeMiQgYW5kIGdldCBhIGZpdHRlZCByZXN1bHRzIGZvciBlYWNoIHN1YmplY3RdDQoNCmBgYHtyfQ0KIyMjIyMjI0Fub3ZhICh1c2luZyB0eXBlIDMgc3VtIG9mIHNxdWFyZXMgc2FtZSBhcyBkZWZhdWx0IGluIFNQU1MvU0FTKQ0KQW5vdmEuTW9kZWwxIDwtYW92X2NhcihIYXBweVBlcmNlbnRaIH4gU29jaWFsKlRpbWVTdGVwICsgRXJyb3IoU3ViamVjdC8oU29jaWFsKlRpbWVTdGVwKSksIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGEgPSBIYXBweURhdGEsIGFub3ZhX3RhYmxlID0gbGlzdChjb3JyZWN0aW9uID0gIm5vbmUiLCBNU0UgPSBGQUxTRSkpDQpBbm92YS5Nb2RlbDENCiMjIyMjIyBHZW5lcmFsIExpbmVhciBNb2RlbCAoc2FtZSBhcyBBTk9WQSwgYnV0IHVzZXMgdHlwZSAxIHN1bSBvZiBzcXVhcmVzKSANCkdMTS5tb2RlbDwtZ2xtKEhhcHB5UGVyY2VudFogfiBTb2NpYWwqVGltZVN0ZXAsIGRhdGE9SGFwcHlEYXRhKQ0KI2ZpdCBkYXRhDQpIYXBweURhdGEkR0xNZml0dGVkPC1maXR0ZWQoR0xNLm1vZGVsKQ0KI21hbnVhbGx5IGV4dHJhY3QgUjIgdG8gY29tcGFyZSB0byBtaXhlZCBtb2RlbHMNCnN1bW1hcnkobG0oSGFwcHlQZXJjZW50WiB+IEdMTWZpdHRlZCwgZGF0YT1IYXBweURhdGEpKSRyLnNxdWFyZWQNCmBgYA0KDQotIFdlIGNhbiBmaXQgdGhlIEdMTS9BTk9WQSBlc3RpbWF0ZSBmb3IgZXZlcnkgc2luZ2xlIHN1YmplY3QNCi0gSG93IGlzIG91ciBBTk9WQSBkb2luZyBpbiBwcmVkaWN0aW5nIGVhY2ggc3ViamVjdD8gDQotIEFOT1ZBIGlzIG5vdCBwcmVkaWN0aW5nIHN1YmplY3QgbGV2ZWw7IGl0IGlzIHRlc3RpbmcvcHJlZGljdGluZyBhdCB0aGUgYXZlcmFnZSBsZXZlbCBvZiB0aGUgZGF0YQ0KLSBIb3dldmVyLCBpZiB3ZSBoYXZlIGxvbmdpdHVkaW5hbC1saWtlIGRhdGEgKGxpa2UgaW4gdGhpcyBzdHVkeSkgd2h5IG5vdCBtYWtlIHByZWRpY3Rpb25zIGF0IHRoZSBzdWJqZWN0LWxldmVsPyANCg0KYGBge3J9DQpGaXR0ZWRHbG1QbG90MSA8LWdncGxvdCgpKw0KICBmYWNldF9ncmlkKFN1YmplY3QgfiBTb2NpYWwsIGxhYmVsbGVyPWxhYmVsX2JvdGgpKw0KICBnZW9tX2xpbmUoZGF0YSA9IEhhcHB5RGF0YSwgYWVzKHggPSBUaW1lU3RlcCwgeSA9R0xNZml0dGVkKSkrDQogIGdlb21fcG9pbnQoZGF0YSA9IEhhcHB5RGF0YSwgYWVzKHggPSBUaW1lU3RlcCwgeSA9SGFwcHlQZXJjZW50WiwgZ3JvdXA9U3ViamVjdCxjb2xvdXIgPSBTdWJqZWN0KSxzaXplPTMpKw0KICB4bGFiKCJUaW1lIFN0ZXAiKSt5bGFiKCJIYXBwaW5lc3MiKQ0KRml0dGVkR2xtUGxvdDENCmBgYA0KDQojIFJhbmRvbSBFZmZlY3QgTW9kZWxpbmc6IEZpdHRpbmcgYXQgZGlmZmVyZW50IGxldmVscw0KLSBMaW5lYXIgcmVncmVzc2lvbiBhc3N1bWVzIG9uZSBtZWFzdXJlbWVudCBwZXIgc3ViamVjdCwgYnV0IHdoYXQgaXMgeW91IGhhdmUgbXVsdGlwbGUgbWVhc3VyZW1lbnRzIHBlciBzdWJqZWN0PyANCi0gSWYgd2UgaGF2ZSBsb25naXR1ZGluYWwgZGF0YSBwZXIgc3ViamVjdCB3ZSBjb3VsZCBlc3RpbWF0ZSBhIHNsb3BlIHBlciBzdWJqZWN0IQ0KLSBXZSBjb3VsZCB0aGVuIGV4dHJhY3QgdGhlIHNsb3BlcyBlc3RpbWF0ZXMgcGVyIHN1YmplY3QgW2FuZCBkbyBzdGF0aXN0aWNzIG9uIHRoZSBzbG9wZXMgcGVyIHN1YmplY3RdDQotIEJ1dCB3ZSBsb3NlIHRoZSB2YXJpYW5jZXMgYXJvdW5kIHRob3NlIHNsb3BlcyEgU3VjaCBhcyBiZWxvdy4NCg0KYGBge3J9DQpGaXR0ZWRsbVBsb3QgPC1nZ3Bsb3QoZGF0YSA9IEhhcHB5RGF0YSwgYWVzKHggPSBUaW1lU3RlcCwgeSA9SGFwcHlQZXJjZW50WiwgZ3JvdXA9U3ViamVjdCkpKw0KICBmYWNldF9ncmlkKFN1YmplY3QgfiBTb2NpYWwsIGxhYmVsbGVyPWxhYmVsX2JvdGgpKw0KICBnZW9tX3BvaW50KGFlcyhjb2xvdXIgPSBTdWJqZWN0KSwgc2l6ZT0zKSsNCiAgZ2VvbV9zbW9vdGgobWV0aG9kID0gImxtIixzZT1GQUxTRSwgY29sb3IgPSJibGFjayIsIHNpemU9LjUpICsNCiAgeGxhYigiVGltZSBTdGVwIikreWxhYigiSGFwcGluZXNzIikNCkZpdHRlZGxtUGxvdA0KYGBgDQoNCiMjIFJhbmRvbSBpbnRlcmNlcHRzDQotIFNvIHdlIG5lZWQgdG8gZ2V0IGEgcmVncmVzc2lvbiBtb2RlbCB0byB0cnkgdG8gZml0IGF0IHRoZSBzdWJqZWN0IGxldmVsDQotIGAoMXxTdWJqZWN0KWAgbWVhbnMgZWFjaCBzdWJqZWN0IGNhbiBoYXZlIHRoZWlyIG93biBpbnRlcmNlcHQNCi0gYFJFTUw9RkFMU0VgIG1lYW5zIGZpdCB0aGUgd2l0aCBtYXhpbXVtIGxpa2VsaWhvb2QgW2xpa2Ugd2UgZGlkIHdpdGggR0xNXQ0KLSBZb3UgY2FuIHNlZSB0aGUgcmFuZG9tLCBmaXhlZCB0ZXJtcywgYW5kIGNvcnJlbGF0aW9ucyBiZXR3ZWVuIHRoZSBmaXhlZCB0ZXJtcyANCg0KYGBge3J9DQpNb2RlbC4xPC1sbWVyKEhhcHB5UGVyY2VudFogflNvY2lhbCpUaW1lU3RlcA0KICAgICAgICAgICAgICAgICAgICsoMXxTdWJqZWN0KSwgIA0KICAgICAgICAgICAgICAgICAgIGRhdGE9SGFwcHlEYXRhLCBSRU1MPUZBTFNFKQ0Kc3VtbWFyeShNb2RlbC4xKQ0KYGBgDQoNCg0KLSBOZXh0IHdlIGV4YW1pbmUgdGhlIGludGVyY2VwdHMgZm9yIGVhY2ggcGVyc29uDQotIFRoZSB2YXIgYW5kIHNkIG9mIHRoZXNlIHZhbHVlcyBhcmUgd2hhdCB3YXMgc2VlbiBpbiB0aGUgdGFibGUgYWJvdmUNCg0KYGBge3J9DQpyYW5lZihNb2RlbC4xKQ0KYGBgDQoNCi0gV2UgY2FuIGV4YW1pbmUgYW4gJFJeMiQgZm9yIHRoZSBtb2RlbHMsIGJ1dCB3ZSBoYXZlIGEgcHJvYmxlbQ0KLSBJcyB0aGUgJFJeMiQgY29taW5nIGZyb20gdGhlIGZpeGVkIG9yIHJhbmRvbSBlZmZlY3Q/IFdlIGNhbiBwYXJzZSB0aGF0Og0KLSBUaGUgbWFyZ2luYWwgaXMgdGhlICRSXjIkIGZvciB0aGUgZml4ZWQgZWZmZWN0cw0KLSBUaGUgY29uZGl0aW9uYWwgJFJeMiQgaW5jbHVkZXMgcmFuZG9tICsgZml4ZWQgZWZmZWN0cw0KLSBUaGVyZSBpcyBubyBzdGFuZGFyZCBhcyB0byB3aGljaCB0byByZXBvcnQgJFJeMiQgYW5kIG1vc3QgcGVvcGxlIG5ldmVyIHJlcG9ydCB0aGVzZSBhdCBhbGwgYXMgaWYgdGhlIHJhbmRvbSBlZmZlY3RzIGFyZSBjb21wbGV4IChuZXN0ZWQsIG9yIGNyb3NzZWQpIHRoZSBtZWFuaW5nIG9mIHRoZW0gY2hhbmdlcw0KLSBXZSB3aWxsIGV4cGxvcmUgdGhpcyBtb3JlIG5leHQgd2VlayAoYnV0IHRvZGF5IEkgd2lsbCB1c2UgdGhlbSB0byBoZWxwIG1ha2Ugc2Vuc2Ugb2Ygd2hhdCB0aGUgcmFuZG9tIGVmZmVjdHMgYXJlIGRvaW5nKQ0KDQoNCmBgYHtyfQ0KcnNxdWFyZWQuZ2xtbShNb2RlbC4xKQ0KYGBgDQoNCi0gTGV0J3MgcGxvdCB0aGUgbW9kZWwgZml0DQoNCmBgYHtyfQ0KSGFwcHlEYXRhJE1vZGVsLjEuZml0dGVkPC1wcmVkaWN0KE1vZGVsLjEpDQpGaXR0ZWRsbVBsb3QxIDwtZ2dwbG90KCkrDQogIGZhY2V0X2dyaWQoU3ViamVjdCB+IFNvY2lhbCwgbGFiZWxsZXI9bGFiZWxfYm90aCkrDQogIGdlb21fbGluZShkYXRhID0gSGFwcHlEYXRhLCBhZXMoeCA9IFRpbWVTdGVwLCB5ID1Nb2RlbC4xLmZpdHRlZCkpKw0KICBnZW9tX3BvaW50KGRhdGEgPSBIYXBweURhdGEsIGFlcyh4ID0gVGltZVN0ZXAsIHkgPUhhcHB5UGVyY2VudFosIGdyb3VwPVN1YmplY3QsY29sb3VyID0gU3ViamVjdCksIHNpemU9MykrDQogIHhsYWIoIlRpbWUgU3RlcCIpK3lsYWIoIkhhcHBpbmVzcyIpDQpGaXR0ZWRsbVBsb3QxDQpgYGANCg0KIyMgUmFuZG9tIGludGVyY2VwdCBhbmQgc2xvcGVzIGZvciBzb2NpYWwgY29uZGl0aW9uIA0KLSBNb2RlbCAyIHRyZWF0cyB0aGF0IFNvY2lhbCBjb25kaXRpb24gYXMgZnJlZSB0byB2YXJ5IGJldHdlZW4gU3ViamVjdHMsIGJ1dCBhY2NvdW50cyBmb3IgdGhlIGNvcnJlbGF0aW9uIHdpdGhpbiBTdWJqZWN0cyBiZXR3ZWVuIHRoZSBTb2NpYWwgY29uZGl0aW9uKQ0KLSBgKDErU29jaWFsfFN1YmplY3QpYCBtZWFucyBlYWNoIHN1YmplY3QgY2FuIGhhdmUgdGhlaXIgaW50ZXJjZXB0IGFuZCBzb2NpYWwgY29uZGl0aW9uIGNhbiB2YXJ5IGFzIGZ1bmN0aW9uIG9mIHRoZSBzdWJqZWN0DQotIFRoZSBjb3JyZWxhdGlvbnMgb2YgdGhlIHJhbmRvbSB0ZXJtcyBhcmUgdGhlIGNvcnJlbGF0aW9ucyBvZiBlYWNoIHN1YmplY3RzIGludGVyY2VwdCBhbmQgc2xvcGUgb2Ygc29jaWFsIGNvbmRpdGlvbg0KDQpgYGB7cn0NCk1vZGVsLjI8LWxtZXIoSGFwcHlQZXJjZW50WiB+U29jaWFsKlRpbWVTdGVwDQogICAgICAgICAgICAgICAgICAgICAgICsoMStTb2NpYWx8U3ViamVjdCksICANCiAgICAgICAgICAgICAgICAgICAgICAgZGF0YT1IYXBweURhdGEsIFJFTUw9RkFMU0UpDQpzdW1tYXJ5KE1vZGVsLjIpDQpyYW5lZihNb2RlbC4yKQ0KYGBgDQoNCi0gVGhlICRSXjIkIGZvciBmaXhlZCBhbmQgcmFuZG9tDQoNCmBgYHtyfQ0KcnNxdWFyZWQuZ2xtbShNb2RlbC4yKQ0KYGBgDQoNCi0gUGxvdCB0aGUgbW9kZWwgcmVzdWx0cw0KDQpgYGB7cn0NCkhhcHB5RGF0YSRNb2RlbC4yLmZpdHRlZDwtcHJlZGljdChNb2RlbC4yKQ0KRml0dGVkbG1QbG90MiA8LWdncGxvdCgpKw0KICBmYWNldF9ncmlkKFN1YmplY3QgfiBTb2NpYWwsIGxhYmVsbGVyPWxhYmVsX2JvdGgpKw0KICBnZW9tX2xpbmUoZGF0YSA9IEhhcHB5RGF0YSwgYWVzKHggPSBUaW1lU3RlcCwgeT1Nb2RlbC4yLmZpdHRlZCkpKw0KICBnZW9tX3BvaW50KGRhdGEgPSBIYXBweURhdGEsIGFlcyh4ID0gVGltZVN0ZXAsIHkgPUhhcHB5UGVyY2VudFosIGdyb3VwPVN1YmplY3QsY29sb3VyID0gU3ViamVjdCksIHNpemU9MykrDQogICMgIGNvb3JkX2NhcnRlc2lhbih5bGltID0gYyguMDMsLjA3NCkpKyANCiAgeGxhYigiVGltZSBTdGVwIikreWxhYigiSGFwcGluZXNzIikNCkZpdHRlZGxtUGxvdDINCmBgYA0KDQoNCiMjIFJhbmRvbSBpbnRlcmNlcHRzIGFuZCBzbG9wZXMgZm9yIHRpbWUgIA0KLSBNb2RlbCAzIHRyZWF0cyB0aGF0IHRpbWVzdGVwcyBhcyBmcmVlIHRvIHZhcnkgYmV0d2VlbiBTdWJqZWN0cywgYnV0IGFjY291bnRzIGZvciB0aGUgY29ycmVsYXRpb24gd2l0aGluIFN1YmplY3RzIGFjcm9zcyB0aW1lDQotIGAoMStUaW1lU3RlcHxTdWJqZWN0KWAgbWVhbnMgZWFjaCBzdWJqZWN0IGNhbiBoYXZlIHRoZWlyIGludGVyY2VwdCBhbmQgdGltZSBzbG9wZSBjYW4gdmFyeSBhcyBmdW5jdGlvbiBvZiB0aGUgc3ViamVjdA0KLSBUaGUgY29ycmVsYXRpb25zIG9mIHRoZSByYW5kb20gdGVybXMgYXJlIHRoZSBjb3JyZWxhdGlvbnMgb2YgZWFjaCBzdWJqZWN0cyBpbnRlcmNlcHQgYW5kIHNsb3BlIG9mIHRpbWUgDQoNCmBgYHtyfQ0KTW9kZWwuMzwtbG1lcihIYXBweVBlcmNlbnRaIH5Tb2NpYWwqVGltZVN0ZXANCiAgICAgICAgICAgICAgICAgICAgICAgKygxK1RpbWVTdGVwfFN1YmplY3QpLCAgDQogICAgICAgICAgICAgICAgICAgICAgIGRhdGE9SGFwcHlEYXRhLCBSRU1MPUZBTFNFKQ0Kc3VtbWFyeShNb2RlbC4zKQ0KcmFuZWYoTW9kZWwuMykNCmBgYA0KDQotIFRoZSAkUl4yJCBmb3IgZml4ZWQgYW5kIHJhbmRvbQ0KDQpgYGB7cn0NCnJzcXVhcmVkLmdsbW0oTW9kZWwuMykNCmBgYA0KDQotIFBsb3QgdGhlIG1vZGVsIHJlc3VsdHMNCg0KYGBge3J9DQpIYXBweURhdGEkTW9kZWwuMy5maXR0ZWQ8LXByZWRpY3QoTW9kZWwuMykNCkZpdHRlZGxtUGxvdDMgPC1nZ3Bsb3QoKSsNCiAgZmFjZXRfZ3JpZChTdWJqZWN0IH4gU29jaWFsLCBsYWJlbGxlcj1sYWJlbF9ib3RoKSsNCiAgZ2VvbV9saW5lKGRhdGEgPSBIYXBweURhdGEsIGFlcyh4ID0gVGltZVN0ZXAsIHkgPU1vZGVsLjMuZml0dGVkKSkrDQogIGdlb21fcG9pbnQoZGF0YSA9IEhhcHB5RGF0YSwgYWVzKHggPSBUaW1lU3RlcCwgeSA9SGFwcHlQZXJjZW50WiwgZ3JvdXA9U3ViamVjdCxjb2xvdXIgPSBTdWJqZWN0KSwgc2l6ZT0zKSsNCiAgIyAgY29vcmRfY2FydGVzaWFuKHlsaW0gPSBjKC4wMywuMDc0KSkrIA0KICB4bGFiKCJUaW1lIFN0ZXAiKSt5bGFiKCJIYXBwaW5lc3MiKQ0KRml0dGVkbG1QbG90Mw0KYGBgDQoNCg0KIyMgUmFuZG9tIGludGVyY2VwdHMgYW5kIHNsb3BlcyBmb3IgdGltZSBhbmQgc29jaWFsIChidXQgaW5kZXBlbmRlbnQpICANCi0gTW9kZWwgNCB0cmVhdHMgdGhhdCB0aW1lc3RlcHMgYW5kIFNvY2lhbCBhcyBmcmVlIHRvIHZhcnkgYmV0d2VlbiBTdWJqZWN0cyBidXQgYWNjb3VudHMgZm9yIHRoZSBjb3JyZWxhdGlvbiB3aXRoaW4gU3ViamVjdHMgYWNyb3NzIHRpbWVzIGFuZCBzb2NpYWwNCi0gYCgxK1NvY2lhbHxTdWJqZWN0KSsoMCtUaW1lU3RlcHxTdWJqZWN0KWAgbWVhbnMgZWFjaCBzdWJqZWN0IGNhbiBoYXZlIHRoZWlyIGludGVyY2VwdCBhbmQgY2FuIGhhdmUgdGhlaXIgc2xvcGUgb2Ygc29jaWFsIGNvbmRpdGlvbg0KLSBBbHNvLCB0aW1lIHNsb3BlIGNhbiB2YXJ5IGFzIGZ1bmN0aW9uIG9mIHRoZSBzdWJqZWN0LCBidXQgaXQgdHJlYXRzIHRoZSB2YXJpYW5jZSBiZXR3ZWVuIHNvY2lhbCBhbmQgdGltZSBhcyAqKmluZGVwZW5kZW50KioNCg0KYGBge3J9DQpNb2RlbC40PC1sbWVyKEhhcHB5UGVyY2VudFogflNvY2lhbCpUaW1lU3RlcA0KICAgICAgICAgICAgICAgICAgICsoMStTb2NpYWx8U3ViamVjdCkrKDArVGltZVN0ZXB8U3ViamVjdCksICANCiAgICAgICAgICAgICAgICAgICBkYXRhPUhhcHB5RGF0YSwgUkVNTD1GQUxTRSkNCnN1bW1hcnkoTW9kZWwuNCkNCnJhbmVmKE1vZGVsLjQpDQpgYGANCg0KLSBUaGUgJFJeMiQgZm9yIGZpeGVkIGFuZCByYW5kb20NCg0KYGBge3J9DQpyc3F1YXJlZC5nbG1tKE1vZGVsLjQpDQpgYGANCg0KLSBQbG90IHRoZSBtb2RlbCByZXN1bHRzDQoNCmBgYHtyfQ0KSGFwcHlEYXRhJE1vZGVsLjQuZml0dGVkPC1wcmVkaWN0KE1vZGVsLjQpDQpGaXR0ZWRsbVBsb3Q0IDwtZ2dwbG90KCkrDQogIGZhY2V0X2dyaWQoU3ViamVjdCB+IFNvY2lhbCwgbGFiZWxsZXI9bGFiZWxfYm90aCkrDQogIGdlb21fbGluZShkYXRhID0gSGFwcHlEYXRhLCBhZXMoeCA9IFRpbWVTdGVwLCB5ID1Nb2RlbC40LmZpdHRlZCkpKw0KICBnZW9tX3BvaW50KGRhdGEgPSBIYXBweURhdGEsIGFlcyh4ID0gVGltZVN0ZXAsIHkgPUhhcHB5UGVyY2VudFosIGdyb3VwPVN1YmplY3QsY29sb3VyID0gU3ViamVjdCksIHNpemU9MykrDQogICMgIGNvb3JkX2NhcnRlc2lhbih5bGltID0gYyguMDMsLjA3NCkpKyANCiAgeGxhYigiVGltZSBTdGVwIikreWxhYigiSGFwcGluZXNzIikNCkZpdHRlZGxtUGxvdDQNCmBgYA0KDQojIyBSYW5kb20gaW50ZXJjZXB0cyBhbmQgc2xvcGVzIGZvciB0aW1lIGFuZCBzb2NpYWwgKGJ1dCBub3QgaW5kZXBlbmRlbnQpIA0KLSBNb2RlbCA1IHRyZWF0cyB0aGF0IHRpbWVzdGVwcyBhbmQgU29jaWFsIGFzIGZyZWUgdG8gdmFyeSBiZXR3ZWVuIFN1YmplY3RzIGJ1dCBhY2NvdW50cyBmb3IgdGhlIGNvcnJlbGF0aW9uIHdpdGhpbiBTdWJqZWN0cyBhY3Jvc3MgdGltZSBhbmQgU29jaWFsDQotIGAoMStTb2NpYWwrIFRpbWVTdGVwfFN1YmplY3QpYCBtZWFucyBlYWNoIHN1YmplY3QgY2FuIGhhdmUgdGhlaXIgaW50ZXJjZXB0IGFuZCBjYW4gaGF2ZSB0aGVpciBzbG9wZSBvZiBzb2NpYWwgY29uZGl0aW9uDQotIEFsc28sIHRpbWUgc2xvcGUgY2FuIHZhcnkgYXMgZnVuY3Rpb24gb2YgdGhlIHN1YmplY3QsIGJ1dCBpdCB0cmVhdHMgdGhlIHZhcmlhbmNlIGJldHdlZW4gc29jaWFsIGFuZCB0aW1lIGFzICoqY29ycmVsYXRlZCoqDQoNCmBgYHtyfQ0KTW9kZWwuNTwtbG1lcihIYXBweVBlcmNlbnRaIH5Tb2NpYWwqVGltZVN0ZXANCiAgICAgICAgICAgICAgICAgICAgICAgKygxK1NvY2lhbCtUaW1lU3RlcHxTdWJqZWN0KSwgIA0KICAgICAgICAgICAgICAgICAgICAgICBkYXRhPUhhcHB5RGF0YSwgUkVNTD1GQUxTRSkNCnN1bW1hcnkoTW9kZWwuNSkNCnJhbmVmKE1vZGVsLjUpDQpgYGANCg0KLSBUaGUgJFJeMiQgZm9yIGZpeGVkIGFuZCByYW5kb20NCg0KYGBge3J9DQpyc3F1YXJlZC5nbG1tKE1vZGVsLjUpDQpgYGANCg0KLSBQbG90IHRoZSBtb2RlbCByZXN1bHRzDQoNCmBgYHtyfQ0KSGFwcHlEYXRhJE1vZGVsLjUuZml0dGVkPC1wcmVkaWN0KE1vZGVsLjUpDQpGaXR0ZWRsbVBsb3Q1IDwtZ2dwbG90KCkrDQogIGZhY2V0X2dyaWQoU3ViamVjdCB+IFNvY2lhbCwgbGFiZWxsZXI9bGFiZWxfYm90aCkrDQogIGdlb21fbGluZShkYXRhID0gSGFwcHlEYXRhLCBhZXMoeCA9IFRpbWVTdGVwLCB5ID1Nb2RlbC41LmZpdHRlZCkpKw0KICBnZW9tX3BvaW50KGRhdGEgPSBIYXBweURhdGEsIGFlcyh4ID0gVGltZVN0ZXAsIHkgPUhhcHB5UGVyY2VudFosIGdyb3VwPVN1YmplY3QsY29sb3VyID0gU3ViamVjdCksIHNpemU9MykrDQogICMgIGNvb3JkX2NhcnRlc2lhbih5bGltID0gYyguMDMsLjA3NCkpKyANCiAgeGxhYigiVGltZSBTdGVwIikreWxhYigiSGFwcGluZXNzIikNCkZpdHRlZGxtUGxvdDUNCmBgYA0KDQoNCg0KIyMgUmFuZG9tIGludGVyY2VwdHMgYW5kIHNsb3BlcyBmb3IgdGltZSB4IHNvY2lhbA0KLSBNb2RlbCA2IHRyZWF0cyB0aGF0IHRpbWVzdGVwcywgU29jaWFsIGFuZCB0aW1lc3RlcHM6U29jaWFsIChpbnRlcmFjdGlvbikgYXMgZnJlZSB0byB2YXJ5IGJldHdlZW4gU3ViamVjdHMgYnV0IGFjY291bnRzIGZvciB0aGUgY29ycmVsYXRpb24gd2l0aGluIFN1YmplY3RzIGFjcm9zcyB0aW1lIGFuZCBTb2NpYWwNCi0gYCgxK1NvY2lhbCpUaW1lU3RlcHxTdWJqZWN0KWAgbWVhbnMgYCgxK1NvY2lhbCtUaW1lU3RlcCtTb2NpYWw6VGltZVN0ZXB8U3ViamVjdClgIGVhY2ggc3ViamVjdCBjYW4gaGF2ZSB0aGVpciBpbnRlcmNlcHQsIHRoZWlyIHNsb3BlIG9mIHNvY2lhbCBjb25kaXRpb24sIHRoZWlyIHRpbWUgc2xvcGUsIGFuZCB0aGVpciBpbnRlcmFjdGlvbiBiZXR3ZWVuIHNvY2lhbCBhbmQgdGltZQ0KLSBBbHNvLCBhbGwgcmFuZG9tIHNsb3BlcyBhbmQgaW50ZXJjZXB0cyBjYW4gYmUgKipjb3JyZWxhdGVkKioNCi0gVGhpcyBpcyBjYWxsZWQgdGhlICptYXhpbXVtIG1vZGVsKiBCYXJyIGV0IGFsLiwgMjAxMyAodGhlc2UgZG8gbm90IGFsd2F5cyBmaXQgYmVzdCBzZWUgQmF0ZXMgZXQgYWwuLCAyMDE1KQ0KDQpgYGB7cn0NCk1vZGVsLjY8LWxtZXIoSGFwcHlQZXJjZW50WiB+U29jaWFsKlRpbWVTdGVwDQogICAgICAgICAgICAgICAgICAgKygxK1NvY2lhbCpUaW1lU3RlcHxTdWJqZWN0KSwgIA0KICAgICAgICAgICAgICAgICAgIGRhdGE9SGFwcHlEYXRhLCBSRU1MPUZBTFNFKQ0Kc3VtbWFyeShNb2RlbC42KQ0KcmFuZWYoTW9kZWwuNikNCmBgYA0KDQotIFRoZSAkUl4yJCBmb3IgZml4ZWQgYW5kIHJhbmRvbQ0KDQpgYGB7cn0NCnJzcXVhcmVkLmdsbW0oTW9kZWwuNikNCmBgYA0KDQotIFBsb3QgdGhlIG1vZGVsIHJlc3VsdHMNCg0KYGBge3J9DQpIYXBweURhdGEkTW9kZWwuNi5maXR0ZWQ8LXByZWRpY3QoTW9kZWwuNikNCkZpdHRlZGxtUGxvdDYgPC1nZ3Bsb3QoKSsNCiAgZmFjZXRfZ3JpZChTdWJqZWN0IH4gU29jaWFsLCBsYWJlbGxlcj1sYWJlbF9ib3RoKSsNCiAgZ2VvbV9saW5lKGRhdGEgPSBIYXBweURhdGEsIGFlcyh4ID0gVGltZVN0ZXAsIHkgPU1vZGVsLjYuZml0dGVkKSkrDQogIGdlb21fcG9pbnQoZGF0YSA9IEhhcHB5RGF0YSwgYWVzKHggPSBUaW1lU3RlcCwgeSA9SGFwcHlQZXJjZW50WiwgZ3JvdXA9U3ViamVjdCxjb2xvdXIgPSBTdWJqZWN0KSwgc2l6ZT0zKSsNCiAgIyAgY29vcmRfY2FydGVzaWFuKHlsaW0gPSBjKC4wMywuMDc0KSkrIA0KICB4bGFiKCJUaW1lIFN0ZXAiKSt5bGFiKCJIYXBwaW5lc3MiKQ0KRml0dGVkbG1QbG90Ng0KYGBgDQoNCg0KDQojIyBIb3cgdG8gc2VsZWN0IGEgcmFuZG9tIHN0cnVjdHVyZQ0KLSBGaXJzdCwgdGhlc2UgbW9kZWxzIG9mdGVuIGZhaWwgdG8gY29udmVyZ2UNCi0gQ29udmVyZ2VuY2UgbWVhbiB0aGF0IHRoZSAqKm9wdGltaXplcioqIChudW1lcmljYWwgbWV0aG9kIHRvIGZpbmQgdGhlIHBhcmFtZXRlcnMpIGhhcyBmYWlsZWQgdG8gcmVhY2ggYW4gb3B0aW1hbCBvciBzdGFibGUgc29sdXRpb24NCi0gVGhlcmUgYXJlIG1hbnkgb3B0aW1pemVycyB0byBzZWxlY3QgZnJvbSBpbiBSIGlmIHlvdSBuZWVkIHRvIGNoYW5nZSBpdCAoaHR0cHM6Ly9yc3R1ZGlvLXB1YnMtc3RhdGljLnMzLmFtYXpvbmF3cy5jb20vMzM2NTNfNTdmYzdiOGU1ZDQ4NGM5MDliNjE1ZDg2MzNjMDFkNTEuaHRtbCkNCi0gRmFpbHVyZXMgbWlnaHQgbm90IGp1c3QgYmUgYSBudW1lcmljYWwgb3B0aW1pemF0aW9uIHByb2JsZW0sIGJ1dCBhIHByb2JsZW0gd2l0aCB5b3VyIHNwZWNpZmljYXRpb24gb2YgdGhlIHJhbmRvbSBlZmZlY3RzIA0KLSBZb3UgY291bGQgb3Zlci1zcGVjaWZ5IHlvdXIgcmFuZG9tIGVmZmVjdHMgaW4gYSB3YXkgdGhhdCBpcyBub3QgKnBhcnNpbW9uaW91cyogd2l0aCB0aGUgdHJ1ZSByYW5kb20gZWZmZWN0cyBpbiB0aGUgcG9wdWxhdGlvbg0KLSBZb3UgY291bGQgc2V0IHRoZW0gdXAgaW4gc3VjaCBhIHdheSB0aGF0IGRvIG5vdCByZWZsZWN0IHRoZSBleHBlcmltZW50IA0KDQojIyMgUHJhY3RpY2FsIFNvbHV0aW9uIHRvIHJhbmRvbSBzdHJ1Y3R1cmUNCi0gVGhpcyBpcyBhIGJhdHRsZWZpZWxkIGluIExNTQ0KLSBUaGUgY3VycmVudCBiZXN0IHByYWN0aWNlIGlzIHRvIHRyeSB0byBtYXRjaCB5b3VyIGRlc2lnbiBhcyBiZXN0IGFzIHlvdSBjYW4NCi0gSWYgdGhhdCBmYWlscywgdHJ5IGRpZmZlcmVudCBvcHRpbWl6ZXINCi0gSWYgdGhhdCBmYWlscywgcmVkdWNlIHRoZSBjb21wbGV4aXR5IG9mIHRoZSByYW5kb20gZWZmZWN0IChyZW1vdmUgcmFuZG9tIGNvcnJlbGF0aW9ucyBvbmUgYXQgYSB0aW1lKQ0KLSBJZiB0aGF0IGZhaWxzLCBmaWd1cmUgb3V0IHdoaWNoIHRlcm0gaXMgY2F1c2luZyB0aGUgcHJvYmxlbSBhbmQgcmVtb3ZlIGl0DQotIElmIHlvdSB1bmRlci1zcGVjaWZ5LCB5b3UgaW5mbGF0ZSB5b3VyIHR5cGUgSSBlcnJvciwgc28gaXQgYSBiYWxhbmNpbmcgYWN0DQoNCg0KIyBGaXhlZCBlZmZlY3RzDQotIEp1c3QgbGlrZSB5b3VyIGxpbmVhciByZWdyZXNzaW9uIHlvdSBjYW4gcnVuIHRoZXNlIG1vZGVscyBoaWVyYXJjaGljYWxseSwgb25jZSB0aGUgcmFuZG9tIGVmZmVjdHMgYXJlIGluIHBsYWNlDQotIHlvdSBqdXN0IHRlc3QgYmV0d2VlbiBtb2RlbCBmaXRzIHVzaW5nIGEgZGV2aWVuY2UgdGVzdCAobGlrZSB3ZSBkaWQgd2l0aCBHTE0pIGBhbm92YShtb2RlbC4xLCBtb2RlbC4yKWANCg0KIyMgUGxvdCB0aGUgZmluYWwgbW9kZWwgY29ycmVjdGx5DQotIFRoZSBtZXRob2QgSSB1c2VkIHdhcyB0byBwcmVkaWN0IHRoZSBkYXRhLCBidXQgdGhhdCB3b3JrcyB3ZWxsIGlmIHlvdSBoYXZlIG5vIGNvdmFyaWF0ZXMNCi0gSWYgeW91IGhhdmUgY292YXJpYXRlcywgaXRzIGJlc3QgdG8gdXNlIHRoZSBlZmZlY3RzIHBhY2thZ2UganVzdCBsaWtlIHdlIGRpZCB3aXRoIGxpbmVhciByZWdyZXNzaW9uDQotIEFsc28geW91IGhhdmUgdG8gZGVjaWRlIHRvIHBsb3QgYm90aCBmaXhlZCArIHJhbmRvbSBvciBqdXN0IHRoZSBmaXhlZCBlZmZlY3RzDQotIE1vc3QgcGVvcGxlIG9ubHkgcGxvdCB0aGUgZml4ZWQgZWZmZWN0LCBhcyB0aGV5IHdhbnQgdG8gY29udHJvbCBmb3IgdGhlIHJhbmRvbSBlZmZlY3RzIA0KLSBIb3dldmVyLCBpZiBzb21lb25lIGFza3MgeW91IHRvIHBsb3QgdGhlIGRhdGEgYW5kIHRoZSBtb2RlbCBmaXRzIHRvZ2V0aGVyLCB5b3Ugd2lsbCBuZWVkIHRvIHRlbGwgdGhlbSBpdCBjb21wbGljYXRlZCAoYXMgeW91IGFyZSBmb3JjZWQgdG8gcGxvdCBmaXhlZCBhbmQgcmFuZG9tLiBCYXNpY2FsbHkgeW91IHVzZSB0aGUgY29kZSBhYm92ZSBJIGhhdmUgYmVlbiBzaG93aW5nIHlvdSkNCi0gQmVsb3cgaXMgaG93IHRvIHBsb3QgZml4ZWQgb25seSAoYW5kIGNvbnRyb2xpbmcgZm9yIGNvdmFyaWF0ZXMgaWYgeW91IHRoZW0pDQotIEkgaGF2ZSB1c2VkIGdncGxvdCB0aGlzIHRpbWUgKHVubGlrZSBvdXIgcGFzdCBsZWN0dXJlcyB0byBtYWtlIGl0IHByZXR0eSkNCg0KYGBge3IsZmlnLmhlaWdodD00LGZpZy53aWR0aD00fQ0KbGlicmFyeShlZmZlY3RzKQ0KRmluYWwuRml4ZWQ8LWVmZmVjdChjKCJTb2NpYWwqVGltZVN0ZXAiKSwgTW9kZWwuNiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgIHhsZXZlbHM9bGlzdChUaW1lU3RlcD1zZXEoMCw0LDEpKSkNCg0KIyBZb3UgaGF2ZSB0byBjb252ZXJ0IHRoZSBvdXRwdXQgdG8gYSBkYXRhZnJhbWUNCkZpbmFsLkZpeGVkPC1hcy5kYXRhLmZyYW1lKEZpbmFsLkZpeGVkKQ0KDQpGaW5hbC5GaXhlZC5QbG90IDwtZ2dwbG90KGRhdGEgPSBGaW5hbC5GaXhlZCwgYWVzKHggPSBUaW1lU3RlcCwgeSA9Zml0LCBncm91cD1Tb2NpYWwpKSsNCiAgY29vcmRfY2FydGVzaWFuKHhsaW09YygwLDQpLHlsaW0gPSBjKDAsLjcpKSsgDQogIGdlb21fbGluZShhZXMoY29sb3I9U29jaWFsKSwgc2l6ZT0yKSsNCiAgZ2VvbV9yaWJib24oYWVzKHltaW49Zml0LXNlLCB5bWF4PWZpdCtzZSxmaWxsPVNvY2lhbCksYWxwaGE9LjIpKw0KICB4bGFiKCJUaW1lIikrDQogIHlsYWIoIkhhcHBpbmVzc1xuc2NvcmUiKSsNCiAgc2NhbGVfY29sb3JfbWFudWFsKHZhbHVlcz1jKCJibHVlIiwgInJlZCIpKSsNCiAgc2NhbGVfZmlsbF9tYW51YWwodmFsdWVzPWMoImJsdWUiLCAicmVkIikpKw0KICB0aGVtZV9idygpKw0KICB0aGVtZSh0ZXh0PWVsZW1lbnRfdGV4dChmYWNlPSJib2xkIiwgc2l6ZT0xMiksDQogICAgICAgIHBhbmVsLmdyaWQubWFqb3IgPSBlbGVtZW50X2JsYW5rKCksIA0KICAgICAgICBwYW5lbC5ncmlkLm1pbm9yID0gZWxlbWVudF9ibGFuaygpLCANCiAgICAgICAgcGFuZWwuYm9yZGVyID0gZWxlbWVudF9yZWN0KGZpbGwgPSBOQSwgY29sb3VyID0gIk5BIiksDQogICAgICAgIGF4aXMubGluZSA9IGVsZW1lbnRfbGluZShzaXplID0gMSwgY29sb3VyID0gImdyZXk4MCIpLA0KICAgICAgICBsZWdlbmQudGl0bGU9ZWxlbWVudF9ibGFuaygpLA0KICAgICAgICBsZWdlbmQucG9zaXRpb24gPSBjKC4yLCAuOTIpKQ0KRmluYWwuRml4ZWQuUGxvdA0KDQpgYGANCg0KDQojIFAtdmFsdWVzDQotIFB2YWx1ZXMgYXJlIGhvdGx5IGRpc3B1dGVkIGluIE1peGVkIG1vZGVscy4gV2h5Pw0KLSBOdWxsLWh5cG90aGVzaXMgdGVzdGluZyBhbmQgbWl4ZWQgbW9kZWxzIGFyZSBjdXJyZW50IG1ham9yIGJhdHRsZWZpZWxkDQotIEdMTSB1c2VzIHN0YW5kYXJkIGVycm9yIHRvIGRldGVybWluZSBzaWduaWZpY2FuY2U6IFJlcXVpcmVkIGRlZ3JlZXMgb2YgZnJlZWRvbSB0byBjb3JyZWN0IHotZGlzdHJpYnV0aW9uIGZvciAic2FtcGxpbmcgZXJyb3IiIHRvIHByb2R1Y2UgdC9GIHRhYmxlcy4gIA0KLSBBc3N1bWVzIG5vcm1hbGl0eQ0KLSAiTW9kZXJuIiBtZXRob2RzIFtzdWNoIGFzIGJvb3RzdHJhcHBpbmddIGZvciBwdmFsdWUgZGV0ZXJtaW5hdGlvbiBoYXZlIGZvY3VzZWQgb24gd29ya2luZyB0b3dhcmQgZGlzdHJpYnV0aW9uIGFwcHJvcHJpYXRlIGFzc3VtcHRpb25zIA0KLSBCYXllc2lhbidzIHVzZSBtb3JlIGNvbXBsZXggZm9ybXMgb2YgYm9vdHN0cmFwcGluZyAoTUNNQykgdG8gZnVydGhlciByZWZpbmUgdGhpcyBtZXRob2RzDQotIGZvciBNaXhlZCBNb2RlbHMgd2UgaGF2ZSBEZWdyZWVzIG9mIGZyZWVkb20gcHJvYmxlbSBmb3IgdGhlIGVycm9yIHRlcm0uICBXaHk/DQoNCiMjIEV4YW1wbGUgb2YgREYgcGFyc2luZyAgDQotIFNhbXBsZSA1MDAsIDExdGhlIGdyYWRlcnMgdGFrZSB0aGUgU0FUIGZyb20gMyBzY2hvb2xzIG5lc3RlZCBpbiAyIGVjb25vbWljIGRpc3RyaWN0cyAgDQoNCmBgYHtyLCBlY2hvID0gRkFMU0UsIG91dC53aWR0aCA9ICIzMDBweCJ9DQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiUmVncmVzc2lvbkNsYXNzL0wxMl9NaXhlZC9WZW5uU2Nob29scy5qcGciKQ0KYGBgDQoNCg0KLSBSYW5kb20gc2FtcGxpbmcgb2YgNTAwIHRvdGFsIGZyb20gMyBzY2hvb2xzIFtub3RlLCBzY2hvb2wgMyBpcyB0d2ljZSBhcyBsYXJnZSBzY2hvb2wgMSBhbmQgMl0gDQotIEFOT1ZBIERlc2lnbjogMSBmYWN0b3I6IFNjaG9vbCBPUiBkaXN0cmljdCBbdGhleSBwcm92aWRlIGFuIHVuYmFsYW5jZWQgZGVzaWduIHNvIHRoZXkgY2FuIG9ubHkgYW5hbHl6ZWQgb25lIGF0IHRpbWVdDQotIEFzc3VtZSBob21vZ2VuZWl0eSBvZiB2YXJpYW5jZSBvZiBTQVQgc2NvcmUgYWNyb3NzIHNjaG9vbHMgYW5kIGRpc3RyaWN0cyBbYmFkIGFzc3VtcHRpb25dDQotIEFOT1ZBIGNhbiBvbmx5IGFzayAxIGJhc2ljIHF1ZXN0aW9uOiBBcmUgdGhlIG1lYW5zIGJldHdlZW4gdGhlIHNjaG9vbHMgb3IgZGlzdHJpY3RzIGRpZmZlcmVudA0KDQojIyMgQU5PVkEgcHJvYmxlbT8NCi0gV2h5IGlzIEFOT1ZBIGJhZCBoZXJlPyANCi0gU2Nob29sIDEgaXMgcHJlZG9taW5hdGVseSBwb29yIHZzIFNjaG9vbCAyIHdoaWNoIGlzIHJpY2gsIHNvIHNjaG9vbCAxIGhhcyBsZXNzIGZ1bmRpbmcsIGxhcmdlciBjbGFzc3Jvb21zIHNpemVzLCBtb3JlIHNvY2lhbCBpc3N1ZXMgYXQgaG9tZSBhbmQgaW4gc2Nob29sLCBsZXNzIHRlYWNoZXJzLCBldGMuIA0KLSBUaGlzIGNvdWxkIGltcGFjdCBib3RoIG1lYW4gdGVzdCBzY29yZXMgYW5kIHZhcmlhbmNlIG9mIHRlc3Qgc2NvcmVzICANCi0gU3R1ZGVudHMgZnJvbSB0aGUgcG9vciBkaXN0cmljdCBtaWdodCBoYXZlIG1vcmUgZGl2ZXJzZSBzY29yZXMgW2hpZ2ggdmFyaWFuY2VdDQotIFN0dWRlbnQgZnJvbSByaWNoIGRpc3RyaWN0IG1pZ2h0IGhhdmUgbW9yZSBzaW1pbGFyIHNjb3JlcywgYnV0IGhpZ2hlciBhdmVyYWdlIHNjb3JlcyBbbG93IHZhcmlhbmNlXQ0KLSBQb29yIHN0dWRlbnQgaW4gdGhlIFJpY2ggc2Nob29sIG1pZ2h0IGFsc28gaGF2ZSBjaGFuZ2VzIGluIHZhcmlhbmNlIG9yIG1lYW4gdGVzdCBzY29yZXMsIHdoaWNoIGxvb2sgdmVyeSBkaWZmZXJlbnQgZnJvbSB0aGUgcG9vciBvciBtaXhlZCBzY2hvb2wuICANCg0KIyMjIE1peGVkIFNvbHV0aW9uDQotIE1peGVkIG1vZGVsIHNvbHV0aW9uIHRvIHNhbWUgZXhhY3QgZGF0YSBzZXQ/DQotIEZpeGVkIGZhY3RvcnM6ICBTY2hvb2wgYW5kIERpc3RyaWN0IFtzdHVkZW50IGJlbG9uZ3MgdG9dDQotIFNBVCBzY29yZSBNRUFOUyBjb3VsZCBiZSBkaWZmZXJlbnQgZGVwZW5kaW5nIG9uIHRoZSBzY2hvb2wgYSBraWQgZ29lcyB0byBhbmQgd2hhdCBkaXN0cmljdCB0aGV5IGFyZSBmcm9tLiAgIA0KLSBSYW5kb20gRmFjdG9yczogIERpc3RyaWN0LCBTY2hvb2wsIFN0dWRlbnQgDQotIFNBVCBzY29yZSBWQVJJQU5DRVMgY291bGQgYmUgZGlmZmVyZW50IGRlcGVuZGluZyBvbiB0aGUgc2Nob29sIGEga2lkIGdvZXMgdG8gYW5kIHdoYXQgZGlzdHJpY3QgdGhleSBhcmUgZnJvbS4gICANCi0gVGhpcyBhbGxvd3MgZWFjaCBraWQgImdyb3VwIiB0byBoYXZlIGl0cyBvd24gbWVhbiBhbmQgdmFyaWFuY2U6DQotIGNhbGN1bGF0ZWQgYWJvdXQgdGhlaXIgY2xhc3NpZmljYXRpb24gYmFzZWQgb24gc2Nob29sIGFuZCBkaXN0cmljdCBbc28gZXJyb3IgdGVybSB2YXJpZXMgYXMgYSBwcm9kdWN0IG9mIHRoaXMgY3VzdG9tIGNsYXNzaWZpY2F0aW9uXSANCi0gU28ga2lkcyBmcm9tIHRoZSBzY2hvb2wgMSwgcG9vciBkaXN0cmljdCBhcmUgImFsbG93ZWQiIHRvIGRpZmZlciBmcm9tIGtpZHMgZnJvbSBzY2hvb2wgMSwgcmljaCBkaXN0cmljdCBmcm9tIHNjaG9vbCAyIHBvb3IgZGlzdHJpY3QsIGV0Yy4uIA0KLSBJbiB0aGUgY2FzZSB3aGVuIGFsbCB0aGUgdmFyaWFibGVzIGFyZSBjYXRlZ29yaWNhbCwgdGhlIERGICJzZWVtcyIgbG9naWNhbCwgaW4gdGhhdCB0aGUgcHJvcGVyIE1TZXJyb3IgdGVybXMgY291bGQgYmUgY2FsY3VsYXRlZC4gSG93ZXZlciB0aGUgcmFuZG9tIGZhY3RvcnMgaGF2ZSBiZWVuIG5lc3RlZC4gDQotIFNvIHdlIGNhbm5vdCBrbm93IGhvdyBtYW55IG9mIHRoZSBwZW9wbGUgd2VudCBpbnRvIHBhcnNpbmcgdGhlIHZhcmlhbmNlIGZvciBlYWNoIGxldmVsIG9mIHRoZSByYW5kb20gZXJyb3IgdGVybS4gIA0KLSBXZSBjb3VsZCBtYWtlIGFzc3VtcHRpb25zLCBidXQgc29tZSBzdGF0aXN0aWNpYW5zIHNheSBvdmVyIHRoZWlyIGRlYWQgYm9kaWVzLiANCi0gQU5PVkEgaXMgYXNzdW1lZCB0byBiZSByb2J1c3QgYWdhaW5zdCBtYW55IHZpb2xhdGlvbnMsIGJ1dCB0aGF0J3MgbW9zdGx5IG9ubHkgdHJ1ZSBpbiB3ZWxsIGNvbnRyb2xsZWQgYmFsYW5jZWQgZGVzaWducy4gICANCi0gV2hlbiBBTk9WQSBpcyBzdHJldGNoZWQgaW50byBtaXhlZCBtb2RlbCB0ZXJyaXRvcnksIGl0IGNhbiByZXN1bHQgaW4gdW5yZWFzb25hYmxlIGxldmVscyBvZiB0eXBlIDEgZXJyb3IuICANCi0gU28gd2hhdCBkbyB3ZSBkbz8gDQoNCg0KIyMgR2V0dGluZyBQdmFsdWVzDQojIyMgUGF0aCBBOiBEb24ndCBwdXNoIHlvdXIgYXNzdW1wdGlvbnMgb24gbWUhDQoNCjEuICBNb2RlbGluZyBGaXR0aW5nIHRlc3RpbmcgDQotIExpa2VsaWhvb2QgcmF0aW8gdGVzdCBbY2hpLXNxdWFyZSBkaXN0cmlidXRpb25dDQotIEQgPSAyIC0gbG4gKExpa2VsaWhvb2Qgb2YgbnVsbCBtb2RlbCAvIExpa2VsaWhvb2Qgb2YgYWx0ZXJuYXRpdmUgbW9kZWxdIA0KLSBBZGQvUmVtb3ZlIGZpeGVkIGZhY3RvcnMgYW5kIHRlc3QgdG8gc2VlIGlmIG1vZGVsICJmaXRzIHRoZSBkYXRhIGJldHRlciIgDQotIFN1Z2dlc3RlZCBieSBCYXRlcyBhbmQgQm9sa2VyIFtsZWFzdCBhc3N1bXB0aW9uc10NCi0gRkFTVCBhbmQgd2VsbCBhY2NlcHRlZCBbYSBiaXQgYW50aS1jb25zZXJ2YXRpdmUgc29tZSBhcmd1ZV0NCi0gdG8gZG8gdGhpcyB5b3Ugc2ltcGx5IGhhdmUgdG8gYGFub3ZhKG1vZGVsLjEsIG1vZGVsLjIpYCwganVzdCBsaWtlIHdlIGRpZCB3aXRoIHRlc3RpbmcgYmV0d2VlbiAkUl4yJCBiZXR3ZWVuIG1vZGVsIGZpdHMNCg0KMi4gQm9vdHN0cmFwcGluZyBvZiBDb25maWRlbmNlIEludGVydmFscyBbY2FuIGJlIHBhcmFtZXRyaWMgb3Igc2VtaS1wYXJhbWV0cmljXQ0KLSBSZS1zYW1wbGluZyBtZXRob2QuIFVzZXMgdGhlIGRpc3RyaWJ1dGlvbnMgaW4geW91ciBkYXRhIHRvIGRldGVybWluZSBjb25maWRlbmNlIGludGVydmFscw0KLSBDb21tb24gbWV0aG9kIGluIG1vZGVybiBzdGF0cy4gQSBtdWNoIGJldHRlciBtZXRob2QgdGhlbiBhc3N1bWluZyBub3JtYWxpdHkgYW5kIHQtZGlzdHJpYnV0aW9ucyANCi0gVkVSWSwgVkVSWSBTTE9XLiAgDQotIFBhcg0KYGBge3J9DQojIHBhcmFtZXRyaWMNCk1vZGVsLkZpbmFsLkNJLmJvb3QgPC0gY29uZmludChNb2RlbC42LCBtZXRob2Q9ImJvb3QiLG5zaW09MTAwLGJvb3QudHlwZSA9IGMoInBlcmMiKSkgDQpNb2RlbC5GaW5hbC5DSS5ib290DQoNCmBgYA0KDQojIyMgUGF0aCBCOiAgTGV0J3MgaG9sZCBoYW5kcywgcHV0IG91ciBmaW5nZXJzIGluIG91ciBlYXJzLCBhbmQgYXNzdW1lISANCg0KMy4gQXNzdW1lIHogZGlzdHJpYnV0aW9ucyB0byBnZXQgcHZhbHVlcw0KLSBWZXJ5LCB2ZXJ5IGNvbW1vbiBhcHByb2FjaA0KLSBJZiB5b3UgbnVtYmVyIG9mIG9ic2VydmF0aW9ucyBpcyBoaWdoLCBqdXN0IGFzc3VtZSBERiAtPiBpbmYgW3NvIHQgPiAxLjk2LCBpcyBzaWduaWZpY2FudF0NCi0gVGhpcyBpcyBwb3NzaWJsZSBiZWNhdXNlIHlvdSBhc3N1bWUgeW91ciByYW5kb20gZWZmZWN0cyBoYXZlIGNvbnRyb2wgZm9yIHRoZSByYW5kb20gdmFyaWF0aW9ucyBpbiB0aGUgcG9wdWxhdGlvbnMgYW5kIHlvdXIgdmFsdWVzIGdvb2QgZXN0aW1hdGVzIG9mIHRoZSBwb3B1bGF0aW9ucw0KLSBTaW11bGF0aW9uIHN0dWRpZXMgYnkgQmFyciBldCBhbC4gc2hvdyB0aGlzIHRvIGJlIGEgcmVhc29uYWJsZSBhc3N1bXB0aW9uIElGIGFuZCBPTkxZIE9GIHJhbmRvbSBzdHJ1Y3R1cmUgaXMgTUFYSU1BTCwgYnV0IFRoaXMgaXMgSE9UTFkgZGViYXRlZCBieSBCYXRlcw0KDQo0LiBFc3RpbWF0ZSBkZWdyZWVzIG9mIGZyZWVkb20NCi0gU0FTL1NQU1MgdXNlcyBTYXR0ZXJ0aHdhaXRlIGFwcHJveGltYXRpb25zIA0KLSBIb3dldmVyLCB0aGVyZSBhcmUgYWxzbyBLZW53YXJkLVJvZ2VyIGFwcHJveGltYXRpb25zIFtzZWUgV2VzdGZhbGwsIEtlbm55LCAmIEp1ZGQsIDIwMTRdDQotIEVpdGhlciBhcmUgYWNjZXB0YWJsZQ0KLSBNYWtlcyBhIG51bWJlciBvZiBhc3N1bXB0aW9ucywgd2lsbCBtaXJyb3IgcmVzdWx0cyBvZiB0cmFkaXRpb25hbCBBTk9WQSBmYWlybHkgY2xvc2VseSBbYXMgeW91IHdvdWxkIGV4cGVjdCB3aGVuIHlvdSBtYWtlIHNpbWlsYXIgYXNzdW1wdGlvbnMgYWJvdXQgREZdDQoNCltOb3RlOiBZb3UgaGF2ZSB0byByZWZpdCB0aGUgbW9kZWwgd2l0aCB0aGUgbG1lclRlc3QgcGFja2FnZV0NCg0KYGBge3J9DQpsaWJyYXJ5KGxtZXJUZXN0KSAjbWl4ZWQgbW9kZWwgYWRkLW9uIHBhY2thZ2UgdG8gDQpNb2RlbC5GaW5hbDwtbG1lcihIYXBweVBlcmNlbnRaIH5Tb2NpYWwqVGltZVN0ZXANCiAgICAgICAgICAgICAgICAgICsoMStTb2NpYWwqVGltZVN0ZXB8U3ViamVjdCksICANCiAgICAgICAgICAgICAgICAgIGRhdGE9SGFwcHlEYXRhLCBSRU1MPUZBTFNFKQ0KDQpzdW1tYXJ5KE1vZGVsLkZpbmFsLGRkZiA9ICJsbWU0IikNCnN1bW1hcnkoTW9kZWwuRmluYWwsIGRkZj0iU2F0dGVydGh3YWl0ZSIpICMjIFNQU1MvU0FTIG1ldGhvZA0Kc3VtbWFyeShNb2RlbC5GaW5hbCxkZGYgPSAiS2Vud2FyZC1Sb2dlciIpICMgS2VubmV5IGV0IGFsIHN1Z2dlc3RlZCBtZXRob2QNCmBgYA0KDQotIEJlc3QgcHJhY3RpY2UgaXMgdG8gZG8gbW9kZWwgZml0dGluZyAob3B0aW9uIDEpIGFuZCB2ZXJpZnkgaXQgd2l0aCBvcHRpb25zIDIgT1IgMy4gDQotIE9wdGlvbiA0IGlzIHVzZWZ1bCBpZiBqb3VybmFsIGRlbWFuZHMgREYsIGJ1dCBpdCBjYW4gYmUgdmVyeSBjb25zZXJ2YXRpdmUgd2hlbiB5b3UgaGF2ZSBsb3RzIG9mIHZhcmlhbmNlIGlzc3Vlcw0KLSBCYXRlcyBzYXlzLCBQdmFsdWVzIGFyZSB3YXN0ZSBvZiBlZmZvcnQgYW5kIERGcyBzaG91bGQgbm90IGJlIGNhbGN1bGF0ZWQgKGJ1dCB5b3Ugd2lsbCBoYXZlIHRvIGp1ZGdlIGZvciB5b3Vyc2VsZiB3aGF0IG1ha2VzIHNlbnNlKQ0KDQojIE1peGVkIHRvIEFOT1ZBDQotIElmIHlvdSB3YW50ZWQgdG8gcmVwb3J0IG1haW4gZWZmZWN0cyBhbmQgaW50ZXJhY3Rpb25zLCBpbnN0ZWFkIG9mIHNsb3BlcyB5b3UgbWlnaHQgY29udmVydCB5b3VyIG1peGVkIG1vZGVsIGludG8gYW4gQU5PVkENCi0gQnV0IHJlbWVtYmVyLCB5b3VyIHNsb3BlcyBvZiB0aW1lIHJldmVydCB0byBiZWluZyBmYWN0b3JzIGFuZCBub3QgY29udGludW91cyB2YXJpYWJsZXMNCg0KYGBge3J9DQphbm92YShNb2RlbC5GaW5hbCxkZGYgPSAiS2Vud2FyZC1Sb2dlciIpICMgS2VubmV5IGV0IGFsIHN1Z2dlc3RlZCBtZXRob2QNCmBgYA0KDQoNCg0KPHNjcmlwdD4NCiAgKGZ1bmN0aW9uKGkscyxvLGcscixhLG0pe2lbJ0dvb2dsZUFuYWx5dGljc09iamVjdCddPXI7aVtyXT1pW3JdfHxmdW5jdGlvbigpew0KICAoaVtyXS5xPWlbcl0ucXx8W10pLnB1c2goYXJndW1lbnRzKX0saVtyXS5sPTEqbmV3IERhdGUoKTthPXMuY3JlYXRlRWxlbWVudChvKSwNCiAgbT1zLmdldEVsZW1lbnRzQnlUYWdOYW1lKG8pWzBdO2EuYXN5bmM9MTthLnNyYz1nO20ucGFyZW50Tm9kZS5pbnNlcnRCZWZvcmUoYSxtKQ0KICB9KSh3aW5kb3csZG9jdW1lbnQsJ3NjcmlwdCcsJ2h0dHBzOi8vd3d3Lmdvb2dsZS1hbmFseXRpY3MuY29tL2FuYWx5dGljcy5qcycsJ2dhJyk7DQoNCiAgZ2EoJ2NyZWF0ZScsICdVQS05MDQxNTE2MC0xJywgJ2F1dG8nKTsNCiAgZ2EoJ3NlbmQnLCAncGFnZXZpZXcnKTsNCg0KPC9zY3JpcHQ+