SparkR vs sparklyr for interacting with Spark from R

This post grew out of some notes I was making on the differences between SparkR and sparklyr, two packages that provide an R interface to Spark. I’m currently working on a project where I’ll be interacting with data in Spark, so wanted to get a sense of options using R. Those unfamiliar with sparklyr might benefit from reading the first half of this previous post, where I cover the idea of having R objects for connections to Spark DataFrames. [Read More]
Code  Spark  R  sparklyr 

Machine learning and k-fold cross validation with sparklyr

Update, 2019. I have now written an updated post on cross-validation with sparklyr, as well as a follow-up on using cross-validation for feature selection. These posts would be better to read as the code here no longer works following changes to sparklyr. In this post I’m going to run through a brief example of using sparklyr in R. This package provides a way to connect to Spark from within R, while using the dplyr functions we all know and love. [Read More]

Modelling working memory precision in R

Functions for modelling continuous response working memory tasks

I have translated Paul Bays’ Matlab functions for modeling continuous response data into R . Bays’ guide to the Matlab functions and their usage can be found here. I have a detailed guide to the R functions and their usage over on Github. A typical precision task has participants recall a feature of stimulus on a continuous scale rather than using classic correct/incorrect scoring. The model described in Bayes et al. [Read More]

Cutoff function

A function for rounding values in R

In R Markdown one is often faced with situations where a value that can vary substantially in automatically pulled out. For example, we might want to pull a p-value out of an linear model object. It’s a chore to have to manually edit p-values so that they conform with reporting conventions (e.g. p < .001). I’ve written a little function to deal with this issues by taking a value and returning, say, "= . [Read More]