data.frame interview questions
Top data.frame frequently asked interview questions
When deleting a column in a DataFrame I use:
del df['column_name']
and this works great. Why can't I use:
del df.column_name
As you can access the column/Series as df.column_name, I expect this to work.
Source: (StackOverflow)
I have a nested list of data. Its length is 132 and each item is a list of length 20. Is there a quick way to convert this structure into a data frame that has 132 rows and 20 columns of data?
I am new to R, so I figure there is probably an easy way to do this. I searched here on Stack Overflow and couldn’t find a similar question, so I apologize if I missed it. Some sample data:
l <- replicate(
132,
list(sample(letters, 20)),
simplify = FALSE
)
Source: (StackOverflow)
Does anyone know how to remove an entire column from a data.frame in R? For example if I am given this data.frame:
> head(data)
chr genome region
1 chr1 hg19_refGene CDS
2 chr1 hg19_refGene exon
3 chr1 hg19_refGene CDS
4 chr1 hg19_refGene exon
5 chr1 hg19_refGene CDS
6 chr1 hg19_refGene exon
and I want to remove the 2nd column.
Source: (StackOverflow)
I think I am using plyr incorrectly. Could someone please tell me if this is 'efficient' plyr code?
require(plyr)
plyr <- function(dd) ddply(dd, .(price), summarise, ss=sum(volume))
A little context: I have a few large aggregation problems and I have noted that they were each taking some time. In trying to solve the issues, I became interested in the performance of various aggregation procedures in R.
I tested a few aggregation methods - and found myself waiting around all day.
When I finally got results back, I discovered a huge gap between the plyr method and the others - which makes me think that I've done something dead wrong.
I ran the following code (I thought I'd check out the new dataframe package while I was at it):
require(plyr)
require(data.table)
require(dataframe)
require(rbenchmark)
require(xts)
plyr <- function(dd) ddply(dd, .(price), summarise, ss=sum(volume))
t.apply <- function(dd) unlist(tapply(dd$volume, dd$price, sum))
t.apply.x <- function(dd) unlist(tapply(dd[,2], dd[,1], sum))
l.apply <- function(dd) unlist(lapply(split(dd$volume, dd$price), sum))
l.apply.x <- function(dd) unlist(lapply(split(dd[,2], dd[,1]), sum))
b.y <- function(dd) unlist(by(dd$volume, dd$price, sum))
b.y.x <- function(dd) unlist(by(dd[,2], dd[,1], sum))
agg <- function(dd) aggregate(dd$volume, list(dd$price), sum)
agg.x <- function(dd) aggregate(dd[,2], list(dd[,1]), sum)
dtd <- function(dd) dd[, sum(volume), by=(price)]
obs <- c(5e1, 5e2, 5e3, 5e4, 5e5, 5e6, 5e6, 5e7, 5e8)
timS <- timeBasedSeq('20110101 083000/20120101 083000')
bmkRL <- list(NULL)
for (i in 1:5){
tt <- timS[1:obs[i]]
for (j in 1:8){
pxl <- seq(0.9, 1.1, by= (1.1 - 0.9)/floor(obs[i]/(11-j)))
px <- sample(pxl, length(tt), replace=TRUE)
vol <- rnorm(length(tt), 1000, 100)
d.df <- base::data.frame(time=tt, price=px, volume=vol)
d.dfp <- dataframe::data.frame(time=tt, price=px, volume=vol)
d.matrix <- as.matrix(d.df[,-1])
d.dt <- data.table(d.df)
listLabel <- paste('i=',i, 'j=',j)
bmkRL[[listLabel]] <- benchmark(plyr(d.df), plyr(d.dfp), t.apply(d.df),
t.apply(d.dfp), t.apply.x(d.matrix),
l.apply(d.df), l.apply(d.dfp), l.apply.x(d.matrix),
b.y(d.df), b.y(d.dfp), b.y.x(d.matrix), agg(d.df),
agg(d.dfp), agg.x(d.matrix), dtd(d.dt),
columns =c('test', 'elapsed', 'relative'),
replications = 10,
order = 'elapsed')
}
}
The test was supposed to check up to 5e8, but it took too long - mostly due to plyr. The 5e5 the final table shows the problem:
$`i= 5 j= 8`
test elapsed relative
15 dtd(d.dt) 4.156 1.000000
6 l.apply(d.df) 15.687 3.774543
7 l.apply(d.dfp) 16.066 3.865736
8 l.apply.x(d.matrix) 16.659 4.008422
4 t.apply(d.dfp) 21.387 5.146054
3 t.apply(d.df) 21.488 5.170356
5 t.apply.x(d.matrix) 22.014 5.296920
13 agg(d.dfp) 32.254 7.760828
14 agg.x(d.matrix) 32.435 7.804379
12 agg(d.df) 32.593 7.842397
10 b.y(d.dfp) 98.006 23.581809
11 b.y.x(d.matrix) 98.134 23.612608
9 b.y(d.df) 98.337 23.661453
1 plyr(d.df) 9384.135 2257.972810
2 plyr(d.dfp) 9384.448 2258.048123
Is this right? Why is plyr 2250x slower than data.table? And why didn't using the new data frame package make a difference?
The session info is:
> sessionInfo()
R version 2.15.1 (2012-06-22)
Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] xts_0.8-6 zoo_1.7-7 rbenchmark_0.3 dataframe_2.5 data.table_1.8.1 plyr_1.7.1
loaded via a namespace (and not attached):
[1] grid_2.15.1 lattice_0.20-6 tools_2.15.1
Source: (StackOverflow)
I have a dataframe in pandas which I would like to write to a CSV file. I am doing this using:
df.to_csv('out.csv')
And getting the error:
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03b1' in position 20: ordinal not in range(128)
Is there any way to get around this easily (i.e. I have unicode characters in my data frame)? And is there a way to write to a tab delimited file instead of a CSV using e.g. a 'to-tab' method (that I dont think exists)?
Source: (StackOverflow)
I made a data.frame in R that is not very big, but it takes quite some time to build. I would to save it as a file, which I can than again open in R?
Source: (StackOverflow)
I'm trying to initialize a data.frame without any rows. Basically, I want to specify the data types for each column and name them, but not have any rows created as a result.
The best I've been able to do so far is something like:
df <- data.frame(Date=as.Date("01/01/2000", format="%m/%d/%Y"), File="", User="", stringsAsFactors=FALSE)
df <- df[-1,]
Which creates a data.frame with a single row containing all of the data types and column names I wanted, but also creates a useless row which then needs to be removed.
Is there a better way to do this?
Source: (StackOverflow)
Given two data frames:
df1 = data.frame(CustomerId = c(1:6), Product = c(rep("Toaster", 3), rep("Radio", 3)))
df2 = data.frame(CustomerId = c(2, 4, 6), State = c(rep("Alabama", 2), rep("Ohio", 1)))
df1
# CustomerId Product
# 1 Toaster
# 2 Toaster
# 3 Toaster
# 4 Radio
# 5 Radio
# 6 Radio
df2
# CustomerId State
# 2 Alabama
# 4 Alabama
# 6 Ohio
How can I do database style, i.e., sql style, joins? That is, how do I get:
- An inner join of df1 and df2:
Return only the rows in which the left table have matching keys in the right table.
- An outer join of df1 and df2:
Returns all rows from both tables, join records from the left which have matching keys in the right table.
- A left outer join (or simply left join) of df1 and df2
Return all rows from the left table, and any rows with matching keys from the right table.
- A right outer join of df1 and df2
Return all rows from the right table, and any rows with matching keys from the left table.
Extra credit:
How can I do a sql style select statement?
Source: (StackOverflow)
I have an R data frame with 6 columns, and I want to create a new dataframe that only has three of the columns.
Assuming my data frame is df, and I want to extract columns A, B, and E, this is the only command I can figure out:
data.frame(df$A,df$B,df$E)
Is there a more compact way of doing this?
Source: (StackOverflow)
I want to sort a data.frame by multiple columns in R. For example, with the data.frame below I would like to sort by column z (descending) then by column b (ascending):
dd <- data.frame(b = factor(c("Hi", "Med", "Hi", "Low"),
levels = c("Low", "Med", "Hi"), ordered = TRUE),
x = c("A", "D", "A", "C"), y = c(8, 3, 9, 9),
z = c(1, 1, 1, 2))
dd
b x y z
1 Hi A 8 1
2 Med D 3 1
3 Hi A 9 1
4 Low C 9 2
Source: (StackOverflow)
I'd like to remove the lines in this data frame that contain NAs across all columns. Below is my example data frame.
gene hsap mmul mmus rnor cfam
1 ENSG00000208234 0 NA NA NA NA
2 ENSG00000199674 0 2 2 2 2
3 ENSG00000221622 0 NA NA NA NA
4 ENSG00000207604 0 NA NA 1 2
5 ENSG00000207431 0 NA NA NA NA
6 ENSG00000221312 0 1 2 3 2
Basically, I'd like to get a data frame such as the following.
gene hsap mmul mmus rnor cfam
2 ENSG00000199674 0 2 2 2 2
6 ENSG00000221312 0 1 2 3 2
Also, I'd like to know how to only filter for some columns, so I can also get a data frame like this:
gene hsap mmul mmus rnor cfam
2 ENSG00000199674 0 2 2 2 2
4 ENSG00000207604 0 NA NA 1 2
6 ENSG00000221312 0 1 2 3 2
Source: (StackOverflow)
I have very large tables (30 million rows) that I would like to load as a dataframes in R. read.table() has a lot of convenient features, but it seems like there is a lot of logic in the implementation that would slow things down. In my case, I am assuming I know the types of the columns ahead of time, the table does not contain any column headers or row names, and does not have any pathological characters that I have to worry about.
I know that reading in a table as a list using scan() can be quite fast, e.g.:
datalist <- scan('myfile',sep='\t',list(url='',popularity=0,mintime=0,maxtime=0)))
But some of my attempts to convert this to a dataframe appear to decrease the performance of the above by a factor of 6:
df <- as.data.frame(scan('myfile',sep='\t',list(url='',popularity=0,mintime=0,maxtime=0))))
Is there a better way of doing this? Or quite possibly completely different approach to the problem?
Source: (StackOverflow)
I have a data frame containing a factor. When I create a subset of this data frame using subset() or another indexing function, a new data frame is created. However, the factor variable retains all of its original levels -- even when they do not exist in the new data frame.
This creates headaches when doing faceted plotting or using functions that rely on factor levels.
What is the most succinct way to remove levels from a factor in my new data frame?
Here's my example:
df <- data.frame(letters=letters[1:5],
numbers=seq(1:5))
levels(df$letters)
## [1] "a" "b" "c" "d" "e"
subdf <- subset(df, numbers <= 3)
## letters numbers
## 1 a 1
## 2 b 2
## 3 c 3
## but the levels are still there!
levels(subdf$letters)
## [1] "a" "b" "c" "d" "e"
Source: (StackOverflow)