I’m still fairly new to Julia, even though I’ve been trying to learn it for a few years. It’s extremely powerful (fast, expressive, … whatever metric you want to use) but with that comes some complexity.
I saw this post in my feed and it seemed like a great bite-sized chunk of code to learn from. I think I understand everything that’s happening, even if I certainly couldn’t write that myself, with one exception.
The connection that for
all() is equivalent to
minimum() (it’s false as soon as there is one 0, otherwise it’s true) and
any() is equivalent to
maximum() (if there’s a 1 it’s true) took me a moment, but seems pretty cool. That wasn’t the problem I had.
The bit that surprised me was that for
ByRow calculations on a
minimum() is faster than
all(). The reason this is so surprising for me is that I understand
all() from an R-perspective and my understanding was that
all() could short-circuit because as soon as it sees a
FALSE it can ignore any other values - the result is guaranteed to be
FALSE (yes, yes, up to missingness). Surely, a calculation of
minimum() needs to evaluate every value at least once (?). Where this might (must?) fall apart is that I’m thinking purely of vectors. Sure enough, checking out some timings on a vector in Julia shows
all() is near-instantaneous (after compilation)
x = rand(Bool, 100_000_000) @time all(x) 0.009047 seconds (218 allocations: 9.531 KiB, 99.85% compilation time) false @time all(x) 0.000002 seconds false @time minimum(x) 0.091183 seconds (85.03 k allocations: 4.461 MiB, 41.98% compilation time) false @time minimum(x) 0.052287 seconds false
I get similar results, expectedly, from R
x <- sample(c(TRUE, FALSE), 1e8, replace = TRUE) microbenchmark::microbenchmark( min = max(x), any = any(x), times = 10 ) # Unit: nanoseconds # expr min lq mean median uq max neval # min 208741173 210539351 223219500.3 212388892 222673528 285974960 10 # any 160 187 2403.4 295 5095 7451 10
So, what’s going on? I think the answer is that we’re not dealing with just a vector, it’s rows from a
DataFrame, right? Now, from the R side, that’s complicated enough -
rowwise() is a necessary thing because R stores a
data.frame as a list of vectors representing columns, so extracting a row means slicing across those.
I can reproduce the speedup in Julia (and honestly, I struggle to find a clean and fast way to do it in R) but the statement “This time things are very fast, as row-wise aggregation for maximum and minimum is optimized.” got me thinking - where should I have learned that? Google isn’t showing me any relevant results, so is this just a known thing? I can imagine that such an optimization for doing this might exist, but can anyone provide a reference or guide?? The author of the blog post used this optimization in a StackOverflow answer without challenge (no reference provided) so I feel like it’s potentially just something I should know.