Software, Physics, Data, Mountains

...and other random associations

Develop Spark Apps on Yarn Using Docker

At svds, we’ll often run spark on yarn in production. Add some artful tuning and this works pretty well. However, developers typically build and test spark application in standalone mode… not on yarn.

Rather than get bitten by the ideosyncracies involved in running spark on yarn -vs- standalone when you go to deploy, here’s a way to set up a development environment for spark that more closely mimics how it’s used in the wild.

SSH Tips and Tricks

Notes from a lunch-and-learn talk. It’s a little weak without the narrative, but I’ll post it here for reference anyway.


  • config
  • tunnels
    • forward
    • reverse
  • proxies
  • ssh + tmux = <3
    • ssh, then tmux
    • tmux, then ssh

Juju Demo Videos

I Put together a series of demo videos using juju-0.7 for oscon.

These are really interesting in that they involve migrating environments between providers. This works slightly differently on the newer juju-1.x series, but the idea’s still sound.

There’s no sound on these… they’re raw video backups for demoing juju (in case we lost networking during the demo).

migrate local to hp

embedded youtube video Jfnxl1Kh9SY

CharmSchool Hangout - Charming From Scratch

Continuing the series of regular CharmSchool Hangouts. In last week’s video we wrote a charm from scratch…

embedded youtube video NQmxuzdc4Zg

Starting from a simple node.js application, we put together “just enough” charm to get things working. Watch for future episodes where we’ll refactor and refine both the application and the charm.

CharmSchool Hangouts

We’re doing a series of regular CharmSchools on G+ hangouts. Last week we did an intro to juju and charming…

embedded youtube video yRcqSjOGweo

There’re links to the whole series of charmschool hangouts in the juju video archive where we also have videos and screencasts of demos, talks, and any other charm schools we’ve been able to capture on video.

Running the LinuxPlumbers Conference Schedule With Juju

Written by Mark Mims and Chris Johnston

Hey, so last month we ran scheduling for the Linux Plumbers Conference entirely on juju!

Here’s a little background on the experience.

Along the way, we’ll go into a little more detail about running juju in production than the particular problem at hand might warrant. It’s a basic stack of services that’s only alive for 6-months or so… but this discussion applies to bigger longer-running production infrastructures too, so it’s worth going over here.