January 2nd. I’m ready for the smartest show on TV to be back.
When Caching Bites Back
We have an application on our site that was rewritten a few years back by a developer who is no longer with the company. He attempted to do some “smart” caching things to make it fast, but I think had a fundamental lack of understanding of how caching, or at least memcached works.
Memcached is a really nifty, stable, memory-based key-value store. The most common use for it is caching the results of expensive operations. Let’s say you have some data you pull from a database that doesn’t change frequently. You’d cache it in memcached for some period of time so that you don’t have to hit the database frequently.
A couple of things to note about memcached. Most folks run it on a number of boxes on the network, so you still have to go across the network to get the data. [1] Memcached also, by default, has a 1MB limit on the objects/data you store in it. [2] Store lots of stuff in it, keep it in smaller objects (that you don’t mind throwing across the network), and you’ll see a pretty nice performance boost.
Unless … someone decides to not cache little things. And instead caches a big thing.
We started to notice some degradation in performance over the past few months. It finally got bad enough that I had to take a look. It only took a little big of debugging to determine that the way the caching was implemented wasn’t helping us: it was actively hurting us. Rather than caching entries individually, it was loading up an entire set of the data and trying to cache a massive chunk of data. Which, since it was larger than the 1MB limit, would fail.
You’d end up with something like this:
- Hey, do I have this item in the cache?
- Nope, let’s generate the giant object so we can cache it
- Send it to the server to cache it
- Nope, it’s too big, can’t cache it
- Oh well, onto the next item … do I have it in the cache?
Turns out, this wasn’t just impacting performance. It was hammering our network.
The top of that graph is about 400Mb/s. The drop off is when we rolled out the change to fix the caching (to cache individual elements rather than the entire object0. It was, nearly instantaneously, a 250Mb/s drop in network traffic.
The lesson here? Know how to use your cache layer.
More Detail on Dropbox Backup
As mentioned in yesterday’s post, I’ve taken to backing up the important guts of my server via Dropbox. It turned out to be very easy, and gives me the added benefit of not having to do any sort of versioning: I get that for free with Dropbox. Plus, it seamlessly integrates with my existing backup routines.
So, how do I do it? It’s honestly very simple.
First, I generate some backups. I run these out of cron every morning.
mysqldump -u root wordpress | gzip -c > /path/to/backup/blogdb.sql.gz
tar czf /path/to/backup/apache2.tgz /etc/apache2/*
I do this for my web databases, any important config files, crontabs, etc. (The actual sites are already backed up, since I develop them locally.)
Once they’re dropped off in the backup location, it’s just a matter of having the script come along and copy them to Dropbox. I chose to write it in ruby. Honestly, my code is 90% of what you find in Dropbox’s example. Here it is, in all of its glory:
require 'dropbox_sdk'
# Dropbox app key, secret, token
APP_KEY = 'get your own'
APP_SECRET = 'get your own'
ACCESS_TOKEN = 'get your own'
BACKUP_DIR = '/path/to/backup/'
# Open our Dropbox Client
client = DropboxClient.new(ACCESS_TOKEN)
# Our hardcoded list of files
files = ['list of files to backup']
# For each file, let's upload it
files.each do |filename|
file = File.new(BACKUP_DIR + filename)
# Send file to dropbox -- last param tells us to overwrite the previous one
response = client.put_file('/' + filename, file, true)
end
That’s it. I don’t really do any error checking (I should, and probably will some day, but I don’t today). I should probably store the key/secret/token in another place, but since my app was created and only has access to one Dropbox folder, and I can revoke access at any time, it’s not too much of a risk. Eventually, when I get really ambitious, I’ll have the files list be dynamic, not static. But for now, it’s less than 20 lines of code to backup important files.
That’s good enough for me.
Remote Server Backup via Dropbox
This site runs on a VPS server sitting out in a data center. I try hard to keep the server reasonably backed up, similar to how I keep my desktop backed up, but it’s something I’ve not been nearly as diligent about.
Previously, here was how I kept my VPS backed up:
- Paid daily server snapshot (i.e. I could restore my server to exactly what it looked like yesterday)
- Database dumps of my important mysql databases that I would remember to copy off to another server every couple of weeks (i.e. not particularly reliable)
- A backup of my important web stuff over FTP and git when I remembered to do it (i.e. not particularly reliable)
So, in the grand scheme of things, not very reliable.
All I really wanted was a simple backup where I could easily get the important files and configs off of the server onto my local machine, so that it would get included with my normal, robust, multi-location backup strategy. The normal method to do that would be rsync, but that means running my machine all the time. Not a big deal for my desktop machine, but that’s also not the machine that gets updated and backed up to Amazon Glacier daily.
Really, what I wanted is to use Dropbox on my server, without having to run Dropbox on the server.
It turns out, that’s pretty easy [1].
There are a few different solutions out there [2], but for me, I wanted a bit of control over it. Dropbox’s Ruby API is really, really easy to use. My total script is about 22 lines (I’ll go into that, in the future). Very simply:
- I’ve created a Dropbox app that is tied to a single directory in my Dropbox account
- I’ve got a number of crons that backup data into a location on my server
- My little 22 line Dropbox script runs once a day and uploads that backed up data into my Dropbox account
So, right there, I’ve now backed up my data daily to Dropbox, which is a big win over what I had before. Even if my computer dies or is turned off for a month, I’m still backing up my data. But, even better, when my various computers are turned on, I’m getting the backed up data synced down to those machines. So, I’m getting multiple copies of my backup data, and that backup data is getting included in the places that those machines are backed up (Time Machine, Glacier, etc.).
I’m not sure why I didn’t think of this before. It took all of about 60 minutes to get the entire thing working, including building the little Dropbox backup app that runs on the server.
Updated NBA Points Created Page
For the 7 people who care, I’ve finally updated my NBA Points Created page [1] to include the 2013–2014 season. Early in the season, the raw numbers clearly favor big men, but that effect should even out over the course of the season. The per–40 numbers are definitely a better barometer for overall value (I think), and at least pass the sniff test.
I’ve not really updated any of the coefficients in the last few years, and it probably could use a slight tweak to reflect the changes in the pace of the game, but as a quick and dirty barometer, it’s at least interesting to look at.
-
The original Points Created description
The LinkedIn Conundrum
Every few weeks, I log into LinkedIn and to accept/reject connections and clear out the stupid notices of people who’ve said I can do something well.
(I don’t think the people are stupid. I think the notices are stupid and spammy. Which, I think, you could say about pretty much 99% of LinkedIn’s email communications.)
I clear all the notifications, flags, messages, requests for human blood from LinkedIn and then I notice my profile is out-of-date. As I go to update it, I encounter The LinkedIn Conundrum.
“What’s The LinkedIn Condundrum?”, you ask? Good question.
The LinkedIn Conundrum (or at least, what I see as the conundrum …) is the desire to update your profile so that it at least reflects reality, but then the fear to do so because you don’t want it to look to your colleagues that you’re shining up your resume to look for a job.[1] Some people clearly don’t have this fear, and they update their profile one hundred times a day, outlining every new thing they’ve done in their job (“skills: getting coffee four times a day without anyone noticing I’m away from my desk”). There are certainly others, and I fall into this category, where I struggle over whether to update my profile at all, fearing that someone will assume I’m looking for a job. So instead, my profile stays there, frozen in carbonite, forever out of date.
-
The corollary to The LinkedIn Conundrum is the same fear whenever you accept the connection of a recruiter on LinkedIn, which clearly makes people think you’re talking to a recruiter, when in reality, you’re just accepting some connection to a recruiter who reached out to you, or you’re using for sourcing. Of course, none of this would be a problem if it was socially acceptable to not connect with everyone on LinkedIn, but that doesn’t seem to be the societal norm. Though, I do, on occasion, not accept a connection, if only to exert some feeble power over the LinkedIn borg.
Typhoon: NPR Tiny Desk Concert
I mentioned Typhoon a few posts ago. I think they’ve put out my favorite album of the year, and certainly put on my favorite show of the year (thus far).
Just watch this. If you like music, at all, you’re going to smile. Harmonies, hand claps, horns, whistling; it’s all there. Seriously, just watch and smile.
Sometimes It Just Takes a Little Prodding
I used to run. There was a point in time at a previous job, where I wasn’t super happy, nor particularly engaged, and I could get out of the office at a normal time and go running. Distance running has never been my thing, but I would run 4–5 miles a day, 5 or 6 days a week.
Over the past few years, I’ve fallen into some bad habits. I’d go through phases where I’d run a few miles a couple of times a week; then it was running a few miles at the gym a couple of times a week. Until the last month, it was run every couple of weeks and pretend that the exercise I get from softball and basketball is enough to get by.
Periodically, I’d use something like Runkeeper to try to motivate me. “Hey, you should go running and try to do better than you did yesterday.” The downside—I’d often get frustrated because going from running 4–5 miles a day to 2–3 miles every couple of weeks was a big step back and seeing that displayed to me when I started the app was demoralizing, not encouraging.
I went through these cycles for a couple of years.
Then I got my new iPhone 5S with its M7 chip.
And, honestly, this isn’t any sort of amazing revelation, but just using an app like Pedometer++[1] is enough to help get me up and moving a bit more. It just measures your steps each day, and you can set a goal. It’s a Fitbit[2], but comes in a device that’s already in my pocket 90% of the day. It’s just enough prodding. I open the app, see I only have a few thousand steps, and I’ll go for a run (I went as far as to buy a thingy to carry my phone while I run). Or I’ll walk the long way to grab dinner.
It’s not much. But I’m sitting here on 9,300 steps for the day (which doesn’t count the hour and a half of basketball, or hour plus of softball – so I’m really probably closer to 20,000 steps), and I’ll probably go walk around a bit to get over 10,000 steps for the day.
It’s so incredibly stupid that I am motivated by a little number, and not by simply getting in better shape or generally being healthier.
But, it turns out, that seeing the little values go to 10,000 is just enough prodding.
-
I’ve avoided the Fitbit or Nike Fuel or any of those things forever, for a couple of reasons. One, I hate carrying things. I don’t wear jewelry or watches. I try to have as little in my pockets as possible. But, I think I should have joined the crowd and grabbed one of those things. I’d likely have figured out that I’m motivated by seeing round numbers a long time ago.
A Couple of Quick Mavericks iCloud Keychain Gotchas
I’ve been using Mavericks for a couple of days now. I’m sure I’ll have more thoughts about it later (particularly about the multi-monitor support). But, right now, here’s a couple of gotchas that I found when using the new iCloud Keychain.
Turn on the ability to save passwords even when a website requests you not
Both on the desktop, in Safari, and in Mobile Safari on your iOS device, there’s a setting that controls whether or not you can autofill a password with your iCloud Keychain password. Since you probably want to control when you can autofill the password (not the website), you’ll want to override the default.
On the desktop, that’s in Safari’s Preferences, under Passwords. Down the bottom, there’s a little checkbox that says “Allow Autofill even for websites that request passwords not be saved”. Check that box.
On your iOS device, go to Settings -> Safari -> Passwords & AutoFill, and turn on “Always Allow”.
Now you’ll never get that error message that you can’t use your saved password.
iCloud Keychain Doesn’t Save the Right Password if You Use PwdHash
I use a Safari Extension called PwdHash (that I actually created, built off of the open-source PwdHash extensions). PwdHash does a nice thing where it generates a new password for you on a site, based off of a password you know. That way, no matter what computer you’re on, or what browser you’re in (assuming you can find a PwdHash extension or bookmarklet), you can login to a site.
It’s pretty handy.
However, the way the Safari extension works (and this may be my fault), it picks up the password I typed in (my “master” password) and not the resulting hashed password after the extension runs.
…which means it ends up syncing the wrong password. That’s very possibly my fault, but could also be just a mechanism of how Safari/iCloud Keychain decides what to sync. It does mean that I need to be careful when saving passwords[1].
-
However, as I go around creating new passwords, Safari’s new “suggested password” feature may make my need for the PwdHash extension moot.
Typhoon at Brighton Music Hall (9/29)
It’s taken me a few weeks to sort out what I wanted to say about seeing Typhoon live for the first time. They put out my favorite song of 2011, and I think White Lighter is the current favorite to be my top album of the year, with three or four songs on it that I might like better than anything else I’ve heard this year. But, I’d never seen them live (which is somewhat rare for bands I really enjoy). They’d never really toured the East Coast.
Radiation City opened for them, and it was one of those great concert moments where you get exposed to a really good band that you’ve never heard of, and you begin to realize “Hey, this crowd is pretty cool right now. Into the music, not much talking, no one pushing to the front.”[1] I think most people who were there were as excited as I was to be there, and people were, sort of, on their best behavior.
Radiation City played a really great set of synth-backed pop with lots of harmonies—right up my alley—but a bit slower than you might expect. Worth checking out on Spotify.
Then Typhoon came out. And from song one, they just owned it. Eleven band members, strings, horns, multiple drummers, and vocals and lyrics that just sort of wring emotions out of you.
It was this big, cathartic release. A bunch of people waiting to see a band, the band coming out and being better than anyone probably expected, and the crowd just reacting with sheer joy. That joy was clear to the band, who acknowledged the crowd’s boisterous response more than once, giving the old “you’re the best crowd we’ve ever played in front of” spiel, but saying it like they meant it, which would just make the crowd react even more.
I know, I know. This all sounds ridiculous. But, sometimes you talk to people who really love Phish (or, if they’re a bit older, the Grateful Dead; it’s always a jam band), and they tell you about these shows where everything happened a certain way, and the lights were just right, and the crowd was just right, and the music segued just right.
That’s what this show felt like. Everything kind of all came together—a band firing on all cylinders, a crowd that had waited years to see the band—and it lead to a really, really wonderful show.
-
Complete aside about concert crowds. They mostly suck. Obviously, it depends on the bands you see, but, if you’re seeing a band that attracts a college audience, it’s going to suck. College kids are college kids, and I now feel horribly old saying this, but they’re shitheads. I mean, we all were in college. But, when you go to a show for a band like Ra Ra Riot, you don’t expect to see bros jumping up on the stage and then stage diving. Or when seeing Clap Your Heads Say Yeah, you don’t expect to spend the entire show with a bunch of college hipsters pushing their way to the front to dance spastically because they “love the band.” We’re all there for the band, dipshit.
Nada Surf attracts really great crowds that make the experience that much better. When I saw The Presidents of the USA recently, that was a great crowd full of people who loved the band. The crowd for Typhoon was one of best crowds I’ve experienced. Dancing and clapping for the right songs; getting quiet for the places that needed to be quiet.