CleverBlogName

I wasn’t a keynoter. Or even a regular presenter. I was just doing a talk at a miniconf. It was still an un-nerving enough experience that I went to see my doctor on the Thursday before I flew to Tullamarine to make sure the chest pains I was having weren’t the onset of a heart attack. It almost would have been a relief if they had been.

As you can see, I did make it, and I didn’t drop dead on stage.

Normally I’m pretty comfortable about speaking in front of people. To the point where, for example, last year I needed to double the time I’d been told I would be allocated, and spoke extemporaneously from the bullet-points I’d listed on a bit of paper, only looking at them once. Or, 4 years ago, spoke at a funeral after leaving my speech at home. Give me a run-up and I can usually stand up and talk about most anything I care about on short notice, and probably for longer than you envisaged when you asked.

So why so nervous? There’s a simple reason: the audience.

Rusty Russell & Matt Evans

Three Cool Projects

• Spark - command-line tool that generates sparks
• Plover - An open-source stenography alternative.
• Homebrew Cray-1A http://chrisfenton/homebrew-cray-1a/

In The Beginning

• “The year was 1976, the hair was long, the shoes were tall.”
• It’s a multi-user machine, it has two teletype machines in front of it!
• Bring on the PDP-11 simulator!

Comparisons

• cat, grep, and ls are the punching bags.
• “Bigger is better. grep is 20 times bigger.”
• “cd only came in in V7.” “I edited the shell so I could have cd.”
• cat was written in assembler in 1976.
• In V6 arguments were in-line.
• The first thing you notice is that they are memory-concious. Although Rusty points out that they don’t bother with system calls; they also just use assembler because it’s natural.
• Rusty re-implemented cat with bug/behaviour and it was only twice is big in C. Modern cat is big in part becuase of more features, error messages, and so on.
• We pay a 30% memory penalty if we use -O2 instead of -Os.
• But -Os is slower by about 6% for these simple utilities.
• Automated runtime analysis tells us 99% of the instructions are used at some point, with only one instruction ever being used. 1% bloat!
• Even going to V7 in 1979 ls has doubled in size. cat only uses 57% of its instructions.
• ...but if you built static cat instead of shared libraries it pulls in another 700KB of glibc dependcies!
• There’s a dependency graph. It looks like scribble.
• It includes TLS (in case you need to fetch from Reddit).
• When we instrument cat on x86 we find that we use... um... 2% of it. Bugger. 0
• On a whole-system analysis there’s 33 MB of wasted RAM. Not much compared to all the memory.
• But there may be a TLB hit.
• Of course, 16-bit vs 64-bit is unfair. So Rusty guessimated the change in Text and Data segments. There’s some big growth, around 50%.
• By way of comparison 32-bit to 64-bit Ubuntu is only 9%.
• If you pull the old code to an Ubuntu system you actually cut the text segment (if you’ve got a stripped-down glibc). ELF, on the other hand, adds stuff, mostly to force on-page alignment. Even so, cat is only marginally bigger that 64-bit PDP cat. Not too bad.
• grep embiggened significantly.
• ls was already complex, with 10 flags. We also have to grow buffers for moving from 14 byte filenames to 255 bytes. It doesn’t use malloc, doing funky magic to grow the program when it starts running out. You kind of need it to use malloc() nowadays. So you grow 120%, because of a combination the changes.

Backporting

• Turns out GNU ls has 60-odd options. A survey of Rusty’s friends says 11 of them were never used.
• So some of this size are the extra options.
• For cat it’s easy: remove all the options and error reporting.
• Cat does some odd malloc() behaviour to have aliged, page-size buffers.
• Backported cat is still bigger than forward-ported cat.
• ls required Vast Surgery.
• It grabs system to the nanosecond so it can show entries more than 6 months old differently.
• It’s much faster, although it’s probably down to LOCALE complexity slowing up non-backported.
• There’s a 60% penality. But that’s for portability, 64-bit and so on.
• 400-odd% bloat? That’s the extra features.

Conclusions

• Most people aren’t prepared to go to the same lengths to keep things small.
• asmutils - reimplementation of *ix utils, but it’s not actually that efficient: it loses all the gains in BSS bloat that they don’t botherr measuring. Bummber.
• Features are the reason for growth.

Sarah Novotny

Origin of the talk was when a customer rang with a complaint that a site was wrong, but Sarah couldn’t find a problem, and this provoked her into thinking about where data can ad should be cached.

Why Cache?

We want to move data as close to the end used, while retaining ACID-style guarantees. The abandonment rate after 7 seconds is huge. We need reliable speed.

Count Them.

• CPU Caches - L1, L2, L3.
• Filesystem/VM caches.
• Controller caching.
• Disk caches. Great, but disks now lie about whether the write is there or on the platter.
• SSD hybrids.
• RAM on disks.
• DB caching.
• memcached or other application level caching.
• Protocol level caching, e.g. HTTP, DNS.
• Transparent proxies.

• Browser caching.

• A lot of this is transparent to you.

• Sometimes stuff ignores semantics around caching, too.
• Users often don’t have the knowledge to bypass bad caching by e.g. doing a browser force-refresh.

A Short Diversion

• DBA/SA background means Sarah cares a lot about ACID demantics around data.
• Will therefore focus on the DB

Which Caches are Redundant

• Some caching is redundant.
• The tackle the same functions, but are either redundant or even harmful. Battery-backed controller caches are good and cache disks. Disk caches cache, but are unlikely to be “safe”.
• You need to ensure durability in those cases.
• For MySQL you also have InnoDB query caches and buffer caches.

Why do we keep doing this? Because we want things to go faster! But there’s a conflict between the DB cache and filesystem cache, too. You’re double-buffering. They aren’t particular dangerous on modern filesystems, but it’s an inefficient use of memory and CPU to manage both sets of caches.

Which Caches are Risky

• Expiries not set well on memcached will result in data being lost; Sarah is of the opinion you should only use this for temp data.
• Hypervisors often cache disk in memory, without advising the guest what happens here.
• Disks lie! They are reporting writes suceeding when they aren’t on the platter for reals.
• RAID controllers lie, but at least they lie with battery backup (if you spent the money), so you’re probably OK.
• The last two are really toxic, because you can end up losing data on power failure. Sarah recommends controlled power failures to test this.
• TURN YOUR DISK CACHE OFF IF YOU VALUE YOUR DATA.
• MySQL generally does better if it bypasses the FS cache for direct-attached storage. However, for SAN-attached disk you should leave FS caching on.

Benchmarking

• You need to be careful when benchmarking, but in general it’s good and you can never do enough.
• It’s not magic. You just need to do it right.
• Don’t do bad benchmarks that just e.g. exercise your cache.
• You need to touch the slowest part of the system. Force pessimistic scenarios, e.g. when your controller cache goes offline.
• You also want to test the normal production case, with real data sets and a workload that looks like production behaviour.
• You can test in prod, but you should use proper staging hardware that’s similar.
• Benchmarking with real data also exercises your backups if you populate from them.
• You can also use a replica/DR server on a short-term basis. Breaking replication and then restoring is good practise for this.

Monitoring

• Only monitor the stuff you want.
• Test multiple layers in your infrastructure and that you test both what the end customer sees, as well as each touch point along the way.
• Monitoring is an evolving case; treat it like you’d treat unit testing in software.
• There’s no boilerplate. Every system is unique.
• So many tools.

• Origins of Performance Co-Pilot (PCP): Frustration with SNMP.
• Started by SGI.
• Top does not scale.
• Open source toolkit for system level performance analysis: live and historical data, extensible, distribted.
• Many agents talking to a master daemon; many clients then talk to the daemon.
• One common protocol for real-time and historical data.
• Real time conditional learning via a little language.
• Log analysis - some nice tools that will e.g. let you compare points in time.
• GUIs. Everyone loves GUIs.
• Alerting functionality.

PCP compared to SNMP

• SNMP is extensible. To a fault. PCP is extensible, but in a more limited fashion.
• Naming is likewise marginally more limited than SNMP.
• SNMP MIBs have the metadata; in PCP the metadata is in the agent. There’s little more effort up-front when building agents.

Why Build a Bridge?

• They use PCP extensively.
• But you can’t use it with non-general purpose devices.
• PCP does a great job for monitoring, alerting, trending.
• but not for e.g. switches, SANs, server LOMs, and so on.
• The rest of the world uses SNMP.
• There’s no general purpose SNMP gateway.

How it Went

• Side-by-side logging and use of the PCP tools.
• Dynamic, ad-hoc interface that doesn’t need rewriting and reconfiguring ahead of time - this wasn’t possible, unfortunately.
• There’s a solid TODO, mostly in fixing the PCP perl libraries.
• Add dynamic mappings.
• Use the MIBs for number/name mappings.
• Not in production yet.