Vinnie doesn’t let go of a question. “OK, Robert, I got that a computer’s internal network is sorta like a horse’s sinews, tying muscle and bone together. An’ I got that a computer’s processors of whichever kind are like a horse’s muscles. But what does for a computer what bones do for a horse?”
“The ‘bones’ are a bit of a stretch, Vinnie. Data’s one possibility, memory or storage is the other one.”
Vinnie takes the bait. “Horse muscles move horse bones. The processors move data, so data’s got to be the bones.”
For the sake of argument, I come back. “But when the electricity turns off, the data goes away, right? Memory’s still there, so memory must be the bones. Or is it storage? What’s the difference between memory and storage?”
“You’ve put your finger on it, Sy — persistence. If the data’s retained when the power’s off, like on a hard drive, it’s in storage. Otherwise it’s in memory. Setting aside power glitches, of course — a bad glitch can even kill some kinds of storage and the data it’s holding, which is one reason for doing backups. As a general rule, memory is smaller, more expensive and much faster than storage so there’s a trade-off. If you want a lot of speed, load up on fast memory but it’ll cost you cash and resilience.”
“I’ll bet that’s where your special skills come in handy, right, Robert?”
“Pretty much, Vinnie. The trick is to get the right data into the right kind of memory at the right time.”
“The right kind…?”
“Ohhhyeah, there’s a whole hierarchy out there — on-chip memory essentially inside the processor, on-board memory on separate chips, off-board memory and storage…. It goes on all the way out to The Cloud if you’re set up that way. There’s even special memory for keeping track of which data is where in the other memories. The internal network plays into it, too — the data bus to a given memory could be just a byte wide or many times fatter, which makes a big difference in access speed. The hardware takes care of some data placement automatically, but a lot of it we can affect with the software. That’s mostly where I come in.”
“Doin’ what? The hardware’s pretty much what your boss already bought, not much you can tinker with there. The bits are zoomin’ around inside at electronic speeds, you can’t pick and choose where to put ’em.”
“Yes, we can, if we’re smart and careful. You know Michael Corleone’s line, ‘Keep your friends close but your enemies closer‘? With us it’s ‘Keep your next data byte close but your next program instruction closer.'”
“Whuzzat mean?”
“What you want to do is have bytes ready for the processor as soon as it’s ready to work with them. That means predicting which bytes it’ll want next and getting those to the top of the memory pyramid. Programs do a lot of short loops, enough that standard architectures have separate instruction memories just for that.”
“So how do you do that predicting? Like Vinnie said, things move fast in there.”
“You design for patterns. My favorite is sequential-and-discard. When you’re watching a movie you look at frames in series and you rarely go back. In the computer we deliver sequential bytes in an orderly manner to fast memory but we don’t have to worry about storing them back out again. Easy-peasy. Sequential-and-store is also highly predictable but then you have to down-copy, too.”
“Yeah, either way the data just flows through. What others?”
“Periodic is useful if you can arrange your program and data to exploit it. If you know a just-used series of bytes are going to be relevant again soon, you try to reserve enough close-in memory to hold onto them. Data references tend to spread out but sometimes you can tilt the odds by clumping together related bytes that are likely to be used together — like all weather data for one location.”
“What if you don’t have any of those patterns?”
“Worst case scenario. You guess periodic, buy lots of memory and cross your fingers.”
~~ Rich Olcott
Hard Science Ain't Hard 
