Lessons From Computer Organization

I recently took a class on computer organization at UCLA and it opened my eyes to how computers really work. It can be difficult to see past all the complexity in frameworks, libraries, languages, and design patterns, but when you get down to the core of what programming is really about, it’s really quite beautiful. I think there is a lot of value in learning how underlying systems work in every field because even though we rarely interact with them, they inform almost every higher-level decisions we make. So here are a few lessons I took away from this class that I think are worth remembering.

  1. It’s all a matter of perspective. Data types aren’t encoded into assembly. It only sees things as data and instructions so the instructions determine how the data is interpreted.
  2. Keep your code small and reusable - RISC vs. CISC also reused code means a smaller binary. Also simpler programs often run faster, loop unrolling.
  3. User input and output from your system (although output is usually user input too) are often the sources of vulnerabilities. It is difficult to break into a self-contained program. only when chaotic input from the outside world enters your system is there an opportunity for exploitation. (for example, is that buffer really only going to hold 40 bytes). Watch the assumptions you make about what your user because you will find most often that those assumptions are not guaranteed. Guard around it carefully.
  4. Caching is one of the most important concepts in programming. Algorithms can only take you so far if caching hasn’t been optimized, and often the two go hand in hand - you need to optimize your algorithm for caching. Data retrieval is likely one of the slowest parts of any program, especially those which heavily rely on databases. It’s easy to feel good after bringing your code closer to your users with things like serverless technology that runs programs on the edge, but there is no real payoff unless you’ve also thought about moving your data closer (think replicas, redis). It doesn’t matter if your algorithm finishes the task 2x as fast as before if fetching the data took 100x as long as the program. Also caching is meaningless if your program doesn’t account for how it is structured, code must be written to take advantage of temporal and spatial locality. Data accesses should be close to each other (i.e. one fetch can get all of it). Think SQL, your queries become more complex and slower as you have to access more tables. It should also be used soon after each other so that it is more likely to be in hot cache paths. Think Redis, data is more likely to be there if it was accessed only a few minutes before instead of a few hours.
  5. What you see is not what you get. Just because you write code in a certain order does not mean it will execute in that order. Thus, guarantees matter. Pay attention to what APIs, libraries, platforms, and even compilers are guaranteeing you when you use them. Don’t assume that your thoughts match reality.
  6. Concurrency can result in order of magnitude performance increases. And yes this is obvious but I think it’s still important to take explicit note of. Before assembly is executed, compilers take note of data dependencies between instructions and move them around so they can be executed concurrently by the CPU. And even on top of that, CPUs have instruction pipelines in which instructions are pre-fetched and can be executed according to what resources are currently available on the CPU. We should think the same way albeit on a more macro level. It pays off to extract false data dependencies and parallelize calls to them.
  7. Calls to outside systems are not simply a dependence, they are a transfer of control unless you have a mechanism to take back control. When you perform an action that depends on the operating system, a compiler will encode the syscall into your instructions (like open()-ing a file). When the CPU executes the syscall, your program loses control (it will pause execution) until the operating system returns it. Similarly, if you call an API, your program may be waiting on its results. You are no longer driving your program, that external API is. Of course most systems will not fall prey to this because built-in measures such as request timeouts guard quite well against it, but it’s always good for your program to remain the main player in executing a task and have explicit code to return control so you know what the guarantees are.
  8. The low-level stuff has a huge impact. This does not mean learning assembly, although it would definitely be useful. It means learning how computers actually work. And I don’t claim to even be close to that but even knowing as little as I know from this one class can take you quite far in terms of knowing how to approach questions of optimization or in general thinking about how solve problems in a way that is consistent with how computers function.