PComp Labs Week 9
Project 2: Autolock

I built my Project 2 -- a pressure-sending autolocking mechanism for my computer -- according to last week's plan, with basically no modifications or extensions. It turns out that doing just the basics reliably was tough enough.

This project buildout was particularly long. Instead of organizing my process and the challenges that came up entirely chronologically, I will group parts together based on theme, for clarity.

How it started

First, I'd like to give a shoutout to my friends at Affinity, my former employer, for inspiring me to take computer security more seriously, in possibly the most fun way possible:

After this happened, I created an open offer for anyone to challenge my security practices. This eventually spread throughout the office, with various people offering other prizes, with the most common one being donuts. (We called it getting "donutted.")

Was this challenge fun? Yes! Did it make me better at keeping my computer locked all the time? Well, maybe not. I got "donutted" fairly frequently, even more than a year later:

This experience has made me think more about computer security, but I'd say that it hasn't actually made me better at it. That's why I've been especially excited to create a device that can help me with it.

Now, as for how it's going...

App architecture
  • I made no modifications to the Arduino code! This was because (1) why not! and (2) I realized that having the "ASCII pressure meter" was an incredibly useful debugging tool -- and it is used in some of the pictures to follow. I instead wrote JS code to parse the this pressure meter -- simply by counting the number of # characters that were shown in each line of the readout.
  • The computer code is written in TypeScript using Deno, which is basically like Node.js with a more modern API and with TypeScript. I've been using Deno off and on for the past year (Deno 1.0 was released in May 2020) and it has been very nice to work with.
  • There are some operating system functions that are done outside of JS: locking the screen, querying whether the screen is locked, and querying time since last HID interaction. Deno makes it easy to call and communicate with external commands. Even reading serial output from the Arduino is done using cat because it was so easy!
Locking the screen reliably and only when necessary

There are command-line commands on macOS to put the computer to sleep, to shut it down, to restart it, to turn off the display. But for locking the screen? Not so much. The most reliable method I could find was simply invoking the appropriate keyboard shortcut: 

tell application id "com.apple.systemevents"
  keystroke "q" using {control down, command down}
end

I noticed that in a naive implementation, we would be attempting to lock the screen when the screen is already locked. How do we detect when the user has unlocked the screen?

First, I found the following script to directly show whether the screen is locked: 

import Quartz
cg_session = Quartz.CGSessionCopyCurrentDictionary()

One problem -- it is in Python, and the import Quartz step takes a long time! I decided to make this subscript a long-running process and allow it to periodically send the results of the screen-lock query back to the Deno process. Implementing handshaking allowed the Python process to not use too much CPU!

Something about the script was still not working. (Late note: I think the problem might have been that I followed the first piece of code in the answer without reading it all.) I added a second line of defense to check whether we're ready to autolock again: we check whether the user has interacted with the system. I found a script that prints out the number of seconds since the user last interacted with the computer. 

ioreg -c IOHIDSystem | awk '/HIDIdleTime/ {print $NF/1000000000; exit}'
Detection algorithm

Last week, it seemed easy enough to check whether the pressure sensors underneath the computer indicate that we should lock the computer. Lots of pressure = don't lock, little pressure = lock, and that's it, right? In practice, getting this right was the trickest part of the build.

I initially used a hardcoded pressure threshold to determine whether we should lock, but (1) the input data was fairly noisy and (2) the threshold kept changing depending on whether the laptop is placed.

To address the first issue, I added a rolling average calculation -- the program would only lock the computer if the average pressure over the past 100 time periods dropped below the threshold.

To combat the second issue, I added a calibration step -- when the user starts the program, they are prompted to remove their hands from the computer, so that the computer can sense what "hands off" "feels like".

This got much better results, but there would still be false positives when I was resting my hand on the computer but wasn't moving it around much, and false negatives when the calibration period just happened to be slightly less pressure-y than average.

After many algorithm iterations, I realized that a better measure might actually be that standard deviation of pressure over the past few time periods. When a person has their hand on the computer, there will be significantly more movement than there would be otherwise, almost invariably leading to a higher pressure variance.

The only exception occurred when the person was apply so much pressure that it stabilized their stance. I added in a significantly scaled-down average pressure to solve this problem.

The final formula: stdev(pressure) + 0.1 * avg(pressure). When this "score" drops below the score from the calibration period, we should lock the computer.

Hardware
Circuit wiring

I... kind of wanted to avoid soldering for this project. But I wanted to avoid using the huge breadboard, and I wanted to create a solid product. I found out that you can insert the wires into each other for a reliable connection.

Running sensor wires along computer underside

I need to have two sensors on two sides of the computer, so for one of the sensors, I would need to run wires from one side of the computer to the other. My laptop has very little clearance below it -- having wires below it would push it up. I realized that an elegant solution would be to use aluminum foil as wires.

Shell

I looked around for good enclosing materials. I ended up using the same piece of foam as the one I used in Project 1!

Final setup

It works!

Making sure the sensors still work!

Code 
import { readLines } from "https://deno.land/std@0.77.0/io/bufio.ts";

let textEncoder = new TextEncoder();
let textDecoder = new TextDecoder();

function sum(items: number[]) {
  return items.reduce((acc, c) => acc + c, 0);
}

function average(items: number[]) {
  return sum(items) / items.length;
}

async function writeFixedLine(line: string) {
  await Deno.stdout.write(textEncoder.encode("\r\u001b[K" + line));
}

function stdev(array: number[]) {
  const n = array.length;
  const mean = array.reduce((a, b) => a + b) / n;
  return Math.sqrt(
    array.map((x) => Math.pow(x - mean, 2)).reduce((a, b) => a + b) / n,
  );
}

async function delay(ms: number) {
  await new Promise((resolve) => setTimeout(resolve, ms));
}

async function osascript(script: string) {
  let process = Deno.run({
    cmd: ["osascript", "-e", script],
    stdout: "null",
  });
  await process.status();
}

async function lockScreen() {
  // Fake lock
  await osascript(`
    display dialog "Now we should lock!" buttons {"OK"} default button "OK"
  `);
  // Real lock
  // await osascript(`
  //   tell application id "com.apple.systemevents"
  //     keystroke "q" using {control down, command down}
  //   end
  // `);
}

class IsScreenLockedCheck {
  process: Deno.Process<{
    cmd: [string, string];
    stdin: "piped";
    stdout: "piped";
  }>;

  constructor() {
    this.process = Deno.run({
      cmd: ["python", "check_if_screen_locked.py"],
      stdin: "piped",
      stdout: "piped",
    });
  }

  async check() {
    await this.process.stdin.write(textEncoder.encode("\n"));
    for await (let line of readLines(this.process.stdout)) {
      return line === "True";
    }
  }
}

async function getHidInactiveTime(): Promise<number> {
  // https://stackoverflow.com/a/17966890
  let process = Deno.run({
    cmd: [
      "sh",
      "-c",
      "ioreg -c IOHIDSystem | awk '/HIDIdleTime/ {print $NF/1000000000; exit}'",
    ],
    stdout: "piped",
  });
  if (!(await process.status()).success) {
    throw new Error();
  }
  for await (let line of readLines(process.stdout)) {
    return Number(line);
  }
  throw new Error();
}

function serialProcess() {
  return Deno.run({
    cmd: ["bash", "-c", "cat /dev/cu.usbmodem*"],
    stdout: "piped",
  });
}

// https://stackoverflow.com/a/57364353
type Await<T> = T extends {
  then(onfulfilled?: (value: infer U) => unknown): unknown;
} ? U
  : T;

// type Process = Await<ReturnType<typeof serialProcess>>;

async function* readLevels(signal?: AbortSignal) {
  let process = serialProcess();
  if (signal) {
    signal.onabort = () => {
      process.kill(9);
    };
  }
  for await (const line of readLines(process.stdout)) {
    yield (line.match(/#/g) || []).length;
  }
}

async function calibrate(
  steps: number,
): Promise<number> {
  console.log("Remove your hands from the computer.");
  await delay(1000);

  let score = undefined;

  let recentLevels = [];
  let controller = new AbortController();
  for await (let level of readLevels(controller.signal)) {
    recentLevels.push(level);

    score = calculateScore(recentLevels);

    await writeFixedLine(`Calibrating... ${score.toFixed(3)}...`);

    if (recentLevels.length >= steps) {
      controller.abort();
      break;
    }
  }

  console.log(" done!");
  return score!;
}

function calculateScore(recentLevels: number[]) {
  return Math.min(...recentLevels) * 0.1 + stdev(recentLevels);
}

async function* readLevelAndScore(steps: number, signal?: AbortSignal) {
  let recentLevels: number[] = [];
  for await (let level of readLevels(signal)) {
    let isInitial = recentLevels.length < steps;
    recentLevels.push(level);
    if (!isInitial) {
      recentLevels.shift();
    }
    let score = calculateScore(recentLevels);
    yield [level, score, isInitial] as [number, number, boolean];
  }
}

async function run(steps: number, threshold: number) {
  console.log("Place your hands back on the computer.");

  while (true) {
    // console.log("Waiting for pressure.");
    // for await (let level of readLevels(process)) {
    //   if (level > threshold) break;
    // }

    console.log();
    console.log("Starting autolock.");

    let controller = new AbortController();
    for await (
      let [level, score, isInitial] of readLevelAndScore(
        steps,
        controller.signal,
      )
    ) {
      let startingText = isInitial ? " still collecting initial data" : "";
      await writeFixedLine(
        `Listening... ` +
          `level: ${level}  ` +
          `score: ${score.toFixed(3)}${startingText}...`,
      );

      if (!isInitial && score < threshold) {
        console.log();
        console.log("Locking!");
        console.log();
        await delay(200);
        await lockScreen();
        controller.abort();
        break;
      }
    }

    await delay(1000);

    while (await isScreenLockedCheck.check()) {
      await delay(1000);
    }
    // Sometimes this is buggy, so we do a second check.
    while ((await getHidInactiveTime()) > 1) {
      await delay(500);
    }

    console.log("Screen unlocked.");
  }
}

let isScreenLockedCheck = new IsScreenLockedCheck();
let threshold = await calibrate(100) + 0.05;
console.log(`Will sleep when score < ${threshold}`);
console.log();

run(100, threshold);
]]>
Code 
import Quartz
import sys

try:
  while True:
    raw_input()
    # https://stackoverflow.com/a/11511419
    cg_session = Quartz.CGSessionCopyCurrentDictionary()
    print(cg_session.valueForKeyPath_("CGSSessionScreenIsLocked"))
    sys.stdout.flush()

except KeyboardInterrupt:
  pass
Adjusting sensitivity

Pressure sensors below an already fairly-heavy object were not as sensitive as I imagined -- I remaining issue I had was the computer was autolocking sometimes when I was using my trackpad with a single finger, super lightly. I added a piece of foam below the computer for added elasticity and this solved the problem.

Feedback from class
  • I feel like this has a ton of practical applications1 Love it
  • Wow, I love how you used calculations instead of hard-coding thresholds that allow for flexibility
  • I feel like this is pretty accessible which I like, but what happens if someone is unable to put enough pressure? Is there a low threshold that remaps to a different scale?
  • So cool how you made all your parts into such minimal style! Love the idea!