Andrew Yurisich a collection of things

Silence Noisy Selenium Server Ouput in Travis CI

If you use Travis CI to run your end to end tests, you may want to start with something simple and run a local selenium server to communicate with the browser. Although SauceLabs is a really powerful tool for mature, cross-browser projects with solid testing procedures, it can be a bit of a burden to sign up for, pay for, and configure it for Travis CI runs. For something like the project I'm working on, we only care about testing against a modern install of Firefox, and doing this is as simple as adding the following lines to your project's .travis.yml file.

language: node_js
node_js:
- '0.10'
before_install: npm install -g protractor@1.0.0 mocha@1.18.2
install:
- npm install
- webdriver-manager update --standalone
before_script:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- webdriver-manager start &
script:
- protractor protractor.conf.js

This is a bare-bones setup for running (free) end to end tests using only the Travis VM, and on first glance appears normal. And you'd be right thinking this. Although that last command does start a background selenium server, it has an awful side effect of polluting the logs with output containing javascript that runs against the browser during the test run.

For example, here is just one of the many tests that run on every pull request of the encore-ui project I've been contributing to lately.

  rxDiskSize
17:01:46.133 INFO - Executing: [get: data:text/html,<html></html>])
17:01:46.168 INFO - Done: [get: data:text/html,<html></html>]
17:01:46.175 INFO - Executing: [execute script: window.name = "NG_DEFER_BOOTSTRAP!" + window.name;window.location.replace("http://localhost:9001/#/component/rxDiskSize");, []])
17:01:46.231 INFO - Done: [execute script: window.name = "NG_DEFER_BOOTSTRAP!" + window.name;window.location.replace("http://localhost:9001/#/component/rxDiskSize");, []]
17:01:46.261 INFO - Executing: [execute script: return window.location.href;, []])
17:01:46.520 INFO - Done: [execute script: return window.location.href;, []]
17:01:46.555 INFO - Executing: [execute async script: try { return (function (attempts, asyncCallback) {
  var callback = function(args) {
    setTimeout(function() {
      asyncCallback(args);
    }, 0);
  };
  var check = function(n) {
    try {
      if (window.angular && window.angular.resumeBootstrap) {
        callback([true, null]);
      } else if (n < 1) {
        if (window.angular) {
          callback([false, 'angular never provided resumeBootstrap']);
        } else {
          callback([false, 'retries looking for angular exceeded']);
        }
      } else {
        window.setTimeout(function() {check(n - 1);}, 1000);
      }
    } catch (e) {
      callback([false, e]);
    }
  };
  check(attempts);
}).apply(this, arguments); }
catch(e) { throw (e instanceof Error) ? e : new Error(e); }, [10]])
17:01:46.567 INFO - Done: [execute async script: try { return (function (attempts, asyncCallback) {
  var callback = function(args) {
    setTimeout(function() {
      asyncCallback(args);
    }, 0);
  };
  var check = function(n) {
    try {
      if (window.angular && window.angular.resumeBootstrap) {
        callback([true, null]);
      } else if (n < 1) {
        if (window.angular) {
          callback([false, 'angular never provided resumeBootstrap']);
        } else {
          callback([false, 'retries looking for angular exceeded']);
        }
      } else {
        window.setTimeout(function() {check(n - 1);}, 1000);
      }
    } catch (e) {
      callback([false, e]);
    }
  };
  check(attempts);
}).apply(this, arguments); }
catch(e) { throw (e instanceof Error) ? e : new Error(e); }, [10]]
17:01:46.580 INFO - Executing: [execute script: angular.resumeBootstrap(arguments[0]);, [[]]])
17:01:46.990 INFO - Done: [execute script: angular.resumeBootstrap(arguments[0]);, [[]]]
17:01:47.022 INFO - Executing: [execute async script: try { return (function (selector, callback) {
  var el = document.querySelector(selector);
  try {
    angular.element(el).injector().get('$browser').
        notifyWhenNoOutstandingRequests(callback);
  } catch (e) {
    callback(e);
  }
}).apply(this, arguments); }
catch(e) { throw (e instanceof Error) ? e : new Error(e); }, [body]])
17:01:47.032 INFO - Done: [execute async script: try { return (function (selector, callback) {
  var el = document.querySelector(selector);
  try {
    angular.element(el).injector().get('$browser').
        notifyWhenNoOutstandingRequests(callback);
  } catch (e) {
    callback(e);
  }
}).apply(this, arguments); }
catch(e) { throw (e instanceof Error) ? e : new Error(e); }, [body]]
17:01:47.059 INFO - Executing: [find elements: By.selector: .component-demo ul li])
17:01:47.071 INFO - Done: [find elements: By.selector: .component-demo ul li]
17:01:47.101 INFO - Executing: [get text: 36 [[FirefoxDriver: firefox on LINUX (54c8e519-0211-45d5-a1f2-64bb2526e652)] -> css selector: .component-demo ul li]])
17:01:47.114 INFO - Done: [get text: 36 [[FirefoxDriver: firefox on LINUX (54c8e519-0211-45d5-a1f2-64bb2526e652)] -> css selector: .component-demo ul li]]
  ✓ should still have 420 GB as test data on the page

Even though I feel it goes without saying, this kind of output is not something I look forward to seeing, especially when my end to end tests are breaking the build. In fact, many of the pull request test runs wind up maxing out the logs at 10000 lines, potentially killing all ability to debug a broken test. This issue can compound when faced with a bug that isn't reproducible on my machine, but appears to break in the CI environment.

Fortunately, you can literally change one line in your project's .travis.yml file to prevent this sort of behavior.

before_script:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
# Immune to logouts, but not VM deprovisions!
- nohup bash -c "webdriver-manager start 2>&1 &"

The end result is a clean, informative test run report that looks just the one you saw on your development machine before you pushed up your branch (at least, hopefully you ran the end to end tests before you pushed up your branch).

$ protractor protractor.conf.js

Using the selenium server at http://localhost:4444/wd/hub
  rxDiskSize
  ✓ should still have 420 GB as test data on the page
  ✓ should convert 420 GB back to gigabytes
  ✓ should still have 125 TB as test data on the page
  ✓ should convert 125 TB back to gigabytes
  ✓ should still have 171.337 PB as test data on the page
  ✓ should convert 171.337 PB back to gigabytes
  ✓ should still have 420 GB as test data on the page
  ✓ should convert 420 GB back to gigabytes
  ✓ should still have 125 TB as test data on the page
  ✓ should convert 125 TB back to gigabytes
  ✓ should still have 171.337 PB as test data on the page
  ✓ should convert 171.337 PB back to gigabytes

  12 passing (3s)

The command "protractor protractor.conf.js" exited with 0.

Full disclosure: I found this helpful tip while browsing codeship.io's excellent documentation, which provides a service similar to Travis CI, and is also my personal preference for my side projects. However, this article was written for those who, like myself, find themselves encountering Travis CI in a project that they don't control.

Github Pull Request Statuses with drone.io

The free CICD site drone.io is a really useful, lightweight replacement for Jenkins for free, open source repositories. Being purpose built for just one thing means setting up a new project takes literally a tenth of the time compared to Jenkins.

But one thing it is lacking is the ability to run CICD tests against new pull requests that have had master merged into it first, and updating the status in the pull request while it's doing that. Most Google searches for help will uncover forum posts with users requesting this feature, followed by promises from the drone.io team to get around to adding this functionality in "six to eight weeks".

Fortunately, you can roll these things out yourself. Just follow these steps:

  • Create an OAuth Token from your github user profile.
    • You only need to enable the public_repo and repo:status scopes.
  • In your drone.io settings page, paste the token you got from github into the "Environment Variables" textbox.

For example:

drone.io project settings page.
  • Next, add some code that will allow you to replicate the behavior that is available in other CICD providers, such as TravisCI.
function postStatus {
  curl -so /dev/null -X POST -H "Authorization: token $OAUTH_TOKEN" -d "{\"state\": \"$1\", \"target_url\": \"$DRONE_BUILD_URL\", \"description\": \"Built and tested on drone.io\", \"context\": \"Built and tested on drone.io\"}" https://api.github.com/repos/:YOUR_NAME:/:YOUR_REPO:/statuses/$DRONE_COMMIT;
}

postStatus pending
git merge master $DRONE_BRANCH
if [ $? -ne "0" ]; then
  echo "Unable to merge master..."
  postStatus error
  exit 1
fi

Finally, add code for your specific project to actually do some testing. Here's a simple example that you might use for a node app.

npm install --loglevel=warn
if [ $? -ne "0" ]; then
  postStatus error
  exit 1
fi

npm test
if [ $? -eq "0" ]; then
  postStatus success
else
  postStatus failure
fi

That's it! Once your vanilla setup with drone.io is set up, everything else should just work.

Occam's Razor as a Debugging Tool

On Friday, I had finished my tasks for the day a little early and decided to check up on the co-workers around me, just to see if any of them had anything blocking them, or anything interesting they'd like to show off to me. It turns out one had been stuck on an issue for the last 45 minutes. He had at least fifteen tabs open, sighed a lot, and was visibly exhausted. Basically, he was sending about a clear a signal as one can that they're debugging some odd behavior. The issue at hand revolved around a really pleasant function, as far as troubleshooting goes.

var transformUser = function (users) {
    return _.omit(users, '_links');
};

Suddenly, this function had started returning a near empty (but successful) response on staging. Nobody was sure why, and seeing how his team had promoted a build to staging earlier that day, he had been digging deep into his code base, and its dependencies, trying to find the culprit.

I immediately asked him to show me what the correct response should look like. He made a call directly to the upstream api using Postman, and returned a list of users that would be passed into this function. This was the exact same call his app was making. So, we copied the JSON payload, ssh'd into staging, and used the node command under the project's directory, ensuring the dependencies were an exact match. Sure enough, we found that lodash's omit function still worked just fine. But what was the next step?

I said, "change the function to this".

var transformUser = function (users) {
    return users;
    // return _.omit(users, '_links');
};

He looked at me like I had just asked him to double check if his computer was still connected to the internet. At first he ignored me, saying he just showed me the users call in Postman, and introduced a couple other leads that he'd been looking into now that we'd ruled out lodash. But I insisted, telling him it'd only take a couple of seconds. He did, and ran the broken call again. To his surprise, and my amusement, something like this came back.

"\"{\"users\":{\"_links\":{\"data\":{...}}},{\"parents\":\"[...]\"}}\""

Apparently, in an attempt to be more RESTful, the api returning the users response had added custom content types that his Postman client either didn't need or had been configured to use, but had not been updated in the application, which (I suppose) had stricter requirements for using the correct content-type when requesting resources. A few hours later, the upstream api did him a favor and allowed a request to specify application/json as its desired content-type and still receive a json encoded response, fixing the root issue with no code change needed.

Occam's Razor has many interpretations. Just as you can use it for developing new features (or in this case, api tests!), you can also use it for debugging. Start by validating the most trivial, essential assumptions surrounding the issue before venturing into trickier territory. As much fun as it is to track down and isolate odd behavior in bizarre places (and brag about it after the fact), the reality is that most bugs are far more likely to be pedestrian ones like this.

Don't Link that Line Number!

Today I saw a co-worker take a link from a specific line number in our team's github repository and place it in the comments section of a user story. This is very helpful for those who need to see a quick reference to the code that could be causing a defect, but it's wrong. Here's what you need to do instead.

First, the original setup had a link that looked like this.

https://github.com/Droogans/.emacs.d/blob/mac/init.el#L135-L138

A snippet from a current github file.

Looks great, right? Here's that same link a few weeks later.

A snippet from an out of date github file.

The problem with linking to line numbers is that if someone were to add some new code above line number 135, your link will still point there, regardless if the new code makes any sense. It's not anchored to the code, just to the line numbers in the file.

Here's what you do instead. First, highlight the code that you want to link to, just like in the examples above. Then, tap the "y" key to jump to the last commit found for that region.

That's it! You've now anchored the code in question to an immutable reference in github's history of your project. No matter what happens, this commit will remain unique forever, and you won't have to worry about having to work out what that old reference used to point to.

Use your Editor for Pull Request Reviews

When reviewing pull requests, I've noticed it's way too easy to scan through the green and red lines, compare them quickly in your mind, and make hasty decisions about what's changed in the code.

Here's a secret: when code changes, it changes in more places than what's shown in the diff. It sounds strange, but it's true. You look at the diff of a two line change, scan the green line, the red line, and think, "This is doing the same thing, just a little differently. And it fixes a bug. Looks good to me". You merge it, and later that week, end up looking at the file on an unrelated task while following a stack trace. Suddenly, you see the method in the context of the file: the usage of this other class' method is borderline abusive. What's it doing here? It wasn't meant to be used in these types of situations! Who did this? You run git blame, and trace it back to a pull request that you merged just a few days ago.

If this hasn't happened to you, it will eventually, given your team and codebase grow to such a size as to make it impossible for one person to have mastery over all of it. So in an effort to reduce the silo effect, you may participate in code reviews. The number one way to do this is in the web interface provided by github.

A sample of github's web based diff view.

Although convenient, it lacks several things:

  1. Your keyboard is useless in this view.
  2. Your color scheme is probably not the same.
  3. Your font is likely different than what you're used to.
  4. The context of the surrounding code is weak at best.

And most importantly, reading diffs is hard. I imagine a developer spends at best a tiny fraction of their time looking at code through the lens of a diff file. The overwhelming majority of the time is spent reading code in the context of wielding it, via an editor of some kind. Looking at code in an editor forces you to stop reading the code and start comprehending it instead.

Reading diffs is like shopping for clothes online. You can get a good idea of what you think it's like, but until it arrives in a more tangible form, you won't know how it fits and feels overall. So from now on, consider adding an extra 30 seconds to your code review process, and fetch that branch, check it out, and take it for a spin. I guarantee you'll notice a difference in your review process right away. You don't write your code using the hilariously inadequate github web interface, so why should you settle for reading it in a similarly crippled fashion?

I would go as far to say that github should expose an option for me to completely disable diff previews for pull requests, and their respective commits on my team. Instead, it would only tell you what files changed, and where. It's up to you to see what those changes were.

There should only be one way to look at code: in the same circumstances that it was written in, and in the same environment it will be interacted with from now on.