Andrew Yurisich a collection of things

Staying Sharp in Software Development

This is my fifth year writing software. As I continue my development in the trade, I find myself spending more and more time interacting with people who's level of contribution now exceeds that of an "individual contributor". They are much too talented to be writing any more code, so their superiors would like to say. Their time would be much better spent discussing the problems that some given software is trying to solve, and ideally, never write any to begin with. In those cases where software is needed, they spend their efforts scaffolding out initial solutions, architecting the patterns for others to fill in with details. I would argue that these all play a part in a successful version of the software development life cycle.

I have no issue going on the record by publicly stating that I have no interest in being a member of this group. In fact, I'll go as far as saying that the only way to stay truly sharp in the ever-evolving world of software development is to periodically reset your career to that of a junior hire every couple of years. Any analogy that you might construct to convince me otherwise is likely to apply poorly to writing software. We are not generals in a war, where "boots on the ground" can't be wasted on the likes of our best and brightest. Oftentimes, there isn't a clean way to truly solve (or even understand) a problem without becoming the entry-level developer who invariably gets tasked with that whole "details" part of the process. I find it odd that we leave our best and brightest to identify the obvious parts of the problem, while leaving the costliest, most error prone mistakes to be made by those who are in the prime of their mistake-making part of their careers.

For an example, take the common case: business software. At the highest, highest levels, there is a need for some payroll system to be put in place for a new company. For someone at a director level, there is a tax issue preventing them with going through a typical payroll servicing company. For an architect, there are tax disadvantages by using pre-fabricated payroll software, so it'll need to be coded in house. At this point, much of the hard work is done, in a sense. The only thing left is to actually make that list of exceedingly more detailed requirements into the most detailed requirements — computer software. Compare this with something like research, where those at the highest levels decide there is a practical business need to build self-driving cars. A director discovers that the technologies behind this don't exist yet, or at best, are new, unvetted, and rough. Where do you begin with those requirements? No amount of architecting can suddenly make computer vision easier. Those are hard details, and they must be orchestrated in a way that makes sense when conforming to something as massive and complex as the motor roadway laws of California.

A great developer will be able to see the forest through the trees, while running through it with a group of four people. Obviously, you can't do it forever, but again, many analogies fail here. It's a lot like hunting, except the barrier to exit is much, much lower. You can be a great hunter, and know where the biggest game tend to gather, but your insights will only take those doing the actual hunting so far. When you're no longer the one stringing the bow, it's easy to forget that the hardest part of hunting is "don't miss". If you find yourself suddenly supporting a village too large to do the hunting yourself anymore, it may be time to start designing better bows and arrows for them to use.

What do you have to lose in a skill like software development? Your ability to type? Your eyesight? Those things last for seventy years or more in a healthy citizen living in a first-world country. The only major loss you face as a real threat is your ability to stay active in a world that is dominated by the new technologies, authors, tools and workflows that can too easily slip from your grasp as you become too senior of a developer.

The Wolf of Wallstreet: ★ ★ ★ ★ ☆

Poster from the movie 'The Wolf of Wall Street'

What impressed me most about this film is that the book it was based on was written by Jordan Belfort, further proving that you can get rich writing a book about getting rich. Although the gratuitous nudity and drug use make it difficult to publicly recommend, the charm of a rags to riches story is certainly enhanced here by revisiting the rags, and the trip back and forth is conveyed with plenty of style.

I felt Leonardo DiCaprio's performance was judged too harshly by those saying he over acted in hopes of sealing an academy award in his rendition of Jordan Belfort. The beginning of the film, which features his character working through a brief stage of worldly naivety, is completely convincing and serves as a stark contrast to the middle of the film, which left him barely recognizable in my eyes. That alone demonstrates his versatility in portraying both the pedestrian sections of the script, as well as the "Oscar baiting" scenes.

In terms of what could be improved, the time ran a bit too long at 3 hours. It may have been served some favors by reducing a few of the many pep rallies that went above and beyond in establishing the mood of a money-hungry investment firm. I found myself re-watching parts of the film, but consistently skipping those scenes in particular (or at least large parts of them). I also felt the connection between Matthew McConaughey's character and Leonardo DiCaprio's character was too thin to keep re-establishing throughout the film. Perhaps if his mentorship had grown past anything but a single drug-fueled lunch, I would have less of a problem believing the impact it had on Jordan's development throughout the film.

Alternative Github Tags

Most github users are aware of tagging issues and pull requests to organize them better. Some also know that it's a good idea to create your own tags to suit your team's individual needs. Our team uses some tags that help deal with communication in larger projects with many collaborators. They're not the usual permanent issue labels, but rather, stateful indicators that alert teammates what the status of a pull request is at a glance.

The Background

Like many experienced git users, I make branches for nearly everything. And as an experienced git user in a team-based environment, I tend to stagger those branches to introduce large changes in the following order.

  1. Outline the impact, typically by changing the directory structure.
  2. Submit blank tests that serve as a rough contract of the change.
  3. Write code that would allow the tests to pass.
  4. Write tests.

The advantages of this approach are huge. Pull requests generally hover around ~200 lines added or removed at any given time. Each phase of the full change never breaks any build automation, and if it does, there's an opportunity to discuss the value, the approach, or the technique used at that point, or at any point before it. You can also get a good idea of what needs to happen in subsequent pull requests as you go along. For me, having 700 or 900 lines of code changes in a single pull request is unacceptable (unless you're checking in machine generated data).

The downsides are pretty obvious too — both myself and the reviewer are in charge of juggling possibly four or more pull requests all at once. If there's a typo in the second pull request, I'll have to introduce it there, and rebase all following branches on it again. This can lead to excessive force-pushing, and extra communication with any reviewers who like to check out the code locally as they review.

The Summary

One of the main issues I had with this workflow involved the previously mentioned communication overhead. I'd see three or four of my own pull requests open, and four comments split up over two of them. I'd need to constantly remind myself which branch needed what changes, especially for routine tasks like rebasing against master. We discussed the idea of having tags that don't represent the essence of an entire issue, but rather, the current state of an open pull request.

Eventually we agreed to add the following tags. They've been very helpful for us, and you should consider adding them to your project as well.

  • Pull Request Reviewed with Comments

A persistent "you've got mail" for the pull request. Once cleared, this means the submitter has responded to comments.

  • Pull Request Merge on CI/CD Passes

Helpful for when you want to let everyone know that the pull request is good, and no other actions need to be taken once the automation runs with a clean exit. I've had a great time merging two or three pull requests I knew nothing about this way.

  • Pull Request Needs Rebase

For when a pull request stays open too long and another pull request has been merged ahead of it that makes a conflict.

  • Pull Request Needs Squash

Lets the pull request owner know that they've addressed reviewer feedback successfully, and need to tidy up the commit log by melding all commits into one.

Once any of the above tasks are completed, the submitter removes the tag.

More Traditional Issue Tags

Aside from those temporary tags, here are other handy "traditional" tags to add to your github pull requests.

  • Needs Discussion

A signal for other reviewers to weigh in on a possible disruptive change to the codebase.

  • Large/Small Pull Request

Subjectively applied to pull requests that will likely be prioritized due to its size. It's nice to know which pull requests will take 5 minutes to review, and which will take 5 hours.

  • 3rd Party Version Change

Makes it more apparent to those who review your code locally to refresh their third party dependencies before trying to vet the changes.

  • Breaking Change

Introduces a backwards-incompatible change to anyone depending on the code featured in the pull request.

Silence Noisy Selenium Server Ouput in Travis CI

If you use Travis CI to run your end to end tests, you may want to start with something simple and run a local selenium server to communicate with the browser. Although SauceLabs is a really powerful tool for mature, cross-browser projects with solid testing procedures, it can be a bit of a burden to sign up for, pay for, and configure it for Travis CI runs. For something like the project I'm working on, we only care about testing against a modern install of Firefox, and doing this is as simple as adding the following lines to your project's .travis.yml file.

language: node_js
node_js:
- '0.10'
before_install: npm install -g protractor@1.0.0 mocha@1.18.2
install:
- npm install
- webdriver-manager update --standalone
before_script:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- webdriver-manager start &
script:
- protractor protractor.conf.js

This is a bare-bones setup for running (free) end to end tests using only the Travis VM, and on first glance appears normal. And you'd be right thinking this. Although that last command does start a background selenium server, it has an awful side effect of polluting the logs with output containing javascript that runs against the browser during the test run.

For example, here is just one of the many tests that run on every pull request of the encore-ui project I've been contributing to lately.

  rxDiskSize
17:01:46.133 INFO - Executing: [get: data:text/html,<html></html>])
17:01:46.168 INFO - Done: [get: data:text/html,<html></html>]
17:01:46.175 INFO - Executing: [execute script: window.name = "NG_DEFER_BOOTSTRAP!" + window.name;window.location.replace("http://localhost:9001/#/component/rxDiskSize");, []])
17:01:46.231 INFO - Done: [execute script: window.name = "NG_DEFER_BOOTSTRAP!" + window.name;window.location.replace("http://localhost:9001/#/component/rxDiskSize");, []]
17:01:46.261 INFO - Executing: [execute script: return window.location.href;, []])
17:01:46.520 INFO - Done: [execute script: return window.location.href;, []]
17:01:46.555 INFO - Executing: [execute async script: try { return (function (attempts, asyncCallback) {
  var callback = function(args) {
    setTimeout(function() {
      asyncCallback(args);
    }, 0);
  };
  var check = function(n) {
    try {
      if (window.angular && window.angular.resumeBootstrap) {
        callback([true, null]);
      } else if (n < 1) {
        if (window.angular) {
          callback([false, 'angular never provided resumeBootstrap']);
        } else {
          callback([false, 'retries looking for angular exceeded']);
        }
      } else {
        window.setTimeout(function() {check(n - 1);}, 1000);
      }
    } catch (e) {
      callback([false, e]);
    }
  };
  check(attempts);
}).apply(this, arguments); }
catch(e) { throw (e instanceof Error) ? e : new Error(e); }, [10]])
17:01:46.567 INFO - Done: [execute async script: try { return (function (attempts, asyncCallback) {
  var callback = function(args) {
    setTimeout(function() {
      asyncCallback(args);
    }, 0);
  };
  var check = function(n) {
    try {
      if (window.angular && window.angular.resumeBootstrap) {
        callback([true, null]);
      } else if (n < 1) {
        if (window.angular) {
          callback([false, 'angular never provided resumeBootstrap']);
        } else {
          callback([false, 'retries looking for angular exceeded']);
        }
      } else {
        window.setTimeout(function() {check(n - 1);}, 1000);
      }
    } catch (e) {
      callback([false, e]);
    }
  };
  check(attempts);
}).apply(this, arguments); }
catch(e) { throw (e instanceof Error) ? e : new Error(e); }, [10]]
17:01:46.580 INFO - Executing: [execute script: angular.resumeBootstrap(arguments[0]);, [[]]])
17:01:46.990 INFO - Done: [execute script: angular.resumeBootstrap(arguments[0]);, [[]]]
17:01:47.022 INFO - Executing: [execute async script: try { return (function (selector, callback) {
  var el = document.querySelector(selector);
  try {
    angular.element(el).injector().get('$browser').
        notifyWhenNoOutstandingRequests(callback);
  } catch (e) {
    callback(e);
  }
}).apply(this, arguments); }
catch(e) { throw (e instanceof Error) ? e : new Error(e); }, [body]])
17:01:47.032 INFO - Done: [execute async script: try { return (function (selector, callback) {
  var el = document.querySelector(selector);
  try {
    angular.element(el).injector().get('$browser').
        notifyWhenNoOutstandingRequests(callback);
  } catch (e) {
    callback(e);
  }
}).apply(this, arguments); }
catch(e) { throw (e instanceof Error) ? e : new Error(e); }, [body]]
17:01:47.059 INFO - Executing: [find elements: By.selector: .component-demo ul li])
17:01:47.071 INFO - Done: [find elements: By.selector: .component-demo ul li]
17:01:47.101 INFO - Executing: [get text: 36 [[FirefoxDriver: firefox on LINUX (54c8e519-0211-45d5-a1f2-64bb2526e652)] -> css selector: .component-demo ul li]])
17:01:47.114 INFO - Done: [get text: 36 [[FirefoxDriver: firefox on LINUX (54c8e519-0211-45d5-a1f2-64bb2526e652)] -> css selector: .component-demo ul li]]
  ✓ should still have 420 GB as test data on the page

Even though I feel it goes without saying, this kind of output is not something I look forward to seeing, especially when my end to end tests are breaking the build. In fact, many of the pull request test runs wind up maxing out the logs at 10000 lines, potentially killing all ability to debug a broken test. This issue can compound when faced with a bug that isn't reproducible on my machine, but appears to break in the CI environment.

Fortunately, you can literally change one line in your project's .travis.yml file to prevent this sort of behavior.

before_script:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
# Immune to logouts, but not VM deprovisions!
- nohup bash -c "webdriver-manager start 2>&1 &"

The end result is a clean, informative test run report that looks just the one you saw on your development machine before you pushed up your branch (at least, hopefully you ran the end to end tests before you pushed up your branch).

$ protractor protractor.conf.js

Using the selenium server at http://localhost:4444/wd/hub
  rxDiskSize
  ✓ should still have 420 GB as test data on the page
  ✓ should convert 420 GB back to gigabytes
  ✓ should still have 125 TB as test data on the page
  ✓ should convert 125 TB back to gigabytes
  ✓ should still have 171.337 PB as test data on the page
  ✓ should convert 171.337 PB back to gigabytes
  ✓ should still have 420 GB as test data on the page
  ✓ should convert 420 GB back to gigabytes
  ✓ should still have 125 TB as test data on the page
  ✓ should convert 125 TB back to gigabytes
  ✓ should still have 171.337 PB as test data on the page
  ✓ should convert 171.337 PB back to gigabytes

  12 passing (3s)

The command "protractor protractor.conf.js" exited with 0.

Full disclosure: I found this helpful tip while browsing codeship.io's excellent documentation, which provides a service similar to Travis CI, and is also my personal preference for my side projects. However, this article was written for those who, like myself, find themselves encountering Travis CI in a project that they don't control.

Github Pull Request Statuses with drone.io

The free CICD site drone.io is a really useful, lightweight replacement for Jenkins for free, open source repositories. Being purpose built for just one thing means setting up a new project takes literally a tenth of the time compared to Jenkins.

But one thing it is lacking is the ability to run CICD tests against new pull requests that have had master merged into it first, and updating the status in the pull request while it's doing that. Most Google searches for help will uncover forum posts with users requesting this feature, followed by promises from the drone.io team to get around to adding this functionality in "six to eight weeks".

Fortunately, you can roll these things out yourself. Just follow these steps:

  • Create an OAuth Token from your github user profile.
    • You only need to enable the public_repo and repo:status scopes.
  • In your drone.io settings page, paste the token you got from github into the "Environment Variables" textbox.

For example:

drone.io project settings page.
  • Next, add some code that will allow you to replicate the behavior that is available in other CICD providers, such as TravisCI.
function postStatus {
  curl -so /dev/null -X POST -H "Authorization: token $OAUTH_TOKEN" -d "{\"state\": \"$1\", \"target_url\": \"$DRONE_BUILD_URL\", \"description\": \"Built and tested on drone.io\", \"context\": \"Built and tested on drone.io\"}" https://api.github.com/repos/:YOUR_NAME:/:YOUR_REPO:/statuses/$DRONE_COMMIT;
}

postStatus pending
git merge master $DRONE_BRANCH
if [ $? -ne "0" ]; then
  echo "Unable to merge master..."
  postStatus error
  exit 1
fi

Finally, add code for your specific project to actually do some testing. Here's a simple example that you might use for a node app.

npm install --loglevel=warn
if [ $? -ne "0" ]; then
  postStatus error
  exit 1
fi

npm test
if [ $? -eq "0" ]; then
  postStatus success
else
  postStatus failure
fi

That's it! Once your vanilla setup with drone.io is set up, everything else should just work.