atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[Engineering Health]]></title>
  <link href="http://engineering.cerner.com/atom.xml" rel="self"/>
  <link href="http://engineering.cerner.com/"/>
  <updated>2017-03-24T15:37:22-05:00</updated>
  <id>http://engineering.cerner.com/</id>
  <author>
    <name><![CDATA[Cerner]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[Engineers on the Road for SMART and FHIR (CHC 2016)]]></title>
    <link href="http://engineering.cerner.com/blog/engineers-on-the-road-for-smart-and-fhir/"/>
    <updated>2017-03-24T00:00:00-05:00</updated>
    <id>http://engineering.cerner.com/blog/engineers-on-the-road-for-smart-and-fhir</id>
    <content type="html"><![CDATA[<p>Over the past few years, Cerner has been developing a standards-based platform on top of our Cerner Millennium® EHR.  Rather than roll our own API, we’ve been using the <a href="http://hl7.org/fhir/overview-dev.html">HL7® FHIR® standard</a>.  For app integration, we’ve been using the <a href="http://docs.smarthealthit.org/">SMART® on FHIR specification</a>.  If you’re not immediately familiar with those acronyms, you’re definitely not alone.</p>

<p>As we were developing the services, we were also fielding questions from 3rd party developers, answering a lot of questions internally, and trying to keep up with the specifications themselves. We knew that we would need to help provide education around our ecosystem, our implementation of the FHIR and SMART standards to developers since the standards are new and evolving.</p>

<p>This need will become much more pressing because <a href="https://www.healthit.gov/providers-professionals/how-attain-meaningful-use">Meaningful Use 3</a> will trigger deployment of our implementations out to most of our Millennium EHRs.  <a href="https://www.cdc.gov/ehrmeaningfuluse/introduction.html">Meaningful Use</a> introduces requirements from the Centers for Medicare and Medicaid (CMS) to modernize the US healthcare infrastructure.  This is nothing new, but we’ve been hard at work to support stage 3, which includes the requirement for patients to access their data via an API.  In order to attest, most of our clients will then require our implementation of the FHIR API.</p>

<p>This past November offered up two exciting opportunities to provide hands-on instruction and support, which we’ll cover in separate posts.  In this first post, we want to highlight the work we did at our annual <a href="https://blogs.cerner.com/blog/CHC16-Summary-UK/">Cerner Health Conference</a>.  We decided that this year’s event would be a great opportunity to offer a small Code Learning Lab &ndash; providing education for some of the APIs that Cerner has available.</p>

<p>The Code Learning Lab was run from November 14th &ndash; 17th in Kansas City with our annual CHC conference. The goal: give developers a hands-on training session with the different APIs that are part of the Cerner Open Developer Experience (<em>code</em>). One of the tracks covered SMART and FHIR specifically, while another went over some of the APIs available with our HealtheIntent and HealtheLife platforms.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2017-03-24-engineers-on-the-road-for-smart-and-fhir/smart-and-fhir-01.jpg"></p>

<p>We saw participation from many of our clients and partners, from both the US and Canada. Over the four days, the lab was in the format of an overview or &ldquo;lecture&rdquo; followed by a longer hands-on lab to put what participants were learning to use. During the labs, our engineers would walk around to check on how everyone was doing, answer questions, and help troubleshoot.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2017-03-24-engineers-on-the-road-for-smart-and-fhir/smart-and-fhir-02.jpg"></p>

<p>We received a lot of comments and excitement from everyone participating around the fact that our engineering team itself was involved in the learning lab. It was also very exciting for the engineering team to watch everyone try out our newer developer tools, read our documentation, and then put it to practice. Not only were the participants able to actually see the data coming back, but there were a lot of conversations that occurred that will help us improve the class (and our tooling) in the future to make the event better. In the end, it was a great experience for everyone involved!</p>

<p>If you’re interested, you can check out some of the presentations and labs that were created for the learning lab here: <a href="https://github.com/cerner/ignite-learning-lab">https://github.com/cerner/ignite-learning-lab</a> and our SMART tutorial: <a href="http://engineering.cerner.com/smart-on-fhir-tutorial/#prerequisites">http://engineering.cerner.com/smart-on-fhir-tutorial/#prerequisites</a></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[One Cerner Associate's Contributions in the Tech Industry]]></title>
    <link href="http://engineering.cerner.com/blog/micah-whitacre-recognition-blog/"/>
    <updated>2017-03-16T00:00:00-05:00</updated>
    <id>http://engineering.cerner.com/blog/micah-whitacre-recognition-blog</id>
    <content type="html"><![CDATA[<h3>Background:</h3>

<p>Micah is currently a software architect in Population Health Development in Healthe Intent Development at Cerner. In 2013, Micah Whitacre received committer status on the <a href="https://crunch.apache.org/">Apache Crunch</a> project, and is now a Project Management Committee (PMC) member of the project.</p>

<h3>How has Cerner been been involved in Apache Crunch?</h3>

<p>In 2012 and 2013, Cerner started using this project within several solutions. We also wanted to give back to the community and decided to invest time in helping answer questions and concerns, and overall project needs. We wanted to get involved in small community like <a href="https://crunch.apache.org/">Apache Crunch</a> since it brought great value to our Cerner solutions.</p>

<h3>What does it mean to be receive committer status on Open Source?</h3>

<p>In 2013, I was selected as a committer on <a href="https://crunch.apache.org/">Apache Crunch</a>. In reference to the technical aspect, Apache projects have some form of criteria for when to accept a committer to a project. Election to being a committer is the result of consistent and high quality contributions to the project, through documentation, code, or helping to build the overall community.</p>

<p>Being selected as a committer was an unexpected honor. It&rsquo;s also an additional responsibility for my involvement with the project. I closely watch for questions and make enhancements and functionality code changes. There are fourteen other committers on the project as well.</p>

<h3>What does it mean to be an Apache Crunch PMC?</h3>

<p>After being a committer for over a year, I was selected as a PMC of <a href="https://crunch.apache.org/">Apache Crunch</a>. The goal of a PMC is to set the pace and the direction of the community, as well as facilitate how the community is interacting. PMCs help drive releases and help grow the committee. Some of my other responsibilities also include participating in discussions about voting people into the community, new releases of code, and which functionalities need to get fixed.</p>

<h3>What are your favorite parts about being a committer?</h3>

<p>One of the coolest parts about being a committer is making new connections with larger tech community. It forces you to build new relationships and talk to people you may have not spoken to before. It&rsquo;s cool being able to meet new people who don&rsquo;t work at the company you&rsquo;re working for. I also enjoy being able to look at the project as a whole and figure out the right way to solve something, rather than solving it right now.</p>

<p>I&rsquo;ve also spoken I&rsquo;ve spoken at ApacheCon twice on <a href="https://crunch.apache.org/">Apache Crunch</a>, which have been awesome opportunities.</p>

<h3>How else are you involved in the tech industry?</h3>

<p>I also like to be involved in our development culture at Cerner, helping improve our internal community and increasing awareness of Cerner in the industry.  I&rsquo;ve been a member of the planning committee and have spoken at our internal conference, DevCon, given internal tech talks at Cerner, and have also spoken at industry conferences like Kafka Summit 2016 and Midwest.io 2014. I&rsquo;ve also written an Engineering Health blog (Scaling People with <a href="https://crunch.apache.org/">Apache Crunch</a>). I am an open source reviewer for Cerner projects, helping review and provide feedback on Cerner open source projects prior to them being published.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Automated Deployment With Apache Kafka]]></title>
    <link href="http://engineering.cerner.com/blog/automated-deployment-with-apache-kafka/"/>
    <updated>2017-02-16T00:00:00-06:00</updated>
    <id>http://engineering.cerner.com/blog/automated-deployment-with-apache-kafka</id>
    <content type="html"><![CDATA[<p>It&rsquo;s likely not a surprise that Cerner would use <a href="http://kafka.apache.org/">Apache Kafka</a>
as we have used a number of related technologies like <a href="http://engineering.cerner.com/2013/02/composable-mapreduce-with-hadoop-and-crunch/">Apache Hadoop</a>
along with its Map/Reduce, HDFS and even <a href="http://engineering.cerner.com/2013/02/near-real-time-processing-over-hadoop-and-hbase/">Apache HBase</a>.
Our team first started using Apache Kafka in 2014
when Kafka 0.8 first came out. Since then we&rsquo;ve expanded to using Kafka for a
number of different use cases
(<a href="http://blog.cloudera.com/blog/2014/11/how-cerner-uses-cdh-with-apache-kafka/">1</a>, <a href="https://www.confluent.io/kafka-summit-2016-users-ingesting-complex-healthcare-data-with-apache-kafka">2</a>)
and it has become a core piece of Cerner&rsquo;s infrastructure.</p>

<p>Just like the <a href="http://engineering.cerner.com/blog/deploying-web-services-with-apache-tomcat-and-chef/">applications we create</a>,
we also needed to automate the deployment of Kafka to handle the ever growing
amount of operations work and to ensure a high level of consistency with our
environments. <a href="https://www.chef.io/chef/">Chef</a> was an obvious choice for us since
its been our deployment tool of choice for the last few years. We quickly put together a
<a href="https://github.com/cerner/cerner_kafka">cookbook</a> to help us automate the
deployment of the many Kafka clusters here at Cerner. We have
<a href="https://github.com/cerner/cerner_kafka/blob/master/CHANGELOG.md">continued to support and use this cookbook</a>
after open-sourcing it upgrading it to handle newer versions of Kafka (0.9, 0.10),
handle Kerberos authentication, and <a href="https://github.com/cerner/cerner_kafka/issues?q=is%3Aissue%20is%3Aclosed%20">many other improvements</a>.</p>

<p>If you use Apache Kafka, feel free to <a href="https://github.com/cerner/cerner_kafka">try it out</a>
and let us know how it works for you.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Beadledom - Simple Java Framework for Building REST APIs]]></title>
    <link href="http://engineering.cerner.com/blog/beadledom-simple-java-framework-for-building-rest-apis/"/>
    <updated>2017-02-15T00:00:00-06:00</updated>
    <id>http://engineering.cerner.com/blog/beadledom-simple-java-framework-for-building-rest-apis</id>
    <content type="html"><![CDATA[<p>Beadledom is a framework for building Java based web services. It bundles several open sourced components required for building JAX-RS services.</p>

<h3>Why?</h3>

<p>HealtheIntent, Cerner’s population health management platform, started 3.5 years ago. We went through the process of investigating different technologies for the platform. We decided on using Java for building  services and arrived on a set of libraries that we believed work well together.</p>

<p>The long history of Java has led to an ocean of libraries that are available to Java developers. Picking up the right set of tools is very important to get the job done the right way. The trickiest part is to know what  “the right set of tools” are. This is only the first of many tough decisions that a developer makes during the lifecycle of a project.</p>

<p>If choosing the libraries for a project is a game, then getting them all to play well together is a whole new game. Beadledom not only brings all the awesome java libraries together but also binds them well, making the developer&rsquo;s life easy. It gives a good head start to the developer by letting them start from a strong foundation.</p>

<p>At Cerner we understand how important and tedious these decisions are and we always want to contribute back to the community. So we open sourced Beadledom because it  helped us develop our platform quickly and consistently. We hope that others in the community can also leverage its power.</p>

<h3>What?</h3>

<p>Beadledom uses several open source projects that are widely used and well maintained. Below is a list of a few of the major components that Beadledom uses.</p>

<p>Guice for  gluing  and bootstrapping components;
Jackson for JSON serialization/deserialization;
Resteasy for JAX-RS implementation;
Stagemonitor for Monitoring and metrics;
Swagger for API Documentation;
Apache Commons-Configuration for handling different types of configurations consistently.
Where?
You can find the source code on GitHub.  Please find our documentation at <a href="http://engineering.cerner.com/beadledom/.">http://engineering.cerner.com/beadledom/.</a></p>

<h3>Who?</h3>

<p>Below are the core developers of Beadledom:
John Leacox
Sundeep Paruvu
Nimesh Subramanian
Brian van de Boogaard
Supriya Lal
Here is the complete list of contributors who made Beadledom awesome.</p>

<h3>Cookie Cake Time!</h3>

<p>Cerner takes pride in our open source contributions. When teams contribute to the community Cerner rewards them with a cake. Here is the cookie cake for Beadledom.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2017-02-15-beadledom-simple-java-framework-for-building-rest-apis/beadledom-01.png" title="Beadledom Cookie Cake" ></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[ShipIt VII Day: Winter 2016]]></title>
    <link href="http://engineering.cerner.com/blog/shipit-vii-day-winter-2016/"/>
    <updated>2017-01-18T00:00:00-06:00</updated>
    <id>http://engineering.cerner.com/blog/shipit-vii-day-winter-2016</id>
    <content type="html"><![CDATA[<p>Cerner’s 7th ShipIt Day took place on December 8th and 9th. ShipIt is a 24-hour hackathon with the
goal of building something awesome, usable, and value-adding within a single day.  The event was
hosted at our DevCenter in the Realization Campus, a large open space that hosts our DevAcademy (to
learn more about our DevAcademy, check out this
<a href="http://engineering.cerner.com/2013/08/devacademy/">earlier post</a>).  We had 18 teams made up of
associates from different Cerner organizations. The DevCenter was stocked with all kinds of snacks,
which of course included a traditional drink: Monster Energy. Participants worked hard all day long
and when dinner came around they were ready to dig in. Good thing we ordered 8 twenty-six inch
pizzas and 40 breadsticks from a local Kansas City favorite, <a href="http://www.pizza51.com/">Pizza 51</a>.
Look how big these pizzas were!</p>

<div align="center">
<table>
  <tr>
    <td><img src="http://engineering.cerner.com/assets/2017-01-18-shipit-vii-day-winter-2016/shipit-01.jpg"></td>
    <td><img src="http://engineering.cerner.com/assets/2017-01-18-shipit-vii-day-winter-2016/shipit-03.jpg"></td>
  </tr>
</table>
</div>


<p>It was predicted we hit a record in Kansas City for the most pizza in an elevator at this time, but
we were not able to get an official count!</p>

<div style="float:left;" >
<img src="http://engineering.cerner.com/assets/2017-01-18-shipit-vii-day-winter-2016/shipit-04.png">
</div>


<p>When getting pizzas of this size, there was a significant amount to fuel the innovation that was
happening. To break it down:</p>

<p>A = πr<sup>2</sup> = π x 132 ≈ 530.93 square inches = 3.68 square feet x 8 = 29.44 square feet of pizza. Which
is larger than a twin bed size of solid pizza, and slightly smaller than a queen bed. A perfect
amount for a ShipIt event! Teams took a break from their work throughout the evening by playing
Nintendo and cheering on the <a href="http://www.chiefs.com/">Chiefs</a>!</p>

<div align="center">
<table>
  <tr>
    <td><img src="http://engineering.cerner.com/assets/2017-01-18-shipit-vii-day-winter-2016/shipit-00.jpg"></td>
    <td><img src="http://engineering.cerner.com/assets/2017-01-18-shipit-vii-day-winter-2016/shipit-07.jpg"></td>
  </tr>
</table>
</div>


<p>There were some impressive projects that came out of ShipIt Day VII. <em>Team GLADoS</em>, made up of Kyle
Harper, Sarah Harper, Snehit Gajjar, and Andy Quangvan took first place, and the coveted Golden
Keyboard, with their impressive idea.</p>

<blockquote><p>This ShipIt, we created a new <a href="https://developer.amazon.com/alexa">Alexa skill</a> called Scribe.
With our new skill we integrated the
<a href="https://www.amazon.com/All-New-Echo-Dot-2nd-Generation/dp/B01DFKC2SO/ref=cp_aucc_ods">Amazon Echo and Echo Dot</a>
smart speakers with <a href="https://code.cerner.com/">Cerner&rsquo;s FHIR implementation</a>. This integration allows
a patient or care provider to access clinical data by simply speaking to the Echo.</p></blockquote>

<p>Judge Sean Griffin was impressed. &ldquo;Really cool, awesome idea. Great innovation! The applicability
towards data entry could definitely be useful.&rdquo;</p>

<div align="center">
<img src="http://engineering.cerner.com/assets/2017-01-18-shipit-vii-day-winter-2016/shipit-08.jpg">
<sub>Team GLADos with the Golden Keyboard</sub>
</div>


<p>Second place went to <em>Team Trogdor</em> (Kyle Lipke, Derek Dobler, Nikki Justice, Mike Harrison,
and Nimesh Subramanain).</p>

<blockquote><p>Troubleshooting errors or misconfigurations in Millennium OAuth can be time consuming. Missing
information or misconfiguration can lead to hours of checking various sources of information. This
web page tool allows us to perform a quick and accurate diagnostic check for what pieces are
available and not available. Built with Ruby on <a href="http://www.sinatrarb.com/">Sinatra framework</a>.
Made to allow for easy additions to various checks that ETS (Emerging Technology Services group)
needs to do.</p></blockquote>

<p>Judge Jim Dwyer noted, &ldquo;This is a great tool to help drive down operational costs and reduce TCO.&rdquo;</p>

<div align="center">
<img src="http://engineering.cerner.com/assets/2017-01-18-shipit-vii-day-winter-2016/shipit-05.jpg">
<sub>Team Trogdor</sub>
</div>


<p>Third place winners, <em>Team 402 Cheeseballs Required</em> (Andy Nelson, Venkatesh Sridharan, Ian Kottman,
Nate Schile, and Anthony Ross) created an application for tracking running mileage for the
<a href="https://www.youtube.com/watch?v=w18fgEcMi7w">Healthe Fitness Center</a>.</p>

<blockquote><p>Our goal was to make the process of submitting running milestones for the Cerner Running Club
easier. The current process involved writing down how far you ran in a binder, and then one of the
gym staff checking that binder occasionally to see when you accomplished a milestone so they could
send a prize. We decided to make a web application that would integrate with Strava that would
simplify tracking how far you&rsquo;ve run. <a href="https://www.strava.com/">Strava</a> is a platform that
aggregates data from multiple fitness apps, such as <a href="https://www.fitbit.com/">Fitbit</a> and
<a href="https://buy.garmin.com/en-US/US/cIntoSports-cRunning-p1.html">Garmin</a>.
We used the <a href="https://strava.github.io/api/">Strava API</a> to pull a person&rsquo;s running data so
we could total up their mileage, regardless of what fitness tracker they used. In order to
integrate with Strava&rsquo;s public API we had to have an externally facing application. We decided to
use an <a href="https://aws.amazon.com/ec2/">Amazon EC2</a> instance to do this, since it was easy and cheap.
We used <a href="http://rubyonrails.org/">Rails</a> to create our web app since that was the web framework
we had the most experience in. First, we would ask a person to connect to their Strava account,
and then we pulled their running data and totaled up their mileage. If a person had reached a
milestone they could press a button to send an email to the gym staff that included their
running log, their total mileage, and what milestones they had achieved.</p></blockquote>

<div align="center">
<img class="center" src="http://engineering.cerner.com/assets/2017-01-18-shipit-vii-day-winter-2016/shipit-06.jpg">
<sub>Team 402 Cheeseballs Required</sub>
</div>


<p>By Friday at 10 a.m. the participants were ready to present (for the most part). Cerner associates
from different campuses watched the event via livestream and voted for People’s choice. Those
results are as followed&hellip;</p>

<p><strong>Favorite team name</strong>: <em>I Shipped My Pants</em> &ndash; Brandon Inman and Steven Goldberg</p>

<blockquote><p>Improvements made to the existing
<a href="http://techblog.netflix.com/2016/08/vizceral-open-source.html">Vizceral-based implementation</a>
of HealtheIntent Intuition
Engineering.  Improvements made by the team included features to make it easier to switch between
streaming and batch processing, more timely updates of data, and several cosmetic features to
improve user experience and differentiate the tool from other Vizceral apps.  Voice activation was
added as a &ldquo;fun&rdquo; feature.</p></blockquote>

<p><strong>Favorite Presentation</strong>: <em>Guys on FHIR</em> &ndash; Bhaumik Aniruddha, Bhagat Parthiv, Vetri Selvi Vairamutha,
Neil Pfeiffer, Sai Praveen Gampa</p>

<blockquote><p>Our ShipIt project was a NICU SMART App, intended for the NICU unit. The app will be used to get a
live video of the baby inside an incubator alongside the vitals of the baby. This will prevent
anyone from disturbing the sleeping pattern of the newborn. The live feedback can also be viewed
by the parents which will ease the mental tension. The future scope of this project is to hook up
with an app, which can read the facial expression of the baby.</p></blockquote>

<p><strong>Best Project</strong>: <em>RSA (Readmission Security Admission)</em> &ndash; Kristopher Williams, Karthik Nimmagadda, Sai
Inampudi</p>

<blockquote><p>Our project was to create an intuitive dashboard that lets the user visualize the network traffic
for readmission solution&rsquo;s services for a given time period (hour/day/week/month currently). This
dashboard will allow the user to easily and quickly identify abnormal behavior (e.g. service goes
down, or service experiencing a lot of errors) and is designed in such a way as to return results
extremely fast for longer time spans, compared to ad-hoc splunk queries.</p></blockquote>

<div align="center">
<img src="http://engineering.cerner.com/assets/2017-01-18-shipit-vii-day-winter-2016/shipit-02.jpg">
<sub>People’s Choice Winners</sub>
</div>


<p>ShipIt Day continues to be a great way for Cerner associates to work on projects they don’t normally
have time for, meet people in different organizations and learn something new. Participant
Kristopher Williams said:</p>

<blockquote><p>My favorite part of ShipIt Day was being able to work on something entirely different, and in my
case, with a different group of people. Just a fresh change of pace.</p></blockquote>

<p>Thank you to all our participants and a special thanks to our judges Jenni Syed
(<a href="https://twitter.com/JenniSyed">@JenniSyed</a>), Yegor Hanov, Sean Griffin
(<a href="https://twitter.com/trenchguinea">@trenchguinea</a>), and Jim Dwyer
(<a href="https://twitter.com/episodicdata">@episodicdata</a>).</p>

<p>Check out some additional highlights of our event in this video:</p>

<iframe width="560" height="315" src="//www.youtube.com/embed/iqTp0dmLgUk" frameborder="0" allowfullscreen></iframe>


<p>If you are interested in reading more about previous ShipIt Day events, see these earlier posts:</p>

<ul>
<li><a href="http://engineering.cerner.com/blog/fall-2016-shipit-day/">Fall 2016 ShipIt Day</a></li>
<li><a href="http://engineering.cerner.com/blog/spring-2016-shipit-day/">Spring 2016 ShipIt Day</a></li>
<li><a href="http://engineering.cerner.com/blog/shipit-hackathon-mplus/">ShipIt &ndash; 24-hour hackathon for Millennium+ Platform Dev</a></li>
</ul>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Alan and Grace: An Origin Story]]></title>
    <link href="http://engineering.cerner.com/blog/alan-and-grace-an-origin-story/"/>
    <updated>2016-11-23T00:00:00-06:00</updated>
    <id>http://engineering.cerner.com/blog/alan-and-grace-an-origin-story</id>
    <content type="html"><![CDATA[<p><img class="center" src="http://engineering.cerner.com/assets/2016-11-23-alan-and-grace-an-origin-story/alan-grace.png" title="Alan and Grace" ></p>

<p>Meet Alan and Grace. These lighthearted, 8-bit characters were nominated in
early 2014 to become the iconic mascots of Engineering at Cerner. Have you ever
wondered who these 80’s video gaming inspired characters on our Engineering
t-shirts are? Why were they chosen as mascots? What do they represent? The
countless hours spent pondering these questions will be no more. Pull up a
chair as we dive into the mystery behind the dynamic duo known as Alan and
Grace.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-11-23-alan-and-grace-an-origin-story/shirt-laptop.png" title="Laptop and t-shirt" ></p>

<p>Believe it or not, the enjoyable pair highlighted on a myriad of our engineering
materials were named after two real life innovators: Alan Turing and Grace
Hopper. Alan and Grace were visionaries in their time, championing advances
that shaped the computer science innovations of today. These icons were chosen
based on their contributions to computer science and the moxie they showed while
doing it.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-11-23-alan-and-grace-an-origin-story/alan-8bit.png" title="Alan" ></p>

<p>Alan, our spiky haired friend who embodies an 80’s meets hipster vibe, is named
after the English mathematician Alan Mathison Turing (1912-1954). Turing is
known to some as the father of theoretical computer science and artificial
intelligence. Alan had an unquenchable mind and was in constant pursuit of
knowledge, devoting his life to a vocation in science and mathematics as a
computer scientist, mathematician, logician, cryptanalyst, and theoretical
biologist. Among his many contributions is the notable Alan Turing Enigma
machine. The machine was used to help break the enigma code used in German
naval communications during World War II. The concept of the Turing Enigma
machine has become the foundation of the modern theory of computation and
computability. Alan and his Enigma machine have received some recent fame as
featured in the Imitation Game, a 2014 American historical drama-thriller
loosely based on the biography Alan Turing: The Enigma. It can be argued that
computer science would not be what it is today without the mathematical
imagination of Alan Turing. Cue the mic drop now.</p>

<div align="center">
<img class="left" src="http://engineering.cerner.com/assets/2016-11-23-alan-and-grace-an-origin-story/alan-turing.jpg" title="Alan" >
<img class="right" src="http://engineering.cerner.com/assets/2016-11-23-alan-and-grace-an-origin-story/turing-machine.jpg" title="Turing Machine" >

<sub>Sources: <a href="http://www.alanturing.net/turing_archive/graphics/photos%20of%20Turing/pages/alan1_psd.htm">alanturing.net</a>, <a href="http://worldnews.nbcnews.com/_news/2013/12/23/22025978-queen-pardons-computing-giant-alan-turing-59-years-after-his-suicide">nbcnews.com</a></sub>
</div>


<hr />

<p><img class="center" src="http://engineering.cerner.com/assets/2016-11-23-alan-and-grace-an-origin-story/grace-8bit.png" title="Grace" ></p>

<p>Cerner Engineering’s jovial female mascot appears as if she’s ready to take on
the world. The fearless spirit of the mascot encompases the nature of its muse
for whom she was named, Grace Brewster Murray Hopper (1906-1992). Grace Hopper
was an American mathematician, computer programmer, military leader, and a woman
clearly ahead of her time. Grace received a Ph.D. in mathematics, one of the
first women to earn this level of education. Grace’s path to computer science
began while serving in the U.S. Naval Reserve as one of the first to program the
Harvard Mark I computer. Grace went on to create the first compiler for
computer languages leading to COBOL, a widely adapted language that would be
used around the world. But Grace wasn’t finished yet. At the age of 60, she
was recalled to active duty where she standardized communication between
different computer languages. When Grace retired at the age of 79 as a rear
admiral, she was the oldest serving officer in the service. In 1991, Grace was
the first women awarded the National Medal of Technology. Throughout her life,
Grace encouraged young people to learn how to program. Grace’s legacy lives on
through the Grace Murray Hopper Award, the Grace Hopper Celebration of Women in
Computing Conference, the University of Missouri’s computer museum “Grace’s
Place”, and now Cerner’s Engineering mascot. Grace continues to serve as an
inspiration to women in tech everywhere. Was there any other choice?</p>

<div align="center">
<img class="left" src="http://engineering.cerner.com/assets/2016-11-23-alan-and-grace-an-origin-story/grace-hopper.jpg" title="Grace Hopper" >
<img class="right" src="http://engineering.cerner.com/assets/2016-11-23-alan-and-grace-an-origin-story/grace-hopper-2.jpg" title="Grace Hopper" >

<sub>Sources: <a href="http://www.womenatworkmuseum.org/envision-engineering.html">womenatworkmuseum.org</a>, <a href="https://en.wikipedia.org/wiki/Grace_Hopper">wikipedia.org</a></sub>
</div>


<p>Alan and Grace represent a beacon of inspiration, providing icons that
encapsulate and promote a culture of innovation, forward thinking, and just
plain fun. The creation of mascots Alan and Grace has been a big win for the
passionate Cerner engineers dedicated to fostering Cerner’s Engineering culture.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-11-23-alan-and-grace-an-origin-story/alan-grace-2.png" title="Alan and Grace" ></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Clara Rules Joins Cerner's Open Source]]></title>
    <link href="http://engineering.cerner.com/blog/clara-rules-joins-cerner-open-source/"/>
    <updated>2016-11-11T00:00:00-06:00</updated>
    <id>http://engineering.cerner.com/blog/clara-rules-joins-cerner-open-source</id>
    <content type="html"><![CDATA[<p>Sometimes a small experiment on the side can grow into something valuable. We at Cerner have long
used forward-chaining rules engines to express and execute clinical knowledge, but we’ve also had to
extend or work around the capabilities of such engines. Engines targeting business users just
weren’t expressive enough to model some of our logic. To meet this need we are making Clara Rules an
open source project driven by Cerner Engineering.</p>

<p>Clara took an unusual path, starting as a minimal implementation of the
<a href="https://en.wikipedia.org/wiki/Rete_algorithm">Rete algorithm</a> to help my
own understanding. After sharing this with others at Cerner, we started to see an opportunity to
gain the modularity and reasoning advantages of a rules engine, while preserving the expressiveness
and power of a modern programming language. We wanted to combine the best ideas of rules engines
with good software engineering practices. You can see the result of this on the
<a href="http://www.clara-rules.org">Clara Rules home page</a>.</p>

<p><a href="https://github.com/mrrodriguez">Mike Rodriguez</a> has been essential in turning Clara from an
experiment into a valuable system to fill a gap for rules engines which are oriented towards
developers. More recently, <a href="https://github.com/WilliamParker">Will Parker</a>
has also made significant contributions to help make Clara fast and reliable. They made it possible for Clara
to help analyze millions of medical records every day. Therefore Mike and Will are joining as the
initial committers to Cerner Engineering’s Clara project. We will welcome other contributors to make
this step in the future.</p>

<p>You can find project <a href="https://github.com/cerner/clara-rules">source code on GitHub</a>, and the
documentation on the <a href="http://www.clara-rules.org">clara-rules.org</a> site.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Fall 2016 ShipIt Day]]></title>
    <link href="http://engineering.cerner.com/blog/fall-2016-shipit-day/"/>
    <updated>2016-11-08T00:00:00-06:00</updated>
    <id>http://engineering.cerner.com/blog/fall-2016-shipit-day</id>
    <content type="html"><![CDATA[<p>Another ShipIt Day was held on September 8th-9th at the Realization Campus at Cerner. ShipIt is a
24-hour hackathon with the goal of building something awesome, usable, and value-adding within a
single day. This fall marked our sixth ShipIt, which has has grown significantly since our first
event.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-11-08-fall-2016-shipit-day/shipit-sign.jpg" title="Announcement" ></p>

<p>The event kicked off in the morning at the auditorium of our Realization campus. At this point,
teams were already formed, instructions were given, and shirts were handed out. This ShipIt day came
with cool new t-shirts for all participants featuring the ShipIt squirrel (<a href="https://www.quora.com/GitHub-What-is-the-significance-of-the-Ship-It-squirrel">inspired by GitHub</a>)!</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-11-08-fall-2016-shipit-day/shipit-squirrel-shirt.jpg" title="ShipIt shirt" ></p>

<p>Teams broke out to their work spaces and got to work on their projects. One team, <strong>LearnStack</strong>,
decided to focus their project on associate development. The judges greatly appreciated their
approach to this very real problem, and decided to award them the third place prize.</p>

<p><strong>LearnStack</strong> &ndash; Jake LaCombe, Daniel Stucky, Sujay Sudheendra</p>

<blockquote><p>Developers can get to a struggling point when they try to increase their experience level. Sure,
getting started with any language can be very easy to do. Rails for example has their
<a href="https://www.railstutorial.org/">impressive tutorial from Michael Hartl</a> for getting started
with the framework. However, what happens when you
get done with those beginning steps? What does it take to go from a beginner to an intermediate
Rails developer, and then again from Intermediate to Advanced?</p>

<p>The answer here is LearnStack. LearnStack takes a list of resources that are recommended by other
developers, and rated appropriately based on the experience level of the material. Developers can
easily share resources related to beginner, intermediate, and advanced knowledge of a particular
language, process, framework, or whatever it may be. No need to keep doing endless searches online
to figure out how to get to the next level! Let LearnStack point you to the resources you need!”</p></blockquote>

<p>Another team decided to tackle the time it takes to provision a virtual machine. This team called
themselves, &ldquo;All your base are belong to us,&rdquo; and they were awarded second place.</p>

<p><strong>All your base are belong to us</strong> &ndash; Chris Soukup, Alex Harder, Ben Hemphill</p>

<blockquote><p>The current situation for OS image creation is that it is done infrequently, in an ad-hoc manner,
with little consistency among the various hypervisors. This means that we take the base image and
then use <a href="https://www.chef.io/">Chef</a> to upgrade the image to the desired state. This adds significant
amounts of time to the process of provisioning a virtual machine.</p>

<p>Our project was to use <a href="https://www.packer.io/">Packer</a> to create a generic image with all of the
universal settings that we set with our base OS Chef recipe. This image can then be distributed to all
the hypervisors and made available to consumers. Since the image would be generated regularly, the
provisioning time could be significantly decreased by not having to immediately upgrade the system. Additionally,
since the universal settings are already applied, we save considerable amounts of time on the OS preparation
as well. We took provisioning time for a RedHat 7.2 <a href="https://www.openstack.org/">OpenStack</a> VM from 12 minutes,
50 seconds to 4 minutes, 1 second, a 320% decrease in time!</p>

<p>Future enhancements will allow us to automatically generate these images, distribute them, and make
them available to consumers. This will allow a consuming team to naturally stay current on OS
patches at provisioning time, decrease the human effort in creating these OS images, and provide a
consistent experience for teams across Vagrant, VMWare, OpenStack, EC2, Docker, etc.”</p></blockquote>

<p>For the first time we had a unanimous winner for both the judges and the participant winners. The
<strong>206 Partial Content</strong> team not only won two giant tubs of cheese balls, but also the all important
<a href="http://engineering.cerner.com/blog/shipit-hackathon-mplus/">Golden Keyboard</a> and bragging rights
until the next ShipIt Day. <strong>206 Partial Content</strong> decided to solve a problem to &ldquo;enable solution designers,
engineers, and grandparents to easily create, stage, and process data for testing scenarios required for
verification without being dependent on upstream processing or rules.&rdquo;</p>

<p><strong>206 Partial Content</strong> &ndash; Madhur Sikri, Ian Kottman, Venkatesh Sridharan, Andy Nelson, Brian van de
Boogard</p>

<blockquote><p>Our project was primarily driven by two rock star engineers on our team, Ian Kottman and Brian van
de Boogaard. We coined our project Sinecure, which means a position requiring little or no work but
giving the holder status or financial benefit. The goal of the project was to find a solution to
creating test data more easily and efficiently for anyone on our team to consume.”</p></blockquote>

<p>Thanks to all who participated and congratulations to our winners. Until next time, at ShipIt VII in
December!</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-11-08-fall-2016-shipit-day/shipit-winners.jpg" title="Winners" ></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Building a Unified UI Component Library: Lessons Learned]]></title>
    <link href="http://engineering.cerner.com/blog/building-a-unified-ui-component-library/"/>
    <updated>2016-10-13T00:00:00-05:00</updated>
    <id>http://engineering.cerner.com/blog/building-a-unified-ui-component-library</id>
    <content type="html"><![CDATA[<p>Cerner is building an open source, mobile-first user interface component library for healthcare applications. Pulling years of experience and lessons learned across multiple teams together, we are creating something great. We are very excited to bring this solution to the public and to be contributing back to the community!</p>

<p>We didn’t simply decide to create this library, we started with segmented UI libraries focused on different aspects of the company which had smaller isolated needs. Mistakes were made along the way and we have learned much from them. Let’s take a look at where we started and how we got to where we are now.</p>

<h2>History</h2>

<p>In 2013, a group of engineers with strong front-end skills was put together with the task of creating a reusable component library for Cerner’s HealtheIntent platform. Twitter Bootstrap was used initially, but quickly outgrown as the needs and designs of UX expanded. The project this team built is called Blue Steel. It featured everything from simple typography to interactive data visualizations while adhering to UX guidelines and requirements.</p>

<p>Blue Steel is a Rails gem that provides some basic helpers, CSS, JavaScript, and site templates to simplify layout and correct HTML usage. Blue Steel abstracts complex components to keep things simple and because we didn’t feel it was necessary to abstract native HTML elements.</p>

<p>In its infancy, the project was well received; other platforms saw the value and wanted to take advantage of the work being done. To accommodate the additional platforms, Blue Steel was merged with another internal UI library called Style Guide. The best features of each library were pulled together, sometimes one overriding the other, sometimes merging feature sets. The merged project was called Terra.</p>

<h2>Problems</h2>

<p>The approach of merging existing components saved time up-front, but came with a steep cost. The library was fragmented in the approaches taken during design and development. As such, it became difficult to work on without pre-existing familiarity. It was fairly obvious that we put two frameworks together on a time constraint.</p>

<p>Blue Steel still exists but is comprised primarily of Terra components. It persists to provide HealtheIntent-specific styles and functionality, while also being a Rails wrapper around Terra. Terra has been historically kept below version 1.0.0 for rapid development, while Blue Steel has been above 1.0.0. This strategy has caused a lot of pain for the consumers of both Terra and Blue Steel.</p>

<p>Keeping Terra below version 1.0.0 allowed breaking changes to occur without cutting a major release. Although this made development easier in some aspects, it damaged consumer trust. We always attempted to communicate breaking changes, but it didn’t always happen and it wasn’t always clear. Any time a team had to upgrade, they had to be prepared to fix their application which caused them not to trust us.</p>

<p>The issue was even worse in Blue Steel since it had to accommodate for Terra’s breaking changes. Blue Steel would consume a version of Terra with breaking changes and would still release as a minor update by providing styles and hooks to work around the breaking changes. Deprecation schedules and documentation were written to help keep Blue Steel backwards compatible. Unfortunately, it often wasn’t; sometimes, breaking changes would only manifest in an application.</p>

<p>Blue Steel and Terra were both tested separately with their own documentation sites. These tests were extensive and thorough but could not accommodate the complexity of the various applications consuming them. Breaking changes would creep into applications even when we thought everything was backwards compatible in Blue Steel.</p>

<p>In Terra, the preference was to style on HTML elements, states, and ARIA-roles whenever possible since they carried far more meaning than CSS classes. Unfortunately, this form of styling is somewhat global in nature. It was easy for style collisions to occur between components within Terra, consumer styles, and 3rd party libraries.</p>

<p>Terra was developed in such a way that it discouraged consumers from building custom components. The thinking was that applications would write little to no custom CSS or JavaScript. This didn’t scale well as needs evolved and application developers had to move ahead of the UI library. Terra components had a very high CSS specificity and were difficult to override which forced consumers to write even more complex CSS to override it. In turn, many bugs were introduced to consumer applications.</p>

<p>Terra was also built as a monolith with very little modularity in place. Consumers had the option of taking it all, figuring out how to make a custom build, or not using it. This caused applications which only needed a subset of functionality to become bloated.</p>

<h2>Solutions</h2>

<p>Today, Terra is in the process of being open sourced. We’ve looked extensively at the issues above and are taking measures to address all of them:</p>

<ul>
<li>Each component is in its own repository.</li>
<li>The SUIT CSS convention is being used for all Terra components.</li>
<li>Base componentry is being developed first.</li>
<li>Components are being kept minimal and composable.</li>
<li>Each component is being built to be a minimum viable product and released as 1.0.0 to follow <a href="http://semver.org/">SemVer</a> up front.</li>
<li>Only as components become complex and specific do we introduce framework opinionation. Base components will be framework agnostic.</li>
<li>Everything will have helpers to abstract underlying HTML structures.</li>
</ul>


<p>SUIT CSS bans the use of styling hooks which are considered global and effectively breaks the cascade in a global sense (you can leverage it within a component). At the root node of each component, only classes can be used. As a result, consumers can rest easy knowing that styles will not leak out of components and they have low specificity which allows them to be overridden as needed.</p>

<p>By starting with base components (buttons, images, etc), we effectively create a set of building blocks to build more complex web components. By keeping them small, composable, and framework agnostic, they can be used anywhere with confidence. Following <a href="http://semver.org/">SemVer</a> from the start will boost consumer confidence when using our components.</p>

<p>Introducing opinion into more complex components enables us to build better and more maintainable components with our most common use cases in mind. This does not preclude consumers who do not wish to consume the frameworks and libraries we use. By keeping each component in its own repository, it’s possible to create alternative versions of componentry to meet application needs. Additionally, keeping each component in its own repo allows for individual versioning, limiting scope of change, and makes it very easy for consumers to omit what they don’t need.</p>

<p>Finally, providing helpers for all components will enable easy and consistent development. It will be possible to build complex components solely out of the smaller components leveraging helpers to output the correct HTML. The abstraction also makes it possible to update the implementation in a compatible way while making it so developers don’t have to be aware of complexities like semantic HTML and accessibility. This will enable developers to build solid applications easily and quickly.</p>

<h2>Lessons Learned</h2>

<ul>
<li>When creating a reusable library, avoid global state or styles.</li>
<li>Create an abstraction or facade to the underlying implementation regardless of simplicity.</li>
<li>Keep your library modular.</li>
<li>Start with <a href="http://semver.org/">SemVer</a> and stick to it.</li>
<li>Start simple then reuse and compose.</li>
<li>Keep your library cohesive by developing consistently across components.</li>
<li>Don’t build with any particular framework(s) in mind when you can; the web is constantly evolving.</li>
</ul>


<p>We’ve learned a lot and are excited to open source the results of our work and learning. Keep an eye on <a href="https://github.com/cerner/">github.com/cerner</a> and <a href="http://engineering.cerner.com/">engineering.cerner.com</a> while we build out this new library. It’s going to be great!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Michelle Brush Receives the Rising Trendsetter STEMMY Award!]]></title>
    <link href="http://engineering.cerner.com/blog/michelle-brush-receives-the-rising-trendsetter-stemmy-award/"/>
    <updated>2016-10-02T00:00:00-05:00</updated>
    <id>http://engineering.cerner.com/blog/michelle-brush-receives-the-rising-trendsetter-stemmy-award</id>
    <content type="html"><![CDATA[<p>We are excited to congratulate our own Michelle Brush for receiving the Rising Trendsetter STEMMY award. The Rising Trendsetter Award is given to a woman in STEMM areas with less than 20 years of experience and has demonstrated significant achievements early in their career.</p>

<p>Michelle is a director at Cerner who manages an engineering team in population health as well as a team dedicated to improving Cerner’s development culture. One of Michelle’s most notable career accomplishments was her leadership role in reconstructing Cerner’s software engineer development program in 2013, which she personally oversaw until the end of 2014 (read about the details of this program in her <a href="http://engineering.cerner.com/2013/08/devacademy/">blog</a>). In 2014, she transitioned into an executive role in the Population Health platform, where she fields particularly tricky support issues to help resolve data inconsistencies. Since 2014, Michelle has also managed the Development Culture team at Cerner, a team that focuses on improving associate morale through cultural empowerment, encouraging and providing opportunities for collaboration and innovation, and growing Cerner’s presence in the technical community.</p>

<p>Michelle actively promotes women entering and growing their STEMM careers through her leadership in the community and at Cerner. She is an integral member of Cerner’s women in tech committee, which has the goal to provide development, networking and leadership opportunities to women in technical roles in a supportive and inclusive environment that promotes authenticity and values differences. Additionally, she is a member of several other leadership councils ranging from helping design the new software engineer campus, improving development culture across the company, and intern planning.</p>

<p>In the community, Michelle is also a local chapter leader of <a href="https://www.girldevelopit.com/chapters/kansas-city">Girl Develop It</a>, a nonprofit organization that provides affordable programs for adult women interested in learning web and software development. Michelle is also an organizer of <a href="http://www.midwest.io/">Midwest.io</a>, a two-day industry tech conference in Kansas City that brings together developers across the Midwest for an eclectic collection of talks covering the latest trends, best practices, and research in the field of computing. As a member of the organizing team, she defines the vision for the conference, which includes finding speakers, promoting the conference in the industry, and ensuring diversity is top of mind at the conference.</p>

<p>Michelle also actively speaks at tech conferences, and has spoken at conferences like OSCON, CityCode, O’Reilly’s Software Architecture Conference, and Strange Loop. Michelle is also a MSU Computer Science Board Member.</p>

<p>Congratulations, Michelle!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[DevCon 2016 Word Cloud]]></title>
    <link href="http://engineering.cerner.com/blog/devcon-word-cloud/"/>
    <updated>2016-09-28T00:00:00-05:00</updated>
    <id>http://engineering.cerner.com/blog/devcon-word-cloud</id>
    <content type="html"><![CDATA[<p>Every year we hold an internal developers conference called <a href="http://engineering.cerner.com/2013/08/devcon/">DevCon</a>. This year
we had 295 submissions for talks, ranging from a deep technical dive into the inner workings of Kafka to a reflection on the power of office pranks.‌ I wondered what, if anything, do these submissions have in common? Are there any common themes/topics being discussed? To find out I decided to visualize the talk submissions in a word cloud to create an easily understandable (and hopefully aesthetically pleasing) view of what topics were most common.</p>

<p>The first step was some basic data cleaning so I could get a good set of words to visualize. I used Python to read in the submissions into a single string, lowercase all words, and then remove all common contractions.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="c"># read all submissions into a single string</span>
</span><span class='line'><span class="n">text</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s">&#39;./submissions.txt&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="c"># lower case</span>
</span><span class='line'><span class="n">text</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="c"># remove all contractions to prevent situations like &quot;we&#39;ll&quot; becoming &quot;well&quot; once punctuation is removed</span>
</span><span class='line'><span class="c"># normally a regex tokenizer could be used for this but that assumes single quote is only ever used</span>
</span><span class='line'><span class="c"># in contractions</span>
</span><span class='line'><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">&quot;./resources/contractions.txt&quot;</span><span class="p">,</span> <span class="s">&quot;r&quot;</span><span class="p">)</span> <span class="k">as</span> <span class="nb">file</span><span class="p">:</span>
</span><span class='line'>    <span class="n">contractions</span> <span class="o">=</span> <span class="nb">file</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">splitlines</span><span class="p">()</span>
</span><span class='line'>    <span class="k">for</span> <span class="n">contraction</span> <span class="ow">in</span> <span class="n">contractions</span><span class="p">:</span>
</span><span class='line'>        <span class="n">text</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">contraction</span><span class="p">,</span> <span class="s">&quot;&quot;</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'>
</span><span class='line'><span class="n">Next</span> <span class="n">I</span> <span class="n">replaced</span> <span class="nb">all</span> <span class="n">punctuation</span> <span class="k">with</span> <span class="n">spaces</span><span class="o">.</span> <span class="n">I</span> <span class="n">did</span> <span class="ow">not</span> <span class="n">just</span> <span class="n">remove</span> <span class="n">the</span> <span class="n">punctuation</span> <span class="n">so</span> <span class="n">special</span> <span class="n">cases</span> <span class="n">such</span> <span class="k">as</span> <span class="n">hyphenated</span><span class="o">-</span><span class="n">words</span> <span class="ow">and</span> <span class="n">the</span> <span class="n">phrase</span> <span class="s">&quot;and/or&quot;</span> <span class="n">are</span> <span class="n">preserved</span><span class="o">.</span>
</span><span class='line'>
</span><span class='line'><span class="c"># replace all punctuation with spaces.</span>
</span><span class='line'><span class="k">for</span> <span class="n">char</span> <span class="ow">in</span> <span class="nb">set</span><span class="p">(</span><span class="n">string</span><span class="o">.</span><span class="n">punctuation</span><span class="p">):</span>
</span><span class='line'>    <span class="n">text</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">char</span><span class="p">,</span> <span class="s">&quot; &quot;</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>


<p>At this point I had 32,987 total words. A word cloud of this dataset was too noisy, predominated by common words like &ldquo;the&rdquo; and &ldquo;talk&rdquo;. In natural language processing these unwanted common words are called &ldquo;stop words&rdquo;. To remove them I made a list out of the submissions and then only kept words that were not in a list of common English words, supplemented by words specific to this data set, such as &ldquo;Cerner&rdquo; and &ldquo;presentation&rdquo;.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="c"># get list of all words</span>
</span><span class='line'><span class="n">submission_words</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="s">&quot;[^\w]&quot;</span><span class="p">,</span> <span class="s">&quot; &quot;</span><span class="p">,</span> <span class="n">text</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'><span class="c"># only keep words that are not in the stop words set</span>
</span><span class='line'><span class="n">stopwords</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s">&#39;./resources/stopwords.txt&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</span><span class='line'><span class="n">stopwords_set</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="n">stopwords</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">&quot;</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">))</span>
</span><span class='line'><span class="n">filtered_word_list</span> <span class="o">=</span> <span class="p">[</span><span class="n">w</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">submission_words</span> <span class="k">if</span> <span class="n">w</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">stopwords_set</span><span class="p">]</span>
</span></code></pre></td></tr></table></div></figure>


<p>This left me with a list of 14,291 words and 4,264 distinct words. When visualized I noticed many similar words were taking the highest spots, such as &ldquo;team&rdquo; and &ldquo;teams&rdquo; or &ldquo;technology&rdquo; and &ldquo;technologies&rdquo;. To remove some of this noise I turned to lemmatization. Lemmatization is the process of finding a canonical representation of a word, i.e. its lemma. For example, the lemma for the words &ldquo;runs&rdquo; and &ldquo;running&rdquo; is run. I used the popular Python library Natural Language Toolkit for lemmatization.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">lemmatizer</span> <span class="o">=</span> <span class="n">WordNetLemmatizer</span><span class="p">()</span>
</span><span class='line'><span class="n">reduced_list</span> <span class="o">=</span> <span class="p">[]</span>
</span><span class='line'><span class="c"># find the lemma of each word</span>
</span><span class='line'><span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">filtered_word_list</span><span class="p">:</span>
</span><span class='line'>  <span class="n">reduced_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">lemmatizer</span><span class="o">.</span><span class="n">lemmatize</span><span class="p">(</span><span class="n">word</span><span class="p">))</span>
</span></code></pre></td></tr></table></div></figure>


<p>This reduced the dataset to 3,834 distinct words. Now that the dataset was cleaned, common words filtered out, and similar words combined it was time to create the word cloud using a project called <a href="https://github.com/amueller/word_cloud">word_cloud</a>.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='python'><span class='line'><span class="n">cloud</span> <span class="o">=</span> <span class="n">WordCloud</span><span class="p">(</span><span class="n">relative_scaling</span><span class="o">=.</span><span class="mi">5</span><span class="p">,</span> <span class="n">height</span><span class="o">=</span><span class="mi">1024</span><span class="p">,</span> <span class="n">width</span><span class="o">=</span><span class="mi">760</span><span class="p">)</span>
</span><span class='line'><span class="n">cloud</span><span class="o">.</span><span class="n">generate</span><span class="p">(</span><span class="n">submissions</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="c"># write to file</span>
</span><span class='line'><span class="n">cloud</span><span class="o">.</span><span class="n">to_file</span><span class="p">(</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">__file__</span><span class="p">),</span> <span class="s">&quot;cloud.png&quot;</span><span class="p">))</span>
</span><span class='line'>
</span><span class='line'><span class="c"># show word cloud using matplotlib</span>
</span><span class='line'><span class="n">plt</span><span class="o">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">cloud</span><span class="p">)</span>
</span><span class='line'><span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">(</span><span class="s">&quot;off&quot;</span><span class="p">)</span>
</span><span class='line'><span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>


<p><img class="center" src="http://engineering.cerner.com/assets/2016-09-28-devcon-word-cloud/word_cloud.png" title="Word Cloud" ></p>

<p>Success! It is obvious what some of the most common themes are, such as &ldquo;data&rdquo; and &ldquo;team&rdquo;. Possible future steps for cleaning up the data would be grouping noun phrases, such as &ldquo;software engineer&rdquo; into single words or possibly removing common spelling mistakes using a spell checking library like <a href="https://pythonhosted.org/pyenchant/">PyEnchant</a>.</p>

<p>You can checkout the full code on <a href="https://github.com/ikottman/devcon-word-cloud">Github</a>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Difficult Debugging: Learning From Utter Nonsense]]></title>
    <link href="http://engineering.cerner.com/blog/advanced-debugging/"/>
    <updated>2016-07-13T00:00:00-05:00</updated>
    <id>http://engineering.cerner.com/blog/difficult-debugging</id>
    <content type="html"><![CDATA[<h2>Preface</h2>

<p>As software engineers we invest time into thinking about the problems we are trying to solve every day. Sometimes the problems we face are small or simple. Other times they are large and complex. Either way, we can usually draw from our experience to find a path forward.</p>

<p>While coding, we run into a lot of different bugs. Often, they are simple typos or a misuse of an API. These problems do not bog us down very much, and they are quick and easy to fix. Even complex bugs, while requiring investigation, often follow a familiar pattern(s) that help us identify a path to a solution.</p>

<p>Given this experience, it’s easy for us to think that we are good at debugging.</p>

<p>We are not. We are good at pattern recognition and recognizing similarities to problems we’ve faced in the past. Because we recognize the pattern, we can quickly debug most of the bugs we come across in our day-to-day work.</p>

<p>Which is great. But today, we will be exploring the types of bugs that do not come up in our day-to-day work.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-07-13-difficult-debugging/99-bugs-in-the-code.jpg" title="'Grumpy Cat meme: fixed one bug added 28'" ></p>

<p align="center">
  Image source:
  <a href="http://weknowmemes.com/2014/04/99-bugs-in-the-code/">weknowmemes.com</a>
</p>


<h2>How Bugs Can Mislead Investigative Efforts</h2>

<p>When a bug is brand new to us, we haven’t had the opportunity to derive a pattern from it yet because we have no experience to draw upon to create that pattern(s). Thus, we must investigate using a limited set of tools:</p>

<ul>
<li>Error messages</li>
<li>Stack traces</li>
<li>Guessing and check debugging</li>
<li>Colleagues</li>
<li>Google</li>
</ul>


<p>Often enough, that set of tools will guide you to a solution within a reasonable amount of time. Be warned, sometimes the solutions online solve a similar problem, but not your root problem. Testing is always in order with new bugs (or any bug more complex than a typo).</p>

<p>Debugging gets even trickier and more time consuming when we encounter a bug that resembles a pattern that we’ve seen before. Once, I came across a bug which looked like two other things, but turned out to be a third unexpected thing.</p>

<h3>Treemaps and IE8, and Fonts, Oh My</h3>

<p>One of our teams created an interactive DOM based treemap which worked in Internet Explorer 8 (IE8) and above. It wasn’t lightning fast in IE8, but it worked sufficiently well with a reasonably complex dataset. That is until the browser started crashing intermittently for some of our users.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-07-13-difficult-debugging/treemap.png" title="'Example treemap'" ></p>

<p align="center">Example treemap with dummy data</p>


<p>There were a few things that we (four engineers) noticed right away:</p>

<ul>
<li>It only crashed in IE8</li>
<li>The presence of a treemap seemed to be a driving factor</li>
<li>Crashing was intermittent</li>
</ul>


<p>Our initial investigation was good. We isolated the problem browser and determined that there was something wrong with the treemap, which was causing the issues, we were on our way to solving this problem &ndash; or so we thought.</p>

<p>Given the intermittent nature of the problem, our experiences in the past, and the patterns we had observed, we assumed we had a memory leak, a race condition, or both. In an attempt to rule one of those options out, we tested with small and large datasets for the treemap.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-07-13-difficult-debugging/15k_treemap.png" title="'Treemap with 15,000 cells'" ></p>

<p align="center">Treemap with 15,000 cells in it rendered by Chrome</p>


<p>The above image serves as proof that modern browsers were unaffected by the bug we were experiencing.</p>

<p>Convinced that the treemap was the issue and that it had to be a memory leak or a race condition, we created automated tooling using sIEve, virtual machines, and AutoIt scripts. The automation enabled us to test dozens of scenarios hundreds of times easily to gain metrics which might help illuminate where, in the code, we should look next.</p>

<p>We chose different sized datasets thinking that smaller datasets would not trigger a memory leak while a large one would. Unfortunately, it crashed either way and we were no closer to figuring out what was wrong.</p>

<p>After several more days of investigation we were just as perplexed as the day we started. We had a plethora of information which told us nothing. At this point our capacity had to be reduced down to just one developer &ndash; me. Frustrated with the resultless weeks spent investigating the issue, I decided step back and take a completely different approach.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-07-13-difficult-debugging/500_treemap.png" title="'Treemap with 500 cells'" ></p>

<p align="center">IE8 managed to render a 500 cell treemap without crashing</p>


<p>To re-evaluate everything we did, I removed all of the assets from our application to minimize moving parts; regardless of how unlikely a variable was, I wanted to remove it. I ran our test scripts against the application to find that, unsurprisingly, the browser did not crash. I then added JS and CSS assets separately to find that neither alone caused the browser to crash. As a sanity check, I added both back to the application and it would reliably crash. This implied that there was some interplay between CSS and JS which was causing the browser to crash.</p>

<p>In our application, it was easier to add CSS file by file than it was for JS so I went that route. Along the way, I noticed that we were sending the wrong X-UA meta tag so I got sidetracked and fixed that to no avail. Eventually, I added back our Font Awesome CSS which caused the browser to crash. Thinking there was no way that a font could be the issue, I added and removed various pieces of our CSS to determine if it was the cause of the problem. I tried changing selectors, changing the load order, and everything else I could think of to no avail. After a while, frustrated, I commented out every line referencing Font Awesome and the browser stopped crashing. At this point, I just started adding code back line by line until the browser started crashing. What I found made no sense:</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>&:before {
</span><span class='line'>  font-family: 'FontAwesome';
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>I looked over this for a while and eventually noticed that we were using single quotes when we referenced the font while the font declaration</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>@font-face {
</span><span class='line'>  font-family: "FontAwesome";
</span><span class='line'>  src: url(@fontAwesomeEotPath);
</span><span class='line'>  font-weight: normal;
</span><span class='line'>  font-style: normal;
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>


<p>was using double quotes.</p>

<p>Having exhausted most other options already, I tried making the quotes match to see if the browser would crash and, to my surprise, it wouldn’t. It didn’t matter whether I used double or single quotes as long as they matched. I added back all of the remaining assets and tested this again; sure enough, the app did not crash even after 200 tests.</p>

<p><strong>Yes, it’s true. It wasn’t a race condition or a memory leak within the treemap, or anything else we thought it might be. It was mismatched quotes.</strong> I would never have guessed this in a million years.</p>

<p>I did not stop here! Although I solved the problem, I needed to understand what was going on. I was also very curious which part of the JS was causing issues with Font Awesome. I reintroduced the old CSS and started testing the JavaScript. The application broke as soon as I re-introduced Modernizr. The treemap was, at this point, seemingly faultless as I was able to reproduce the crashes without it.</p>

<p>I researched mismatched quotes and Modernizr online to try and get a better understanding of what was going on. I found several articles that detailed similar issues, but did not identify the root cause. Eventually I found a post on StackOverflow that enabled me to connect the dots.</p>

<p>Browsers do a lot for developers under the hood and IE8 treats different types of quotes differently and kicks off an error handling subroutine to smooth it over. If that error handling subroutine occurs while Modernizr is attempting to shim the browser for HTML5 compatibility, the browser will crash.</p>

<p>We weren’t wrong in that we were facing a race condition, but we were very wrong about our presumption about the treemap. The only interplay the treemap had was that it caused Modernizr to take longer to apply the shim thus widening the window of opportunity for the race condition to apply.</p>

<p>We worked on this from December 20th to January 9th, a total of twenty days!</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-07-13-difficult-debugging/sad_panda.png" title="'Meme: 20 days of debugging makes me a sad panda'" ></p>

<p align="center">
  Image Source:
  <a href="https://www.linkedin.com/in/jessicazehring">Jessica Zehring</a>
</p>


<h2>How To Tackle Perplexing Bugs</h2>

<p>Oftentimes, our ability to recognize coding patterns enables us to identify and solve bugs in a timely manner. As the example bug shows, however, following them can lead you down a rabbit hole. At face value, twenty days were spent working on that bug, but when you consider that there were up to four developers involved at a time, the lost time actually equates to one to three months. Once you factor in opportunity cost on top of that, you look at two to six months of time lost to debugging this issue.</p>

<p>That is a disturbingly significant amount of time to have lost on a bug. It was a difficult bug to solve, but it could have been solved faster had we recognized that our past experience was insufficient to solve this problem. There are some key points to take away from debugging something perplexing like the example bug we looked at:</p>

<ol>
<li>Isolate the problem. Find a single way that the bug can be consistently replicated and remove any variance that can trip you up.</li>
<li>Automate as much as you can. There are likely to be many tests; let the computer handle that work for you. While working on the example bug, we ran thousands of tests over those twenty days.</li>
<li>Remove as many variables as possible and keep a list of what’s remaining.</li>
<li>Change one variable at a time. If you change many things simultaneously, you won’t know what yielded which results.</li>
<li>Test, document, and repeat. When you change a single variable, record all of your findings. How did the results change? Eventually new patterns will present themselves.</li>
<li>Narrow the scope. As your testing reveals patterns, narrow down to a smaller area. In the example bug, we went from “somewhere in our static assets” to “something in our CSS and JS” to “something in this particular file” to “it’s this particular line”.</li>
<li>Repeat and keep repeating. Be meticulous in recording your findings. Even if you don’t see a pattern, someone else studying your results might.</li>
</ol>


<p>When you find that past experience is not yielding results in a timely manner (you’ll have to define what that means for yourself), take a step back and follow the steps above. Doing this allows your past experience to help you when it applies and gets it out of the way when it doesn’t. Don’t see how far the rabbit hole goes when you can scientifically figure it out.</p>

<p>Additionally, do not blindly trust answers found online. When working on the example bug, we found several examples of people correctly identifying a similar problems to ours, none of which correctly identified the underlying cause. Treat online resources as what they are &ndash; resources. Sometimes they have the answer you seek while other times they are the dots you must connect yourself.</p>

<h2>Tools and Techniques Which Will Help With Day-to-Day Developing and Debugging</h2>

<h3>Linters</h3>

<p>Static analysis, or linting, is a technique to catch common coding mistakes quickly. A linter will evaluate raw source code and give back a report detailing what needs to be fixed. Most languages have some form of linting software available for them. Since this post is front-end oriented, below are a few linters available for JS:</p>

<ul>
<li><a href="http://eslint.org/">ESLint</a></li>
<li><a href="http://jshint.com/">JSHint</a></li>
<li><a href="http://jscs.info/">JSCS</a></li>
</ul>


<p>All of these tools will notify you when you make typos, aren’t using variables, fail to follow a predefined set of coding styles, and more. They are very fast and are your first line of defense against day-to-day bugs. Linters will catch mistakes much faster than a human.</p>

<h4>Example</h4>

<p>I have a friend who is a very talented software engineer and mathematician. One day he called me to see if I could help him debug an issue he had been struggling with for about 19 hours. His Angular application kept spin locking (freezing up) and he couldn’t figure out why.  We scrolled through the code and luckily for him, a piece of it caught my eye:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class='js'><span class='line'><span class="k">for</span> <span class="p">(</span><span class="kd">var</span> <span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>  <span class="k">for</span> <span class="p">(</span><span class="kd">var</span> <span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o">&lt;</span> <span class="mi">5</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>    <span class="c1">// logic</span>
</span><span class='line'>  <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>


<p>He had been working on C++ code prior to this project which features block scoping while JavaScript (ES5) only features function scoping. A linter would have immediately caught this mistake and informed him of it, saving 19 hours of debugging.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-07-13-difficult-debugging/semi-colon.jpg" title="'Meme: Semi-colons are the hide and seek champion since 1958'" ></p>

<p align="center">
  Image source:
  <a href="https://www.reddit.com/r/ProgrammerHumor/comments/3dqj35/several_hours_of_debugging_later/">reddit.com</a>
  by Wilsam239
</p>


<h3>Automated Testing Frameworks</h3>

<p>While static analysis is your first line of defense, tests for your code are your second line of defense. Tests should depict the behavior of your codebase and should fail if you deviate from that contract. Just like linters, most languages support automated testing. There are several frameworks that you can leverage for testing JS, a few are:</p>

<ul>
<li><a href="https://mochajs.org/">Mocha</a></li>
<li><a href="http://jasmine.github.io/">Jasmine</a></li>
<li><a href="https://qunitjs.com/">QUnit</a></li>
</ul>


<p>Defining a set of behaviors for your code helps to protect you from unintentional changes and side effects. Tests will take longer to run than a linter, but are still frequently quicker at catching mistakes than a human. Since automated unit tests are like a contract for behavior, they can also help keep developers unfamiliar with the behavioral contract from making mistakes.</p>

<h3>Debugging Tools</h3>

<p>When automated approaches are insufficient, it’s time to start the manual debugging process. Every programming language has a debugging toolset that can be used to work through code with bugs. Since this post is front-end oriented, we’ll look at browser tools.</p>

<p>You could use JS to alert variables at different times, but this is radically inefficient. Today’s modern browsers have a plethora of tools available to you starting with the debugger keyword for JS code. When the browser’s console is open, the debugger statement acts as a breakpoint; it does nothing otherwise. Although simple, it’s very powerful as it gives easy control of adding/removing breakpoints to step through the code.</p>

<p>Atop the debugger keyword lay all of the browser tools. These enable you to step through code, inspect code, profile code runtimes, inject assertions, and even monitor the network traffic relating to the current page. Different browsers have different tools; below are some guides for debugging in different browsers:</p>

<ul>
<li><a href="https://developer.chrome.com/devtools">Chrome</a></li>
<li><a href="https://developer.apple.com/safari/tools/">Safari</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Tools">Firefox</a></li>
<li><a href="https://developer.microsoft.com/en-us/microsoft-edge/platform/documentation/f12-devtools-guide/">Edge</a></li>
</ul>


<h4>Automated Robustness Testing</h4>

<p>When you are in the process of manually debugging code, it is useful to automate what you can. For example, when testing the IE8 crashes, we used virtual machines and an AutoIt script to recreate and test the scenario as well as to keep logs of what happened for future reference.</p>

<p>Whenever possible, always automate recreating the scenario(s) for testing purposes as it:</p>

<ul>
<li>Speeds up the process</li>
<li>Removes human error</li>
<li>Ensures consistently gathered metrics</li>
</ul>


<h2>Conclusion</h2>

<p>Many bugs will follow the pattern(s) of another bug and this can help you debug them quickly. However, this is not always the case; tools and automated tests will help you speed up your debugging time, but cannot keep you from chasing ghosts.</p>

<p>Remember, the following steps are the key to solving difficult bugs without losing a lot of time and effort:</p>

<ol>
<li>Isolate the problem</li>
<li>Automate</li>
<li>Remove as many variables as possible</li>
<li>Change one variable at a time</li>
<li>Test, document, and repeat</li>
<li>Narrow the scope</li>
<li>Repeat</li>
</ol>


<p>When faced with a challenging bug watch how much time and effort you give to it. When it’s time to step back and re-evaluate your approach, do so and success will follow.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[RailsConf 2016 Recap]]></title>
    <link href="http://engineering.cerner.com/blog/railsconf-2016-recap/"/>
    <updated>2016-05-19T00:00:00-05:00</updated>
    <id>http://engineering.cerner.com/blog/railsconf-2016-recap</id>
    <content type="html"><![CDATA[<p>At Cerner, we love Ruby and Rails and use it prominently in our cloud based solutions. When we found out RailsConf 2016 was going to be held in the hometown of Cerner&rsquo;s headquarters in Kansas City, Missouri, we were excited for the opportunity to support the conference by attending as well as being a sponsor and hosting one of the official after parties.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/welcome.jpg" title="'Welcome To RailsConf'" ></p>

<p>RailsConf 2016 took place in the heart of downtown Kansas City. From day one, the atmosphere was buzzing. We usually send a few engineers to RailsConf every year, but since it was in Kansas City, we were lucky enough to send 50+ engineers to soak up all of the knowledge and news that was being shared across three session-filled days.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/cerner-engineers.jpg" title="'Cerner engineers" ></p>

<p>We were a silver sponsor this year, which provided us with a booth space where we could share Cerner&rsquo;s story with Rails developers from across the world.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/cerner-booth.jpg" title="'Cerner booth'" ></p>

<p>In addition to meeting all of the wonderful Rails developers and getting the chance to tell them about Cerner and how we are huge into Rails, we also had some fun at our booth with some multiplayer gaming action&hellip;</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/cerner-booth-multiplayer.jpg" title="'Multiplayer gaming'" ></p>

<p>&hellip;and our daily raffles. We gave away some some awesome BBQ from Joe&rsquo;s KC and a Phantom 3 Professional drone. Congratulations to both winners.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/joes-bbq-winner.jpg" title="&#34;Joe's BBQ Winner&#34;" alt="&#34;Joe's BBQ Winner&#34;"></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/drone-winner.jpg" title="'Phantom 3 Drone Winner'" ></p>

<p>And for even more fun, we hosted one of the official after parties after the closing Day 1 keynote at No Other Pub. During the party, RailsConf attendees got a chance to mingle and talk to our developers and architects in a casual setting…and the free food and drinks certainly made for a great time.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/cerner-after-party.jpg" title="'Cerner after party'" ></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/cerner-after-party-crowd.jpg" title="'Cerner after party crowd'" ></p>

<p>In addition to all of the fun, one of our distinguished engineers, Nathan Beyer, gave a talk about lessons we&rsquo;ve learned using Rails at Cerner.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/nathan-beyer-talk.jpg" title="'Pragmatic Lessons of Rails & Ruby in the Enterprise'" ></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/nathan-beyer-talk2.jpg" title="'Nathan Beyer presenting'" ></p>

<p>RailsConf 2016 was a great event, and the fact that it was in our backyard, was icing on the cake. We wish it would be held in Kansas City every year, but we are already looking forward to <a href="https://twitter.com/railsconf/status/728712538897027072">RailsConf 2017 in sunny Phoenix, AZ</a>.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-05-19-railsconf-2016-recap/railsconf.jpg" title="'RailsConf'" ></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Spring 2016 Shipit Day]]></title>
    <link href="http://engineering.cerner.com/blog/spring-2016-shipit-day/"/>
    <updated>2016-04-18T00:00:00-05:00</updated>
    <id>http://engineering.cerner.com/blog/spring-2016-shipit-day</id>
    <content type="html"><![CDATA[<p>ShipIt Day is an all day, all night hackathon with the goal of building something awesome, usable, and value-adding
within the given 24 hours. This Spring marked our fifth ShipIt Day, with participation and reach growing with each
event.</p>

<p>We kicked off the event at 10 AM on Thursday, when teams broke out and got to work and stopped only for food and
caffeine. Many stayed until a few hours after dinner arrived, several stayed all night and took naps in their workspace,
and a few worked through the whole night. The event was initially announced early in February, to give teams time to
work this into their project plans. The schedule was to start at 10 am on a Thursday and wrap-up at 10 AM on Friday.
Teams then presented their awesome projects and then were free to leave for the weekend (and catch-up on some sleep).
Each team was free to choose the project they wanted to work on, with the limitation added that they should work on
something which can be deployed somewhere in 24 hours (there were bonus points involved for deployed projects).  The
winning prize not only included bragging rights, but also the <em>Golden Keyboard</em>, which will be a traveling trophy.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-04-18-spring-2016-shipit-day/shipit-start.png" title="Kickoff" ></p>

<p>The event wrapped up on April 1st at Continuous Campus. We had over 50 participants from nine different organizations
split into teams. They survived several April Fools pranks, two cases of <a href="https://www.monsterenergy.com/">Monster</a>, and 18 extra large pizzas before
presenting their projects to a room full of other participants, judges, and over 50 people watching the live broadcast
online.</p>

<p>By the time 10 AM on Friday rolled around, the group was looking sleepy but excited for presentations. We moved over to
a bigger room where a live streaming service was set up for associates at other campuses to watch presentations. A
voting form was sent out to the entire audience, and a scoresheet was given to the judges. All 12 teams got through
their presentations despite their lack of sleep, and the judges were thoroughly impressed. One judge, K.K. Kailasam made
the remark,</p>

<blockquote><p>You guys made it very difficult for us. It was extremely exciting to see the passion and excitement in each of you guys coming up and doing something outside of normal work. I’m totally humbled and very impressed.</p></blockquote>

<p>The people’s choice votes were tallied and judges decided on the top three winners.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-04-18-spring-2016-shipit-day/shipit-judging.png" title="Judging" ></p>

<h1>Winning Projects</h1>

<h3>People’s Choice</h3>

<p>Team Mantri (Dinesh Rauniyar, Raju Karki, Dinesh Bajracharya, Mahesh Acharya, Prannav Shrestha)</p>

<p>Spoor is a <a href="https://www.google.com/chrome/">Chrome</a> extension that scans through tools like <a href="https://github.com/">GitHub</a>, <a href="https://www.atlassian.com/software/jira">JIRA</a>, <a href="https://www.atlassian.com/software/crucible">Crucible</a> and <a href="https://jenkins.io/index.html">Jenkins</a> and presents relevant
information to us. Just with a single click, you can view all of the notifications from these tools and drill down to
the detailed and hyperlinked description of each one of them.</p>

<h3>Third Place</h3>

<p>Suicide Cupcakes (Travis Collins, Daniel Lloyd, Andy Quangvan, Gared Seats)</p>

<blockquote><p>The ETS_SHIV Ship IT project came about due to the frustration of being locked into a chroot jail on an ssh jump box
without any ability to lookup the necessary connection info (fully qualified domain name / IP Address) necessary to
connect to a node. Shiv is a tool that will query a read-only file system with all the node objects represented as
directories and text files.  The script writes out the files and directories to a file system that is bind mounted read-
only inside chroot.  Now users inside the chroot jail can run simple a command like; shiv cloud.  This would return the
FQDN/IP address of all nodes with a <a href="https://www.chef.io/">Chef</a> node name containing the the string “cloud”.</p></blockquote>

<h3>Second Place</h3>

<p>Really Tardy Measurement mechanicS (Jeff Compton, Himanshi Gulati, Adam Splitter, Timothy Waszac, Sriram
Vimaraju)</p>

<blockquote><p>The <em>Response Time Measurement System (RTMS)</em> framework, which our team owns, is dependent on many moving parts, such as: <a href="http://www.cerner.com/about_cerner/partnerships/operational_system_management/">Olympus</a>, <a href="http://hadoop.apache.org/">Hadoop</a> and <a href="https://jenkins.io/index.html">Jenkins</a>
in order to produce a timer. In addition, using <a href="http://www8.hp.com/us/en/software-solutions/advanced-sql-big-data-analytics/index.html">Vertica</a>, <a href="http://www.tableau.com/">Tableau</a>, <a href="https://lightson.cerner.com/">LightsOn Network</a>, and the Olympus portal for reporting
timer data.  This framework has been consolidated into a single application which can be used for development or
troubleshooting of timers.  Currently the application supports <a href="http://www.splunk.com/">Splunk</a> (for <a href="http://www.cerner.com/newsroom.aspx?id=17179872330">Millennium+</a>), <a href="http://www.cerner.com/solutions/hospitals_and_health_systems/">Millennium</a> application-produced
checkpoint CSV files, or produce timers real-time as a Millennium application sends checkpoint data to it.  After timers
have been produced, the user can inspect the composition of the timer by exposing checkpoints, incomplete timers, and
chronological representation of a timer and other timers which may have affected it in a Gantt chart.</p></blockquote>

<p>This application will improve the accuracy of timer definitions developed, improve speed of development on engineering,
and empower engineers to develop and troubleshoot their timers without depending on the entire RTMS framework.</p>

<h3>First Place</h3>

<p>Wolfe Packe (Bryan Baugher, Micah Whitacre, Christian Duranleau, Michael Barker, Alex Hostetler)</p>

<blockquote><p>Kafkaesque is a Web UI to help manage and operate <a href="http://kafka.apache.org/">Apache Kafka</a>. It provides insightful statistics about topics,
partitions and brokers and is able to balance and re-assign partitions to different brokers.</p></blockquote>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-04-18-spring-2016-shipit-day/shipit-awards.png" width="550" title="Awards" ></p>

<p>ShipIt Day has become an incredibly valuable experience for participants, giving them a chance to not only focus on
projects that they wouldn’t normally have time for, but allowing them to meet people from other teams (in different
organizations) and learn from those they normally wouldn’t work with. To quote a participant, Daniel Lloyd:</p>

<blockquote><p>ShipIt day was a blast.  It was fun, it was serious, it was fun again… We got to make some new friends in IP and
participate in an event that spanned both teams and organizations.  That in itself is enough to recommend ShipIt day to
anyone.  It provides so much perspective into how the other teams operate and the cool things other teams and
organizations are doing.</p></blockquote>

<p>Thanks to all that participated, and we’ll see you again in the summer!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Deploying Web Services With Apache Tomcat and Chef]]></title>
    <link href="http://engineering.cerner.com/blog/deploying-web-services-with-apache-tomcat-and-chef/"/>
    <updated>2016-02-04T00:00:00-06:00</updated>
    <id>http://engineering.cerner.com/blog/deploying-web-services-with-apache-tomcat-and-chef</id>
    <content type="html"><![CDATA[<p>Open source is an important part of our engineering culture and we love when we’re able to contribute back to the community. We recently open sourced our <a href="https://github.com/cerner/cerner_tomcat">Tomcat Chef Cookbook</a>, which we use to automate deploying many of our web services here at Cerner. The cookbook is meant to be simple, yet flexible enough to change most anything about <a href="http://tomcat.apache.org/">Apache Tomcat</a> to your liking. It supports installing any version of Tomcat, configuring any file within the Tomcat installation and deploying web applications.</p>

<p>The cerner_tomcat cookbook is one of our oldest cookbooks which was originally written when we first started using <a href="https://www.chef.io/">Chef</a>. We created it as the other community Tomcat cookbooks didn’t include all the features we needed. As of today, the <a href="https://github.com/chef-cookbooks/tomcat">Tomcat cookbook provided by Chef</a> seems to be the predominant cookbook in the community; however, it still lacks the functionality and flexibility we need such as deploying the applications themselves.</p>

<p>One of the biggest problems when using Chef to configure Tomcat is that Tomcat uses XML for configuration which does not map very well to Chef’s Ruby Hash-based attributes like JSON, YAML or properties files do. Our first implementation was very similar to the <a href="https://github.com/chef-cookbooks/tomcat/blob/v1.0.1/templates/default/server.xml.erb">Chef community cookbook</a> in which the cookbook provides a configuration template with a static set of properties. We quickly learned that this solution didn’t scale for us given the varying configuration needs of applications and the ever-changing configuration options across Tomcat versions. Our approach now is to allow applications to provide configuration templates themselves, allowing this cookbook to be leveraged by the various teams deploying Tomcat applications.</p>

<p>Our cerner_tomcat cookbook is one of our most used cookbooks here at Cerner. We would love for you to try out our cookbook and let us know how we can improve it.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Girls in Technology Movement - Hour of Code]]></title>
    <link href="http://engineering.cerner.com/blog/girls-in-technology-movement-hour-of-code/"/>
    <updated>2016-01-20T00:00:00-06:00</updated>
    <id>http://engineering.cerner.com/blog/girls-in-technology-movement-hour-of-code</id>
    <content type="html"><![CDATA[<p>As a female Software Engineer, I know the benefits of having other females in the field to fellowship with as well as the benefits of early exposure to coding and the technology field. The Girls in Technology Movement Hour of Code event held December 9th, 2015 at Cerner’s Innovation Campus was a great opportunity for girls from nearby schools to gather and hear a little about the technology industry, the opportunities available for their future, meet some women in the industry, and get a hands-on introduction to coding.</p>

<p>The girls worked in pairs to complete exercises using coding constructs.  The exercises from <a href="https://code.org/learn">code.org</a> followed a Star Wars theme and included things like moving a robot and escaping Stormtroopers while earning points for certain events. The intent of the event was for it to be easy to learn even for those who have no experience with code. Blocks of code had to be put together to accomplish the tasks. They could then run the code to see what happens and how it needed to be adjusted to work next time if the desired outcome wasn’t reached the first run through. A few pairs were enjoying it so much that they finished early and used the time to work on the Minecraft one as well. It was great to see them enjoying it so much that they didn’t want to stop.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-01-20-girls-in-technology-movement-hour-of-code/photo1.jpg">
<img class="center" src="http://engineering.cerner.com/assets/2016-01-20-girls-in-technology-movement-hour-of-code/photo2.jpg">
<img class="center" src="http://engineering.cerner.com/assets/2016-01-20-girls-in-technology-movement-hour-of-code/photo3.jpg"></p>

<p>The girls at this event came for various reasons, from genuine interest in technology and coding to just wanting to get out of school and go on a fun trip. And who can blame them, right? It was a fun event! They learned teamwork through working in pairs, bouncing ideas back and forth to figure out the solution to the problem. Some of them would get excited when they figured out the harder problems. It reminded me of that feeling I get when working on a project and finally figuring out a tough part that had been stumping me for awhile. If you work with code, you know that feeling.</p>

<p>After the exercises, the girls had the opportunity to talk with a female Cerner engineer. This was a chance for them to ask questions and learn more about what it’s like to be a software engineer. I spoke with a senior in high school who asked things about what college I attended, what types of classes I took, how I like my job, and what I do. I was able to share some of the fun events we have at Cerner like DevCon and Programmer’s Day.</p>

<p>I didn’t have any exposure to coding in school until I got to college, so seeing these middle and high school girls get the opportunity to see what coding and the technology industry is like is awesome. I’m glad to see something like this being put together and to have the chance to be a part of it.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2016-01-20-girls-in-technology-movement-hour-of-code/photo4.jpg"></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Identifying Network ACL Issues With Chef Locally]]></title>
    <link href="http://engineering.cerner.com/blog/identifying-network-acl-issues-with-chef-locally/"/>
    <updated>2015-11-17T00:00:00-06:00</updated>
    <id>http://engineering.cerner.com/blog/identifying-network-acl-issues-with-chef-locally</id>
    <content type="html"><![CDATA[<p>At Cerner, we use <a href="https://www.chef.io/">Chef</a> to automate our deployments across a very large distributed system comprised of many thousands of nodes. While we employ <a href="http://serverspec.org/">ServerSpec</a> and <a href="http://kitchen.ci/">Test Kitchen</a> extensively to test our Chef recipes, this still doesn&rsquo;t identify all possible failure scenarios. In a large distributed system such as ours, we occasionally run into connectivity failures between machines due to network ACL issues. This could be due to a variety of reasons. The simplest being that you didn’t know what all you were connecting to. The second being that you didn’t ask for the ACLs to be opened early enough, or at all. And finally (and most frustrating), you asked and were granted an ACL change but there still exists an issue (like an unfollowed redirect). When we took a look at these issues, we saw an opportunity for improvement in our Chef testing.</p>

<p>This is why we created <a href="https://github.com/cerner/ops_tcpdump_handler">ops_tcpdump_handler</a>, an open source Chef cookbook that is helping us identify network connectivity issues during Chef test runs.</p>

<p>As explained in the project README, when you add this cookbook into a Test Kitchen or <a href="https://www.vagrantup.com/">Vagrant</a> run list, you&rsquo;ll see an output similar to below that shows connection attempts between nodes. This will not only help you understand what nodes you&rsquo;re connecting to in a Chef run, but also diagnose any connectivity issues that arise.</p>

<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>==&gt; node01: attempted to connect to: proxy06.fedoraproject.org using http
</span><span class='line'>==&gt; node01: attempted to connect to: mirror.sfo12.us.leaseweb.net using http
</span><span class='line'>==&gt; node01: attempted to connect to: li63-48.members.linode.com using https
</span><span class='line'>==&gt; node01: attempted to connect to: 23.235.40.133 using https
</span><span class='line'>==&gt; node01: attempted to connect to: github.com using https
</span><span class='line'>==&gt; node01: attempted to connect to: codeload.github.com using https
</span><span class='line'>==&gt; node01: attempted to connect to: s3-1.amazonaws.com using https</span></code></pre></td></tr></table></div></figure>


<p>We&rsquo;re currently using this cookbook in our Chef test runs to identify and help troubleshoot connectivity issues between nodes. Next, we&rsquo;re going to incorporate this into our continuous delivery pipeline to verify connectivity first before proceeding with the deployment. We decided to open source this work to get feedback from the community and to hear your experience with these types of issues. We&rsquo;d love to hear your experience with incorporating this cookbook into your pipeline or other approaches you&rsquo;ve taken with these types of issues.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[JavaScript Logging: We Can Do Better!]]></title>
    <link href="http://engineering.cerner.com/blog/javascript-logging-we-can-do-better/"/>
    <updated>2015-07-20T00:00:00-05:00</updated>
    <id>http://engineering.cerner.com/blog/javascript-logging-we-can-do-better</id>
    <content type="html"><![CDATA[<p>Currently in the world of JavaScript these options
are what we most commonly use to generate logs:</p>

<ul>
<li><code>console.log</code></li>
<li><code>console.info</code></li>
<li><code>console.warn</code></li>
<li><code>console.error</code></li>
</ul>


<p>These are actually pretty good in most modern browsers. Even
if you go back to Internet Explorer 8 <code>console.log</code> and friends
work as long as you have the developer tools open.</p>

<p>Given that we have these logging utilities what is the problem
with using them? When in local development these are just fine
for helping debug and speed up development. They can be used to help
you quickly catch errors or see where you&rsquo;re starting to go astray
when using a library.</p>

<p><code>console.log</code> and friends allow you to see
what&rsquo;s going on and leave notes for other developers in the
future. This is okay for local development. However, what do
you do once you move into production? Almost everyone removes
<code>console</code> commands before code is served in production.</p>

<p>Without console commands in production how do you have the same
level of logging you&rsquo;re used to with standard applications?
When using a web server you get to see most every error that
occurs. Each <code>500</code> is logged to an error log file for every
error that occurs for every user. This is not something that
really exist for JavaScript. By the nature of how web browsers
work we do not get any errors that occur for the end user.</p>

<p>Here are the issues with JavaScript logging today:</p>

<ul>
<li>Local development is the only way to see the errors.</li>
<li>Logs are distributed across many clients.</li>
<li>Errors usually lack local stack context.</li>
<li>We do not know when a user sees an error.</li>
</ul>


<p>Given the problems listed above you may ask: &ldquo;Why should I care?&rdquo;
We&rsquo;ve gotten along for years without getting JavaScript errors.
Try not to follow this line of flawed reasoning. Despite spending years
without tracking analytics about how people use our sites, we
now view those tracking analytics as invaluable. Once you start
seeing your JavaScript errors at the same rate and volume as your
server side errors you will view those logs as invaluable. Most
importantly developers will finally be empowered to provide proactive fixes
for JavaScript errors the same way we can fix server side errors
in a proactive fashion.</p>

<p>Imagine this scenario, you have an advanced search feature in your
application.  This search feature works two fold: it has an
AJAX call to fill out the search results as well as a two-layer UI
with a drop down that shows the results. When a result is clicked, it
opens a more detailed modal of those results.</p>

<p>In most cases this kind of interaction is JavaScript heavy. How
do you know when the searches fail due to a scripting error, instead
of a network drop on the client? What can you do to be proactive
about issues occurring here?</p>

<p>We&rsquo;ve released a logging framework, <a href="https://github.com/cerner/canadarm/">Canadarm</a>,
to make identifying and handling these kinds of situations
easy. Now each time a script error occurs you&rsquo;ll get to see it. As
long as the client can connect to the Internet and execute JavaScript
you&rsquo;ll get to see what went wrong. A common issue you may not realize
in local testing is a Unicode search error. This logger will tell
you what error occurred as well as the language and encoding used
to read your page.</p>

<p>Below are a some topics that are likely to cross your mind.
This post will cover each of them in detail.</p>

<ul>
<li>What does Canadarm do?</li>
<li>How does Canadarm work?</li>
<li>Who has used Canadarm?</li>
<li>Has Canadarm helped solve any problems?</li>
<li>When can I use it?</li>
</ul>


<p>This post will cover all of these questions in detail.</p>

<h2>What does Canadarm do?</h2>

<p>Canadarm makes it easy to send logs to a remote server.
This alone is nothing novel and isn&rsquo;t all that impressive. It&rsquo;s fairly easy
to setup a <code>try/catch</code> around your code and send that error to
a server via a <code>GET</code> request. The real advantage to Canadarm comes in
what it does to catch the errors.</p>

<p>Canadarm has three ways to gather errors:</p>

<ul>
<li>Automatically catch all <code>window.onerror</code> events (least useful due to lack of context)</li>
<li>Automatically catch all errors that occur when events fire (most useful because it &ldquo;just works&rdquo;)</li>
<li>Manually <code>watch</code> or <code>attempt</code> individual function calls</li>
</ul>


<p>These modes allow you to write your code and not have to worry about
logging or catching errors yourself. Any global errors will be caught
and more specifically <em>all</em> errors bound to events will be caught.
The ability to catch errors related to events is the most useful feature of Canadarm.</p>

<p>Most errors that will occur on your web pages happen when a user performs
some sort of action. Canadarm is able to provide you with context specific
error messages by automatically hooking into and monitoring functions
bound to events.</p>

<p>With the <code>Canadarm.watch</code> and <code>Canadarm.attempt</code> functions, you have the power to individually
monitor specific functions. Let&rsquo;s say you have a function that gets
called without an event being fired. You can call <code>attempt</code> on that function which
will immediately invoke the function. If an error occurs, the error will be logged.
With <code>watch</code> you can watch a function once and every time it
throws an error later during execution the error will be logged.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
</pre></td><td class='code'><pre><code class='js'><span class='line'><span class="kd">function</span> <span class="nx">fastMath</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="kd">var</span> <span class="nx">addedItems</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">i</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">for</span> <span class="p">(</span><span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o">&lt;</span> <span class="nx">arguments</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>        <span class="nx">addedItems</span> <span class="o">+=</span> <span class="nx">argument</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span> <span class="c1">// This typo will throw an error when called.</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">return</span> <span class="nx">addedItems</span><span class="p">;</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// Immediately attempt to execute fast math with the arguments 1,2,3</span>
</span><span class='line'><span class="nx">Canadarm</span><span class="p">.</span><span class="nx">attempt</span><span class="p">(</span><span class="nx">fastMath</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// Override fastMath with the watched version.</span>
</span><span class='line'><span class="nx">fastMath</span> <span class="o">=</span> <span class="nx">Canadarm</span><span class="p">.</span><span class="nx">watch</span><span class="p">(</span><span class="nx">fastMath</span><span class="p">);</span>
</span></code></pre></td></tr></table></div></figure>


<p>If you don&rsquo;t want Canadarm to automatically log global errors
and/or event based errors you can opt-out of this feature. With
<code>watch</code> and <code>attempt</code> you can write your JavaScript how
you want to and not worry about what is going on within Canadarm.</p>

<p>Finally, you get out of Canadarm what you really wanted from <code>console</code> functions.
You can log specific error messages at the point you want to via
these logging commands:</p>

<ul>
<li><code>Canadarm.debug(msg, error)</code></li>
<li><code>Canadarm.info(msg, error)</code></li>
<li><code>Canadarm.warn(msg, error)</code></li>
<li><code>Canadarm.error(msg, error)</code></li>
<li><code>Canadarm.fatal(msg, error)</code></li>
</ul>


<p>Optionally, you can provide two more arguments after <code>msg</code> and <code>error</code>. <code>data</code>
followed by <code>options</code>. You can see the usage of these arguments over in the
<a href="https://github.com/cerner/canadarm/">Canadarm documentation</a>. Specifically, <code>data</code> is the most useful
here. <code>data</code> allows you to pass an extra object that will get its values passed
to the appenders. The default appender included in Canadarm will log all these
values for you as key-value pairs.</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
</pre></td><td class='code'><pre><code class='js'><span class='line'><span class="kd">function</span> <span class="nx">addPositives</span><span class="p">()</span> <span class="p">{</span>
</span><span class='line'>    <span class="kd">var</span> <span class="nx">value</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>    <span class="k">try</span> <span class="p">{</span>
</span><span class='line'>        <span class="kd">var</span> <span class="nx">addedItems</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="nx">i</span><span class="p">;</span>
</span><span class='line'>
</span><span class='line'>        <span class="c1">// Add all values together.</span>
</span><span class='line'>        <span class="k">for</span> <span class="p">(</span><span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o">&lt;</span> <span class="nx">arguments</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>            <span class="nx">value</span> <span class="o">=</span> <span class="nx">arguments</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span>
</span><span class='line'>
</span><span class='line'>            <span class="c1">// If the value is negative we throw an error.</span>
</span><span class='line'>            <span class="k">if</span> <span class="p">(</span><span class="nx">value</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>                <span class="k">throw</span> <span class="nb">Error</span><span class="p">();</span>
</span><span class='line'>            <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>            <span class="nx">addedItems</span> <span class="o">+=</span> <span class="nx">value</span><span class="p">;</span>
</span><span class='line'>        <span class="p">}</span>
</span><span class='line'>
</span><span class='line'>        <span class="k">return</span> <span class="nx">addedItems</span><span class="p">;</span>
</span><span class='line'>    <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
</span><span class='line'>        <span class="c1">// This gives a very specific error, likely relating to business logic of a</span>
</span><span class='line'>        <span class="c1">// case that should not occur.</span>
</span><span class='line'>        <span class="nx">Canadarm</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="s1">&#39;A negative value &#39;</span> <span class="o">+</span> <span class="nx">value</span> <span class="o">+</span> <span class="s1">&#39; was given.&#39;</span><span class="p">,</span> <span class="nx">e</span><span class="p">)</span>
</span><span class='line'>        <span class="k">return</span> <span class="kc">undefined</span><span class="p">;</span>
</span><span class='line'>    <span class="p">}</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'>
</span><span class='line'><span class="c1">// In the console (if the console handler is enabled) you will see the error message.</span>
</span><span class='line'><span class="nx">addPositives</span><span class="p">();</span>
</span></code></pre></td></tr></table></div></figure>


<p>To find out more on how to configure and use Canadarm go and
check out its <a href="https://github.com/cerner/canadarm/">documentation</a>. It&rsquo;s pretty easy though. You
only need to include the Canadarm code and then configure the logger. As
seen on the Canadarm readme, you can do the following to get a working
local logger:</p>

<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='js'><span class='line'><span class="nx">Canadarm</span><span class="p">.</span><span class="nx">init</span><span class="p">({</span>
</span><span class='line'>  <span class="nx">onError</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span>
</span><span class='line'>  <span class="nx">wrapEvents</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span>
</span><span class='line'>  <span class="nx">logLevel</span><span class="o">:</span> <span class="nx">Canadarm</span><span class="p">.</span><span class="nx">level</span><span class="p">.</span><span class="nx">DEBUG</span><span class="p">,</span>
</span><span class='line'>  <span class="nx">appenders</span><span class="o">:</span> <span class="p">[</span>
</span><span class='line'>    <span class="nx">Canadarm</span><span class="p">.</span><span class="nx">Appender</span><span class="p">.</span><span class="nx">standardLogAppender</span>
</span><span class='line'>  <span class="p">],</span>
</span><span class='line'>  <span class="nx">handlers</span><span class="o">:</span> <span class="p">[</span>
</span><span class='line'>    <span class="nx">Canadarm</span><span class="p">.</span><span class="nx">Handler</span><span class="p">.</span><span class="nx">consoleLogHandler</span><span class="p">,</span>
</span><span class='line'>    <span class="nx">Canadarm</span><span class="p">.</span><span class="nx">Handler</span><span class="p">.</span><span class="nx">beaconLogHandler</span><span class="p">(</span><span class="s1">&#39;http://example.com/beacon_url&#39;</span><span class="p">)</span>
</span><span class='line'>  <span class="p">]</span>
</span><span class='line'><span class="p">});</span>
</span></code></pre></td></tr></table></div></figure>


<p>Now you&rsquo;ll see all logged errors in your console with all the information the
standardLogAppender provides. Obviously you want more than local logs. Next you&rsquo;ll
see how our teams have used this logger.</p>

<h2>How does this work?</h2>

<p>Canadarm is fairly simple. The logger catches an error and then
sends that error to a central server. Under the
covers it uses Appenders and Handlers as the mechanisms to achieve
this result.</p>

<h3>Appenders</h3>

<p>An appender works as a way to process an error or log event that occurs.
The appender has this signature: <code>appender(level, exception, message, data)</code>.</p>

<ul>
<li><code>level</code> &ndash; Level of the log, one of <code>DEBUG</code>, <code>INFO</code>, <code>WARN</code>, <code>ERROR</code>, <code>FATAL</code></li>
<li><code>exception</code> &ndash; An actual JavaScript <code>Error</code> <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error">object</a>.</li>
<li><code>message</code> &ndash; Text message of the logged error.</li>
<li><code>data</code> &ndash; Extra information to provide to the appender, usually this is not used.</li>
</ul>


<p>An appender must return an object. The object should contain simple data types.
They are single key/value pairs, usually strings. The return value of
an appender is then passed to a handler.</p>

<h3>Handlers</h3>

<p>Handlers take action on the objects produces by the appenders. A handler&rsquo;s
job is to send the results of the appenders somewhere. By default
there are two handlers that come out of the box with Canadarm:
a <a href="https://github.com/cerner/canadarm/#console-log-handler">console handler</a> that logs all errors
to the console and a <a href="https://github.com/cerner/canadarm/#beacon-log-handler">beacon handler</a> that sends
all errors to a given URL end point.</p>

<h3>Appenders &amp; Handlers</h3>

<p>Appenders and handlers work together to create your logs.
Here&rsquo;s the break down of what happens during an error or logging event:</p>

<ol>
<li>Error or log event happens</li>
<li><em>Every</em> appender is iterated over in order (duplicate keys will be replaced with the value of a later appender)</li>
<li>A final object is created from the output of all appenders</li>
<li>The final object is passed to <em>every</em> handler</li>
<li>Each handler usually sends this information somewhere (e.g. console or remote server)</li>
</ol>


<p>That&rsquo;s it for how the logger works on the client. The real power comes when
you combine this log gathering with the <code><a href="https://github.com/cerner/canadarm/blob/master/lib/handler/beacon.js">BeaconHandler</a></code>.
The logs gathered are then sent to a server. The server receiving these logs should
be writing them out to a file that is then read into a logging system. We currently use a
simple Apache server and treat its access logs as our JavaScript error logs. We then send
the logs to a log aggregation tool, <a href="http://www.splunk.com/en_us/download.html">Splunk</a>.</p>

<h2>Who has used it?</h2>

<p>We have a few applications that have begun using Canadarm.</p>

<ul>
<li>HealtheLife &ndash; website for patients to manage their health</li>
<li>Internal Sites &ndash; a few sites we use internally for a few things (e.g. code review, cheat sheets etc.)</li>
</ul>


<h3>HealtheLife</h3>

<p>HealtheLife was the first client facing application to go into production
using Canadarm. They have over 3 million users and at any given moment they
usually have at least one thousand concurrent users. These metrics matter
for two reasons: first, it shows us that Canadarm
works at scale without causing issues to the application, second, we
have been able to see trends in JavaScript errors occurring in this application.</p>

<p>For those who thought &ldquo;why should I care?&rdquo; when it comes to JavaScript logs
this go live was an interesting story. Within the first 20 minutes we
noticed errors that occurred on every page load. Specifically this error
was a reference to <code>$</code> (jQuery) before it was defined. Since this was in
an analytics tracking snippet, and at the end of a script tag, it did not
cause an end user impact, beyond eating processing time to handle an
error on every page.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-07-20-javascript-logging-we-can-do-better/healthe_life_errors.png"></p>

<p>However, it did mean that analytics were not getting tracked how
the application intended. In fact, without this logger in place the
application would have happily continued along with no indication certain actions
where never taking place. Since the analytics tool did not report the
expected user interactions, it appeared as if features of the application
where not getting used, or worse, that the analytics were faulty.</p>

<p>The actual messages in the errors for this application are interesting. Since
HealtheLife is used in many countries with many different
locales, they support various languages. Because of the various
supported languages and users being able to use their browsers in any locale they want, we
had a few interesting logs messages. Specifically we have had a few logs
come across in English, Spanish, German and more. It was kind of eye opening to know that errors are actually
translated within a browser.</p>

<h3>Internal Sites</h3>

<p>Currently a few internal sites are
using Canadarm for local development and integration environments.
Most interesting so far for has been looking at the logs and seeing
who has been copy pasting code around.</p>

<p>Interestingly enough I found some random logs on our Splunk dashboard in dev.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-07-20-javascript-logging-we-can-do-better/logs_for_cheat_sheet_app.png"></p>

<p>Which lead me to github, specifically a github pages site.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-07-20-javascript-logging-we-can-do-better/cheet_sheets_app.png"></p>

<p>Seeing the application and where the logs said the application lived I was able to find the source code.
The code then lead me to the owner of the application. At that point I was able
to contact the owner and get the issue fixed. Finding another application&rsquo;s errors and
letting the owner know about is an interesting experience. The whole interaction was
cool because it was not a use case we had considered when building Canadarm.</p>

<h2>Has it helped solve any problems?</h2>

<p>As mentioned before this has helped to point out two issues:
one for HealtheLife and another for an internal application.
Pointing out issues is not enough to fix them though. Also, Canadarm
does not solve problems on its own. You get the most
out of logging when you use a tool to aggregate those logs. We&rsquo;ve
been using Splunk to aggregate our logs.</p>

<p>Combined with the searching and reporting of Splunk we&rsquo;ve
been able to leverage the logs generated by Canadarm to see
a few common trends in our code. Canadarm has helped
us to see a few common problems we have:</p>

<ol>
<li>Locally we produce a lot of JavaScript errors</li>
<li>We often introduce new errors when we write visualizations</li>
<li>Referencing variables before they exist</li>
<li>New frameworks are hard to get a handle on</li>
</ol>


<p>Using Canadarm to generate logs doesn’t solve problems on its own.
It&rsquo;s when we combine those logs with the
reporting capabilities of Splunk that we can see trends and
identify areas we need to improve upon in our development.</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-07-20-javascript-logging-we-can-do-better/bar_errors.png"></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-07-20-javascript-logging-we-can-do-better/bar_errors_hover.png"></p>

<p>On our teams it has shown that we need to get better at defining
our APIs for data visualizations. I&rsquo;ve been able to see many errors
from our developers when they first try to update or modify any of
our visualizations. Without this logging in place it would not be
so obvious that our current API is not working well for others.
This gives us empirical data that developers are having issues
using our software. Without this data, we&rsquo;d have to rely on complaints
and hope people reported the issues they encountered when using our
code.</p>

<p>As a more concrete example, recently one of our teams has begun to try and use react. By analyzing
the logs it&rsquo;s easy to see that we have had some issues getting a handle
on how react works. Over the few weeks of react work we could clearly
see a number of errors. This shows we need
a lot more training on how to properly use react. Further it shows me
that if we plan to adopt react as our frontend framework that we need
to put together a &ldquo;gotcha&rdquo; or &ldquo;tips and tricks&rdquo; guide for getting
started.</p>

<p>While the logger is not directly &ldquo;solving&rdquo; problems it is helping
to illuminate issues we are seeing in local development as well as
in production. By shining a light on these issues we are able to
move forward and solve these problems ourselves. Sometimes the
problems may be solved by additional training or they may be
solved by code changes. Most importantly Canadarm is opening our
eyes to the kinds of issues we&rsquo;ve had for years with JavaScript. We
can no longer ignore these issue because we have solid empirical evidence
showing us our problems.</p>

<h2>When can I use it?</h2>

<p>After reading along this far hopefully you&rsquo;re thinking: &ldquo;This
all sounds great! When and how can I get started?&rdquo;. It&rsquo;s pretty easy:</p>

<ol>
<li>Include <a href="https://github.com/cerner/canadarm/">Canadarm</a> in your JavaScript (as early as possible)</li>
<li>Configure Canadarm</li>
<li>Have a server to handle logs</li>
<li>Have a reporting tool on top of your logs</li>
</ol>


<h3>Standalone applications</h3>

<p>If you are a public facing Internet application or a small startup,
Canadarm can still be a great investment. Ideally you do not need
to worry about steps 3 and 4 above. Simple Log management solutions,
such as <a href="http://loggly.com">Loggly</a> should be enough for your needs.</p>

<p>Here&rsquo;s what I did to get a quick setup working:</p>

<ol>
<li>Setup a <a href="https://www.loggly.com/signup/">free account</a></li>
<li>Fill out info in the pop up</li>
<li>Go to <code>https://YOUR_SUB_DOMAIN_HERE.loggly.com/sources/setup/https</code></li>
<li>See the &ldquo;step 2&rdquo; section you should have a URL to copy that looks something like:</li>
</ol>


<p>  <a href="http://logs-01.loggly.com/inputs/WWWWWWW-55555-5555555-55WW55-WWWWW55555/tag/http/">http://logs-01.loggly.com/inputs/WWWWWWW-55555-5555555-55WW55-WWWWW55555/tag/http/</a>
5. Configure Canadarm with this end point to see the logs:</p>

<pre><code>Canadarm.init({
  onError: true,
  wrapEvents: true,
  appenders: [
    Canadarm.Appender.standardLogAppender
  ],
  handlers: [
    Canadarm.Handler.beaconLogHandler('http://logs-01.loggly.com/inputs/WWWWWWW-55555-5555555-55WW55-WWWWW55555/tag/http/'),
    Canadarm.Handler.consoleLogHandler
  ]
});
</code></pre>

<p>After this setup it was pretty easy to get some graphs going. For example
you can easily see what errors occurred by message in this pie chart:</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-07-20-javascript-logging-we-can-do-better/loggly_pie.png"></p>

<p>Even easier is getting to view the raw output of a given event message:</p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-07-20-javascript-logging-we-can-do-better/loggly_raw_msg.png"></p>

<p>Loggly is a great tool to use for large and small projects. A big bonus for
anyone starting to use Loggly for their JavaScript logs it that they can
begin to use Loggly for their other logs (if they are not already).
While Loggly may not be ideal for every situation when it comes to logging,
it is really handy when you do not have the resources, money, or time to
setup your own log aggregation tool.</p>

<h2>Summary</h2>

<p>From desktop, to mobile, to embedded devices, web browsers can be seen
everywhere. With the help of <a href="https://github.com/cerner/canadarm/">Canadarm</a> we can now see
what exactly is happening within our applications. An entire world of
client side errors and issues can now be properly managed and acted
upon. Combine these logs with an aggregation tool such as <a href="http://www.splunk.com/en_us/download.html">Splunk</a> or
<a href="http://loggly.com">Loggly</a> and you have enabled operational intelligence.</p>

<p>The next time a user logs an issue for a JavaScript error you can
respond by telling them you&rsquo;ve seen the error and are already working
to correct it. Gone are the days of reactive fixes. Now you can worry
about proactive solutions.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Managing 30,000 Logging Events Per Day With Splunk]]></title>
    <link href="http://engineering.cerner.com/blog/managing-30000-logging-events-per-day-with-splunk/"/>
    <updated>2015-01-29T00:00:00-06:00</updated>
    <id>http://engineering.cerner.com/blog/managing-30000-logging-events-per-day-with-splunk</id>
    <content type="html"><![CDATA[<p>Our team works on a patient facing web application with a thousand live clients with 2,315,000+ users. On an average, the high traffic results into more than 40,000 visits and 300,000 page views daily generating about 30,000 logging events. A considerable portion of these events are of information and warning level in order to aid proactive monitoring or identify potential issues due to clients’ misconfiguration.</p>

<h2>Before Splunk</h2>

<p>To handle this large volume of logging, our team created a rotational support role to manually monitor the logs at regular intervals daily. We built a custom log viewer that would aggregate application logs and the engineer on the support role was expected to watch this tool manually to identify issues. Although we were able to identify problems such as bad client builds or service errors, it was not very efficient nor accurate in quickly determining end user impact stats. Since there was no way to tag previously identified and resolved issues, often times newer support engineers lacked knowledge to react to a problem. This led to unnecessary escalation and engagement of next tier support.</p>

<p><strong>Below: Our old log aggregator used to identify the occurrences (2) of a log with the stack trace.</strong></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-01-29-managing-30000-logging-events-per-day/Log-Aggregator.png"></p>

<h2>Splunk Round 1</h2>

<p>Once we migrated to Splunk we were very excited about the capabilities it offered, especially around searching and data visualization. In addition to searching logs more effectively, we were able to extract meaningful information from our logs unlike before. Splunk gave us the ability to identify client and end user impact down to the user id across all events in our logs [see image below]. This helped us gain more insight into our problems and trends in terms of impact to users. For a particular problem, we were able to quickly conclude whether all clients were affected, whether clients affected were over a virtual network only, and if the issue was isolated to a specific user. This information gave us the ability to determine the impact of issues coming into our logs, including the area of the site being impacted and frequency.</p>

<p><strong>Below: Once we extracted meaningful fields in our logs, we could identify impact. In this case, an issue is spread across 9 orgs and and 28 users.</strong></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-01-29-managing-30000-logging-events-per-day/Selected-Fields.png"></p>

<p>Although we had crossed some of the hurdles which made log monitoring difficult shortly after moving to Splunk, monitoring logs for issues was still not an easy job. It was possible to overlook issues since there was no effective way of studying trends. Initially, we created dashboards which helped identify organizations having problems. This was slightly useful but failed to depict more important graphical representation of the different types of occurring issues for a particular client or for all clients at a given time.</p>

<p><strong>Below: Reports like these weren&rsquo;t very helpful. Clients with more users tend to have more errors, so this trend doesn&rsquo;t necessarily indicate a client is experiencing a downtime.</strong></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-01-29-managing-30000-logging-events-per-day/Report-users.png"></p>

<h2>Splunk Round 2</h2>

<p>It didn&rsquo;t take us long to realize that we had to get a better handle on our logs to stay on top of increasing traffic caused by a growing user base. Although we were able to identify frequently occurring errors, we still needed a more effective way to identify known issues, service issues, configuration issues and application issues. In order to do that, we needed something more meaningful than a stack trace to track issues.  We needed to tag events, and to do that, we turned to the eventtypes feature offered by Splunk.</p>

<p>Eventtypes are applied at search time and allow you to create events to tag search results with. Because they are applied at search time, we were able to add new event types and have them applied historically throughout our logs. This also gave us the ability to tweak our event types to add more known issues as we continued identifying them. Once we successfully gauged a way to take advantage of eventtypes, we came up with a query that created a stacked timechart of eventtypes where eventtypes represented known issues. Once we reached the improved level of production monitoring, the following had to be done:</p>

<ol>
<li>Create an eventtype with least priority that catches all problems and label it &ldquo;Unknown Issue.&rdquo;</li>
<li>Go through “Unknown Issues&#8221; and create prioritized eventtypes that describe the problem in english. Once an issue is logged in our bug tracking system, tag the eventtype with that id for easy tracking.</li>
<li>Repeat daily.</li>
</ol>


<p><strong>Below: Eventtypes give us the ability to see known problems that happen over time. We can even see known problems broken down by client.</strong></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-01-29-managing-30000-logging-events-per-day/Eventtype-1.png"></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-01-29-managing-30000-logging-events-per-day/Eventtype-2.png"></p>

<p>Once we had our frequently occurring problems categorized, we were able to break it down even further. We could identify problems caused by configuration in our application layer, problems that required escalation or if client side contacts needed to be engaged.</p>

<p><strong>Below: We now have the ability to track impact to users from clients not taking a service package [left], or from improper Service Configuration [right].</strong></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-01-29-managing-30000-logging-events-per-day/Impact-to-users-from-clients-1.png"></p>

<p><img class="center" src="http://engineering.cerner.com/assets/2015-01-29-managing-30000-logging-events-per-day/Impact-to-users-from-clients-2.png"></p>

<h3>Alerting</h3>

<p>We&rsquo;ve also started taking advantage of Splunk&rsquo;s alerting. With its powerful searching abilities, we have scheduled searches that trigger an alert when a particular condition is met. For example, when a client has misconfigured certain credentials that cause authentication errors all over the site, we can engage support immediately to get it resolved.</p>

<h2>What&rsquo;s Next?</h2>

<p>Although we have a better understanding of our logs now, it can get even better. We plan on continually categorizing our logs so that monitoring our system becomes really simple for everyone. Once all of our known issues are categorized, we wish to have a scheduled search that can identify anomalies in the logs. This would be highly beneficial to find out if a release introduces issues.</p>

<p>Since our site is dependent on multiple services, most of the service problems are resolved by escalated support. We are currently working on identifying problems with known resolutions  along with  the people that need to be contacted to perform the resolution steps. Eventually we would like to send alerts/emails from Splunk to Cerner’s general support directly for these issues.</p>

<p>We also plan on integrating Jira into splunk with the help of the Splunk Jira app. This will give us the ability to not only track issues in our logs, but also view their current status (investigation, assigned, fixed, resolved). This closes the loop on finding new issues, tracking their impact, and finally their resolution until the end. Splunk has been extremely exciting to work on and has been an invaluable asset to our team. We&rsquo;d love to continue the conversations on how we can improve our usage of Splunk and how others are using it as well.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Cerner and the Apache Software Foundation]]></title>
    <link href="http://engineering.cerner.com/blog/cerner-and-the-apache-software-foundation/"/>
    <updated>2014-10-28T00:00:00-05:00</updated>
    <id>http://engineering.cerner.com/blog/cerner-and-the-apache-software-foundation</id>
    <content type="html"><![CDATA[<p>At the beginning of this year, <a href="http://engineering.cerner.com/2014/01/sponsoring-the-apache-software-foundation/">we announced that Cerner became a bronze-level sponsor</a> of the non-profit <a href="http://www.apache.org/">Apache Software Foundation</a> (ASF). Many of the open source projects we use and contribute to are under the ASF umbrella, so supporting the mission and work of the ASF is important to us.</p>

<p>We&rsquo;re happy to announce that <a href="http://www.apache.org/foundation/thanks.html">Cerner has now increased our sponsorship of the ASF to become a silver-level sponsor</a>. Open source continues to play an integral role in both our architecture and engineering culture. We&rsquo;ve blogged and spoken at conferences about how several ASF projects are core foundational components in our architecture and several of our <a href="http://www.youtube.com/playlist?list=PLSti19ysyJtDYa2JdSjVYWaKhuul1jPUP">tech talks</a> have focused on ASF projects.</p>

<p>Further increasing our sponsorship of the ASF reaffirms our continued support for an organization that provides homes for numerous open source projects that are important not only to us, but the larger development community.</p>

<p><a href="http://sponsor.apache.org/"><img class="center" src="http://engineering.cerner.com/assets/2014-10-28-cerner-and-the-apache-software-foundation/asf-sponsorship-silver-logo.jpg" title="ASF Silver Sponsorship" ></a></p>
]]></content>
  </entry>
  
</feed>