Jekyll2021-02-20T03:22:34+00:00https://polyglot.jamie.ly/feed.xmlPolyglot{"twitter"=>"jamiely"}New Blog at Codesmith2019-10-01T00:00:00+00:002019-10-01T00:00:00+00:00https://polyglot.jamie.ly/programming/2019/10/01/new-blog<p>As I’ve done <a href="https://antiquity.jamie.ly/uncategorized/new-blog-location/">before</a>,
I’ll be starting a new blog at <a href="https://codesmith.jamie.ly">Codesmith</a>. See that
site for future writing!</p>
<p><img src="/assets/codesmith_blog.png" alt="" /></p>JamieAs I’ve done before, I’ll be starting a new blog at Codesmith. See that site for future writing!Handling JSON with jq2019-09-01T00:00:00+00:002019-09-01T00:00:00+00:00https://polyglot.jamie.ly/programming/2019/09/01/jq<h1 id="intro">Intro</h1>
<p><a href="https://stedolan.github.io/jq/">jq</a> is a power tool for working with JSON on
the command-line. It’s incredibly useful when working with JSON data such as
JSON documents or API responses. It’s one of several popular tools for
manipulating JSON including <a href="http://jmespath.org/">JMESPath</a> and
<a href="https://jsonnet.org/">Jsonnet</a>. Although I have become particular fond of
<code class="highlighter-rouge">jq</code>, you might want to look into <code class="highlighter-rouge">JMESPath</code> for its integration into <a href="https://aws.amazon.com/">Amazon Web Services (AWS)</a> and
<a href="https://azure.microsoft.com/en-us/">Azure</a> command-line tools.</p>
<p>If you don’t feel like installing <code class="highlighter-rouge">jq</code> on your machine, you can follow along
on <a href="https://jqplay.org/">jq play</a>.</p>
<h1 id="extracting-data">Extracting Data</h1>
<p>We can extract data from a JSON document like the one below. Let’s say this
data is stored in <code class="highlighter-rouge">data.json</code>.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Ada"</span><span class="p">,</span><span class="w">
</span><span class="s2">"reports"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"Boris"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Becky"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Booker"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Booker"</span><span class="p">,</span><span class="w">
</span><span class="s2">"reports"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"Carly"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Chris"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Carly"</span><span class="p">,</span><span class="w">
</span><span class="s2">"reports"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"Diana"</span><span class="p">,</span><span class="w">
</span><span class="s2">"David"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>
<p>Let’s say we want all of the supervisor’s names:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% jq <span class="s1">'. [] .name'</span> < data.json
<span class="s2">"Ada"</span>
<span class="s2">"Booker"</span>
<span class="s2">"Carly"</span>
</code></pre></div></div>
<p>If we don’t want the quotes, we can use <code class="highlighter-rouge">--raw-output</code> (or <code class="highlighter-rouge">-r</code>), which is a command-line flag
used to output raw values rather than JSON data:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% jq <span class="s1">'. [] .name'</span> <span class="nt">-r</span> < data.json
Ada
Booker
Carly
</code></pre></div></div>
<p>We can get all of the reports:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% jq <span class="s1">'. [] .reports []'</span> <span class="nt">-r</span> < data.json
Boris
Becky
Booker
Carly
Chris
Diana
David
</code></pre></div></div>
<p>We can get all of the employees together by combining the two filters:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% jq <span class="s1">'. [] .reports [], . [] .name'</span> <span class="nt">-r</span> < data.json
Boris
Becky
Booker
Carly
Chris
Diana
David
Ada
Booker
Carly
</code></pre></div></div>
<p>We can pipe data into <code class="highlighter-rouge">jq</code> from an API result in order to pretty print it. We
can also use this in scripting scenarios.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w">
</span><span class="err">%</span><span class="w"> </span><span class="err">curl</span><span class="w"> </span><span class="err">http</span><span class="p">:</span><span class="err">//dummy.restapiexample.com/api/v</span><span class="mi">1</span><span class="err">/employees</span><span class="w"> </span><span class="err">|</span><span class="w"> </span><span class="err">jq</span><span class="w"> </span><span class="err">.</span><span class="w">
</span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"41393"</span><span class="p">,</span><span class="w">
</span><span class="s2">"employee_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Fernando"</span><span class="p">,</span><span class="w">
</span><span class="s2">"employee_salary"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4061"</span><span class="p">,</span><span class="w">
</span><span class="s2">"employee_age"</span><span class="p">:</span><span class="w"> </span><span class="s2">"68"</span><span class="p">,</span><span class="w">
</span><span class="s2">"profile_image"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"41394"</span><span class="p">,</span><span class="w">
</span><span class="s2">"employee_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"SLKJDLKJ"</span><span class="p">,</span><span class="w">
</span><span class="s2">"employee_salary"</span><span class="p">:</span><span class="w"> </span><span class="s2">"32"</span><span class="p">,</span><span class="w">
</span><span class="s2">"employee_age"</span><span class="p">:</span><span class="w"> </span><span class="s2">"43"</span><span class="p">,</span><span class="w">
</span><span class="s2">"profile_image"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="err">...</span><span class="w">
</span></code></pre></div></div>
<h1 id="transformations">Transformations</h1>
<p><code class="highlighter-rouge">jq</code> has powerful transformation capabilties. Using the example above, we can
build new objects from the data:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% jq <span class="s1">'. [] | { firstName: .name }'</span> < data.json
</code></pre></div></div>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"firstName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Ada"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"firstName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Booker"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"firstName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Carly"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>The <code class="highlighter-rouge">map</code> filter allows us to process array items in place. It works similarly to the <a href="https://en.wikipedia.org/wiki/Map_(higher-order_function)"><code class="highlighter-rouge">map</code> higher-order function</a> as defined in many other languages.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% jq <span class="s1">'map({firstName: .name}) < data.json
</span></code></pre></div></div>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"firstName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Ada"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"firstName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Booker"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"firstName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Carly"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>
<p>We can filter the results returned by applying selectors to each object of an
array. We do this using the <code class="highlighter-rouge">select</code> filter.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% jq <span class="s1">'map(select(.name == "Ada"))'</span> < data.json
</code></pre></div></div>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Ada"</span><span class="p">,</span><span class="w">
</span><span class="s2">"reports"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"Boris"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Becky"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Booker"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>
<p>And we can combine selectors with transformations:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jq <span class="s1">'map(select(.name == "Ada") | { firstName: .name }) '</span> < data.json
</code></pre></div></div>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"firstName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Ada"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>
<h1 id="functions">Functions</h1>
<p><code class="highlighter-rouge">jq</code> comes with a bunch of built-in functions we can use to transform and
filter data, such as reversing arrays:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% jq <span class="s1">' . | reverse[] |.name'</span> <span class="nt">-r</span> < data.json
Carly
Booker
Ada
</code></pre></div></div>
<p>Filtering based on content using the <code class="highlighter-rouge">startswith</code> function:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% jq <span class="s1">' . [] | .name | select(. | startswith("A")) '</span> <span class="nt">-r</span> < data.json
Ada
</code></pre></div></div>
<p>String interpolation using a back-slash pattern:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% jq <span class="s1">' map("Hi, \(.name)")[]'</span> <span class="nt">-r</span> < data.json
Hi, Ada
Hi, Booker
Hi, Carly
</code></pre></div></div>
<p>And so many other functions you should investigate in <a href="https://stedolan.github.io/jq/manual">the manual</a>.</p>
<h1 id="working-with-aws-results">Working with AWS Results</h1>
<p>We can use <code class="highlighter-rouge">jq</code> to work with AWS API data. Assuming you have an AWS account, you
can install <a href="https://aws.amazon.com/cli/">awscli</a> to work with the AWS APIs
from the command-line. One of the simplest things you can do is list publicly
available <a href="https://aws.amazon.com/ec2/">AWS Elastic Compute Cloud (EC2)</a> <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html">images</a>. These are images you can use to create new virtual machines. This takes a long time to run, so we’ll save the
results in a local file.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws ec2 describe-images <span class="nt">--output</span> json <span class="o">></span> aws_images.json
</code></pre></div></div>
<p>The image data looks something like this:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"Images"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"Architecture"</span><span class="p">:</span><span class="w"> </span><span class="s2">"i386"</span><span class="p">,</span><span class="w">
</span><span class="s2">"CreationDate"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="s2">"ImageId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"aki-00806369"</span><span class="p">,</span><span class="w">
</span><span class="s2">"ImageLocation"</span><span class="p">:</span><span class="w"> </span><span class="s2">"karmic-kernel-zul/ubuntu-kernel-2.6.31-300-ec2-i386-20091001-test-04.manifest.xml"</span><span class="p">,</span><span class="w">
</span><span class="s2">"ImageType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kernel"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Public"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"OwnerId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"099720109477"</span><span class="p">,</span><span class="w">
</span><span class="s2">"State"</span><span class="p">:</span><span class="w"> </span><span class="s2">"available"</span><span class="p">,</span><span class="w">
</span><span class="s2">"BlockDeviceMappings"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w">
</span><span class="s2">"Hypervisor"</span><span class="p">:</span><span class="w"> </span><span class="s2">"xen"</span><span class="p">,</span><span class="w">
</span><span class="s2">"RootDeviceType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"instance-store"</span><span class="p">,</span><span class="w">
</span><span class="s2">"VirtualizationType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"paravirtual"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"Architecture"</span><span class="p">:</span><span class="w"> </span><span class="s2">"i386"</span><span class="p">,</span><span class="w">
</span><span class="s2">"CreationDate"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
</span><span class="s2">"ImageId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"aki-00896a69"</span><span class="p">,</span><span class="w">
</span><span class="s2">"ImageLocation"</span><span class="p">:</span><span class="w"> </span><span class="s2">"karmic-kernel-zul/ubuntu-kernel-2.6.31-300-ec2-i386-20091002-test-04.manifest.xml"</span><span class="p">,</span><span class="w">
</span><span class="s2">"ImageType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kernel"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Public"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"OwnerId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"099720109477"</span><span class="p">,</span><span class="w">
</span><span class="s2">"State"</span><span class="p">:</span><span class="w"> </span><span class="s2">"available"</span><span class="p">,</span><span class="w">
</span><span class="s2">"BlockDeviceMappings"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w">
</span><span class="s2">"Hypervisor"</span><span class="p">:</span><span class="w"> </span><span class="s2">"xen"</span><span class="p">,</span><span class="w">
</span><span class="s2">"RootDeviceType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"instance-store"</span><span class="p">,</span><span class="w">
</span><span class="s2">"VirtualizationType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"paravirtual"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="err">//</span><span class="w"> </span><span class="err">...</span><span class="w"> </span><span class="err">more</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>I’ve cut out most of the results since when I ran it, there were 123,521
public images. We can take limit the results to the first 1000 images by using
the following:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat </span>aws_images.json | jq <span class="s1">'.Images |= .[0:1000]'</span> <span class="o">></span> short_aws_images.json
</code></pre></div></div>
<p>If we want to get a list of the providers of Ubuntu images, and the versions
of those images, we can use the <code class="highlighter-rouge">.ImageLocation</code> property of each image dictionary, and
use a <a href="https://en.wikipedia.org/wiki/Regular_expression">regular expression</a> to parse that location, capturing information about
the Ubuntu release and version. The <code class="highlighter-rouge">capture</code> function allows us to use
<a href="https://www.regular-expressions.info/refext.html">named capture groups</a> to
transform the results into a new JSON object.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat </span>short_aws_images.json <span class="se">\</span>
| jq <span class="s1">'.Images[] | select(.ImageLocation | contains("ubuntu")) | .ImageLocation | {ImageLocation: .} + capture("^(?<provider>\\w+)\/.*ubuntu(?:-(?<release>trusty|xenial|bionic|eoan))?-(?<version>(?:[\\d\\.]+|daily))")'</span>
</code></pre></div></div>
<p>The command is a little hard to read, but we can save it as a filter file
that can be loaded via the <code class="highlighter-rouge">--from-file</code> (or <code class="highlighter-rouge">-f</code>) flag. We’ll save it as
<code class="highlighter-rouge">ubuntu.jq_filter</code> with the contents below. One big advantage of storing the
filter as a file is that we can use comments to clarify what we are trying to
do.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.Images[] <span class="c"># unpack the array at the `Images` key</span>
| <span class="k">select</span><span class="o">(</span>.ImageLocation | contains<span class="o">(</span><span class="s2">"ubuntu"</span><span class="o">))</span> <span class="c"># select all objects where the `ImageLocation` key contains ubuntu</span>
| .ImageLocation <span class="c"># grab the `ImageLocation` key value from each object</span>
| <span class="o">{</span>ImageLocation: .<span class="o">}</span> <span class="c"># Define a new object with the ImageLocation key and value</span>
<span class="c"># then add that to the object returned by this capture, which will be an object</span>
<span class="c"># where each key-value pair corresponds to a named capture group, like:</span>
<span class="c"># { "provider": "abc", "release": trusty, "version": "12.0.0" }</span>
+ capture<span class="o">(</span><span class="s2">"(?# we match the start of the string)^(?# then, we grab the provider, which is all the text before the first forward-slash)(?<provider></span><span class="se">\\</span><span class="s2">w+)</span><span class="se">\/</span><span class="s2">(?# after the first backslash, we ignore everything until the last occurrence of ubuntu).*ubuntu(?# next, we may optionally have a release name )(?:-(?<release>trusty|xenial|bionic|eoan))?-(?# finally, we have a numeric version or the word daily)(?<version>(?:[</span><span class="se">\\</span><span class="s2">d</span><span class="se">\\</span><span class="s2">.]+|daily))"</span><span class="o">)</span>
</code></pre></div></div>
<p>Unfortunately, the regular expression is a bit hard to read. <code class="highlighter-rouge">jq</code> uses the
<a href="https://github.com/kkos/oniguruma">Oniguruma regular expression library</a>
which <a href="https://github.com/stedolan/jq/wiki/Docs-for-Oniguruma-Regular-Expressions-(RE.txt)">seems to allow
comments</a>
defined using a <code class="highlighter-rouge">(?#...)</code> group. Although this is better than having no
comments, it would be even better to be able to use
<a href="https://www.regular-expressions.info/freespacing.html">free-spacing</a> to
allow multi-line regular expressions with comments. I’m not sure whether the
problem is that Oniguruma doesn’t support this or that there’s something
lacking in <code class="highlighter-rouge">jq</code>’s implementation. Below, I show the regular expression using
free-spacing with additional comments and escape characters removed.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>^ # we match the start of the string
# then, we grab the provider, which is all the text before the first forward-slash
(?<provider>\w+)
/
# after the first backslash, we ignore everything until the last occurrence of ubuntu
.*ubuntu
# next, we may optionally have a release name
(?:- # a dash is required
(?<release>trusty|xenial|bionic|eoan))?
- # then a dash
# finally, we have a numeric version or the word daily
(?<version>
(?:[\d\.]+ # the numeric version may contain digits and periods
|daily)) # or it can be the word "daily"
</code></pre></div></div>
<p>Then, we can use the filter file like so:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat </span>short_aws_images.json <span class="se">\</span>
| jq <span class="nt">-f</span> ubuntu.jq_filter
</code></pre></div></div>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"ImageLocation"</span><span class="p">:</span><span class="w"> </span><span class="s2">"099720109477/ubuntu/images/ebs/ubuntu-trusty-14.04-i386-server-20180627"</span><span class="p">,</span><span class="w">
</span><span class="s2">"provider"</span><span class="p">:</span><span class="w"> </span><span class="s2">"099720109477"</span><span class="p">,</span><span class="w">
</span><span class="s2">"release"</span><span class="p">:</span><span class="w"> </span><span class="s2">"trusty"</span><span class="p">,</span><span class="w">
</span><span class="s2">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"14.04"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"ImageLocation"</span><span class="p">:</span><span class="w"> </span><span class="s2">"099720109477/ubuntu/images-testing/ebs-ssd/ubuntu-xenial-daily-amd64-server-20170823.1"</span><span class="p">,</span><span class="w">
</span><span class="s2">"provider"</span><span class="p">:</span><span class="w"> </span><span class="s2">"099720109477"</span><span class="p">,</span><span class="w">
</span><span class="s2">"release"</span><span class="p">:</span><span class="w"> </span><span class="s2">"xenial"</span><span class="p">,</span><span class="w">
</span><span class="s2">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"daily"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"ImageLocation"</span><span class="p">:</span><span class="w"> </span><span class="s2">"099720109477/ubuntu/images-testing/hvm-ssd/ubuntu-xenial-daily-amd64-server-20181102"</span><span class="p">,</span><span class="w">
</span><span class="s2">"provider"</span><span class="p">:</span><span class="w"> </span><span class="s2">"099720109477"</span><span class="p">,</span><span class="w">
</span><span class="s2">"release"</span><span class="p">:</span><span class="w"> </span><span class="s2">"xenial"</span><span class="p">,</span><span class="w">
</span><span class="s2">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"daily"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="err">//</span><span class="w"> </span><span class="err">more</span><span class="w"> </span><span class="err">...</span><span class="w">
</span></code></pre></div></div>
<h1 id="working-with-elasticsearch-results">Working with Elasticsearch results</h1>
<p><a href="https://www.elastic.co/">Elasticsearch</a> is a distributed full-text search engine based on <a href="https://lucene.apache.org/solr/">Apache Solr</a>.
It has an easy to use API you can use to issue queries and retrieve data.
This <a href="https://dzone.com/articles/23-useful-elasticsearch-example-queries">great article on useful Elasticsearch
queries</a>
has some example data that we’ll use in our example. We’ll save this into a
file called <code class="highlighter-rouge">books.json</code>.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"hits"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"_index"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bookdb_index"</span><span class="p">,</span><span class="w">
</span><span class="s2">"_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"book"</span><span class="p">,</span><span class="w">
</span><span class="s2">"_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4"</span><span class="p">,</span><span class="w">
</span><span class="s2">"_score"</span><span class="p">:</span><span class="w"> </span><span class="mf">1.3278645</span><span class="p">,</span><span class="w">
</span><span class="s2">"_source"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Solr in Action"</span><span class="p">,</span><span class="w">
</span><span class="s2">"authors"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"trey grainger"</span><span class="p">,</span><span class="w">
</span><span class="s2">"timothy potter"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"summary"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Comprehensive guide to implementing a scalable search engine using Apache Solr"</span><span class="p">,</span><span class="w">
</span><span class="s2">"publish_date"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2014-04-05"</span><span class="p">,</span><span class="w">
</span><span class="s2">"num_reviews"</span><span class="p">:</span><span class="w"> </span><span class="mi">23</span><span class="p">,</span><span class="w">
</span><span class="s2">"publisher"</span><span class="p">:</span><span class="w"> </span><span class="s2">"manning"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"_index"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bookdb_index"</span><span class="p">,</span><span class="w">
</span><span class="s2">"_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"book"</span><span class="p">,</span><span class="w">
</span><span class="s2">"_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w">
</span><span class="s2">"_score"</span><span class="p">:</span><span class="w"> </span><span class="mf">1.2871116</span><span class="p">,</span><span class="w">
</span><span class="s2">"_source"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Elasticsearch: The Definitive Guide"</span><span class="p">,</span><span class="w">
</span><span class="s2">"authors"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"clinton gormley"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zachary tong"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"summary"</span><span class="p">:</span><span class="w"> </span><span class="s2">"A distibuted real-time search and analytics engine"</span><span class="p">,</span><span class="w">
</span><span class="s2">"publish_date"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2015-02-07"</span><span class="p">,</span><span class="w">
</span><span class="s2">"num_reviews"</span><span class="p">:</span><span class="w"> </span><span class="mi">20</span><span class="p">,</span><span class="w">
</span><span class="s2">"publisher"</span><span class="p">:</span><span class="w"> </span><span class="s2">"oreilly"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Let’s try two things with the data. Firstly, we want to transform the
original result into a JSON array of authors of all books. First, we’ll get
retrieve the array referenced by the <code class="highlighter-rouge">hits</code> key. Then, we change each “hit”
in the array and replace it with the value of the <code class="highlighter-rouge">_source.authors</code> key.
Then, we <code class="highlighter-rouge">flatten</code> the resulting multi-dimensional array. Finally, we <code class="highlighter-rouge">sort</code>
the array and only return <code class="highlighter-rouge">unique</code> results. This is not necessary but will
typically make the output cleaner.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat books.json | jq '.hits | map(._source.authors) | flatten | sort | unique'
</code></pre></div></div>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
</span><span class="s2">"clinton gormley"</span><span class="p">,</span><span class="w">
</span><span class="s2">"timothy potter"</span><span class="p">,</span><span class="w">
</span><span class="s2">"trey grainger"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zachary tong"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>
<p>Let’s try obtaining the total number of reviews of all of the books.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">cat</span><span class="w"> </span><span class="err">books.json</span><span class="w"> </span><span class="err">|</span><span class="w"> </span><span class="err">jq</span><span class="w"> </span><span class="err">'.hits</span><span class="w"> </span><span class="err">|</span><span class="w"> </span><span class="err">map(._source.num_reviews)</span><span class="w"> </span><span class="err">|</span><span class="w"> </span><span class="err">add'</span><span class="w">
</span></code></pre></div></div>
<p>First, we get the array referenced by the <code class="highlighter-rouge">hits</code> key. Then we <code class="highlighter-rouge">map</code> over that
array, returning the number of reviews. Then, we add all of those values. The
answer is <code class="highlighter-rouge">43</code>.</p>
<h1 id="conclusion">Conclusion</h1>
<p><code class="highlighter-rouge">jq</code> is an incredibly useful program that makes working with JSON data easy.
It’s the missing tool for manipulating JSON results from the command-line.
It’s a great tool to use in scripts especially when dealing with APIs that
return JSON. <code class="highlighter-rouge">jq</code> is an incredibly rich tool with its own language and
idioms. Not only can you filter and transform data, save filters in files,
but there are also lots of things I didn’t talk about including working with
streaming JSON, defining functions, and creating your own modules of custom
filters. While it’s not the only JSON tool that you should look at, it will
go a long way in filling that empty spot in your toolbox.</p>JamieIntroThe Problem With Management’s Workhorse2019-08-01T00:00:00+00:002019-08-01T00:00:00+00:00https://polyglot.jamie.ly/career/2019/08/01/management-vs-employees<h1 id="why">Why?</h1>
<p>Ever since my first few years as a software developer, I’d always wondered why
it was that no matter how much work I finished, and how well-received the work
was, that I always seemed to get more and more work to do. It seemed like
rather than being rewarded with a break from long hours, I’d get more and more
work that would require continued long hours. As my career has progressed, and
I’ve experienced other roles such as project manager, build engineer, senior
developer, team lead, manager, and director, I’ve learned the answer to this
question and many others.</p>
<h1 id="management">Management</h1>
<p>Although I was a team lead before I was ever a manager, I didn’t really
understand the point of view of a manager until I became one. This is still
really suprising to me, and I’m not sure whether it was just some form of
naivety or a general failure of empathy on my part. My experience with
management had two components. Before I became a manager, I had completed an
MBA, but that academic experience didn’t fully illuminate what goes through the
mind of a manager. When I finally became a manager, it was working with my
employees through both their accomplishments and failures that started to give
me a deeper understanding of what I’d been through in my career. With that
description of my experience, I’ll answer a few questions that always plagued me
in my early years.</p>
<h1 id="why-is-the-employee-who-completes-the-most-work-given-the-most-work">Why is the employee who completes the most work given the most work?</h1>
<p>Often managers will give the most work to the employee who does the most work.
This might seem like a tautology, but hear me out. There is a line of reasoning
like Newton’s first law that someone who does something is likely to continue
to do something. If you have someone who commonly works long hours, they are
likely to continue working long hours. If you have someone who does quality
work, they are likely to continue doing quality work. As a manager, it is
really easy to give the most work to your workhorse and rely on them, and this
is why a lot of managers will do it–because it is the easy thing. People will
always gravitate towards making the easy choice.</p>
<p>Where this becomes a problem is when you have someone like I was who didn’t
understand the reason behind it. I actually thought that my work was lacking or
that my contribution was lacking, when that was certainly not the case, as
indicated by glowing performance reviews.</p>
<p>I think it’s also problematic when it starts to become exploitation. Perhaps
the manager knows that the employee avoids confrontation and so won’t push back
on unreasonable requests. Maybe the manager knows the employee is stressed out
from financial troubles and decides the employee will accept a smaller raise.</p>
<h1 id="why-is-the-workhorse-not-be-commensurately-compensated">Why is the workhorse not be commensurately compensated?</h1>
<p>People often wonder why they are not paid more. Everyone wants to be paid more,
but if there is someone who does 80% of the work, why would they not be paid
some large percentage more than someone at their same level on the team?</p>
<p>There are a number of reasons why this might be the case. Firstly, especially
in companies with greater headcount, Human Resources often has a tight rein on
salary ranges by grade. This is for compliance reasons such as equal opportunity
and other related legislation. It also makes sense when you think of it in terms
of standardization. Someone in Human Resources, might find it is easier to compare
people when they are arranged in tiers and to try to constrain people within
boundaries by those tiers.</p>
<p>Many managers, rather than having studied or specialized in management, are
promoted through the ranks having previously been in some other sort of
non-leadership role. Worse is that companies rarely have some sort of training
program for new managers. These managers might lack the sort of flexible thinking
that may be required within the framework of a big company. The bigger the company,
the more a manager might be constrained in what she can do with traditional
compensation. Even in this rigid atmosphere, managers typically have other things
they can do including:</p>
<ul>
<li>Being more flexible with working hours and remote work</li>
<li>Encouraging and giving time for more professional development such as
conferences, hack days, 20% time, and mentoring</li>
<li>Assigning the type of work that an employee most desires</li>
<li>Using technologies that the employee might want to learn</li>
<li>Creating at atmosphere that encourages positivity, teamwork, respect, and
fulfillment</li>
<li>Listening and being responsive to all of the employees needs</li>
</ul>
<h1 id="why-does-management-make-choices-that-make-the-work-more-difficult">Why does management make choices that make the work more difficult?</h1>
<p>Middle managers are often trying to make sense of company goals and the goals
of their direct superior, then try to direct their teams towards those goals.
Hopefully their own goals align to these goals as well.</p>
<p>An individual contributor often has goals and priorities that have accreted
slowly during their tenure. While this sort of experience makes someone an SME
and the go-to person during periods of stability, during times of large
organizational changes it can be characterized as calcification. Employees each
have goals that may not align with organizational goals. When organizational
goals conflcit with personal goals, it’s a recipe for disaster.</p>
<p>One of the biggest reasons employees find it difficult to make sense of
management’s decisions is misalignment. Often middle management has existing
stakeholders whose goals will be misaligned with those of the IT or Product.
Middle managers also have personal goals that may be tangential to
organizational goals. During my stint as manager, I wanted to keep on doing
development even though it conflicted with the goals of the organization and
team. Decisions I made during this time may have been seemed mystifying to
my reports, especially since I didn’t communicate my personal goals to them.</p>
<h1 id="what-to-do">What to do</h1>
<p>As someone who has experienced all of these problems, and has gone full circle
from an individual contributor to leadership positions and back to an
individual contributor, there are certain things I do to address these issues.</p>
<p>Firstly, I try to understand clearly the expectations of the team. Although I
tend towards being a workaholic, if that’s not the culture of the team, it
doesn’t benefit anyone in the long run for me to work long hours. In fact, it
definitely hurts my home life, stunts my personal development, and sets
unrealistic expectations for the team and our stakeholders. Time spent
completing a task for work means less time spent learning new technologies and
staying up to date with current technologies. There’s also a limit to how much
someone can grind, so it’s best to save the grinding for times when it is
absolutely needed such as (hopefully infrequent) fire fighting. Spending extra
hours working when your team members are not can also breed discontent and a
feeling of resentment to those who aren’t putting in the same number of hours.
This is counter-productive because rationally, work should be about output and
productivity, not raw hours.</p>
<p>Next, I try to align my own goals with those of the team, my manager, and the
organization, in that order. Although I think organizational goals are more
important than any one person’s goals, there needs to be a level of trust in
your manager that they are steering you towards that goal. This is the same
kind of trust that a manager gives you when they aren’t micromanaging you. While
there can rarely be 100% alignment, the task of introspection and subsequent
analysis is healthy. An example of this, is when I started professional
development, I had a great attachment to the code I wrote. I’d resent it when
people criticized my code, rewrote it, or when it wasn’t used (as
in the case of a cancelled project). Becoming a better developer and team
player has meant letting go of the attachment to my code in favor of a more
team and organizational view. I have learned to accept when my work is discarded
for the better of the organization; however, this doesn’t mean that I don’t care
about my work! Instead, I try to discover how my work is furthering the
organization and find myself often pushing back when the connection isn’t
clear.</p>
<p>Finally, I constantly try and struggle to be more empathetic to my team members
and manager. Patience, empathy, and love are they keys to leading a good life.
With respect to team members, I assume each person is doing her best and try to
give them all the time I can spare. Being generous with my time is something
that I have struggled with. For my direct manager, I avoid creating too many
expectations of them (especially if they don’t have formal training or don’t
have a huge amount of experience being a manager). I also try to remove myself
as a concern for my manager by anticipating their needs and doing my job well.
What I ask in return from my team members and manager is respect and patience,
in turn.</p>{"twitter"=>"jamiely"}Why?Grafana SQLite Migration to PostgreSQL2019-07-01T00:00:00+00:002019-07-01T00:00:00+00:00https://polyglot.jamie.ly/programming/2019/07/01/grafana-sqlite-to-postgres<h1 id="introduction">Introduction</h1>
<p><a href="https://grafana.com/"> Grafana </a> is a monitoring and dashboard system that
supports metrics backends like <a href="https://prometheus.io/"> Prometheus </a>. This
post talks about the process of migrating from using a <a href="https://www.sqlite.org/index.html"> SQLite
</a> backend for Grafana to <a href="https://www.postgresql.org/"> PostgreSQL
</a>. An initial Grafana installation (at least the <a href="https://www.docker.com/">
Docker </a> image) defaults to using SQLite. While this
is nice to get up and running quickly and try out Grafana, to make Grafana
highly available (HA), it’s better to use a database backend like MySQL or
PostgreSQL.</p>
<h1 id="high-availability">High Availability</h1>
<p>Although it’s easy to dismiss HA for an internal monitoring service like
Grafana, when implemented well, monitoring becomes increasingly crucial to the
health of an organization’s business and infrastructure. As something like
Grafana becomes more important as the pulse of the organization, even brief
service interruptions become increasingly visible. If you are running a
container orchestration service such as <a href="https://aws.amazon.com/ecs/"> AWS Elastic Container Service (ECS)
</a> or <a href="https://kubernetes.io/"> Kubernetes (k8s) </a>,
then running Grafana with a an <a href="https://aws.amazon.com/ebs/"> EBS </a> <a href="https://aws.amazon.com/blogs/compute/amazon-ecs-and-docker-volume-drivers-amazon-ebs/"> Docker
volume on ECS
</a>
or <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/"> Persistent Volume (PV)
</a> on k8s is an
HA option that may be “good enough.” In this post, I assume that Grafana has
been installed via an OS package or via Docker container running SQLite.</p>
<h1 id="aside-postgresql-ha">Aside: PostgreSQL HA</h1>
<p>This talk of HA begs the question, “is your database HA?” There are quite a
number of HA solutions for PostgreSQL including <a href="https://wiki.postgresql.org/wiki/Multimaster"> multi-master
</a> (although I’ve never tried it)
and <a href="https://www.postgresql.org/docs/current/hot-standby.html"> hot standby </a>
allowing reads on the standby node. If you’re running on AWS, you might as well
run <a href="https://aws.amazon.com/rds/"> RDS </a> with <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html"> multiple availability zone
failover
</a>.</p>
<h1 id="sqlite-to-postgresql-migration">SQLite to PostgreSQL Migration</h1>
<p>The migration from SQLite to PostgreSQL is made much easier by the existence of
the <a href="https://pgloader.readthedocs.io/en/latest/"><code class="highlighter-rouge">pgloader</code></a> project which
loads data into PostgreSQL from various sources. We can copy the SQLite
database from Grafana’s data directory <code class="highlighter-rouge">/var/lib/grafana/grafana.db</code> and use
that in the migration. We create a <code class="highlighter-rouge">main.load</code> script like the one below,
lifted almost directly from the SQLite example <a href="https://pgloader.readthedocs.io/en/latest/ref/sqlite.html">in the pgloader
documentation</a>.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">load</span> <span class="k">database</span>
<span class="k">from</span> <span class="n">sqlite</span><span class="p">:</span><span class="o">///</span><span class="n">path</span><span class="o">/</span><span class="k">to</span><span class="o">/</span><span class="n">grafana</span><span class="p">.</span><span class="n">db</span>
<span class="k">into</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">//</span><span class="n">username</span><span class="p">:</span><span class="n">password</span><span class="o">@</span><span class="n">hostname</span><span class="o">/</span><span class="n">grafana</span>
<span class="k">with</span> <span class="n">include</span> <span class="k">drop</span><span class="p">,</span> <span class="k">create</span> <span class="n">tables</span><span class="p">,</span> <span class="k">create</span> <span class="n">indexes</span><span class="p">,</span> <span class="k">reset</span> <span class="n">sequences</span>
<span class="k">set</span> <span class="n">work_mem</span> <span class="k">to</span> <span class="s1">'16MB'</span><span class="p">,</span> <span class="n">maintenance_work_mem</span> <span class="k">to</span> <span class="s1">'512 MB'</span><span class="p">;</span>
</code></pre></div></div>
<h1 id="schema-issues">Schema Issues</h1>
<p>It turns out that this naive approach results in a schema that is incompatible
with what Grafana expects. For example, this will result in a field in the
<code class="highlighter-rouge">alert</code> table, <code class="highlighter-rouge">silenced</code>, defined as a <code class="highlighter-rouge">bigint</code> rather than a <code class="highlighter-rouge">boolean</code>. We
know this if we allow Grafana to create a schema by allowing it to run its own
migrations again a clean PostgreSQL database. Below, we <code class="highlighter-rouge">diff</code> the schema that
is created by <code class="highlighter-rouge">pgloader</code> versus the one created by Grafana. Notice differences
in <code class="highlighter-rouge">integer</code> versus <code class="highlighter-rouge">bigint</code> fields, <code class="highlighter-rouge">varchar</code> versus <code class="highlighter-rouge">text</code> fields, and
differences in <code class="highlighter-rouge">NULL</code>able fields.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="mi">2</span><span class="p">,</span><span class="mi">15</span><span class="n">c2</span><span class="p">,</span><span class="mi">15</span>
<span class="o"><</span> <span class="n">id</span> <span class="n">integer</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="k">version</span> <span class="n">bigint</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">dashboard_id</span> <span class="n">bigint</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">panel_id</span> <span class="n">bigint</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">org_id</span> <span class="n">bigint</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">name</span> <span class="n">character</span> <span class="n">varying</span><span class="p">(</span><span class="mi">255</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">message</span> <span class="n">text</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="k">state</span> <span class="n">character</span> <span class="n">varying</span><span class="p">(</span><span class="mi">190</span><span class="p">)</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">settings</span> <span class="n">text</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">frequency</span> <span class="n">bigint</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="k">handler</span> <span class="n">bigint</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">severity</span> <span class="n">text</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">silenced</span> <span class="n">boolean</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">execution_error</span> <span class="n">text</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="c1">---</span>
<span class="o">></span> <span class="n">id</span> <span class="n">bigint</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o">></span> <span class="k">version</span> <span class="n">bigint</span><span class="p">,</span>
<span class="o">></span> <span class="n">dashboard_id</span> <span class="n">bigint</span><span class="p">,</span>
<span class="o">></span> <span class="n">panel_id</span> <span class="n">bigint</span><span class="p">,</span>
<span class="o">></span> <span class="n">org_id</span> <span class="n">bigint</span><span class="p">,</span>
<span class="o">></span> <span class="n">name</span> <span class="n">text</span><span class="p">,</span>
<span class="o">></span> <span class="n">message</span> <span class="n">text</span><span class="p">,</span>
<span class="o">></span> <span class="k">state</span> <span class="n">text</span><span class="p">,</span>
<span class="o">></span> <span class="n">settings</span> <span class="n">text</span><span class="p">,</span>
<span class="o">></span> <span class="n">frequency</span> <span class="n">bigint</span><span class="p">,</span>
<span class="o">></span> <span class="k">handler</span> <span class="n">bigint</span><span class="p">,</span>
<span class="o">></span> <span class="n">severity</span> <span class="n">text</span><span class="p">,</span>
<span class="o">></span> <span class="n">silenced</span> <span class="n">bigint</span><span class="p">,</span>
<span class="o">></span> <span class="n">execution_error</span> <span class="n">text</span><span class="p">,</span>
<span class="mi">17</span><span class="p">,</span><span class="mi">21</span><span class="n">c17</span><span class="p">,</span><span class="mi">21</span>
<span class="o"><</span> <span class="n">eval_date</span> <span class="k">timestamp</span> <span class="k">without</span> <span class="n">time</span> <span class="k">zone</span><span class="p">,</span>
<span class="o"><</span> <span class="n">new_state_date</span> <span class="k">timestamp</span> <span class="k">without</span> <span class="n">time</span> <span class="k">zone</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">state_changes</span> <span class="n">integer</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">created</span> <span class="k">timestamp</span> <span class="k">without</span> <span class="n">time</span> <span class="k">zone</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="o"><</span> <span class="n">updated</span> <span class="k">timestamp</span> <span class="k">without</span> <span class="n">time</span> <span class="k">zone</span> <span class="k">NOT</span> <span class="k">NULL</span>
<span class="c1">---</span>
<span class="o">></span> <span class="n">eval_date</span> <span class="k">timestamp</span> <span class="k">with</span> <span class="n">time</span> <span class="k">zone</span><span class="p">,</span>
<span class="o">></span> <span class="n">new_state_date</span> <span class="k">timestamp</span> <span class="k">with</span> <span class="n">time</span> <span class="k">zone</span><span class="p">,</span>
<span class="o">></span> <span class="n">state_changes</span> <span class="n">bigint</span><span class="p">,</span>
<span class="o">></span> <span class="n">created</span> <span class="k">timestamp</span> <span class="k">with</span> <span class="n">time</span> <span class="k">zone</span><span class="p">,</span>
<span class="o">></span> <span class="n">updated</span> <span class="k">timestamp</span> <span class="k">with</span> <span class="n">time</span> <span class="k">zone</span>
</code></pre></div></div>
<h1 id="changing-the-approach">Changing the Approach</h1>
<p>This schema comparison suggests the correct procedure to follow. Instead of
allowing <code class="highlighter-rouge">pgloader</code> to create the PostgreSQL schema, we need to export data
from SQLite into the PostgreSQL schema generated by Grafana. The first step is
to spin up an instance of Grafana pointed at some PostgreSQL database. One easy
way to do this is to use <a href="https://docs.docker.com/compose/"> docker-compose </a>
to spin up a set of linked containers. We can define a <code class="highlighter-rouge">docker-compose.yml</code>
with the following contents that will spin up two containers on a shared
network.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3'</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">postgres</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">postgres:11</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">POSTGRES_PASSWORD=dummy</span>
<span class="na">grafana</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">grafana/grafana:5.3.4</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">grafana</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">3000:3000</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">../volumes/grafana/data:/var/lib/grafana:z</span>
<span class="na">env_file</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">./grafana.env</span>
</code></pre></div></div>
<p>In the <code class="highlighter-rouge">./grafana.env</code>, we must configure Grafana to use the <code class="highlighter-rouge">postgres</code> service
defined above.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GF_DATABASE_URL=postgres://postgres:dummy@postgres/grafana
</code></pre></div></div>
<p>Note that the Grafana container will initially fail because the PostgreSQL
has not been initialized. Once the <code class="highlighter-rouge">postgres</code> service container intializes, we
create the Grafana database. First, we initiate a <code class="highlighter-rouge">psql</code> prompt inside the
<code class="highlighter-rouge">postgres</code> container:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker-compose <span class="nb">exec </span>postgres psql <span class="nt">-h</span> 127.0.0.1 <span class="nt">-U</span> postgres
</code></pre></div></div>
<p>Then we issue the creation command:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">psql</span><span class="o">></span> <span class="k">CREATE</span> <span class="k">DATABASE</span> <span class="n">grafana</span><span class="p">;</span>
</code></pre></div></div>
<p>Next, we try restarting the Grafana container so that it will try to connect:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker-compose restart grafana
</code></pre></div></div>
<p>The container should start and stay active, and a migration will create the
schema and data in the <code class="highlighter-rouge">grafana</code> database. We can export that schema by running
<code class="highlighter-rouge">pg_dump</code> in the <code class="highlighter-rouge">postgres</code> container.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> docker-compose <span class="nb">exec </span>postgres <span class="se">\</span>
pg_dump <span class="nt">--schema-only</span> <span class="nt">-h</span> 127.0.0.1 <span class="nt">-U</span> postgres grafana <span class="se">\</span>
| <span class="nb">tee </span>schema.sql
</code></pre></div></div>
<h1 id="retrying-the-load">Retrying the Load</h1>
<p>Then, we modify <code class="highlighter-rouge">main.load</code> to avoid recreating the schema.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">load</span> <span class="k">database</span>
<span class="k">from</span> <span class="n">sqlite</span><span class="p">:</span><span class="o">///</span><span class="k">data</span><span class="o">/</span><span class="n">grafana</span><span class="p">.</span><span class="n">db</span>
<span class="k">into</span> <span class="n">postgresql</span><span class="p">:</span><span class="o">//</span><span class="n">username</span><span class="p">:</span><span class="n">password</span><span class="o">@</span><span class="n">hostname</span><span class="o">/</span><span class="n">grafana</span>
<span class="k">with</span> <span class="k">data</span> <span class="k">only</span><span class="p">,</span> <span class="k">reset</span> <span class="n">sequences</span>
<span class="k">set</span> <span class="n">work_mem</span> <span class="k">to</span> <span class="s1">'16MB'</span><span class="p">,</span> <span class="n">maintenance_work_mem</span> <span class="k">to</span> <span class="s1">'512 MB'</span><span class="p">;</span>
</code></pre></div></div>
<p>Rerunning <code class="highlighter-rouge">pgloader</code> with this modified script, against a database initialized
with the schema generated by Grafana completes the migration.</p>
<h1 id="conclusion">Conclusion</h1>
<p>In this post, we discussed how to migrate a Grafana database from SQLite to
PostgreSQL. While <code class="highlighter-rouge">pgloader</code> made the process easier, there were initial
difficulties using the correct schema. Relying on Grafana’s migrations resolved
the issues.</p>{"twitter"=>"jamiely"}IntroductionMy Top 3 Interview Questions to Ask Employers2019-06-01T00:00:00+00:002019-06-01T00:00:00+00:00https://polyglot.jamie.ly/programming/2019/06/01/3-interview-questions<p>I fully acknowledge that I’ve had a lot of <a href="https://www.jamie.ly/resume">different
jobs</a>. I’ve had ample opportunity to experience a
good variety of different work environments and employers. I’m also getting to
the point in my career where I can be a little more choosy about where I work.
This is a far-cry from the job hunt after college, where I almost cried having
received a single offer (I didn’t know if I’d receive any, and I accepted right
away). I have also been a team lead and manager, so that experience informs
my view on some of these topics. This post lists the top three questions that I
like to ask when I am applying for a job. There are some questions that are
important but more basic, like, “do you provide free filtered or spring water?”
that won’t be on this list because they are simple and the answer is very
clear. I also think it is important to most people that a company provide free
drinking water, so there’s not much to talk about. Believe it or not, I have
heard of companies not offering this “benefit”.</p>
<p>When I am applying, it is common for me to ask many more questions than I am
asked. I like to think this is because my resume, answers to technical
questions, and answers regarding my experience and behavior are that clear
and informative. Limiting the number of topics or questions I’m listing to three
is difficult, but doable by focusing on the things most important to me. Here
are the three things that are most important to me right now:</p>
<ul>
<li>Family</li>
<li>“Interesting” work</li>
<li>A collegial team</li>
</ul>
<h1 id="family">Family</h1>
<p>I consider myself a family-first kind of person. I do have children, and I need
to be able to take sick-days if they are sick, run errands like taking them to
regular health checkups, and I want to be able to attend their school
functions. While it is illegal for employers to ask you if you have children,
I always mention that I do. I don’t recommend this for other people. For me, I
want to give employers the opportunity to <strong>illegally</strong> discriminate against me
(because of my parental status). I am a very practical person and I assume that
if they would not hire me becase I am a parent, that I would not want to work
there. Here’s my question:</p>
<blockquote>
<p>When your child has an event at school, such as a class party, a special
mother-daughter event, or something like a field day, is it easy for you to
miss work and attend? Would you use PTO or make up the hours? Can you tell me
about a time when your manager made you feel like she respected your work
and time?</p>
</blockquote>
<p>This type of question can be asked in many different ways, but there are key
points to tease out. For people on your team, or in a similar position, is it
frowned upon to take off in the middle of the day? If you do take off are you
forced to use PTO? Can you just make up the hours? This question is
particularly relevant for those of us who are expected to be loosely on call on
evenings and weekends anyway. There are managers who expect you respond to late
night, unscheduled calls, but balk at short, scheduled errands run in the
middle of the day. Finally, is time with your family something that is given
begrudgingly, or do people really care about their own families and yours?</p>
<p>People without children might argue that there is incidental discrimination
against them because they don’t have the same opportunties to flex time. I’d
say, why not? A person should be able to flex time to do something important to
them. Unfortunately, not all managers would agree, and that’s why you have to
ask the right questions.</p>
<h1 id="interesting-work">“Interesting” work</h1>
<p>I tend to avoid positions working on well-established products with only minor
feature additions. I enjoy working on major new features, tricky bugs, and
technical challenges. I’ve worked on these types of products and at some point
the features become things like:</p>
<ul>
<li>Change the marketing text in the alpha quadrant</li>
<li>Make the button red instead of blue</li>
<li>Adjust the timeout from 60 to 120 seconds</li>
</ul>
<p>having completed all of these tasks quickly, I am left with no specific tasks
to work on, and enter a period of what some would call yak-shaving where I do
useful but ultimately not asked-for work:</p>
<ul>
<li>Mock the 3rd party API for unit testing and QA</li>
<li>Reduce integration test time by 50%</li>
<li>Add static code analysis and related reporting to the CI pipeline</li>
</ul>
<p>While initially interesting, creating and working on my own pet tasks for a
project starts to lose its appeal for me after a couple months.</p>
<p>My question seeks to find out whether the position will be like this:</p>
<blockquote>
<p>Will I be working on a single product? If so, how often does the product have
major new features? How often do you rework the product? How long on average
does a team member stay to work on this product? What are some interesting
technical challenges that you’ve had with this product in the past month?
What about the past year?</p>
</blockquote>
<p>People may have left the team to work on other, more interesting projects. The
product may be winding down, so there may be limited new work on it. Maybe the
product has no users, so there have been very few bug reports.</p>
<h1 id="a-collegial-team">A collegial team</h1>
<p>I’ve worked loosely solo, doing my own requirements gathering and full-stack
development; on a team, partially solo, but with shared
responsibilitiesl; and as part of a team, working on the same product, work
split by features. I don’t have a strong preference in terms of work, but I
think I can contribute more to an organization the more I work with others, so
that’s the type of position I gravitate towards.</p>
<p>I want to work with a team that shows respect to each other, that is proud of
their work, and that seeks to improve. This question tries to determine
how closely the team matches that desire.</p>
<blockquote>
<p>Has there ever been a time when one team member had a conflict with another
team member? How was it resolved? How does the team communicate? How does the
team share knowledge? How would you characterize the team dynamic? Can you
tell me about a time when there was a problem with the product, and the team
worked together to resolve it? How does the team do continuing education?
How does the team and its members receive feedback?</p>
</blockquote>
<p>Many interviewers wouldn’t answer something directly about disrespect within
the team (even though they would expect you to be candid), so using the
ambiguous <em>conflict</em> will help. Sometimes the team does not use group chat to
communicate (my preferred method of team communication), instead communicating
mostly one-on-one. Sometimes they all sit around the same table and will talk
aloud as needed. Teams may be split into two factions by geography and although
the team is logically one team, it is really two teams. Do the members of the
team learn together? How do they share knowledge? If I do something wrong, will
my coworkers be comfortable telling me? Do the senior team members and
architects feel unassailable (in terms of their ideas)?</p>
<h1 id="conclusion">Conclusion</h1>
<p>These are questions I ask based on what is important to me. They may have no
value for you depending on your priorities, your career goals, or your
experiences so far. I think that senior engineers should be more critical of
their prospective employers and treat interviews as a two-way street.</p>{"twitter"=>"jamiely"}I fully acknowledge that I’ve had a lot of different jobs. I’ve had ample opportunity to experience a good variety of different work environments and employers. I’m also getting to the point in my career where I can be a little more choosy about where I work. This is a far-cry from the job hunt after college, where I almost cried having received a single offer (I didn’t know if I’d receive any, and I accepted right away). I have also been a team lead and manager, so that experience informs my view on some of these topics. This post lists the top three questions that I like to ask when I am applying for a job. There are some questions that are important but more basic, like, “do you provide free filtered or spring water?” that won’t be on this list because they are simple and the answer is very clear. I also think it is important to most people that a company provide free drinking water, so there’s not much to talk about. Believe it or not, I have heard of companies not offering this “benefit”.Test-Driven Development via Expression Parsing2019-05-01T00:00:00+00:002019-05-01T00:00:00+00:00https://polyglot.jamie.ly/programming/2019/05/01/tdd-polish-notation<h1 id="intro">Intro</h1>
<p>The practice of <a href="https://en.wikipedia.org/wiki/Test-driven_development">Test-Driven Development
(TDD)</a> can sound like an
excercise is self-flagellation. It makes sense considering the mainstream
attitude towards writing automated tests as a chore akin to writing
documentation, rather than a part of development as integral as the code
itself. {1} But if part of the distaste is fear of the unknown, then exposure
and practice may be acclimatizing forces. This post will introduce the practice
of TDD by implementing a Polish notation parser in JavaScript (ES6). We will
implement a simple testing framework, and implement several features using the
TDD methodology.</p>
<h1 id="polish-notation">Polish Notation</h1>
<p>You may be familiar with <a href="https://en.wikipedia.org/wiki/Polish_notation">Polish
notation</a> if you have ever seen
lisp-like code. In Polish notation, the operator comes first, followed by the
operands.</p>
<p>We want to implement a Polish notation parser. The first step is to come up
with some <a href="https://en.wikipedia.org/wiki/User_story">user stories</a>. We’ll
implement them one at a time using TDD.</p>
<ul>
<li>As a user, I want to enter a number
and have it recognized as a number</li>
<li>As a user, I want to enter an expression in Polish notation
to add two numbers and view the sum
eg: + 1 2 = 3</li>
<li>As a user, I want to enter an expression
to multiply numbers and view the product
eg: * 1 2 = 2</li>
<li>As a user, I want to be able to
both multiply and add groups.
eg: (* (+ 1 2) 5)</li>
</ul>
<h1 id="setup">Setup</h1>
<p>Since many of us learned JavaScript by using it in the browser, our testing
framework will be browser-based. Although running the tests using NodeJS is
simpler from a <a href="https://en.wikipedia.org/wiki/Continuous_integration">Continuous Integration
(CI)</a> perspective, my
hope is using a browser rather than command-line will be a gentler
introduction. We start with a simple function called <code class="highlighter-rouge">runTest</code>, which will take
the name of a test function (in the global scope), and run that function. It’s
expected that the test function will return true if the test passes, and
something <a href="https://developer.mozilla.org/en-US/docs/Glossary/Falsy">falsy</a>
otherwise. If there is an exception, the test will fail. When the test passes,
we write to the page the word “passed” in green. When it fails, we write the
word “FAILED” in red caps.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">runTest</span><span class="p">(</span><span class="nx">testName</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">passed</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nx">passed</span> <span class="o">=</span> <span class="nb">window</span><span class="p">[</span><span class="nx">testName</span><span class="p">]()</span>
<span class="p">}</span>
<span class="k">catch</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">passed</span> <span class="p">?</span> <span class="s1">'<span style="color: green">passed</span>'</span> <span class="p">:</span> <span class="s1">'<b style="color:red">FAILED</b>'</span><span class="p">;</span>
<span class="nb">document</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="nx">testName</span> <span class="o">+</span> <span class="s1">' '</span> <span class="o">+</span> <span class="nx">result</span> <span class="o">+</span> <span class="s1">'<br>'</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Since we will be adding tests incrementally, we will create an array of test
names, and execute <code class="highlighter-rouge">runTest</code> against each name. Another way to do things would
be to get a list of functions in the global scope beginning with a certain
prefix like <code class="highlighter-rouge">test</code>, and just execute all of them. I want to avoid as much
<a href="https://en.wikipedia.org/wiki/Magic_(programming)">magic</a> as we can in this
post, so we will just manually specify them.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">tests</span> <span class="o">=</span> <span class="p">[];</span>
<span class="nx">tests</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(</span><span class="nx">runTest</span><span class="p">);</span>
</code></pre></div></div>
<p>This code is placed inside a script block and saved in a file called
<code class="highlighter-rouge">expressions.html</code>. We’ll open this page in our browser, and reload it
whenever we want to run the tests.</p>
<h1 id="adding-numbers-and-starting-with-tdd">Adding Numbers and Starting with TDD</h1>
<p>Let’s start with the first user story:</p>
<blockquote>
<p>As a user, I want to enter a number and have it recognized as a number.</p>
</blockquote>
<p>In TDD, we write a failing test for this first and add it to the tests array.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">testNum</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="s2">"3"</span><span class="p">)</span> <span class="o">===</span> <span class="mi">3</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">tests</span> <span class="o">=</span> <span class="p">[</span>
<span class="s1">'testNum'</span>
<span class="p">];</span>
</code></pre></div></div>
<p>We will assume that there will be a function called <code class="highlighter-rouge">evaluateExpression</code>. When
we call it with the string <code class="highlighter-rouge">3</code>, we should get back the number <code class="highlighter-rouge">3</code>. Reloading
the browser to run the test, we should see that the test failed. Opening the
developer console to view the console logs, we see that the error was:</p>
<blockquote>
<p>evaluateExpression is not defined</p>
</blockquote>
<p>That makes sense since we didn’t write it yet. Let’s do this now. One of the
principles of TDD is to get the test “green” (or passing) as soon as possible.
That means we should provide the simplest solution that makes the test pass.
Once that is done, we can modify our function to make it better. This is called
the <a href="https://blog.cleancoder.com/uncle-bob/2014/12/17/TheCyclesOfTDD.html">“red, green, refactor”
cycle</a>.
The test starts out “red”. We make it “green” as simply and quickly as
possible. Then, we refactor the code to remove duplication, satisfy edge cases,
and bring it to the final state, all the while keeping the test green as much
as possible.</p>
<p>To do this for the test above, we will just make <code class="highlighter-rouge">evaluateExpression</code> return
the number <code class="highlighter-rouge">3</code> directly.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">evaluateExpression</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="mi">3</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This may seem silly since the implementation will be just a one-liner; but,
when starting out, it’s important to get a feel for the process. When we come
to a feature that is not so easily implemented, we’ll want to keep in mind that
the task is not to complete a fully-formed feature, but to make the test green.
Some people including <a href="https://en.wikipedia.org/wiki/Kent_Beck">Kent Beck</a>, the
creator of TDD, have said that this sort of rigorous cycle helps keep them on
task and working towards the goal, rather than getting bogged down by the
ambiguity of a feature. We’ll refactor this code to remove duplication by
removing the <code class="highlighter-rouge">3</code> that occurs in both the test and the return of the function.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="nx">expr</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nb">parseInt</span><span class="p">(</span><span class="nx">expr</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Great, we have implemented the first user story using TDD. Let’s move on to the
next.</p>
<p><a href="https://github.com/jamiely/tdd/tree/story-numbers">refs/tags/story-numbers</a></p>
<h1 id="addition">Addition</h1>
<blockquote>
<p>As a user, I want to enter an expression in Polish notation
to add two numbers and view the sum
eg: + 1 2 = 3</p>
</blockquote>
<p>Remember the red, green, refactor cycle. Let’s create the failing test and
add it to tests array.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">testAdd</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="s2">"+ 1 2"</span><span class="p">)</span> <span class="o">==</span> <span class="mi">3</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">tests</span> <span class="o">=</span> <span class="p">[</span>
<span class="s1">'testNum'</span><span class="p">,</span>
<span class="s1">'testAdd'</span>
<span class="p">];</span>
</code></pre></div></div>
<p>Let’s implement this in the simplest way possible. We modify the
<code class="highlighter-rouge">evaluateExpression</code> function to return a hard-coded <code class="highlighter-rouge">3</code> if the expression
starts with a plus-sign.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="nx">expr</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// if the first character is a plus</span>
<span class="k">if</span><span class="p">(</span><span class="nx">expr</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">===</span> <span class="s1">'+'</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="mi">3</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nb">parseInt</span><span class="p">(</span><span class="nx">expr</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Reloading to check, this makes the tests pass. It’s time to refactor. We will
create a function called <code class="highlighter-rouge">sum</code>, which we pass operands to. Keeping in mind that
we want to keep the test green for as long as possible, we limit the changes
we make, and move the hard-coded <code class="highlighter-rouge">3</code> to the sum function.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">function</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="nx">expr</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="nx">expr</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">===</span> <span class="s1">'+'</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// return the result of summing "parts"</span>
<span class="c1">// whatever that turns out to be</span>
<span class="k">return</span> <span class="nx">sum</span><span class="p">(</span><span class="nx">parts</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nb">parseInt</span><span class="p">(</span><span class="nx">expr</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">sum</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="mi">3</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Focusing on the sum function, we want this to just accept an array of
numbers, and return the sum. Implementing that looks like:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">sum</span><span class="p">(</span><span class="nx">operands</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">operands</span><span class="p">.</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span><span class="p">)</span> <span class="o">=></span> <span class="nx">a</span> <span class="o">+</span> <span class="nx">b</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This is similar to using a loop:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">sum</span><span class="p">(</span><span class="nx">operands</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">value</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">for</span><span class="p">(</span><span class="kd">var</span> <span class="nx">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="nx">i</span> <span class="o"><</span> <span class="nx">operands</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> <span class="nx">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">value</span> <span class="o">+=</span> <span class="nx">operands</span><span class="p">[</span><span class="nx">i</span><span class="p">];</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">value</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Saving the <code class="highlighter-rouge">sum</code> definition makes the test red, but we can make it green
again by passing evaluated numbers to the <code class="highlighter-rouge">sum</code> function.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="nx">expr</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="nx">expr</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">===</span> <span class="s1">'+'</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// split a string into an array by white-space</span>
<span class="kd">var</span> <span class="nx">parts</span> <span class="o">=</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="sr">/</span><span class="se">\s</span><span class="sr">+/</span><span class="p">);</span>
<span class="c1">// throw away the plus by removing the first-element of the array</span>
<span class="nx">parts</span><span class="p">.</span><span class="nx">shift</span><span class="p">();</span>
<span class="c1">// and convert each element to a number</span>
<span class="nx">parts</span> <span class="o">=</span> <span class="nx">parts</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">i</span><span class="p">)</span> <span class="o">=></span> <span class="nb">parseInt</span><span class="p">(</span><span class="nx">i</span><span class="p">));</span>
<span class="k">return</span> <span class="nx">sum</span><span class="p">(</span><span class="nx">parts</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nb">parseInt</span><span class="p">(</span><span class="nx">expr</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>There’s a small nitpick here. We call <code class="highlighter-rouge">parseInt</code> twice. Can we
<a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a> this up?
Although we might not always want to do so, DRY-ing in this case will help us
with future user stories. We can improve this by calling <code class="highlighter-rouge">evaluateExpression</code>
recursively.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="nx">expr</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="nx">expr</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">===</span> <span class="s1">'+'</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">parts</span> <span class="o">=</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="sr">/</span><span class="se">\s</span><span class="sr">+/</span><span class="p">);</span>
<span class="nx">parts</span><span class="p">.</span><span class="nx">shift</span><span class="p">();</span>
<span class="c1">// call evaluateExpression recursively to convert each element to a number</span>
<span class="nx">parts</span> <span class="o">=</span> <span class="nx">parts</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">evaluateExpression</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">sum</span><span class="p">(</span><span class="nx">parts</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nb">parseInt</span><span class="p">(</span><span class="nx">expr</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>On the recursive call, the function will NOT detect a plus at the beginning of
the string, so will fall through to the last case, calling <code class="highlighter-rouge">parseInt</code>.</p>
<p><img src="/assets/tdd_polish_recursion1.png" alt="Recursion diagram" /></p>
<p><a href="https://github.com/jamiely/tdd/tree/story-addition">refs/tags/story-addition</a></p>
<h1 id="multiplication">Multiplication</h1>
<p>The next user story is similar to the last, so it should be really easy to
“make green”, but there will be a big opportunity to refactor.</p>
<blockquote>
<p>As a user, I want to enter an expression
to multiply numbers and view the product
eg: * 1 2 = 2</p>
</blockquote>
<p>The test is similar to the last. Make sure to add it to the tests array!</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">testMultiply</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="s2">"* 1 2"</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Making it green is super simple because we can just copypasta most of what we
did before.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="nx">expr</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="nx">expr</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">===</span> <span class="s1">'+'</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">parts</span> <span class="o">=</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="sr">/</span><span class="se">\s</span><span class="sr">+/</span><span class="p">);</span>
<span class="nx">parts</span><span class="p">.</span><span class="nx">shift</span><span class="p">();</span>
<span class="nx">parts</span> <span class="o">=</span> <span class="nx">parts</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">evaluateExpression</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">sum</span><span class="p">(</span><span class="nx">parts</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// we just copied this whole part from the branch above 🤮</span>
<span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="nx">expr</span><span class="p">.</span><span class="nx">charAt</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">===</span> <span class="s1">'*'</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">parts</span> <span class="o">=</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="sr">/</span><span class="se">\s</span><span class="sr">+/</span><span class="p">);</span>
<span class="nx">parts</span><span class="p">.</span><span class="nx">shift</span><span class="p">();</span>
<span class="nx">parts</span> <span class="o">=</span> <span class="nx">parts</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">evaluateExpression</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">product</span><span class="p">(</span><span class="nx">parts</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nb">parseInt</span><span class="p">(</span><span class="nx">expr</span><span class="p">);</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">product</span><span class="p">(</span><span class="nx">operands</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">operands</span><span class="p">.</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span><span class="p">)</span> <span class="o">=></span> <span class="nx">a</span> <span class="o">*</span> <span class="nx">b</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This makes the tests pass, but it’s time to refactor. Since we’ve copypasta’d,
there should be a lot we can do. We extract the common lines out of the
branches, and modify the final return to look at only the first token.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="nx">expr</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">parts</span> <span class="o">=</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="sr">/</span><span class="se">\s</span><span class="sr">+/</span><span class="p">);</span>
<span class="kd">var</span> <span class="nx">first</span> <span class="o">=</span> <span class="nx">parts</span><span class="p">.</span><span class="nx">shift</span><span class="p">();</span>
<span class="nx">parts</span> <span class="o">=</span> <span class="nx">parts</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">evaluateExpression</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="nx">first</span> <span class="o">===</span> <span class="s1">'+'</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">sum</span><span class="p">(</span><span class="nx">parts</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="nx">first</span> <span class="o">===</span> <span class="s1">'*'</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">product</span><span class="p">(</span><span class="nx">parts</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nb">parseInt</span><span class="p">(</span><span class="nx">first</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a href="https://github.com/jamiely/tdd/tree/story-multiplication">refs/tags/story-multiplication</a></p>
<h1 id="parenthetical-expressions">Parenthetical expressions</h1>
<p>Finally, we are ready to implement the first complex user story, which is the
biggest one yet.</p>
<blockquote>
<p>As a user, I want to be able to
both multiply and add groups.
eg: (* (+ 1 2) 5)</p>
</blockquote>
<p>This is a bigger user story because there are some things implied in it that
are not explicitly stated (a common problem in software estimation).
Although there is only one nested grouped expression shown, it’s implied that
there can be any number of nested grouped expressions, and to any level,
meaning you could have something like:</p>
<div class="language-clojure highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="nb">*</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="p">(</span><span class="nb">+</span><span class="w"> </span><span class="p">(</span><span class="nb">*</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nb">+</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">2</span><span class="p">)))</span><span class="w">
</span></code></pre></div></div>
<p>Another thing that has been implied is that whitespace is not required next to
parentheses. To keep the implementation simple, we will reject this implication
explicitly, and require that there be spaces between all parentheses,
operators, and operands. Despite these implications, we will start with a
simpler test.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">testAddGroup</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="s2">"( + 1 2 )"</span><span class="p">)</span> <span class="o">===</span> <span class="mi">3</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The naive approach to implementing this might have us just strip the
parentheses from the expression and use our function as we did before. Since I
don’t think this helps us move the needle at all, I’m going to skip a bit
ahead. When we see an open-parenthesis, we will assume that we are starting a
new expression. The operator should be the next token. We will process tokens
until we come to a close-parenthesis. When we come to a close-parenthesis, we
will have all the operands evaluated, and we can apply the operator to the
operands. We’ll continue processing the list of tokens wherever we left off.</p>
<p>We wind up refactoring what we have to separate
<a href="https://en.wikipedia.org/wiki/Lexical_analysis#Tokenization">tokenization</a> and
evaluation. We also combine the operator branches, and use a dictionary to
lookup the operator function we want (<code class="highlighter-rouge">sum</code> or <code class="highlighter-rouge">product</code>). We also change the
operand evaluation to use a <code class="highlighter-rouge">while</code> loop instead of a <code class="highlighter-rouge">map</code>. This will be
important later for recursively evaluating all of the operands when we come to
nested groups. This relies on <code class="highlighter-rouge">tokens</code> being passed by reference to recursive
calls.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">var</span> <span class="nx">ops</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'+'</span><span class="p">:</span> <span class="nx">sum</span><span class="p">,</span> <span class="s1">'*'</span><span class="p">:</span> <span class="nx">product</span><span class="p">};</span>
<span class="kd">function</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="nx">expr</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">tokens</span> <span class="o">=</span> <span class="nx">expr</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="sr">/</span><span class="se">\s</span><span class="sr">+/</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">_evaluateExpression</span><span class="p">(</span><span class="nx">tokens</span><span class="p">)[</span><span class="mi">0</span><span class="p">];</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">_evaluateExpression</span><span class="p">(</span><span class="nx">tokens</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">first</span> <span class="o">=</span> <span class="nx">tokens</span><span class="p">.</span><span class="nx">shift</span><span class="p">();</span>
<span class="kd">var</span> <span class="nx">op</span> <span class="o">=</span> <span class="nx">ops</span><span class="p">[</span><span class="nx">first</span><span class="p">];</span>
<span class="k">if</span><span class="p">(</span><span class="nx">op</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// This doesn't help us right now, but later this</span>
<span class="c1">// control structure will allow us to keep processing</span>
<span class="c1">// operands until we run out of them. We're relying</span>
<span class="c1">// on the `tokens` field being passed by reference,</span>
<span class="c1">// so that we keep on removing tokens from the</span>
<span class="c1">// original list, until it is empty.</span>
<span class="kd">var</span> <span class="nx">operands</span> <span class="o">=</span> <span class="p">[];</span>
<span class="k">do</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">_evaluateExpression</span><span class="p">(</span><span class="nx">tokens</span><span class="p">);</span>
<span class="nx">operands</span> <span class="o">=</span> <span class="nx">operands</span><span class="p">.</span><span class="nx">concat</span><span class="p">(</span><span class="nx">result</span><span class="p">);</span>
<span class="p">}</span> <span class="k">while</span><span class="p">(</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span><span class="p">);</span>
<span class="k">return</span> <span class="p">[</span><span class="nx">op</span><span class="p">(</span><span class="nx">operands</span><span class="p">)];</span>
<span class="p">}</span>
<span class="k">return</span> <span class="p">[</span><span class="nb">parseInt</span><span class="p">(</span><span class="nx">first</span><span class="p">)];</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I should stop a moment and note that these particular steps and formulation are
derived <em>after the fact</em>. I’ve already implemented this, and I want to show the
cleanest way of arriving at my final solution. When I first implemented this
via TDD, I was stuck at this step for awhile, and the <code class="highlighter-rouge">testAddGroup</code> test
stayed red too long, and the other tests went red for awhile too. I don’t have
an easy answer as to what to do if you find yourself in this situation, except
that recursive solutions are often like this. Recursion can be like this by
definition, because you define some base cases, and then the big recursive step
that is sometimes like catching lightning in a bottle.</p>
<p><a href="https://github.com/jamiely/tdd/tree/story-groups">refs/tags/story-groups</a></p>
<p>With that, we can finish our implementation by adding branches that deal with
the parentheses.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">_evaluateExpression</span><span class="p">(</span><span class="nx">tokens</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// elided</span>
<span class="k">if</span><span class="p">(</span><span class="nx">op</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// elided</span>
<span class="kd">var</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">_evaluateExpression</span><span class="p">(</span><span class="nx">tokens</span><span class="p">);</span>
<span class="nx">operands</span> <span class="o">=</span> <span class="nx">operands</span><span class="p">.</span><span class="nx">concat</span><span class="p">(</span><span class="nx">result</span><span class="p">);</span>
<span class="c1">// elided</span>
<span class="p">}</span>
<span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="nx">first</span> <span class="o">===</span> <span class="s1">'('</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">_evaluateExpression</span><span class="p">(</span><span class="nx">tokens</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="nx">first</span> <span class="o">===</span> <span class="s1">')'</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">[];</span>
<span class="p">}</span>
<span class="c1">// elided</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If we encounter an open-parenthesis, we must recurse in order to handle a
nested expression. If we encounter a close-parenthesis, we return an empty
array. In the current implementation, this will be concatenated to the
operands array, and so leave it unchanged. This is a problem that will become
visible after we add the next test, but this makes all the current tests pass.</p>
<h1 id="nested-groups">Nested groups</h1>
<p>We’ll now test nested groups.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">testMultGroupNested</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">evaluateExpression</span><span class="p">(</span><span class="s2">"( + ( * 1 2 ) ( + 2 2 ) )"</span><span class="p">)</span> <span class="o">===</span> <span class="mi">6</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This test fails for the reason I hinted at earlier. When we see a
close-parenthesis, instead of just adding the empty array to our operands, we
must take it as some sign that we need to end our recursion, and allow some
parent-level to process the remaining tokens. We’ll modify our code to do this.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">_evaluateExpression</span><span class="p">(</span><span class="nx">tokens</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// elided</span>
<span class="k">if</span><span class="p">(</span><span class="nx">op</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">operands</span> <span class="o">=</span> <span class="p">[];</span>
<span class="k">do</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">_evaluateExpression</span><span class="p">(</span><span class="nx">tokens</span><span class="p">);</span>
<span class="c1">// we have to stop processing if we see an empty array,</span>
<span class="c1">// allowing the caller to process further tokens.</span>
<span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="nx">result</span><span class="p">.</span><span class="nx">length</span><span class="p">)</span> <span class="p">{</span> <span class="k">break</span><span class="p">;</span> <span class="p">}</span>
<span class="nx">operands</span> <span class="o">=</span> <span class="nx">operands</span><span class="p">.</span><span class="nx">concat</span><span class="p">(</span><span class="nx">result</span><span class="p">);</span>
<span class="p">}</span> <span class="k">while</span><span class="p">(</span><span class="nx">tokens</span><span class="p">.</span><span class="nx">length</span><span class="p">);</span>
<span class="k">return</span> <span class="p">[</span><span class="nx">op</span><span class="p">(</span><span class="nx">operands</span><span class="p">)];</span>
<span class="p">}</span>
<span class="c1">// elided</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This makes our tests pass. If we were implementing this feature for a project,
we’d add more test cases with deeper nesting just to make sure it works right.</p>
<p><a href="https://github.com/jamiely/tdd/tree/story-nested-groups">refs/tags/story-nested-groups</a></p>
<h1 id="conclusion">Conclusion</h1>
<p>In this post, we implemented a Polish notation parser by using TDD. TDD is
a method of testing whereby you write tests first, and follow a red, green,
refactor cycle in writing your code. Some say that it improves productivity
and results in cleaner code. I think that it’s an important methodology to
practice in order to learn how to write testable code, but I don’t write most
of my code using TDD. Where I do like to use TDD is where the requirements are
very clear, and I have a somewhat clear idea of what the implementation should
be. I also avoid using it when implementing primarily UI-level features like
animations. Still, it’s important to know how to use TDD to test features
like these, and doing so has made me a better programmer.</p>
<h1 id="notes">Notes</h1>
<p>{1}</p>
<p>I wrote about this before in <a href="/programming/2019/02/04/tdd.html">Let’s Test</a></p>{"twitter"=>"jamiely"}IntroTracer, a Swift Drawing View2019-04-01T00:00:00+00:002019-04-01T00:00:00+00:00https://polyglot.jamie.ly/programming/2019/04/01/tracer-a-swift-drawing-view<h1 id="introduction">Introduction</h1>
<p>A common way for children’s apps to teach letters is to have someone draw
the letter, the app tracking the user’s drawing. For this article, we’ll
implement this functionality by subclassing <code class="highlighter-rouge">UIView</code>, adding features
incrementally, adding tests, managing Cocoapod dependencies, and using
fastlane. You can follow along using the source code on
<a href="https://github.com/jamiely/Tracer">GitHub</a>. Along the way, I’ve added links to
git tags, pointers to the code at specific points in time.</p>
<h1 id="touches">Touches</h1>
<p>There are several <code class="highlighter-rouge">UIView</code> callbacks which deal with users touching the view.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">override</span> <span class="kd">func</span> <span class="nf">touchesBegan</span><span class="p">(</span><span class="n">_</span> <span class="nv">touches</span><span class="p">:</span> <span class="kt">Set</span><span class="o"><</span><span class="kt">UITouch</span><span class="o">></span><span class="p">,</span> <span class="n">with</span> <span class="nv">event</span><span class="p">:</span> <span class="kt">UIEvent</span><span class="p">?)</span>
<span class="k">override</span> <span class="kd">func</span> <span class="nf">touchesMoved</span><span class="p">(</span><span class="n">_</span> <span class="nv">touches</span><span class="p">:</span> <span class="kt">Set</span><span class="o"><</span><span class="kt">UITouch</span><span class="o">></span><span class="p">,</span> <span class="n">with</span> <span class="nv">event</span><span class="p">:</span> <span class="kt">UIEvent</span><span class="p">?)</span>
<span class="k">override</span> <span class="kd">func</span> <span class="nf">touchesEnded</span><span class="p">(</span><span class="n">_</span> <span class="nv">touches</span><span class="p">:</span> <span class="kt">Set</span><span class="o"><</span><span class="kt">UITouch</span><span class="o">></span><span class="p">,</span> <span class="n">with</span> <span class="nv">event</span><span class="p">:</span> <span class="kt">UIEvent</span><span class="p">?)</span>
</code></pre></div></div>
<p>When we detect that someone has moved their touch, we can grab the list of
touches, which includes the current and previous position of the touch.
We’ll store those in a <code class="highlighter-rouge">struct Line</code>.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">struct</span> <span class="kt">Line</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">start</span><span class="p">:</span> <span class="kt">CGPoint</span>
<span class="k">let</span> <span class="nv">end</span><span class="p">:</span> <span class="kt">CGPoint</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Then, we’ll draw these lines in the the <code class="highlighter-rouge">UIView</code>’s <code class="highlighter-rouge">draw</code> call. The drawing
functionality is similar to other drawing APIs like HTML canvas.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">override</span> <span class="kd">func</span> <span class="nf">draw</span><span class="p">(</span><span class="n">_</span> <span class="nv">rect</span><span class="p">:</span> <span class="kt">CGRect</span><span class="p">)</span> <span class="p">{</span>
<span class="n">lines</span><span class="o">.</span><span class="nf">forEach</span><span class="p">(</span><span class="n">drawLine</span><span class="p">)</span>
<span class="p">}</span>
<span class="kd">private</span> <span class="kd">func</span> <span class="nf">drawLine</span><span class="p">(</span><span class="nv">line</span><span class="p">:</span> <span class="kt">Line</span><span class="p">)</span> <span class="p">{</span>
<span class="k">guard</span> <span class="k">let</span> <span class="nv">context</span> <span class="o">=</span> <span class="kt">UIGraphicsGetCurrentContext</span><span class="p">()</span> <span class="k">else</span> <span class="p">{</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"ERROR: no context available"</span><span class="p">)</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="n">context</span><span class="o">.</span><span class="nf">move</span><span class="p">(</span><span class="nv">to</span><span class="p">:</span> <span class="n">line</span><span class="o">.</span><span class="n">start</span><span class="p">)</span>
<span class="n">context</span><span class="o">.</span><span class="nf">addLine</span><span class="p">(</span><span class="nv">to</span><span class="p">:</span> <span class="n">line</span><span class="o">.</span><span class="n">end</span><span class="p">)</span>
<span class="n">context</span><span class="o">.</span><span class="nf">setStrokeColor</span><span class="p">(</span><span class="kt">UIColor</span><span class="o">.</span><span class="n">black</span><span class="o">.</span><span class="n">cgColor</span><span class="p">)</span>
<span class="n">context</span><span class="o">.</span><span class="nf">strokePath</span><span class="p">()</span>
<span class="kt">UIGraphicsEndImageContext</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a href="https://github.com/jamiely/tracer/releases/tag/line-drawing">refs/tags/line-drawing</a></p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/g7SyzN9jgsI" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<h1 id="out-of-bounds">Out of Bounds</h1>
<p>We want to implement a feature where a user has to follow a particular path.
If the user goes beyond some threshold of this path, then we want to detect
that in order to give the user some sort of indication.</p>
<p>There are at least two ways we can implement this:</p>
<ol>
<li>We can allow the user to draw as long as any point drawn is within a
threshold from any point in the expected path.</li>
<li>We can track the progress of the user as they draw, and note which points
on the expected path have been drawn, and require that the user continue
on the path. In this way, we only need to check parts of the expected path
that we have not drawn yet.</li>
</ol>
<p>We’ll implement #1 first since it is simpler.</p>
<p>To implement this, we’ll first need to accept some expected path. Although
we may want smooth paths at some point using bezier paths, we’ll keep things
simple and only support straight lines between segments. We’ll display the path
under our drawing to guide our touch. Since this expected path won’t change,
we’ll draw this on an image. We’ll display the image on an image view behind
our drawing.</p>
<p><img src="/assets/2019-03-23-14-03-25.png" alt="" /></p>
<p>Next, when we draw, we will compare the points we draw, and calculate the
distance from each of the points in the expected path. This is fairly
inefficient but there are ways of making it quicker. One way is to partition
the space into a set of areas and note which points in the expected path fall
into these areas. (Related to
<a href="https://en.wikipedia.org/wiki/Binary_space_partitioning">BSP</a>.) For now, we
will just use the trivial
method of doing this. If the line that we draw is beyond some threshold
distance from the closest point on the expected path, then we will give some
indication (by drawing the line red).</p>
<p>We create a property that will draw the expected path on the image associated
with a <code class="highlighter-rouge">UIImageView</code> when it is set.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">var</span> <span class="nv">expectedPath</span><span class="p">:</span> <span class="kt">Array</span><span class="o"><</span><span class="kt">CGPoint</span><span class="o">></span> <span class="p">{</span>
<span class="k">get</span> <span class="p">{</span> <span class="k">return</span> <span class="n">_expectedPath</span> <span class="p">}</span>
<span class="k">set</span> <span class="p">{</span>
<span class="n">_expectedPath</span> <span class="o">=</span> <span class="n">newValue</span>
<span class="nf">drawExpectedPath</span><span class="p">(</span><span class="nv">points</span><span class="p">:</span> <span class="n">newValue</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kd">private</span> <span class="kd">func</span> <span class="nf">drawExpectedPath</span><span class="p">(</span><span class="nv">points</span><span class="p">:</span> <span class="kt">Array</span><span class="o"><</span><span class="kt">CGPoint</span><span class="o">></span><span class="p">)</span> <span class="p">{</span>
<span class="kt">UIGraphicsBeginImageContext</span><span class="p">(</span><span class="n">expectedPathView</span><span class="o">.</span><span class="n">bounds</span><span class="o">.</span><span class="n">size</span><span class="p">)</span>
<span class="k">guard</span> <span class="k">var</span> <span class="nv">last</span> <span class="o">=</span> <span class="n">points</span><span class="o">.</span><span class="n">first</span><span class="p">,</span>
<span class="k">let</span> <span class="nv">context</span> <span class="o">=</span> <span class="kt">UIGraphicsGetCurrentContext</span><span class="p">()</span> <span class="k">else</span> <span class="p">{</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"There should be at least one point"</span><span class="p">)</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="n">context</span><span class="o">.</span><span class="nf">setFillColor</span><span class="p">(</span><span class="kt">UIColor</span><span class="o">.</span><span class="n">white</span><span class="o">.</span><span class="n">cgColor</span><span class="p">)</span>
<span class="n">context</span><span class="o">.</span><span class="nf">fill</span><span class="p">(</span><span class="n">expectedPathView</span><span class="o">.</span><span class="n">bounds</span><span class="p">)</span>
<span class="n">context</span><span class="o">.</span><span class="nf">setStrokeColor</span><span class="p">(</span><span class="kt">UIColor</span><span class="o">.</span><span class="n">blue</span><span class="o">.</span><span class="n">cgColor</span><span class="p">)</span>
<span class="n">points</span><span class="p">[</span><span class="mi">1</span><span class="o">..<</span><span class="n">points</span><span class="o">.</span><span class="n">count</span><span class="p">]</span><span class="o">.</span><span class="n">forEach</span> <span class="p">{</span> <span class="n">pt</span> <span class="k">in</span>
<span class="n">context</span><span class="o">.</span><span class="nf">move</span><span class="p">(</span><span class="nv">to</span><span class="p">:</span> <span class="n">last</span><span class="p">)</span>
<span class="n">context</span><span class="o">.</span><span class="nf">addLine</span><span class="p">(</span><span class="nv">to</span><span class="p">:</span> <span class="n">pt</span><span class="p">)</span>
<span class="n">context</span><span class="o">.</span><span class="nf">strokePath</span><span class="p">()</span>
<span class="n">last</span> <span class="o">=</span> <span class="n">pt</span>
<span class="p">}</span>
<span class="k">let</span> <span class="nv">image</span> <span class="o">=</span> <span class="kt">UIGraphicsGetImageFromCurrentImageContext</span><span class="p">()</span>
<span class="kt">UIGraphicsEndImageContext</span><span class="p">()</span>
<span class="n">expectedPathView</span><span class="o">.</span><span class="n">image</span> <span class="o">=</span> <span class="n">image</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We change the <code class="highlighter-rouge">touchesMoved</code> call to note the color that should be used for the
line. Black if the line is valid and red if invalid. We’ll add a property in
the <code class="highlighter-rouge">Line</code> struct to hold the color as well.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">override</span> <span class="kd">func</span> <span class="nf">touchesMoved</span><span class="p">(</span><span class="n">_</span> <span class="nv">touches</span><span class="p">:</span> <span class="kt">Set</span><span class="o"><</span><span class="kt">UITouch</span><span class="o">></span><span class="p">,</span> <span class="n">with</span> <span class="nv">event</span><span class="p">:</span> <span class="kt">UIEvent</span><span class="p">?)</span> <span class="p">{</span>
<span class="n">touches</span><span class="o">.</span><span class="n">forEach</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">start</span> <span class="o">=</span> <span class="nv">$0</span><span class="o">.</span><span class="nf">previousLocation</span><span class="p">(</span><span class="nv">in</span><span class="p">:</span> <span class="k">self</span><span class="p">)</span>
<span class="k">let</span> <span class="nv">end</span> <span class="o">=</span> <span class="nv">$0</span><span class="o">.</span><span class="nf">location</span><span class="p">(</span><span class="nv">in</span><span class="p">:</span> <span class="k">self</span><span class="p">)</span>
<span class="k">let</span> <span class="nv">color</span> <span class="o">=</span> <span class="nf">colorForPoints</span><span class="p">(</span><span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">)</span>
<span class="k">self</span><span class="o">.</span><span class="n">lines</span><span class="o">.</span><span class="nf">append</span><span class="p">(</span><span class="kt">Line</span><span class="p">(</span><span class="nv">start</span><span class="p">:</span> <span class="n">start</span><span class="p">,</span> <span class="nv">end</span><span class="p">:</span> <span class="n">end</span><span class="p">,</span> <span class="nv">color</span><span class="p">:</span> <span class="n">color</span><span class="p">))</span>
<span class="p">}</span>
<span class="nf">setNeedsDisplay</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p><code class="highlighter-rouge">colorForPoints</code> just checks if both of the line’s points are within some
threshold of any of the points in the expected path, returning black if so, and
otherwise red.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="kd">func</span> <span class="nf">colorForPoints</span><span class="p">(</span><span class="n">_</span> <span class="nv">pts</span><span class="p">:</span> <span class="kt">CGPoint</span><span class="o">...</span><span class="p">)</span> <span class="o">-></span> <span class="kt">CGColor</span> <span class="p">{</span>
<span class="k">if</span> <span class="n">pts</span><span class="o">.</span><span class="nf">allSatisfy</span><span class="p">(</span><span class="n">isPointWithinBounds</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="kt">UIColor</span><span class="o">.</span><span class="n">black</span><span class="o">.</span><span class="n">cgColor</span>
<span class="p">}</span>
<span class="k">return</span> <span class="kt">UIColor</span><span class="o">.</span><span class="n">red</span><span class="o">.</span><span class="n">cgColor</span>
<span class="p">}</span>
<span class="kd">private</span> <span class="kd">func</span> <span class="nf">isPointWithinBounds</span><span class="p">(</span><span class="n">_</span> <span class="nv">pt</span><span class="p">:</span> <span class="kt">CGPoint</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Bool</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">threshold</span><span class="p">:</span> <span class="kt">CGFloat</span> <span class="o">=</span> <span class="mi">75</span>
<span class="k">return</span> <span class="n">expectedPath</span><span class="o">.</span><span class="n">contains</span> <span class="p">{</span> <span class="n">ept</span> <span class="k">in</span>
<span class="k">let</span> <span class="nv">dx</span> <span class="o">=</span> <span class="n">pt</span><span class="o">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">ept</span><span class="o">.</span><span class="n">x</span>
<span class="k">let</span> <span class="nv">dy</span> <span class="o">=</span> <span class="n">pt</span><span class="o">.</span><span class="n">y</span> <span class="o">-</span> <span class="n">ept</span><span class="o">.</span><span class="n">y</span>
<span class="k">let</span> <span class="nv">distance</span> <span class="o">=</span> <span class="nf">sqrt</span><span class="p">(</span><span class="n">dx</span> <span class="o">*</span> <span class="n">dx</span> <span class="o">+</span> <span class="n">dy</span> <span class="o">*</span> <span class="n">dy</span><span class="p">)</span>
<span class="k">return</span> <span class="n">distance</span> <span class="o"><</span> <span class="n">threshold</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p><img src="/assets/tracer_red_line.png" alt="" /></p>
<p><a href="https://github.com/jamiely/tracer/tree/red-lines">refs/tags/red-lines</a></p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/ZzuO4-Euju8" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<h1 id="refining-features">Refining Features</h1>
<p>Although we’ve made good progress, this is somewhat unimpressive. There are
two problems.</p>
<ul>
<li>Firstly, things would look a lot better with some nice image
outlining the path we have to trace, like a big letter T (we pick T because it
is easy to path).</li>
<li>Secondly, when we specify a path and threshold, we want the
view to figure out waypoints between the given path so that the distance
between any two points is less than the threshold. This will ensure that the
red lines we generate are what we’d expect.</li>
</ul>
<p>We add a <a href="https://openclipart.org/detail/288906/floral-alphabet-t">T made of
flowers</a>, centered in
the window to make things look a bit better. Note that although the image is
aspect fit, that’s a problem since we’ll have to scale the path accordingly.
Until we do that, it is better to fix the dimensions of the image.</p>
<p><a href="https://github.com/jamiely/tracer/tree/t-path">refs/tags/t-path</a></p>
<p>To add waypoints, we use a recursive method. We take the points of a line segment
and calculate the distance between them. If the distance between them is larger
than the threshold, we find the midpoint. This point will be added to the
expected path. We then find any waypoints that need to be added between
the start and midpoint, and midpoint and end. In this way, we recursively add
points until we have enough points to cover the distance.</p>
<p><a href="https://github.com/jamiely/tracer/tree/waypoints">refs/tags/waypoints</a></p>
<p><img src="/assets/tracer_t_flowers.png" alt="" /></p>
<h1 id="indicating-success">Indicating Success</h1>
<p>We need to know when we have completed tracing the figure. We will define this
as having a point close to each of the points in the expected path (with
waypoints). We’ll use the same threshold for error as described above. When we
complete tracing the figure, we will change the color of the drawn line to
green.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="k">override</span> <span class="kd">func</span> <span class="nf">draw</span><span class="p">(</span><span class="n">_</span> <span class="nv">rect</span><span class="p">:</span> <span class="kt">CGRect</span><span class="p">)</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">overrideColor</span> <span class="o">=</span> <span class="n">isComplete</span> <span class="p">?</span> <span class="kt">UIColor</span><span class="o">.</span><span class="n">green</span><span class="o">.</span><span class="nv">cgColor</span> <span class="p">:</span> <span class="kc">nil</span>
<span class="c1">// elided</span>
<span class="n">lines</span><span class="o">.</span><span class="n">forEach</span> <span class="p">{</span>
<span class="nf">drawLine</span><span class="p">(</span><span class="nv">context</span><span class="p">:</span> <span class="n">context</span><span class="p">,</span> <span class="nv">line</span><span class="p">:</span> <span class="nv">$0</span><span class="p">,</span> <span class="nv">overrideColor</span><span class="p">:</span> <span class="n">overrideColor</span><span class="p">)</span>
<span class="p">}</span>
<span class="c1">// elided</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a href="https://github.com/jamiely/tracer/tree/green-lines">refs/tags/green-lines</a></p>
<p><img src="/assets/tracer_green_line.png" alt="" /></p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/aUFJw1ZBq5I" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>Although we don’t need to implement smooth curves to make this usable, it would
be nice to support multiple paths, in order to support disconnected paths. We
will do that by taking a list of expected paths instead of just a single path.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="k">var</span> <span class="nv">_expectedPaths</span><span class="p">:</span> <span class="kt">Array</span><span class="o"><</span><span class="kt">Path</span><span class="o">></span> <span class="o">=</span> <span class="p">[]</span>
<span class="kd">private</span> <span class="k">var</span> <span class="nv">_expectedPathsWithWaypoints</span><span class="p">:</span> <span class="kt">Array</span><span class="o"><</span><span class="kt">Path</span><span class="o">></span> <span class="o">=</span> <span class="p">[]</span>
<span class="kd">private</span> <span class="k">var</span> <span class="nv">pendingPoints</span><span class="p">:</span> <span class="kt">Array</span><span class="o"><</span><span class="kt">CGPoint</span><span class="o">></span> <span class="o">=</span> <span class="p">[]</span>
<span class="kd">private</span> <span class="k">var</span> <span class="nv">isComplete</span><span class="p">:</span> <span class="kt">Bool</span> <span class="p">{</span> <span class="k">return</span> <span class="n">pendingPoints</span><span class="o">.</span><span class="n">isEmpty</span> <span class="p">}</span>
<span class="c1">// elided</span>
<span class="kd">public</span> <span class="k">var</span> <span class="nv">expectedPaths</span><span class="p">:</span> <span class="kt">Array</span><span class="o"><</span><span class="kt">Path</span><span class="o">></span> <span class="p">{</span>
<span class="k">set</span> <span class="p">{</span>
<span class="n">_expectedPaths</span> <span class="o">=</span> <span class="n">newValue</span>
<span class="n">_expectedPathsWithWaypoints</span> <span class="o">=</span> <span class="c1">// elided</span>
<span class="n">pendingPoints</span> <span class="o">=</span> <span class="c1">// the points of all the paths with waypoints</span>
<span class="c1">// elided</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kd">public</span> <span class="k">override</span> <span class="kd">func</span> <span class="nf">touchesMoved</span><span class="p">(</span><span class="n">_</span> <span class="nv">touches</span><span class="p">:</span> <span class="kt">Set</span><span class="o"><</span><span class="kt">UITouch</span><span class="o">></span><span class="p">,</span> <span class="n">with</span> <span class="nv">event</span><span class="p">:</span> <span class="kt">UIEvent</span><span class="p">?)</span> <span class="p">{</span>
<span class="n">touches</span><span class="o">.</span><span class="n">forEach</span> <span class="p">{</span>
<span class="c1">// elided</span>
<span class="nf">removeFromPending</span><span class="p">(</span><span class="nv">pt</span><span class="p">:</span> <span class="n">start</span><span class="p">)</span>
<span class="nf">removeFromPending</span><span class="p">(</span><span class="nv">pt</span><span class="p">:</span> <span class="n">end</span><span class="p">)</span>
<span class="p">}</span>
<span class="c1">// elided</span>
<span class="p">}</span>
<span class="kd">private</span> <span class="kd">func</span> <span class="nf">removeFromPending</span><span class="p">(</span><span class="nv">pt</span><span class="p">:</span> <span class="kt">CGPoint</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// elided, remove pending point if passed point is close to it</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a href="https://github.com/jamiely/tracer/tree/multiple-paths">refs/tags/multiple-paths</a></p>
<p>This is all working pretty well. One last visual thing we’ll implement to
make things look better is we’ll draw the expected path with a wide stroke
twice the error threshold. This will add a nice visual indicator of where
we are allowed to make mistakes.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">context</span><span class="o">.</span><span class="nf">setLineWidth</span><span class="p">(</span><span class="n">maxDistance</span> <span class="o">*</span> <span class="mi">2</span><span class="p">)</span>
<span class="n">context</span><span class="o">.</span><span class="nf">setLineCap</span><span class="p">(</span><span class="o">.</span><span class="n">round</span><span class="p">)</span>
<span class="n">context</span><span class="o">.</span><span class="nf">setStrokeColor</span><span class="p">(</span><span class="kt">UIColor</span><span class="o">.</span><span class="n">gray</span><span class="o">.</span><span class="n">cgColor</span><span class="p">)</span>
</code></pre></div></div>
<p><a href="https://github.com/jamiely/tracer/tree/wide-guide">refs/tags/wide-guide</a></p>
<p>We’ll also add stars at key points in the path that show the points we need to
pass through. We accept some image to be the image we display at each path
point. When given the image, we create <code class="highlighter-rouge">UIImageView</code> subviews placed at each
point, sized using the <code class="highlighter-rouge">maxDistance</code> variable and centered at the point. If the
image is <code class="highlighter-rouge">nil</code>‘ed, we remove each <code class="highlighter-rouge">UIImageView</code> from the view and empty the array
of references.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">var</span> <span class="nv">keyPointImage</span><span class="p">:</span> <span class="kt">UIImage</span><span class="p">?</span> <span class="p">{</span>
<span class="k">get</span> <span class="p">{</span> <span class="k">return</span> <span class="n">_keyPointImage</span> <span class="p">}</span>
<span class="k">set</span> <span class="p">{</span>
<span class="n">_keyPointImage</span> <span class="o">=</span> <span class="n">newValue</span>
<span class="k">guard</span> <span class="k">let</span> <span class="nv">image</span> <span class="o">=</span> <span class="n">newValue</span> <span class="k">else</span> <span class="p">{</span>
<span class="n">imageViews</span><span class="o">.</span><span class="n">forEach</span><span class="p">{</span><span class="nv">$0</span><span class="o">.</span><span class="nf">removeFromSuperview</span><span class="p">()}</span>
<span class="n">imageViews</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="k">let</span> <span class="nv">imageSize</span> <span class="o">=</span> <span class="kt">CGSize</span><span class="p">(</span><span class="nv">width</span><span class="p">:</span> <span class="n">maxDistance</span><span class="p">,</span> <span class="nv">height</span><span class="p">:</span> <span class="n">maxDistance</span><span class="p">)</span>
<span class="k">let</span> <span class="nv">offset</span> <span class="o">=</span> <span class="n">maxDistance</span> <span class="o">/</span> <span class="mf">2.0</span>
<span class="n">_expectedPaths</span><span class="o">.</span><span class="n">compactMap</span><span class="p">{</span><span class="nv">$0</span><span class="o">.</span><span class="n">points</span><span class="p">}</span><span class="o">.</span><span class="nf">joined</span><span class="p">()</span><span class="o">.</span><span class="n">forEach</span> <span class="p">{</span> <span class="n">pt</span> <span class="k">in</span>
<span class="k">let</span> <span class="nv">imageView</span> <span class="o">=</span> <span class="kt">UIImageView</span><span class="p">(</span><span class="nv">image</span><span class="p">:</span> <span class="n">image</span><span class="p">)</span>
<span class="nf">addSubview</span><span class="p">(</span><span class="n">imageView</span><span class="p">)</span>
<span class="k">let</span> <span class="nv">offsetPt</span> <span class="o">=</span> <span class="kt">CGPoint</span><span class="p">(</span><span class="nv">x</span><span class="p">:</span> <span class="n">pt</span><span class="o">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">offset</span><span class="p">,</span> <span class="nv">y</span><span class="p">:</span> <span class="n">pt</span><span class="o">.</span><span class="n">y</span> <span class="o">-</span> <span class="n">offset</span><span class="p">)</span>
<span class="n">imageView</span><span class="o">.</span><span class="n">frame</span> <span class="o">=</span> <span class="kt">CGRect</span><span class="p">(</span><span class="nv">origin</span><span class="p">:</span> <span class="n">offsetPt</span><span class="p">,</span> <span class="nv">size</span><span class="p">:</span> <span class="n">imageSize</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a href="https://github.com/jamiely/tracer/tree/star-points">refs/tags/star-points</a></p>
<p><img src="/assets/tracer_green_line_wide.png" alt="" /></p>
<h1 id="unit-testing">Unit Testing</h1>
<p>We have created something quite usable and the code is clean enough that it
can be augmented; however, we did a bunch of refactoring in the last step
and things are more messy than they were. Since code tends towards entropy,
we want to make sure what we’ve done is protected against future mistakes by
creating some tests. Although most of what we’ve done is visual, and uses
built-in parts of Cocoa Touch, we can cover it with tests by isolating the
changes we make and testing them. Let’s start first with the code that we used
to place the stars. The code is mostly straightforward, but we can certainly
test that given a point and <code class="highlighter-rouge">maxDistance</code>, we return the expected frame
(<code class="highlighter-rouge">CGRect</code>).</p>
<p>We separate the code above that creates <code class="highlighter-rouge">UIImageView</code>s and sets their frame
into two separate parts.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="nv">frames</span> <span class="o">=</span> <span class="n">_expectedPaths</span><span class="o">.</span><span class="n">compactMap</span><span class="p">{</span><span class="nv">$0</span><span class="o">.</span><span class="n">points</span><span class="p">}</span><span class="o">.</span><span class="nf">joined</span><span class="p">()</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span> <span class="n">pt</span> <span class="o">-></span> <span class="kt">CGRect</span> <span class="k">in</span>
<span class="k">let</span> <span class="nv">offsetPt</span> <span class="o">=</span> <span class="kt">CGPoint</span><span class="p">(</span><span class="nv">x</span><span class="p">:</span> <span class="n">pt</span><span class="o">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">offset</span><span class="p">,</span> <span class="nv">y</span><span class="p">:</span> <span class="n">pt</span><span class="o">.</span><span class="n">y</span> <span class="o">-</span> <span class="n">offset</span><span class="p">)</span>
<span class="k">return</span> <span class="kt">CGRect</span><span class="p">(</span><span class="nv">origin</span><span class="p">:</span> <span class="n">offsetPt</span><span class="p">,</span> <span class="nv">size</span><span class="p">:</span> <span class="n">imageSize</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">frames</span><span class="o">.</span><span class="n">forEach</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">imageView</span> <span class="o">=</span> <span class="kt">UIImageView</span><span class="p">(</span><span class="nv">image</span><span class="p">:</span> <span class="n">image</span><span class="p">)</span>
<span class="n">imageView</span><span class="o">.</span><span class="n">frame</span> <span class="o">=</span> <span class="nv">$0</span>
<span class="nf">addSubview</span><span class="p">(</span><span class="n">imageView</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Now, we can easily extract a method that calculates a frame from a point and
max distance. We make it <code class="highlighter-rouge">public static</code> so that we can call it from the test without
instantiating the <code class="highlighter-rouge">TracerView</code>.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// elided code</span>
<span class="k">let</span> <span class="nv">frames</span> <span class="o">=</span> <span class="n">_expectedPaths</span><span class="o">.</span><span class="n">compactMap</span><span class="p">{</span><span class="nv">$0</span><span class="o">.</span><span class="n">points</span><span class="p">}</span><span class="o">.</span><span class="nf">joined</span><span class="p">()</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span>
<span class="kt">TraceView</span><span class="o">.</span><span class="nf">getFrameFrom</span><span class="p">(</span><span class="nv">maxDistance</span><span class="p">:</span> <span class="n">maxDistance</span><span class="p">,</span> <span class="nv">andPt</span><span class="p">:</span> <span class="nv">$0</span><span class="p">)</span>
<span class="p">}</span>
<span class="c1">// elided code</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">func</span> <span class="nf">getFrameFrom</span><span class="p">(</span><span class="nv">maxDistance</span><span class="p">:</span> <span class="kt">CGFloat</span><span class="p">,</span> <span class="n">andPt</span> <span class="nv">pt</span><span class="p">:</span> <span class="kt">CGPoint</span><span class="p">)</span> <span class="o">-></span> <span class="kt">CGRect</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">imageSize</span> <span class="o">=</span> <span class="kt">CGSize</span><span class="p">(</span><span class="nv">width</span><span class="p">:</span> <span class="n">maxDistance</span><span class="p">,</span> <span class="nv">height</span><span class="p">:</span> <span class="n">maxDistance</span><span class="p">)</span>
<span class="k">let</span> <span class="nv">offset</span> <span class="o">=</span> <span class="n">maxDistance</span> <span class="o">/</span> <span class="mf">2.0</span>
<span class="k">let</span> <span class="nv">offsetPt</span> <span class="o">=</span> <span class="kt">CGPoint</span><span class="p">(</span><span class="nv">x</span><span class="p">:</span> <span class="n">pt</span><span class="o">.</span><span class="n">x</span> <span class="o">-</span> <span class="n">offset</span><span class="p">,</span> <span class="nv">y</span><span class="p">:</span> <span class="n">pt</span><span class="o">.</span><span class="n">y</span> <span class="o">-</span> <span class="n">offset</span><span class="p">)</span>
<span class="k">return</span> <span class="kt">CGRect</span><span class="p">(</span><span class="nv">origin</span><span class="p">:</span> <span class="n">offsetPt</span><span class="p">,</span> <span class="nv">size</span><span class="p">:</span> <span class="n">imageSize</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Then we can add our first test:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="kt">TracerTests</span><span class="p">:</span> <span class="kt">XCTestCase</span> <span class="p">{</span>
<span class="kd">func</span> <span class="nf">testGetFrame</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">rect</span> <span class="o">=</span> <span class="kt">TraceView</span><span class="o">.</span><span class="nf">getFrameFrom</span><span class="p">(</span>
<span class="nv">maxDistance</span><span class="p">:</span> <span class="mi">10</span><span class="p">,</span> <span class="nv">andPt</span><span class="p">:</span> <span class="kt">CGPoint</span><span class="p">(</span><span class="nv">x</span><span class="p">:</span> <span class="mi">100</span><span class="p">,</span> <span class="nv">y</span><span class="p">:</span> <span class="mi">300</span><span class="p">))</span>
<span class="kt">XCTAssertEqual</span><span class="p">(</span><span class="n">rect</span><span class="p">,</span> <span class="kt">CGRect</span><span class="p">(</span><span class="nv">x</span><span class="p">:</span> <span class="mi">95</span><span class="p">,</span> <span class="nv">y</span><span class="p">:</span> <span class="mi">295</span><span class="p">,</span> <span class="nv">width</span><span class="p">:</span> <span class="mi">10</span><span class="p">,</span> <span class="nv">height</span><span class="p">:</span> <span class="mi">10</span><span class="p">),</span>
<span class="s">"The rects should be equal"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p><code class="highlighter-rouge">calculateWayPoints</code> is even easier to test because it doesn’t make use of
any member variables. We can just declare it and all of the methods it calls
as static, then create a test for it.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">testCalculateWaypoints</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">points</span> <span class="o">=</span> <span class="kt">TraceView</span><span class="o">.</span><span class="nf">calculateWaypoints</span><span class="p">(</span><span class="nv">maxDistance</span><span class="p">:</span> <span class="mi">10</span><span class="p">,</span>
<span class="nv">start</span><span class="p">:</span> <span class="kt">CGPoint</span><span class="p">(</span><span class="nv">x</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="nv">y</span><span class="p">:</span> <span class="mi">0</span><span class="p">),</span> <span class="nv">end</span><span class="p">:</span> <span class="kt">CGPoint</span><span class="p">(</span><span class="nv">x</span><span class="p">:</span> <span class="mi">20</span><span class="p">,</span> <span class="nv">y</span><span class="p">:</span> <span class="mi">20</span><span class="p">))</span>
<span class="kt">XCTAssertEqual</span><span class="p">(</span><span class="n">points</span><span class="p">,</span> <span class="p">[</span>
<span class="kt">CGPoint</span><span class="p">(</span><span class="nv">x</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="nv">y</span><span class="p">:</span> <span class="mi">5</span><span class="p">),</span>
<span class="kt">CGPoint</span><span class="p">(</span><span class="nv">x</span><span class="p">:</span> <span class="mi">10</span><span class="p">,</span> <span class="nv">y</span><span class="p">:</span> <span class="mi">10</span><span class="p">),</span>
<span class="kt">CGPoint</span><span class="p">(</span><span class="nv">x</span><span class="p">:</span> <span class="mi">15</span><span class="p">,</span> <span class="nv">y</span><span class="p">:</span> <span class="mi">15</span><span class="p">)</span>
<span class="p">],</span> <span class="s">"waypoints should be the same"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Since I know that the method works, an easy way to implement the tests is to
just run the method with some logical input, then write a failing
<code class="highlighter-rouge">XCTAssertEqual</code>. The console will note the actual value, and I can just add
that into the test if it makes sense. This way of testing is not bulletproof.
There are edge cases that we will miss, but at least we can guard against
regressions in the future.</p>
<p><a href="https://github.com/jamiely/tracer/tree/unit-tests">refs/tags/unit-tests</a></p>
<h1 id="dependency-management">Dependency Management</h1>
<p>One of the best ways to test edge cases is using property-based testing, which
relies on the testing system to generate test cases. We can do just that with by using a
<a href="https://en.wikipedia.org/wiki/QuickCheck">QuickCheck-like</a> library. In order
to import it, we will setup our project with
<a href="https://cocoapods.org/">Cocoapods</a> for dependency management. See the notes
for more about Cocoapods and some of the tools that follow{3}.</p>
<p>We make sure <a href="https://bundler.io/">bundler</a>, the Ruby package manager, is
installed using the <code class="highlighter-rouge">gem</code> command{2}.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>gem <span class="nb">install </span>bundler
</code></pre></div></div>
<p>We create a <code class="highlighter-rouge">Gemfile</code> in order to install the <code class="highlighter-rouge">cocoapods</code> gem.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Gemfile</span>
<span class="n">source</span> <span class="s2">"https://rubygems.org"</span>
<span class="n">gem</span> <span class="s1">'cocoapods'</span>
</code></pre></div></div>
<p>Then we install the <code class="highlighter-rouge">cocoapods</code> gem using Bundler, and create a new <code class="highlighter-rouge">Podfile</code>,
where we can define our iOS dependencies.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle <span class="nb">install
</span>bundle <span class="nb">exec </span>pod init
</code></pre></div></div>
<p>We add <code class="highlighter-rouge">SwiftCheck</code> to the tests target block of our new <code class="highlighter-rouge">Podfile</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Podfile</span>
<span class="c1"># elided</span>
<span class="n">target</span> <span class="s1">'TracerTests'</span> <span class="k">do</span>
<span class="n">inherit!</span> <span class="ss">:search_paths</span>
<span class="n">pod</span> <span class="s1">'SwiftCheck'</span>
<span class="k">end</span>
<span class="c1"># elided</span>
</code></pre></div></div>
<p>Then run <code class="highlighter-rouge">bundle exec pod install</code> to install the <code class="highlighter-rouge">SwiftCheck</code> dependency. Upon
executing that command, Cocoapods will create an <code class="highlighter-rouge">xcproject</code> to build the
dependencies into a framework that is included in your original project. It
packages this project along with our original project in a new workspace file
called <code class="highlighter-rouge">Tracer.xcworkspace</code>. From now on, we’ll use the workspace to build our
project since we’re relying on the Cocoapods-managed dependencies.</p>
<p><a href="https://github.com/jamiely/tracer/tree/cocoapod-init">refs/tags/cococapod-init</a></p>
<h1 id="property-based-testing">Property-based Testing</h1>
<p><a href="https://github.com/typelift/SwiftCheck"><code class="highlighter-rouge">SwiftCheck</code></a> is a QuickCheck-like
library for Swift. We will write a property-based test that will test the
following:</p>
<ul>
<li>For all max distances at least 1, and</li>
<li>Starting points with positive x and y, and</li>
<li>Ending points with positive x and y</li>
<li>Generate waypoints that are less than the max distance apart</li>
</ul>
<p><code class="highlighter-rouge">SwiftCheck</code> is smart enough to test edge cases, like what happens when the
max distance is 1 or the maximum <code class="highlighter-rouge">CGFloat</code>, or when both the starting and
ending point are (0, 0). Here’s how we will define the test:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">testWaypointDistanceProperty</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">property</span><span class="p">(</span><span class="s">"a description"</span><span class="p">)</span>
<span class="c1">// for all combinations of max distance and start and end points</span>
<span class="o"><-</span> <span class="n">forAll</span> <span class="p">{</span> <span class="p">(</span><span class="n">maxDistance</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">)</span> <span class="k">in</span>
<span class="k">let</span> <span class="nv">waypoints</span> <span class="o">=</span> <span class="c1">// calculate waypoints</span>
<span class="k">var</span> <span class="nv">previous</span> <span class="o">=</span> <span class="n">start</span>
<span class="c1">// starting with the starting point and the first waypoint,</span>
<span class="c1">// confirm that the distance between each neighboring point is</span>
<span class="c1">// less than the max distance</span>
<span class="nf">return</span> <span class="p">(</span><span class="n">waypoints</span> <span class="o">+</span> <span class="p">[</span><span class="n">end</span><span class="p">])</span><span class="o">.</span><span class="n">allSatisfy</span> <span class="p">{</span> <span class="n">current</span> <span class="k">in</span>
<span class="k">let</span> <span class="nv">distance</span> <span class="o">=</span> <span class="kt">TraceView</span><span class="o">.</span><span class="nf">getDistance</span><span class="p">(</span><span class="nv">start</span><span class="p">:</span> <span class="n">previous</span><span class="p">,</span> <span class="nv">end</span><span class="p">:</span> <span class="n">current</span><span class="p">)</span>
<span class="n">previous</span> <span class="o">=</span> <span class="n">current</span>
<span class="k">return</span> <span class="n">distance</span> <span class="o"><=</span> <span class="n">maxDistance</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In order for this test to run, we have to tell <code class="highlighter-rouge">SwiftCheck</code> how to generate
the required data types <code class="highlighter-rouge">CGFloat</code> and <code class="highlighter-rouge">CGPoint</code>. One way to generate an arbitrary
<code class="highlighter-rouge">CGFloat</code> is to map over the existing <code class="highlighter-rouge">SwiftCheck</code>-provided generator for <code class="highlighter-rouge">Double</code>.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">extension</span> <span class="kt">CGFloat</span><span class="p">:</span> <span class="kt">Arbitrary</span> <span class="p">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="k">var</span> <span class="nv">arbitrary</span> <span class="p">:</span> <span class="kt">Gen</span><span class="o"><</span><span class="kt">CGFloat</span><span class="o">></span> <span class="p">{</span>
<span class="k">return</span> <span class="kt">Double</span><span class="o">.</span><span class="n">arbitrary</span><span class="o">.</span><span class="n">map</span><span class="p">{</span><span class="kt">CGFloat</span><span class="p">(</span><span class="nv">$0</span><span class="p">)}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>But note that we want max distance to always be at least 1. Although we could
embed this into the <code class="highlighter-rouge">Arbitrary</code>{4} implementation we have added, we could not use
it as an arbitrary for <code class="highlighter-rouge">CGFloat</code>s in general. Instead, we apply a predicate to
the <code class="highlighter-rouge">Gen</code>erator{5} we just defined and let-bind the result.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="nv">floatsGTE1</span> <span class="o">=</span> <span class="kt">CGFloat</span><span class="o">.</span><span class="n">arbitrary</span><span class="o">.</span><span class="n">suchThat</span> <span class="p">{</span> <span class="nv">$0</span> <span class="o">>=</span> <span class="mi">1</span> <span class="p">}</span>
</code></pre></div></div>
<p>We’ll also need only positive <code class="highlighter-rouge">CGFloat</code> instances to create the <code class="highlighter-rouge">CGPoint</code>s we
need for the test.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="nv">positiveFloats</span> <span class="o">=</span> <span class="kt">CGFloat</span><span class="o">.</span><span class="n">arbitrary</span><span class="o">.</span><span class="n">suchThat</span> <span class="p">{</span> <span class="nv">$0</span> <span class="o">>=</span> <span class="mi">0</span> <span class="p">}</span>
</code></pre></div></div>
<p>We’ll let-bind a <code class="highlighter-rouge">CGPoint</code> generator by zipping two <code class="highlighter-rouge">positiveFloats</code> and
applying the result to the <code class="highlighter-rouge">CGPoint</code> constructor.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="nv">positivePoints</span> <span class="o">=</span> <span class="kt">Gen</span><span class="o"><</span><span class="p">(</span><span class="kt">CGFloat</span><span class="p">,</span> <span class="kt">CGFloat</span><span class="p">)</span><span class="o">>.</span><span class="nf">zip</span><span class="p">(</span>
<span class="n">positiveFloats</span><span class="p">,</span> <span class="n">positiveFloats</span><span class="p">)</span><span class="o">.</span><span class="nf">map</span><span class="p">(</span><span class="kt">CGPoint</span><span class="o">.</span><span class="kd">init</span><span class="p">)</span>
</code></pre></div></div>
<p>And finally, although there are type signatures that allow passing <code class="highlighter-rouge">Gen</code>
instances to <code class="highlighter-rouge">forAll</code>, I couldn’t get the type signatures to work out. Intead,
in order to enforce using the exact <code class="highlighter-rouge">Gen</code> instances I need, I <a href="https://refactoring.com/catalog/introduceParameterObject.html">introduced a
parameter
object</a>.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">struct</span> <span class="kt">LineAndDistance</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">maxDistance</span><span class="p">:</span> <span class="kt">CGFloat</span>
<span class="k">let</span> <span class="nv">start</span><span class="p">:</span> <span class="kt">CGPoint</span>
<span class="k">let</span> <span class="nv">end</span><span class="p">:</span> <span class="kt">CGPoint</span>
<span class="p">}</span>
</code></pre></div></div>
<p>and defined its <code class="highlighter-rouge">Arbitrary</code> implementation by zipping the generators above,
and mapping the result using the <code class="highlighter-rouge">LineAndDistance</code> default constructor.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">extension</span> <span class="kt">LineAndDistance</span> <span class="p">:</span> <span class="kt">Arbitrary</span> <span class="p">{</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="k">var</span> <span class="nv">arbitrary</span> <span class="p">:</span> <span class="kt">Gen</span><span class="o"><</span><span class="kt">LineAndDistance</span><span class="o">></span> <span class="p">{</span>
<span class="k">return</span> <span class="kt">Gen</span><span class="o"><</span><span class="p">(</span><span class="kt">CGFloat</span><span class="p">,</span> <span class="kt">CGPoint</span><span class="p">,</span> <span class="kt">CGPoint</span><span class="p">)</span><span class="o">>.</span><span class="nf">zip</span><span class="p">(</span>
<span class="n">floatsGTE1</span><span class="p">,</span>
<span class="n">positivePoints</span><span class="p">,</span>
<span class="n">positivePoints</span><span class="p">)</span><span class="o">.</span><span class="nf">map</span><span class="p">(</span><span class="kt">LineAndDistance</span><span class="o">.</span><span class="kd">init</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The final test code becomes:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">testWaypointDistanceProperty</span><span class="p">()</span> <span class="p">{</span>
<span class="k">var</span> <span class="nv">counter</span> <span class="o">=</span> <span class="mi">0</span>
<span class="nf">property</span><span class="p">(</span><span class="s">"waypoints are created so that points are not more than max distance apart"</span><span class="p">)</span>
<span class="o"><-</span> <span class="n">forAll</span> <span class="p">{</span> <span class="p">(</span><span class="nv">args</span><span class="p">:</span> <span class="kt">LineAndDistance</span><span class="p">)</span> <span class="k">in</span>
<span class="k">let</span> <span class="nv">maxDistance</span> <span class="o">=</span> <span class="n">args</span><span class="o">.</span><span class="n">maxDistance</span>
<span class="k">let</span> <span class="nv">start</span> <span class="o">=</span> <span class="n">args</span><span class="o">.</span><span class="n">start</span>
<span class="k">let</span> <span class="nv">end</span> <span class="o">=</span> <span class="n">args</span><span class="o">.</span><span class="n">end</span>
<span class="k">let</span> <span class="nv">waypoints</span> <span class="o">=</span> <span class="kt">TraceView</span><span class="o">.</span><span class="nf">calculateWaypoints</span><span class="p">(</span>
<span class="nv">maxDistance</span><span class="p">:</span> <span class="n">maxDistance</span><span class="p">,</span> <span class="nv">start</span><span class="p">:</span> <span class="n">start</span><span class="p">,</span> <span class="nv">end</span><span class="p">:</span> <span class="n">end</span><span class="p">)</span>
<span class="k">var</span> <span class="nv">previous</span> <span class="o">=</span> <span class="n">start</span>
<span class="nf">return</span> <span class="p">(</span><span class="n">waypoints</span> <span class="o">+</span> <span class="p">[</span><span class="n">end</span><span class="p">])</span><span class="o">.</span><span class="n">allSatisfy</span> <span class="p">{</span> <span class="n">current</span> <span class="k">in</span>
<span class="k">let</span> <span class="nv">distance</span> <span class="o">=</span> <span class="kt">TraceView</span><span class="o">.</span><span class="nf">getDistance</span><span class="p">(</span><span class="nv">start</span><span class="p">:</span> <span class="n">previous</span><span class="p">,</span> <span class="nv">end</span><span class="p">:</span> <span class="n">current</span><span class="p">)</span>
<span class="n">previous</span> <span class="o">=</span> <span class="n">current</span>
<span class="k">return</span> <span class="n">distance</span> <span class="o"><=</span> <span class="n">maxDistance</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p><a href="https://github.com/jamiely/tracer/tree/property-based-test">refs/tags/property-based-test</a></p>
<h1 id="adding-fastlane">Adding Fastlane</h1>
<p>It’s great that we can have a little more confidence in our code with these
new tests. We want to be able to run our test via command line so that we can
implement a <a href="https://codefresh.io/continuous-integration/continuous-integration-delivery-pipeline-important/">CI
pipeline</a>,
and the command for that is:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>xcodebuild -workspace Tracer.xcworkspace \
-scheme Tracer \
-sdk iphonesimulator \
-destination 'platform=iOS Simulator,name=iPhone XR' \
test
</code></pre></div></div>
<p>There are two small problems with this.</p>
<ol>
<li>It gives us messy build output like {1}</li>
<li>We have to repeat those arguments whenever we want to test</li>
</ol>
<p>In order to address these issues, we’ll use <a href="https://docs.fastlane.tools/actions/scan/"><code class="highlighter-rouge">fastlane scan</code></a>
to do the testing. First we’ll add the fastlane gem to our <code class="highlighter-rouge">Gemfile</code>, then
<code class="highlighter-rouge">bundle install</code>, then <code class="highlighter-rouge">bundle exec fastlane scan</code>. The command will figure out
how we want to run the tests, and if it doesn’t use the proper defaults,
we can save the settings in a file. This also results in more nicely
formatted output.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[07:14:02]: ▸ 2019-03-21 07:14:02.528 xcodebuild[56939:3115294] IDETestOperationsObserverDebug: Writing diagnostic log for test session
to:
[07:14:02]: ▸ /Users/jly/Library/Developer/Xcode/DerivedData/Tracer-bxjzmxdcrauuvpherayrwqtaedvd/Logs/Test/Run-Tracer-2019.03.21_07-13-18
--0400.xcresult/2_Test/Diagnostics/TracerTests-238BE08F-B889-4811-9774-DD6E49600588/TracerTests-13341B9C-FF27-40F8-92CB-FEEB663A6721/Sess
ion-TracerTests-2019-03-21_071402-wfPSuG.log
[07:14:02]: ▸ 2019-03-21 07:14:02.528 xcodebuild[56939:3112940] [MT] IDETestOperationsObserverDebug: (A66F349B-071E-4093-A9F3-D4451530DE8
1) Beginning test session TracerTests-A66F349B-071E-4093-A9F3-D4451530DE81 at 2019-03-21 07:14:02.529 with Xcode 10B61 on target <DVTiPho
neSimulator: 0x7fbf1b941240> {
[07:14:02]: ▸ SimDevice: iPhone 5s (CCAC9692-E629-4EDE-815C-9D99DEAA8E96, iOS 12.1, Booted)
[07:14:02]: ▸ } (12.1 (16B91))
[07:14:06]: ▸ All tests
[07:14:06]: ▸ Test Suite TracerTests.xctest started
[07:14:06]: ▸ TracerTests
[07:14:06]: ▸ ✓ testCalculateWaypoints (0.002 seconds)
[07:14:06]: ▸ ✓ testGetFrame (0.000 seconds)
[07:14:06]: ▸ ✓ testWaypointDistanceProperty (0.035 seconds)
[07:14:06]: ▸ Executed 3 tests, with 0 failures (0 unexpected) in 0.038 (0.041) seconds
</code></pre></div></div>
<p><a href="https://github.com/jamiely/tracer/tree/fastlane">refs/tags/fastlane</a></p>
<h1 id="continuous-integration-ci">Continuous Integration (CI)</h1>
<p>Now that we have a relatively simple way of running tests, we can setup CI
for our project. CI is useful in making sure that our code is always working,
by running tests each time we check in our code.</p>
<p>If you have your own hardware, such as a spare Mac lying around, you can
setup something opensource such as <a href="https://jenkins.io/">Jenkins</a> or
<a href="https://about.gitlab.com/">Gitlab</a> to kick off the build, setting up the
spare Mac as a job runner. If you’d rather not go through the trouble of
setting up your own CI pipeline, there are a number of service providers that
offer CI services and macOS build agents. We’ll try to add our repository to
two different CI services.</p>
<p>First, <a href="http://circleci.com/">CircleCI</a> is one of the most popular hosted CI
providers right now. It works via a declarative yaml file that has various
shortcuts that make it simple to read and build projects, but also allows
you to run arbitrary commands. They offer a free trial to get started. We can
add support for CircleCI by adding the following yaml file to our project.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># .circleci/config.yml</span>
<span class="na">version</span><span class="pi">:</span> <span class="s">2</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">build-and-test</span><span class="pi">:</span>
<span class="na">macos</span><span class="pi">:</span>
<span class="na">xcode</span><span class="pi">:</span> <span class="s2">"</span><span class="s">10.1.0"</span>
<span class="na">shell</span><span class="pi">:</span> <span class="s">/bin/bash --login -o pipefail</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">checkout</span>
<span class="pi">-</span> <span class="na">run</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">Install required Ruby gems</span>
<span class="na">command</span><span class="pi">:</span> <span class="s">bundle install</span>
<span class="pi">-</span> <span class="na">run</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">Install CocoaPods</span>
<span class="na">command</span><span class="pi">:</span> <span class="s">bundle exec fastlane run cocoapods</span>
<span class="pi">-</span> <span class="na">run</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">Build and run tests</span>
<span class="na">command</span><span class="pi">:</span> <span class="s">bundle exec fastlane scan</span>
<span class="na">workflows</span><span class="pi">:</span>
<span class="na">version</span><span class="pi">:</span> <span class="s">2</span>
<span class="na">build-and-test</span><span class="pi">:</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">build-and-test</span>
</code></pre></div></div>
<p>Once we have added and committed this file to our repository, and linked our
repository with CircleCI via their web interface, it will start to build. The
output looks like this:</p>
<p><img src="/assets/tracer_circleci.png" alt="CircleCI successful build" /></p>
<p>CircleCI will build the project whenever a commit is pushed to master.</p>
<p><a href="https://github.com/jamiely/tracer/tree/circle-ci">refs/tags/circle-ci</a></p>
<p>If you have trouble getting on CircleCI’s free plan like I did,
<a href="https://azure.microsoft.com/en-us/services/devops/">Azure DevOps</a> is
another option that supports free macOS pipelines and is easy to get started with.</p>
<p>We can get started by defining a yaml file in our project containing
a definition of the image we will use and the build steps. Azure DevOps allows us
to choose which yaml file defines our build, so the name doesn’t matter.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># .azure-pipelines/main.yml</span>
<span class="na">pool</span><span class="pi">:</span>
<span class="na">vmImage</span><span class="pi">:</span> <span class="s1">'</span><span class="s">macOS-10.13'</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">CocoaPods@0</span>
<span class="na">displayName</span><span class="pi">:</span> <span class="s1">'</span><span class="s">pod</span><span class="nv"> </span><span class="s">install</span><span class="nv"> </span><span class="s">using</span><span class="nv"> </span><span class="s">the</span><span class="nv"> </span><span class="s">CocoaPods</span><span class="nv"> </span><span class="s">task</span><span class="nv"> </span><span class="s">with</span><span class="nv"> </span><span class="s">defaults'</span>
<span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">ShellScript@2</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">scriptPath</span><span class="pi">:</span> <span class="s">.azure-pipelines/test</span>
</code></pre></div></div>
<p>Although Azure DevOps does support a task that will test an XCode project,
and even allow you to <a href="https://docs.microsoft.com/en-us/azure/devops/pipelines/languages/xcode?view=azure-devops#testing-on-azure-hosted-devices">test on real devices</a>,
I didn’t want to make the testing step much different than what we already have.
So, I run a shell script in the pipeline that runs <code class="highlighter-rouge">fastlane scan</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># .azure-pipelines/test</span>
<span class="c"># When this script gets run, the context will be the</span>
<span class="c"># script directory, so we cd into the root directory</span>
<span class="nb">cd</span> ..
bundle update <span class="nt">--bundler</span>
bundle <span class="nb">install</span>
<span class="c"># This step is new. Since there are multiple XCode</span>
<span class="c"># versions on their macOS image, we have to specify</span>
<span class="c"># which one we want to use</span>
<span class="nb">export </span><span class="nv">DEVELOPER_DIR</span><span class="o">=</span><span class="s2">"/Applications/Xcode_10.1.app"</span>
bundle <span class="nb">exec </span>fastlane scan <span class="nt">--output_directory</span> output/scan
</code></pre></div></div>
<p>The pipeline run looks like this:</p>
<p><img src="/assets/tracer_azure_devops.png" alt="Azure DevOps Pipeline View" /></p>
<p><a href="https://github.com/jamiely/tracer/tree/azure-pipelines">refs/tags/azure-pipelines</a></p>
<h1 id="framework-distribution">Framework Distribution</h1>
<p>We just added <code class="highlighter-rouge">Cocoapods</code> in order to add and maintain dependencies
in our project. Wouldn’t it be nice if we could distribute this project as a
Cocoapod so that it was easy for people to use? If we want to do this, we’ll
have to make several changes:</p>
<ul>
<li>Restructure the project to produce a <code class="highlighter-rouge">Cocoa Touch Framework</code> product. This
is a library that can be included into other projects.</li>
<li>Optionally, move the code that shows the view into some sort of “Demo” project.</li>
<li>Create the necessary files to define our <code class="highlighter-rouge">Cocoapod</code> and publish it to the
<code class="highlighter-rouge">Cocoapod</code> repository.</li>
</ul>
<p>First, we will move most of our files into a subdirectory called <code class="highlighter-rouge">Demo</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">-p</span> Demo
git <span class="nb">mv </span>Tracer Tracer.xcodeproj TracerUITests Demo
<span class="nb">cp</span> <span class="nt">-r</span> Gemfile<span class="k">*</span> Podfile<span class="k">*</span> Demo
</code></pre></div></div>
<p>Opening the project from the <code class="highlighter-rouge">Demo</code> folder, we’ll use XCode to rename the
project and each of the targets. We’ll postpone further changes until we have
the framework project working. We will create a new project in the root
directory called Tracer. (To do this we wil create the project in a temporary
directory, then copy the files into the root directory.) We’ll choose “Cocoa
Touch Framework” as the template. We’ll move <code class="highlighter-rouge">TraceView.swift</code> and
<code class="highlighter-rouge">TracerTests</code> from the <code class="highlighter-rouge">Demo</code> folder to source folders in the root directory.
We will add a <code class="highlighter-rouge">Podfile</code> that installs <code class="highlighter-rouge">SwiftCheck</code>, run <code class="highlighter-rouge">bundle exec pod
install</code> in the root directory, and open the <code class="highlighter-rouge">Tracer.xcworkspace</code>. We’ll make
sure that it builds correctly, then add a <code class="highlighter-rouge">Tracer.podspec</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Pod</span><span class="o">::</span><span class="no">Spec</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span> <span class="o">|</span><span class="n">s</span><span class="o">|</span>
<span class="n">s</span><span class="p">.</span><span class="nf">name</span> <span class="o">=</span> <span class="s2">"Tracer"</span>
<span class="n">s</span><span class="p">.</span><span class="nf">version</span> <span class="o">=</span> <span class="s2">"0.1.0"</span>
<span class="c1"># elided</span>
<span class="n">s</span><span class="p">.</span><span class="nf">ios</span><span class="p">.</span><span class="nf">deployment_target</span> <span class="o">=</span> <span class="s2">"11.0"</span>
<span class="n">s</span><span class="p">.</span><span class="nf">source</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">:git</span> <span class="o">=></span> <span class="s2">"https://github.com/jamiely/Tracer.git"</span><span class="p">,</span> <span class="ss">:tag</span> <span class="o">=></span> <span class="s2">"</span><span class="si">#{</span><span class="n">s</span><span class="p">.</span><span class="nf">version</span><span class="si">}</span><span class="s2">"</span><span class="p">}</span>
<span class="n">s</span><span class="p">.</span><span class="nf">source_files</span> <span class="o">=</span> <span class="s2">"Sources/**/*.swift"</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now that we have made a <code class="highlighter-rouge">Tracer</code> pod, we’ll finish modifications to the <code class="highlighter-rouge">Demo</code>
project. We’ll edit the <code class="highlighter-rouge">Podfile</code> to remove unnecessary targets and update the
<code class="highlighter-rouge">Podfile</code> to include the <code class="highlighter-rouge">Tracer</code> dependency. We use the <code class="highlighter-rouge">path</code> argument to
refer to the source files in the root directory.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">platform</span> <span class="ss">:ios</span><span class="p">,</span> <span class="s1">'11.0'</span>
<span class="n">target</span> <span class="s1">'TracerDemo'</span> <span class="k">do</span>
<span class="c1"># Comment the next line if you're not using Swift and don't want to use dynamic frameworks</span>
<span class="n">use_frameworks!</span>
<span class="n">pod</span> <span class="s1">'Tracer'</span><span class="p">,</span> <span class="ss">path: </span><span class="s1">'../'</span>
<span class="k">end</span>
</code></pre></div></div>
<p>We’ll rerun <code class="highlighter-rouge">bundle exec pod install</code> to generate <code class="highlighter-rouge">Demo.xcworkspace</code>. We may
have to delete some Framworks and Products added by the previous Podfile
targets, and modify the location of the <code class="highlighter-rouge">Info.plist</code> file in the Build
Settings. We’ll also have to add an <code class="highlighter-rouge">import Tracer</code> to the
<code class="highlighter-rouge">ViewController.swift</code>. Compiling, we’ll notice some issues we have to resolve
dealing with visibility of the types. We have to add <code class="highlighter-rouge">public</code> to classes,
structs, and some methods to make them visible outside of the framework. We’ll
build and run <code class="highlighter-rouge">Demo.xcworkspace</code> to make sure that everything works.</p>
<p><a href="https://github.com/jamiely/tracer/tree/cocoapod-distribution">refs/tags/cocoapod-distribution</a></p>
<h1 id="next-steps">Next Steps</h1>
<p>There are many places we could continue at this point. Since I don’t have a
specific use case for the project right now, there’s no clear direction for me
to take. One of the very next steps that would probably be needed to make this
more usable is to create some delegate protocol that allows further customization
of this view’s functionality. For example, we could have delegates that can
be used for simple customizations such as:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">protocol</span> <span class="kt">TracerDelegate</span> <span class="p">{</span>
<span class="c1">/// Returns the color that should be used for the passed line</span>
<span class="kd">func</span> <span class="nf">colorForLine</span><span class="p">(</span><span class="nv">line</span><span class="p">:</span> <span class="kt">Line</span><span class="p">)</span> <span class="o">-></span> <span class="kt">UIColor</span>
<span class="c1">/// Controls whether lines should be shown</span>
<span class="kd">func</span> <span class="nf">shouldDrawLine</span><span class="p">(</span><span class="nv">line</span><span class="p">:</span> <span class="kt">Line</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Bool</span>
<span class="p">}</span>
</code></pre></div></div>
<p>or a delegate that can be used to customize everything:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">protocol</span> <span class="kt">TracerDelegate</span> <span class="p">{</span>
<span class="c1">/// Customize how lines are drawn</span>
<span class="kd">func</span> <span class="nf">drawLine</span><span class="p">(</span><span class="nv">context</span><span class="p">:</span> <span class="kt">CGContext</span><span class="p">,</span> <span class="nv">line</span><span class="p">:</span> <span class="kt">Line</span><span class="p">,</span> <span class="nv">overrideColor</span><span class="p">:</span> <span class="kt">CGColor</span><span class="p">?,</span>
<span class="n">forView</span> <span class="nv">view</span><span class="p">:</span> <span class="kt">TracerView</span><span class="p">)</span>
<span class="c1">/// Customize the drawing of expected paths</span>
<span class="kd">func</span> <span class="nf">drawExpectedPaths</span><span class="p">(</span><span class="nv">paths</span><span class="p">:</span> <span class="kt">Array</span><span class="o"><</span><span class="kt">Path</span><span class="o">></span><span class="p">,</span> <span class="n">forView</span> <span class="nv">view</span><span class="p">:</span> <span class="kt">TracerView</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<h1 id="conclusion">Conclusion</h1>
<p>In this article, we started with a concept and implemented that idea piece by
piece into something that works. There’s a lot more we can do with this idea,
so feel free to fork the project. We implemented unit tests, and a
property-based test using <code class="highlighter-rouge">SwiftCheck</code>. We saw how to manage dependencies
using <code class="highlighter-rouge">Cocoapods</code>. We made test runs more straightforward using <code class="highlighter-rouge">fastlane</code>
and setup CI with two hosted solutions. Finally, we saw how to package our
own Framework as a Cocoapod.</p>
<p>The source code for this project is available on
<a href="https://www.github.com/jamiely/tracer">GitHub</a>, and you can add this Cocoapod
to your own project by including this in your <code class="highlighter-rouge">Podfile</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">pod</span> <span class="s1">'Tracer'</span><span class="p">,</span> <span class="ss">:git</span> <span class="o">=></span> <span class="s1">'https://github.com/jamiely/Tracer.git'</span>
</code></pre></div></div>
<h1 id="notes">Notes</h1>
<p>{1}:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/usr/bin/codesign -r- --display /Users/jamie/Library/Developer/Xcode/DerivedData/Tracer-bxjzmxdcrauuvpherayrwqtaedvd/Build/Products/Debug-i
phonesimulator/Tracer.app/PlugIns/TracerTests.xctest/Frameworks/libswiftFoundation.dylib
Code signature of /Users/jamie/Library/Developer/Xcode/DerivedData/Tracer-bxjzmxdcrauuvpherayrwqtaedvd/Build/Products/Debug-iphonesimulator
/Tracer.app/PlugIns/TracerTests.xctest/Frameworks/libswiftDispatch.dylib is unchanged; keeping original
Code signature of /Users/jamie/Library/Developer/Xcode/DerivedData/Tracer-bxjzmxdcrauuvpherayrwqtaedvd/Build/Products/Debug-iphonesimulator
/Tracer.app/PlugIns/TracerTests.xctest/Frameworks/libswiftObjectiveC.dylib is unchanged; keeping original
Code signature of /Users/jamie/Library/Developer/Xcode/DerivedData/Tracer-bxjzmxdcrauuvphera
Object: <IDESchemeActionTestAttachment: 0x7fa5853bc3f0>
Method: -_savePayload:
Thread: <NSThread: 0x7fa58510b4b0>{number = 1, name = main}
Please file a bug at https://bugreport.apple.com with this warning message and any useful information you can provide.
t = 4.93s Tear Down
Test Case '-[TracerUITests.TracerUITests testExample]' passed (5.128 seconds).
Test Suite 'TracerUITests' passed at 2019-03-21 06:56:28.103.
Executed 1 test, with 0 failures (0 unexpected) in 5.128 (5.129) seconds
Test Suite 'TracerUITests.xctest' passed at 2019-03-21 06:56:28.104.
Executed 1 test, with 0 failures (0 unexpected) in 5.128 (5.131) seconds
Test Suite 'All tests' passed at 2019-03-21 06:56:28.105.
Executed 1 test, with 0 failures (0 unexpected) in 5.128 (5.133) seconds
</code></pre></div></div>
<p>{2}: Assuming we are using System ruby, that is, the ruby that comes with an
install of macOS.</p>
<p>{3}:</p>
<h2 id="bundler">Bundler</h2>
<p>We use <code class="highlighter-rouge">bundler</code> to manage ruby dependencies called <code class="highlighter-rouge">gem</code>s. <code class="highlighter-rouge">Cocoapods</code> is a ruby
gem that is itself an iOS dependency manager. So, we use <code class="highlighter-rouge">bundler</code> to make sure
all the contributors to the <code class="highlighter-rouge">Tracer</code> project are using the same version of
<code class="highlighter-rouge">Cocoapods</code>. In practice, I’ve never run into an issue where someone using a
different version of <code class="highlighter-rouge">Cocoapods</code> than I was caused an issue.</p>
<h2 id="cocoapods">Cocoapods</h2>
<p><code class="highlighter-rouge">Cocoapods</code> is one of a few possibilities for managing dependencies. There’s
also manual management, which will work decently until you want to upgrade
libraries or track the version you are using. There’s
<a href="https://github.com/Carthage/Carthage">Carthage</a> which is less invasive than
<code class="highlighter-rouge">Cocoapods</code>. And although it doesn’t seem pratical to use yet (at least for iOS
projects), there is <a href="https://swift.org/package-manager/">Swift Package
Manager</a>.</p>
<p>One thing I like about <code class="highlighter-rouge">Cocoapods</code> is that there is a catalogue of pods that
is searchable. In practice, this often yields out of date and unused pods. I
often just wind up performing a web search for a library I want, and checking
if there is a Cocoapod spec.</p>
<h2 id="fastlane">Fastlane</h2>
<p>Although we just use Fastlane here for testing, it is the swiss army knife of
iOS build automation. Check it out if you do anything iOS build related.</p>
<p>{4}: <a href="https://github.com/typelift/SwiftCheck/blob/master/Sources/SwiftCheck/Arbitrary.swift"><code class="highlighter-rouge">protocol Arbitrary</code></a></p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">/// A type that implements random generation and shrinking of values.</span>
<span class="c1">///</span>
<span class="c1">/// While testing, SwiftCheck will invoke `arbitrary` a given amount of times</span>
<span class="c1">/// (usually 100 if the default settings are used). During that time, the</span>
<span class="c1">/// callee has an opportunity to call through to any data or sources of</span>
<span class="c1">/// randomness it needs to return what it deems an "Arbitrary" value.</span>
<span class="c1">///</span>
<span class="c1">/// Shrinking is reduction in the complexity of a tested value to remove noise</span>
<span class="c1">/// and present a minimal counterexample when a property fails. A shrink</span>
<span class="c1">/// necessitates returning a list of all possible "smaller" values for</span>
<span class="c1">/// SwiftCheck to run through. As long as each individual value in the returned</span>
<span class="c1">/// list is less than or equal to the size of the input value, and is not a</span>
<span class="c1">/// duplicate of the input value, a minimal case should be reached fairly</span>
<span class="c1">/// efficiently. Shrinking is an optional extension of normal testing. If no</span>
<span class="c1">/// implementation of `shrink` is provided, SwiftCheck will default to an empty</span>
<span class="c1">/// one - that is, no shrinking will occur.</span>
<span class="c1">///</span>
<span class="c1">/// As an example, take `Array`'s implementation of shrink:</span>
<span class="c1">///</span>
<span class="c1">/// Arbitrary.shrink([1, 2, 3])</span>
<span class="c1">/// > [[], [2,3], [1,3], [1,2], [0,2,3], [1,0,3], [1,1,3], [1,2,0], [1,2,2]]</span>
<span class="c1">///</span>
<span class="c1">/// SwiftCheck will search each case forward, one-by-one, and continue shrinking</span>
<span class="c1">/// until it has reached a case it deems minimal enough to present.</span>
<span class="c1">///</span>
<span class="c1">/// SwiftCheck implements a number of generators for common Swift Standard</span>
<span class="c1">/// Library types for convenience. If more fine-grained testing is required see</span>
<span class="c1">/// `Modifiers.swift` for an example of how to define a "Modifier" type to</span>
<span class="c1">/// implement it.</span>
<span class="kd">public</span> <span class="kd">protocol</span> <span class="kt">Arbitrary</span> <span class="p">{</span>
<span class="c1">/// The generator for this particular type.</span>
<span class="c1">///</span>
<span class="c1">/// This function should call out to any sources of randomness or state</span>
<span class="c1">/// necessary to generate values. It should not, however, be written as a</span>
<span class="c1">/// deterministic function. If such a generator is needed, combinators are</span>
<span class="c1">/// provided in `Gen.swift`.</span>
<span class="kd">static</span> <span class="k">var</span> <span class="nv">arbitrary</span> <span class="p">:</span> <span class="kt">Gen</span><span class="o"><</span><span class="k">Self</span><span class="o">></span> <span class="p">{</span> <span class="k">get</span> <span class="p">}</span>
<span class="c1">/// An optional shrinking function. If this function goes unimplemented, it</span>
<span class="c1">/// is the same as returning the empty list.</span>
<span class="c1">///</span>
<span class="c1">/// Shrunken values must be less than or equal to the "size" of the original</span>
<span class="c1">/// type but never the same as the value provided to this function (or a loop</span>
<span class="c1">/// will form in the shrinker). It is recommended that they be presented</span>
<span class="c1">/// smallest to largest to speed up the overall shrinking process.</span>
<span class="kd">static</span> <span class="kd">func</span> <span class="nf">shrink</span><span class="p">(</span><span class="nv">_</span> <span class="p">:</span> <span class="k">Self</span><span class="p">)</span> <span class="o">-></span> <span class="p">[</span><span class="k">Self</span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div></div>
<p>{5}: <a href="https://github.com/typelift/SwiftCheck/blob/master/Sources/SwiftCheck/Gen.swift"><code class="highlighter-rouge">Gen<A></code></a></p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">//</span>
<span class="c1">// Gen.swift</span>
<span class="c1">// SwiftCheck</span>
<span class="c1">//</span>
<span class="c1">// Created by Robert Widmann on 7/31/14.</span>
<span class="c1">// Copyright (c) 2015 TypeLift. All rights reserved.</span>
<span class="c1">//</span>
<span class="c1">/// `Gen` represents a generator for random arbitrary values of type `A`.</span>
<span class="c1">///</span>
<span class="c1">/// `Gen` wraps a function that, when given a random number generator and a</span>
<span class="c1">/// size, can be used to control the distribution of resultant values. A</span>
<span class="c1">/// generator relies on its size to help control aspects like the length of</span>
<span class="c1">/// generated arrays and the magnitude of integral values.</span>
<span class="kd">public</span> <span class="kd">struct</span> <span class="kt">Gen</span><span class="o"><</span><span class="kt">A</span><span class="o">></span> <span class="p">{</span>
</code></pre></div></div>{"twitter"=>"jamiely"}IntroductionAnsible Testing2019-03-01T09:00:00+00:002019-03-01T09:00:00+00:00https://polyglot.jamie.ly/programming/2019/03/01/ansible-testing<h1 id="introduction">Introduction</h1>
<p>This article describes the configuration management tool
<a href="https://www.ansible.com/">Ansible</a> and some of its basic concepts. Then it
discusses how to test Ansible code. Refer to the version chart below. It
assumes familiarity with the concept of configuration management,
programming, and testing.</p>
<table>
<thead>
<tr>
<th>Software</th>
<th>Version</th>
</tr>
</thead>
<tbody>
<tr>
<td>Python</td>
<td>3.7.2</td>
</tr>
<tr>
<td>Ansible</td>
<td>2.7.5</td>
</tr>
<tr>
<td>Molecule</td>
<td>2.18.1</td>
</tr>
<tr>
<td>CentOS</td>
<td>7.5</td>
</tr>
</tbody>
</table>
<h1 id="what-is-ansible">What is Ansible?</h1>
<!-- **TODO: picture of ansible communication (faster than light communication)
between hosts** -->
<p><a href="https://www.ansible.com/">Ansible</a> is a <a href="https://en.wikipedia.org/wiki/Configuration_management">configuration
management</a> tool
which can be used to manage software installations. Configuration management
helps us to ensure that software installations are documented, consistent,
and repeatable. Let’s say you want to copy the files for your web application
to a server and make sure that its runtime is installed (such as
<a href="https://www.ruby-lang.org/en/">Ruby</a> or <a href="https://docs.microsoft.com/en-us/dotnet/core/">.NET
Core</a>). While there are many
who might deploy manually, it is convenient to be able to do this in an
automated way triggered by an event such as an update to source code. Ansible
is just one of many tools that can help with this automation and is often
compared to alternatives such as <a href="https://www.chef.io/">Chef</a>,
<a href="https://puppet.com/">Puppet</a>, and
<a href="https://docs.saltstack.com/en/develop/topics/states/index.html">SaltStack</a>.</p>
<p>We use Ansible by defining a
<a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks.html">Playbook</a>
which defines a set of <em>tasks</em> to perform and a set of <em>hosts</em> to target.
When we apply the Playbook, each <em>task</em> will be applied to each <em>host</em>. Some
common use cases include:</p>
<ul>
<li>Executing remote commands</li>
<li>Copying and downloading files</li>
<li>Interaction with cloud providers</li>
<li>User, group, and permission management</li>
</ul>
<p>In order to create some measure of isolation and reuse, a developer can break
a Playbook into one or many
<a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html">Roles</a>.
For example, if we primarily deploy Ruby applications, we might create a role
called <code class="highlighter-rouge">webapp_prerequisites</code> which will prepare a host to run a Ruby
application by installing the desired Ruby version and installing
<a href="http://www.haproxy.org/">HAProxy</a> to use as a local HTTP proxy. (For a brief
discussion on why we would want to do this, see {2}). This Role can then be
referenced by each of the Playbooks that install our individual Ruby
applications.</p>
<!-- **TODO: picture of playbook split into multiple roles** -->
<p>If you want to follow along with the examples below and setup your environment,
see {7}.</p>
<h1 id="creating-a-role">Creating a Role</h1>
<p>Let’s create a simple role that installs Ruby, so that we can run a simple <a href="http://sinatrarb.com/">
Sinatra </a> or <a href="https://rubyonrails.org/"> Rails </a>
application, and installs HAProxy to proxy requests to the application. In
the Roles directory of our Ansible installation, we will create a directory
<code class="highlighter-rouge">webapp_prerequisites</code> that contains a file <code class="highlighter-rouge">tasks/main.yml</code> with the
following contents:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install Ruby</span>
<span class="na">package</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">ruby</span>
<span class="na">state</span><span class="pi">:</span> <span class="s">present</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install HAProxy</span>
<span class="na">package</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">haproxy</span>
<span class="na">state</span><span class="pi">:</span> <span class="s">present</span>
</code></pre></div></div>
<p>This YAML file contains a list of tasks that will install Ruby and HAProxy a
system with some caveats {3}. You can follow along with the code in this
article via this repository {8}.</p>
<h1 id="haproxy-virtual-ip-failover">HAProxy Virtual IP Failover</h1>
<p>In order to demonstrate a more complex Role, we will create a Role that sets
up two HAProxy hosts. One of the hosts will be active, and one will be a hot
standby. The active HAProxy host will be bound to a <a href="https://en.wikipedia.org/wiki/Virtual_IP_address">virtual
IP</a> (VIP), and if there is
some problem with the active host, the system with failover to the second
HAProxy. In this scenario, the second host will claim the VIP.</p>
<p>Keepalived is a daemon that provides generic failover capabilities. We can
configure it to monitor for the presence of an <code class="highlighter-rouge">haproxy</code> process. If one
isn’t found, Keepalived peers can communicate with each other in order to
coordinate assigning the VIP.</p>
<!-- **TODO: picture of haproxy failover** -->
<p>First, we have to install HAProxy on each host. We will make the assumption
we’re running on CentOS. We also want the HAProxy service to be enabled
and started.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install haproxy</span>
<span class="na">package</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">haproxy</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Enable haproxy</span>
<span class="na">systemd</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">haproxy</span>
<span class="na">state</span><span class="pi">:</span> <span class="s">started</span>
<span class="na">enabled</span><span class="pi">:</span> <span class="no">true</span>
</code></pre></div></div>
<p>We typically want HAProxy to proxy another sevice. For these purposes, we’ll
assume we’re proxying <a href="http://example.com/">http://example.com/</a>. We’ll
create a configuration file to transfer to the remote host.</p>
<pre><code class="language-haproxy"># haproxy.cfg
defaults
mode http
frontend my_frontend
bind *:80
default_backend my_backend
backend my_backend
# We need to send this request header otherwise
# the upstream server won't respond.
http-request set-header Host example.com
server example example.com:80
</code></pre>
<p>We need a task to copy this configuration file to the appropriate location on
the remote host:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Copy haproxy.cfg</span>
<span class="na">copy</span><span class="pi">:</span>
<span class="na">src</span><span class="pi">:</span> <span class="s">haproxy.cfg</span>
<span class="na">dest</span><span class="pi">:</span> <span class="s">/etc/haproxy/haproxy.cfg</span>
<span class="na">notify</span><span class="pi">:</span> <span class="s">reload haproxy</span>
</code></pre></div></div>
<p>Whenever the configuration file is changed by our <code class="highlighter-rouge">copy</code> task, we want to
queue a reload of the <code class="highlighter-rouge">haproxy</code> service. {6}</p>
<p>Having configured HAProxy, we need to install and configure Keepalived.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Install keepalived</span>
<span class="na">package</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">keepalived</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Setup keepalived service</span>
<span class="na">systemd</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">keepalived</span>
<span class="na">state</span><span class="pi">:</span> <span class="s">started</span>
<span class="na">enabled</span><span class="pi">:</span> <span class="no">true</span>
</code></pre></div></div>
<p>Finally, we have to configure Keepalived with a
<a href="http://jinja.pocoo.org/">Jinja</a> template file. {1} It’s somewhat involved but
the gist is we want to configure Keepalived to claim the VIP whenever HAProxy
goes down. We templatize the configuration file because it will differ
between <code class="highlighter-rouge">host1</code> and <code class="highlighter-rouge">host2</code>. {9}</p>
<p>We also need an Ansible task to create this file on the hosts:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Create keepalived configuration</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">src</span><span class="pi">:</span> <span class="s">keepalived.conf.j2</span>
<span class="na">dest</span><span class="pi">:</span> <span class="s">/etc/keepalived/keepalived.conf</span>
</code></pre></div></div>
<!-- **TODO: picture of how haproxy and keepalived interact with each other** -->
<p>That completes the role. For the full source see {8}. Now that we’re done,
how do we know it works? We can run it against a host we create manually but
an even better way to do things is to create an automated test.</p>
<h1 id="testing-roles">Testing Roles</h1>
<p>Just like other pieces of software, Ansible Roles can be tested to confirm
their operation. The reasons you might want an automated test are described
in <a href="/blog/programming/2019/02/04/tdd.html">a previous blog post</a>. It’s important to test Roles in
some way because they often provide the base components of your system like
runtimes, HTTP servers, and file and directory structures. Especially if
multiple people are working on the same Role, the risk of adding the wrong
configuration to a file or breaking a Jinja configuration template is high.</p>
<p>The standard way to test Ansible Roles is using
<a href="https://molecule.readthedocs.io/en/latest/">Molecule</a>. Molecule is a testing
framework designed for Ansible. It works by adding a <code class="highlighter-rouge">molecule</code> directory to
each Role, which contains instructions for how to provision testing hosts,
Playbooks to apply the Role, and tests to run against each provisioned
testing host. If you have an existing Role, this directory can be
automatically generated by installing <code class="highlighter-rouge">molecule</code>, then running the following
command inside the Role directory. For example, we can add molecule to the
<code class="highlighter-rouge">webapp_prerequisites</code> Role by running the following command from inside the
<code class="highlighter-rouge">webapp_prerequisites</code> directory.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% molecule init scenario <span class="se">\</span>
<span class="nt">--driver-name</span> vagrant <span class="se">\</span>
<span class="nt">--scenario-name</span> default <span class="se">\</span>
<span class="nt">--role-name</span> webapp_prerequisites
</code></pre></div></div>
<p>In the command above, we tell <code class="highlighter-rouge">molecule</code> that we want to create a new testing
scenario called <code class="highlighter-rouge">default</code>, and to provision testing hosts using
<a href="https://www.vagrantup.com/">Vagrant</a>. Vagrant makes it easy to
programmatically provision virtual machines (VMs). {4}</p>
<p>Afterwards, from inside the Role directory, we can run the tests using
<code class="highlighter-rouge">molecule test</code>. The <code class="highlighter-rouge">molecule test</code> command will provision the testing
infrastructure, in this case VMs running on
<a href="https://www.virtualbox.org/">VirtualBox</a>, prepare and configure the hosts
using Ansible, run tests against it using Python’s
<code class="highlighter-rouge">[testinfra](https://testinfra.readthedocs.io/en/latest/)</code> framework, then
destroy the testing infrastructure, reporting the results of the test. This
can take quite awhile when using virtual machines, so when I am in the
process of developing a Role, I use <code class="highlighter-rouge">molecule verify</code>, which will run all
those steps without destroying the infrastructure. I can repeatedly run
<code class="highlighter-rouge">verify</code> as I develop the Role.</p>
<!-- **TODO: picture of molecule with glasses checking things off a clipboard** -->
<p>Given the example Role definition above, we may want to test that HAProxy has
been properly installed by running a simple command and checking its exit
code. We can login to an instance provisioned by Molecule using <code class="highlighter-rouge">molecule
login</code>.</p>
<p>We first run a command that should return the version of HAProxy that is
installed, then we <code class="highlighter-rouge">echo</code> the exit code stored in <code class="highlighter-rouge">$?</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% molecule login
<span class="nt">--</span><span class="o">></span> Validating schema /.../ansible/roles/ecri.haproxy/molecule/default/molecule.yml.
Validation completed successfully.
<span class="o">[</span>vagrant@instance ~]<span class="nv">$ </span>haproxy <span class="nt">-v</span>
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>
<span class="o">[</span>vagrant@instance ~]<span class="nv">$ </span><span class="nb">echo</span> <span class="nv">$?</span>
0
</code></pre></div></div>
<p>We can have Molecule test this automatically by writing a test using the
<code class="highlighter-rouge">testinfra</code> Python provisioner testing framework. In this article, we’ll be
using testinfra 1.14.1. In <code class="highlighter-rouge">molecule/tests/test_default.py</code>, we can add the
following test, that does the same thing we did manually above.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_haproxy_installed</span><span class="p">(</span><span class="n">host</span><span class="p">):</span>
<span class="n">cmd</span> <span class="o">=</span> <span class="n">host</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">"/usr/sbin/haproxy -v"</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">cmd</span><span class="o">.</span><span class="n">rc</span> <span class="o">==</span> <span class="mi">0</span>
</code></pre></div></div>
<p>After running <code class="highlighter-rouge">molecule verify</code>, we get a message like</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tests/test_default.py::test_haproxy_installed[ansible://instance] PASSED [100%]
=========================== 1 passed in 1.95 seconds ===========================
Verifier completed successfully.
</code></pre></div></div>
<p><code class="highlighter-rouge">testinfra</code> has an API for retrieving hosts, running commands on those hosts,
and obtaining information about the host (facts). For example, we can run a
simple test to see if HAProxy is running and enabled on start with the
following code:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_services_running_and_enabled</span><span class="p">(</span><span class="n">host</span><span class="p">):</span>
<span class="n">service</span> <span class="o">=</span> <span class="n">host</span><span class="o">.</span><span class="n">service</span><span class="p">(</span><span class="s">'haproxy'</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">service</span><span class="o">.</span><span class="n">is_running</span>
<span class="k">assert</span> <span class="n">service</span><span class="o">.</span><span class="n">is_enabled</span>
</code></pre></div></div>
<p>We can run arbitrary commands on the host, retrieving the stdout, stderr, and
return code. This gives us a lot of flexibility in running our tests. For
example, to determine if HAProxy is serving requests, we can use <code class="highlighter-rouge">curl</code> to
test connectivity while logged onto the host:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_redirection</span><span class="p">(</span><span class="n">host</span><span class="p">):</span>
<span class="n">cmd</span> <span class="o">=</span> <span class="n">host</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="s">"curl -v http://localhost"</span><span class="p">)</span>
<span class="k">assert</span> <span class="s">"HTTP/1.1 200"</span> <span class="ow">in</span> <span class="n">cmd</span><span class="o">.</span><span class="n">stderr</span>
</code></pre></div></div>
<p>In this test, we confirm that something is responding on the default <code class="highlighter-rouge">http</code>
port 80, and that it responds with a 200 status code. If it’s present, the
text will be in the <code class="highlighter-rouge">stderr</code> of the command. We use Python’s <code class="highlighter-rouge">assert</code>
statement that will throw an error if the following condition is false.</p>
<!-- **TODO: picture molecule giving seal of approval to something** -->
<h1 id="multi-machine-testing">Multi-machine testing</h1>
<p>The test above is useful, and you can imagine the more complicated tests we
can create to verify an instance. Still more complex scenarios arise in the
case of multi-host testing.</p>
<p>In order to spin up multiple VMs, we modify the <code class="highlighter-rouge">molecule.yml</code> definition to
include multiple <code class="highlighter-rouge">platforms</code>:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">platforms</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">host1</span>
<span class="na">box</span><span class="pi">:</span> <span class="s">bento/centos-7.5</span>
<span class="na">instance_raw_config_args</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">vm.network</span><span class="nv"> </span><span class="s">:private_network,</span><span class="nv"> </span><span class="s">ip:</span><span class="nv"> </span><span class="s">"192.168.3.9"'</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">host2</span>
<span class="na">box</span><span class="pi">:</span> <span class="s">bento/centos-7.5</span>
<span class="na">instance_raw_config_args</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">vm.network</span><span class="nv"> </span><span class="s">:private_network,</span><span class="nv"> </span><span class="s">ip:</span><span class="nv"> </span><span class="s">"192.168.3.10"'</span>
</code></pre></div></div>
<p>We assign them static IPs <code class="highlighter-rouge">192.168.3.9</code> and <code class="highlighter-rouge">192.168.3.10</code> in a private network so
that they can communicate with each other. A private network limits access to
the VM to only the VM host machine {5}.</p>
<p>When we run tests involving multiple hosts, we may need to setup the test by
performing actions across hosts. <code class="highlighter-rouge">testinfra</code> will run the tests for each
host, so we need a way to get a reference to a host that is not the one being
tested. For this article, we will call the host currently being tested by
<code class="highlighter-rouge">testinfra</code> the <strong>current testing host</strong>. <code class="highlighter-rouge">testinfra</code> typically passes a
<a href="https://testinfra.readthedocs.io/en/latest/modules.html#host"><code class="highlighter-rouge">Host</code></a>
instance to each test method as the first argument:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_something</span><span class="p">(</span><span class="n">host</span><span class="p">):</span> <span class="c"># <-- the host arg is a Host instance</span>
<span class="c"># test definition</span>
</code></pre></div></div>
<p>We can use this <code class="highlighter-rouge">Host</code> instance to retrieve other hosts we previously defined
in the <code class="highlighter-rouge">platforms</code> section of <code class="highlighter-rouge">molecule.yml</code>. We call <code class="highlighter-rouge">host.get_host</code> with a
URI that refers to the host we want. When we are provisioning hosts with
Ansible, it looks something like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">my_host</span> <span class="o">=</span> <span class="n">host</span><span class="o">.</span><span class="n">get_host</span><span class="p">(</span><span class="s">"ansible://my_host_name?ansible_inventory="</span> <span class="o">+</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="s">'MOLECULE_INVENTORY_FILE'</span><span class="p">])</span>
</code></pre></div></div>
<p><code class="highlighter-rouge">get_host</code> takes a <a href="https://en.wikipedia.org/wiki/Uniform_Resource_Identifier"> URI whose scheme
</a> in the
provisioner <code class="highlighter-rouge">ansible</code>. <code class="highlighter-rouge">my_host_name</code> is one of the host names defined in
<code class="highlighter-rouge">platforms</code>. We tell Ansible the location of the inventory file that Molecule
has generated via the <code class="highlighter-rouge">ansible_inventory</code> query parameter. Ansible requires a
list of hosts to operate on, which is commonly referred to as the inventory.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ansible://my_host_name?ansbile_inventory=/tmp/inventory.txt
└─┬──┘ └─┬──────┘ └───────────────────────────┬┘
scheme path query parameter
</code></pre></div></div>
<p>You may want to only run the test from a single host. Although I don’t
know of a way to do this, we can skip tests when running on the wrong host.
For example, if we only want a test to run against <code class="highlighter-rouge">host1</code>, then we can check
the hostname of the <em>current testing host</em>, and return from the test method
if it doesn’t match <code class="highlighter-rouge">host1</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_only_host1</span><span class="p">(</span><span class="n">host</span><span class="p">):</span>
<span class="n">ansible_facts</span> <span class="o">=</span> <span class="n">host</span><span class="o">.</span><span class="n">ansible</span><span class="o">.</span><span class="n">get_variables</span><span class="p">()</span>
<span class="k">if</span> <span class="n">ansible_facts</span><span class="p">[</span><span class="s">'inventory_hostname'</span><span class="p">]</span> <span class="o">!=</span> <span class="s">'host1'</span><span class="p">:</span>
<span class="k">return</span>
</code></pre></div></div>
<p>In this case, we are using Ansible, so we request the facts about the host
from Ansible. This includes the <code class="highlighter-rouge">inventory_hostname</code>, which is the hostname
that Ansible refers to the host by. This may be something different than what
the host uses to refer to itself.</p>
<p>In order to test the HAProxy VIP Playbook, we want to create a test that will
confirm that the VIP still responds after the master <code class="highlighter-rouge">host1</code> stops
responding. It doesn’t matter which host is the <em>current testing host</em> as
long as we confirm that when the <code class="highlighter-rouge">keepalived-</code>master <code class="highlighter-rouge">host1</code>’s, HAProxy
server goes down, whichever host is associated to the VIP is still serving
requests. In the test, we will assume we AREN’T running on <code class="highlighter-rouge">host1</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_vip_active_after_one_node_goes_down</span><span class="p">(</span><span class="n">host</span><span class="p">):</span>
<span class="c"># if this is not host1, return</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">service_haproxy</span><span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="s">'stop'</span><span class="p">)</span>
<span class="n">wait_awhile</span><span class="p">()</span>
<span class="n">test_vip</span><span class="p">()</span>
<span class="k">finally</span><span class="p">:</span>
<span class="c"># After the test, make sure that haproxy is started again</span>
<span class="n">service_haproxy</span><span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="s">'start'</span><span class="p">)</span>
<span class="n">wait_awhile</span><span class="p">()</span>
</code></pre></div></div>
<p>The <code class="highlighter-rouge">test_vip</code> function just does a local curl and hits the VIP:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_vip</span><span class="p">():</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="k">assert</span> <span class="mi">0</span> <span class="o">==</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">call</span><span class="p">([</span> <span class="s">"/usr/bin/curl"</span><span class="p">,</span> <span class="s">"-vs"</span><span class="p">,</span>
<span class="s">"--connect-timeout"</span><span class="p">,</span> <span class="s">"1"</span><span class="p">,</span> <span class="s">"http://192.168.3.2"</span><span class="p">])</span>
</code></pre></div></div>
<p>For detail on what <code class="highlighter-rouge">service_haproxy</code> and <code class="highlighter-rouge">wait_awhile</code>, which are somewhat
self-explantory, see {10}.</p>
<h1 id="conclusion">Conclusion</h1>
<p>In this article, we learned a little about what Ansbile is and what it can do.
We learned about how to create Roles, and how to test them. We also
learned how to write a complex multi-host testing scenario using
Molecule and <code class="highlighter-rouge">test_infra</code>.</p>
<h1 id="endnotes">Endnotes</h1>
<p>{1}: Jinja is the template language that Ansible uses. It uses curly braces
to denote variables, and has looping and conditional constructs.</p>
<p>{2}: Local HTTP Proxies are useful as middleware to provide functionality
that we might not want to build into our application. For example, HAProxy
has features related to routing, http header injection, Lua-scripting,
SSL, and throttling built into it can provide to an application without
the need to make the application more complex by building it into the application.
For example, if the application needed simple basic authentication,
this could be provided by a local HAProxy instance proxying the application.
This is made even more relevant when one uses containers. Using containers,
it’s incredibly simple to add sidecar containers that add functionality.</p>
<p>{3}: The method we use to install Ruby and HAProxy results in versions
that are dependent upon the underlying package manager and repositories
installed on the host. For example, CentOS 7.5’s default repositories only
include up to HAProxy 1.5, but the most recent version as of this writing is
HAProxy 1.8.</p>
<p>{4}: Vagrant is useful for creating VMs for development. Vagrant itself is
compatible with a number of local and remote VM providers such as VirtualBox,
VMWare products, and Amazon Web Services (AWS).</p>
<p>{5}: The terminology is a bit confusing, but in this article, we call a host some
logical computer running an OS. This is different than a <code class="highlighter-rouge">VM host</code> which
is a machine that runs Virtual Machines.</p>
<p>{6}: In Ansible, Handlers are used to queue actions to happen at a later
time. We know we want to reload HAProxy if the configuration changes, but we
also might want to reload HAProxy if we have a change in an SSL certificate.
We don’t want to reload HAProxy twice, so we just queue this reload to happen
later.</p>
<p>{7}: <strong>Setting Up Ansible</strong></p>
<p>The source code at {8} provides a <code class="highlighter-rouge">requirements.txt</code> file which lists the
Python dependencies needed to follow along with the example. We can use
<a href="https://virtualenv.pypa.io/en/latest/"><code class="highlighter-rouge">virtualenv</code></a> to install these
dependencies into an isolated python environment. On MacOS, we can use <a href="https://brew.sh/">
<code class="highlighter-rouge">homebrew</code> </a> to install Python 3 via</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% brew <span class="nb">install </span>python
</code></pre></div></div>
<p>After making sure it is on the <code class="highlighter-rouge">PATH</code>, we can confirm our Python version using:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>% python <span class="nt">-V</span>
Python 3.7.2
</code></pre></div></div>
<p>Then, we can install <code class="highlighter-rouge">virtualenv</code> globally using <code class="highlighter-rouge">pip</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install virtualenv
</code></pre></div></div>
<p>We can create a <code class="highlighter-rouge">virtualenv</code> environment and activate it using something like:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>virtualenv .python
source .python/bin/activate
</code></pre></div></div>
<p>Afterwards, we can install any requirements provided in the <code class="highlighter-rouge">requirements.txt</code> like:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install -r requirements.txt
</code></pre></div></div>
<p>{8}: GitHub repository for examples: <a href="https://github.com/jamiely/ansible_haproxy_vip">https://github.com/jamiely/ansible_haproxy_vip</a></p>
<p>{9}: Keepalived Jinja Configuration template: https://github.com/jamiely/ansible_haproxy_vip/blob/master/templates/keepalived.conf.j2</p>
<p>{10}: VIP test using <code class="highlighter-rouge">test_infra</code>: https://github.com/jamiely/ansible_haproxy_vip/blob/master/molecule/default/tests/test_default.py</p>
<!--
- What is ansible?
- What are some alternatives to ansible?
- How can we use ansible?
- What are common use cases?
- Why is it important to test Ansible roles?
- What kinds of issues can we experience?
- How do we test Ansible roles?
- How do we use Molecule?
- How can we test installing haproxy?
- How can we test setting up VIP with HAProxy?
- Docker + Ansible?
- Virtualenv?
-
TODO:
- asciicast
- video
- screenshot
Ideas for cover image
- Molecule sensing/testing a black box?
-->{"twitter"=>"jamiely"}IntroductionLet’s Test2019-02-04T00:00:00+00:002019-02-04T00:00:00+00:00https://polyglot.jamie.ly/programming/2019/02/04/tdd<h1 id="intro">Intro</h1>
<p>The art of unit testing and
<a href="https://en.wikipedia.org/wiki/Test-driven_development"> Test-driven Development (TDD) </a>
is considered
obscure to some but is incredibly useful to the practie of software
development. We all know that testing of some kind is important to producing
quality software, yet our actions do not reflect this knowledge.</p>
<h1 id="are-tests-necessary">Are tests necessary?</h1>
<p>Some of us test the features we’re working on manually before sending the
code along for acceptance testing. While this type of manual testing is
important when implementing a new feature, the lack of a corresponding
automated test is problematic for future maintenance because it is easier
for regressions to occur. It can be argued there are some cases where
lack of automated tests is okay:</p>
<ul>
<li>There are lots of very experienced and skilled developers</li>
<li>The code is already written in a very maintainable and clean way</li>
<li>The code is throwaway code</li>
<li>We want to frontload development at the cost of future development,
as in the case of a simple MVP.</li>
</ul>
<p>The main caveat to all of the above scenarios is they focus on the now rather
than the future. While we don’t want to go yak-shaving by attempting to
future proof everything, there is a concern that if we’re always focusing on
the now, then we may never focus on the future.</p>
<h1 id="why-dont-we-write-tests">Why don’t we write tests?</h1>
<p>For those who consider tests important, why is it that we do not write tests?
There are a number of specific reasons for every scenario, but they can be
boiled down to a few categories:</p>
<ul>
<li>Time</li>
<li>Ignorance</li>
<li>Incentive</li>
</ul>
<p>If testing is not part of the company culture, developers are rarely
explicitly given <strong>time</strong> to write tests. It’s also common for developers to
feel bogged down by the amount of work they have to do–adding writing
automated tests would just be additional work. Especially when automated
testing is something new for a developer, work can take several times longer
to complete.</p>
<p>Writing automated tests and writing testable code is a discipline unto
itself. It’s no wonder that sometimes we don’t write tests because we don’t
know how to. We are <strong>ignorant</strong> of how to do it. Even more difficult is
adding tests to legacy code.</p>
<p>Sometimes management disincentivizes testing. Imagine that you were rewarded
for every bug you fix. This would create an <strong>incentive</strong> for you to avoid
writing automated tests (this might allow more regression-related bugs
through). Praising someone for fighting a fire in production, when it was
preventable through automated testing is another kid of disincentive.
Finally, ignoring the tests that people write and not creating a testing
culture can be a disincentive. Those that are externally motivated may not
feel rewarded for something perceived as “extra” work.</p>
<h1 id="how-do-we-start-writing-more-tests">How do we start writing more tests?</h1>
<p>Although I don’t believe in 100% coverage, at the minimum, I try to write
tests for important bits of functionality, for things I’d be spending a lot
of time manually testing, or code that I expect to be regression-sensitive. I
don’t always want to write tests for the reasons stated above, and I have
some thoughts that might help people that feel the same way.</p>
<p>I’ve always gotten good performance reviews. While this does not mean I’m a
good developer, it should mean that I get my work done on time and as expected.
Still, when I compare my productivity and ability now compared to when I first
started in my career, it is worlds apart. I know that the code I wrote as a
junior was less secure, less performant, had more issues, and
was less maintainable. The one constant between then and now is that I’ve always
felt like I had a lot of work to do. My experience has informed several
opinions I hold.</p>
<p><em>No matter how much work I have, there is a limited amount I can do.</em> I
remember always being busy, no matter how much work I was actually finishing.
This tells me I could’ve finished fewer tasks (especially when I’ve worked
long hours), or put more quality time into fewer tasks without much change in
my perceived performance.</p>
<p><em>The scope of my code has increased over time, but the time to implement
things hasn’t drastically changed.</em> As I’ve improved over my career, the time
to implement a feature such as a web form hasn’t changed much. What’s changed
is the quality of the feature I deliver. For example, things like validation,
santization, security, performance, and, <em>yes</em>, testing are things I include
in features by default.</p>
<p><em>It’s uncommon for management to explicitly give time to test.</em> I’ve
rarely if ever been alotted time for testing. Rather than being because
management does not care about quality, I think this is more because
management expects me to take care of quality in the ways I see fit. It
would be like management giving me more time to make something secure.
I think it’s expected that I make things secure by default unless
otherwise directed to totally disregard security, and I consider testing
to be a lot like that.</p>
<p>There are two things I’ve done to address my own ignorance with writing
tests. Firstly, while generally I don’t use TDD in
my daily work, I think it is incredibly useful to have experience with it,
and know how to do it, in order to gain a better understanding of how to
write testable code. Where I tend to use TDD is where I am very clear about
how I want the interface for a feature to look when it is done. For example,
I often use TDD with code that does string parsing. There is usually a
function like <code class="highlighter-rouge">parse</code> which returns a string, and the inputs and outputs are
well defined. I also prefer to use TDD for programming challenges like
<a href="http://adventofcode.com/">Advent of Code</a>.
Secondly, I always recommend
<a href="https://www.goodreads.com/book/show/44936.Refactoring"><em>Refactoring</em></a>
and
<a href="https://www.goodreads.com/book/show/44919.Working_Effectively_with_Legacy_Code"><em>Working Effectively with Legacy Code</em></a>
as two must-read books. They’ve both helped me to learn how to add
tests to existing code.</p>
<p>It’s hard to deal with systemic disincentives to testing. At best, the
disincentives are incidental. At worst, they may be designed to make
middle-management look good, but I haven’t ever encountered these dark
management patterns. Most of what I have seen has been positive attitudes
towards fire-fighting and a lack of a testing culture. Although systemic
incentivization may be something difficult for an individaul to address, I
try to adopt some internal incentives for writing tests. One such incentive
is that if I break something, I will probably get blamed for it or have to
fix it. An existing test makes this easier. Even if I am not the one who
broke it, sometimes the original developer will be asked to fix an issue,
even if someone else broke it. Another incentive is that a test serves as
documentation about how something should work and makes returning to a
project easier.</p>
<h1 id="towards-testing">Towards Testing</h1>
<p>I’ve never been in an organization that placed a heavy emphasis on testing.
Despite this, I have become a proponent of automated testing. I think that
100% code coverage is impractical, but there must exist some idyllic waypoint
between TDD and 100% coverage and having zero tests. There are only a few
main reasons why people don’t test, and there are clear ways to address those
reasons. Although it is hard to enact systemic change as a front-line
developer, there are a number of things that an individual can do to start
writing automated tests. Whether we start writing more tests is predicated on
whether we even agree that automated tests are important, but I think we can
all agree that testing is an important part of creating quality software.</p>{"twitter"=>"jamiely"}IntroAdvent of Code 20182019-01-02T00:00:00+00:002019-01-02T00:00:00+00:00https://polyglot.jamie.ly/programming/2019/01/02/advent-of-code-2018<p>Although I first heard of Advent of Code last year, this is
the first year I participated. <a href="https://adventofcode.com/">Advent of Code</a> is a set of
Christmas-themed puzzles arranged as an
<a href="https://en.wikipedia.org/wiki/Advent_calendar">Advent Calendar</a>,
with a new programming puzzle released daily starting December 1st,
leading up until Christmas.</p>
<p>I wanted to brush up on Python 3 syntax, having been using Python 2 for
awhile and I decided to start this year’s exercises using a
<a href="https://jupyter.org/">Juypter notebook</a>.
I thought that this would be a good way to share my code with others.
Unfortunately, after completing a few exercises, I noted the process of
developing a solution was fairly difficult. I wanted to run unit tests as a
part of developing solutions since I was using Test-driven Development (TDD).
However, as you can see if you take a look at the Jupyter notebooks for the
initial solutions <a href="https://github.com/jamiely/adventofcode/blob/master/2018/day1.ipynb">1</a>, <a href="https://github.com/jamiely/adventofcode/blob/master/2018/day2.ipynb">2</a>, <a href="https://github.com/jamiely/adventofcode/blob/master/2018/day3.ipynb">3</a>, this wound up being cumbersome.
Specifically, the cycle of rerunning the tests was tedious. When I use TDD, I
really want to rerun the tests often, and even run them automatically when
files change. In a Jupyter notebook, to rerun the code, you re-execute blocks
or cells of code manually.</p>
<p>I think that where a Jupyter notebook would be more useful is with something
more exploratory, trying to capture a stream of consciousness as one explores
a problem space. With these puzzles, especially since there are test cases,
there was little of that type of exploration. Having only completed half the
puzzles, there may be more problems requiring this type of exploration in the
latter half.</p>
<p>Another problem that I had with using a Jupyter notebook is code reuse. There
seem to be three straightforward paths to take. Either write the common code
as a regular python file and import it into two notebooks,
<a href="https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Importing%20Notebooks.html">import a Jupyter notebook as a module</a>,
or use a single notebook for all of the puzzles. I might’ve tried importing
another notebook as a module, but the previous problem with running tests
forced me away from using Jupyter.</p>
<p>I wound up switching to use plain Python files, and running unit tests on
file changes using the <a href="http://eradman.com/entrproject/"><code class="highlighter-rouge">entr</code></a> command. It
accepts a list of files from <code class="highlighter-rouge">stdin</code>, and will run a given command when one of
those files is changed. I used it like:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>find day8<span class="k">*</span> | entr python day8.test.py
</code></pre></div></div>
<p>which would pipe all the files beginning with <code class="highlighter-rouge">day8</code> to <code class="highlighter-rouge">entr</code>. When any of the files
changed, <code class="highlighter-rouge">entre</code> would run <code class="highlighter-rouge">day8.test.py</code>. In this way, I could continually evaluate
the status of tests and use a
<a href="https://blog.cleancoder.com/uncle-bob/2014/12/17/TheCyclesOfTDD.html">red-green-refactor cycle</a>
to implement the required features.</p>
<p>As of this writing, I have completed 12 of this year’s puzzles. The code can be found on
<a href="https://github.com/jamiely/adventofcode/tree/master/2018">GitHub</a>.
My favorite puzzle was
<a href="https://github.com/jamiely/adventofcode/blob/master/2018/day10.py">Day 10</a>,
which asks you to find the message spelled out
by stars. To do this, one had to read input star velocities, then figure out
when the stars would converge into an alphabetic passphrase. This turned out
to be the moment in time when the bounding box of the starfield reached its
smallest size. I also wound up creating a video of the transition by
streaming dynamic images created using
<a href="https://pillow.readthedocs.io/en/5.3.x/"><code class="highlighter-rouge">Pillow</code></a> to
<a href="https://www.ffmpeg.org/">ffmpeg</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/VWQ3NOmIjBk" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>I found Day 8’s puzzle to be one of the easiest. It was basically parsing input
into a tree, and was <a href="https://github.com/jamiely/adventofcode/blob/master/2018/day8.py#L90">easily solved recursively</a>.
<a href="https://github.com/jamiely/adventofcode/blob/master/2018/day9.py">Day 9</a>
gave me some trouble initially. The puzzle involved a game using a ring of
marbles. Each turn involves inserting a marble somewhere in the ring. While a
performant solution used a doubly-linked list, I started with a regular list,
and when insertions became an obvious issue (it seems the default list
implementation has linear time insertions), used
<a href="https://pypi.org/project/blist/"><code class="highlighter-rouge">blist</code></a>.
While I was happy enough with my solution, I saw on reddit that someone used
a <a href="https://docs.python.org/2/library/collections.html#collections.deque"><code class="highlighter-rouge">deque</code></a>
and its <code class="highlighter-rouge">rotate</code> function to great purpose. This is a certainly a case where
knowing the right data-structure made the problem simple.</p>
<p>I found <a href="https://github.com/jamiely/adventofcode/blob/master/2018/day6.py">Day 6</a>
to be the hardest puzzle of those I’ve done. It asks you to find the areas of influence of
each coordinate in a set. The difficulty I had was figuring out what to do with coordinates
with infinite areas of influence. I eventually figured out that any coordinates on the border
of the bounding box of all coordinates would have infinite areas, and hence should be
ignored. This was a very simple solution that just took a bit of thought.</p>
<p>Advent of Code is a great resource if you are interested in programming puzzles.
I’m very interested in puzzles and games with programming elements such as
<a href="http://www.zachtronics.com/spacechem/">SpaceChem</a> and <a href="https://www.factorio.com/">Factorio</a>
but these games sometimes are too far removed from actual programming to be
of practical use (they are are fun though). The nice thing about programming
puzzles and challenges like Advent of Code or
<a href="https://projecteuler.net/">Project Euler</a> is that you can actually practice or
learn a language that you can use to work on real projects.</p>{"twitter"=>"jamiely"}Although I first heard of Advent of Code last year, this is the first year I participated. Advent of Code is a set of Christmas-themed puzzles arranged as an Advent Calendar, with a new programming puzzle released daily starting December 1st, leading up until Christmas.