bd808.comhttps://bd808.com/2017-04-17T04:18:24+00:00Making Django migrations that work with MySQL 5.5 and utf8mb42017-04-17T04:18:24+00:002017-04-17T04:18:24+00:00Bryan Davistag:bd808.com,2017-04-17:/blog/2017/04/17/making-django-migrations-that-work-with-mysql-55-and-utf8mb4/<p>I like <a href="https://en.wikipedia.org/wiki/Django_(web_framework)">Django</a>, <a href="https://en.wikipedia.org/wiki/MySQL">MySQL</a>, and <a href="https://en.wikipedia.org/wiki/Unicode">Unicode</a>, but getting all three to play together
nicely can sometimes be a bit challenging. One of the more annoying things is
getting Django to make a <a href="https://docs.djangoproject.com/en/1.8/topics/migrations/">migration</a> that will create a 255 character
<code>CharField</code> that is encoded using the <a href="https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html">utf8mb4</a> character set and indexed.</p>
<p>I like <a href="https://en.wikipedia.org/wiki/Django_(web_framework)">Django</a>, <a href="https://en.wikipedia.org/wiki/MySQL">MySQL</a>, and <a href="https://en.wikipedia.org/wiki/Unicode">Unicode</a>, but getting all three to play together
nicely can sometimes be a bit challenging. One of the more annoying things is
getting Django to make a <a href="https://docs.djangoproject.com/en/1.8/topics/migrations/">migration</a> that will create a 255 character
<code>CharField</code> that is encoded using the <a href="https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html">utf8mb4</a> character set and indexed.</p>
<p>Out of the box, MySQL's <a href="https://dev.mysql.com/doc/refman/5.5/en/innodb-storage-engine.html">InnoDB</a> table type has a maximum index length or 767
bytes. This is enough to index 255 characters in the <a href="https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8.html">utf8</a> encoding, but that
encoding won't work for storing any Unicode data from the <a href="https://en.wikipedia.org/wiki/Plane_(Unicode)#Supplementary_Multilingual_Plane">Supplementary
Multilingual Plane</a>. That means you can't put a <a href="http://emojipedia.org/unicorn-face/">unicorn face</a> (🦄) or
a <a href="http://emojipedia.org/slice-of-pizza/">slice of pizza</a> (🍕) into a column using this encoding. Changing to the
utf8mb4 character set will allow you to store four byte code points, but only
index 191 characters.</p>
<p>With MySQL 5.5.14 and later you can raise this limit to 3072 bytes by using
the <a href="https://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_large_prefix">innodb_large_prefix</a> setting along with the Barracuda file format, file
per table storage, and dynamic row format. The first three can all be set
server wide, but the row format for the table needs to be provided in
a <code>CREATE TABLE</code> or <code>ALTER TABLE</code> statement as a <code>ROW_FORMAT=DYNAMIC</code>
attribute.</p>
<p>Django does not have a feature flag or setting for adding the
needed attribute. I've worked around this before by using manual database
hacking, but today I figured out a hack that you can manually apply to your
Django migration files to work around it. The trick is to edit the migration
so that the initial field creation uses a length that will fit in the 767 byte
limit, and then add a <code>RunSQL</code> to change the table's row format and an
<code>AlterField</code> to increase the field length.</p>
<div class="highlight"><pre><span class="code-line"><span></span> <span class="n">operations</span> <span class="o">=</span> <span class="p">[</span></span>
<span class="code-line"> <span class="n">migrations</span><span class="o">.</span><span class="n">CreateModel</span><span class="p">(</span></span>
<span class="code-line"> <span class="n">name</span><span class="o">=</span><span class="s1">'UnicodeHack'</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">fields</span><span class="o">=</span><span class="p">[</span></span>
<span class="code-line"> <span class="p">(</span><span class="s1">'hack'</span><span class="p">,</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">unique</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">max_length</span><span class="o">=</span><span class="mi">128</span><span class="p">)),</span></span>
<span class="code-line"> <span class="p">],</span></span>
<span class="code-line"> <span class="p">),</span></span>
<span class="code-line"> <span class="n">migrations</span><span class="o">.</span><span class="n">RunSQL</span><span class="p">(</span><span class="s1">'ALTER TABLE unicodehack ROW_FORMAT = DYNAMIC;'</span><span class="p">),</span></span>
<span class="code-line"> <span class="n">migrations</span><span class="o">.</span><span class="n">AlterField</span><span class="p">(</span></span>
<span class="code-line"> <span class="n">model_name</span><span class="o">=</span><span class="s1">'unicodehack'</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">name</span><span class="o">=</span><span class="s1">'hack'</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">field</span><span class="o">=</span><span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">unique</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">max_length</span><span class="o">=</span><span class="mi">255</span><span class="p">),</span></span>
<span class="code-line"> <span class="p">),</span></span>
<span class="code-line"> <span class="p">]</span></span>
</pre></div>SASL auth with python-irc2017-03-01T06:48:04+00:002017-03-01T06:48:04+00:00Bryan Davistag:bd808.com,2017-03-01:/blog/2017/03/01/sasl-auth-with-python-irc/<p>I maintain a couple of IRC bots that help out with Wikimedia devops tasks.
<a href="https://wikitech.wikimedia.org/wiki/Tool:Jouncebot">Jouncebot</a> was a bot I started helping with when <a href="https://github.com/mattofak">@mattofak</a> moved on to
other projects. Later I developed <a href="https://wikitech.wikimedia.org/wiki/Tool:Stashbot">Stashbot</a> as a replacement for using the
<a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-irc.html">Logstash</a> that collected data for my <a href="https://tools.wmflabs.org/sal/">SAL</a> tool in <a href="https://wikitech.wikimedia.org/wiki/Portal:Tool_Labs">Tool
Labs</a>.</p>
<p>Both bots are built using the awesome <a href="https://pypi.python.org/pypi/irc">irc python library</a> from <a href="https://github.com/jaraco">Jason
Coombs</a>. I've copied various core irc behaviors from one bot to the other as
I've discovered and fixed various bugs in how I was using the library.
I finally got around to extracting these core parts into a Python library of
it's own that I have named "IRC Bot Behavior Bundle" or <a href="https://python-ib3.readthedocs.io/en/latest/index.html">IB3</a> for short.</p>
<p>I maintain a couple of IRC bots that help out with Wikimedia devops tasks.
<a href="https://wikitech.wikimedia.org/wiki/Tool:Jouncebot">Jouncebot</a> was a bot I started helping with when <a href="https://github.com/mattofak">@mattofak</a> moved on to
other projects. Later I developed <a href="https://wikitech.wikimedia.org/wiki/Tool:Stashbot">Stashbot</a> as a replacement for using the
<a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-irc.html">Logstash</a> that collected data for my <a href="https://tools.wmflabs.org/sal/">SAL</a> tool in <a href="https://wikitech.wikimedia.org/wiki/Portal:Tool_Labs">Tool
Labs</a>.</p>
<p>Both bots are built using the awesome <a href="https://pypi.python.org/pypi/irc">irc python library</a> from <a href="https://github.com/jaraco">Jason
Coombs</a>. I've copied various core irc behaviors from one bot to the other as
I've discovered and fixed various bugs in how I was using the library.
I finally got around to extracting these core parts into a Python library of
it's own that I have named "IRC Bot Behavior Bundle" or <a href="https://python-ib3.readthedocs.io/en/latest/index.html">IB3</a> for short.</p>
<p>The IB3 library provides a collection of <a href="https://en.wikipedia.org/wiki/Mixin">mixin</a> classes that can be used to
extend an <code>irc.bot.SingleServerIRCBot</code> instance to do things like:</p>
<ul>
<li>Encrypt connections using SSL</li>
<li>Authenticate to Freenode</li>
<li>Join channels slowly to avoid flood bans</li>
<li>Ping the upstream IRC server to check for connection liveness</li>
<li>Rejoin channels when kicked</li>
<li>Regain primary nickname after receiving a <code>ERR_NICKNAMEINUSE</code> message</li>
</ul>
<p>All of these behaviors are pretty battle tested from months/years of use in
one or the other of my bots.</p>
<p>IB3 has one sexy new addition, <a href="http://ircv3.net/specs/extensions/sasl-3.1.html">SASL</a> PLAIN authentication. SASL is an IRC v3
protocol extension that allows a client to authenticate at the time of
connection. This method lets you authenticate before your connection becomes
visible to other clients on the server. It also seems to be a bit faster than
the normal exchange with NickServ.</p>
<p>Making a basic bot that uses SASL auth is pretty easy using the library:</p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="c1"># This program is free software: you can redistribute it and/or modify it</span></span>
<span class="code-line"><span class="c1"># under the terms of the GNU General Public License as published by the Free</span></span>
<span class="code-line"><span class="c1"># Software Foundation, either version 3 of the License, or (at your option)</span></span>
<span class="code-line"><span class="c1"># any later version.</span></span>
<span class="code-line"><span class="c1">#</span></span>
<span class="code-line"><span class="c1"># This program is distributed in the hope that it will be useful, but WITHOUT</span></span>
<span class="code-line"><span class="c1"># ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or</span></span>
<span class="code-line"><span class="c1"># FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for</span></span>
<span class="code-line"><span class="c1"># more details.</span></span>
<span class="code-line"><span class="c1">#</span></span>
<span class="code-line"><span class="c1"># You should have received a copy of the GNU General Public License along with</span></span>
<span class="code-line"><span class="c1"># this program. If not, see <http://www.gnu.org/licenses/>.</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">ib3.auth</span></span>
<span class="code-line"><span class="kn">import</span> <span class="nn">irc.bot</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="n">NICKNAME</span><span class="o">=</span><span class="s1">'your account name here'</span></span>
<span class="code-line"><span class="n">PASSWORD</span><span class="o">=</span><span class="s1">'your password here'</span></span>
<span class="code-line"><span class="n">CHANNELS</span><span class="o">=</span><span class="p">[</span><span class="s1">'##sasl_test'</span><span class="p">]</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="k">class</span> <span class="nc">ExampleSaslBot</span><span class="p">(</span><span class="n">ib3</span><span class="o">.</span><span class="n">auth</span><span class="o">.</span><span class="n">SASL</span><span class="p">,</span> <span class="n">irc</span><span class="o">.</span><span class="n">bot</span><span class="o">.</span><span class="n">SingleServerIRCBot</span><span class="p">):</span></span>
<span class="code-line"> <span class="c1"># Add your ``on_*`` handlers here</span></span>
<span class="code-line"> <span class="k">pass</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="n">bot</span> <span class="o">=</span> <span class="n">ExampleSaslBot</span><span class="p">(</span></span>
<span class="code-line"> <span class="n">server_list</span><span class="o">=</span><span class="p">[(</span><span class="s1">'chat.freenode.net'</span><span class="p">,</span> <span class="mi">6667</span><span class="p">)],</span></span>
<span class="code-line"> <span class="n">nickname</span><span class="o">=</span><span class="n">NICKNAME</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">realname</span><span class="o">=</span><span class="n">NICKNAME</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">ident_password</span><span class="o">=</span><span class="n">PASSWORD</span><span class="p">,</span></span>
<span class="code-line"> <span class="n">channels</span><span class="o">=</span><span class="n">CHANNELS</span><span class="p">,</span></span>
<span class="code-line"><span class="p">)</span></span>
<span class="code-line"><span class="n">bot</span><span class="o">.</span><span class="n">start</span><span class="p">()</span></span>
</pre></div>
<p>The <code>ib3.auth.SASL</code> mixin will take care of these things for you behind the
scenes:</p>
<ul>
<li>Send <code>CAP REQ :sasl</code> as soon as <code>SingleServerIRCBot</code> knows it has connected</li>
<li>Listen for a <code>CAP ACK :sasl</code> response from the server</li>
<li>Send an <code>AUTHENTICATE PLAIN</code> message to start the auth handshake</li>
<li>Wait for an <code>AUTHENTICATE +</code> response</li>
<li>Send <code>AUTHENTICATE <base64 encoded 'username\0username\0password'></code> SASL
PLAIN request</li>
<li>Wait for a <code>903 :SASL authentication successful</code> response</li>
<li>Send a <code>CAP END</code> message to finish the handshake</li>
</ul>
<p>Both Jouncebot and Stashbot have been using this code for a few weeks with no
problems yet. If you try it out and find issues, please <a href="https://github.com/bd808/python-ib3/issues">report a bug</a> and
I'll see if I can figure out how to make things work better.</p>Switching from Octopress to Pelican2017-03-01T04:19:06+00:002017-03-01T04:19:06+00:00Bryan Davistag:bd808.com,2017-03-01:/blog/2017/03/01/switching-from-octopress-to-pelican/<p>I fell down a rabbit hole a few days ago. I wanted to write a blog post about
<a href="https://python-ib3.readthedocs.io/">my new irc library</a>, but the <a href="//github.com/rbenv/rbenv">rbenv</a> I had setup to run <a href="http://octopress.org/">Octopress</a> was all
messed up. I stated poking around to try and remind myself how to get it
working again and eventually decided that I should really look for a new
static site generator written in language that I like to use. I ended up
picking <a href="https://blog.getpelican.com/">Pelican</a>.</p>
<p>I fell down a rabbit hole a few days ago. I wanted to write a blog post about
<a href="https://python-ib3.readthedocs.io/">my new irc library</a>, but the <a href="//github.com/rbenv/rbenv">rbenv</a> I had setup to run <a href="http://octopress.org/">Octopress</a> was all
messed up. I stated poking around to try and remind myself how to get it
working again and eventually decided that I should really look for a new
static site generator written in language that I like to use. I ended up
picking <a href="https://blog.getpelican.com/">Pelican</a>.</p>
<p>There are <a href="https://www.google.com/search?q=octopress+to+pelican">plenty of blog posts</a> already that cover the basics, so I won't try
to give a complete walk through. I mostly used the guides by <a href="https://jakevdp.github.io/blog/2013/05/07/migrating-from-octopress-to-pelican/">Jake Vanderplas</a>
and <a href="http://jhshi.me/2015/10/11/migrating-from-octopress-to-pelican/">Jinghao Shi</a> along with the <a href="http://docs.getpelican.com/en/stable/">manual</a>. I have blogged before about how
I setup Octopress to make <a href="/blog/2012/04/14/using-github-issues-for-comments/">GitHub issues for comments</a>. I ported this
functionality to Pelican with a couple of commits:</p>
<ul>
<li><a href="https://github.com/bd808/bd808.github.com/commit/412c0b3fc45dacda2bd2800ca5b2d8a49d9ee46e">bd808/bd808.github.com@412c0b3</a> adds the javascript and template changes
needed to render comments and a link to the GitHub issue for a given post.</li>
<li><a href="https://github.com/bd808/bd808.github.com/commit/fe45c78fd96577923f958f1c743f8572c0714829">bd808/bd808.github.com@fe45c78</a> adds a <code>new_post</code> target to my <a href="http://www.fabfile.org/">fabric</a>
file which creates an issue in the GitHub project and adds the needed
metadata to a stub Markdown file.</li>
</ul>Puppet file recurse pitfall2014-09-30T20:44:13-06:002014-09-30T20:44:13-06:00Bryan Davistag:bd808.com,2014-09-30:/blog/2014/09/30/puppet-file-recurse-pitfall/<p><a href="http://puppetlabs.com/">Puppet</a> has become my go to system management tool in no small part because
it is the tool that the operations group at <a href="https://wikimediafoundation.org/wiki/Home">$DAYJOB</a> has standardized on
for our production infrastructure management. It took quite a while for me to
get the hang of how Puppet does what it does, but today I'd say I'm a fairly
decent Puppet programmer. Every once in a while however I stumble on something
new and surprising.</p>
<p><a href="http://puppetlabs.com/">Puppet</a> has become my go to system management tool in no small part because
it is the tool that the operations group at <a href="https://wikimediafoundation.org/wiki/Home">$DAYJOB</a> has standardized on
for our production infrastructure management. It took quite a while for me to
get the hang of how Puppet does what it does, but today I'd say I'm a fairly
decent Puppet programmer. Every once in a while however I stumble on something
new and surprising.</p>
<p>A couple of weeks ago I got an interesting bug report from a user about
a collection of Puppet manifests I help manage. The bug was that his testing
server was pegged at 99% CPU utilization for multiple minutes during each
<code>puppet agent</code> run. The bug reporter did a great job of investigating and had
also found that <code>strace</code> showed a repetitive stream of <code>stat()</code> calls while
the process was hogging the CPU.</p>
<p>This also turned out to the be the great kind of bug that was reproducible.
The first testing server I tried the steps from the bug report on showed the
exact same symptoms. I grabbed some very verbose logs by turning on the
<code>--debug</code> logging in <code>puppet agent</code> and logging all of the system calls with
<code>strace</code> at the same time:</p>
<div class="highlight"><pre><span class="code-line"><span></span>$ <span class="nv">TZ</span><span class="o">=</span>UTC strace /usr/bin/ruby /usr/bin/puppet agent --onetime --verbose <span class="se">\</span></span>
<span class="code-line"> --no-daemonize --no-splay --debug <span class="m">2</span>><span class="p">&</span><span class="m">1</span> <span class="p">|</span></span>
<span class="code-line"> tee /tmp/loud-puppet-strace.log</span>
</pre></div>
<p>Looking at the <code>strace</code> messages there was clearly a pattern of <code>stat()</code> calls
for <code>.rb</code> files in unexpected numbers. Puppet was pretty obviously searching
for ruby files that were related to several <a href="https://docs.puppetlabs.com/learning/definedtypes.html">defined types</a> implemented in
our manifests. The log was full of lines like
<code>stat("/var/lib/puppet/lib/puppet/type/git::clone.rb")</code>. A little
searching led me to <a href="https://tickets.puppetlabs.com/browse/PUP-2924">PUP-2924</a> which explained that Puppet was checking to
see if the type had been implemented as a <a href="https://docs.puppetlabs.com/guides/custom_types.html">custom type</a> in Ruby code first
before looking for a defined type in the Puppet manifests. In our case, there
were 17 possible paths for a Ruby class to be loaded from which led to 17
failed stat calls for each defined type in the manifest.</p>
<p>What this did not explain however what why there were so many checks for our
<code>git::clone</code> resource. Two million, two hundred ninety three thousand, six
hundred and seventy seven calls to <code>stat()</code> for the same collection of files
in this one puppet run. Insanity!</p>
<div class="highlight"><pre><span class="code-line"><span></span>$ grep stat<span class="se">\(</span> loud-puppet-strace.log <span class="p">|</span> grep git::clone <span class="p">|</span> wc -l</span>
<span class="code-line"><span class="m">2293677</span></span>
</pre></div>
<p>So now I knew what was happening, but I needed to dig deeper to try and figure
out why it was happening. For this I wanted even more verbose <code>puppet agent</code>
output.</p>
<div class="highlight"><pre><span class="code-line"><span></span>$ <span class="nv">TZ</span><span class="o">=</span>UTC /usr/bin/ruby /usr/bin/puppet agent --onetime --verbose <span class="se">\</span></span>
<span class="code-line"> --no-daemonize --no-splay --debug --trace --evaltrace --noop <span class="m">2</span>><span class="p">&</span><span class="m">1</span> <span class="p">|</span></span>
<span class="code-line"> tee /tmp/puppet-noop.log</span>
</pre></div>
<p>I watched this run happen in real time and took note of what was logged just
before the long pause in logging which accompanied each CPU utilization spike
that I now knew correlated to the outrageous number of <code>stat()</code> calls.</p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="n">Info</span><span class="o">:</span> <span class="n">Git</span><span class="o">::</span><span class="n">Clone</span><span class="o">[</span><span class="n">vagrant</span><span class="o">]:</span> <span class="n">Starting</span> <span class="n">to</span> <span class="n">evaluate</span> <span class="n">the</span> <span class="n">resource</span></span>
<span class="code-line"><span class="n">Info</span><span class="o">:</span> <span class="n">Git</span><span class="o">::</span><span class="n">Clone</span><span class="o">[</span><span class="n">vagrant</span><span class="o">]:</span> <span class="n">Evaluated</span> <span class="k">in</span> <span class="mf">0.01</span> <span class="n">seconds</span></span>
<span class="code-line"><span class="o">[...</span> <span class="n">long</span> <span class="n">pause</span> <span class="n">here</span> <span class="o">...]</span></span>
<span class="code-line"><span class="n">Info</span><span class="o">:</span> <span class="sr">/Stage[main]/Labs_vagrant/File[/srv/</span><span class="n">vagrant</span><span class="o">]:</span> <span class="n">Starting</span> <span class="n">to</span> <span class="n">evaluate</span> <span class="n">the</span> <span class="n">resource</span></span>
<span class="code-line"><span class="n">Info</span><span class="o">:</span> <span class="sr">/Stage[main]/Labs_vagrant/File[/srv/</span><span class="n">vagrant</span><span class="o">]:</span> <span class="n">Evaluated</span> <span class="k">in</span> <span class="mf">0.00</span> <span class="n">seconds</span></span>
</pre></div>
<p>This led to my ah ha moment and an eventual fix. The <code>File[/srv/vagrant]</code>
resource had a definition that looked something like this:</p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="k">file</span> <span class="p">{</span> <span class="s">'/srv/vagrant'</span><span class="p">:</span></span>
<span class="code-line"> <span class="na">recurse</span> <span class="o">=></span> <span class="k">true</span><span class="p">,</span></span>
<span class="code-line"> <span class="na">owner</span> <span class="o">=></span> <span class="s">'vagrant'</span><span class="p">,</span></span>
<span class="code-line"> <span class="na">group</span> <span class="o">=></span> <span class="s">'www-data'</span><span class="p">,</span></span>
<span class="code-line"> <span class="na">require</span> <span class="o">=></span> <span class="na">Git</span><span class="p">::</span><span class="na">Clone</span><span class="p">[</span><span class="s">'vagrant'</span><span class="p">],</span></span>
<span class="code-line"><span class="p">}</span></span>
</pre></div>
<p>The intent of this was to recursively manage the ownership of files in the
/srv/vagrant directory. Seems pretty simple right? <code>chown -R vagrant:www-data
/srv/vagrant</code> would do the same thing at a command prompt.</p>
<p>It turns out however that what Puppet does under the hood is more complicated.
The <code>recurse => true</code> flag makes Puppet do the equivalent of a <code>find</code> command
on the /srv/vagrant directory and then create a new File resource for each file
and directory found that replicates the other settings of the parent type.</p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="k">file</span> <span class="p">{</span> <span class="s">'/srv/vagrant/file1'</span><span class="p">:</span></span>
<span class="code-line"> <span class="na">owner</span> <span class="o">=></span> <span class="s">'vagrant'</span><span class="p">,</span></span>
<span class="code-line"> <span class="na">group</span> <span class="o">=></span> <span class="s">'www-data'</span><span class="p">,</span></span>
<span class="code-line"> <span class="na">require</span> <span class="o">=></span> <span class="na">Git</span><span class="p">::</span><span class="na">Clone</span><span class="p">[</span><span class="s">'vagrant'</span><span class="p">],</span></span>
<span class="code-line"><span class="p">}</span></span>
<span class="code-line"><span class="k">file</span> <span class="p">{</span> <span class="s">'/srv/vagrant/file2'</span><span class="p">:</span></span>
<span class="code-line"> <span class="na">owner</span> <span class="o">=></span> <span class="s">'vagrant'</span><span class="p">,</span></span>
<span class="code-line"> <span class="na">group</span> <span class="o">=></span> <span class="s">'www-data'</span><span class="p">,</span></span>
<span class="code-line"> <span class="na">require</span> <span class="o">=></span> <span class="na">Git</span><span class="p">::</span><span class="na">Clone</span><span class="p">[</span><span class="s">'vagrant'</span><span class="p">],</span></span>
<span class="code-line"><span class="p">}</span><span class="c"></span></span>
<span class="code-line"><span class="c"># ... Lots and lots more file resources here ...</span></span>
<span class="code-line"><span class="k">file</span> <span class="p">{</span> <span class="s">'/srv/vagrant/subdir/subdir/subdir/fileN'</span><span class="p">:</span></span>
<span class="code-line"> <span class="na">owner</span> <span class="o">=></span> <span class="s">'vagrant'</span><span class="p">,</span></span>
<span class="code-line"> <span class="na">group</span> <span class="o">=></span> <span class="s">'www-data'</span><span class="p">,</span></span>
<span class="code-line"> <span class="na">require</span> <span class="o">=></span> <span class="na">Git</span><span class="p">::</span><span class="na">Clone</span><span class="p">[</span><span class="s">'vagrant'</span><span class="p">],</span></span>
<span class="code-line"><span class="p">}</span></span>
</pre></div>
<p>All of these resources are added to the internal DAG (Directed Acyclic Graph)
and then evaluated one by one. Our /srv/vagrant directory can have a lot of
files beneath it. In my testing server there turned out to be about 135,000
files. So Puppet added 135,000 extra nodes to the DAG and as it placed each
one it called <code>stat()</code> 17 times to see if there was a Ruby class providing the
<code>git::clone</code> resource that Puppet wanted to ensure that the new File resource
followed.</p>
<p><strong><em>YIKES!</em></strong></p>
<p>I think there are probably several opportunities here for optimizations in the
Puppet implementation itself. Caching the implementation of the <code>git::clone</code>
resource would be one that comes to mind pretty quickly. Making recursive File
resources operate based on one node rather than N would be another. There is
probably some kind of graph insertion change that could be made as well. If
I was more comfortable with Ruby I might take a stab at one or more of these
myself.</p>
<p>To fix the bug at hand however I looked around and found that we really didn't
need to bother with the recursive <code>chown</code> at all, so I was able to remove the
whole <code>File[/srv/vagrant]</code> resource from the manifest and let our <code>git::clone</code>
implementation create the directory when it performed the initial git
repository clone operation.</p>GnuPG key transition statement2014-05-15T22:33:39-06:002014-05-15T22:33:39-06:00Bryan Davistag:bd808.com,2014-05-15:/blog/2014/05/15/gnupg-key-transition-statement/<div class="highlight"><pre><span class="code-line"><span></span>-----BEGIN PGP SIGNED MESSAGE-----</span>
<span class="code-line">Hash: SHA1,SHA512</span>
<span class="code-line"></span>
<span class="code-line">I am transitioning GPG keys from an old 1024-bit DSA key to a new</span>
<span class="code-line">4096-bit RSA key. The old key will continue to be valid for some time,</span>
<span class="code-line">but I prefer all new correspondence to be encrypted to the new key, and</span>
<span class="code-line">will …</span></pre></div><div class="highlight"><pre><span class="code-line"><span></span>-----BEGIN PGP SIGNED MESSAGE-----</span>
<span class="code-line">Hash: SHA1,SHA512</span>
<span class="code-line"></span>
<span class="code-line">I am transitioning GPG keys from an old 1024-bit DSA key to a new</span>
<span class="code-line">4096-bit RSA key. The old key will continue to be valid for some time,</span>
<span class="code-line">but I prefer all new correspondence to be encrypted to the new key, and</span>
<span class="code-line">will be making all signatures going forward with the new key.</span>
<span class="code-line"></span>
<span class="code-line">This transition document is signed with both keys to validate the</span>
<span class="code-line">transition.</span>
<span class="code-line"></span>
<span class="code-line">If you have signed my old key, I would appreciate signatures on my new</span>
<span class="code-line">key as well, provided that your signing policy permits that without</span>
<span class="code-line">re-authenticating me.</span>
<span class="code-line"></span>
<span class="code-line">The old key, which I am transitioning away from, is:</span>
<span class="code-line"></span>
<span class="code-line"> pub 1024D/0x41E5C23F0F8E76D6 [created: 2004-10-14]</span>
<span class="code-line"> Key fingerprint = FE97 560A 1C17 F268 1A20 5B80 41E5 C23F 0F8E 76D6</span>
<span class="code-line"></span>
<span class="code-line">The new key, to which I am transitioning, is:</span>
<span class="code-line"></span>
<span class="code-line"> pub 4096R/0xC139E10FD9F20FC1 [created: 2014-05-16]</span>
<span class="code-line"> Key fingerprint = 7DFA 4AEF AC15 8BFC 151D 2DD8 C139 E10F D9F2 0FC1</span>
<span class="code-line"></span>
<span class="code-line">To fetch the full new key from a public key server using GnuPG, run:</span>
<span class="code-line"></span>
<span class="code-line"> gpg --keyserver keys.gnupg.net --recv-key 0xC139E10FD9F20FC1</span>
<span class="code-line"></span>
<span class="code-line">If you have already validated my old key, you can then validate that the</span>
<span class="code-line">new key is signed by my old key:</span>
<span class="code-line"></span>
<span class="code-line"> gpg --check-sigs 0xC139E10FD9F20FC1</span>
<span class="code-line"></span>
<span class="code-line">If you then want to sign my new key, a simple and safe way to do that is</span>
<span class="code-line">by using caff as follows:</span>
<span class="code-line"></span>
<span class="code-line"> caff 0xC139E10FD9F20FC1</span>
<span class="code-line"></span>
<span class="code-line">Please contact me via e-mail at &lt;bd808@bd808.com&gt; if you have any</span>
<span class="code-line">questions about this document or this transition.</span>
<span class="code-line"></span>
<span class="code-line">Bryan Davis</span>
<span class="code-line">&lt;bd808@bd808.com&gt;</span>
<span class="code-line">2014-05-15</span>
<span class="code-line"></span>
<span class="code-line">-----BEGIN PGP SIGNATURE-----</span>
<span class="code-line"></span>
<span class="code-line">iEYEARECAAYFAlN1m3cACgkQQeXCPw+OdtYXEwCfXUThM0JsPacy1bCBQ6rZpWRY</span>
<span class="code-line">dAcAoIMg91zhQlgo2DJCu3o9BUzCqEJuiQIcBAEBCgAGBQJTdZt3AAoJEEhMmO+k</span>
<span class="code-line">BO60xv0QAJEV8VYVqpIdEoZWRYw6sGJVmkTCs2rC4OC68/W+1e41hgPqE+i+6ACU</span>
<span class="code-line">2MhwusMQhsBu1QpyeWXTOEPMU4rvwwlMeQnIlg+DEFGn2k3qJfxeYooO1Ni9n0US</span>
<span class="code-line">fb676RByWnaAZUYPebNrmTvk5bv/M5BSU8XDfPmDsFk5hzeOa1j1kw9Loffr74LL</span>
<span class="code-line">LJozHb8Uj9fMZj1f8SzqlhyqPVUWqF3AEE3Dl14Wl2FH507ZzpMwuOetj65KxeiJ</span>
<span class="code-line">Iee2Hhu6TvQcqs6erxMrsVFxuYz9s1eJzo7feEL22Z8Nm46KSF6x43lpt8ebiKDU</span>
<span class="code-line">zxzdjLBRQOYf3KcCHE2HvbGxPqEfKkwmCJcd1a3Bd/7sgPXrKsJbeCg8LD2x8aTT</span>
<span class="code-line">DHGXUEVbMv8r3qMAlKXxJ8iBJ9AvdG0nKneVJ8gB6YkCPlSuDlh2bL3CrMPQ5Db+</span>
<span class="code-line">vtI0EwuGHSMocWX5cns3t31/iWdoOJ/8lvXJoauT+TVmenmhQ0mU71+whVlnahhr</span>
<span class="code-line">fhKqsZHM4Nryve8LOntndzAIRUK9EZom1ZGfxzEgfgheg0boMfbk9+dS38zVxjmx</span>
<span class="code-line">EZ4JuTVvAUv4ZgG553JaKed278wNPxdXSqaXggV+HAceFkaW80M6uQhvOCXX+T05</span>
<span class="code-line">1HCfl3sQmGkYZ1f3DPrcur0jm+PkHPB4Jw29wogBFU0d7dDJQ0qv</span>
<span class="code-line">=l8Kz</span>
<span class="code-line">-----END PGP SIGNATURE-----</span>
</pre></div>How do you know when you're done?2014-01-14T21:48:20-07:002014-01-14T21:48:20-07:00Bryan Davistag:bd808.com,2014-01-14:/blog/2014/01/14/how-do-you-know-when-youre-done/<p>In scrum a story is "Done" when it meets the team's shared "Definition of
Done". The definition of done is roughly a list of requirements that all parts
of the software increment must adhere to to be called complete. Like most
things in scrum the implementation details are left to the team to decide.
When I was first working with scrum I had a hard time finding examples of what
a typical definition of done would include. Most scrum authors (and even many
trainers) wave their hands and say that it's too specific to the team and
their environment to generalize.</p>
<p>In scrum a story is "Done" when it meets the team's shared "Definition of
Done". The definition of done is roughly a list of requirements that all parts
of the software increment must adhere to to be called complete. Like most
things in scrum the implementation details are left to the team to decide.
When I was first working with scrum I had a hard time finding examples of what
a typical definition of done would include. Most scrum authors (and even many
trainers) wave their hands and say that it's too specific to the team and
their environment to generalize.</p>
<p>Intellectually I agree with this, but pragmatically I think that having some
sort of rough draft of ideas to start from makes writing the first draft
easier. This particular definition of Done is written from the perspective of
a cross functional team responsible for implementing features in a product. It
does not include Done criteria for the operations or support teams that will
maintain the deployed software or assist customers in its use. It does however
include deliverables that must be produced by the development team to support
those additional teams.</p>
<p>This list taken as a whole looks pretty daunting. It turns out that producing
production ready software is hard work. It is such hard work that it takes
a group of well trained individuals working as a team to complete properly.
This list is a recipe that can and should be used by the team to ensure that
they produce an increment that is worthy of their combined energy. When used
properly it will increase the reputation and worth of the team, their product
and the organization.</p>
<h2>Done with grooming a story</h2>
<p>A groomed story is <strong><em>clear, feasible and testable</em></strong>.</p>
<dl>
<dt>Business Goal described</dt>
<dd>Why will we build this?</dd>
<dt>Acceptance criteria defined</dt>
<dd>What will it do?</dd>
<dt>Tasks identified</dt>
<dd>How will we do it?</dd>
<dt>Story points estimated</dt>
<dd>What will it cost?</dd>
</dl>
<p>It may take several iterations to achieve this level of clarity. In fact
anything that can be quickly groomed is necessarily trivial. It may still take
significant time to implement, but it would have to be a variation on work
that has already been done that is understood by the whole team.</p>
<p>Themes, Epics and large stories will need to be decomposed into smaller parts.
This must happen recursively until the smallest parts are describable using
the criteria established above.</p>
<p>Spikes or other research may need to be done to remove uncertainty about new
tech or legacy impact. These things are stories in their own right and should
be treated as such. R&D must be a traceable expense and is just as important as
the final product/feature.</p>
<h2>Done with a story</h2>
<dl>
<dt>Everything from "Done with grooming a story"</dt>
<dd>A story must be groomed before it can be implemented.</dd>
<dt>Design complete</dt>
<dd>Design is not one size fits all. Some stories must have UML and detailed
functional descriptions. Others will only need a statement of "do this just
like we always do an X feature." The level of design required should be
determined during grooming by the team.</dd>
<dt>Design artifacts in wiki/bug tracker/other</dt>
<dd>Design isn't complete until it's tangible artifacts are available to the
team and the business.</dd>
<dt>Design reviewed by peers</dt>
<dd>Similar to a code review, design should get a once over by at least one
tangentially involved party to ensure that the level of detail is appropriate
to the story and that the proposed implementation makes sense.</dd>
<dt>Code complete</dt>
<dd>All code for the story has been written.</dd>
<dt>Unit tests written</dt>
<dd>Unit tests have been written to verify that the code works at the most basic
level. This can be done via TDD or code-then-test as best suits the team and
the story.</dd>
<dt>All code checked into version control</dt>
<dd>Feature code and tests are committed to version control.</dd>
<dt>All unit tests passing</dt>
<dd>Unit tests are passing in all testable environments.</dd>
<dt>Automated code checks passing</dt>
<dd>Coding style, lint and other common automated code quality measurements are passing
according to the organization's definition of passing.</dd>
<dt>CI tests passing</dt>
<dd>Automated tests in the continuous integration environment are passing.</dd>
<dt>Peer code review completed</dt>
<dd>A code review has been completed involving at least one tangentially
involved party.</dd>
<dt>Material defects from code review addressed</dt>
<dd>All questions and defects raised in the code review have been addressed.</dd>
<dt>All acceptance tests (manual and automated) identified, written and passing.</dt>
<dd>Given/When/Then style or other detailed acceptance tests for the story have
been written and verified either with automated tests or manual testing.
Automated tests are preferred as they do not increase the overall manual
testing load of the product.</dd>
<dt>Help/documentation updated</dt>
<dd>"Just enough" help and documentation has been produced so that the feature
can be used by clients, maintained by developers and supported by customer
service and operations.</dd>
<dt>Release notes updated</dt>
<dd>Deliverable artifacts and deployment procedures have been documented.</dd>
</dl>
<h2>Done with a sprint</h2>
<p><dl>
<dt>Everything from "Done with a story"</dt>
<dd>All stories in the sprint must be done (or returned to the backlog) for the
sprint to be done.</dd>
<dt>Released to beta/integration environment</dt>
<dd>The deliverables identified in the release notes for the Sprint must be
deployed in the beta/integration environment. </dd>
<dt>Demoed in beta/integration environment (UAT)</dt>
<dd>The demonstration of the increment to Product Owner and other Stakeholders
must be performed from the beta/integration environment.</dd>
<dt>Approved by Stakeholders</dt>
<dd>The increment must be approved following UAT.</dd>
<dt>CI/automated tests passing</dt>
<dd>All automated tests against the product must be passing. </dd>
<dt>Integration tests passing</dt>
<dd>Manual integration tests for the product must be passing.</dd>
<dt>Regression tests passing</dt>
<dd>Manual regression tests for the product must be passing.</dd>
<dt>Code coverage for automated tests meets acceptable guidelines.</dt>
<dd>Code coverage measurements for unit tests must be within acceptable ranges.</dd>
<dt>Performance tests passing</dt>
<dd>Performance/scaling tests must return results within acceptable ranges.</dd>
<dt>Diagrams/documentation updated to match final state</dt>
<dd>Documentation for design, implementation, deployment, support and use must
be updated to match the completed increment.</dd>
<dt>Bugs closed or pushed into backlog</dt>
<dd>Defects identified in UAT, QA and development must be resolved or appended
to the backlog for Product Owner triage.</dd>
<dt>Unfinished stories pushed into backlog</dt>
<dd>Any work in the sprint which does not meet this definition of done will be
returned to the backlog. The Sprint isn't done as long as any non-done issues
are associated with it.</dd>
</dl></p>
<h2>Done with a QA/staging release</h2>
<p><dl>
<dt>Everything from "Done with a sprint"</dt>
<dd>All Sprints that are to be included in the release must be Done.</dd>
<dt>Operations guide updated and approved by Ops</dt>
<dd>The support documentation delivered to Ops via the wiki must be updated and
those updates must be approved (UAT) by the Operations team.</dd>
<dt>Automated tests passing</dt>
<dd>All automated tests available for the QA/Staging environment must be
passing.</dd>
</dl></p>
<h2>Done with a production release</h2>
<p><dl>
<dt>Everything from "Done with a QA/Staging Release"</dt>
<dd>A successful QA/Staging release is a prerequisite for a Production release.</dd>
<dt>Stress/Load tests passing</dt>
<dd>Stress/Load testing in the QA/Staging environment must return results within
acceptable ranges.</dd>
<dt>Network/Component diagrams updated</dt>
<dd>Documentation for design, implementation, deployment, support and use must
be updated to match the proposed release.</dd>
</dl></p>FileVault2 Hacks2013-12-09T21:35:52-07:002013-12-09T21:35:52-07:00Bryan Davistag:bd808.com,2013-12-09:/blog/2013/12/09/filevault2-hacks/<p>Mac OS X 10.7 introduced a whole disk encryption service called
<a href="http://support.apple.com/kb/ht4790">FileVault2</a>. This allows you to use AES 128 encryption to protect your
data. This is a great feature but it has a few small drawbacks. It uses the
password of your primary user account to unlock the system. I'm a fan of
strong passwords but for encryption I'd prefer to use a longer pass phrase for
increased entropy. Second the EFI-boot screen that is used to get the password
to decrypt the disk shows the display name of all usersthat can unlock the
system rather than blank fields for both username and password. This leaks
information that I would really rather not leak. Fortunately I've found
a little hack to work around both of these issues.</p>
<p>Mac OS X 10.7 introduced a whole disk encryption service called
<a href="http://support.apple.com/kb/ht4790">FileVault2</a>. This allows you to use AES 128 encryption to protect your
data. This is a great feature but it has a few small drawbacks. It uses the
password of your primary user account to unlock the system. I'm a fan of
strong passwords but for encryption I'd prefer to use a longer pass phrase for
increased entropy. Second the EFI-boot screen that is used to get the password
to decrypt the disk shows the display name of all usersthat can unlock the
system rather than blank fields for both username and password. This leaks
information that I would really rather not leak. Fortunately I've found
a little hack to work around both of these issues.</p>
<p>The key to my fix lies in this statement from the documentation:</p>
<blockquote>
<p>Users not enabled for FileVault unlock are only able to log into the
computer after an unlock-enabled user has started or unlocked the drive.
Once unlocked, the drive remains unlocked and available to all users, until
the computer is restarted.
<small><a href="http://support.apple.com/kb/ht4790">"OS X: About FileVault 2"</a></small></p>
</blockquote>
<p>My fix is to create a new local user account that will only be used to unlock
the disk encryption key. This will provide a fix for both issues. Since this
account won't be my primary account I can give it a much longer password
without risk of <a href="https://en.wikipedia.org/wiki/Repetitive_strain_injury">RSI</a> every time that OS X prompts me for an administrator
password to install or update software. I can also give the user an innocuous
display name to be shown on the unlock screen.</p>
<ol>
<li>Create a new account from the <em>Users & Groups</em> control panel:</li>
<li>New Account: Standard</li>
<li>Full Name: <strong><em>*</em></strong><strong>*</strong></li>
<li>Account name: encrypt</li>
<li>Password: omg this is a really long passphrase for me to remember</li>
<li>Follow the <a href="http://support.apple.com/kb/ht4790">instructions</a> for enabling FileVault 2 and chose the new
user as the only user who can unlock the disk.</li>
</ol>
<p>If you already have FileVault 2 enabled you will need to remove the decryption
right from the existing users. The easiest way I've found to do this is using
the <code>fdesetup</code> command line tool. <code>sudo fdesetup list</code> will show you the
accounts that are enabled. <code>sudo fdesetup remove -user bd808</code> will remove the
certificate for the <em>bd808</em> user.</p>
<p>One last step is to make the new <em>encrypt</em> user log out as soon as they log
in. This will return control to the normal OS X login system where you can
configure the login screen to display username and password prompts instead of
a list of local user accounts. There are probably several ways to do this, but
I chose to make a small application that executes this apple script command:</p>
<p><em>logout</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="k">ignoring </span><span class="nb">application responses</span></span>
<span class="code-line"> <span class="k">tell</span> <span class="nb">application</span> <span class="s2">"loginwindow"</span> <span class="k">to</span> «<span class="nb">event aevtlogo</span>»</span>
<span class="code-line"><span class="k">end</span> <span class="k">ignoring</span></span>
</pre></div>Yaml 1.1.1 PECL Module Released2013-11-18T22:20:43-07:002013-11-18T22:20:43-07:00Bryan Davistag:bd808.com,2013-11-18:/blog/2013/11/18/yaml-111-pecl-module-released/<p>I'm glad to announce that I finally got around to releasing the bug fix
version of the <a href="http://pecl.php.net/package/yaml">YAML PECL module</a> that
I announced on 2013-04-23. Version 1.1.1 fixes several long standing bugs:</p>
<ul>
<li><a href="https://bugs.php.net/bug.php?id=61770">#61770</a> Crash on nonunicode character</li>
<li><a href="https://bugs.php.net/bug.php?id=61923">#61923</a> Detect_scalar_type() is not aware of base 60 representation</li>
<li><a href="https://bugs.php.net/bug.php?id=63086">#63086</a> Compiling PHP with YAML as static extension fails</li>
<li><a href="https://bugs.php.net/bug.php?id=64019">#64019</a> Segmentation fault if yaml anchor ends with a colon</li>
<li><a href="https://bugs.php.net/bug.php?id=64694">#64694</a> Segfault when array used as mapping key</li>
</ul>
<p>I'm glad to announce that I finally got around to releasing the bug fix
version of the <a href="http://pecl.php.net/package/yaml">YAML PECL module</a> that
I announced on 2013-04-23. Version 1.1.1 fixes several long standing bugs:</p>
<ul>
<li><a href="https://bugs.php.net/bug.php?id=61770">#61770</a> Crash on nonunicode character</li>
<li><a href="https://bugs.php.net/bug.php?id=61923">#61923</a> Detect_scalar_type() is not aware of base 60 representation</li>
<li><a href="https://bugs.php.net/bug.php?id=63086">#63086</a> Compiling PHP with YAML as static extension fails</li>
<li><a href="https://bugs.php.net/bug.php?id=64019">#64019</a> Segmentation fault if yaml anchor ends with a colon</li>
<li><a href="https://bugs.php.net/bug.php?id=64694">#64694</a> Segfault when array used as mapping key</li>
</ul>
<p>It also includes a small but important patch from a community member who
discovered that I had left the <code>yaml_emit_file()</code> method marked as
unimplemented when it actually was fully functional.</p>
<p>I hope the users of this extension will find the changes to be useful. I also
welcome bug reports, feature requests and patches from the community. I would
especially appreciate it if someone found the time become the maintainer of
a Debian package for the project to make it a little easier for some users to
install.</p>Planning Work in a Sprint2013-10-27T20:05:00+00:002013-10-27T20:05:00+00:00Bryan Davistag:bd808.com,2013-10-27:/blog/2013/10/27/planning-work-in-a-sprint/<p>We've been having some discussions at <code>$DAYJOB</code> about process and
methodologies. The topic of late is <a href="https://scrum.org/">scrum</a> and how it may
or may not be helpful for the particular group I work with. I've been
providing some anecdotal input based my past experience with scrum and other
methodologies/frameworks/practices and asking questions about what problems
the group is hoping to find new solutions for.</p>
<p>I started to write a big wall o' text™ email about a particular topic and then
decided that maybe a blog post would be a better way to work through my idea.
So dear reader<sup id="fnref-1"><a class="footnote-ref" href="#fn-1">1</a></sup>, here are some of my highly
opinionated and mostly unsubstantiated thoughts about a process that a group
of people could use to plan a scrum sprint (<em>or really any other iterative
unit of work)</em>.</p>
<p>We've been having some discussions at <code>$DAYJOB</code> about process and
methodologies. The topic of late is <a href="https://scrum.org/">scrum</a> and how it may
or may not be helpful for the particular group I work with. I've been
providing some anecdotal input based my past experience with scrum and other
methodologies/frameworks/practices and asking questions about what problems
the group is hoping to find new solutions for.</p>
<p>I started to write a big wall o' text™ email about a particular topic and then
decided that maybe a blog post would be a better way to work through my idea.
So dear reader<sup id="fnref-1"><a class="footnote-ref" href="#fn-1">1</a></sup>, here are some of my highly
opinionated and mostly unsubstantiated thoughts about a process that a group
of people could use to plan a scrum sprint (<em>or really any other iterative
unit of work)</em>.</p>
<h2>Pick some work you think you can get done</h2>
<p>Step one, pick some work. Sounds easy, but pick it from where? Well that's
a damn good question but one that's going to depend on your environment. For
the sake of this post let's assume that you have access to an ordered list of
features that need to be implemented. Let's further assume that the team and
the stakeholders have talked about these features a little and that the team
has reached a general consensus about how big the top few features are
relative to features the team has worked on in the past. Scrum calls this
a "groomed backlog", but you can call it whatever you'd like.</p>
<p>Now that you know where the work comes from, pick some. How much? Well, as
much as the team thinks it can get done in the iteration. Without knowing your
team and the length of the iteration and how tricky the problems are I can't
tell you. Just go with your collective gut and pick some. If you pick too
little you can always come back and get more. If you pick too much the team
can use that experiential data to adjust when choosing for the next iteration.
Just pick some work for now and adjust in the future based on what happens
during the iteration<sup id="fnref-2"><a class="footnote-ref" href="#fn-2">2</a></sup>.</p>
<h2>Figure out what ties the work together</h2>
<p>Step two, come up with a narrative about why you chose the work. A list of
features and bugs you want to implement is a great start, but you can do
better. It will be a lot easier for the team to make good choices during the
iteration if they have a more noble goal than "cross off all the things on
this list." If the goal is just to get each item done it's more likely that
people will think of each part in isolation rather than thinking about how
this work builds on what came before and enables more enhancements in the
future.</p>
<p>This step may lead you to switch out some of the things you've chosen with
other things that are in the backlog to make a more cohesive story. That's ok
as long as you keep the most important thing. After all that's the MOST
important thing; if you get it done plus some other stuff and everything works
people should be happy.</p>
<p>This narrative you've created and the work that supports it are the forecast
for the iteration. The product owner can take this information back to the
rest of the stakeholders and tell them what to expect to hear about in the
demonstration meeting at the end of the iteration. Be careful not to tell
them that all of this work will be completed. The team has said they'll try
to do this but they can't promise that it will get done any more than the
stakeholders can promise how much money will be raised or how many new
customers will be acquired.</p>
<h2>Figure out how to do the work</h2>
<p>The last step before you close the planning meeting and get back to "real
work" is to figure out how to actually <em>do</em> the work. We're talking about
agile practices here so nobody should expect a gantt chart chart or
a architecture document, but <strike>anarchists</strike> agile teams need
enough of a plan to do today's work efficiently. The product owner doesn't
need to stick around for this half of the meeting. The team should have enough
information from the feature descriptions already given to make the tactical
plan.</p>
<p>I'm sure there are other methods that would work as well, but I've personally
had success with a process that starts with finding dependencies. The team
looks at the stories and tries to determine their rough interdependencies. The
goal here is to identify communication interfaces that need to be specified
and sequential implementation order dependencies. The team also looks for
areas of uncertainty that could be resolved with tech spikes and/or further
investigation of the requirements.</p>
<p>Once you've got the dependencies sorted, start breaking down the most obvious
starting point features. Make a list of the smaller tasks that need
to be completed to finish the feature. Repeat the process by breaking those
tasks down into even smaller tasks. Stop when the leaf tasks are "small
enough". My rule of thumb is that something that feels like it will take 3-5
ideal hours is small enough. Getting smaller than that early on is probably
a waste of time, but staying larger leaves more uncertainty and risk in
the plan. Scrum calls this step "story decomposition".</p>
<p>The list of decomposed tasks that the team has created is the start of the
iteration backlog. Just like the product backlog this needs to be put in order
so that when a team member or pair needs more work to do they can just pull
the next most important thing in their area of expertise off of the backlog.
You'll reorder the list as the iteration progresses, but get started by
ordering the tasks you just decomposed.</p>
<p>If you only have a few features to break down, continue to do the work as
a group. If there are quite a few to get through you can split up into
appropriately skilled groups and work in parallel. Depending on your team and
the time you have left in the meeting (two hours per week of iteration is
a suggested total duration), you may have time to outline all of the features.
You need to at least outline enough to keep the whole team occupied for the
rest of today and tomorrow.</p>
<p>If you have some high risk things to accomplish in the iteration try to break
them down as early as you can so that someone (or some pair) can start on the
tech spikes or API design or whatever sooner rather than later. Don't forget
to put a "decompose feature X" task onto the backlog for any stories that you
didn't have time to get to by the end of the time box.</p>
<h2>Get to work</h2>
<p>Now you've got a list of features to implement, a narrative about why these
things go together and at least a day or two of granular tasks to start
working on. Each team member or pair now needs to select one thing to begin
working on. Start by choosing the highest priority task that you have the
skill set to accomplish. When you get <strong>Done</strong><sup id="fnref-3"><a class="footnote-ref" href="#fn-3">3</a></sup> with the task you've taken
come back to the backlog and chose another. Don't forget to mark the things
you are working on as in progress by whatever tracking mechanism the team is
using so you and another team member don't duplicate the work.</p>
<p>Whew. That would have been a nasty email to read. I hope you like it better as
a blog post. Don't forget to use inspection and adaptation to refine this
process so that it works well for your team. I think I've given a reasonable
outline of a process that has worked for me in the past, but never be afraid
to look for ways to improve.</p>
<hr>
<div class="footnote">
<hr>
<ol>
<li id="fn-1">
<p><em>Hi Mom!</em> <a class="footnote-backref" href="#fnref-1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn-2">
<p><em>"Inspect and adapt" is a common refrain in scrum.</em> <a class="footnote-backref" href="#fnref-2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn-3">
<p><em>"Definition of Done" is a topic for <a href="/blog/2014/01/14/how-do-you-know-when-youre-done/">another post</a>.</em> <a class="footnote-backref" href="#fnref-3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Creating a Self-signed Code Certificate for XCode2013-10-21T21:38:00+00:002013-10-21T21:38:00+00:00Bryan Davistag:bd808.com,2013-10-21:/blog/2013/10/21/creating-a-self-signed-code-certificate-for-xcode/<p>I wanted to make my own build of <a href="http://www.codeux.com/textual/">Textual</a>
the other day and needed a code signing certificate to complete the build.
I decided to make single, long-lived certificate to that I could reuse for
building multiple applications.</p>
<p>I wanted to make my own build of <a href="http://www.codeux.com/textual/">Textual</a>
the other day and needed a code signing certificate to complete the build.
I decided to make single, long-lived certificate to that I could reuse for
building multiple applications.</p>
<ol>
<li>
<p>Open the "Keychain Access" application</p>
<p><code>bash
open -a "Keychain Access"</code></p>
</li>
<li>
<p>Application menu > Certificate Assistant > Create a Certificate...</p>
<p><img alt="Create a Certificate" src="/static/blog/create-certificate-menu.png"></p>
</li>
<li>
<p>Configure your new certificate</p>
<p><img alt="" src="/static/blog/ca-1.png"></p>
<ul>
<li>Name: Self-signed Applications</li>
<li>Identity Type: Self Signed Root</li>
<li>Certificate Type: Code Signing</li>
<li>[x] Let me override defaults</li>
<li>Continue</li>
<li>Change expiration date</li>
</ul>
<p><img alt="" src="/static/blog/ca-2.png"></p>
<ul>
<li>Validity Period (days): 3650</li>
<li>Continue</li>
</ul>
</li>
<li>
<p>Just keep hitting Continue to accept defaults from here on out</p>
<p><img alt="" src="/static/blog/ca-last.png"></p>
</li>
</ol>
<p>Note: Xcode seems to cache certificate info on startup. If you had XCode open while you created this certificate, restart it.</p>
<p>I have since used this same certificate to build
<a href="http://growl.info/documentation/developer/growl-source-install.php">Growl</a>
and a couple of other apps. I'm thinking that I'll export the public
certificate and import it on my other OSX hosts so I can share the compiled
binaries from machine to machine without needing to recompile them.</p>Managing my laptop with Boxen2013-10-14T22:11:00+00:002013-10-14T22:11:00+00:00Bryan Davistag:bd808.com,2013-10-14:/blog/2013/10/14/managing-my-laptop-with-boxen/<p><a href="https://boxen.github.com/">Boxen</a> is a framework and collection of libraries created by the fine folks
at <a href="https://github.com/">GitHub</a> to make setting up and managing Mac OS X computers easy and
repeatable. Rather than a simple set of shell scripts or other provisioning
tools, Boxen uses <a href="https://puppetlabs.com/">Puppet</a> to automate installing and configuring software.
I don't have the time or space to explain how great Puppet is a configuration
management is, so you'll have to trust me or go do your own research.</p>
<p>Anybody could take a stab at rolling their own collection of Puppet manifests
to manage their laptop or their corporate install base. That's actually
exactly what GitHub did to create Boxen. Having tried (and failed) at doing
just that before I was pretty impressed when I gave Boxen a test drive. GitHub
has not only provided a system that "works for them"; they have also managed
to engineer a reasonably extensible solution for a very complex problem.</p>
<p>You can use your favorite search engine to find folks who can wax poetic about
the magnitude of this accomplishment. Let's get on with a description of what
I've been able to do with it.</p>
<p><a href="https://boxen.github.com/">Boxen</a> is a framework and collection of libraries created by the fine folks
at <a href="https://github.com/">GitHub</a> to make setting up and managing Mac OS X computers easy and
repeatable. Rather than a simple set of shell scripts or other provisioning
tools, Boxen uses <a href="https://puppetlabs.com/">Puppet</a> to automate installing and configuring software.
I don't have the time or space to explain how great Puppet is a configuration
management is, so you'll have to trust me or go do your own research.</p>
<p>Anybody could take a stab at rolling their own collection of Puppet manifests
to manage their laptop or their corporate install base. That's actually
exactly what GitHub did to create Boxen. Having tried (and failed) at doing
just that before I was pretty impressed when I gave Boxen a test drive. GitHub
has not only provided a system that "works for them"; they have also managed
to engineer a reasonably extensible solution for a very complex problem.</p>
<p>You can use your favorite search engine to find folks who can wax poetic about
the magnitude of this accomplishment. Let's get on with a description of what
I've been able to do with it.</p>
<p>I'm using Boxen to manage my <code>$DAYJOB</code> laptop. This was a great place to start
because I had a brand new laptop that needed to be setup and a brand new tool
to use to do it. I started by following the <a href="https://github.com/boxen/our-boxen">bootstrapping instructions</a> to
create <a href="https://github.com/bd808/my-boxen">my own copy of the template project</a>. I made a few changes to the
<a href="https://github.com/bd808/my-boxen/blob/master/manifests/site.pp">site manifest</a> and then started working on a <a href="https://github.com/bd808/my-boxen/blob/master/modules/people/manifests/bd808.pp">manifest for myself</a>.</p>
<p>Along the way I decided I didn't like a few of the decisions that the Boxen
architects had made. As I pointed out earlier, the team behind Boxen
anticipated this and changing most things is as easy as forking a repo, making
your change and updating the <a href="https://github.com/bd808/my-boxen/blob/master/Puppetfile">Puppetfile</a> in your Boxen project.</p>
<p>At the moment I have customized or created these repositories:</p>
<ul>
<li><a href="https://github.com/bd808/my-boxen">my-boxen</a>: My fork of <a href="https://github.com/boxen/our-boxen">boxen/our-boxen</a>.</li>
<li><a href="https://github.com/bd808/puppet-boxen">puppet-boxen</a>: Fork of the core
<a href="https://github.com/boxen/puppet-boxen">boxen/puppet-boxen</a> modules that
installs <a href="http://brew.sh/">Homebrew</a> in <code>/usr/local</code> instead of under
<code>/opt/boxen</code>.</li>
<li><a href="https://github.com/bd808/puppet-dnsmasq">puppet-dnsmasq</a>: Fork of
<a href="https://github.com/boxen/puppet-dnsmasq">boxen/dnsmasq</a> that uses the stock
Homebrew <code>dnsmasq</code> install and provides <code>dnsmasq::address</code> to configure new
address mappings.</li>
<li><a href="https://github.com/bd808/puppet-geektool">puppet-geektool</a>: Original
module to install <a href="http://projects.tynsoe.org/en/geektool/">GeekTool</a>.</li>
<li><a href="https://github.com/bd808/puppet-git">puppet-git</a>: Fork of
<a href="https://github.com/boxen/puppet-git">boxen/puppet-git</a> to use the stock
Homebrew version of <a href="http://git-scm.com/">git</a>.</li>
<li><a href="https://github.com/bd808/puppet-growl">puppet-growl</a>: Fork of
<a href="https://github.com/petems/puppet-growl">petems/puppet-growl</a> that installs
an aging version of <a href="http://growl.info/">Growl</a>. I've since abandoned this
in favor a <a href="http://growl.info/documentation/developer/growl-source-install.php">self-compiled version</a> which I should figure out how to Puppetize.</li>
<li><a href="https://github.com/bd808/puppet-homebrew">puppet-homebrew</a>: Fork of
<a href="https://github.com/nybblr/puppet-homebrew">nybblr/puppet-homebrew</a> that
adds support for installing in <code>/usr/local</code> and using custom Homebrew
<a href="https://github.com/mxcl/homebrew/wiki/brew-tap">taps</a>.</li>
<li><a href="https://github.com/bd808/puppet-monolingual">puppet-monolingual</a>: Original
module to install <a href="http://monolingual.sourceforge.net/">Monolingual</a>.</li>
<li><a href="https://github.com/bd808/puppet-osx">puppet-osx</a>: Fork of
<a href="https://github.com/codec/puppet-osx">codec/puppet-osx</a> that pulls in
patches from <a href="https://github.com/joebadmo/puppet-osx">joebadmo/puppet-osx</a>
and adds a few system settings of my own.</li>
<li><a href="https://github.com/bd808/puppet-slimbatterymonitor">puppet-slimbatterymonitor</a>: Original module to install <a href="http://www.orange-carb.org/SBM/">SlimBatteryMonitor</a>.</li>
</ul>
<p>The one thing I most wish someone would figure out how to do with Boxen/Puppet
is install apps from the <a href="https://www.apple.com/osx/apps/app-store.html">Mac App Store</a>.</p>Hacking GitHub Contributions Calendar2013-04-17T21:06:00+00:002013-04-17T21:06:00+00:00Bryan Davistag:bd808.com,2013-04-17:/blog/2013/04/17/hacking-github-contributions-calendar/<p>GitHub profile pages include a neat visualization of commit history that they
call the "<a href="https://help.github.com/articles/viewing-contributions#contributions-calendar">contributions calendar</a>". This 53x7 grid shows the number of
commits and other GitHub interactions that the user performed on each day for
the last year.</p>
<p><img alt="Example graph" src="/static/blog/timeline.png"></p>
<p>Each cell in the graph is shaded with one of 5 possible colors. These colors
correspond to the <a href="https://en.wikipedia.org/wiki/Quartile">quartiles</a> of the <a href="https://en.wikipedia.org/wiki/Normal_distribution">normal distribution</a> over the range
<code>[0, max(v)]</code> where <code>v</code> is the sum of issues opened, pull requests proposed and commits authored per day.</p>
<p>GitHub profile pages include a neat visualization of commit history that they
call the "<a href="https://help.github.com/articles/viewing-contributions#contributions-calendar">contributions calendar</a>". This 53x7 grid shows the number of
commits and other GitHub interactions that the user performed on each day for
the last year.</p>
<p><img alt="Example graph" src="/static/blog/timeline.png"></p>
<p>Each cell in the graph is shaded with one of 5 possible colors. These colors
correspond to the <a href="https://en.wikipedia.org/wiki/Quartile">quartiles</a> of the <a href="https://en.wikipedia.org/wiki/Normal_distribution">normal distribution</a> over the range
<code>[0, max(v)]</code> where <code>v</code> is the sum of issues opened, pull requests proposed and commits authored per day.</p>
<p>If your all time high for the last year was 100 contributions in a single day,
the cells would color like this:</p>
<table>
<thead>
<tr>
<th align="left">Contributions</th>
<th align="center">Color</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">0</td>
<td align="center"><img alt="gray" src="/static/blog/eeeeee.png" title="#eeeeee"></td>
</tr>
<tr>
<td align="left">1 - 24</td>
<td align="center"><img alt="pale green" src="/static/blog/d6e685.png" title="#d6e685"></td>
</tr>
<tr>
<td align="left">25 - 49</td>
<td align="center"><img alt="light green" src="/static/blog/8cc665.png" title="#8cc665"></td>
</tr>
<tr>
<td align="left">50 - 74</td>
<td align="center"><img alt="green" src="/static/blog/44a340.png" title="#44a340"></td>
</tr>
<tr>
<td align="left">75+</td>
<td align="center"><img alt="dark green" src="/static/blog/1e6823.png" title="#1e6823"></td>
</tr>
</tbody>
</table>
<p>A tweet got me interested in the possibility of gaming the interaction data to
control the display:</p>
<blockquote class="twitter-tweet"><p>GitHub users might find this guy's contribution graph interesting/funny: <a href="https://t.co/xOFjLbqUK2" title="https://github.com/will">github.com/will</a></p>— Peter Cooper (@peterc) <a href="https://twitter.com/peterc/status/322636613018607617">April 12, 2013</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p><a href="https://github.com/will">@will</a> has done something to make his calendar spell "WILL" over and over.
Looking at his contribution activity list it was pretty obvious that this
trick had something to do with the <a href="https://github.com/will/githubprofilecheat">will/githubprofilecheat</a> and/or
<a href="https://github.com/will/githubprofilecheat2">will/githubprofilecheat2</a> repositories.</p>
<p>I did some digging in the <a href="http://git-scm.com/docs">git documentation</a> to see how hard it is to fake
the date on a commit. It turns out that it's as easy as setting an environment
variable. The <code>GIT_AUTHOR_DATE</code> and <code>GIT_COMMITTER_DATE</code> environment variables
can be used to provide <a href="http://git-scm.com/docs/git-commit-tree#_commit_information">git-commit-tree</a> with dates for the author and
commit dates that are attached to each commit object.</p>
<p>Armed with this bit of trivia I decided that I would try to do something
interesting with my contributions graph. I didn't just want to copy <a href="https://github.com/will">@will</a>
and write my name in the graph. I decided that I would pay homage to my
gravatar instead and make a series of <a href="https://en.wikipedia.org/wiki/Glider_%28Conway%27s_Life%29">gliders</a> that ran across the
timeline. The result of my experiment can be seen in the image at the top of
this post.</p>
<p>The script that I used to generate the commits with faked dates is available
in my <a href="https://github.com/bd808/profile-life">bd808/profile-life</a> repository.</p>
<p>The script takes the path to a pattern file and an optional start date as
arguments.</p>
<div class="highlight"><pre><span class="code-line"><span></span>./bin/pattern-to-commits.sh patterns/glider.cells <span class="m">2012</span>-04-15 <span class="p">|</span> sh</span>
</pre></div>
<p>The pattern file is expected to be in the <a href="http://www.conwaylife.com/wiki/Plaintext">plaintext</a> Life format. This
format allows you to specify an on/off pattern. When a cell is "on" the script
will output 23 commits (one per hour) for the corresponding day. "Off" cells
won't generate any commit activity.</p>
<div class="highlight"><pre><span class="code-line"><span></span>!Name: Profile Glider Train</span>
<span class="code-line">!A simulation of a glider cruising across the contributions timeline.</span>
<span class="code-line">O.O...O.O....O..................................................................</span>
<span class="code-line">.OO.O.O..OO...O.O.O...O.O....O..................................................</span>
<span class="code-line">.O...OO.OO..OOO..OO.O.O..OO...O.O.O...O.O....O..................................</span>
<span class="code-line">.................O...OO.OO..OOO..OO.O.O..OO...O.O.O...O.O....O..................</span>
<span class="code-line">.................................O...OO.OO..OOO..OO.O.O..OO...O.O.O...O.O....O..</span>
<span class="code-line">.................................................O...OO.OO..OOO..OO.O.O..OO...O.</span>
<span class="code-line">.................................................................O...OO.OO..OOO.</span>
</pre></div>
<p>The script reads in this file a column at a time using the <code>cut</code> command. It
loops over the characters from the column and when it finds an <code>O</code> it echos
commit commands to stdout:</p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="nv">GIT_AUTHOR_DATE</span><span class="o">=</span><span class="s1">'2013-04-17T20:00'</span> <span class="nv">GIT_COMMITTER_DATE</span><span class="o">=</span><span class="s1">'2013-04-17T20:00'</span> <span class="se">\</span></span>
<span class="code-line">git commit --allow-empty -m <span class="s1">'2013-04-17T20:00'</span></span>
</pre></div>
<p>This output can be piped to bash to apply the commits to the repository.</p>
<p>An interesting extension of this script would be to support all 5 possible
colors. It would also be nice if the script read your current contribution
history to determine how many commits are necessary to hit the 4th quartile
every time. For now these additions are left as an exercise for the reader. :)</p>Using GitHub issues for comments2012-04-14T20:22:00+00:002012-04-14T20:22:00+00:00Bryan Davistag:bd808.com,2012-04-14:/blog/2012/04/14/using-github-issues-for-comments/<p>I was inspired by <a href="http://ivanzuzak.info/2011/02/18/github-hosted-comments-for-github-hosted-blogs.html">Ivan Zuzak's post</a> to try using GitHub issues on the <a href="https://github.com/bd808/bd808.github.com">repository for this blog</a> to collect and display reader comments. I'm using <a href="http://octopress.org/">Octopress</a> to generate the site, so I decided to make some customizations to make applying Ivan's ideas easy for me.</p>
<p>I started by adding a new configuration setting to my <code>_config.yml</code> file: <code>github_comments: true</code>. I'll use this configuration switch to turn the new feature on in other places in the codebase.</p>
<p>I was inspired by <a href="http://ivanzuzak.info/2011/02/18/github-hosted-comments-for-github-hosted-blogs.html">Ivan Zuzak's post</a> to try using GitHub issues on the <a href="https://github.com/bd808/bd808.github.com">repository for this blog</a> to collect and display reader comments. I'm using <a href="http://octopress.org/">Octopress</a> to generate the site, so I decided to make some customizations to make applying Ivan's ideas easy for me.</p>
<p>I started by adding a new configuration setting to my <code>_config.yml</code> file: <code>github_comments: true</code>. I'll use this configuration switch to turn the new feature on in other places in the codebase.</p>
<p>Next I changed the <a href="https://github.com/mojombo/jekyll/wiki/Liquid-Extensions">Liquid template</a> in source/_layout/post.html to include a link to the comment thread for the post. I added this block right after the existing disqus rendering block:</p>
<p><em>source/_layout/post.html</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="x">{% if site.github_comments and page.github_issue_id %}</span></span>
<span class="code-line"><span class="x"><section id="comments"></span></span>
<span class="code-line"><span class="x"> <header></span></span>
<span class="code-line"><span class="x"> <h2>Comments</h2></span></span>
<span class="code-line"><span class="x"> <p>Visit <a href="https://github.com/{{site.github_user}}/{{site.github_user}}.github.com/issues/{{page.github_issue_id}}">this post's issue page on GitHub</a> to add a comment.</p></span></span>
<span class="code-line"><span class="x"> </header></span></span>
<span class="code-line"><span class="x"></section></span></span>
<span class="code-line"><span class="x">{% endif %}</span></span>
</pre></div>
<p>If the <code>github_comments: true</code> flag is set and the <a href="https://github.com/mojombo/jekyll/wiki/yaml-front-matter">yaml front
matter</a> for the post
contains a <code>github_issue_id: N</code> setting, this block with display a link to
issue N in the associated GitHub repository.</p>
<p>Next I wanted to display any current comments. I use a slightly tweaked
version of Ivan's javascript to do this.</p>
<p><em>source/_includes/github_comments.html</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="x">{% if site.github_comments and page.comments == true %}</span></span>
<span class="code-line"><span class="x"><script type="text/javascript"></span></span>
<span class="code-line"><span class="x">$.ajax({</span></span>
<span class="code-line"><span class="x"> url: "https://api.github.com/repos/{{site.github_user}}/{{site.github_user}}.github.com/issues/{{page.github_issue_id}}/comments"</span></span>
<span class="code-line"><span class="x"> , method: "get"</span></span>
<span class="code-line"><span class="x"> , headers: { Accept: "application/vnd.github.full+json" }</span></span>
<span class="code-line"><span class="x"> , error: function(e){}</span></span>
<span class="code-line"><span class="x"> , success: function(resp){</span></span>
<span class="code-line"><span class="x"> var cuser, cuserlink, clink, cbody, cavatarlink, cdate;</span></span>
<span class="code-line"><span class="x"> for (var i=0; i<resp.length; i++) {</span></span>
<span class="code-line"><span class="x"> cuser = resp[i].user.login;</span></span>
<span class="code-line"><span class="x"> cuserlink = "https://github.com/" + resp[i].user.login;</span></span>
<span class="code-line"><span class="x"> clink = "https://github.com/{{site.github_user}}/{{site.github_user}}.github.com/issues/{{page.github_issue_id}}#issuecomment-" + resp[i].url.substring(resp[i].url.lastIndexOf("/")+1);</span></span>
<span class="code-line"><span class="x"> cbody = resp[i].body_html;</span></span>
<span class="code-line"><span class="x"> cavatarlink = resp[i].user.avatar_url;</span></span>
<span class="code-line"><span class="x"> cdate = (new Date(resp[i].created_at)).toLocaleString();</span></span>
<span class="code-line"></span>
<span class="code-line"><span class="x"> $("#comments").append('<div class="comment"><div class="comment-header"><a class="comment-user" href="' + cuserlink + '"><img class="comment-gravatar" src="' + cavatarlink + '" alt="" width="20" height="20"> ' + cuser + '</a><a class="comment-date" href="' + clink + '">' + cdate + '</a></div><div class="comment-body">' + cbody + '</div></div>');</span></span>
<span class="code-line"><span class="x"> }</span></span>
<span class="code-line"><span class="x"> }</span></span>
<span class="code-line"><span class="x">});</span></span>
<span class="code-line"><span class="x"></script></span></span>
<span class="code-line"><span class="x">{% endif %}</span></span>
</pre></div>
<p>I added an include for this new file in source/_includes/after_footer.html to
get it tacked on to each page:</p>
<p><em>source/_includes/after_footer.html</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="x">{% include github_comments.html %}</span></span>
</pre></div>
<p>Those changes plus the OAuth application configuration described in Ivan's post have the blog all setup for comments. The only problem is that I have to remember to manually create an issue on the GitHub side and add it to the yaml front matter for the post. Being a lazy programmer I wanted to get rid of that burden as well. Lucky for me Octopress already has a Rake task that sets up a new blog post. The changes I made here aren't pretty, but they are pragmatic.</p>
<p><em>Rakefile</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="k">def</span> <span class="nf">create_comment_issue</span><span class="p">(</span><span class="n">title</span><span class="p">,</span> <span class="n">url</span><span class="p">)</span></span>
<span class="code-line"> <span class="nb">require</span> <span class="s1">'octopi'</span></span>
<span class="code-line"> <span class="kp">include</span> <span class="no">Octopi</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">authenticated</span> <span class="ss">:config</span> <span class="o">=></span> <span class="s2">"_github.yml"</span> <span class="k">do</span></span>
<span class="code-line"> <span class="n">user</span> <span class="o">=</span> <span class="no">User</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s2">"bd808"</span><span class="p">)</span></span>
<span class="code-line"> <span class="n">repo</span> <span class="o">=</span> <span class="n">user</span><span class="o">.</span><span class="n">repository</span><span class="p">(</span><span class="ss">:name</span> <span class="o">=></span> <span class="s2">"bd808.github.com"</span><span class="p">)</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">issue</span> <span class="o">=</span> <span class="no">Issue</span><span class="o">.</span><span class="n">open</span> <span class="ss">:user</span> <span class="o">=></span> <span class="n">user</span><span class="p">,</span> <span class="ss">:repo</span> <span class="o">=></span> <span class="n">repo</span><span class="p">,</span></span>
<span class="code-line"> <span class="ss">:params</span> <span class="o">=></span> <span class="p">{</span></span>
<span class="code-line"> <span class="ss">:title</span> <span class="o">=></span> <span class="n">title</span><span class="p">,</span></span>
<span class="code-line"> <span class="ss">:body</span> <span class="o">=></span> <span class="s2">"Reader comments on [</span><span class="si">#{</span><span class="n">title</span><span class="si">}</span><span class="s2">](</span><span class="si">#{</span><span class="n">url</span><span class="si">}</span><span class="s2">)"</span></span>
<span class="code-line"> <span class="p">}</span></span>
<span class="code-line"> <span class="nb">puts</span> <span class="s2">"Successfully opened issue </span><span class="se">\#</span><span class="si">#{</span><span class="n">issue</span><span class="o">.</span><span class="n">number</span><span class="si">}</span><span class="s2">"</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="n">labels</span> <span class="o">=</span> <span class="n">issue</span><span class="o">.</span><span class="n">add_label</span> <span class="s2">"blog-post"</span></span>
<span class="code-line"></span>
<span class="code-line"> <span class="k">return</span> <span class="n">issue</span><span class="o">.</span><span class="n">number</span></span>
<span class="code-line"> <span class="k">end</span></span>
<span class="code-line"><span class="k">end</span></span>
</pre></div>
<p>I plugged this function into the existing <code>new_post</code> task so that it will
create an issue and plug it's id into the front matter for the new post
automatically when I run a command like <code>rake new_post["Using GitHub Issues
for Comments"]</code>:</p>
<p><em>source/_posts/2012-04-14-using-github-issues-for-comments.markdown</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="nn">---</span></span>
<span class="code-line"><span class="l l-Scalar l-Scalar-Plain">layout</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">post</span></span>
<span class="code-line"><span class="l l-Scalar l-Scalar-Plain">title</span><span class="p p-Indicator">:</span> <span class="s">"Using</span><span class="nv"> </span><span class="s">GitHub</span><span class="nv"> </span><span class="s">issues</span><span class="nv"> </span><span class="s">for</span><span class="nv"> </span><span class="s">comments"</span></span>
<span class="code-line"><span class="l l-Scalar l-Scalar-Plain">date</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">2012-04-14 20:22</span></span>
<span class="code-line"><span class="l l-Scalar l-Scalar-Plain">comments</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">true</span></span>
<span class="code-line"><span class="l l-Scalar l-Scalar-Plain">github_issue_id</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">7</span></span>
<span class="code-line"><span class="l l-Scalar l-Scalar-Plain">categories</span><span class="p p-Indicator">:</span> </span>
<span class="code-line"><span class="nn">---</span></span>
</pre></div>Generating an Apparently Random Unique Sequence2012-03-31T15:40:00+00:002012-03-31T15:40:00+00:00Bryan Davistag:bd808.com,2012-03-31:/blog/2012/03/31/generating-an-apparently-random-unique-sequence/<p>Using a sequentially increasing counter to generate an id token is easy.
Database sequences and auto-number columns make it fairly trivial to
implement. If that isn't available a simple file or shared memory counter can
be implemented in minutes. Displaying such a number to a client however may
give them more information than you would really like them to have about the
number of ids you are allocating per unit time. We'd really like to obfuscate
the id somehow while retaining the uniqueness of the original sequence.</p>
<p>One way to do this is to use a combination of multiplication and modulo
arithmetic to map the sequence number into a constrained set. With careful
choice of the multiplicative constant and the modulo value the resulting
number can be made to wander rather effectively over the entire space of the
target set.</p>
<p>Using a sequentially increasing counter to generate an id token is easy.
Database sequences and auto-number columns make it fairly trivial to
implement. If that isn't available a simple file or shared memory counter can
be implemented in minutes. Displaying such a number to a client however may
give them more information than you would really like them to have about the
number of ids you are allocating per unit time. We'd really like to obfuscate
the id somehow while retaining the uniqueness of the original sequence.</p>
<p>One way to do this is to use a combination of multiplication and modulo
arithmetic to map the sequence number into a constrained set. With careful
choice of the multiplicative constant and the modulo value the resulting
number can be made to wander rather effectively over the entire space of the
target set.</p>
<p>The basic math looks like this: <code>f(n) := (n * p) % q</code></p>
<ul>
<li><code>n</code> := input sequence value</li>
<li><code>p</code> := step size</li>
<li><code>q</code> := maximum result size</li>
</ul>
<p><code>p</code> and <code>q</code> must be chosen such that:</p>
<ul>
<li><code>p</code> < <code>q</code></li>
<li><code>p</code> * <code>q</code> < arithmetic limit (2^31, 2^32, 2^63, 2^64, ... depending on the precision of the underlying system)</li>
<li><code>p</code> ⊥ <code>q</code> (<a href="https://en.wikipedia.org/wiki/Coprime">coprime</a> or relatively prime)</li>
</ul>
<p>With <code>p := 5</code> and <code>q := 12</code> our function will generate this output:
<table class="table table-bordered">
<tr><th>n</th><td>1 </td><td> 2 </td><td> 3 </td><td> 4 </td><td> 5 </td><td> 6 </td><td> 7 </td><td> 8 </td><td> 9 </td><td> 10 </td><td> 11 </td></tr>
<tr><th>f(n)</th><td>5 </td><td> 10 </td><td> 3 </td><td> 8 </td><td> 1 </td><td> 6 </td><td> 11 </td><td> 4 </td><td> 9 </td><td> 2 </td><td> 7 </td></tr>
</table></p>
<p>Change <code>p</code> to 7 and you'll get:
<table class="table table-bordered">
<tr><th>n</th><td>1 </td><td> 2 </td><td> 3 </td><td> 4 </td><td> 5 </td><td> 6 </td><td> 7 </td><td> 8 </td><td> 9 </td><td> 10 </td><td> 11 </td></tr>
<tr><th>f(n)</th><td>7</td><td> 2</td><td> 9</td><td> 4</td><td> 11</td><td> 6</td><td> 1</td><td> 8</td><td> 3</td><td> 10</td><td> 5</td></tr>
</table></p>
<p>The rational for keeping <code>p * q < limit</code> is that as <code>n</code> approaches <code>q</code> the
initial multiplication will approach <code>p * q</code> and if this calculation overflows
the available precision the result will wrap back into a previously traversed
space causing duplication. The same sort of thing will occur if <code>p</code> and <code>q</code>
are not coprime. The result of the modulo will exhibit a period equivalent to
the GCD<sup id="fnref-GCD"><a class="footnote-ref" href="#fn-GCD">1</a></sup> of <code>p</code> and <code>q</code> rather than mapping the entire range of <code>q</code>
evenly.</p>
<p>Careful choice of <code>p</code> and <code>q</code> are key to getting a good spread in the output
of the function and maintaining the uniqueness of the result. One easy way to
ensure that the chosen coefficients are coprime is to make them both be prime
powers of prime numbers (eg 9^17, 13^11, 13^15, 19^7, ...).</p>
<p>This method is a type of <a href="https://en.wikipedia.org/wiki/Linear_congruential_generator">Linear congruential generator</a> almost exactly equivalent to the <a href="https://en.wikipedia.org/wiki/Park%E2%80%93Miller_RNG">Park–Miller random number generator</a>.</p>
<h2>Examples</h2>
<p><em>PHP</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="cp"><?php</span></span>
<span class="code-line"><span class="sd">/**</span></span>
<span class="code-line"><span class="sd"> * Obfuscate an id generated from a linear sequence.</span></span>
<span class="code-line"><span class="sd"> *</span></span>
<span class="code-line"><span class="sd"> * @param int $n Input value</span></span>
<span class="code-line"><span class="sd"> * @param int $p Random walk step size</span></span>
<span class="code-line"><span class="sd"> * @param int $q Maximum result value</span></span>
<span class="code-line"><span class="sd"> * @return int Obfuscated result</span></span>
<span class="code-line"><span class="sd"> */</span></span>
<span class="code-line"><span class="k">function</span> <span class="nf">obfuscate_id</span> <span class="p">(</span><span class="nv">$n</span><span class="p">,</span> <span class="nv">$p</span><span class="p">,</span> <span class="nv">$q</span><span class="p">)</span> <span class="p">{</span></span>
<span class="code-line"> <span class="k">return</span> <span class="p">(</span><span class="nv">$n</span> <span class="o">*</span> <span class="nv">$p</span><span class="p">)</span> <span class="o">%</span> <span class="nv">$q</span><span class="p">;</span></span>
<span class="code-line"><span class="p">}</span></span>
</pre></div>
<p><em>PL/SQL</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="k">FUNCTION</span> <span class="n">obfuscate_id</span> <span class="p">(</span><span class="n">n</span> <span class="nb">NUMBER</span><span class="p">,</span> <span class="n">p</span> <span class="nb">NUMBER</span><span class="p">,</span> <span class="n">q</span> <span class="nb">NUMBER</span><span class="p">)</span> <span class="k">RETURN</span> <span class="nb">NUMBER</span> <span class="k">IS</span></span>
<span class="code-line"><span class="k">BEGIN</span></span>
<span class="code-line"> <span class="k">RETURN</span> <span class="k">MOD</span><span class="p">(</span><span class="n">n</span> <span class="o">*</span> <span class="n">p</span><span class="p">,</span> <span class="n">q</span><span class="p">);</span></span>
<span class="code-line"><span class="k">END</span> <span class="n">f</span><span class="p">;</span></span>
</pre></div>
<hr>
<p>Thanks to <a href="http://www.timbarber.org/">Tim</a> for explaining all of this to me several times without becoming annoyed at the parts I wasn't getting.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn-GCD">
<p>Greatest Common Divisor <a class="footnote-backref" href="#fnref-GCD" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>FizzBuzz — the wrong way to do it2012-01-18T21:47:00+00:002012-01-18T21:47:00+00:00Bryan Davistag:bd808.com,2012-01-18:/blog/2012/01/18/fizzbuzz-the-wrong-way-to-do-it/<blockquote>
<p>Write a program that prints the numbers from 1 to 100. But for multiples of
three print "Fizz" instead of the number and for the multiples of five print
"Buzz". For numbers which are multiples of both three and five print
"FizzBuzz".
<small><a href="http://imranontech.com/2007/01/24/using-fizzbuzz-to-find-developers-who-grok-coding/">imranontech [at] googlemail.com</a></small></p>
</blockquote>
<blockquote>
<p>Write a program that prints the numbers from 1 to 100. But for multiples of
three print "Fizz" instead of the number and for the multiples of five print
"Buzz". For numbers which are multiples of both three and five print
"FizzBuzz".
<small><a href="http://imranontech.com/2007/01/24/using-fizzbuzz-to-find-developers-who-grok-coding/">imranontech [at] googlemail.com</a></small></p>
</blockquote>
<p><em>Python</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="ch">#!/usr/bin/env python</span></span>
<span class="code-line"><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">101</span><span class="p">):</span></span>
<span class="code-line"> <span class="k">print</span> <span class="p">(</span><span class="ow">not</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">3</span><span class="p">)</span> <span class="o">*</span> <span class="s2">"Fizz"</span> <span class="o">+</span> <span class="p">(</span><span class="ow">not</span> <span class="n">i</span> <span class="o">%</span> <span class="mi">5</span><span class="p">)</span> <span class="o">*</span> <span class="s2">"Buzz"</span> <span class="ow">or</span> <span class="n">i</span></span>
</pre></div>
<p><em>PHP</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="cp"><?php</span></span>
<span class="code-line"><span class="nv">$p</span> <span class="o">=</span> <span class="s2">"printf"</span><span class="p">;</span> <span class="nv">$r</span> <span class="o">=</span> <span class="s2">"str_repeat"</span><span class="p">;</span></span>
<span class="code-line"><span class="k">for</span> <span class="p">(</span><span class="nv">$i</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="nv">$i</span> <span class="o"><=</span> <span class="mi">100</span><span class="p">;</span> <span class="nv">$i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span></span>
<span class="code-line"> <span class="nv">$p</span><span class="p">(</span><span class="s2">"%s</span><span class="se">\n</span><span class="s2">"</span><span class="p">,</span> <span class="nv">$r</span><span class="p">(</span><span class="nv">$i</span><span class="p">,</span> <span class="nv">$p</span><span class="p">(</span><span class="s2">"%s%s"</span><span class="p">,</span></span>
<span class="code-line"> <span class="nv">$r</span><span class="p">(</span><span class="s2">"Fizz"</span><span class="p">,</span> <span class="o">!</span><span class="p">(</span><span class="nv">$i</span> <span class="o">%</span> <span class="mi">3</span><span class="p">)),</span> <span class="nv">$r</span><span class="p">(</span><span class="s2">"Buzz"</span><span class="p">,</span> <span class="o">!</span><span class="p">(</span><span class="nv">$i</span> <span class="o">%</span> <span class="mi">5</span><span class="p">)))</span> <span class="o">==</span> <span class="mi">0</span><span class="p">));</span></span>
<span class="code-line"><span class="p">}</span></span>
</pre></div>
<p><em>Bash</em></p>
<div class="highlight"><pre><span class="code-line"><span></span><span class="ch">#!/usr/bin/env bash</span></span>
<span class="code-line"><span class="k">for</span> i in <span class="o">{</span><span class="m">1</span>..100<span class="o">}</span><span class="p">;</span> <span class="k">do</span></span>
<span class="code-line"> <span class="k">if</span> <span class="o">[</span> <span class="nv">0</span> <span class="o">=</span> <span class="k">$((</span><span class="nv">$i</span> <span class="o">%</span> <span class="m">15</span><span class="k">))</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> <span class="nb">echo</span> <span class="s2">"FizzBuzz"</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">elif</span> <span class="o">[</span> <span class="nv">0</span> <span class="o">=</span> <span class="k">$((</span><span class="nv">$i</span> <span class="o">%</span> <span class="m">3</span><span class="k">))</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> <span class="nb">echo</span> <span class="s2">"Fizz"</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">elif</span> <span class="o">[</span> <span class="nv">0</span> <span class="o">=</span> <span class="k">$((</span><span class="nv">$i</span> <span class="o">%</span> <span class="m">5</span><span class="k">))</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> <span class="nb">echo</span> <span class="s2">"Buzz"</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">else</span> <span class="nb">echo</span> <span class="nv">$i</span><span class="p">;</span></span>
<span class="code-line"> <span class="k">fi</span></span>
<span class="code-line"><span class="k">done</span></span>
</pre></div>