Turns out…Ruby’s not the memory hog I’d previously thought. After some research and experimentation, I’ve found jemalloc to offer significant memory savings while at least preserving performance, if not improving it as well.

The Problem

At Bark, we poll external APIs for millions of monitored social media, text, and emails. This is all done through Sidekiq background jobs. Even though Ruby doesn’t truly allow parallelism, we see great benefit with Sidekiq concurrency as the jobs wait for external APIs to respond. The API responses can often be large, not to mention any media they might include. As a result, we see the memory usage of our Sidekiq workers increase until they’re ultimately killed and restarted by systemd.

The following shows a common memory usage pattern for our queue servers:

Two things to notice:

1. Memory increased quickly - The rise of memory happens immediately after the processes are restarted. We deploy multiple times a day, but this was especially problematic on the weekends when deploys are happening less frequently

2. Memory wasn’t reused until restarted - The jaggedness of graph towards the center is the result of the memory limits we imposed on the systemd processes, causing them to be killed and ultimately restarted until they later reach the configured max memory setting again. Because the processes didn’t appear to be reusing memory, we saw this happen just a few minutes after being restarted.

The Solution

As the author of a multi-threaded background processing library, I frequently see reports of memory leaks in Rails applications. As a Sidekiq user, this one caught my attention. It starts as a classic memory leak report, but later turns towards deeper issues in the underlying operating system, not in the application. With Nate Berkopec’s post on Ruby memory usage in multi-threaded applications referenced, the reporter found switching to jemalloc to fix their issue.

jemalloc describes itself as:

a general purpose malloc(3) implementation that emphasizes fragmentation avoidance and scalable concurrency support

The description targets our use-case and issues with the current memory allocator. We were seeing terrible fragmentation when using Sidekiq (concurrent workers).

How to use jemalloc

Ruby can use jemalloc a few different ways. It can be compiled with jemalloc, but we already had Ruby installed and were interested in trying it with the least amount of infrastructure changes.

It turns out Ruby will attempt to use jemalloc if the well-document environment variable LD_PRELOAD is set.

Our Sidekiq servers use Ubuntu 16.04, so we started by installing jemalloc:

sudo apt-get install libjemalloc-dev

From there, we configured the LD_PRELOAD environment variable by adding the following to /etc/environment:

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1

Note: The location of jemalloc may vary depending on version and/or Linux distribution.

Benchmark

We benchmarked jemalloc on just one of the queue servers. This would allow us to do a true comparison against similar activity.

As we can see, the difference is drastic – over 4x decrease in memory usage!

The more impressive detail was the consistency. Total memory usage doesn’t waver much. Processing large payloads and media, I assumed we’d continue to see the peaks and valleys common to processing social media content. The sidekiq processes using jemalloc show a better ability to use previously allocated memory.

Roll it in to production

With similar behavior over a 3 day period, we concluded to roll it out to the remaining queue servers.

The reduced memory usage continues to be impressive, all without any noticeable negative trade-offs.

Conclusion

We were surprised by the significant decrease in memory usage by switching to jemalloc. Based on the other reports, we assumed it be reasonable, but not a 4x decrease.

Even after looking at these graphs for the last couple days, the differences seem too good to be true. But all is well and it’s hard to imagine NOT doing this for any Ruby server we deploy in the future.

Give it a shot. I’d love to see your results.

As Bark continues to grow, we became interested in how the number of jobs enqueued and retrying trended over time. Using AWS Lambda to post this data to CloudWatch, we were able to visualize this data over time.

The Problem

Sidekiq offers a way to visual the jobs processed over time when on the dashboard. In fact, this graph was one of my first open source contributions.

Unfortunately, these graph don’t show the number of retries from 2 am last night, or how long it took to exhaust the queues when 2 million jobs were created.

Historical queue data is important if our application does a lot of background work and number of users is growing. Seeing these performance characteristics over time can help us be more prepared to add more background workers or scale our infrastructure in a way to stay ahead when our application is growing quickly.

The Solution

Because Bark is on AWS, we naturally looked to their tools for assistance. We already use CloudWatch to store data about memory, disk, and CPU usage for each server. This has served us well and allows us to set alarms for certain thresholds and graph this data over time:

Knowing we’d have similar data for queue usage, we figured we could do the same with Sidekiq.

Sidekiq Queue Data Endpoint

If you remember from the last article on monitoring Sidekiq retries using AWS Lambda, we setup an endpoint in our application to return Sidekiq stats:

require 'sidekiq/api'

class SidekiqQueuesController < ApplicationController
  skip_before_action :require_authentication

  def index
    base_stats = Sidekiq::Stats.new
    stats = {
       enqueued: base_stats.enqueued,
       queues: base_stats.queues,
       busy: Sidekiq::Workers.new.size,
       retries: base_stats.retry_size
    }

    render json: stats
  end
end

along with the route:

resources :sidekiq_queues, only: [:index]

Using this resource, we need to poll at some regular interval and store the results.

AWS Lambda Function

AWS Lambda functions are perfect for one-off functions that feel like a burden to maintain in our application.

For the trigger, we’ll use “CloudWatch Events - Schedule”:

From here, we’ll enter a name and description for our rule and define its rate (I chose every 5 minutes). Enable the trigger and we’ll move to defining our code. Next, we’ll give the function a name and choose the latest NodeJS as the runtime. Within the inline editor, we’ll use the following code:

var AWS = require('aws-sdk');
var url = require('url');
var https = require('https');

if (typeof Promise === 'undefined') {
  AWS.config.setPromisesDependency(require('bluebird'));
}

var cloudwatch = new AWS.CloudWatch();

sidekiqUrl = '[Sidekiq stat URL]'

var logMetric = function(attr, value) {
    var params = {
        MetricData: [
            {
                MetricName: attr,
                Dimensions: [
                    {
                        Name: "App",
                        Value: "www"
                    }
                ],
                Timestamp: new Date(),
                Unit: "Count",
                Value: value
            }
        ],
        Namespace: "Queues"
    };

    return cloudwatch.putMetricData(params).promise();
}

var getQueueStats = function(statsUrl) {
    return new Promise(function(resolve, reject) {
        var options = url.parse(statsUrl);
        options.headers = {
            'Accept': 'application/json',
        };
        var req = https.request(options, function(res){
            var body = '';

            res.setEncoding('utf8');

            //another chunk of data has been recieved, so append it to `str`
            res.on('data', function (chunk) {
                body += chunk;
            });

            //the whole response has been recieved
            res.on('end', function () {
                resolve(JSON.parse(body));
            });
        });

        req.on('error', function(e) {
           reject(e);
        });

        req.end();
    });
}

exports.handler = function(event, context) {
    getQueueStats(sidekiqUrl).then(function(stats) {
        console.log('STATS: ', stats);

        var retryPromise = logMetric("Retries", stats.retries);
        var busyPromise = logMetric("Busy", stats.busy);
        var enqueuedPromise = logMetric("Enqueued", stats.enqueued);

        Promise.all([retryPromise, busyPromise, enqueuedPromise]).then(function(values) {
            console.log(values);
            context.succeed();
        }).catch(function(err){
            console.error(err);
            context.fail("Server error when processing message: " + err );
        });
    })
    .catch(function(err) {
        console.error(err);
        context.fail("Failed to get stats from HTTP request: " + err );
    });
};

Note: sidekiqURL need to be defined with appropriate values for this to work.

Within CloudWatch, we’re defining a new namespace (“Queues”) where our data will live. Within this namespace, we’ll segregate these stats by the Dimension App. As we can see, we chose www for this value. If we wanted to monitor the queues of a few servers, each one could use a unique App name.

Review and save the Lambda function and we’re all set!

Graphing Sidekiq Queue Data

Once the function has run a few times, under CloudWatch –> Metrics, we’ll see the following custom namespace:

From here, we’ll choose the name of our app (www) and graph the values of each of these values over whatever timespan we want:

Conclusion

I’ve found AWS lamba to be a great place for endpoints/functionality that feels cumbersome to include in my applications. Bringing deeper visibility to our Sidekiq queues has given us the ability to see usage trends we weren’t aware of throughout the day. This will help us preemptively add infrastructure resources to keep up with our growth.

Using PhantomJS, we were able to take screenshots of a modified version of the application’s child analytics page and include the image in the email sent to the parent. The email now contains everything the parent can see from the application, all without leaving their inbox.

The Problem

If you’ve every attempted to style an HTML email with anything more than text, you’re sadly familiar with its limitations. Tables and other elements from the 90’s are the only tools we have to maintain cross-platform compatibility. One of those tools, the subject of this post, is images.

Our weekly recap email contained a line chart illustrating the number of messages the child exchanged during the past week. While this was somewhat helpful to parents, it didn’t tell the full story.

While this email does include a graph, it’s the result of calling out to a service that rendered the graph, stored it, and returned the URL to include as an image. While this service worked well for simple illustrations, it didn’t provide us the flexibility we had with modern web tools and visualizations. Aside from that, the styling of the charts is limited.

Elsewhere on Bark, we had already built the full story through other lists and illustrations.

Recreating the same lists and charts just for the email felt like a duplication nightmare and vulnerable to becoming stale. We wouldn’t be able to use the same rendering because most of the charts rendered SVGs, which aren’t compatible with most email clients. Additionally, there were a handful of CSS styles needed for the page that while possible to include in the email, felt excessive.

Stepping back from the problem, we realized we wanted the majority of the analytics page, just embedded in the email. Was there a way to do that without rewriting it for email clients?

The Solution

We could take a screenshot of the analytics page and embed it as an image in the recap email.

wkhtmltoimage

Our first attempt was using wkhtmltoimage and the IMGKit ruby gem. Aside from the headaches of installing a working OSX version of wkhtmltoimage due to a regression, getting a working configuration was non-trivial.

wkhtmltoimage doesn’t parse CSS or JavaScript, so those would have to be explicitly included. Since Bark uses the asset pipeline, we’d have to get the latest version of the compiled assets both on development and production. This proved to be extremely difficult under the default configuration given how each group is compiled. We use Nginx to serve our assets in the production and it felt weird to have a configuration we would hope worked when we pushed to production.

After spending almost a full day trying to get the right combination of settings, we gave up. There had to be a better way…

Saas FTW

Frankly, our next step was to look for a Saas service that provided this functionality. Certainly I should be able to send a URL to an API, and they’d return an image, perhaps with some configuration options for size and response. To our surprise, there were none (based on a 15 minute internet search. If you know of one, we’d love to hear about it). There were plenty of services focused on rendering PDFs geared towards invoices and other documents one would want to email customers.

PhantomJS

We were reminded of Capybara’s ability to capture screenshots on failed test runs. After poking around this functionality, phantomjs emerged as a potential solution.

If we installed phantomjs on to the server and ran a command line script to capture the screenshots prior to sending the email, we could inline include those images in the email.

Installation of phantomjs was simplified using the phantomjs-gem ruby gem, which installs the appropriate phantomjs binary for the operating system and provides an API (#run) to execute commands.

Script the Screenshot

Using a screenshot example from the PhantomJS github repo, we put together a script (vendor/assets/javascripts/phantom-screenshot.js) to capture the analytics page:

#!/bin/sh

var page   = require('webpage').create();
var system = require('system');
page.viewportSize = { width: 550, height: 600 };
page.zoomFactor = 0.85;

page.onError = function(msg, trace) {
  var msgStack = ['ERROR: ' + msg];
  if (trace && trace.length) {
    msgStack.push('TRACE:');
    trace.forEach(function(t) {
      msgStack.push(' -> ' + t.file + ': ' + t.line + (t.function ? ' (in function "' + t.function +'")' : ''));
    });
  }

  console.error(msgStack.join('\n'));
};

page.open(system.args[1], function(status) {
  if (status !== 'success') {
    console.log('Unable to load the address!');
    phantom.exit(1);
  } else {
    window.setTimeout(function () {
      page.render(system.args[2]);
      phantom.exit();
    }, 2000);
  }
});

Note: a variety of the settings (viewPortSize, zoomFactor, and timeout) were found using trial and error for our particular situation.

We use Sidekiq to enqueue the thousands of recap emails sent to parents each week. Because this approach relies on using our existing website as the source data for the screenshot, we have to be conscious of spreading the job processing over a certain period of time, so we don’t overload the application for regular users.

Create the Screenshot

With this script in hand, now we can use the following class to create the image for each child:

class RecapAnalytics
  ScreenshotError = Class.new(StandardError)

  def initialize(analytics_url:)
    @analytics_url = analytics_url
  end

  def file_path
    unless create_screenshot
      raise ScreenshotError.new("Unable to complete analytics screenshot")
    end

    temp_file_path
  end

  def create_screenshot
    Phantomjs.run screenshot_script, analytics_url, temp_file_path
  end

  private

  attr_reader :analytics_url

  def screenshot_script
    Rails.root.join('vendor', 'assets', 'javascripts', 'phantom-screenshot.js').to_s
  end

  def temp_file_path
    @temp_file_path ||= begin
      file = Tempfile.new("child-analytics")
      file.path + ".png"
    end
  end
end

For each child, we’ll provide the URL to the child’s analytics page and run the following file_path method to return the path of the screenshot:

RecapAnalytics.new(analytics_url: "https://www.bark.us/children/XXX/analytics").file_path

Adding as an Inline Email Attachment

With an image for each child, we can iterate through each child and inline include the image in the mailer:

file_path = RecapAnalytics.new(analytics_url: "https://www.bark.us/children/XXX/analytics").file_path
attachments.inline["#{child.first_name}.png"] = File.read(file_path)

Then in the email template, we can include the following to render the image:

 <%= link_to image_tag(attachments["#{child.first_name}.png"].url), child_url(child) %>

Conclusion

PhantomJS proved to be the simplest solution for the job. With a small script and no further configuration, we were able to lean on the analytics page we’d already built to improve the Bark recap emails.

Parents will now have more visibility in to their child’s online activity without leaving their inbox.

As a remote company, we use Slack to stay in touch with everyone AND to manage/monitor our infrastructure (hello #chatops). We can deploy from Slack (we don’t generally, we have full CI) and be notified of infrastructure and application errors.

When Sidekiq retries accumulate, it’s a good indication that something more severe might be wrong. Rather than get an email we won’t see for 30 minutes, we decided to integrate these notifications in to Slack. In doing so, we found AWS Lambda to be a lightweight solution to tie the monitoring of Sidekiq and notifications in Slack together.

The Problem

Bark is background job-heavy. The web application is a glorified CRUD app that sets up the data needed to poll a child’s social media feed and monitor for potential issues. The best-case scenario for a parent is that they will never hear from us.

Because Bark’s background jobs commonly interact with 3rd-party APIs, failures aren’t a big surprise. APIs can be down, network connections can fail – Sidekiq’s retry logic protects us from transient network errors. Under normal circumstances, jobs retry and ultimately run successfully after subsequent attempts. These are non-issues and something we don’t need an engineer to investigate.

There are times when retries accumulate, giving us a strong indication that something more severe may be wrong. Initially, we setup New Relic to notify us of an increased error rate. This worked for simple cases, but was sometimes a false positive. As a result, we started to ignore them, which potentially masked more important issues.

We soon realized one of the gauges of application health was the number of retries in the Sidekiq queue. We have the Sidekiq Web UI mounted within our admin application, so we’d browse there a few times a day to make sure the number of retries weren’t outside our expectations (in this case < 50 were acceptable).

This wasn’t a great use of our time. Ideally, we wanted a Slack notification when the number of Sidekiq retries was > 50.

The Solution

Because Bark is on AWS, we naturally looked to their tools for assistance. In this case, we needed something that would poll Sidekiq, check the number of retries, and POST to Slack if the number of retries was > 50.

There were a few options:

1. Add the Sidekiq polling and Slack notification logic to our main application and setup a Cron job
2. Create a new satellite application that ONLY does the above (microservices???)
3. Setup an AWS Lambda function to handle the above logic

The first two options would’ve worked, but I was hesistant to add complexity to our main application. I was also hesitant to have to manage another application (ie. updates, etc.) for something that seemed simple.

Option “AWS Lambda” won! Let’s take a look at the implementation.

Sidekiq Queue Data Endpoint

First, we need to expose the number of Sideki retries somehow. As I mentioned above, the Sidekiq web UI is mounted in our admin application, but behind an authentication layer that would’ve been non-trivial to publicly expose.

Instead, we created a new Rails route to respond with some basic details about the Sidekiq system.

require 'sidekiq/api'

class SidekiqQueuesController < ApplicationController
  skip_before_action :require_authentication

  def index
    base_stats = Sidekiq::Stats.new
    stats = {
       enqueued: base_stats.enqueued,
       queues: base_stats.queues,
       busy: Sidekiq::Workers.new.size,
       retries: base_stats.retry_size
    }

    render json: stats
  end
end

along with the route:

resources :sidekiq_queues, only: [:index]

As you can see, the endpoint is public (there’s no job args or names exposed – just counts). The code digs in to the Sidekiq API to interrogate the size of queues.

Slack Incoming WebHook

We want to be able to POST to Slack when the number of Sidekiq retries are > 50. To do this, we’ll setup a custom incoming webhook integration in Slack.

We’ll start by choose Apps & integrations from within the main Slack options. From here, choose Manage in the top right, and then Custom Integrations on the left. You’ll have 2 options:

1. Incoming WebHooks
2. Slash Commands

We’ll choose Incoming Webhooks and choose Add Configuration to add a new one. From here, we’ll supply the information needed to specify the channel where the notifications will appear and how they look.

The most important of this step is to get the Webhook URL. This will be the URL we POST to from within our Lambda function when retries are above our acceptable threshold.

AWS Lambda Function

Now that we have our endpoint to expose the number of retries (among other things) and the Slack webhook URL to POST to, we need to setup the AWS Lambda function to tie to the two together. We’ll start by creating a new Lambda function with the defaults – using the latest Node.

For the trigger, we’ll use “CloudWatch Events - Schedule”:

From here, we’ll enter a name and description for our rule and define its rate (I chose every 5 minutes). Enable the trigger and we’ll move to defining our code. Next, we’ll give the function a name and choose the latest NodeJS as the runtime. Within the inline editor, we’ll use the following code:

var AWS = require('aws-sdk');
var url = require('url');
var https = require('https');
var sidekiqURL, hookUrl, slackChannel, retryThreshold;

sidekiqUrl = '[Sidekiq queue JSON endpoint]'
hookUrl = '[Slack Incoming WebHooks URL w/ token]';
slackChannel = '#operations';  // Enter the Slack channel to send a message to
retryThreshold = 50;

var postMessageToSlack = function(message, callback) {
    var body = JSON.stringify(message);
    var options = url.parse(hookUrl);
    options.method = 'POST';
    options.headers = {
        'Content-Type': 'application/json',
        'Content-Length': Buffer.byteLength(body),
    };

    var postReq = https.request(options, function(res) {
        var chunks = [];
        res.setEncoding('utf8');
        res.on('data', function(chunk) {
            return chunks.push(chunk);
        });
        res.on('end', function() {
            var body = chunks.join('');
            if (callback) {
                callback({
                    body: body,
                    statusCode: res.statusCode,
                    statusMessage: res.statusMessage
                });
            }
        });
        return res;
    });

    postReq.write(body);
    postReq.end();
};

var getQueueStats = function(callback) {
    var options = url.parse(sidekiqUrl);
    options.headers = {
        'Accept': 'application/json',
    };

    var getReq = https.request(options, function(res){
        var body = '';

        res.setEncoding('utf8');

        //another chunk of data has been recieved, so append it to `str`
        res.on('data', function (chunk) {
            body += chunk;
        });

        //the whole response has been recieved, so we just print it out here
        res.on('end', function () {
            if (callback) {
                callback({
                    body: JSON.parse(body),
                    statusCode: res.statusCode,
                    statusMessage: res.statusMessage
                });
            }
        });
    })

    getReq.end();
}

var processEvent = function(event, context) {
    getQueueStats(function(stats){
        console.log('STATS: ', stats.body);

        var retries = stats.body.retries;

        if (retries > retryThreshold) {
            var slackMessage = {
                channel: slackChannel,
                text: "www Sidekiq retries - " + retries
            };

            postMessageToSlack(slackMessage, function(response) {
                if (response.statusCode < 400) {
                    console.info('Message posted successfully');
                    context.succeed();
                } else if (response.statusCode < 500) {
                    console.error("Error posting message to Slack API: " + response.statusCode + " - " + response.statusMessage);
                    context.succeed();  // Don't retry because the error is due to a problem with the request
                } else {
                    // Let Lambda retry
                    context.fail("Server error when processing message: " + response.statusCode + " - " + response.statusMessage);
                }
            });
        } else {
            console.info('Sidekiq retries were ' + retries + ' . Below threshold.');
            context.succeed();
        }
    })
};

exports.handler = function(event, context) {
    processEvent(event, context);
};

Note: sidekiqURL and hookURL need to be defined with appropriate values for this to work.

Review and save the Lambda function and we’re all set!

Review

We can review the Lambda function logs on CloudWatch. Go to CloudWatch and choose “Logs” from the left menu. From here, we’ll click the link to the name of our Lambda function:

From here, logs for each invocation of the Lambda function will be grouped in to a log stream:

Grouped by time, each link will contain multiple invocations. A single execution is wrapped with a START and END, as shown in the logs. Messages in between will be calls to console.log from within our function. We logged the results of the Sidekiq queue poll for debugging purposes, so you can see that below:

This was invocation where the number of retries were < 50, and a result, didn’t need to POST to Slack. Let’s take a look at the opposite:

We can see the Message posted successfully log indicating our message was successfully sent to Slack’s incoming webhook.

Finally, here’s what the resulting message looks like in Slack when the number of Sidekiq retries are above our threshold:

Conclusion

Using new tools is fun, but not when it brings operational complexity. I’ve personally found AWS lamba to be a great place for endpoints/functionality that feels cumbersome to include in my applications. Bringing these notifications in to Slack has been a big win for our team. We took a previously untrustworthy notification (NewRelic error rate) and brought some clarity to the state and health of our applications.

Fast forward almost 15 years and I’m on the opposite end of the spectrum – I build a product I want to exist, a parent like me is the target user and furthermore, I have equity in the company.

This spectrum of motivation and responsibility encapsulates all different types of software development jobs. Personal preferences play as large part in defining motivation. This articles explores how caring about what I was building changed my perception of work and the questions I asked myself to get there.

A Unique Time in Technology

Knowledge workers..is that we’re called? As most job boards illustrate, I can’t recall another time in my career when there’s been such demand for developers. Recognizing the talent pool is limited in most areas, companies are hiring remote workers to grow their engineering teams.

If we, as developers, are in such high demand, why do we settle for anything short of our dream job? It’s the perfect economic time to make a change if you don’t feel fulfilled. Who knows if there’ll be another time where we have so much leverage. You deserve a job you love, that also fits your professional ambitions.

The Soapbox Test

Most software developers I know are introverts. They generally like writing code, shipping new features, but avoid meetings at all costs. The rise of slack and other remote-focused tooling further increases the human contact void that most developers experience.

My experience with sales people have been the polar opposite. They thrive on human interaction. Sales people, no matter the company, generally position their product/services as the perfect fit for you and your company. That’s ultimately their job – match the customers’ need(s) with a solution they hopefully sell.

It’s not surprising that being in a sales role for a company/product that you love and genuinely want to see succeed is much easier than one you could care less about. This is where roles at a small company often cross-over.

Not too long ago I took a break from writing code to pitch Bark at TechCrunch Disrupt. I’m not in sales person don’t strive to be. But because I deeply care about Bark and the differences we hope to make, becoming an advocate for the company is easy. I have to find a way to articulate what we do, how we do it, and why we do it. And who better than me? I spend the majority of my days already thinking about it, which makes me more qualified than anyone else.

Imagine you have a microphone in your hand, a crowd of 2,000 people and 2 minutes to tell them what you’re building and why they should care. Does it feel weird or slimy?

If they answer is “yes”, you should find another job where you’re a natural advocate. If your love for what you’re building isn’t genuine, you’re doing yourself a disservice. Passion and care have a way to turn even the most anti-sales people in to advocates.

The Deathbed Test

Doctors and nurses that spend a lot of time around people in the latter stages of their life get an unusual look at the regrets of those in care. They’re often able to apply the lessons to their own life.

If you read about the most common regrets among the dying, it’s something like “I wish I spent more time with my family” or “I wish I didn’t work as much”. Work occupies a large majority of our lives. Assuming we agree to work a somewhat standard career, the next question is “Are we happy with what we’re working on?”

The deathbed question is a useful one beyond your career decisions.

Imagine you have 1 more day to live and you’re left thinking about all the choices you’ve made over your lifetime. Would you be happy with the job you have today or the job you’re thinking about taking tomorrow?

If the answer is “No”, it’s time to find something better. Life can be short. Don’t waste it on a job you’re less than excited about.

The Lottery Test

The media constantly reminds us about successful entrepreneurs like Mark Zuckerberg, Bill Gates and Elon Musk who don’t have to work, but do. A common thread amongst those founders is their genuine belief in what they’re creating. Given their financial freedom, they could spend hours sun-bathing on private beaches around the world or traveling to the most remote places on the planet, but they choose to work. In fact, my guess is that none of them refer to what they do as “work”.

If you had unlimited financial freedom, like a sizable lottery, would you continue doing what you’re doing now?

Whether its the position, company, lack of control, or strict hour requirements, most of us would probably adjust our current position in at least a small way. If you’d continue doing exactly what you’re currently doing after winning the lottery, you’ve won. This, of all the questions, is the ultimate. If you can answer “Yes” to this one, kudos to you. You’re probably happier than 99.999% of people in the world.

Ditch the Product, Love the Craft

I know what you’re thinking, you don’t love what you’re building, so where do you go from here? Not everyone has the freedom and ability to choose a position where they’re 100% happy. There will always be personal trade-offs, some of which might leave you working for a company/product you don’t care about and don’t have the luxury to leave anytime soon. I’ve been there before – ultimately leaving – but I made it work for a period of time.

One of the ways I separated myself was to focus on the software development craft. If you’re like me, you care about designing things in a maintainable and reliable way. Whether it’s good design, well-written tests, or using the latest and greatest, shifting your focus away from a product you have no interest levels the playing field.

Take processing incoming email for instance, whether you’re making an internal tool for an ISP or adding email features to your latest drip campaign software, the challenge is the same. Perhaps you work for the less interesting of the two (based on personal preference), removing yourself from the end product where the feature will be present allows you to invest yourself 100% in to making the best programming decisions possible. Being content with this approach requires you have a love for software and everything that comes with it. I’ve done this dozens of times and it’s generally gotten me out of a funk. Looking back, I’ve always been proud of what I’ve accomplished after investing myself 100% in the craft.

Career Kickstarter

Whether you’ve just graduated college or a relevant developer bootcamp, you don’t always have the luxury of being picky. You take the best job available at the time, no matter the product, and get some experience under your belt. This is reasonable and expected (I think there’s value in targeting the more interesting companies even if you’re just starting, but depending on your interests, a job with an interesting company may not always be possible). Within 2-3 years (or sooner!), you’ll have the experience needed to be a little more picky. It’s worth keeping an eye on the handful of companies that do interest you in case opportunities arise there in the future.

Niche Guru

The other way this comes up is because there’s a team or specific developer you want to work with, no matter the company/product. If your interests lie within a specific niche, what better than to learn from the best. Find the most influential person in that niche and try to work with them, attempting to suck every ounce of know-how out of them. This requires putting your own ego and opinions to the side for a bit. The two times I’ve done this in my career, I came out on top.

Summary

Happiness is subjective. Every career decision is the result of personally quantifiable trade-offs. When I look back on my career thus far, the happiest I’ve been has coincided with how fulfilled I’ve felt about my current position. I’ve recognized this and made changes when necessary. I envy those who can hate their job and block it out when they’re not at work. I can’t. Rather than deal with it, I’ve tried to avoid hating my job. Each step I take gets me closer to what I imagine being the perfect job, if there is such a thing.

Care about what you build. Your life will be better for it.

If you’re upgrading to a newer version of Rails and have a bunch of legacy JavaScript code, I can imagine this being difficult. But if you’re green-fielding a new application, there’s no reason not to take advantage of it. I wrote extensively about how to organize JavaScript in a Rails application with Turbolinks enabled. If you’re struggling to get your JavaScript code to work as expected on clicks through the application, take a look at that post. I continue to use that organization pattern for all my applications and it never lets me down.

With Turbolinks enabled, interacting with an application feels smooth and fast. No more full page refreshes.

Every once in awhile we’ll stumble on a page request takes longer than others. Rather than having the user sit there thinking nothing is happening, we can offer better feedback through a loading progress bar, specifically nprogress. I’ve found it to be the perfect companion to Turbolinks to create a great user experience.

The Problem

In a traditional web application, when we click a link or submit a form, we get a loading spinner where the favicon typically appears. We might also see text in the status bar saying “Connecting…” or “Loading…”. These are the loading indications that internet users have become accustomed to.

By adopting Turbolinks, we not longer get those loading feedback mechanisms because the request for the new page is asynchronous. Once the request is complete, the new content is rendered in place of the previous page’s body element. For fast page loads, this isn’t a problem. However, if you have applications like mine, every once in awhile, you might have a page request take a few seconds (reasons for this are beyond the scope of this article). In those cases, a user might click a link and sit there for 2-3 sec. without any indication the page is loading. While Turbolinks generally improves the user experience of our application, having no user feedback for several seconds is not ideal (ideally, you’d want to address a page request that takes multiple seconds). This is where nprogress can help.

The Solution

nprogress is a progress loading indicator, like what you see on YouTube.

Like other JavaScript libraries, there’s a Ruby Gem that vendors the code and includes it in the Rails asset pipeline.

We’ll first add nprogress-rails to our Gemfile:

gem "nprogress-rails"

Bundle to install the new gem:

$ bundle install

Now with nprogress installed, we need to include the JavaScript in our application. We’ll do this by adding the following the app/assets/javascripts/application.js manifest:

//= require nprogress
//= require nprogress-turbolinks

We first include the nprogress JavaScript source, and then an adapter that’ll hook the Turbolinks request to the progress indicator.

Note: If you’re familiar with Turbolinks and its events, you’ll recognize the events triggered.

By default, the nprogress loading bar is anchored to the top of the browser window, but we need to include some CSS to make this work. Let’s open the app/assets/stylesheets/application.scss manifest file and add the following:

*= require nprogress
*= require nprogress-bootstrap

Note: Including nprogress-bootstrap isn’t necessary if you don’t use Bootstrap in your application. I typically do, so I’m going to include it.

At this point, we’ll have a working loading indicator. But what if we want to tweak the styles to match your application’s theme?

Customizing Nprogress Styles

Because the nprgress styles are Sass, we can overwrite the variables for customization.

There are 3 variables available to overwite:

• $nprogress-color
• $nprogress-height
• $nprogress-zindex

For Bark, we have an aqua accent color with use throughout the site. It made sense for the nprogress loading indicator to be that same color.

Back in our app/assets/stylesheets/application.scss, I overwrote the variable before including the nprogress source code:

$nprogress-color: #37c8c9;

@import "nprogress";
@import "nprogress-bootstrap";

Summary

I’ve found nprogress to be a great companion library to Turbolinks. The two libraries together provide a much better user experience over full page refreshes. Turbolinks helps asynchronously load the page content that’s changing and nprogress gives the user feedback that their request is in progress. Now, even when a user has to suffer through multi-second page loads, at least they’ll know it’s not broken and don’t have to click again.

The latest version of Turbolinks has a progress bar built-in. I’m looking forward to removing the dependency if it performs similarly.

What are Post-Install Messages?

As Rubyists, we have plenty experiences installing gems. By running gem install rails, we’re asking Rubygems to install the gem named rails on to our system.

The typical output of installing a gem with no other dependencies (assuming it completes successfully) is minimal:

$ gem install so_meta
Successfully installed so_meta-0.1
1 gem installed

As you can see, we ran gem install so_meta and the output confirmed the install, with nothing more.

If you’ve used the HTTParty gem, you’ve probably seen the additional line of output it generates when you run gem install httparty:

$ gem install httparty
When you HTTParty, you must party hard!
Successfully installed httparty-0.13.7
1 gem installed

Where did When you HTTParty, you must party hard! come from? It turns out the source of that text was a [post-install message defined in the gemspec.

Now, I know what you’re probably thinking…what good is that message? That’s up for debate. In fact, that specific message in HTTParty has been the source of much debate over the years.

How to configure a Post-Install Message

As we’ve seen before, the gemspec file (located at the root of the gem) defines the specification of a Ruby gem. Using bundler to bootstrap a new gem will automatically create this file. Here’s an example of a default gemspec file created by bundler using the command bundle gem brandon (brandon being the name of my fake gem):

# coding: utf-8
lib = File.expand_path('../lib', __FILE__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
require 'brandon/version'

Gem::Specification.new do |spec|
  spec.name          = "brandon"
  spec.version       = Brandon::VERSION
  spec.authors       = ["Brandon Hilkert"]
  spec.email         = ["brandonhilkert@gmail.com"]

  spec.summary       = %q{TODO: Write a short summary, because Rubygems requires one.}
  spec.description   = %q{TODO: Write a longer description or delete this line.}
  spec.homepage      = "TODO: Put your gem's website or public repo URL here."

  # Prevent pushing this gem to RubyGems.org by setting 'allowed_push_host', or
  # delete this section to allow pushing this gem to any host.
  if spec.respond_to?(:metadata)
    spec.metadata['allowed_push_host'] = "TODO: Set to 'http://mygemserver.com'"
  else
    raise "RubyGems 2.0 or newer is required to protect against public gem pushes."
  end

  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
  spec.bindir        = "exe"
  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
  spec.require_paths = ["lib"]

  spec.add_development_dependency "bundler", "~> 1.11"
  spec.add_development_dependency "rake", "~> 10.0"
  spec.add_development_dependency "minitest", "~> 5.0"
end

Aside from summary, description, and homepage, we can leave the rest of this file intact. These setter attributes on the Gem::Specification.new instance allow us to define the options and metadata necessary to properly configure and release a Ruby gem (see the Rubygems specification reference for an extensive list of options).

As you might have guessed by now, a post-install message is an option available in the gemspec. The value can be a simple string or a more complex heredoc.

The simplest example being:

spec.post_install_message = "My test post-install message."

With that in our gemspec, now when we install our fake gem brandon, we’ll see the following output:

$ gem install brandon
My test post-install message.
Successfully installed brandon-0.1.0
1 gem installed

Easy, huh?

If we wanted to include a more complex message with line breaks and other formatting, another option would be something like:

s.post_install_message = %q{
My test post-install message.

Another post-install message a few lines down.
}

The formatting of these messages can get weird because whitespace is preserved in multiline strings. If you’re looking to include anything more complex than a simple string literal, it’s worth experimenting by installing locally and confirming it’s what you want.

The NewRelic gem is another example that comes to mind that commonly includes more than just a simple string. Looking back at an older version of the NewRelic gem yields the following post_install_message:

s.post_install_message = %q{
Please see http://support.newrelic.com/faqs/docs/ruby-agent-release-notes
for a complete description of the features and enhancements available
in version 2.12 of the Ruby Agent.

For details on this specific release, refer to the CHANGELOG file.

}

Notice the message includes a line break both before and after the message. This will help isolate from our post-install messages when included in the longer output of a command like bundle install. Again, if you’re focused on formatting and getting it right, it’s worth installing locally in to something like a Rails application which yields more output than using gem install [gemname].

When to Use Post-Install Messages

The examples above use post-install messages for different reasons. HTTParty’s message wasn’t for a serious technical or information reason, just a light-hearted message that’s garnered quite a bit of negative attention from users that don’t appreciate it.

My suggestion would be to avoid any non-sensical messages and only provide a post-install message for something like breaking changes or information you think is critical to the usage of your gem. In most cases, post-install messages are most useful when a user is upgrading from an older version of your gem and the new version includes backwards-incompatible changes. Whether is be syntactical changes or core functionality, post-install messages provide us as gem authors a means to keep our users updated.

What to Include in Post-Install Messages

If you’re adhering to semantic versioning and introduce any breaking changes in a major release, a post-install message is a great way to warn users about the changes. However, one thing you want to avoid is enumerating your gem’s full changelog in the message. In most cases, a short notice about the backwards incompatible changes and a URL for more information is enough.

I introduced a new public API in Sucker Punch, which warranted a major release. Because of these backwards-incompatible changes, I added a post-install message to the new version:

$ gem install sucker_punch
Fetching: sucker_punch-2.0.1.gem (100%)
Sucker Punch v2.0 introduces backwards-incompatible changes.
Please see https://github.com/brandonhilkert/sucker_punch/blob/master/CHANGES.md#20 for details.
Successfully installed sucker_punch-2.0.1
1 gem installed

“Sucker Punch v2.0 introduces backwards-incompatible changes” provided the heads up that something was different. The URL in the following line allows the users to see a more extension list of the changes and to make adjustments in their application if necessary.

Summary

In addition to documentation through a README or wiki, post-install messages are a great way to keep users of our gems informed. Having access to the output of their console is a privilege, so use it sparingly. Like the boy who cried wolf, if we include a wall of text with each release of our gem, users will learn to ignore it and that would negatively affect its value for everyone.

When I published the previous article, Mike Perham rightly pointed out the opportunity to support the previous versions’s API through an opt-in module. I was hesitant to include support for the old syntax by default, but allowing a developer to require a file to get the old syntax made complete sense to me. My intent was never to abandon existing Sucker Punch users, but it felt necessary for the success of the project going forward.

The Problem

The following is an example of enqueueing a background job with Sucker Punch using the old syntax:

LogJob.new.async.perform("new_user")

And with the new syntax:

LogJob.perform_async("new_user")

How do we support the old syntax in the new version?

Let’s step back and reminder ourselves of what a typical job class looks like:

class LogJob
  include SuckerPunch::Job

  def perform(event)
    Log.new(event).track
  end
end

Important points to notice:

1. Each job includes the SuckerPunch::Job module to gain access to asynchronous behavior
2. Each job executes its logic using the perform instance method
3. Each job passes arguments needed for its logic as arguments to the perform instance method

The Solution

We’ll start with the test:

# test/sucker_punch/async_syntax_test.rb

require 'test_helper'

module SuckerPunch
  class AsyncSyntaxTest < Minitest::Test
    def setup
      require 'sucker_punch/async_syntax'
    end

    def test_perform_async_runs_job_asynchronously
      arr = []
      latch = Concurrent::CountDownLatch.new
      FakeLatchJob.new.async.perform(arr, latch)
      latch.wait(0.2)
      assert_equal 1, arr.size
    end

    private

    class FakeLatchJob
      include SuckerPunch::Job

      def perform(arr, latch)
        arr.push true
        latch.count_down
      end
    end
  end
end

Note: Some details of this are complex because the job’s code is running in another thread. I’ll walk through those details in a future article.

The basic sequence is: 1. require sucker_punch/async_syntax 2. Execute a background job using the async syntax 3. Assert changes made in that job were successful

Running the tests above, we get the following error:

1) Error:
SuckerPunch::AsyncSyntaxTest#test_perform_async_runs_job_asynchronously:
LoadError: cannot load such file -- sucker_punch/async_syntax
  /Users/bhilkert/Dropbox/code/sucker_punch/test/sucker_punch/async_syntax_test.rb:6:in `require'
  /Users/bhilkert/Dropbox/code/sucker_punch/test/sucker_punch/async_syntax_test.rb:6:in `setup'

1 runs, 0 assertions, 0 failures, 1 errors, 0 skips

Ok, so the file doesn’t exist. Let’s create it and re-run the tests:

1) Error:
SuckerPunch::AsyncSyntaxTest#test_perform_async_runs_job_asynchronously:
NoMethodError: undefined method `async' for #<SuckerPunch::AsyncSyntaxTest::FakeLatchJob:0x007fbc73cbf548>
  /Users/bhilkert/Dropbox/code/sucker_punch/test/sucker_punch/async_syntax_test.rb:12:in `test_perform_async_runs_job_asynchronously'

Progress! The job doesn’t have an async method. Let’s add it:

module SuckerPunch
  module Job
    def async # <--- New method
    end
  end
end

Notice: We’re monkey-patching the SuckerPunch::Job module. This will allow us to add methods to the background job since it’s included in the job.

The tests now:

1) Error:
SuckerPunch::AsyncSyntaxTest#test_perform_async_runs_job_asynchronously:
NoMethodError: undefined method `perform' for nil:NilClass
  /Users/bhilkert/Dropbox/code/sucker_punch/test/sucker_punch/async_syntax_test.rb:12:in `test_perform_async_runs_job_asynchronously'

More progress…the async method we added returns nil, and because of the syntax async.perform, there’s no perform method on the output of async. In short, we need to return something from async that responds to perform and can run the job.

In its most basic form, suppose we create a proxy object that responds to perform:

class AsyncProxy
  def perform
  end
end

We’ll need to do some work in perform to execute the job, but this’ll do for now. Now, let’s integrate this new proxy to our async_syntax.rb file and return a new instance of the proxy from the async method:

module SuckerPunch
  module Job
    def async
      AsyncProxy.new # <--- new instance of the proxy
    end
  end

  class AsyncProxy
    def perform
    end
  end
end

Running our tests gives us the following:

1) Error:
SuckerPunch::AsyncSyntaxTest#test_perform_async_runs_job_asynchronously:
ArgumentError: wrong number of arguments (2 for 0)
  /Users/bhilkert/Dropbox/code/sucker_punch/lib/sucker_punch/async_syntax.rb:9:in `perform'
  /Users/bhilkert/Dropbox/code/sucker_punch/test/sucker_punch/async_syntax_test.rb:12:in `test_perform_async_runs_job_asynchronously'

Now we’re on to something. We see an error related to the number of arguments on the perform method. Because each job’s argument list will be different, we need to find a way to be flexible for whatever’s passed in, something like…the splat operator! Let’s try it:

module SuckerPunch
  module Job
    def async
      AsyncProxy.new
    end
  end

  class AsyncProxy
    def perform(*args) # <--- Adding the splat operator, will handle any # of args
    end
  end
end

The tests now:

1) Failure:
SuckerPunch::AsyncSyntaxTest#test_perform_async_runs_job_asynchronously [/Users/bhilkert/Dropbox/code/sucker_punch/test/sucker_punch/async_syntax_test.rb:14]:
Expected: 1
Actual: 0

At this point, we’ve reached the end of test output suggesting the path forward. This error is saying, “Your assertions failed.”. This is good because it means our syntax implementation will work and it’s just about executing the actual job code in the proxy’s perform method.

We want to leverage our new syntax (perform_async) to run the actual job asynchronously so it passes through the standard code path. To do so, we’ll need a reference to the original job in the proxy object. Let’s pass that to the proxy during instantiation:

module SuckerPunch
  module Job
    def async
      AsyncProxy.new(self) # <--- Pass the job instance
    end
  end

  class AsyncProxy
    def initialize(job) # <--- Handle job passed in
      @job = job
    end

    def perform(*args)
    end
  end
end

Now that the proxy has a reference to the job instance, we can call the perform_async class method to execute the job:

module SuckerPunch
  module Job
    def async
      AsyncProxy.new(self)
    end
  end

  class AsyncProxy
    def initialize(job)
      @job = job
    end

    def perform(*args)
      @job.class.perform_async(*args) # <---- Execute the job
    end
  end
end

Lastly, the tests:

ress ENTER or type command to continue
bundle exec rake test TEST="test/sucker_punch/async_syntax_test.rb"
Run options: --seed 43886

# Running:

.

1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

Success!

Just like that, new users of Sucker Punch will be able to add require 'sucker_punch/async_syntax' to their projects to use the old syntax. This will allow existing projects using Sucker Punch to take advantage of the reworked internals without the need to make sweeping changes to the enqueueing syntax.

Support for the old syntax will be available for foreseeable future via this include. All new code/applications should use the new syntax going forward.

Conclusion

Before realizing a proxy object would work, I tinkered with alias_method and a handful of other approaches to latching on to the job’s perform method and saving it off to execute later. While some combinations of these might have worked, the proxy object solution is simple and elegant. There’s no magic, which means less maintenance going forward. The last thing I want is to make a breaking change, add support for the old syntax and find the support to be bug-ridden.

Ruby is incredibly flexible. Sometimes a 9-line class is enough to get the job done without reaching for an overly complex metaprogramming approach.

Having said all that, Sucker Punch v2 has been released!

Having gotten familiar with Celluloid from my work on Sidekiq, I knew Celluloid had all the pieces to puzzle to make this easier. In fact, one of the earliest incarnations of Sucker Punch wasn’t a gem at all, just some Ruby classes implementing the pieces of Celluloid necessary to put together a background processing queue.

The resulting code was less than ideal. It worked, but didn’t feel like an API that anyone would want to use. From a beginner’s perspective, this would stop adoption in its tracks. This is a common challenge with any code we encounter. No doubt, the Ruby standard library has all the tools necessary to make just about anything we can dream of, but sometimes the result isn’t ideal. It’s the same reason libraries like Rspec and HTTParty can exist. Developers prefer to use simplistic DSLs over convoluted, similarly-functioning code. Ruby has always been a language where developers consistently tout their ability to write code that reads well, feeding the levels of developer happiness.

Why Rewrite Sucker Punch

It started when a version of Celluloid was yanked from RubyGems.org. This resulted in a flurry of tweets and GH issues detailing their inability to bundle their applications.

As of version 0.17, methods in public API changed without supporting documentation. On top of that, the core celluloid gem was split in to a series of child gems causing navigation to be painful.

This made my life as the Sucker Punch maintainer difficult. There were some requests to upgrade Sucker Punch to use Celluloid ~> 0.17 and I feared of what would happen if I did. This caused me to think about what the future of Sucker Punch looked like without Celluloid. I still use Sucker Punch and believe it’s a valuable asset to the community. My goal was to find a way to move it forward productively without experiencing similar pains.

In the end, thanks to some communinity contributions, Sucker Punch 1.6.0 was released with Celluloid 0.17.2 support.

Where to now?

Around that same time, Mike Perham had been writing about his experiences optimizing Sidekiq and whether continuing with Celluloid made sense for Sidekiq. Having less experience with multi-threading, it didn’t make sense for me to reinvent the wheel.

I had been hearing about concurrent-ruby through a variety of outlets, one of which was Rails replacing the existing concurrency latch with similar functionality from concurrent-ruby. After poking around concurrent-ruby, I realized it had all the tools necessary to build a background job processing library. Much like Celluloid in that respect, had the tools, but lacked the simple DSL for the use case.

What if Sucker Punch used concurrent-ruby in place of celluloid?

I can hear what you’re thinking…“What’s the difference? You’re swapping one dependency for another!”. 100% true. The difference was that the little bit of communication I had with the maintainers of concurrent-ruby felt comfortable, easy, and welcoming. And with concurrent-ruby now a dependency of Rails, it’s even more accessible for those using Sucker Punch within a Rails application (a common use case). But like before, there’s no way to be sure that concurrent-ruby won’t cause similar pains/frustrations.

Celluloid Basics

A basic Sucker Punch job looks like:

class LogJob
  include SuckerPunch::Job

  def perform(event)
    Log.new(event).track
  end
end

To run the job asynchronously, we use the following syntax:

LogJob.new.async.perform("new_user")

The most interesting part of this method chain is the async. Removing async, leaves us with a call to a regular instance method.

It so happens that async is a method in Celluloid that causes the next method to execute asynchronously. And this works because by including SuckerPunch::Job, we’re including Celluloid, which gives us the async method on instances of the job class.

Developing APIs

If you’re familiar with the basics of Celluloid, you’ll notice there’s not much to Sucker Punch. It adds the Celluloid functionality to job classes and does some things under the hood to ensure there’s one queue for each job class.

Early in my concurrent-ruby spike, I realized what a mistake to tie Sucker Punch’s API to the API of Celluloid. Tinkering with the idea of removing Celluloid has left Sucker Punch with two options:

1. Continue using the async method with the new dependency
2. Break the existing DSL and create a dependency-independent syntax and try my best to document and support the change through the backwards-incompatible change

Option 1 is the easy way out. Option 2 is more work, far more scary, but the right thing to do.

I decided to abandon my thoughts about previous versions and write as if it were new today. This will be the basis for the next major release of Sucker Punch (2.0.0).

Settling on abandoning the existing API, the next question is, “What should the new API look like?”.

Being a fan of Sidekiq, it didn’t take long for me to realize it could actually make developers lives easier if Sucker Punch’s API was the same.

Switching between Sidekiq and Sucker Punch is not uncommon. I look at Sidekiq as Sucker Punch’s big brother and often suggest people use it instead when the use case makes sense.

If you’re familiar with Sidekiq, using the perform_async class method should look familiar:

LogJob.peform_async("new_user")

So why not use the same for Sucker Punch?

If so, switching between Sidekiq and Sucker Punch would be no more than swapping include Sidekiq::Worker for include SuckerPunch::Job in the job class, aside from the gem installation itself. The result would be less context switching and more opportunity focus on the important parts of the application.

I can hear the same question again, “What’s the difference? You suggested isolating yourself from a dependency’s API and now you’re suggesting using another!”. I look at this one a little differently…

Sidekiq is uniquely positioned in the community as a paid open source project. We’re happy users of Sidekiq Pro and continue to do so for the support. You can certainly get support for the open source version, but one way to ensure Sidekiq is actively maintained is by paying for it. This financial support from us and others decreases the likelihood Mike will choose to abandon it. Mike’s also been public about his long-term interest in maintaining Sidekiq. With all this in mind, I’m willing to bank on its existence as the defacto way to enqueue jobs for background processing.

And if for some reason Sidekiq does disappear, there’s nothing lost on Sucker Punch. There’s no dependency. Just a similar syntax.

Sucker Punch 2.0.0 will have 2 class methods to enqueue jobs:

LogJob.perform_async("new_user")

and

LogJob.perform_in(5.minutes, "new_user")

The latter defining a delayed processing of the perform method 5 minutes from now.

Summary

Settling on a library’s API isn’t easy. Isolating it from underlying dependencies is the best bet for long-term stability. Using the adapter pattern can help create a layer (adapter) between your code and the dependency’s API. But like always, there are always exceptions.

I’m taking a leap of faith that doing what I believe is right won’t leave existing users frustrated, ultimately abandoning Sucker Punch altogether.

Sucker Punch v2.0 is shaping up to be the best release yet. I’m looking forward to sharing it with you.

Sadly, the answer is, “it depends”. The “depends” part is specific to your context. While microservices may seem like the right move for you and your application, it’s also possible it could cause a mess if not done carefully.

This post will explore opportunities for splitting out unique microservices using Sidekiq, without introducing an enterprise message broker like RabbitMQ or Apache Kafka.

When are Microservices right?

Martin Fowler wrote about trade-offs that come when introducing microservices.

The article outlines 6 pros and cons introduced when you moved a microservices-based architecture. The strongest argument for microservices is the strengthening of module boundaries.

Module boundaries are naturally strengthened when we’re forced to move code to another codebase. The result being, in most cases a group of microservices appears to be better constructed than the legacy monolith it was extracted from.

There’s no doubt Rails allows developers to get something up and running very quickly. Sadly, you can do so while making a big mess at the same time. It’s worth noting there’s nothing stopping a monolith from being well constructed. With some discipline, your monolith can be the bright and shiny beauty that DHH wants it to be.

Sidekiq Queues

Ok, ok. You get it. Microservices can be awesome, but they can also make a big mess. I want to tell you about how I recently avoided a big mess without going “all in”.

There’s no hiding I’m a huge Sidekiq fan. It’s my goto solution for background processing.

Sidekiq has the notion of named queues for both jobs and workers. This is great from the standpoint that it allows you to put that unimportant long-running job in a different queue without delayed other important fast-running jobs.

A typical worker might look like:

class ImportantWorker
  include Sidekiq::Worker

  def perform(id)
    # Do the important stuff
  end
end

If we want to send this job to a different queue, we’d add sidekiq_options queue: :important to the worker, resulting in:

class ImportantWorker
  include Sidekiq::Worker
  sidekiq_options queue: :important

  def perform(id)
    # Do the important stuff
  end
end

Now, we need to make sure the worker process that’s running the jobs knows to process jobs off this queue. A typical worker might be invoked with:

bin/sidekiq

Since new jobs going through this worker will end up on the important queue, we want to make sure the worker is processing jobs from the important queue too:

bin/sidekiq -q important -q default

Note: Jobs that don’t specify a queue will go to the default queue. We have to include the default queue when we using the -q option, otherwise the default queue will be ignored in favor of the queue passed to the -q option.

The best part, you don’t even have to have multiple worker processes to process jobs from multiple queues. Furthermore, the important queue can be checked twice as often as the default queue:

bin/sidekiq -q important,2 -q default

This flexibility of where jobs are enqueued and how they’re processed gives us an incredible amount of freedom when building our applications.

Extracting Worker to a Microservice

Let’s assume that we’ve deployed your main application to Heroku. The application uses Sidekiq and we’ve included a Redis add-on. With the addition of the add-on, our application now has a REDIS_URL environment variable that Sidekiq connects to on startup. We have a web process, and worker process. A pretty standard Rails stack:

What’s stopping us from using that same REDIS_URL in another application?

Nothing, actually. And if we consider what we know about the isolation of jobs in queue and workers working on specific queues, there’s nothing stopping us from having workers for a specific queue in a different application altogether.

Remember ImportantWorker, imagine the logic for that job was better left for a different application. We’ll leave that part a little hand-wavey because there still should be a really good reason to do so. But we’ll assume you’ve thought long and hard about this and decided the core application was not a great place for this job logic.

Extracting the worker a separate application might now look something like this:

Enqueueing Jobs with the Sidekiq Client

Typically, to enqueue the ImportantWorker above, we’d call the following from our application:

ImportantWorker.perform_async(1)

This works great when ImportantWorker is defined in our application. With the expanded stack above, ImportantWorker now lives in a new microservice, which means we don’t have access to the ImportantWorker class from within the application. We could define it in the application just so we can enqueue it, with the intent that the application won’t process jobs for that worker, but that feels funny to me.

Rather, we can turn to the underlying Sidekiq client API to enqueue the job instead:

Sidekiq::Client.push(
  "class" => "ImportantWorker",
  "queue" => "important",
  "args" => [1]
)

Note: We have to be sure to define the class as a string "ImportantWorker", otherwise we’ll get an exception during enqueuing because the worker isn’t defined in the application.

Processing Sidekiq Jobs from a Microservice

Now we’re pushing jobs to the important queue, but have nothing in our application to process them. In fact, our worker process isn’t even looking at that queue:

bin/sidekiq -q default

From our microservice, we setup a worker process to ONLY look at the important queue:

bin/sidekiq -q important

We define the ImportantWorker in our microservice:

class ImportantWorker
  include Sidekiq::Worker
  sidekiq_options queue: :important

  def perform(id)
    # Do the important stuff
  end
end

And now when the worker picks jobs out of the important queue, it’ll process them using the ImportantWorker defined above in our microservice.

If we wanted to go one step further, the microservice could then enqueue a job using the Sidekiq client API to a queue that only the core application is working on in order to send communication back the other direction.

Summary

Any architectural decision has risks. Microservices are no exception. Microservices can be easier than an enterprise message broker, cluster of new servers and a handful of devops headaches.

I originally dubbed this the “poor man’s message bus”. With more thought, there’s nothing “poor” about this. Sidekiq has a been a reliable piece of our infrastructure and I have no reason to believe that’ll change, even if we are using it for more than just simple background processing from a single application.

In the previous article in this series, we introduced a billing service to determine which features an account could access. If you remember, the email service was a “fire and forget” operation and was capable of handling throughput delays given its low value to the core application.

This post will explore how we handle synchronous communication for a service like billing where an inline response is required to service a request from the core application.

Background

If you remember from the previous post, we introduced the billing service to an infrastructure that looked like this:

Handling multiple pricing tiers in a SaaS app means you have to control authorization based on account status. Our billing service encapsulates the knowledge of which features correspond to which pricing tier.

For instance, one feature is the ability to send trackable email to contacts in your PipelineDeals account. To service this request, we add an option to the bulk action menu from a list view:

Service Request

Before we can conditionally show this option based on the pricing tier, we have to first make a request to the billing service to get the list of features available to that user.

class Billing::Features
  def initialize(user)
    @user = user
    @account = user.account
  end

  def list
    Rails.cache.fetch("account_#{account.id}_billing_features") do
      response = Billing::Api.get "account/#{account.id}/features"
      response['features']
    end
  end

  private

  attr_reader :user, :account
end

Billing::Api, in this case, is a wrapper around the API calls to handle exceptions and other information like security.

Note: When making synchronous HTTP calls like this, it’s worth considering the failure state and providing a default response set in that case so the user isn’t burdened with a failure page. In this case, one option would be dumb down the features on the page to the most basic tier.

Serving a JSON API

Plenty of articles have been written about how to create a JSON API with Rails, so we won’t rehash those techniques here. Instead, we’ll highlight patterns we’ve used for consistency.

We tend to reserve the root URL namespace for UI-related routes, so we start by creating a unique namespace for the API:

namespace :api do
  resources :account do
    resource :features, only: :show
  end
end

This setup gives us the path /api/account/:account_id/features. We haven’t found a need for versioning internal APIs. If we decided to in the future, we could always add the API version as a request header.

The features endpoint looks like:

class Api::FeaturesController < Api::ApiController
  skip_before_filter :verify_authenticity_token

  def show
    render json: {
      success: true,
      features: AccountFeatures.new(@account_id).list
    }
  end
end

Notice Api::FeaturesController inherits from Api::ApiController. We keep the API-related functionality in this base controller so each endpoint will get access to security and response handling commonalities.

AccountFeatures is a PORO that knows how to list billing features for a particular account. We could’ve queried it straight from an ActiveRecord-based model, but our handling of features is a little more complicated than picking them straight from the database.

Another note here is that we haven’t introduced a serializing library like active_model_serializers or jbuilder. Using render json alone has serviced us well for simple APIs. We reach for something more complex when the response has more attributes than shown above.

Handling Service Response

By introducing Rails.cache, we can serve requests (after the initial) without requiring a call to the billing service.

One of the first things we do is serialize the set of features to JavaScript so our client-side code has access:

<%= javascript_tag do %>
  window.Features = <%= Billing::Features.new(logged_in_user).list.to_json %>;
<% end %>

We also include a helper module in to our Rails views/controllers, so we can handle conditional feature logic:

module Features
  def feature_enabled?(feature)
    Billing::Features.new(logged_in_user).list.include?(feature.to_s)
  end
end

Synchronous Side Effects

When we looked at asynchronous service requests, there was less immediacy associated with the request due to its “fire-and-forget” nature. A synchronous request, on the other hand, will handle all requests to the core application, so scaling can be challenge and infrastructure costs can add up.

In addition to the infrastructure costs, performance can be a factor. If the original page response time was 100ms and we’re adding a synchronous service request that takes another 100ms, all of a sudden we’ve doubled our users’ response times. And while this architectural decision might seem like an optimization, I’m positive none of our users will thank us for making their page load times 2x slower.

Summary

As you can see, there’s little magic to setting up a synchronous service request.

Challenges appear when you consider failure states at every point in the service communication - the service could be down, or the HTTP request itself could fail due to network connectivity. As mentioned above, providing a default response during service failure is a great start to increasing the application’s reliability. Optionally, the circuit break pattern can provide robust handling of network failures.

Part 4 in this series will cover how we manage asynchronous communication between services, specifically around an open source gem we built called Mantle.

In the previous article in this series, we moved the responsibility of emails to a separate Rails application. In order to leverage this new service, we created a PORO to encapsulate the specifics of communicating with our new service by taking advantage of Sidekiq’s built-in retry mechanism to protect from intermittent network issues.

Communication between microservices can be broken down in to 2 categories: synchronous and asynchronous. Understanding when to use each is critical in maintaining a healthy infrastructure. This post will explore details about these two methods of communication and their associated use cases.

Background

Continuing the discussion of our architecture from last time, we have a primary Rails web application serving the majority of our business logic. We now have an additional application that’s only responsibility is the formatting and sending of emails.

In this article, we’ll discuss the addition of our Billing service. The service’s responsibility to is to process transactions related to money. This can come in the form of a trial conversion, adding a seat to an additional account, or deleting users from an existing account, among others.

Like many SaaS applications, PipelineDeals has multiple tiers of service. The most expensive intended for customers needing advanced functionality. Part of the billing service’s responsibility is to manage the knowledge of which features an account can access.

So stepping back to the main PipelineDeals web application, the app has to decide which features to render at page load. Because the billing service is our source of truth for this information, a page load will now require a call to this service to understand which features to render.

This new dependency looks a little different than the email dependency from the previous article. Email has the luxury of not being in the dependency path of a page load. Very few customers will complain if an email is 10 seconds late. On the other hand, they’ll complain immediately if their account won’t load, and rightfully so.

An interesting benefit from having already extracted the email service is that the billing service sends email regarding financial transactions and account changes. Typically, we would have done the same thing for every other Rails app that needed to send email, which was integrate ActionMailer and setup the templates and mailers needed to do the work. In this case, we can add those emails to the email service and use the same communication patterns we do from the main web application to trigger the sending of an email from the billing service. This does require making changes to 2 different projects for a single feature (business logic in billing and mailer in email), but removes the necessity to configure another app to send email properly. We viewed this as a benefit.

Asynchronous Events

As the easier of the two, asynchronous would be any communication not necessary for the request/response cycle to complete. Email is the perfect example. Logging also falls in to this category.

For the networks gurus out there, this would be similar to UDP communication. More of a fire-and-forget approach.

An email, in this case, is triggered due to something like an account sign up. We send a welcome email thanking the customer for signing up and giving them some guidance on how to get the most benefit from the application. Somewhere in the process of signing up, the code triggers an email and passes along the data needed for email template.

As shown in the previous article, the call to send the email looks something like this:

Email.to current_user, :user_welcome

The value in this call is that under the covers, it’s enqueuing a Sidekiq job:

EmailWorker.perform_async(opts)

where opts is a hash of data related to the email and the variables needed for the template.

Note: Because the options are serialized to JSON, values in hash must be simple structures. Objects won’t work here.

As you can see above, the code invoking the Email.to method doesn’t care about what it returns. In fact, it doesn’t return anything we care about at this point. So as long as the method is called, the code can move forward without waiting for the email to finish sending.

Extracting asynchronous operations like this that exist in a code path is a great way to improve performance. There are times, though, where deferring an operation to background job might not make sense.

For example, imagine a user changes the name of a person. They click one of their contact’s names, enter a new name, and click “Save”. It doesn’t make sense to send the task of updating the actual name in the database to a background job because depending on what else is in the queue at that time, the update might not complete until after the next refresh, which would make the user believe their update wasn’t successful. This would be incredibly confusing.

Logging is another perfect candidate for asynchronicity. In most cases, our users don’t care if a log of their actions has been written to the database before their next refresh. It’s information we may want to store, and as a result, can be a fire-and-forget operation. We can rest easy knowing we’ll have that information, soon-ish, and it won’t add any additional overhead to the end user’s request cycle.

The opposite of asynchronous events like this are synchronous events! (surprise right?). Let’s explore how they’re different.

Synchronous Events

We can look at synchronous events as dependencies of the request cycle. We use MySQL as a backend for the main PipelineDeals web application and queries to MySQL would be considered synchronous. In that, in order to successfully fulfill the current request, we require the information returned from MySQL before we can respond.

In most cases, we don’t think of our main datastore as a service. It doesn’t necessarily have a separate application layer on top of it, but it’s behavior and requirements are very much like a service.

If we consider the addition of our billing service above, we require information about the features allowed for a particular account before we can render the page. This allows us to include/exclude modules they should or should not see. The flow goes something like this:

Web request -> lookup account in DB -> Request features from Billing service -> render page

If the request to the billing service didn’t require a response, we would consider this to be an asynchronous service, which might change how we invoke the request for data.

Synchronous communication can happen over a variety of protocols. The most common is a JSON payload over HTTP. In general, it’s not the most performant, but it’s one of the easier to debug and is human-readable, so it tends to be pretty popular.

The synchronous services we’ve setup all communicate over HTTP. Rails APIs are a known thing. We’re familiar with the stack and the dependencies required to set up a common JSON API, which is a large part of the reason it’s our preferred communication protocol between services.

Summary

We’ve simplified the communication between services into these two categories. Knowing this helps dictate the infrastructure and configuration of the applications.

Next time, we’ll take a closer look at the synchronous side and the specifics about the JSON payloads involved to send an email successfully.

The PipelineDeals web application recently celebrated its ninth birthday. It’s seen its fair share of developers, all of whom had their own idea of “clean code”. As a team, we’d been brainstorming ways to wrangle certain areas of the application. The question we’d frequently ask ourselves was “How do we clean up [x] (some neglected feature of the application)?”.

Reasonable solutions ended up being:

1. Rewrite it
2. Rewrite and put it elsewhere

In short, we chose to rewrite many of the hairy areas of the app into separate services communicating over HTTP. It’s been about a year since our first commit in a separate service, and we’ve learned quite a bit since then. This is part 1 in a series of posts related to our transition to microservices.

How we got here

This was us 18 months ago. PipelineDeals was a crufty Rails 2 application that many of us were scared to open. It’d been several years of adding feature upon feature without consistent knowledge, style, or guidance. And it’s probably not surprising we had what we did. Regardless, we needed to fix it.

One of our goals was to move to Rails 3, and later more updated versions, but in order to get there, we had to refactor (or remove) quite a bit of code to make the transition easier.

This, to me, was a huge factor around our decision to move to a more service-focused approach. At this year’s Railsconf keynote, DHH joked about the “majestic monolith” and how many companies prematurely piece out services, all to later suffer pain when they realize it was a premature optimization.

The same could be said for our move. Instead of spinning out separate services, we could have cleaned up the mess we had by refactoring every nasty piece of our app. We could have turned our ugly monolith into a majestic one. But while it would’ve been possible, our team agreed we were better served by more or less starting over. Not in the big-bang rewrite sense, but instead to stand up brand new service apps when we added new features, and when it made sense. “Made sense” is the key here. There have been many times when it didn’t make sense over the past 12 months. But we’re learning and getting better at identifying the things that are good candidates for a more isolated service.

Now what?

Do we wait for the next requested feature or what?

At one of our weekly team hangouts, we watched a talk focused on starting by isolating the responsibility of Email. It was the perfect introduction and motivation for us to get a small win and some experience under our belts. For some reason prior, we didn’t have a great sense of how to start making that transition.

The idea was to take our emails (and there were plenty) and move them to a separate Rails app that’s only responsibilty is sending email. While it sounds trivial, the idea alone introduces a lot of interesting questions: What do we do with those really nasty emails that have 30 instance variables? What do we do if the email service is down? How do we trigger an email to be sent?

Rails new

We created a new Rails 4 app, removed all the stuff we didn’t need and created a golden shrine where emails could flourish…but seriously, that’s all it did. And it did it really well.

The next question was how to send emails from the main application. We’re very happy Sidekiq Pro users, and one of the benefits we love about Sidekiq is the built-in retries. This gives us a layer of reliability outside of the code layer. So rather than build some ad-hoc retry mechanism by creating a counter in ruby, and rescuing failures within a certain range, we shoot off a job. If it fails because the network is down, or the endpoint isn’t available, the job will retry soon after and continue down the happy path. Sidekiq retries are a recurring theme with our infrastructure. We’ve made a number of decisions around the fact that we have this advantage already built-in, and we might as well take advantage of it. More on that to come.

Communicate

The defacto communication method between services is over HTTP. And we did nothing different. Our services use JSON payloads to exchange data, which let’s us easily take advantage of Sidekiq on both ends.

So now, rather than invoking a built-in Rails mailer like:

UserMailer.welcome(current_user).deliver

we invoke a PORO to send off the communication:

Email.to current_user, :user_welcome

where Email is defined as

class Email
  def initialize(users, email_key, opts)
    @users, @email_key, @opts = users, email_key, opts
  end

  def self.to(users, email_key, opts = {})
    new(users, email_key, opts).queue_email
  end

  def queue_email
    opts[:email_key] = email_key
    opts[:to] ||= email_array
    opts[:name] ||= first_users_name
    opts[:user_id] ||= user_id
    opts[:account_id] ||= account_id

    json = JSON.generate(opts)
    RestClient.post(ENV["PIPELINE_EMAIL_URL"], json, :content_type => :json)
  end
end

There’re a number of use-case specific variables above, but the email_key is probably the most important. We used that to describe what email should be invoked on the service.

In the above example, we triggered the welcome email on the UserMailer class. We translated this request into an email key of user_welcome.

This key then gets interpreted by the Email service app and turned into an actual Mailer class and method within it. We could have done this in a variety of ways, but we split the string on the service-side at the _, and the first element described the mailer, the rest the method. So in this case, it gets interpreted as UserMailer#welcome.

One thing this pattern allowed us to do was almost full copy/paste the old mailer methods in to the new Email service application.

Failures, failures, failures

“What if the service is down?” you say, “the email request will fail!” Sure will.

So let’s wrap that request in a Sidekiq job to take advantage of the built-in retries.

Rather than invoke the following method in the email object:

RestClient.post(ENV["PIPELINE_EMAIL_URL"], json, :content_type => :json)

we’ll shoot off a Sidekiq job instead, changing the queue_email method to:

def queue_email
  opts[:email_key] = email_key
  opts[:to] ||= email_array
  opts[:name] ||= first_users_name
  opts[:user_id] ||= user_id
  opts[:account_id] ||= account_id

  EmailWorker.perform_async(opts)
end

There we have it. Network-proof email requests!

Not so fast…

Astute readers will probably recognize that the service-side network communication can potentially also fail. This is becoming a pattern, huh? More communication, more potential for failure and more potential headaches.

On the service side, we have a controller that takes in the request for the email and immediately serializes it to a Sidekiq job:

  def create
    EmailWorker.perform_async(parsed_params)
    head :accepted
  end

  private

  def parse_params
    JSON.parse(request.body) || {}
  end
end

Because we immediately serialize the job to Sidekiq, we’ve successfully acknowledged the job was received, and the main app’s Sidekiq job completes successfully. Now the email service can move on to doing the heavy-lifting in whatever way makes the most sense. In our case, we use Mailgun to send our emails, so the EmailWorker Sidekiq job invokes a new mailer based on the email_key param and sends it off to mailgun for transport. And because it’s wrapped in a Sidekiq job, we can sleep well knowing that the Mailgun request can fail and the job will successfully retry until it goes through.

Summary

Service communication is definitely not for the faint of heart and as a team, we can completely appreciate the challenges that come along with keeping services in sync now—especially having stood up about 8 new services in the last 12 months.

Sidekiq has been the queueing solution we’ve leaned on to keep communication in sync and reliable. We’ve also written a few internal tools that piggy-backy off Sidekiq that we’re really excited share with the community in the near future.

Part II, in this series, will discuss the methods of communication necessary to consider when implementing a service-based architecture.

Here’s a quick rundown of the details:

Is this bundle right for me?

The books in the Ruby Book Bundle span a range from beginner to advanced developers. As long as you understand the Ruby basics, though, you’ll learn a lot from these books.

You’ll probably get the most out of the bundle if you’re an intermediate Ruby dev, focused on building more specific development skills. But if you’re interested in mastering the most important and most challenging parts of Rails, like testing, refactoring, gem-building, metaprogramming/DSL-writing, and app-building, this bundle will be perfect for you.

What will I get?

When you buy the bundle, you’ll immediately get the following to download:

• Build a Ruby Gem (pdf, epub, mobi) with source code and screencasts
• Minitest Cookbook (pdf, epub, mobi) and source code examples
• Practicing Rails (pdf, epub, mobi)
• Fearless Refactoring (pdf)
• Ruby DSL Handbook (pdf, epub, mobi) with cheat sheets, sample code and screencasts
• Rebuilding Rails (pdf)

How long will the sale run?

The bundle will be available for 1 week only - ending July 10th, 11:59PM PDT.

What if I’m not happy with the bundle?

We want you to be happy with what you’ve bought! So if you’re not 100% satisfied with the value you got from the bundle, shoot me an email at brandonhilkert+bundle@gmail.com within 30 days, and I’ll refund your money.

I feel fortunate to be part of such a great group of developers. They’ve provided a ton of value to the community and their books continue to impress me.

Several articles back, I documented how I handle page-specific Javascript in a Rails application. My solution included a third-party jQuery plugin that did some magic on the $(document).ready function in combination with CSS style scoping to limit the functionality.

The plugin worked well for awhile, but with the advent of Turbolinks, the solution felt less and less appropriate. I’ve since settled on some techniques to not only handle page-specific Javascript, but overall organization and structure of Javascript within a Rails application. I’ve used it in a hand full of large applications over the past few months and it’s held up incredibly well.

The Problem

Using “sprinkles” of Javascript throughout a Rails application can get unwieldly fast if we’re not consistent. What we ideally want is some techniques and guidelines that can keep the Javascript organized in our projects. We also don’t want to have to disable Turbolinks to make our application work as we expect.

The Solution

Generally, javascript behavior can be boiled down to the following categories:

1. Behavior that’s “always on”
2. Behavior that’s triggered from a user action

But first, a few things to will help us stay organized…

Class Scoping

I still like to scope the body element of the layout(s) with the controller and action name:

<body class="<%= controller_name %> <%= action_name %>">
  <%= yield %>
</body>

This not only let’s us control access to the DOM through jQuery if we need to, but also provides some top-level styling classes to allow us to easily add page-specific CSS.

In the case we’re working on the proverbial blog posts application, the body tag ends up looking like:

<body class="posts index">
  <%= yield %>
</body>

This gives us the opportunity to scope CSS and Javascripts to all posts-related pages in the controller with the .posts class, or down to the specific page using a combination of both the controller and action: .posts.index.

Default Application Manifest

Here’s the default app/assets/javascripts/application.js:

// This is a manifest file that'll be compiled into application.js, which will include all the files
// listed below.
//
// Any JavaScript/Coffee file within this directory, lib/assets/javascripts, vendor/assets/javascripts,
// or any plugin's vendor/assets/javascripts directory can be referenced here using a relative path.
//
// It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the
// compiled file.
//
// Read Sprockets README (https://github.com/rails/sprockets#sprockets-directives) for details
// about supported directives.
//
//= require jquery
//= require jquery_ujs
//= require turbolinks
//= require_tree .

I start by removing the line //= require_tree .. I do this because if you don’t, the javascript files in the folder will be loaded in alphabetical order. As you’ll see below, there’s an initialization file that needs to be loaded before other Javascript. We’ll also remove the comments from the top of the file to preserve space.

So we’re left with:

//= require jquery
//= require jquery_ujs
//= require turbolinks

Initialization

Let’s start by adding the file app/assets/javascripts/init.coffee with the following:

window.App ||= {}

App.init = ->
  $("a, span, i, div").tooltip()

$(document).on "turbolinks:load", ->
  App.init()

Let’s dig in to each pagef of this:

window.App ||= {}

We’re creating the App object on window so the functionality added to the object is available throughout the application.

Next, we define an init() function on App to initialize common jQuery plugins and other Javascript libraries:

App.init = ->
  $("a, span, i, div").tooltip()

The call to $("a, span, i, div").tooltip() initializes Bootstrap Tooltips. This is an example of the type of libraries that can/should be setup here. Obviously, if you’re not using Bootstrap tooltips, you would haven’t this here, but coupled with the next line, we’ll see why this works.

As many have found out the hard way, when Turbolinks is enabled in a project, jQuery $(document).ready functions don’t get fired from page to page. In order to call the init() function on each page transition, we’ll hook in to the turbolinks:load event:

$(document).on "turbolinks:load", ->
  App.init()

Note: the turbolinks:load transition is also triggered on the well known document ready event, so there’s no need to add any special handling for first page load.

Lastly, we need to add init.coffee to the asset pipeline:

//= require jquery
//= require jquery_ujs
//= require turbolinks
//= require init

“Always On” Javascript Functionality

Now with the defaults out of the way, let’s take a look at adding some behavior.

Let’s assume one of our pages will show a Javascript graph of data. We’ll start by adding a file with a name related to that responsibility.

# app/assets/javascripts/app.chart.coffee

class App.Chart
  constructor: (@el) ->
    # intialize some stuff

  render: ->
    # do some stuff

$(document).on "turbolinks:load", ->
  chart = new App.Chart $("#chart")
  chart.render()

A few things to note here…

Structure

I created a class in the App namespace – the same we initialized in app/assets/javascripts/init.coffee. This gives us an isolated class that has a clear responsiblity. Like our Ruby, we want to do our best to keep its responsibilities to a minimium.

You might notice the file takes the form:

|
|
class definition
|
|


|
invocation
|

While this may seem obvious, it’s an important point to keep in mind. I’ve found it offers a predictable structure that allows me to open any coffeescript file that we’ve written in the project and generally know where to look for what.

Turbolinks-Proof

We called this “Always On” functionality because, as you probably noticed, using the following event listener $(document).on "turbolinks:load", ->, we know with Turbolinks, this gets triggered on every page transition.

Add to Manifest

Because we removed the //= require_tree . line in the default application.js manifest file, we’ll have to add our chart file to be included in the asset pipeline (last line):

//= require jquery
//= require jquery_ujs
//= require turbolinks
//= require init
//= require app.chart

Page-Specific Javascript

Uh oh, so maybe we don’t want the graph to show up on every page! In this case, we’re looking for “Always On” functionality for specific pages ONLY.

We can limit the page pages certain functionality is loaded on by using the classes we added to the body of the layout. In this case, a small conditional to the invocation can prevent this being triggered on pages it shouldn’t be.

$(document).on "turbolinks:load", ->
  return unless $(".posts.index").length > 0
  f = new App.Chart $("#chart")
  f.render()

We added return unless $(".posts.index").length > 0 to make sure App.Chart never gets instantiated if we’re on the .posts.index page. While this may seem verbose, I’ve found that it’s not very common to need page-specific functionality. There are probably plenty of libraries that do something similar, like the one I previously suggested. However, to me, limiting javascript to a single page and very explicit when I read the code, it’s almost never worth dragging in a separate plugin for this. YMMV.

User-Triggered Javascript

This type of Javascript is exactly what you’d think – Javascript invoked as a result of a user clicking or performing some type of action. You’re probably thinking, “I know how to do this, I’ll just add a random file to the javascripts directory and throw in some jQuery”. While this will functionally work just fine, I’ve found that keeping the structure of these files similar will give you great piece of mind going forward.

“data-behavior” Attribute

Let’s assume there’s a link in the user’s account that allows them to update their credit card. In this case, we have the following:

<%= link_to "Update Credit Card", "#", data: { behavior: "update-credit-card" } %>

You’ll probably notice the data-behavior tag being added to the link. This is the key we’ll use to attach the Javascript behavior.

We could have added a unique class to the link:

<%= link_to "Update Credit Card", "#", class: "update-credit-card" %>

or, perhaps, even assign an ID:

<%= link_to "Update Credit Card", "#", id: "update-credit-card" %>

Both of these techniques don’t really indicate whether we added the update-card-card for the use of CSS styling, or to attach Javascript behavior. So in my applications, I leave classes for styling ONLY.

So now to the Javascript:

App.Billing =
  update: ->
    # do some stuff

$(document).on "click", "[data-behavior~=update-credit-card]", =>
  App.Billing.update()

We can use the selector [data-behavior~=update-credit-card] to latch on to the data-behavior tag we defined in the view. We use the on jQuery method to ensure that we’re listening to this event whether the element’s on the page or not. This is what allows us to load this Javascript when on other pages and have it still work when a user clicks through to the page with the actual link on it.

We could latch on to the change event, or whatever is appropriate to the element we’re adding behavior.

Add to Manifest

Again, because Javascripts assets we add to app/assets/javascripts won’t automatically be inserted in to the asset pipeline, we’ll add //= require app.billing to the manifest file:

//= require jquery
//= require jquery_ujs
//= require turbolinks
//= require init
//= require app.chart
//= require app.billing

Summary

Using the techniques above, we can keep the Javascript in our Rails applications organized and predictable. We can rest easy knowing the files will all generally look the same. There’s not been any uses cases where this structure hasn’t worked for me personally.

One thing that makes me feel good about this approach is there’s no real magic or extra plugins. It’s using all the tools we already have in a basic Rails application, which is one less thing to maintain and keep up to date. Less depedencies == less pain down the road.

Turning the clocks back to when you knew much less about something is hard. But putting yourself back in that position can offer valuable insight to the opportunities available and how they might be improved in the future.

How I Started

Most come to the Rails community not knowing much about Ruby. Learning any new technology is hard. And learning a few at the same time is even harder.

This was me in 2009. A relatively new Rails 2.3 app was dropped in to my lap, and despite only having experience with PHP, my job was to aggressively ship new features. I read Agile Web Development with Rails cover to cover and dove in head first. Little did I know it would be one of the best career decisions of my life.

I spent the next few months pounding my head against my desk. The days and weeks of frustration seemed endless. And then…it just went away. The pain I’d endured merged in to an intense desire to dig in harder. There were more light bulbs moments in the months that followed than any other time I can remember.

During those intense months of frustration, I leaned heavily on the Rails and Philly.rb IRC rooms. In the former, I tried not to say anything too stupid. Fortunately, the latter felt more welcoming and approachable and I owe that group a lot of holding my hand through what might have otherwise been a deathwish for me and the Ruby language.

A few questions pop out in my head…I was confused about filtering an ActiveRecord query and was surprised to learn methods that were built in to the Ruby language would do exactly what I want. At the time, if it wasn’t in ActiveRecord, it might as well have not existed to me.

From someone who advocates for using tools without fancy DSL’s, this is hysterical to me. Ruby, of all things, had the answer. At that point, I’m almost certain I’d never seen the Ruby standard library documentation.

Rails Starts Where Ruby Stops

Out of convenience, Rails does a lot to make our experience with the Ruby language easier than it is out of the box. Like 2.hours.ago….none of this is possible if Rails doesn’t monkey patch the Integer class. For someone who doesn’t know any better (me in 2009!), being able to calls #hours on an integer just seems like something the language would do. Because Ruby was created for developer happiness, right?

So perhaps the approach of monkey patching doesn’t offer a clear indication of where functionality is coming from. The flip side of that argument is convenience. If I had to instantiate a time-related class every time I wanted “noon time yesterday”, maybe I’d be slightly less enthralled with my ability to get stuff done in Rails. Perhaps it would cater more to the true OO neckbeards, but also may have resulted in far less adoption. Who knows!

I don’t have strong numbers to back this up, but I’m guessing large majority of developers that get paid to write Ruby, due so within the context of a Rails application. And whether we want to admit it or not, a large reason new developers learn Ruby, is to learn Rails. So does it matter that newcomers don’t know Ruby?

From the standpoint of creating a web application fast and being able to iterate quickly, maybe not. But certainly if the person is interested in understanding the interworkings of what’s happening within the application, knowing where Ruby stops and Rails starts is ideal.

How to Start?

I’ve talked to quite a few people that are new to Ruby and I always struggle to suggest a good start project when they ask. Everyone learns differently and has different interests, but in general, I think there are core-level motivating factors that can keep someone focused and interested.

To me, it’s the following:

1. Does the project have real-world value?
2. Does the project offer immediate feedback?

Why does it matter if the project has real-world value? For one, continuing on something that doesn’t improve our lives is sometimes hard to keep up with. And learning Ruby/Rails is definitely something that’ll take more than a few nights and weekends. If the project we’re driving towards continues to seem desirable, we’ll have a better chance not to lose focus.

Second, I don’t want to get too philosophical, but there are plenty of resources that suggest if you want something badly enough and can visualize the end goal, there’s a higher likelyhood that it’ll come to be. Our desires will be stronger when we can see the end goal and know there is an increased real-world value for this application to be in existence.

The immediate feedback piece shortens the time we’re able to see changes and progress. This brings a lower rate of abandonment and better chance we’ll see the project through.

Rails answers both of these questions with a resounding, “YES!”. Think about it…

Does a Rails application have real world value? Of course it does. It’s a web application. There has been no better time to be focused on Rails, where it be for the web and the backend of a mobile app.

Does the project offer immediate feedback? Sure does! A couple keystrokes and a refresh can give you instant gratifcation in the browser (or the occasional disappointment!).

Whether it’s a command-line tool, game, or other utility, I struggle to find other oppotunities to get people started. Frankly, many newcomers to Rails have never used a terminal before. So why would we suggest a command-line application as a good place to start? This is especially true for someone with little programming experience.

For the experienced developer, it’d be much easier to suggest writing something like a markdown processor, but only because they have the context of another language. At that point, they’re really just comparing the Ruby language to what they already know and figuring out how to translate the things they do know to Ruby.

For a new developer altogether, they would get little value of writing a markdown processor, and in fact, probably seems more like a research project than a sure-fire way to learn a developer-friendly programming language.

So now we’re back to suggesting Rails, but then you consider everything else a newcomer would need familiarity with to traverse the Rails eco-stytem: HTML, CSS, Javascript, Coffeescript, Sass, SQL…

The list goes on and on. Not so easy after all. No wonder people get intimidated and bail. I suppose there’s starting with Sinatra, but the doesn’t remove you from the HTML and CSS requirements. Perhaps those impossible to dodge, given the medium. And even with Sinatra, we often end up creating the functionality that’s in Rails anyway.

I wish I had a better answer.

How would you suggest someone start with Ruby?

Decorators give us a tool to add additional functionality to a class while still keeping the public API consistent. From the perspective of the client, this is a win-win! Not only do they get the added behavior, but they don’t need to call different methods to do so.

The Problem

Our original class accessed data from Stripe AND cached the response for some time period. I accentuated “AND” because it’s generally the word to be on alert for when considering whether functionality can be teased apart in to separate responsibilities.

The question becomes, can we make one class that accesses Stripe data, and another that’s only responsible for caching it?

Of course we can!

The Solution

Let’s start with the most basic form of accessing our Stripe customer data with the Stripe gem:

class AccountsController < ApplicationController
  before_action :require_authentication

  def show
    @customer = Stripe::Customer.retrieve(current_user.stripe_id)
    @invoices = @customer.invoices
    @upcoming_invoice = @customer.upcoming_invoice
  end
end

Extract an Adapter

Because we’re interfacing with a third-party system (Stripe), it makes sense for to create a local adapter to access the Stripe methods. It’s probably not likely we’re going to switch out the official Stripe gem for another one that access the same data, but a better argument might be that we could switch billing systems entirely in the future. And if we make a more generic adapter to our third-party billing system, we would only need to update our adapter when that time comes.

While the adapter optimization may seem like overkill here, we’ll see how that generic adapter helps us implement our caching layer shortly.

Let’s start by removing the notion that it’s Stripe and all and call it Billing. Here we can expose the methods needed from the AccountsController above:

class Billing
  attr_reader :billing_id

  def initialize(billing_id)
    @billing_id = billing_id
  end

  def customer
    Stripe::Customer.retrieve(billing_id)
  end

  def invoices
    customer.invoices
  end

  def upcoming_invoice
    customer.upcoming_invoice
  end
end

There we have it. A simple Billing class that wraps the methods that we used in the first place – no change in functionality. But certainly more organized and isolated.

Let’s now use this new class in the accounts controller from earlier:

class AccountsController < ApplicationController
  before_action :require_authentication

  def show
    billing = Billing.new(current_user.stripe_id)

    @customer = billing.customer
    @invoices = billing.invoices
    @upcoming_invoice = billing.upcoming_invoice
  end
end

Not too bad! At this point we’ve provide the exact same functionality we had before, but we have a class that sits in the middle between the controller and Stripe gem - an adapter if you will.

Create a Decorator

Now that we have our adapter set up, let’s look at how we can add caching behavior to improve the performance of our accounts page.

The most of basic form of a decorator is to pass in the object we’re decorating (Billing), and define the same methods of the billing, but add the additional functionality on top of them.

Let’s create a base form of BillingWithCache that does nothing more than call the host methods:

class BillingWithCache
  def initialize(billing_service)
    @billing_service = billing_service
  end

  def customer
    billing_service.customer
  end

  def invoices
    customer.invoices
  end

  def upcoming_invoice
    customer.upcoming_invoice
  end

  private

  attr_reader :billing_service
end

So while we haven’t added any additional functionality, we have created the ability for this class to be used in place of our existing Billing class because it responds to the same API (#customer, #invoices, #upcoming_invoice).

Integrating this new class with AccountsController looks like:

class AccountsController < ApplicationController
  before_action :require_authentication

  def show
    billing = BillingWithCache.new(Billing.new(current_user.stripe_id))

    @customer = billing.customer
    @invoices = billing.invoices
    @upcoming_invoice = billing.upcoming_invoice
  end
end

As you can see, we only had to change one line – the line where we decorated the original billing class:

BillingWithCache.new(Billing.new(current_user.stripe_id))

I know what you’re thinking, “But it doesn’t actually cache anything!”. You’re right! Let’s dig in to the BillingWithCache class and add that.

Adding Caching Functionality

In order to cache data using Rails.cache, we’re going to need a cache key of some kind. Fortunately, the original Billing class provides a reader for billing_id that will allow us to make this unique to that user.

def cache_key(item)
  "user/#{billing_id}/billing/#{item}"
end

In this case, item can refer to things like "customer", "invoices", or "upcoming_invoice". This gives us a method we can use internally with BillingWithCache to provide a cache key unique to the both the user and the type of data we’re caching.

Adding in the calls to actually cache the data:

class BillingWithCache
  def initialize(billing_service)
    @billing_service = billing_service
  end

  def customer
    key = cache_key("customer")

    Rails.cache.fetch(key, expires: 15.minutes) do
      billing_service.customer
    end
  end

  def invoices
    key = cache_key("invoices")

    Rails.cache.fetch(key, expires: 15.minutes) do
      customer.invoices
    end
  end

  def upcoming_invoice
    key = cache_key("upcoming_invoice")

    Rails.cache.fetch(key, expires: 15.minutes) do
      customer.upcoming_invoice
    end
  end

  private

  attr_reader :billing_service

  def cache_key(item)
    "user/#{billing_service.billing_id}/billing/#{item}"
  end
end

The code above caches the call to each of these methods for 15 minutes. We could go further and move that to an argument with a default value, but I’ll leave as an exercise for another time.

Summary

Separating your application and third-party services helps keeps your applications flexible – offering the freedom to switch to another service when one no longer fits the bill.

Another benefit of an adapter is you have the freedom to name the class and methods whatever you like. The base gem for a service might not have the best names, or it may be that the names don’t make sense when dragged in to your application’s domain. This is a small but important point as applications get larger and its code more complex. The more variable/method names you need to think about when you poke around the code, the harder it’ll be to remember what was going on. Not to mention the pain new developers will have if they acquire the code. Whether it’s you or the next developer, the time you invest in creating great names will be greatly appreciated.

Using decorators in this way makes it easier for clients of the code to avoid change, but keep your applications flexible. The Billing class above was relatively simple – intentionally so. If the class being decorated has more than a few methods, it might be worth incorporating SimpleDelegator to ensure the methods that don’t need additional functionality still continue to respond appropriately.

While Stripe does provide a number of features and plugins that make updating a credit card or signing up for a service simple, there are occasions when data needs to be fetched from Stripe in real-time. For these cases, it’s great to be able to fetch and cache this data before-hand, and only expire if you know there’s been a change.

Combining Sucker Punch with Rails cache allows you to cache Stripe customer data so that billing pages are just as snappy as the rest of the application.

The Pain

Even though Stripe is generally pretty fast, retrieving customer data on the fly can be expensive. In order to optimize page load times, we can look to cache this data before it’s actually used.

If you’re familiary with the Stripe gem, you’ve probably seen something like this:

customer = Stripe::Customer.retrieve(user.stripe_id)

With the response of customer, we can further query customer data with the following methods:

invoices = customer.invoices
upcoming_invoices = customer.upcoming_invoices

If we make all 3 of these method calls on page load, we’d have 3 separate lookups from Stripe. This is pretty common for the typical billing page where you might want to show the customer’s current credit card on file, their past invoices, and charges they can expect for the next invoice.

Three lookups like this could potentially add another second or so to page load, which is not ideal.

So how can we improve this?

The Solution

First, we can move the code to fetch the relevant stripe data in to a class of it’s own, which wraps the notion of caching around the data retrieval.

class StripeCache
  def initialize(user)
    @user = user
  end

  def refresh
    purge_all
    cache_all
    self
  end

  def customer
    return @customer if @customer

    @customer = Rails.cache.fetch(cache_key("customer"), expires_in: 15.minutes) do
      Stripe::Customer.retrieve(user.stripe_id)
    end
  end

  def invoices
    Rails.cache.fetch(cache_key("invoices"), expires_in: 15.minutes) do
      customer.invoices
    end
  end

  def upcoming_invoice
    Rails.cache.fetch(cache_key("upcoming_invoice"), expires_in: 15.minutes) do
      customer.upcoming_invoice
    end
  end

  private

  attr_reader :user

  def cache_all
    customer
    invoices
    upcoming_invoice
  end

  def purge_all
    Rails.cache.delete_matched("#{user.id}/stripe")
  end


  def cache_key(item)
    "user/#{user.id}/stripe/#{item}"
  end
end

To use this on a billing page, we could do:

stripe = StripeCache.new(current_user).refresh

And from the response of that class, we could access the customer, invoices, and upcoming_invoice respectively:

@customer = stripe.customer
@invoices = stripe.invoices
@upcoming_invoice = stripe.invoices

This is great! All future calls to this customer’s Stripe data will be fast – for 15 minutes, of course.

However, the first time the page is load, the user is still burdened with the initial fetch of the data. So the method above works for every request to the billing page after the first.

But let’s be honest, what users are going to the billing page multiple times during a session? Probably not many. So we still need to fix the initial load somehow.

This is where Sucker Punch comes in. Like other Ruby background processing libraries, Sucker Punch allows you to move the processing of code to the background. However, unlike the others, Sucker Punch doesn’t require additional infrastructure like Redis, and doesn’t require a separate worker process to monitor and execute enqueued jobs. Because of this, the time it takes to extract code to a Sucker Punch job and have it incorporated with your application code is much lower.

In this case, rather than send a transactional email or perform some database calculation, we can write a job thats only responsibility is to run the Stripe caching code.

class StripeCacheJob
  include SuckerPunch::Job

  def perform(user)
    StripeCache.new(user).refresh
  end
end

The next question is, when do you run this?

Well, I chose to run it on user login, but you could run it anywhere you think would give you a head start if the user were about to go to the billing page. In my case, on login meant that if they didn’t go to the billing page at all, after 15 minutes the data would be exhausted from the cache anyway, so no hard done.

But if the user did navigate to the billing page during that session, they would have up the latest Stripe customer and invoice data to see – all without a request to stripe on page load.

One other thing to keep in mind is there may be times when we’d want invalidate the Rails cache data. One example would be when the user’s card information is updated. In that case, we can slip in another call to the Stripe cache job, which would invalidate the previous cache and re-request the customer’s billing information:

module Accounts
  class CardsController < ApplicationController
    before_action :require_authentication

    def create
      cust = StripeCache.new(current_user).customer
      cust.save(card: params[:stripeToken])

      StripeCacheJob.new.async.perform(current_user)

      redirect_to account_path, notice: t("card.update.success")
    end
  end
end

Summary

Using Sucker Punch in combination with Rails cache feels like a great way make optimizations to third-party data requests. This article focused on using it to fetch Stripe data, but it could be used with another service just as easily.

The beauty of Sucker Punch is that it doesn’t require a separate worker process to be running in the background. On a platform like Heroku, this saves the cost of an additional dyno.

Sucker Punch excels at background jobs that are relatively fast and if missed, wouldn’t be critical to the operation. In this case, if a cache job is lost, it’s not the end of the world. At worst, the user’s Stripe data would be requested on the fly and the page would be slower than usual. But the majority of the time, the request is fast because the data’s been cached beforehand.

What other jobs have you used Sucker Punch for?

Most applications have ancillary data that’s required to support the main function of the application — think drop-downs with states for shipping or credit card type.

This data is almost always never interesting, but completely necessary for the application to work as expected. So when it comes to time send your little baby to production, only to find your users can’t pay because they can’t pick their credit card type, your world comes crashing down.

If you have those credit card types in fixtures from the start, loading them in to your development of production database is just a rake task away.

The Problem

Let’s assume our application requires us have a list of supported credit card types, and the user is required to pick from the list to pay for the awesome stuff we sell. A sample fixture might look like:

visa:
  name: Visa

mastercard:
  name: Mastercard

amex:
  name: American Express

This is a somewhat trivial example because the name matches what one might expect in a potential transaction record if we had a credit_card_type field or something similar if we denormalized.

Perhaps we have a field credit_card_type_id in a transactions table that references the foreign key of the related CreditCardType record.

So how do we get these records in to our development and production databases?

The Solution

Fortunately, Rails has our backs. The following rake test is available from a default Rails application:

$ bin/rake -T
...
rake db:fixtures:load # Load fixtures into the current environment's database

The db:fixtures:load task is an interesting start, but quickly we realize it might be a little heavy-handed. If this application has users, we probably wouldn’t want to copy them to production. They might, however, be a great starting pointing for development.

So how do we handle getting trivial model data in to production for only specific models?

It turns out that we can specify ONLY the models we want to load by using the FIXTURES environment variable:

rake db:fixtures:load FIXTURES=credit_card_types

Note: The name of the fixture file (usually the database table name) should be used as the value for FIXTURES, not the model name.

With that single command, any environment we specify will immediately get the data for our 3 credit card types.

A word of warning, if we run this command multiple times, it will seed the table multiple times. It’s not idempotent.

Additionally, if we wanted to load more than just a single fixture, we can specify the names of the files separated by commas:

rake db:fixtures:load FIXTURES=credit_card_types,states,cities

Let’s take a quick look at how Rails implements this functionality, specifically the determination of single models:

fixtures_dir = if ENV['FIXTURES_DIR']
                 File.join base_dir, ENV['FIXTURES_DIR']
               else
                 base_dir
               end

fixture_files = if ENV['FIXTURES']
                  ENV['FIXTURES'].split(',')
                else
                  # The use of String#[] here is to support namespaced fixtures
                  Dir["#{fixtures_dir}/**/*.yml"].map {|f| f[(fixtures_dir.size + 1)..-5] }
                end

ActiveRecord::FixtureSet.create_fixtures(fixtures_dir, fixture_files)

If the FIXTURES variable is present, code teases appart the model names and looks in the fixtures directory and loads the YAML fixture file for that table name.

An interesting side note, it’s possible to specify alternate directories for fixture using the FIXTURES_DIR variable. I personally haven taken advantage of this, but could be useful if you want to keep specific fixture files for production that might be different than those that reside in test/fixtures/*.

I wouldn’t suggesting using this approach for anything that needs to reference other foreign keys. When you’re transferring to a different database, foreign keys will not match and your application will likely not work as expected.

Summary

This approach has saved me quite a bit of time in my last few applications. Build it once, use it everywhere. As mentioned above, using this approach to seed database records with a foreign key should be avoided.

Most applications have a number of tasks needed for a developer to get up and running. Combining fixture data with additional seed data placed in db/seeds.rb can give you the best of both worlds, while still ensuring you have robust data to test against.

Mine have been more of the bucket list variety. The first was completing an Ironman. The second, writing a book.

Both felt almost too big initially, but ultimately led to opportunities and lifestyle changes I would’ve never expected once finished. So naturally, with 2014 winding down, the question becomes “what’s the goal for 2015?”.

And the answer is…I don’t know.

It doesn’t mean I’m not going to do anything. In fact, it probably means the opposite. I’m just not going to set out with a specific goal in mind. If halfway through something, I want to stop and do something else, so be it.

I remember during my training for the Ironman (9 months total), I constantly thought about what I would do with my free time when it was over. Every time my alarm went off at 5am, I thought about what it would feel like to get another 2 hours of sleep. It was endless. Nine months was a long time to have those thoughts and, perhaps, why I was so well positioned to write and launch the book (I had 9 months to think about what was next and how to accomplish it).

Always having your sights set on the future can wear on you though.

Without those feelings now, I’m going to let 2015 take me wherever it does. I don’t have any expectations financially or professionally. I’m going to do my best to make the most out of every moment and appreciate more of the small things. It’s so easy to skip over the small things and in many cases, the small things are actually the best things. And we don’t realize it until they’re gone.

2014 was a great year in all. Here are a few of the events that stand out the most:

• my wife and I welcomed our son, Cruz
• my wife and I celebrated our 4 year anniversary
• wrote a book, Build a Ruby Gem
• provided a free email course on building a Ruby gem to 1,218 people
• connected with 2,645 people through my newsletter
• built a funnel optimization service for bloggers selling products
• published 24 articles
• saw Sucker Punch downloaded over 225k times
• celebrated Sucker Punch being integrated in to Rails
• made my first commit to Rails (even if it was small!)
• traveled to Jackson Hole, WY
• built a pay-to-view video platform with a few friends
• built a performance management system
• renovated my kids’ bathroom
• built a deployment system for PipelineDeals and supporting services that I’m very proud of
• built a staging server management application
• built 3 chrome extensions, the latter being my first React app
• started a weekly tech gathering at a local co-working space
• stopped push email notifications on all my devices (strongly recommended)
• saw Defriend Notifier be used by 30,128 people
• started exploring other programming languages, specifically Go and Elixir

I’m excited for 2015!