The standard setup for sending an email is like the following:

 data = Multimap.new
 data[:from] = "My Self <no-reply@example.com>"
 data[:subject] = "Subject"
 data[:to] = "#{recipients.join(', ')}"
 data[:text] = text_version_of_your_email
 data[:html] = html_version_of_your_email
 data[:attachment] = File.new(File.join("files", "attachment.txt"))
 RestClient.post "https://api:#{API_KEY}"\
   "@api.mailgun.net/v2/#{MAILGUN_DOMAIN}/messages",
   data

As this shows, the attachment is expected to be a File that can be read in. So the challenges was to make an S3 object readable. One option, of course, is to do this in two steps: read in the S3 object and use Tempfile to write a file which can then be used as the attachment. This seemed pretty unfortunate. For one thing, I’m running stuff on Heroku, and try to avoid using the file system even for temp files. But primarily, it’s really wasteful to have to write, and then read, a transitory file. The better option, of course, was to see if there was a way to trick the client into reading from S3.

Thanks to some very nice help from Mailgun support (thanks Sasha!), the idea of writing a wrapper seemed feasible, and in fact it wasn’t too bad aside from a couple of tricky issues. A side-effect advantage was that it solved another problem: the naming of the attachment. By default, the name of the attachment is the name of the file, which is pretty ugly if you use a temp file. Not user-friendly.

I’ve put the wrapper file in a Github Gist at https://gist.github.com/masonoise/5624266. It’s pretty short, and there were only a couple of gotchas, which I describe below. The key for this wrapper is to provide the methods that RestClient needs: #read, #path, #original_filename, and #content_type. It’s pretty obvious what #read and #path are for. The attachment naming problem is solved by #original_filename: whatever it returns will be the name of the attachment in the email. It should be clear what #content_type does, but see below for why it’s important.

Using the wrapper is described in the header comment, but it’s mainly a change to give RestClient the wrapper instead of a File object:

data[:attachment] = MailgunS3Attachment.new(file_name, file_key)

The first gotcha was that RestClient calls #read repeatedly for 8124 bytes, and doesn’t pass a block. This forced me to write a crude sort of buffering — the wrapper reads the whole S3 object in, then hands out chunks of it when asked. This isn’t a problem for me because the files I’m dealing with aren’t very large, but it’s something I warn about in the comments. If you have large files, this may be a problem for you, so beware.

The second gotcha that threw me off for a little while is that the value returned by #content_type is important. I haven’t researched exactly why this is, but I found that if I tried to send a Word document but #content_type returns ‘text/plain’, the attachment comes through corrupted. It was easy enough for me to check the filename suffix and set the content type accordingly, but I can imagine cases where this might not work, so this is something else to beware of.

Anyway, this solved the issue for me, and hopefully it’ll be useful for others. There are ways to make this a bit more elegant, but it’s a short piece of code that works. Enjoy.

    $(document).pjax('a[data-pjax]', '#content-panel');
    $(document).on('pjax:complete', function() {
        angular.bootstrap($("#content-panel")[0], ["my-app"]);
    })

That waits for the pjax request to complete, then boots the AngularJS app, and everything is good again.

So, what I wanted to was really easy with plain old jQuery, but it turned out to be more complicated when AngularJS entered the picture. Inside a list of items, for each item I had a DIV into which I was rendering a line graph, using Flotr2, and I had to pass it the JSON data needed for the charting. Before converting my page to AngularJS, I simply iterated my list in Ruby and called item.json_chart_data for each one, and called Flotr2. Not any more; now I’m using ng-repeat on a collection of items in my controller scope. What to do? I could in theory stuff the chart data into my items when my API returns them to my controller, but that was overloading the items themselves. So instead, I created a directive that loads the chart data for each item via an AJAX call.

Here’s the relevant markup from the page:

    <div class="item-graph" id="item-graph-{{item.id}}" style="width: 100px; height: 50px; padding-right: 20px">

        <item-graph item="{{item.id}}"></item-graph>

    </div>

This defines the DIV that Flotr2 is going to draw into, with the width and height, and then inside it is my directive, “item-graph”. I pass the item id into the directive, so that it can make the AJAX call to the server to get the graph data. Now let’s look at the directive:

var app = angular.module('my-app', ['restangular', 'ui.bootstrap']).
  config(function(RestangularProvider) {
    RestangularProvider.setBaseUrl('/api/v1');
})
.controller('MyListCtrl', function($scope, Items) {
  $scope.items = Items;
})
.directive('itemGraph', function($http) {
  return {
    restrict: 'E',
    scope:{
        itemId:'@item'
    },
    link: function(scope, element, attrs) {
      scope.$watch('itemId', function(value) {
          if (value) {
            $http.jsonp("/items/" + value + "/item_graph_data.json?callback=JSON_CALLBACK").then(function(response){
              draw_line_graph(document.getElementById('item-graph-' + value), response.data);
            });
          }
        });
    }
  }
});

This is all of the code for the page; I’ll gloss over the top part since that’s pretty standard AngularJS stuff. It defines the app, injecting the dependencies. I use Restangular for most of the server interaction; it does a really nice job of encapsulating things in a clear, RESTful way. The code configures Restangular with the base URL, since my server-side routes are in the ‘/api/v1′ space. Then we define the controller, and use the Items service (see below) to fetch the items. Then we get into the more interesting part: the directive.

First, note the name of the directive: “itemGraph”. But in the page markup, the tag is “item-graph” — it’s an irritating inconsistency, but just remember that directive names are “camel cased” while the name in your markup is “snake cased”. Thus “item-graph” in the page matches up with “itemGraph” in the code. Whatever. So then in the declaration we inject in the $http service so it can be used later.

Directives basically return an object with a couple of important sections. The first is random configuration, of which the restrict is very important, and this caused me more than a few minutes of debugging. I seemed to have everything wired up, but nothing was happening. That’s because restrict defaults to ‘A’, which specifies that the directive is restricted to attributes. I needed ‘E’ for element. This is described in the AngularJS page on directives (here), but it’s a long, long way down the page buried in details about compiling and the “directive definition object”. Major documentation fail, yes. In any case, don’t make my mistake; as soon as I added this, it started invoking my code.

The second piece needed is the scope, which you can think of as what ties the attributes in your page to your directive. In this case, remember that in my page I specified item="{{item.id}}" in order to pass in the item id. In this scope block we specify itemId:'@item', which says that we want a binding between the local property ‘itemId’ and the attribute ‘item’. The ‘@’ indicates a one-way data binding. I highly recommend reading this great blog post that describes the use of attributes, bindings, and functions. You’ll be glad you did.

Whew, okay, so now we have the directive attached to our element, and we have the attribute bound to a local property so we can use it. How do we use it? Well, it’s a little complicated, but not too bad. First: think of the link() function as what’s invoked when the directive is “activated” for an element. So in here, we want to take the item id that was passed in, and do something. However, it turns out that you don’t simply use itemId directly, or (as a normal person might assume) go with attrs.itemId; instead, we have to use our scope, and “watch” the property. In fairness, this is because often what you’re doing is binding your directive to something in the page that can change, such as a form field. Then by watching it, your directive can respond to changes in that attribute. In my case that wasn’t needed, but I still have to play by the rules. So okay, call scope.$watch() on ‘itemId’, and then write a function to deal with it. Apparently when first loaded, the watch function is called, so then we make sure there is a value (perhaps the directive gets loaded before the DOM is ready? I dunno). If there is, let’s finally do something.

What I wanted to do in my case is make a quick call back to the server, specifying the item id — which, remember, is now represented by “value” since that’s the parameter name on the watch function. When it returns, it calls draw_line_graph(), another function that sets up Flotr2 and does the drawing. It draws the graph into the DOM element passed in, with the data also passed in.

And that’s it. Seems like a lot of code to do what could be done in a couple of lines before, to be honest, but it’s packaged and reusable, and easily copied to do more complex things. One last thing; as promised I wanted to include the Items service which is used to get the initial list of items for the page, just in case someone finds it useful. It’s in another small JS file:

var app = angular.module('my-app');

app.factory('Items', function($http, Restangular) {
  var baseItems = Restangular.all('items');
  return baseItems.getList();
});

That’s all there is to it, using Restangular to get the list. This automagically ends up invoking the server URL ‘/api/v1/items’ and returns the JSON response. Restangular is even nicer when you start getting into nested relationships and other more complex needs.

I’ve made some of this generic (“my-app” and “Items”), since I can’t get show full details of the app I’m working on, but hopefully it will also make it easier for this to be used by others. I hope this saves others who are new to AngularJS some of the pain I had in figuring all of this out.

I quickly reached for the Tire gem, which is very similar to Sunspot if you’re using ActiveRecord. Where Sunspot has you include a “searchable” block, Tire adds a “mapping” block, but the idea is the same — that’s where you tell it what fields to index, and how to do it. For each field you can adjust the data type, boost, and more. You can also tack on a “settings” block to adjust things like the analyzers.

The documentation for Tire is pretty good, but I found that I made a number of mistakes trying to adapt the instructions on the Elasticsearch site to the Tire way of doing things, so I thought I’d write up some of the things I learned in hopes that it can help save time for others. Many thanks to the folks on StackOverflow who answered my questions and pointed me in the right direction.

One starter suggestion is to configure Tire’s debugger, which is really convenient because it will output the request being sent to the ES server as a curl command that you can copy and paste into a terminal for testing. Very handy. I added this to my config/environments/development.rb file:

  Tire.configure do
    logger STDERR, :level => 'debug'
  end

Now on to the model. I’ll call mine Publication, so inside app/models/publication.rb:

class Publication < ActiveRecord::Base
  include Tire::Model::Search
  include Tire::Model::Callbacks

  attr_accessible :title, :isbn, :authors, :abstract, :pub_date

  settings :analysis => {
    :filter  => {
      :ngram_filter => {
        :type => "nGram",
        :min_gram => 2,
        :max_gram => 12
      }
    },
    :analyzer => {
      :index_ngram_analyzer => {
        :type  => "custom",
        :tokenizer  => "standard",
        :filter  => ["lowercase", "ngram_filter"]
      },
      :search_ngram_analyzer => {
        :type  => "custom",
        :tokenizer  => "standard",
        :filter  => ["standard", "lowercase", "ngram_filter"]
      }
    }
  } do
    mapping :_source => { :excludes => ['attachment'] } do
      indexes :id, :type => 'integer'
      indexes :isbn
      [:title, :abstract].each do |attribute|
        indexes attribute, :type => 'string', :index_analyzer => 'index_ngram_analyzer', :search_analyzer => 'search_ngram_analyzer'
      end
      indexes :authors
      indexes :pub_date, :type => 'date'
      indexes :attachment, :type => 'attachment'
    end
  end

  def to_indexed_json
    to_json(:methods => [:attachment])
  end

  def attachment
    if isbn.present?
      path_to_pdf = "/Users/foobar/Documents/docs/#{isbn}.pdf"
      Base64.encode64(open(path_to_pdf) { |pdf| pdf.read })
    end
  end
end

Okay, that’s a lot of code, so let’s look things over bit by bit. Naturally the includes at the top are needed to mix-in the Tire methods. There are two includes so that you can include the calls needed for searching without the callbacks if you don’t need them. The callbacks, though, are what make things work auto-magically when you persist ActiveRecord objects. With those, whenever you call save() on an AR model, the object will be indexed into ES for you.

Next up are two blocks of code: settings, and mapping. The settings block defines a filter, and two analyzers, one for indexing and one for searching. I can’t claim to be enough of an expert yet to fully explain the ramifications of the filter/analyzer options, so rather than risk confusion I’ll just note that this code is there to set up the nGram filter and connect it with two analyzers, index and search, which differ slightly in order to ensure that the standard filter is included for searching. You may want to play with the nGram’s min and max settings to get the matching behavior you want. Note that if you don’t need the nGram filter, you can remove the settings block and let the mapping block stand on its own, in which case the default settings will be used (but you’ll have to change the mapping entry for the :title and :abstract fields, as described below).

The mapping block is the more interesting one, as it defines the fields and indexing behavior. The first line took me some searching and StackOverflow questioning to figure out. The issue that by default, Elasticsearch will put all of the fields you index into its _source storage. Because I’m indexing large PDF documents, the result was that a huge Base64-encoded field was being stored. If I wanted to serve the PDFs out of Elasticsearch that might be okay, but that’s not the plan. The :excludes instruction prevents the attachment field from being stored.

Next are the fields themselves, and I won’t spend much time on these because the Tire documentation does a fine job of explaining these. The only interesting items are the :attachment field and the entry for :title and :abstract — that one specifies that for those fields the custom analyzers defined in the settings block should be used. For :attachment it gets a little bit tricky.

When the indexing is performed, the fields themselves are gathered up by calling the method to_indexed_json(). Normally that will just do a to_json() on your model and then collect the fields. But you can also override it, which we do here. You can see that we add in the method attachment(), which is defined below. So the other fields will be JSONized as normal, as well as the output of the attachment() method. The attachment() method itself uses the ISBN number to open the PDF file, which is read and Base64-encoded. The results of that encoding will be included with the other fields and sent to ES for indexing.

Performing the searching is almost too easy, but there was one bit that threw me off initially, which was getting highlighting to work. The search block in my controller looks like this:

results = Publication.search do
  query { string query_string }
  sort { by :pub_date, 'desc' }
  highlight :title,  ptions => { :tag => "<strong class='highlight'>" }
end

I was trying to test the highlighting and was thrown off by the field names being case-sensitive (see my question on StackOverflow), but this now works. The other key is that the highlighted fields are returned separately from the plain fields, which was odd to see. This means that to display the highlighting I have to check for the field:

results.each do |r|
  r_title = (r.highlight.nil? ? r.title : r.highlight.title[0])
  puts "Title: #{r_title}"
end

If the highlighting is present then it’s used; if not (because the term isn’t present in that field) then the regular field is used. The other handy thing to note is that you can specify the tag with which to wrap the term. The default is “<em>” but I wanted to specify the “highlight” CSS class, as is shown here. This is a really convenient feature.

That covers the basics, though it’s also probably worth sharing just how nice it is to be able to test using curl. For example, I wanted to check how easy it is to have the search call return just a single field (to speed up certain requests), so I tried it first in curl:

curl -XPOST http://localhost:9200/publications/_search\?pretty\=true -d '{
"query": {"query_string": {"query": "Foobar"}},
"fields": ["title"]
}'

That’s of course assuming that ES is running on your local system on port 9200; if not, adjust accordingly.

There you go. I hope this writeup is helpful to folks getting started and it saves you some time.

However, as-provided, Timeline-Setter wants to exist as a command-line utility. You get a data file ready, run the utility, and it generates a set of HTML (and associated) files that you can drop into an app. That’s dandy if you have a set of data that doesn’t change often, or you want to perhaps run a cron to pre-generate a bunch of timelines. We needed something that could generate a timeline for a dynamic set of data, so we weren’t sure Timeline-Setter would work for us. However, looking it over I thought it seemed potentially usable in a dynamic way. I generated a static example using our data, read through what it had created, and deconstructed what it wanted in order to display the timeline, then wrote some code to dynamically generate the JSON data necessary. It wasn’t too difficult, and fairly shortly we had dynamic timelines going. I wanted to share the info here since it’s a pretty nice library that others should get a lot out of.

We’re using Jammit to handle our static assets, so we simply put “public/javascripts/timeline-setter.js” and “public/stylesheets/timeline-setter.css” into our assets.yml file, but you can use whatever standard approach you take to including JS and CSS into your pages. Once that’s done, you’re ready to go.

Timeline-Setter takes a pretty standard approach to placing itself in the page: it takes the ID of a DIV, and that’s the container which will hold the timeline. One note: we needed to include multiple timelines in a single page, so we had to do a little creative naming of the DIVs that hold the timelines, as you’ll see below.

<div id="timeline-<%= author.id %>" class="timeline_holder"></div>
<pre>
<script type="text/javascript">// <![CDATA[
    $(function () {
      var currentTimeline = TimelineSetter.Timeline.boot([<%= author.make_timeline_json %>], {"container":"#timeline-<%= author.id %>", "interval":"", "colorReset":true});
    });

// ]]></script>

The code here creates the DIV, gives it an ID, and then places a Javascript call to the Timeline-Setter boot() function, which tells it to generate the timeline. The first parameter is the JSON holding the data for the timeline; the second is a set of options, passed in as JSON. “container” of course is the ID of the DIV which will contain the timeline. Other options include “interval”, “formatter”, and “colorReset” among others. See the library’s page as listed at the beginning for details of the API, and in particular see the section headed “Configuring the Timeline JavaScript Embed” for the basics of calling the boot() function.

Now of course we need the make_timeline_json() method, which will take our object’s data and create the JSON needed by Timeline-Setter. As an example here, let’s pretend that we’re showing a timeline of books written by an author over the years.

class Author < ActiveRecord::Base
  def make_timeline_json
    timeline_list = []
    if (!birth_date.nil?)
      timeline_list << "{'html':'','date':'#{birth_date}','description':'Birth Date:','link':'','display_date':'','series':'Events','timestamp':#{(birth_date.to_time.to_i * 1000)}}"
    end
    books.each do |book|
      author_list = book.authors.map { |auth| "#{auth['name']}" }.join('; ')
      timeline_list << "{'html':'Authors: #{author_list}
Publisher: #{book.publisher}','date':'#{book.pub_date}','description':'#{book.title}','link':'','display_date':'','series':'Publications','timestamp':#{(book.pub_date.to_time.to_i * 1000)}}"
    end
    if (!death_date.nil?)
      timeline_list << "{'html':'','date':'#{death_date}','description':'Death Date:','link':'','display_date':'','series':'Events','timestamp':#{(death_date.to_time.to_i * 1000)}}"
    end
    return "#{timeline_list.join(',')}"
  end
end

Essentially, this method creates a JSON string containing a set of entries, each representing an event to place on the timeline. Each entry has several fields: html, date, description, link, display_date, series, and timestamp. Not all of these are used here, but with the basics you can experiment further. The important fields are:

• html: This will be displayed in the event’s pop-up when clicked on.
• description: Just what it says.
• link: an optional URL which will be associated with a link in the event pop-up.
• series: which “series” this event belongs to; see below for details on this.
• timestamp: This is the timestamp associated with the event, used to construct the timeline in order.

A note about the “series” parameter: one very nice feature of Timeline-Setter is that you can display more than one set of events in a single timeline. Each set of events is called a “series”. In our example we’re creating two series: “Events” and “Publications”. Each will be shown in a different color, with a title (so the names of the series need to look nice, as they’ll be displayed) and a checkbox so that a viewer can hide and show each series individually. It’s extremely useful.

In the code above, you’ll see that we create the “Birth Date” and “Death Date” events individually, but in the middle we iterate over the books associated with this author. For each book we build a string of authors, semicolon-delimited, just to demonstrate one way to include another list of information in an event’s HTML. I have to admit that I’m not entirely certain why it’s necessary to multiply the timestamps by 1000 to get to the correct time, but it works fine…

And there you are. Hopefully anyone needing an economical, nice-looking timeline with dynamically-generated data can take advantage of this. But certainly, if you can work with static (or infrequently-updated) data, you may be able to use Timeline-Setter out of the box via a cron job or rake task — you could generate the CSV file for its command-line interface, run it, then copy the resulting files into your application. If you need dynamic timelines, though, I hope this post is helpful.

I wrote up an article about using the gem, which is now available on the DeveloperForce site. You can go and check it out at http://wiki.developerforce.com/page/Accessing_Salesforce_Data_From_Ruby. I hope it helps get your started if you’re finding a need to do this sort of work. So many companies rely on Salesforce now for at least their sales pipeline that it can be extremely useful to do things like extract data to show in an internal Rails dashboard, or do more complex reporting. In our case we’re also sourcing data from external places and pushing it into our Salesforce organization so that our sales/support folks can have easy access to it within their Salesforce world.

The perftools.rb gem is of course on GitHub: here. Because I needed to profile a rake task rather than part of our Rails app, I couldn’t use the rack-perftools_profiler, so I had to do it a bit more manually. First thing of course was to add the gem to the Gemfile:

    gem "perftools.rb", "~> 2.0.0"

After a bundle install, I was good to go there. Next I had to put the profiling around the block of code I wanted to investigate:

    require 'perftools'	# At the top of the class

    PerfTools::CpuProfiler.start("/tmp/my_method_profile")
    my_method_call()
    PerfTools::CpuProfiler.stop

It’s pretty easy. I ran my code, and then I had the file /tmp/my_method_profile. But what to do with this binary file? Well, the simplest option is to run pprof.rb on it:

pprof.rb --text /tmp/my_method_profile

That will output a bunch of information, basically a list of the “samples” the profiler took in descending order of percentage of samples in the function — that is, the first entry is the one which was “seen” the most by the profiler. I recommend taking a look at the pprof documentation here.

That textual information is surely useful, but I hoped for more. Using callgrind output seemed very interesting, but I couldn’t use kcachegrind on my Mac. Thanks to this gist I was able to get qcachegrind, a Qt version, running. It’s not quite 100% happy with OS X 10.7 but it works. You’ll need Xcode installed, then follow the directions in that gist. I installed the latest “libs only” version of Qt, 4.8.0. Installing Graphviz was straightforward; you can follow the instructions in the gist or use Homebrew as well. I was then able to do the svn checkout of qcachegrind and build it. Note that the patch gist mentioned to let it open any file is no longer necessary — since that was written, a change to qcachegrind was made to allow it to open any file. Go ahead and do the qmake and make, and you should be good to go.

Okay, so now I could process the profile into callgrind format, and view it:

pprof.rb --callgrind /tmp/my_method_profile > /tmp/my_method_profile.grind
open qcachegrind.app

Using qcachegrind I opened the /tmp/my_method_profile.grind file, and there it was! I could view the call graphs, and get some nice views of what was going on in the code.

If all you want is a straight-forward call graph, though, you can also (once you have Graphviz installed) generate a call graph GIF image with “pprof.rb –gif /tmp/my_method_profile > /tmp/my_method_profile.gif”. Open that GIF and you’ll have a very useful view of what was happening in the block of code you’re profiling.

I hope this quick summary is helpful — it’s not always easy to figure out precisely what’s happening your code, and perftools.rb can help out a great deal.

As the title indicates, the author focuses on using MongoDB to provide the storage back-end for PHP web applications. After a short introduction to the basic concepts behind MongoDB, we get a walk-through of installing MongoDB and getting PHP to talk with it, before starting in on building a blog. It’s a safe example, since a blog is a reasonable candidate for a document store like MongoDB. It also provides a way to address one of the big design questions when using MongoDB, which is when to use embedded documents and when to store references. To my mind the question is glossed over a bit too quickly, but it is discussed.

Additional projects like session management and geolocation get a bit off-track, as a lot of time is spent describing the concepts rather than MongoDB, but the meatier sections that get into topics like Map-Reduce (creating a tag cloud) and web analytics are certainly worthwhile. I did feel that the chapter reviewing two MongoDB management tools could have been skipped, since the information will likely be out of date within a couple of months.

Overall, this is a reasonable beginner’s guide, as its subtitle indicates. There’s a great deal of PHP code filling its pages, which will give you a starting point if you need a boost to get going. Reading through it will give you the basics about MongoDB, and a bit more — hints on indexing, optimizing, and Map-Reduce will keep you running. A lot of the information felt cursory, and I would have appreciated more depth, but that’s probably just me wanting more than a beginner’s introduction. Perhaps more relevant were my concerns about the copy-editing and grammar. I didn’t notice any actual errors, but the grammar is quite rough and it made me wonder. It may make me old-fashioned these days, but I still expect my books to be well-edited and grammatically correct. The issue didn’t get directly in the way of the information to be had, but it’s still a pity. Nonetheless, if you’re a PHP developer and you’re looking to get started with MongoDB, you’ll doubtless find this a useful book.

PHP and MongoDB Web Development from Packt Publishing, also available from Amazon.