Converting ActiveResource Http Requests to Equivalent CURL Commands

Published on:

Say if you want to debug a API call that your application makes to a external API. And also assume you are using rails and ActiveResource to make your API calls.

You will need to enable debugging in Net:HTTP to actually see the HTTP requests that are being made by ActiveResource. And then if you suspect that this is something wrong with the API, you’ll probably have to convert those requests in to curl commands so you can file a bug report with the guys who are responsible for the API.

Recently i had to do what i just described above. And i am documenting it down so i can reference it in the future. Making up curl commands fades from my memory so fast, i really need to document this stuff :)

Printing out the HTTP requests made by ActiveResource

ActiveResource uses Net::HTTP internally to make all the http requests. So what we can do is to monkey patch ActiveResource to enable debugging on Net::HTTP.

I choose to add a initializer that enables debugging.

[config/initializers/debug_connection.rb]
1
2
3
4
5
6
7
8
9
10
11
12
13
class ActiveResource::Connection
  # Creates new Net::HTTP instance for communication with
  # remote service and resources.
  def http
    http = Net::HTTP.new(@site.host, @site.port)
    http.use_ssl = @site.is_a?(URI::HTTPS)
    http.verify_mode = OpenSSL::SSL::VERIFY_NONE if http.use_ssl
    http.read_timeout = @timeout if @timeout
    # Here's the addition that allows you to see the output
    http.set_debug_output $stderr
    return http
  end
end

Ref: How do I view the HTTP response to an ActiveResource request?

After adding this you should start seeing HTTP requests in your logs whenever you make a API call

[log/developmentlog]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
opening connection to localhost...
opened
<- "PUT /v1/identities/4f9249b318065b63f0002b11.json HTTP/1.1\r\nAuthorization: Basic YXJ0aXJpeDpzbk9fIWxuYk84Ng==\r\nContent-Type: application/json\r\nAccept: */*\r\nUser-Agent: Ruby\r\nConnection: close\r\nHost: localhost:4000\r\nContent-Length: 506\r\n\r\n"
<- "{\"identity\":{\"categories\":[\"4f2ad5692418ae215b000066\",\"country:brunei\"],\"dateofbirth\":null,\"dateofdeath\":null,\"external_links\":{\"wikipedia\":\"\",\"wikipedia_cn\":\"\",\"official\":\"http://www.mincom.gov.bn/ \"},\"gender\":\"na\",\"id\":\"4f9249b318065b63f0002b11\",\"keywords\":{\"cn\":[],\"en\":[],\"ru\":[]},\"large_images\":[],\"name\":{\"cn\":\"\",\"en\":\"Ministry of Communications - Brunei Darussalam \",\"ru\":\"\"},\"person\":false,\"small_images\":[],\"text\":{\"cn\":null,\"en\":null,\"ru\":null},\"videos\":[],\"yearofbirth\":1984,\"yearofdeath\":null}}"
-> "HTTP/1.1 200 OK\r\n"
-> "Date: Tue, 08 May 2012 01:43:05 GMT\r\n"
-> "Server: Apache/2.2.14 (Ubuntu)\r\n"
-> "Content-Length: 1586\r\n"
-> "Connection: close\r\n"
-> "Content-Type: application/json\r\n"
-> "\r\n"
reading 1586 bytes...
-> "{\"identity\": {\"yearofdeath\": null, \"dateofdeath\": null, \"name\": {\"ru\": \"\", \"en\": \"Ministry of Communications - Brunei Darussalam \", \"cn\": \"\"}, \"videos\": [], \"person\": false, \"text\": {\"ru\": null, \"en\": null, \"cn\": null}, \"small_images\": [], \"yearofbirth\": 1984, \"metrics\": {\"starrank\": {\"d_t-7\": 224, \"d_t-1\": -15, \"last\": 72293, \"at\": \"2012-05-08T00:00:00\"}, \"starrank:ytd\": {\"d_t-7\": -112, \"d_t-1\": 186, \"last\": 69264, \"at\": \"2012-05-08T00:00:00\"}, \"chart:4f460b8118065b22fa0001a2\": {\"last\": 17, \"d_t-1\": 0, \"d_t-7\": 0}, \"starscore:t-7\": {\"d_t-7\": 2.9921256690808573e-06, \"d_t-1\": -3.8318675103982315e-07, \"last\": 3.923343477289096e-06, \"at\": \"2012-05-08T00:00:00\"}, \"starrank:t-1\": {\"d_t-7\": 1639, \"d_t-1\": 3512, \"last\": 62662, \"at\": \"2012-05-08T00:00:00\"}, \"starrank:t-7\": {\"d_t-7\": -6571, \"d_t-1\": 293, \"last\": 62086, \"at\": \"2012-05-08T00:00:00\"}, \"starscore:t-1\": {\"d_t-7\": 3.5127893557257633e-08, \"d_t-1\": -1.0839413653295542e-06, \"last\": 1.617713164713497e-06, \"at\": \"2012-05-08T00:00:00\"}, \"starscore:ytd\": {\"d_t-7\": 2.4220256669688935e-07, \"d_t-1\": -2.9300947545939653e-08, \"last\": 1.1578785211521211e-06, \"at\": \"2012-05-08T00:00:00\"}, \"starscore\": {\"d_t-7\": -1.4304586890077729e-07, \"d_t-1\": -6.7151807517585576e-09, \"last\": 4.1007696372749711e-06, \"at\": \"2012-05-08T00:00:00\"}"
-> "}, \"dateofbirth\": null, \"large_images\": [], \"external_links\": {\"wikipedia_cn\": \"\", \"wikipedia\": \"\", \"official\": \"http://www.mincom.gov.bn/ \"}, \"gender\": \"na\", \"keywords\": {\"ru\": [], \"en\": [], \"cn\": []}, \"id\": \"4f9249b318065b63f0002b11\", \"categories\": [\"4f2ad5692418ae215b000066\", \"country:brunei\"]}}"
read 1586 bytes
Conn close

Converting the HTTP requests to CURL commands

Now that you know what the HTTP request looks like and what exact params and values are passed to the API call you can construct the CURL command as following. Note that I am looking at a update action which uses the http method PUT to save a edit to a resource (its a RESTful API) in the server.

[CURL command]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
curl 'http://api.guys.com/vx/identities/4f9249b318065b63f0002b11.json' -X PUT -i --user username:password -H 'Content-Type: application/json' -d '{
  "identity":
  {
    "categories":["4f2ad5692418ae215b000066","country:brunei"],
    "dateofbirth":null,
    "dateofdeath":null,
    "external_links":
      {
        "wikipedia":"",
        "wikipedia_cn":"",
        "official":"http://www.mincom.gov.bn/ "
      },
    "gender":"na",
    "id":"4f9249b318065b63f0002b11",
    "keywords":{"cn":[],"en":[],"ru":[]},
    "large_images":[],
    "name":{
        "cn":"",
        "en":"Ministry of Communications - Brunei Darussalam ",
        "ru":""
      },
    "person":false,
    "small_images":[],
    "text":{"cn":null,"en":null,"ru":null},
    "videos":[],
    "yearofbirth":1984,
    "yearofdeath":null
  }
}'

Note that we are sending the input as json. When you have a object with a lot of properties and values this is the way to send things via curl. I am sending a Identity object with all fields and values set.

Optimizing MongoDB (When Used With Mongomapper and Ruby on Rails)

Published on:
Tags:

So I’ve been working on a data entry web application that deals with a huge amount of data. And all of the data is stored in mongodb (we also consume a external API that have a ton of data but I feel that this is not relevant to this discussion). The setup is a bit complicated but the requirement is to improve performance of the DB server. Why? Because its performance is affecting the web application’s performance which means the users are complaining about 502 Bad Gateway Errors!

We’ve tried throwing more CPU/Memory at it but we didn’t see any significant improvements. So the quest for archiving ultimate optimization began! :)

Background

The web server is a running a Ubuntu/Rails3/Nginx/Unicorn/MongoMapper stack. I believe the 502 errors are caused by idle unicorn workers, been killed by unicorn after a period of inactivity. After looking at the logs it seems that some database queries takes a long time to complete and thus end up holding the unicorn workers for far too long resulting in them getting killed. Now I could ‘ask’ unicorn to give more time to idle workers before killing them. This would reduce the number of 502 errors. However thats not the real problem. So I tried looking in to how I can improve the mongodb performance.

Logging

In order for us to find out whats slow, so we can improve it we need to enable some logging.

MongoMapper

If you are using mongomapper you can enable it’s logging functionality like this (note that I only enable logging in the development env. You might want to enable it on production if you are debugging something. But generally you should not as it can affect performance):

#
#config/mongo.rb
#
# enable logging only in development/test environments
if Rails.env.production?
  MongoMapper.connection = Mongo::Connection.new('127.0.0.1', 27017)
else
  mongo_logger = Logger.new('log/mongo_mapper.log')
  MongoMapper.connection = Mongo::Connection.new('127.0.0.1', 27017, :logger => mongo_logger)
end

if defined?(PhusionPassenger)
   PhusionPassenger.on_event(:starting_worker_process) do |forked|
     MongoMapper.connection.connect if forked
   end
end

Nginx

Keep a watch on the nginx error logs for any 502 Gateway Timeout errors. These could possibly indicate db queries that take too long to complete (hence resulting in requests that take too long to complete and then is eventually axed by Unicon master).

You can enable logging in several other places. The mongo documentation describes this clearly.

Profiling

Enabling profiling

You need to enable profile first. Be sure to turn it of once you are done performance tuning for obvious reasons.

From mongodb documentation

Profiling levels are:

0 - off 1 - log slow operations (by default, >100ms is considered slow) 2 - log all operations

So to turn profiling on to log slow (>20ms) operations:

db.setProfilingLevel(1,20)

You can adjust 20ms to a value that suits you.

Spotting slow queries

The following query would get any slow (took more than 5 mili seconds to complete) queries, sort them by the time stamp and pick the latest one.

db.system.profile.find( { millis : { $gt : 5 } } ).sort({ts:-1}).limit(1).pretty()

Looking at whats happening in real time

db.currentOp();

I use this to get a little insight in to whats happening on the db server. If you tend to see a specific query again and again, you might be looking at something that can be optimized.

Read Database Profiler for more information on mongodb profiling.

Indexing

If you are going to only one thing about your database’s performance this should be it. Make sure you have a proper set of indexes covering all your common, complex and time taking queries.

You can read all about indexing in the following articles:

There are lots of information on those article about how to best use indexes. In my case what I learned was that:

  • Its usually a good idea to use “Index per Query”. At least in cases where the query is quite complex.
  • Always use “explain()” to see how your queries are executed. You will catch potential problems earlier this way.
  • Make sure your multi key indexes have the keys in the right order.
  • Add the keys that you are sorting on, as the last field to your index.
  • If you using queries with operators such as $ne (not equals) and then sorting the results, you’ll need to add the keys that you are sorting on as the first keys for the index. If not it will results in “scanAndOrder:true”. Which basically means that your queries will be much slower. Refer this article (linked to google cache)
  • MongoMapper do not support passing ‘hints’ to mongodb queries. Hints allows you to tell mongodb to use a specific index when executing a query. There is a issue for this feature in Github. But I do not see it been implemented anytime soon. And… Why do we need this feature? Well sometimes mongodb just don’t use the index that we want it to. So if you need this feature you might want to look at mongoid instead.

References

Removing ActiveRecord From Your Rails App

Published on:

Recently i switched one of the rails apps that i was working on to mongo from mysql. I got mongomapper running smooth and everything was working fine. But ActiveRecord was still there and i wanted to get rid of it. Please note that this would only apply to rails 3 and up.

Its pretty straight forward:

Remove this line (most probably the first line in the file)…

[config/application.rb]
1
   require 'rails/all'

and replace it with:

[config/application.rb]
1
2
3
4
   require "action_controller/railtie"
  require "action_mailer/railtie"
  require "active_resource/railtie"
  require "rails/test_unit/railtie"

You are basically telling rails what modules (yes, rails is now much more modular) to load. So if you are not using test_unit for example, you would omit the last line as well. We are not loading active_record now (which would have been loaded by rails/all.

If you are creating a new rails app and want to use a different framework you don’t need to get your hands dirty with all this. Just tell the rails generator what ORM you want to use :)

Rails Assets Pipeline Survival Guide

Published on:

Helpful settings

If you have assets pipeline enabled. By default you need to precompile your assets every time you change something (in production env). Having everything precompiled is cool but this will ask rails compile any assets that are not already compiled.

[production.rb]
1
2
   # Don't fallback to assets pipeline if a precompiled asset is missed
   config.assets.compile = true

Restricting JS/CSS to specific pages

If you want some JS/CSS to only be included in a specific set of pages. break the page specific JS/CSS code to different files (ex:- profiles.js) make sure they are not referenced in your manifest file (application.js) add the following code snippets to the necessary files

[application.html.erb]
1
2
3
4
5
6
7
  <head>
   <title>FamousApp</title>
   <%= stylesheet_link_tag "application" %>
   <%= javascript_include_tag "application" %>
   <%= yield :head %>
   <%= csrf_meta_tags %>
  </head>

specific_page.html.erb (the template where you want your JS to appear)

[specific_page.html.erb]
1
2
3
  <% content_for :head do %>
   <%= javascript_include_tag "profiles" %>
  <% end %>

production.rb

[production.rb]
1
2
   # Precompile additional assets (application.js, application.css, and all non-JS/CSS are already added)
   config.assets.precompile += %w( profiles.js )

Deploying with Capistrano

This is easy enough

[deploy.rb]
1
   load 'deploy/assets'

If you ever run across this problem:

[deploy.rb]
1
2
3
   # Disable warning message about missing dirs 'public/[javascripts|images|stylesheets]'
  # src: https://github.com/capistrano/capistrano/issues/79
  set :normalize_asset_timestamps, false

Note: Test and View Available Rails Routes

Published on:
Tags:
See all available routes
rake routes
Check route/path/URL helpers (in rails console)
app.the_route_you_want_to_test_path
OR
app.the_route_you_want_to_test_url

SQL ‘Group by’ + ‘Order by’ in Mongodb

Published on:
Tags:
As a newbie to mongodb i was a bit puzzled when i couldn’t find a equivalent to the SQL group by combined with order by in mongodb.

Group

If you just want to go a SQL group by in mongodb you can use the group() function (refer http://www.mongodb.org/display/DOCS/Aggregation#Aggregation-Group). however the results you get from group function cannot be sorted using the sort() function. This is because group function does not return a cursor (on which you can sort(), limit()…etc), but a single document. You can’t sort the document wich is returned by group but you can sort the document in the client side. This is one way to go about doing a ‘group by + order by’. However if you don’t want to sort things in the client side you can use mapreduce.

MapReduce

This is probably not as simple as you would want it to be. But until mongodb supports a one liner for group by + order by function this is the only way to go about it. You can refer the MapReduce documentation in the official site to get a better understanding about how it works. But if you just want to do a simple group by + order by equivalent in mongodb you can look at the sample bellow.
Mongodb Data Structure
gplus_profiles: id(objectId), have_user_in_circles_count (int)
SQL
SELECT COUNT(id), `have_user_in_circles_count` from `gplus_profiles`GROUP BY `have_user_in_circles_count` ORDER BY 1 DESC
Mongodb
m = function(){
emit(this.have_user_in_circles_count, { count : 1 });
};
r = function(key, values) {
var total = 0;
for ( var i=0; i<values.length; i++ )
total += values[i].count;
return { count:total };
};
res = db.gplus_profiles.mapReduce(m, r, { out : "myoutputt" } );
db.myoutputt.find().sort({value:-1})
Although we have to use mapreduce for the time being, a later version of mongodb will support group by + order by like functionality natively.


		
		
	

Google Plus and Google Profiles

Published on:
Tags:
Was a bit confused about Google Profiles and Google Plus profiles. A quick note for my self about the two.
  • Both google plus and google profiles seems to be using the same user id
  • A google profile may or may not have a google plus account (if the user didn’t opt for google plus)
  • A user who has signed up for both google profile and google plus have one id and one page for both services.
    • which is the google plus profile page.
  • A user who only signed up for google profile has a google profile page (which is different from a google plus profile page).
  • A user who only signed up for google plus has a google profile page which points to his/her google plus page.
  • So in essences google plus is a upgrade on top of the google profile.
Happy Christmas everyone!

Merlion

Published on:
Tags:
Merlion by thekindofme
Merlion, a photo by thekindofme on Flickr.

The Merlion (Malay: Singa-Laut) is a mythical creature with the head of a lion and the body of a fish, used as a mascot of Singapore. Its name combines “mer” meaning the sea and “lion”. The fish body represents Singapore’s origin as a fishing village when it was called Temasek, which means “sea town” in Javanese. The lion head represents Singapore’s original name — Singapura — meaning “lion city” or “kota singa”.
The symbol was designed by Fraser Brunner, a member of the Souvenir Committee and curator of the Van Kleef Aquarium, for the logo of the Singapore Tourism Board (STB) in use from 26 March 1964 to 1997[1] and has been its trademarked symbol since 20 July 1966. Although the STB changed their logo in 1997, the STB Act continues to protect the Merlion symbol.[2] Approval must be received from STB before it can be used. The Merlion appears frequently on STB-approved souvenirs.

From: Wikipedia(http://en.wikipedia.org/wiki/Merlion)

Batch Video Conversion to 3gp Using Ffmpeg

Published on:
Tags:
Some notes for self reference.

Batch video conversion

Using some a simple loop (in bash) for f in *.mp4; do ffmpeg -i “${f}” -b 4000k -s 4cif -vcodec h263 -acodec aac -y “${f}_4000.3gp”; done; Converts all mp4 files in the current folder
Original file name: file_one.mp4
Converted file name: file_one.mp4_4000.3gp

Convert to 3gp

ffmpeg -i file_name -s 4cif -vcodec h263 -acodec aac -y converted.3gp

Convert to with a set bit-rate

When the set bit rate is 4000k (kilo bits) ffmpeg -i file_name -b 4000k -s 4cif -vcodec h263 -acodec aac -y converted.3gp

Setting resolution

Use the -s flag to set the resoultion. For possible values refer: http://ffmpeg.org/ffmpeg-doc.html

Good Old Days [Photo]

Published on:
Tags:
good old days by thekindofme
good old days, a photo by thekindofme on Flickr.

Some where near the tanjong pagar market and food centre. From atop of a HDB void deck at night (around 9.30PM) ;).

0.6 sec at f/5.3 45mm with the nikon 18-55 kit lens.