Ruby process & ActiveRecord data set executing in multi cores

You know what! in one of our (tekSymmetry LLC) projects, we have so many background calculations,
which usually takes so many hours to get fully completed. ever since we have introduced those processes,
we were having problem with it’s execution time. sometimes it get’s in nerve

as you know a single ruby process can use a single processor’s core at a time.
this is probable one of the reasons why muli processes based deployment
strategy is picked by ruby on rails community.

anyway, these days our servers got more than one core! more precisely,
in our case each of our production server got 8 cores based intel xeon processor.

so you see the question rose if we could run those long running expensive process in multicores
our system could have better chance to get faster!.

well this blog post is intended for showing you the technique how we have done it in ruby on rails.

for better understanding, let me give you some hints so you can get the context -

• we have big database table rows!
• processing a single row doesn’t require anything from the same database table.
• we are using linux (in our case debian lenny)

so here is the way we have done it -

1. we took the max rows count for the main query
2. and divided by the number of cores we have
3. then we forked child process with each subset of the rows
4. and executed the logic and related stuffs!
5. on the parent process we initiated a loop where it was checking the newly forked process status
6. if all the pid files (which are generated by the newly forked children) are removed,
   parent process will flag it as successful execution thus it will end the loop.

so you see, it is damn! simple _) and it is working for us _),
it has improved our execution time 8x faster, because of getting 8 cores in new server.

here is the code in ruby how we did it. (we created a helper “multicore_execution_helper.rb“  and included in model, thus execute_in_multicores became usable)

module MulticoreExecutionHelper

  def execute_in_multicores(
      p_cores, p_total_rows, p_model, p_conditions = {}, &block)

    p_cores == 2 if p_cores.to_i == 0
    total_items_per_core = p_total_rows / p_cores
    logger.info "[BATCH-PROCESS-LOG] Total processes - #{p_cores}, " +
                "total rows - #{p_total_rows} [#{total_items_per_core} / 1 core]"

    # Create job id for each process
    job_ids = p_cores.times.collect{|i| rand.to_s }

    # Fork process for each core and execute the block
    p_cores.times do |offset|
      Process.fork do
        logger.info "[BATCH-PROCESS-LOG] Starting thread - #{offset} " +
                    "assigned # #{job_ids[offset]}"

        # Keep job track through the created process pid file.
        pid_file = File.join(RAILS_ROOT, 'tmp/pids/', "#{job_ids[offset]}.pid")
        File.open(pid_file, 'w') {|f| f.puts Process.pid.to_s}

        # Since fork process is created from the sample of the parent
        # process's memory so we need to reconnect all live connections.
        begin
          ActiveRecord::Base.connection.reconnect!

          # Retrieve data from the specific row through the defined
          # offset and limit
          teams = p_model.find(
              :all, {
                   ffset => (offset * total_items_per_core),
                  :limit => total_items_per_core}.merge(p_conditions))

          block.call(teams)
        rescue => $e
          logger.error "[BATCH-PROCESS-LOG] Exception raised during " +
                       "execution - #{$e.inspect}"
        end

        # Remove pid since we are done here!
        FileUtils.rm(pid_file)
      end
    end

    # monitor whether the process is completed or still in progress
    # don't return this method unless all the forked processes have
    # completed their job
    sleep(2)

    while 1 do
      fully_completed = true
      for job_id in job_ids
        pid_file = File.join(RAILS_ROOT, 'tmp/pids/', "#{job_id}.pid")
        if fully_completed && File.exists?(pid_file)
          fully_completed = false
          break
        end
      end

      break if fully_completed
      sleep(2)
      logger.debug '[BATCH-PROCESS-LOG] again...'
    end
  end

end

here is the usages code -

      execute_in_multicores(p_total_cores, SomeStuff.count, SomeStuff) do |some_stuffs|
        # Do.. whatever you wanna do with the stuff here! these are gonna run on multicores!
      end

see it is really simple!  _) if you like it let me know! how much you like it  _)
here you can find the code on github 

best wishes!

Ruby on Rails demo application presentation is picked by slideshare’s editor

Today morning i was informed by an email that my slide on slideshare is picked by their editor to keep it on their featured slides list.

it was really too great things for me. i congrats those slideshare’s guys!

you can check out slide here -

here is the moment i locked it up on screen shot!