so far i can assume, this problem was occurring due to twitter oauth service call validation, since the attached timestamp (through OAuth timestamp or some other header which was carrying request time stamp) was giving far beyond time (ie 30 mins beyond in my case). thus twitter was rejecting the request.

anyway, after synchronizing my system date and time, i can see it’s working.

thanks “baline garrett”

as you know a single ruby process can use a single processor’s core at a time.
this is probable one of the reasons why muli processes based deployment
strategy is picked by ruby on rails community.

anyway, these days our servers got more than one core! more precisely,
in our case each of our production server got 8 cores based intel xeon processor.

so you see the question rose if we could run those long running expensive process in multicores
our system could have better chance to get faster!.

well this blog post is intended for showing you the technique how we have done it in ruby on rails.

for better understanding, let me give you some hints so you can get the context -

• we have big database table rows!
• processing a single row doesn’t require anything from the same database table.
• we are using linux (in our case debian lenny)

so here is the way we have done it -

1. we took the max rows count for the main query
2. and divided by the number of cores we have
3. then we forked child process with each subset of the rows
4. and executed the logic and related stuffs!
5. on the parent process we initiated a loop where it was checking the newly forked process status
6. if all the pid files (which are generated by the newly forked children) are removed,
   parent process will flag it as successful execution thus it will end the loop.

so you see, it is damn! simple _) and it is working for us _),
it has improved our execution time 8x faster, because of getting 8 cores in new server.

here is the code in ruby how we did it. (we created a helper “multicore_execution_helper.rb“  and included in model, thus execute_in_multicores became usable)

module MulticoreExecutionHelper

  def execute_in_multicores(
      p_cores, p_total_rows, p_model, p_conditions = {}, &block)

    p_cores == 2 if p_cores.to_i == 0
    total_items_per_core = p_total_rows / p_cores
    logger.info "[BATCH-PROCESS-LOG] Total processes - #{p_cores}, " +
                "total rows - #{p_total_rows} [#{total_items_per_core} / 1 core]"

    # Create job id for each process
    job_ids = p_cores.times.collect{|i| rand.to_s }

    # Fork process for each core and execute the block
    p_cores.times do |offset|
      Process.fork do
        logger.info "[BATCH-PROCESS-LOG] Starting thread - #{offset} " +
                    "assigned # #{job_ids[offset]}"

        # Keep job track through the created process pid file.
        pid_file = File.join(RAILS_ROOT, 'tmp/pids/', "#{job_ids[offset]}.pid")
        File.open(pid_file, 'w') {|f| f.puts Process.pid.to_s}

        # Since fork process is created from the sample of the parent
        # process's memory so we need to reconnect all live connections.
        begin
          ActiveRecord::Base.connection.reconnect!

          # Retrieve data from the specific row through the defined
          # offset and limit
          teams = p_model.find(
              :all, {
                   ffset => (offset * total_items_per_core),
                  :limit => total_items_per_core}.merge(p_conditions))

          block.call(teams)
        rescue => $e
          logger.error "[BATCH-PROCESS-LOG] Exception raised during " +
                       "execution - #{$e.inspect}"
        end

        # Remove pid since we are done here!
        FileUtils.rm(pid_file)
      end
    end

    # monitor whether the process is completed or still in progress
    # don't return this method unless all the forked processes have
    # completed their job
    sleep(2)

    while 1 do
      fully_completed = true
      for job_id in job_ids
        pid_file = File.join(RAILS_ROOT, 'tmp/pids/', "#{job_id}.pid")
        if fully_completed && File.exists?(pid_file)
          fully_completed = false
          break
        end
      end

      break if fully_completed
      sleep(2)
      logger.debug '[BATCH-PROCESS-LOG] again...'
    end
  end

end

here is the usages code -

      execute_in_multicores(p_total_cores, SomeStuff.count, SomeStuff) do |some_stuffs|
        # Do.. whatever you wanna do with the stuff here! these are gonna run on multicores!
      end

see it is really simple!  _) if you like it let me know! how much you like it  _)
here you can find the code on github 

best wishes!

i had the following code in Rakefile

require ‘cucumber/rake/task’

Cucumber::Rake::Task.new do |t|
profile = ENV['PROFILE'] || ‘default’
t.cucumber_opts = “–profile #{profile}”
end

the fix is just keep the following code only -

require ‘cucumber/rake/task’

Cucumber::Rake::Task.new do |t|
end

so are you still facing problem while you are executing “rake features” but it doesn’t come up with any output?
here is the check list – (this list may grow gradually) -
1. do you have features directory
2. do you have features/steps directory
3. lets say you have “features/transfer.feature” file do you know that you must have “features/steps/transfer_steps.rb” file?
4. do you know “feature name” must be prefixed for steps file?
hope this might help you.

though all these “subversion, git, perforce” methods are not predefined, these will be added while new implemention is added. my skeleton (abstract) implementation was in VersionControl::Service, so whatever implemention (svn, git, perforce, other) comes ahead it will implement method from VersionControl::Service method. since ruby doesn’t support class abstraction so this was the only way  i got in head.

so my skeleton implemention was consist of the following code in summary -

module VersionControl

# log entry class for data object
class LogEntry; end

# local repository information
class Information; end

# abstract service class.
# define service which will be exposed for a
# normal version controlling service.
class Service

# retrieve recent logs from the current directory
# required parameters - *base project path* and other options.
def logs(p_path, p_options = {}); raise "not implemented method" end

# generate diff from mentioned revision number
# or current revision number
# required parameters - *base project path* and other options.
def diff(p_path, p_options = {}); raise "not implemented method" end

# find repository information
def info(p_path, p_options = {}); raise "not implemented method" end
# check out content from version control server
def checkout(p_source, p_path, p_options = {}); raise "not implemented method" end

end

# factory class for supporting different version control implementation
class ServiceFactory; end
end

and the implementation is written in this way -

class ServiceFactory

class < < self
# add subversion factory method implementation
@@subversion_instance = SubversionService.new
def subversion
return @@subversion_instance
end
end
end

so every implementation also push it’s factory method to “VersionControl::ServiceFactory” class. so this way i have implemented dynamic factory method on ruby.
]]> http://we4tech.wordpress.com/2008/05/10/ruby-dynamic-factory-method-implementation/feed/ 1 hasan http://we4tech.wordpress.com/2008/02/10/upcoming-project-mojar_workflow-workflow-engine-in-ruby/ http://we4tech.wordpress.com/2008/02/10/upcoming-project-mojar_workflow-workflow-engine-in-ruby/#comments Sun, 10 Feb 2008 07:59:25 +0000 nhm tanveer hossain khan http://hasan.we4tech.com/upcoming-project-mojar_workflow-workflow-engine-in-ruby/ Continue reading →]]> hi,
we just kicking start a new open source ruby based workflow engine project “mojar workflow“.
we named it after our deshi word “mojar” reason is very clear to
spread out this word.

mojar workflow, is integral solution to execute a flow of business
rules. for example -

you have an action where you have the following set of rules -
1. start transaction
2. verify user account
3. verify user balance
4. verify user dues
5. reduce dues from balance
6. complete transaction

after few days you got a new requirement, where you suppose to reduce
user dues by the 10% because of company new discount policy.
so you have to implement the following rules -
1. start transaction
2. verify user account
3. verify user balance
4. verify user dues
5. reduce dues by 10% of discount
5. reduce discounted dues from balance
6. complete transaction

to implement such scenario you have to again code in your stable
release. but using mojar workflow, you can add that new concern from
the abstract flow maintenance layer. where you can define this flow in
yaml file or an xml document.

keep your eyes on -
http://rubyforge.org/projects/mojarworkflow/

best wishes,

after implementing this stuff, i replaced “render(:partial => …)” with the following method -

render_from_cache_or_render(:cache_key =>”cache key”, :cache_expire_after => ConstantHelper::TAG_CLOUD_EXPIRED_IN, # minutes :partial => “….”)

let’s have a look on my implementation -

def render_from_cache_or_render(p_args)

return render(p_args) if true == p_args[:cache_off]

# check from cache
cache_key = p_args[:cache_key]
cached_content = CacheService.get_cache(cache_key)

if not cached_content.nil? and not cached_content.empty?
return cached_content
else
content = render(p_args)
# cache expire time if defined
cache_expire_time_in_minutes = p_args[:cache_expire_after] || 60
CacheService.add_cache(cache_key, cache_expire_time_in_minutes, content)
return content
end
end

actually, my implemented “CacheService” class is simply storing all cache in a hash map.
when some cache was requested for peek, cache expiry was checked before returning the cached value.

for CacheService implementation look at the bottom of my post.

anyway, after implementing and utilizing this stuff, i gained 70+ requests capability per second. fyi, before applying cache it was around 10 per second.

module Cache
class Item
attr_accessor :key, :expire_time, :content, :created_on

def initialize(p_key, p_expire_time, p_content)
@key = p_key
@expire_time = p_expire_time * 60 # in minutes
@content = p_content
@created_on = Time.now
end
end
end

class CacheService
@@CACHES = {}
@@CACHE_EXPIRE_TIMES = {}

def self.add_cache(p_key, p_expire_time, p_content)
cache_item = Cache::Item.new(p_key, p_expire_time, p_content)
@@CACHES[p_key.to_sym] = cache_item
end

def self.get_cache(p_key)
# load content from cache
cached_content = @@CACHES[p_key.to_sym]
return nil if cached_content.nil?

# verify cache validity
return cached_content.content if not expired?(cached_content)
return nil
end

private
def self.expired?(p_cache)
# find time difference
time_difference = Time.now – p_cache.created_on
return true if time_difference > p_cache.expire_time
end
end

best wishes,

Errno::EINVAL (Invalid argument):
/app/models/index_service.rb:63:in `write’
/app/models/index_service.rb:63:in `puts’

this problem was because of “puts” what i forgot to remove before deploying on test server.
if your deployment on windows service environment and if your code has few “puts” usages, you must face this problem with mongrel

on mongrel group, i found they are working with this, hopefully they will replace puts with logger and other things.
anyway, the quickest solution i had in mind was just use the dynamic behavior of ruby. here is what i did -

def puts(p_args)
logger.debug(p_args)
end

thats all fixed my problem

thank ruby, thanks for dynamic scripting…

here is how my simple example is running-

def test_aop
# apply pointcuts
apply_advices(:before, /^do_.+/, SimpleService, SimpleServiceAdvice, :before_do)
apply_advices(:around, /^do_.+/, SimpleService, SimpleServiceAdvice, :around_do)
apply_advices(:after, /^do_.+/, SimpleService, SimpleServiceAdvice, :after_do)

simple_service = SimpleService.new
simple_service.do_1(“A”)
end

here is the parameter name.

apply_advices(type_of_advice, pointcuts_in_regex, service_class, advice_class, advice_method)

instead of making aspectJ type pointcuts syntax, i have used regex, which is fine for the time being.

if you run this code you will find the following output -

Before execution.
Around {
1 performed – A
}
After execution.

now let’s have a look on my SimpleService class.

class SimpleService

def do_1(p_param)
puts “1 performed – #{p_param}”
end

def do_2
puts “2 performed.”
end

def no_do
puts “No do”
end
end

and here is my aspect class,

class SimpleServiceAdvice

def around_do(p_invoke)
puts “Around { “
output = p_invoke.proceede()
puts “}”
return output
end

def before_do(p_invoke)
puts “Before execution.”
end

def after_do(p_params, p_output)
puts “After execution.”
end
end

my implementation is very straight forward, actually during implementing this stuff, i really felt the strength of meta programming. it is so flexible and so easy that sky is the limit.

here is the attached source code.

much better and complete aop implementation in ruby

few days back, we found our application_helper and few controllers are growing too fast and getting extra fat (lines of code). so we had few refactoring to reduce the extra fat.

now have a look on the code we had with in application_helper.rb taken from tag/v-0.3

this code is not completely visible over the screen snap, this is 340 number of lines. which was the output of our 3 iterations.

though these number of lines are not that much problematic, but we had a scenario which was difficult to make it more concern aware and single concerned.

now have a look on our code which is taken from the current trunk,

Wow, now it is 50 lines only including the header copyright information.
the trick was very simple, we followed the following conventions -

1. find out all related and same concerned functions
2. stick team together in a module
3. include the module to statically import all functions
no integration error, nothing has occurred.
we are happy with this

i think, our ruby learning process is going smooth

i don’t know whether any design pattern or best practice is already exist on this topic.
i came up with something that is really good for me. here is a bit about my task and also the explanation about what i did.

i have a model “category” where i have the following relations -

has_many :category_mappings, :dependent => :destroy
has_many :items, :through => :category_mappings

has_many :categories, :foreign_key => “parent_id”, :dependent => :destroy

has_many :attribute_category_mappings, :dependent => :destroy
has_many :properties, :through => :attribute_category_mappings

belongs_to :category

i was applying :dependent = > :destroy with the mapping model and child categories, which are needs to be removed as a part of the category destroy process.

so i had a messy unit test method to ensure this is working fine. sorry for not keeping my old code snaps otherwise i could show the messy one. anyway here what i wrote later to make it bit cleaner than the previous messy version -

def test_destroy_category_with_dependent
category = Category.find(3)

# prepare for verification
prepare_for_verifying_related_property_mappings(category)
prepare_for_verifying_category_mappings(category)
prepare_for_verifying_child_category(category)

# perform action
category.destroy

# ensure the action
assert_raise(ActiveRecord::RecordNotFound) {
Category.find(category.id)
}

# perform verification
verify_related_property_mappings(category)
verify_related_category_mappings(category)
verify_related_child_category(category)
end

here i just sliced my test method in 2 different roles,

1. preparation stage
2. verification stage

1. preparation stage -
in this stage, i just keep log for current state or other steps which are important for later testing.

2. verification stage -
in this stage, i just verified my new state comparing with the old state (which was kept during preparation stage).
so let’s have a look on the typical code which i wrote in  my preparation and verification stage -

def verify_related_child_category(p_category)
puts “verify related child categories – #{p_category}”
now_category_count = Category.count
assert_equal(true, ((@old_category_count – now_category_count) > 1),
“No category has been removed.”)
end

this seem pretty good for me, as long as i can make my code bunch simple and easy to change.

best wishes,