EzDevInfo.com

Clockwork / Clockworkd produces 'No Rakefile found (looking for: rakefile, Rakefile, rakefile.rb, Rakefile.rb)'

I'm running clockwork in the background using Clockworkd. If I just run clockwork, it executes the rake command successfully. If I run it via clockworkd, it produces an error. I'm not sure why it isn't working. Any advice / suggestions would be appreciated.

Starting clockworkd

RAILS_ENV=production clockworkd -c lib/clock.rb start --log

The error

I, [2013-09-05T17:53:01.035923 #2580]  INFO -- : Triggering 'Get Updates'
rake aborted!
No Rakefile found (looking for: rakefile, Rakefile, rakefile.rb, Rakefile.rb)
/Users/lukesmith/.rvm/gems/ruby-1.9.3-p327/gems/rake-10.1.0/lib/rake/application.rb:632:in `raw_load_rakefile'
/Users/lukesmith/.rvm/gems/ruby-1.9.3-p327/gems/rake-10.1.0/lib/rake/application.rb:94:in `block in load_rakefile'
/Users/lukesmith/.rvm/gems/ruby-1.9.3-p327/gems/rake-10.1.0/lib/rake/application.rb:165:in `standard_exception_handling'
/Users/lukesmith/.rvm/gems/ruby-1.9.3-p327/gems/rake-10.1.0/lib/rake/application.rb:93:in `load_rakefile'
/Users/lukesmith/.rvm/gems/ruby-1.9.3-p327/gems/rake-10.1.0/lib/rake/application.rb:77:in `block in run'
/Users/lukesmith/.rvm/gems/ruby-1.9.3-p327/gems/rake-10.1.0/lib/rake/application.rb:165:in `standard_exception_handling'
/Users/lukesmith/.rvm/gems/ruby-1.9.3-p327/gems/rake-10.1.0/lib/rake/application.rb:75:in `run'
/Users/lukesmith/.rvm/gems/ruby-1.9.3-p327/gems/rake-10.1.0/bin/rake:33:in `<top (required)>'
/Users/lukesmith/.rvm/gems/ruby-1.9.3-p327/bin/rake:23:in `load'
/Users/lukesmith/.rvm/gems/ruby-1.9.3-p327/bin/rake:23:in `<main>'

clock.rb

require 'clockwork'
require File.expand_path('../../config/boot', __FILE__)
require File.expand_path('../../config/environment', __FILE__)

include Clockwork

handler do |job|
  puts "Running #{job}"
end

every(1.hour, 'Get Updates') { `rake get_updates` }

Running Rake Task with Clock Process

In my Heroku application, I have a clock process which needs to periodically execute a rake task.I am using the clockwork gem.Here is the code in lib/clock.rb :

require File.expand_path('../../config/boot',        __FILE__)
require File.expand_path('../../config/environment', __FILE__)
require 'clockwork'

include Clockwork

every(1.day, 'API Data Fetching', :at => '18:31') { `rake build_index` }

Here is the Heroku log snippet:

2013-07-15T18:31:00.716092+00:00 app[clock.1]: I, [2013-07-15T18:31:00.673880 #2]  INFO -- : Triggering 'API Data Fetching'
2013-07-15T18:31:39.131681+00:00 app[clock.1]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/1/4/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/Rakefile:7)
2013-07-15T18:31:39.132841+00:00 app[clock.1]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/1/4/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/Rakefile:7)
2013-07-15T18:31:39.132346+00:00 app[clock.1]: DEPRECATION WARNING: You have Rails 2.3-style plugins in vendor/plugins! Support for these plugins will be removed in Rails 4.0. Move them out and bundle them in your Gemfile, or fold them in to your app as lib/myplugin/* and config/initializers/myplugin.rb. See the release notes for more on this: http://weblog.rubyonrails.org/2012/1/4/rails-3-2-0-rc2-has-been-released. (called from <top (required)> at /app/Rakefile:7)

The task is not being executed and no API data is being fetched. What could be the issue? Is running a rake task from clock.rb bad idea?

Scheduled Jobs and Custom Clock Processes with Clockwork

I am trying to understand how to execute custom code with clockwork. This is the example lib/clock.rb file that Heroku uses in its devcenter document.

require File.expand_path('../../config/boot',        __FILE__)
require File.expand_path('../../config/environment', __FILE__)
require 'clockwork'

include Clockwork

every(4.minutes, 'Queueing interval job') { Delayed::Job.enqueue IntervalJob.new }
every(1.day, 'Queueing scheduled job', :at => '14:17') { Delayed::Job.enqueue ScheduledJob.new }

What is IntervalJob and ScheduledJob? Where are these files supposed to be located? I want to run my own custom job that has access to my database records.

EDIT

This is my /lib/clock.rb

require 'clockwork'
require './config/boot'
require './config/environment'

module Clockwork

  handler do |job|
    puts "Running #{job}"
  end

  every(2.minutes, 'Filtering Streams') { Delayed::Job.enqueue FilterJob.new}
end

This is my /lib/filter_job.rb

  class FilterJob
    def perform
      @streams = Stream.all

      @streams.each do |stream|
      # manipulating stream properties
      end
    end
   end

I get the error:

uninitialized constant Clockwork::FilterJob (NameError)
/app/lib/clock.rb:11:in `block in <module:Clockwork>'

Clockwork does not work as expected

When the task is run, I get always nil object for supplier parameter that I am passing to perform_later method for a reason. The user is set properly. Any advice why supplier object is not created and passed and user object is?

require File.expand_path('../../config/boot',        __FILE__)
require File.expand_path('../../config/environment', __FILE__)
require 'clockwork'

include Clockwork


every(2.minutes, 'Queueing xml feed job'){ 
    @suppliers = Supplier.all
    @user = User.find(1) # admin / system user
    @suppliers.each do |supplier| 
      ImportXmlFeedJob.perform_later(supplier, @user) if !supplier.xml_feed_url.blank?
    end
  }

clockwork gem not running the job when :at => "some time" is given

I am trying to setup clockwork for a recurring job. (Rails 4.2) The following code from the gem read-me in app/clock.rb works and every 30 seconds the background task is fired and email is sent.

require 'clockwork'
require File.expand_path('../../config/boot',        __FILE__)
require File.expand_path('../../config/environment', __FILE__)

module Clockwork

   handler do |job, time|
    puts "Running #{job} , at #{time}"
   end
     #History is a class method which runs the logic and ques the job
   every(30.seconds, 'send_history.job'){History.recur}
end

If I replace the recurring statement to the following then it doesn't work.

 every(1.day, 'send_history.job', :at => '22:00'){History.recur}

Any thoughts? I searched for others with this issue but couldn't find any. I am using delayed_job with Active job for queing. thanks

Schedule clockwork to run only once on start up

I have a task for clockwork gem to run only once on start of clockwork process. How do I achieve that because all tasks defined start with every(1.x,....)

Does Clockwork require its own process?

I've seen people say in a couple of different places that clockwork should be run on its own dyno, with a procfile that might look something like this (example from heroku):

clock: bundle exec clockwork lib/clock.rb

Is there any reason not to run it on the same dyno as the workers? With a process that looks something like this:

worker: bundle exec clockwork clock.rb & bundle exec sidekiq -C config/sidekiq.yml -L log/sidekiq.log

It seems to work fine this way, but I'd like to know the underlying reason why people say not to do it this way. Any guidance would be appreciated. Thanks in advance.

Sidekiq with Clockwork triggers but doesn't process job on heroku

Running: sidekiq 2.1.17, rails 3.2.21

I'm trying to use clockwork to schedule some recurring tasks to be done by sidekiq workers. Running foreman in development, everything runs perfectly as scheduled.

When I deploy to heroku, however, I get the "Triggering 'NameWorker.perform_async'" message in the logs at the appropriate times but then the respective jobs don't actually run.

When I instead call NameWorker.perform_async in a controller action, the job runs as it should. The jobs I'm trying to schedule just contain puts statements to verify that they're working. Anyone have any ideas about what I'm doing wrong? Thanks in advance.

Run ruby scripts using daemon and watch files using daemon process

I have some download/upload ruby scripts. I want to run those ruby scripts using schedulers. So for that I need to create a couple of daemons/schedulers to download stats from various providers and to upload stats to Mysql.

Can anyone suggest better options (gems) for this that might make sense in Ruby?

I want to run daemon and after a daemon starts, it will read the config file and then basically run forever doing whatever tasks it is supposed to do. The daemons should be able to wake up periodically to do things, should be able to wake up at specific times. The daemons should also be able to watch for file system changes… like the uploader should watch for creation of new files(which are ready to upload) under the download path to know it has work to do. The daemons should keep a log file of the major actions they do and failures they encounter. I want some kind of Ruby libraries/tools that help out with this as much as possible.

Let me suggest what kind of options are available.

Currently I'm trying with https://github.com/tomykaira/clockwork gem. But I'm stucked at how can I watch files using daemon process?

Rails clockwork being limited to every 24 hours at most on heroku?

I'm attempting to run jobs on the order of every month in my Rails app hosted on Heroku. My goal was to be able to dynamically schedule recurring jobs based on our web interface (i.e. user signs up for some monthly or weekly or daily action)

My code is based on the readme for clockwork, and works correctly with second and minute frequencies, but once we make the frequency anything longer than a day, a month for example, it executes daily.

Does anyone know why this could be? Heroku seems to have officially endorsed clockwork, so it would be surprising if was on their end, but they also don't mention any support of DB Scheduled tasks, and I know workers restart daily. Would the worker-restart reset the frequency timer on long scheduled tasks?

Code below for reference:

[1] pry(main)> x = TweetSchedulerTask.first
  TweetSchedulerTask Load (0.9ms)  SELECT "tweet_scheduler_tasks".* FROM "tweet_scheduler_tasks" LIMIT 1
=> #<TweetSchedulerTask id: 5, frequency_quantity: 1, frequency_period_id: 6, at: "14:30", created_at: "2014-09-08 05:03:20", updated_at: "2014-09-08 05:03:20", charity_id: 392560474, custom_statuses: nil, active: true, user_id: 2713081644, donation_amount: 100>
[2] pry(main)> x.frequency_period
  FrequencyPeriod Load (1.0ms)  SELECT "frequency_periods".* FROM "frequency_periods" WHERE "frequency_periods"."id" = 6 LIMIT 1
=> #<FrequencyPeriod id: 6, name: "month", created_at: "2014-08-08 02:07:06", updated_at: "2014-08-08 02:07:06">
[3] pry(main)> x.frequency
=> 2592000
[4] pry(main)> 2592000/30
=> 86400
[5] pry(main)> 86400/24
=> 3600
[6] pry(main)> 3600/60
=> 60
[7] pry(main)> x.frequency_period
=> #<FrequencyPeriod id: 6, name: "month", created_at: "2014-08-08 02:07:06", updated_at: "2014-08-08 02:07:06">
[8] pry(main)> x.frequency_period.name
=> "month"
[9] pry(main)> x.frequency_period.name.pluralize
=> "months"
[10] pry(main)> x.frequency_quantity.send x.frequency_period.name.pluralize
=> 2592000

This task is running daily: (from logs)

Sep 09 07:30:01 blahblah-8448 app/sidekiq.1:    TweetSchedulerTask Load (221.6ms)  SELECT 
"tweet_scheduler_tasks".* FROM "tweet_scheduler_tasks" WHERE "tweet_scheduler_tasks"."id" = $1 LIMIT 1  [["id", "5"]] 
Sep 10 07:30:05 blahblah-8448 app/sidekiq.1:    TweetSchedulerTask Load (35.2ms)  SELECT "tweet_scheduler_tasks".* FROM "tweet_scheduler_tasks" WHERE "tweet_scheduler_tasks"."id" = $1 LIMIT 1  [["id", "5"]] 

clock.rb

  Clockwork.manager = ManagerWithDatabaseTasks.new

  sync_database_tasks model: TweetSchedulerTask, every: 1.minute do |instance_job_name|
    # TODO: pmarx make this so it can wait for the second time around to run
    id = instance_job_name.split(':').last
    TweetSchedulerTask.perform_async(id)
  end

Clockwork and Resque

I am trying to run a simple task to test out using resque and clockwork together.

My worker: app/workers/logger_helper.rb

class LoggerHelper
  @queue = :log_queue

  def self.before_perform
    Rails.logger = Logger.new(File.open(Rails.root.join('log', 'resque.log')))
    Rails.logger.level = Logger::DEBUG
  end

  def self.perform
    time = Time.now
    Rails.logger.info("testing #{time}")
  end
end    

My clock.rb file lib/clock.rb

  require File.expand_path('../../config/boot',        __FILE__)
  require File.expand_path('../../config/environment', __FILE__)
  require 'clockwork'

  module Clockwork

  handler do |job|
    Resque.enqueue(job)
  end

  every(10.seconds, 'loggerhelper') {LoggerHelper}
end

Rake file:

require File.expand_path('../config/application', __FILE__)
Rails.application.load_tasks
require 'resque/tasks'
task(:default).clear
task default: [:spec]

task "resque:setup" => :environment do
  QUEUE = '*'
end

First I run resque:setup Second I run clockwork lib/clockwork.rb

I get the following output in the terminal

INFO -- : Triggering 'loggerhelper'
INFO -- : Triggering 'loggerhelper'
...

But nothing writes to the log.

I've tried a combination of things but I don't see any output.

I did run

every(10.seconds, 'loggerhelper') {LoggerHelper.perform}

in the clock.rb file and it does work, but I didn't think that you were supposed to call deliver directly. Also I'm not sure if it's actually running off of the queue or just simply executing.

Laravel disable provider (Clockwork) on fly

I'm having a laravel setup with the Clockwork installed. And I have a artisan command that takes really long time to run, so that clockwork will consume too much memory. I still need that extension. How can I disable it when just running a command?

Logging in delayed_job?

I can't get any log output from delayed_job, and I'm not sure my jobs are starting.

Here's my Procfile:

web:     bundle exec rails server
worker:  bundle exec rake jobs:work
worker:  bundle exec clockwork app/clock.rb

And here's the job:

class ScanningJob
  def perform
    logger.info "logging from delayed_job"
  end

  def after(job)
    Rails.logger.info "logging from after delayed_job"
  end
end

I see that clockwork outputs to system out, and I can see worker executor starting, but I never see my log statements hit. I tried puts as well to no avail.

My clock file is pretty simple:

every(3.seconds, 'refreshlistings') { Delayed::Job.enqueue ScanningJob.new }

I just want to see this working, and lack of logging means I can't. What's going on here?

Heroku clock process: how to ensure jobs weren't skipped?

I'm building a Heroku app that relies on scheduled jobs. We were previously using Heroku Scheduler but clock processes seem more flexible and robust. So now we're using a clock process to enqueue background jobs at specific times/intervals.

Heroku's docs mention that clock dynos, as with all dynos, are restarted at least once per day--and this incurs the risk of a clock process skipping a scheduled job: "Since dynos are restarted at least once a day some logic will need to exist on startup of the clock process to ensure that a job interval wasn’t skipped during the dyno restart." (See https://devcenter.heroku.com/articles/scheduled-jobs-custom-clock-processes)

What are some recommended ways to ensure that scheduled jobs aren't skipped, and to re-enqueue any jobs that were missed?

One possible way is to create a database record whenever a job is run/enqueued, and to check for the presence of expected records at regular intervals within the clock job. The biggest downside to this is that if there's a systemic problem with the clock dyno that causes it to be down for a significant period of time, then I can't do the polling every X hours to ensure that scheduled jobs were successfully run, since that polling happens within the clock dyno.

How have you dealt with the issue of clock dyno resiliency?

Thanks!

Sidekiq & Clock Background Task On Redis, Rails, Heroku Only Updating Some

I am running a background task on many records with Rails, Sidekiq, Redis, and Clockwork on Heroku, but I keep getting this error on many of the records it's trying to update with an API:

ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds

Here are my files:

Unicorn:

worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3)
timeout 15
preload_app true

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
  end
end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end

  if defined?(ActiveRecord::Base)
    config = Rails.application.config.database_configuration[Rails.env]
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    config['pool']            = ENV['DB_POOL'] || 5
    ActiveRecord::Base.establish_connection(config)
  end
end

Database:

production:
  adapter: postgresql
  encoding: unicode
  database: production
  pool: 25
  timeout: 10000

Clock:

every(24.hours, 'Update') do
  sites = Site.order(:id).pluck(:id)
  Site.each do |site|
    UpdatePosts.perform_async(site)
  end
end

UpdatePosts

class UpdatePosts
    include Sidekiq::Worker
    sidekiq_options retry: false

    def perform(site_id)
    ...
    end
end