EzDevInfo.com

Rails/Dragonfly/Apache - Rack::Cache - how to use X-Sendfile?

I'm using Dragonfly to serve processed images for my Rails app. Dragonfly relies on Rack::Cache for future visits to those processed images, so that Dragonfly won't have to process those images again and again, thus wasting CPU time.

My problem starts here: if I'm right that sending a file via Rack::Cache still busies a Rails process, then viewing a page of 30 images, even if these images have a small file size, will tie up the Rails processes pretty quickly. If a couple more visitors come to see that page, then they will experience very slow response times. How do I get these files served via X-Sendfile?

I've set the following in production.rb, but I know these are for the assets from Rails, not the Dragonfly files:

config.serve_static_assets = false
config.action_dispatch.x_sendfile_header = "X-Sendfile"

I know that Rack::Cache somehow supports X-Sendfile (probably through Rack::Sendfile) because it produces a body that responds to #to_path. However, I don't know how to enable this. When I check files that come from Rack::Cache, I don't see any X-Sendfile information:

Date: Wed, 02 Nov 2011 11:38:28 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.9
Content-Disposition: filename="2.JPG"
Cache-Control: public, max-age=31536000
Etag: "3174d486e4df2e78a5ff9174cacbede5787d4660"
X-Content-Digest: c174408eda6e689998a40db0aef4cdd2aedb3b6c
Age: 28315
X-Rack-Cache: fresh
Content-Length: 22377
Status: 200
Content-Type: image/jpeg

I know, based on posts around the net, that I'm supposed to see something like:

X-Sendfile: /path/to/file

In the end I don't know if its Dragonfly or Rack::Cache (or both) that I have to configure. How do I get either Dragonfly and/or Rack::Cache to serve files via X-Sendfile?

Info about my setup:

• Rails 3.1.1
• Passenger 3.0.9
• CentOS
• Sendfile module is installed, as far as I know. I have XSendFile On and XSendFilePath /path/to/app specified in my virtualhost configuration, and Apache doesn't complain about the directive XSendFile not existing.

Thanks!

UPDATE Nov 6, 2011

Based on this old update, as long as Rack::Sendfile is placed in front of Rack::Cache, then X-Sendfile will be used. I did that, and this is how my middleware looks like. The files, however, still don't have the X-Sendfile tag. Again, I don't know if that is a sure-fire way of determining if X-Sendfile is enabled, so I checked the Passenger queue. It seems that the queue is greatly encumbered when I visit a page.

UPDATE Nov 7, 2011

It seems this is purely a Rack::Cache and Rails 3.1 issue. While Rack::Cache supports the use of X-Sendfile through Rack::Sendfile (like I mentioned above, Rack::Cache, when using the Disk EntityStore since that responds_to to_path since the body it returns is a subclass of File), Rails 3.1 uses its own storage solution. Rails 3.1 uses ActiveSupport::Cache::FileStore, which is set by default, if you don't specify anything in your production.rb file.

The problem with FileStore is that the body it returns, to be part of the response to be sent upstream, because that body doesn't respond to to_path. The body is an instance of ActiveSupport::Cache::Entry. You can see here that when the FileStore is asked to read a cached file, it reads it via File.open('/path/to/file') {|f| Marshal.load(f) } which returns an instance of Entry. The value that ultimately gets passed upstream and back to the client, is Entry#value.

My questions

To help me decide whether I should patch this, or to get Rails to use Rack::Cache's own Disk store instead, I have some questions:

1. What's the reason Rack::Cache's own storage solutions weren't used for Rails 3.1? Why does Rails have its own?
2. Is there a reason Marshal is used? Is there a reason that a bytestream of data should be sent back instead?

I got in deeper than I usually go, and will be surprised if I understood things correctly. I hope to find an answer!

Rack::Cache not busting cache after changed etag

I have setup Rack::Cache to work with my rails application, heroku, Dalli and memcachd. Everything works as expected however when I update my content in the backend of my site the cache is still fresh. The etag is changed however this does not seem to bust Rack::Cache. How do I tell Rack::Cache to bust on content updates?

Ignore googlebot for Rack::Cache?

I'm using Rack::Cache, and google search bot triggers all my cache misses, so my cache is always full of pages that don't actually represent the most popular ones on my site, I can't seem to find how to ignore an IP or useragent for caching, has anyone had this problem before?

Disadvantages to rack-cache vs. Varnish in Heroku cedar stack?

The previous 2 Heroku application stacks came with a Varnish layer which automatically reverse-proxy-cached content based on http headers.

The new Heroku cedar stack doesn't have this Varnish layer. Heroku suggests using rack-cache and memcache instead.

Does this have disadvantages compared to the previous stacks with the varnish layer? With rack-cache, aren't there fewer servers serving the caching layer, and in a less optimized way?

• http://devcenter.heroku.com/articles/http-routing
• http://devcenter.heroku.com/articles/http-caching
• http://devcenter.heroku.com/articles/cedar

Bypass Rack::Cache for requests coming from a CDN

I have a Rails 3.2 app, deployed to heroku. I use caching with Rack::Cache and Amazon CloudFront.

For Requests that are served via CloudFront (assets, mostly), the caching layer with Rack::Cache is redundant and I'd rather not use it there (memcached on heroku is expensive)

What is a good way to skip Rack::Cache for these requests?

Rails: cache with substitutes?

I have a partial I am caching, but one part of it is dynamic:

- cache @product do
  .product
    #.....
    .price = format_money(@product.money)

And would like to turn it into something like this:

- cache_subst @product, {price: format_money(@product.money)} do
  .product
    #.....
    .price {{price}}

Is there some gem or method that does this?

Update:

Please don't suggest making money part of the key in any way, thats what we're doing right now.

HTTP cache with Rails + Rack::Cache not invalidating

My Rails 3 app generates pages that will change for a specified period of time and then be static (no changes) for the rest of their lifetime (think: sports scoreboard)

This seems like the perfect opportunity for full page caching, so I opted for Rack::Cache, using the Last-Modified part of the response to indicate when the cache was invalid.

The cache works well - too well. It seems that even when the Last-Modified field is updated with a date/time that is later than the If-Modified-Since field of the request and the response generates a status 200 (as opposed to 304), the browser still loads the version of the page that is cached on the server.

Here is what I see in the server debug logs:

App 16638 stdout: Started GET "/games/2014/2/10" for xx.xx.xx.xxx at 2014-02-11 04:04:11 +0000
App 16638 stdout: Processing by GamesController#index as HTML
App 16638 stdout:   Parameters: {"year"=>"2014", "month"=>"2", "day"=>"10"}
App 16638 stdout:   Game Load (2.8ms)  SELECT "games".* FROM "games" WHERE "games"."date" = '2014-02-10'
App 16638 stdout:   Game Load (2.8ms)  SELECT "games".* FROM "games" WHERE "games"."id" = 877 LIMIT 1
App 16638 stdout:   Team Load (1.3ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 10 LIMIT 1
App 16638 stdout:   Team Load (0.7ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 23 LIMIT 1
App 16638 stdout:   Game Load (4.2ms)  SELECT "games".* FROM "games" WHERE "games"."id" = 875 LIMIT 1
App 16638 stdout:   Team Load (0.6ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 17 LIMIT 1
App 16638 stdout:   Team Load (5.4ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 2 LIMIT 1
App 16638 stdout:   Game Load (7.0ms)  SELECT "games".* FROM "games" WHERE "games"."id" = 874 LIMIT 1
App 16638 stdout:   Team Load (0.7ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 9 LIMIT 1
App 16638 stdout:   Team Load (7.9ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 27 LIMIT 1
App 16638 stdout:   Game Load (0.7ms)  SELECT "games".* FROM "games" WHERE "games"."id" = 876 LIMIT 1
App 16638 stdout:   Team Load (0.4ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 18 LIMIT 1
App 16638 stdout:   Team Load (0.4ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 11 LIMIT 1
App 16638 stdout:   Game Load (0.6ms)  SELECT "games".* FROM "games" WHERE "games"."id" = 873 LIMIT 1
App 16638 stdout:   Team Load (4.0ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 12 LIMIT 1
App 16638 stdout:   Team Load (0.6ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 8 LIMIT 1
App 16638 stdout:   Game Load (5.4ms)  SELECT "games".* FROM "games" WHERE "games"."id" = 872 LIMIT 1
App 16638 stdout:   Team Load (0.6ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 28 LIMIT 1
App 16638 stdout:   Team Load (3.9ms)  SELECT "teams".* FROM "teams" WHERE "teams"."id" = 19 LIMIT 1
App 16638 stdout: Latest game: 2014-02-11 03:44:50 UTC
App 16638 stdout: Cache read: views/xxxx/games/2014/2/10
App 16638 stdout: Dalli::Server#connect 127.0.0.1:11211
App 16638 stdout: Read fragment views/xxxxx/games/2014/2/10 2.6ms
App 16638 stdout: Completed 200 OK in 618.2ms (ActiveRecord: 128.8ms)
App 15100 stderr: cache: [GET /games/2014/2/10] stale, invalid, store
App 16638 stdout: Started GET "/assets/bootstrap.css" for 66.55.150.181 at 2014-02-11 04:04:12 +0000
App 16638 stdout: Served asset /bootstrap.css - 304 Not Modified (12ms)
App 15100 stderr: cache: [GET /assets/bootstrap.css] stale, valid, store
App 15100 stderr: cache: [GET /assets/application-85cd667c6ae785b5d80f452fe6ad811e.js] fresh
App 15100 stderr: cache: [GET /assets/application-50dd9f33494a58a079e7417a68763e42.css] fresh

Here is a sample request/response (see the 200 status):

    Request URL:http://xxxxxxx.com/games/2014/2/10
Request Method:GET
Status Code:200 OK
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Cookie:_gamestory_app_session=BAh7B0kiD3Nlc3Npb25faWQGOgZFVEkiJWIwYjAwNWQ3NTBmYTk4YjQ0YWRjYjEwMWQ5Y2ZjYTA2BjsAVEkiEF9jcx3JmX3Rva2VuBjsARkkiMXdKNWZZUWFhZDVHN3hEeEJjaWxYclp1NmN4OGUrSkI4VzJ2eXdWOUsrc1E9BjsARg%3D%3D--6f99b71bca632560c069c371fdb6a4477a26dfa3
Host:xxxxxxxxx.com
If-Modified-Since:Tue, 11 Feb 2014 03:43:06 GMT
If-None-Match:"e282d8be52fbe437d3c2d3908288100d"
User-Agent:Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.17
Response Headersview source
Age:0
Cache-Control:public
Connection:keep-alive
Content-Length:10411
Content-Type:text/html; charset=utf-8
Date:Tue, 11 Feb 2014 04:04:11 GMT

ETag:"d6d4c1f573a56353e0abbe14de76e306"
Last-Modified:Tue, 11 Feb 2014 03:44:50 GMT
Server:nginx/1.4.4 + Phusion Passenger 4.0.29
Status:200 OK
X-Content-Digest:7b2caeed19cc4674669f4c386271054427342b5e
X-Powered-By:Phusion Passenger 4.0.29
X-Rack-Cache:stale, invalid, store
X-Request-Id:c19584efff9941bf5c456cb68d85b192
X-Runtime:0.661659
X-UA-Compatible:IE=Edge,chrome=1

In the relevant controller, I use a before_filter for the given action and in that filter, I use fresh_when on the last updated object on the page. This seems to generate the right date/times in the response, but I'm including the code for any possible errors (I removed the show action, for brevity):

class GamesController < ApplicationController
  helper ApplicationHelper
  before_filter :set_index_cache_headers, :only => [:index]
  caches_action :index

  def set_index_cache_headers
    @date = Date.today

    if params[:year] && params[:month] && params[:day]
      @date = Date.new(params[:year].to_i, params[:month].to_i, params[:day].to_i)
    else
      most_recent_game = Game.where(status: [:final, :in_progress]).order("date DESC").first

      if most_recent_game != nil
        @date = most_recent_game.date
      end
    end

    @games = Game.where(:date => @date).compact

    @game_infos = []

    @games.each do |g|
      @game_infos.push GameInfo.new(g.id)
    end

    @latest_game = @games.max_by{|g| g.updated_at}

    Rails.logger.debug "Latest game: #{@latest_game.updated_at}"
    fresh_when(@latest_game, public: true)
  end

  def index

  end
end

rails rack cache - how to manually clear all the cache

I cached a page via rack-cache, and now the browser never displays the updated page cuz its showing the cached version.

how do I clear all the cache? (Note: I deleted all the cached pages from the rails_app_root/public directory). I even tried disabling rack-cache but it still shows the cached version of the page.

any ideas?

Problematic caching with rack cache

I am trying to cache a response from a server locally per example shown here.

#!/usr/bin/env ruby

require 'restclient/components'
require 'rack/cache'
RestClient.enable Rack::Cache,
                  :metastore => 'file:/tmp/cache/meta',
                  :entitystore => 'file:/tmp/cache/body',
                  :verbose => true
RestClient.get 'http://www.google.com/intl/en/policies/privacy/?fg=1'

Somehow I always get miss as a response and the folder is empty. I guess caching doesn't work for me. What am I doing wrong?

cache: [GET /intl/en/policies/privacy/?fg=1] miss

Update

The repsponse header contains:

Cache-Control: must-revalidate, private, max-age=0 

So I guess my question is how to intercept a response before it is processed and modify response header fields?

Is my Rack::Cache misconfigured?

I'm caching the most visited page on my Rails app using HTTP caching and etags.

This is reliant on Rack::Cache, and I recently discovered I'm not explicitly setting the storage configuration for Rack::Cache anywhere in my config files or initializers (specifically, I don't have the config.action_dispatch.rack_cache bit of code anywhere, as discussed in this Heroku guide).

My implementation of HTTP caching appears to be working i.e. if a page is not modified, a logged out user gets the pages back with a 304 Not Modified response.

I fired up a Rails console from my server (in development, staging, and production), ran MyApp::Application.config.action_dispatch.rack_cache, and this is what it came back with:

=> {:metastore=>"rails:/", :entitystore=>"rails:/", :verbose=>false}

I've only ever encountered memcache, file:/, or heap:/ as options for metastore and entitystore. Never rails:/ - there's no mention of it here.

Is my Rack::Cache misconfigured? What does rails:/ mean?

Poor performance with Rack::Cache

I have Rack::Cache set up as the very top (last) middleware in my stack. My app is hosted on heroku cedar, so Rack::Cache is responsible for static assets as well.

It's backed by the heroku memcached product, and configured like this:

config.action_dispatch.rack_cache = {
  :metastore    => Dalli::Client.new,
  :entitystore  => 'file:tmp/cache/rack/body',
  :allow_reload => false
}
config.static_cache_control = "public, max-age=2592000"

It's running on thin.

I'm testing performance on an image file, using ab.

ab -n100 -c10 https://example.com/foo.jpg

Looking in my logs, all the requests are a cache hit ("fresh"). But the performance is poor, only around 6 requests/sec. I know that Rack::Cache isn't going to perform as well as a dedicated http proxy, but I certainly expected higher throughput than this.

update I don't how the heck I didn't think of this before, but my benchmark is indeed maxing out my internet connection. If I do the same test with robots.txt instead of a jpg, I get 20 requests/sec (and still max out the downstream).

Rack-Cache: "stale, valid, store"

I use in my rails app a page caching solution along with an expire_at statement.

My page is good for 1 hour and then needs to be redone.

For some reason the first hour when my rails server start its served "fresh" and then when it becomes style EVERY single request afterwards is served "stale, valid, store".... the store meaning it goes to my app server and tries to get a new copy.

I am on Heroku, using unicorn and rack-cache with memcached for the metastore and a file backed caching for the entity store.

Any ideas why that would happen?

on my local comp/dev machine it doesn't exhibit the same symptoms. When the page goes stale, the first request is served "stale, valid, store" and the subsequent ones get served "fresh" until it stales again when an hour passes.

Difference between Rack::Cache and page caching

We are currently updating our sites at work, and I am responsible for choosing/designing our caching strategy.

Our sites are all article based magazine sites, however some of them have a user system for restricted articles that needs subscriptions.

We have used page caching so far (and stored the pages in memcached) with a little bit of javascript. However Im thinking that Rack::Cache or maybe Varnish is a better solution now. As far as i can see, does it work almost the same way performance wise:

• Page caching, caches the full page in memcached and this cache will be served directly from memcached by nginx on future request.
• Rack::Cache, also caches the full page in memcached, owever the cached version is served by the webserver instead of nginx. Rack::Cache uses HTTP-caching headers which means that visitors also will store a local cache in the browsers. Furthermore would it be easy to replace with Varnish, that also uses HTTP-caching headers.

Im i right so far, and does anyone else have some comments on the differences or the performance of the two strategies? It's also possible too use both, but i can see any advantages with this approach as they will cache the same types of pages.

Rails 3.2 Rack::Cache HTTP Headers and Action Caching

Good afternoon,

I've run into some issues trying to combine HTTP caching with Rack::Cache and action caching (on my Heroku-hosted app).

Using them individually, it seems to be working. With action caching enabled, the page loading is snappy, and the log would suggest it is caching. With HTTP caching in the controllers (eTag, last_modified and fresh_when) the proper headers appear to be set.

However, when I try to combine the two, it appears to be action caching, but the HTTP headers are always max_age: 0, must_revalidate. Why is this? Am I doing something wrong?

For example, here's the code in my "home" action:

class StaticPagesController < ApplicationController
  layout 'public'

  caches_action :about, :contact, ......, :home, .....

  ......

  def home
    last_modified = File.mtime("#{Rails.root}/app/views/static_pages/home.html.haml")
    fresh_when last_modified: last_modified , public: true, etag: last_modified
    expires_in 10.seconds, :public => true       
  end

For all intents and purposes, this should have a public cache-control tag with max-age 10 no?

$ curl -I http://myapp-staging.herokuapp.com/

HTTP/1.1 200 OK
Cache-Control: max-age=0, private, must-revalidate
Content-Type: text/html; charset=utf-8
Date: Thu, 24 May 2012 06:50:45 GMT
Etag: "997dacac05aa4c73f5a6861c9f5a9db0"
Status: 200 OK
Vary: Accept-Encoding
X-Rack-Cache: stale, invalid
X-Request-Id: 078d86423f234da1ac41b418825618c2
X-Runtime: 0.005902
X-Ua-Compatible: IE=Edge,chrome=1
Connection: keep-alive

Config Info:

# Use a different cache store in production
config.cache_store = :dalli_store

config.action_dispatch.rack_cache = {
  :verbose      => true,
  :metastore => "memcached://#{ENV['MEMCACHE_SERVERS']}",
  :entitystore => "memcached://#{ENV['MEMCACHE_SERVERS']}"#,
}

In my mind, you should be able to use action caching as well as a reverse proxy correct? I know that they do fairly similar things (if the page changes, both the proxy and the action cache will be invalid and need to be regenerated), but I feel I should be able to have both in there. Or should I get rid of one?

UPDATE

Thanks for the answer below! It seems to work. But to avoid having to write set_XXX_cache_header methods for every controller action, do you see any reason why this wouldn't work?

before_filter :set_http_cache_headers

.....

def set_http_cache_headers
  expires_in 10.seconds, :public => true
  last_modified = File.mtime("#{Rails.root}/app/views/static_pages/#{params[:action]}.html.haml")
  fresh_when last_modified: last_modified , public: true, etag: last_modified
end

Could not find rack-cache-1.0.3 in any of the sources

I have created a very basic Rails 3.1 app, deployed to a box that runs Ruby 1.8.7 (P334) (I can't easily go to 1.9.2. there unfortunately).

After deploying and running 'bundle install' I tried to run a console:

bundle exec rails console

And I get:

Could not find rack-cache-1.0.3 in any of the sources

and the console does not come up.

It seems that this particular version of rack-cache is listed as a dependency by ActionPack 3.1.0.

Can someone explain to me what I need to do to resolve this, i.e. get bundler to attach this version of rack-cache to the project?

Also I read that bundler stores the project-specific GEMs 'somewhere else', i.e. no longer in the global Ruby GEM path. Is there a default location for this project specific place ?

Oh and I also keep getting heaps of 'invalid gemspec' warnings with Rails 3.1, i.e.:

Invalid gemspec in [/usr/local/lib/ruby/gems/1.8/specifications/rack-cache-1.0.3.gemspec]: invalid date format in specification: "2011-08-27 00:00:00.000000000Z"