elasticsearch-rails
Elasticsearch integrations for ActiveModel/Record and Ruby on Rails
I see in the elasticsearch docs you can fetch a document by its ID. Is there any equivalent in elasticsearch rails? I'm feeding by API with "as_indexed_json" and it's a somewhat expensive query, I'd like to return ths JSON straight out of elasticsearch in my API.
Source: (StackOverflow)
I am new to ElasticSearch, but need to use it to return a list of products. Please do not include answers or links to old answers which reference the deprecated tire gem.
gemfile
ruby '2.2.0'
gem 'rails', '4.0.3'
gem 'elasticsearch-model', '~> 0.1.6'
gem 'elasticsearch-rails', '~> 0.1.6'
I have a couple models with relationships. I included the relationships below.
Models and Relationships
product.rb
include Searchable
belongs_to :family
belongs_to :collection
has_many :benefits_products
has_many :benefits, :through => :benefits_products
def as_indexed_json(options={})
as_json(
include: {:benefits => { :only => [ :id, :name ] },
:categories => { :only => [ :id, :name ] } }
)
end
collection.rb
include Searchable
has_many :products
def as_indexed_json(options={})
as_json(
include: [:products]
)
end
family.rb
include Searchable
has_many :products
def as_indexed_json(options={})
as_json(
include: [:products]
)
end
benefit.rb
include Searchable
has_many :benefits_products
has_many :products, :through => :benefits_products
def as_indexed_json(options={})
as_json(
include: [:products]
)
end
Serachable.rb
Is just a concern that includes Elastic search and callbacks in all models
module Searchable
extend ActiveSupport::Concern
included do
include Elasticsearch::Model
include Elasticsearch::Model::Callbacks
settings index: { number_of_shards: 1, number_of_replicas: 0 } do
mapping do
indexes :id, type: 'long'
indexes :name, type: 'string'
indexes :family_id, type: 'long'
indexes :collection_id, type: 'long'
indexes :created_at, type: 'date'
indexes :updated_at, type: 'date'
indexes :benefits, type: 'nested' do
indexes :id, type: 'long'
indexes :name, type: 'string'
end
indexes :categories, type: 'nested' do
indexes :id, type: 'long'
indexes :name, type: 'string'
end
end
end
def self.search(options={})
__set_filters = lambda do |key, f|
@search_definition[:filter][:and] ||= []
@search_definition[:filter][:and] |= [f]
end
@search_definition = {
query: {
filtered: {
query: {
match_all: {}
}
}
},
filter: {}
}
if options[:benefits]
f = { term: { "benefits.id": options[:benefits] } }
__set_filters.(:collection_id, f)
__set_filters.(:family_id, f)
__set_filters.(:categories, f)
end
def as_indexed_json(options={})
as_json(
include: {:benefits => { :only => [ :id, :name ] },
:categories => { :only => [ :id, :name ] } }
)
end
if options[:categories]
...
end
if options[:collection_id]
...
end
if options[:family_id]
...
end
__elasticsearch__.search(@search_definition)
end
end
end
ElasticSearch
I breakdown dash separated slugs into the various families, collections and benefits. I am able to search for products with a specific family or collection and return correct results. I am also able to return results for one benefit, but they don't appear to be accurate. Also searching multiple benefits yields strange results. I would like the "AND" combination of all fields search, but my result doesnt seem to be the result of "AND" or "OR". So this is confusing me as well.
What do I pass to the Product.search method to yield desired results?
Thanks for any help you can provide!
Edit
I have now verified that benefits are indexed on the products. I used curl -XGET 'http://127.0.0.1:9200/products/_search?pretty=1' which produced a json response that looked like this:
{
"id":4,
"name":"product name"
"family_id":16
"collection_id":6
"created_at":"2015-04-13T12:49:42.000Z"
"updated_at":"2015-04-13T12:49:42.000Z"
"benefits":[
{"id":2,"name":"my benefit 2"},
{"id":6,"name":"my benefit 6"},
{"id":7,"name":"my benefit 7"}
],
"categories":[
{"id":2,"name":"category 2"}
]}
},
{...}
Now I just need to figure out how to search for the product with benefits 2,6, AND 7 in ElasticSearch if I wanted the above example product. I am specifically looking for the syntax to submit to the elasticsearch #search method to acquire the results of a nested "AND" query, nested query setup/mappings (to make sure I have not missed anything, and any other relevant info you can think of you troubleshoot this.
Upated
The Searchable concern has been updated to reflect the answer received. I translated the mapping json object to fit in the elasticsearch-model syntax. My remaining confusion occurs when I attempt to translate the query in a similar fashion.
Second Update
I am basic most of my searchable.rb concern off the elasticsearch-rails example app. I have updated searchable.rb to reflect this code, and while I am getting results, they are not the result of an "AND" execution. When I apply two benefits, I get the results from all products that have either benefit.
Source: (StackOverflow)
Here's the query I'd like to get working with elasticsearch-rails. (The query works in Sense). My goal is to return all the buckets for items that have a person whose name begins with the letter B. My first stumbling block is that I can't figure out how to specify that the search_type should be set to count.
GET _search?search_type=count
{
"query": {
"prefix": {
"person": "B"
}
},
"aggs" : {
"facets" : {
"terms" : {
"field" : "person",
"size" : 0,
"order" : { "_term" : "asc" }
}
}
}
}
Source: (StackOverflow)
System:
- Rails 4
- Ruby 2
- Elasticsearch 1.6.0
I'm using Elasticsearch to filter records and calculate statistics for my Active Record models. I'd like my Elasticsearch indices to mirror my Postgres database so that existing records are imported into my indices, new records are indexed as they are created, etc.
I have a concern that two of my models include like so:
# app/models/concerns/foo.rb
module Foo
extend ActiveSupport::Concern
included do
include Elasticsearch::Model
include Elasticsearch::Model::Callbacks
self.import force: true
end
class_methods do
def statistics(filter = {})
# ES extended_stats aggregations
end
end
end
# app/models/bar.rb
# Same for another class
class Bar < ActiveRecord::Base
include Foo
end
However, when I open the Rails console and call Bar.statistics, I get nilin every field. If I then execute Bar.import force: true and then Bar.statistics, I get correct, non-nil values in my extended stats aggregation.
Something else that might be of note: When I opened foo.rb in my Rails console using Pry and then exited, I ran into a Cannot define multiple included blocks for a Concern error (though loading the application works fine).
Am I missing something to make my records auto-import into ES?
Source: (StackOverflow)
I wondering what would be the best practice to perform next task.
I have a search results to display from index action. Every individual record displays in the pop up through show action.
What I would love to do is to execute pop up if there is only one record found.
Here what I already tried.
def index
@companies = Company.search(params[:query]).results
@count = @companies.total
if @count == 1
return
render company_path
end
end
Seems like return, redirect_to or render aren't play well in one action.
Any other thought of doing it?
UPDATE added show action
def show
sleep 1/2
client = Elasticsearch::Client.new host:'127.0.0.1:9200', log: true
response = client.search index: 'companies', body: {query: { match: {_id: params[:id]} } }
@company = response['hits']['hits'][0]['_source']
respond_to do |format|
format.html # show.html.erb
format.js # show.js.erb
format.json { render json: @company }
end
# more code
end
Source: (StackOverflow)
I am using logstash+elasticsearch to index server logs. The Elasticsearch server is running at localhost:9200 with millions of server log docs.
I also have a Rails app running at localhost:3000. I need to connect this rails app to the ES server.
I have read about the "elasticsearch-rails" gem but everywhere i found them using ActiveRecords/ Models. However, i don't think ActiveRecords are required for this. I just need a way to query the ES server index and fetch the documents inside my Rails app.
Is there a way to do this? Can anyone please help me with this situation? please comment if I am not clear with my question.
Thanks in advance.
Source: (StackOverflow)
I'm trying to make a search on a field that contains URL using elastic search Term query. I use elasticsearch-rails the ActiveRecord Persistance Pattern. This is how I try to do it.
total_views = UserAction.search :query=> {
:filtered=> {
:filter=> {
:term=> { action_path:"http://0.0.0.0:3000/tshirt/test" }
}
}
}
It works if there are no '/' or ':' characters. For example when the action_path is just 'tshirt'. The other fields are not analyzed and they work if there are no '/', ':' kinds of characters in the field.
So obviously elastic search tries to analyze it but the problem is they should not be analyzed because mapping is already there.
This my user action class
class UserAction
include Elasticsearch::Persistence::Model
extend Calculations
include Styles
attribute :user_id, Integer
attribute :user_referrer, String, mapping: { index: 'not_analyzed' }
attribute :user_ip, String, mapping: { index: 'not_analyzed' }
attribute :user_country, String, mapping: { index: 'not_analyzed' }
attribute :user_city, String, mapping: { index: 'not_analyzed' }
attribute :user_device, String, mapping: { index: 'not_analyzed' }
attribute :user_agent, String, mapping: { index: 'not_analyzed' }
attribute :user_platform
attribute :user_visitid, Integer
attribute :action_type, String, mapping: { index: 'not_analyzed' }
attribute :action_css, String, mapping: { index: 'not_analyzed' }
attribute :action_text, String, mapping: { index: 'not_analyzed' }
attribute :action_path, String, mapping: { index: 'not_analyzed' }
attribute :share_url, String, mapping: { index: 'not_analyzed' }
attribute :tag
attribute :date
I also tried adding indexes using 'mapping do.." and then "create_index!" but result is the same. Because mapping is there it does create the mapping.
This is my gem file
gem "elasticsearch-model", git: "git://github.com/elasticsearch/elasticsearch-rails.git", require: "elasticsearch/model"
gem "elasticsearch-persistence", git: "git://github.com/elasticsearch/elasticsearch-rails.git", require: "elasticsearch/persistence/model"
gem "elasticsearch-rails"
When I make the search I also see that those fields that are not analyzed.
:reload_on_failure=>false,
:randomize_hosts=>false,
:transport_options=>{}},
@protocol="http",
@reload_after=10000,
@resurrect_after=60,
@serializer=
#<Elasticsearch::Transport::Transport::Serializer::MultiJson:0x007fc4bf9e0e18
@transport=#<Elasticsearch::Transport::Transport::HTTP::Faraday:0x007fc4bf9b35a8 ...>>,
@sniffer=
#<Elasticsearch::Transport::Transport::Sniffer:0x007fc4bf9e0dc8
@timeout=1,
@transport=#<Elasticsearch::Transport::Transport::HTTP::Faraday:0x007fc4bf9b35a8 ...>>,
@tracer=nil>>,
@document_type="user_action",
@index_name="useraction",
@klass=UserAction,
@mapping=
#<Elasticsearch::Model::Indexing::Mappings:0x007fc4bfab18d8
@mapping=
{:created_at=>{:type=>"date"},
:updated_at=>{:type=>"date"},
:user_id=>{:type=>"integer"},
:user_referrer=>{:type=>"string"},
:user_ip=>{:type=>"string"},
:user_country=>{:type=>"string", :index=>"not_analyzed"},
:user_city=>{:type=>"string", :index=>"not_analyzed"},
:user_device=>{:type=>"string", :index=>"not_analyzed"},
:user_agent=>{:type=>"string", :index=>"not_analyzed"},
:user_platform=>{:type=>"string"},
:user_visitid=>{:type=>"integer"},
:action_type=>{:type=>"string", :index=>"not_analyzed"},
:action_css=>{:type=>"string", :index=>"not_analyzed"},
:action_text=>{:type=>"string", :index=>"not_analyzed"},
:action_path=>{:type=>"string", :index=>"not_analyzed"}},
@options={},
@type="user_action">,
@options={:host=>UserAction}>,
@response={"took"=>1, "timed_out"=>false, "_shards"=>{"total"=>4, "successful"=>4, "failed"=>0}, "hits"=>{"total"=>0, "max_score"=>nil, "hits"=>[]}}>
(END)
the initializer file has nothing other than the elastichq connection url.
Data is there in elastichq so I should get the results but can't get any.
user_action 1 AUzH9xKDueQ8OtBQuyQC http://example.org/api/analytics/track
user_actions user_action 1 AUzIAUsvueQ8OtBQuyQg http://0.0.0.0:3000/tshirt/funnel_test2
user_actions user_action 1 AUzH7ay5ueQ8OtBQuyP2 http://example.org/api/analytics/track
user_actions user_action 1 AUzH-HAdueQ8OtBQuyQU http://0.0.0.0:3000/tshirt/test
user_actions user_action 1 AUzIJbCGueQ8OtBQuyQ4 http://example.org/api/analytics/track
user_actions user_action 1 AUzIJbCjueQ8OtBQuyQ5 http://example.org/api/analytics/track
Curl Results from Elastichq
curl -XGET "https://YYYYY:XXXXX@xxxx.qbox.io/user_actions/_mapping"
{
"user_actions": {
"mappings": {
"user_action": {
"properties": {
"action_css": { "type": "string" },
"action_path": { "type": "string" },
"action_text": { "type": "string" },
"action_type": { "type": "string" },
"created_at": { "format": "dateOptionalTime", "type": "date" },
"date": { "type": "string" },
"share_url": { "type": "string" },
"tag": { "type": "string" },
"updated_at": { "format": "dateOptionalTime", "type": "date" },
"user_agent": { "type": "string" },
"user_city": { "type": "string" },
"user_country": { "type": "string" },
"user_device": { "type": "string" },
"user_id": { "type": "long" },
"user_ip": { "type": "string" },
"user_referrer": { "type": "string" },
"user_visitid": { "type": "long" }
}
}
}
}
}
can anybody help me on getting url term search work?
Source: (StackOverflow)
I'm trying to deploy my app and index a model. Everything is working fine until i'm trying to actually get the data from elasticsearch.
My environment is staging but when i perform a search via a rest api i get an error
{
"status": "error",
"messages":
"[404]{\"error\":\"IndexMissingException[[myapp_production_products] missing]\",\"status\":404}"
}
From this:
[myapp_production_products]
i deduced that it's trying to query elasticsearch on wrong environment.
What is causing this?
Source: (StackOverflow)
I'm trying to implement an auto complete using Rails and Elastic Search using the elasticsearch-rails gem.
Say I have the following records:
[{id: 1, name: "John White"},
{id:2, name: "Betty Johnson"}]
Which elastic search method could I use to return both records upon searching "John".
The autocomplete would only return "John White" and it does so without returning id:1.
Source: (StackOverflow)
I have a Piece model with a boolean attribute of published.
I want the search results to only contain Pieces that are plublished: true.
My index action for the PiecesController is:
def index
@list = params[:list]
@sort = params[:sort]
if params[:q]
@pieces = Piece.search(params[:q]).records
else
@pieces = Piece.all_in_category(@list, @sort)
end
end
From searching around it seems that I should overwrite the search method in the Piece controller but I am not sure the correct way of doing this to maintain the current search methods functionality.
What is the best way to filter the elasticsearch results using the elasticsearch-rails gem?
Source: (StackOverflow)
I have an insert-only architecture that I want to index with ElasticSearch, but I'm having trouble figuring out the right way to manage the indexing without having duplicate records.
The database table looks something like this:
id | reference_id | title | created at | approved
-------------------------------------------------------------------------------------------
1 | 1 | Post A | Today at 11:00am | true
2 | 1 | Post A that is changed | Today at 11:30am | true
3 | 1 | Post A with an updated title | Today at 12:00pm | false
3 | 2 | A different post | Today at 12:30am | false
Each row has substantially more information in it, but that's an example of the way the id / reference_id relationship works. Each new post has it's own ID and they are grouped by their reference_id.
The ideal situation would be that only the latest approved version of each post is returned for each reference_id, but that you could search based on the content of any version. However, I would be happy enough in the short term with just returning one copy of each post.
So far, I have looked at using the reference_id as the ElasticSearch _id and updating the document when a new post is created. I've also tried indexing all versions independently and doing the aggregation on the client side. The first approach seems the simplest, but I'm not sure if this can be accomplished using the ElasticSearch versions, nested or parent approach.
So, my question: What is the best way to handle intentionally duplicate items ElasticSearch without showing all duplicates in the results?
Source: (StackOverflow)
I am using elasticsearch-rails and elasticsearch-model gem for searching products in my rails app.
I want to make my search case insensitive and must be independent of pluralization. I researched a lot on google but got a hunch on how to do it using analyzer but not success. So I had to post a new question.
Here is my product model where I want to search to take place
class Product < ActiveRecord::Base
include Elasticsearch::Model
include Elasticsearch::Model::Callbacks
belongs_to :category
belongs_to :sub_category
has_many :variations
settings index: { number_of_shards: 1 } do
mappings dynamic: 'false' do
indexes :name, analyzer: 'snowball'
indexes :description, analyzer: 'snowball'
indexes :category, analyzer: 'snowball'
end
end
def as_indexed_json(options={})
as_json(only: [:name, :description, :brand, :color, :make, :style, :gender],
include: { category: { only: :name}, sub_category: {only: :name}}
)
end
def self.search(query)
__elasticsearch__.search({
query: { query_string: {
query: query,
default_operator: "AND"
}},
})
end
end
- My questions are how do I do search irrespective of pluralization?
- tshirt, T-shirt, Tshirts should all match.
Above all I never found any explanation of the following block given below
settings index: { number_of_shards: 1 } do
mappings dynamic: 'false' do
indexes :name, analyzer: 'snowball', index_options: 'offsets'
indexes :description, analyzer: 'snowball'
indexes :category, analyzer: 'snowball'
end
end
what is the exact purpose of this block for searching.
what does mappings dynamic: 'false' means?
what does indexes :name, analyzer: 'snowball', index_options: 'offsets'
means?
What are the other options that could be added in this block with a little bit of explanation would be of great help for me.
Any links for further research is also helpful.
Thanks
Source: (StackOverflow)
I am using logstash+elasticearch to index server logs. The ES server is running at localhost:9200 with millions od server log docs. I also have a rails app running at localhost:3000. I need to connect this rails app to ES server.
I found elasticsearch-rails gem with examples explained using Rails Model. In my case the scenario is different. I just need to query the ES server index and fetch documents inside my rails app.
Is there a way to do this using the elasticsearch-rails gem? If yes, then how would I do it?
I know my question might sound silly but I really need to learn this ASAP. I am stressing on the elasticsearch-rails gem because it supports will_pagination and thus this will make my job easy inside the rails app. Is it even possible using the elasticsearch-rails gem?
please let me know if I am not clear with my situation.
Thanks in advance.
Source: (StackOverflow)
I have an elasticsearch index 'events' - within that index there's a type 'event'.
event objects have a 'venue' which has various properties, including a 'name' - so the simplified structure is:
event {
venue {
name: "foo"
}
}
Now, i'm using elasticsearch-rails - everything works fine for listing the events, searching etc using the query dsl - but what if i want to list all the events at a venue with a particular name?
I'm assuming something like this should be possible:
Event.search "{ 'query': { 'match': { 'venue.name': '#{params[:v]}' }}}
but i get the following error:
Elasticsearch::Transport::Transport::Errors::BadRequest
followed by a substantial stack trace which contains a lot of this sort of thing:
Was expecting one of:\n \"]\" ...\n \"}\" ...\n ];
ParseExceptions suggesting malformed json - but i'm not sure why.
The simple search
Event.search '{"query" : { "match_all" : {} }}'
works fine, so i'm guessing it's just the structure of the query that's wrong.
I've tried switching single/double quotes around, tried following more closely the example on this page:
https://www.elastic.co/guide/en/elasticsearch/guide/current/denormalization.html
all to no avail, wondered if anyone else had encountered this situation and could suggest how to work this in ruby.
Source: (StackOverflow)