EzDevInfo.com

ElasticSearch RoutingMissingException

I'm trying to populate objects having parent-child relation, but get an error:

[Elastica\Exception\ResponseException] RoutingMissingException[routing is required for [myindex]/[comment]/[12345]]

Excerpt from type conf:

article:
    _source:
        enabled: false
    mappings:
        ...

comment:
    _source:
        enabled: false
    _parent:
        type: article
        property: article_id
        identifier: id
    _routing:
        required: true
        path: article_id
    mappings:
        article:
            type: long
            index: not_analyzed
        ...

Cant understand what I am missing here....

I'm using Symfony2.3, FOSElasticaBundle 3.0, ElasticSearch 1.2.2

elastica search how to convert Elastica\Result to actual doctrine object

I am using FOSElasticaBundle with symfony2 and doctrine 2.

I have trouble understanding how to retrieve actual doctrine objets from a search result. I am under the impression that it is the default behaviour but I get this kind of result :

object(Elastica\Result)[1239]
  protected '_hit' => 
    array (size=5)
      '_index' => string 'foodmeup' (length=8)
      '_type' => string 'recipes' (length=7)
      '_id' => string '2' (length=1)
      '_score' => float 2.2963967
      '_source' => 
        array (size=5)
          'name' => string 'Bavaroise vanille' (length=17)
          'nickName' => string 'Bavaroise vanille' (length=17)
          'content' => null
          'userRecipes' => 
            array (size=1)
              ...
          'tags' => 
            array (size=0)

Here is my FOSElasticaBundle configuration:

#Elastic Search
fos_elastica:
    default_manager: orm
    clients:
        default: { host: localhost, port: 9200 }
    indexes:
        search:
            client: default
            index_name: foodmeup
            types:
                recipes:
                    mappings:
                        name: { type: string, boost: 5}
                        nickName: { type: string }
                        content: { type: string }
                        userRecipes:
                            type: "nested"
                            properties:
                                name: { type: string }
                                content: { type: string }
                        tags:
                            type: "nested"
                            boost: 5
                            properties:
                                name: { type: string }
                    persistence:
                        driver: orm
                        model: AppBundle\Entity\FoodAnalytics\Recipe
                        repository: AppBundle\Repository\FoodAnalytics\RecipeRepository
                        provider: ~
                        finder: ~
                        listener: ~ # by default, listens to "insert", "update" and "delete"

And the code in my controller :

public function searchAction(Request $request)
{
    $search = $request->query->get('search');
    $finder = $this->get('fos_elastica.index.search.recipes');
    $results = $finder->search($search)->getResults();

    return array(
        'search' => $search,
        'results' => $results
    );
}

I understood I could use a custom repository method to get the objects, but before I reach that point, what is the default way to get objects ? (Here I want a Recipe Object, an instance of my model).

Thanks a lot !

Sorting in FOSElasticaBundle

I use FOSElasticaBundle in my project to search on my Player entity. As i only want to search entities with the property isactive on a value 1, i followed the documentation on "Filtering Results and Executing a Default Query": FriendsOfSymfony/FOSElasticaBundle/README.md

$query = new \Elastica\Query\QueryString($searchterm);
$term = new \Elastica\Filter\Term(array('isactive' => true));

$filteredQuery = new \Elastica\Query\Filtered($query, $term);
$players = $this->get('fos_elastica.finder.xxx.player')->find($filteredQuery);

The configuration of my bundle looks like following:

fos_elastica:
clients:
    default: { host: localhost, port: 9200 }
serializer:
    callback_class: FOS\ElasticaBundle\Serializer\Callback
    serializer: serializer
indexes:
    xxx:
        client: default
        types:
            player:
                mappings:
                    firstname: { boost: 3 }
                    lastname: { boost: 3 }
                serializer:
                    groups: [elastica, Default]
                persistence:
                    driver: orm
                    model: xxx\FrontendBundle\Entity\Player
                    listener: ~
                    provider: ~
                    finder: ~

Now i want to do some sorting and cut back the result with limit and offset. How can i achieve this?

I found a solution like

$finalQuery = new \Elastica\Query($boolQuery);
$finalQuery->setSort(array('price' => array('order' => 'asc')));

But i dont have an Elastica\Query object and the AbstractQuery didn't support this method. Same with

$elasticaQuery->addSort($sort);

What to do? Where to read?? ://

(in addition, if we are here already: what does {boost: 3} really do exactly?)

FOSElastica Bundle : retrieving highlights for results

I'm trying to have highlights returned with a global index search with the FOSElastica Bundle.

I have a global index finder in my configuration (yml file):

fos_elastica:
  clients:
    default: { host: %elastic_host%, port: %elastic_port% }
  indexes:
    myIndex:
      client: default
      finder: ~
      types: 
       # different types here

and I use it as per the doc (here) :

$finder = $this->container->get('fos_elastica.finder.myIndex');

// Returns a mixed array of any objects mapped
$results = $finder->find('whatever');

That works perfectly and returns the expected results. Now I would like to highlight the words found in the results using for instance the fast vector highlighter of ES. But I did not find any example or any documentation to do so.

I guess I need to define a more proper \Query object with something like :

$query = new \Elastica\Query();
$query->setHighlights(array("whatever"));
$query->setTerm("whatever");

$results = $finder->find($query);

But I cannot find any information. Any hint ?

Thanks a lot !!

Symfony 2.4 execute Command from Controller

I want to execute the command fos:elastica:populate from my controller.

I tried that code but it doesn't work, i get error = 1 the var_dump show ""

$command = 'fos:elastica:populate';
$app = new Application($this->get('kernel'));
$app->setAutoExit(false);
$input = new StringInput($command);
$output = new ConsoleOutput;
$error = $app->run($input, $output);
var_dump($error);
var_dump(stream_get_contents($output->getStream());

Any ideas ?

I try a different code.....

    $command = $this->get('FosElasticaPopulateService');
    $input = new StringInput('');

    $output = new ConsoleOutput();
    ladybug_dump($input);
    // Run the command
    $retval = $command->run($input, $output);

    if(!$retval)
    {
        echo "Command executed successfully!\n";
    }
    else
    {
        echo "Command was not successful.\n";
    }
    var_dump(stream_get_contents($output->getStream()));

It say: 'The "no-interaction" option does not exist.' at Input ->getOption ('no-interaction') in the PopulateCommand.

if i change my code with:
$input = new StringInput('--no-interaction');

It say: 'The "--no-interaction" option does not exist.' at
'ArgvInput ->addLongOption ('no-interaction', null) '

FOS Elastica -- getting string representation of query

I am unit-testing a repository that uses FOS Elastica and I was wondering if anyone knows how to get the string version of a query, rather than in array form. Here is my repository method:

    /**
     * Creates query object by either first or last name, with given parameters
     *
     * @param $name
     *
     * @param array $params
     *
     * @return Query
     */
    public function findByFirstOrLast($name, array $params)
    {
        $queryString = new QueryString();
        $queryString->setQuery($name);
        $queryString->setFields(array('firstName', 'lastName'));


        $query = new Query();
        $query->setQuery($queryString);
        $query->setSort(array($params['sort'] => array('order' => $params['direction'])));

        return $query;
    }

Assuming $name = 'foo'; (and that I am sorting on id), I believe the corresponding FOS Elastica query should be

{
    "query":
    {
        "query_string":
            {
                "query":
                    "foo",
                    "fields":["firstName","lastName"]
            }
    },
    "sort":
    {
      "id":
          {
              "order":"asc"
          }
    }
}

Does anyone know how to get this json-string representation of the query? It doesn't necessarily have to be in this pretty format either, it can be a one-line string.

Order by timeUpdated, if exists, otherwise timeCreated

I have 2 fields in an elastic type, namely timeCreated and timeUpdated. Now my business case is that we should always order by timeUpdated, unless it's null (or non-existent), which then obviously leaves us with timeCreated, which is always set.

Now by my understanding of how sorting in elasticsearch works, the following won't work because it'll sort by timeUpdated AND timeCreated together, and not in the way I'm looking for.

"query": {
    "match_all": {}
},
"sort": {
    "timeUpdated": {"order": "desc", "missing": "_last"},
    "timeCreated": {"order": "desc"}
}

I'm not really sure whether this is possible or not, but elastic is awesome, so I'm hoping it's a simple thing that I missed in the docs.

I'm using the latest version (1.6 I believe) with PHP using Elastica (though that makes zero difference to the original question).

EDIT

As an example, if I have 3 documents:

1 - {"timeCreated": "2015-07-08T08:00:00Z"},
2 - {"timeCreated": "2015-05-08T10:00:00Z", "timeUpdated": "2015-07-09T08:00:00"}
3 - {"timeCreated": "2015-07-10T08:00:00Z"}

The order it should display in must be 3, 2, 1. What I'm getting is 2, 3, 1.

Sync between Doctrine2 and FOSElasticaBundle is not realtime?

I have url /posts to list latest 10 active posts. When I edit the last post (1st of those 10), the change seem not sync to elasticsearch at realtime.

Here is my code:

    $post = $this->findPostBy....;

    /** @var EntityManager $entityManager */
    $entityManager = $this->getDoctrine()->getManager();
    $post->setStatus(PostModel::STATUS_DELETED);
    $entityManager->persist($post);
    $entityManager->flush();

    return $this->redirect('/posts'); 

After above code is executed, I will be redirected to /posts but the last post which has just been edited and has its status changed to deleted still get listed as latest 10 active posts (which is definitely wrong). It only disappears when I refresh the page (/posts).

Code for /posts:

    /** @var TransformedFinder $finder */
    $finder = $this->container->get('fos_elastica.finder.index_name.post');
    $query = new Query();
    $filter = new Term(array('status' => PostModel::STATUS_ACTIVE));
    $query->setFilter($filter);
    $query->addSort(array('tttAt' => array('order' => 'desc')));
    $posts = $finder->find($query);

My mapping:

        types:
            post:
                mappings:
                    title:
                        type: string
                        analyzer: post_analyzer
                    description:
                        type: string
                        analyzer: post_analyzer
                    status:
                        type: integer
                    tttAt :
                        type : date
                    createdAt :
                        type : date
                persistence:
                    driver: orm
                    model: MyBundle\Entity\Post
                    finder: ~
                    provider: ~
                    listener: ~

Can someone explain me why and how to solve that problem?

Updated: If I use xdebug to delay the redirection in about 1 or 2 seconds, the post will not be listed in /posts any more. So I guest that there was not enough time for ElasticSearch to re-index. Any ideas or solutions?

FOSElasticaBundle perform nested queries

I'm learning to use FOSElasticaBundle, but I can not run a query where there are nested fields.

The purpose is this: to look for a query_string in multiple fields, including fields nested.

My setup is:

structure:
    mappings:
        name: { boost: 9, type: string }
        locality:
            type: "nested"
            properties:
                locality: { boost: 8, type: string }

For now I run this code:

$mainQuery = new \Elastica\Query\Bool();
$searchQuery = new \Elastica\Query\QueryString();
$searchQuery->setParam('query', $query);
$searchQuery->setParam('fields', array(
    'name'
));

$searchNested = new \Elastica\Query\QueryString();
$searchNested->setParam('query', $query);
$searchNested->setParam('fields', array(
    'locality'
));
$nestedQuery = new \Elastica\Query\Nested();
$nestedQuery->setPath('locality');
$nestedQuery->setQuery($searchNested);

$mainQuery->addShould($searchQuery);
$mainQuery->addShould($nestedQuery);

$results = $index->search($mainQuery)->getResults();

But returns no result!

How do I create queries with nested fields?

FOSElasticaBundle issue running populate command in cron job

I built a cron job in Linux to run the following command every 5 minuets to update my elasticsearch index

php app/console fos:elastica:populate --no-reset --no-debug

And I have a nested object mapping for documents attachments as below:

...
types:
   documents:
      mappings:
         Title: ~
         Description: ~
         Attachments:
            type: "object"
               properties:
                  File_Name:
                  content:
                     type: attachment 
      persistence:
         driver: orm
         model: Acme\AcmeBundle\Entity\Documents
         provider: ~
         listener: ~
         finder: ~

It works properly to index my documents in DB and has no issue, except for the attachments. It does not index the attachments when the cron job runs, but if I run the populate command manually it will index my attachments. It's really weird and mixed me up.

One more thing which may help to find a solution: I built same schedule task in windows machine to run the command and it works without any issue

Appreciate it if you could help

Populating FosElasticaBundle running out of php memory, possible memory leak?

I've installed FOSElasticaBundle and have it working with a cross section of my data.

My problem arises in that I have about 14m rows that I need to use to build an index. I ran the populate command and after about 6 hours yesterday it errored out at 10.8% with a memory error:

PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 52277 bytes) in /var/www/html/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php on line 111

As you can see I've set my php memory limit to 2G which should be quite excessive.

The last line before the error looked like

Populating index/entity, 10.8% (1315300/12186320), 36 objects/s (RAM : current=2045Mo peak=2047Mo)

And the current and peak were ticking up with every line, starting around 30mb.

My assumption here is that there is some sort of memory leak? Surely php's memory shouldn't be exhausted by this process. I've also tried the command with some extra parameters

app/console fos:elastica:populate --no-debug --no-reset --env=prod

but as I watch it running the current memory is still ticking up.

Any thoughts on what might be going on here and what I can do to debug it? I found this discussion which sounds like my problem, but doesn't really present a good solution: https://github.com/FriendsOfSymfony/FOSElasticaBundle/issues/82. I'm using doctrine and the default provider.

Thank you-

Symfony FOSElastica ResponseException with Elasticsearch

i installed FOSElastica 3.0 with Composer to my Symfony Project. This is my Config:

fos_elastica:
    clients:
        default: { host: localhost, port: 9200 }
    indexes:
        search:
          types:
            keyword:
                mappings:
                    value:
                      type: string
                persistence:
                    driver: orm
                    model: Anklick\UserPagesBundle\Entity\Keyword
                    provider: ~
                    listener:
                        immediate: ~
                    finder: ~

Pretty simple huh?

Now when i do php app/console fos:elastica:populate

It says me:

Resetting search

[Elastica\Exception\ResponseException]
ClassCastException[java.util.ArrayList cannot be cast to java.util.Map]

Whats the Problem?

EDIT: Same Problem with fos:elastica:reset

Can ElasticSearch _suggest query return multiple fields?

I have the following mapping for an index in which I'm doing a suggest/autocomplete lookup and response.

"mappings": {
    "listing": {
        "_source": {
            "enabled": false
        },
        "dynamic": false,
        "properties": {
            "_all": {
                "enabled": false
            },
            "listingTitle": {
                "type": "completion",
                "index_analyzer": "str_index_analyzer",
                "search_analyzer": "str_search_analyzer"
            },
            "address": {
                "dynamic": false,
                "properties": {
                    "city": {
                        "type": "completion",
                        "index_analyzer": "str_index_analyzer",
                        "search_analyzer": "str_search_analyzer"
                    },
                    "stateOrProvince": {
                        "type": "completion",
                        "index_analyzer": "str_index_analyzer",
                        "search_analyzer": "str_search_analyzer"
                    }
                }
            }
        }
    }
}

This is the endpoint and data for the request: site.dev:9200/listingsuggest/_suggest

{
  "result": {
    "text": "Minn",
    "completion": {
      "field": "address.city"
    }
  }
}

So this works for finding documents that match cities starting with Minn. The issue I'm having is I also want to return the stateOrProvince field value of each document matching "Minn". For example my result set comes back with the following matched words:

Minneapolis
Minnetonka

What I need it to do is return this:

Minneapolis, MN
Minnetonka, MN

The full response currently:

{
    _shards: {
        total: 1
        successful: 1
        failed: 0
    }
    result: [{
        text: Minn
        offset: 0
        length: 4
        options: [{
            text: Minnetonka
            score: 1
        } {
            text: Minneapolis
            score: 1
        }]
    }]
}

Desired full response if possible:

{
    _shards: {
        total: 1
        successful: 1
        failed: 0
    }
    result: [{
        text: Minn
        offset: 0
        length: 4
        options: [{
            text: Minnetonka, MN
            score: 1
        } {
            text: Minneapolis, MN
            score: 1
        }]
    }]
}

Is this, or some variation of that response, possible?

Elastica Results : how to get doctrine object when searching entire index

Continuing my discovery of FOSElasticaBundle.

Now I know how to retrieve actual doctrine entities :

    $finder = $this->get('fos_elastica.finder.website.recipe');
    $results = $finder->find($search);

as opposed to Elastica\Result instances:

    $finder = $this->get('fos_elastica.index.website.recipe');
    $results = $finder->search($search);

It seems however I can call

$finder = $this->get('fos_elastica.index.website');

but I can't call

$finder = $this->get('fos_elastica.finder.website');

So how do I retrieve doctrine object results for a search on the entire index ?

ElasticSearch does not find by UTF8 characters

I'm using ElasticSearch along with Symfony2 and FOSElasticaBundle. Then I try to find anything with ASCII characters only, it works like a charm. Also, I'm using JSON array for data, and it converts unicode characters into something like \u00f8C. Could this affect my problem? My code for search:

function __construct(Request $request, TransformedFinder $artist_finder, TransformedFinder $picture_finder, Router $router)
{
    $this->request = $request;
    $this->a_finder = $artist_finder;
    $this->p_finder = $picture_finder;
    $this->router = $router;
}

/**
 * @param $query
 * @return array
 */
public function search($query)
{
    $locale = $this->request->getLocale();

    $artist_results = $this->a_finder->find($query.'*', self::SEARCH_LIMIT);
    $gallery_results = $this->p_finder->find($query.'*', self::SEARCH_LIMIT);

    $results = array(
        'artist' => array(),
        'gallery' => array()
    );

    foreach($artist_results as $a)
    {
        $results['artist'][] = array(
            'url' => $this->router->generate('artist_page', ['slug' => $a->getSlug()]),
            'name' => $a->getInfo()[$locale]['name']. ' ' .$a->getInfo()[$locale]['surname']
        );
    }

    foreach($gallery_results as $a)
    {
        $results['gallery'][] = array(
            'url' => $this->router->generate('picture_page', ['slug' => $a->getSlug()]),
            'name' => $a->getInfo()[$locale]['title']
        );
    }

    return $results;
}

Also, does ES have ability to see no difference between lithuanian characters, for example "Š" and "S", "Ė" and "E" etc? I mean, I would like to get same results with query like "žąsis" and "zasis".