phirehose
PHP interface to Twitter Streaming API
I'm trying to search hashtags on twitter using the Phirehose.
I'm using this code:
$sc = new FilterTrackConsumer(OAUTH_TOKEN, OAUTH_SECRET, Phirehose::METHOD_FILTER);
$sc->setTrack(array('selfie'));
$sc->consume();
I'm not getting any results using this code.
What am I doing wrong here?
I want to search all of twitter for specified hashtags.
Hope you can help me out here.
Source: (StackOverflow)
after reading through all of the twitter streaming API and Phirehose PHP documentation i've come across something I have yet to do, collect and process data separately.
The logic behind it, If I understand correctly, is to prevent a log jam at the processing phase that will back up the collecting process. I've seen examples before but they basically write right to a MySQL database right after collection which seems to go against what twitter recommends you do.
What I'd like some advice/help on is, what is the best way to handle this and how. It seems that people recommend writing all the data directly to a text file then parsing/processing it with a separate function. But with this method, I'd assume it could be a memory hog.
Here's the catch, it's all going to be running as a daemon/background process. So does anyone have any experience with solving a problem like this, or more specifically, the twitter phirehose library? Thanks!
Some notes:
*The connection will be through a socket so my guess is that the file will constantly be appended? not sure if anyone has any feedback on that
Source: (StackOverflow)
Currently I'm using Phirehose to track keywords in Twitter streams and want to track users as well.
But when I use setFollow() in my checkFilterPredicates() method, Phirehose stops working after the initial connection.
public function checkFilterPredicates() {
$this->setFollow(array(12));
}
And this is the console output:
Phirehose: Connecting to twitter stream: https://stream.twitter.com/1/statuses/f
ilter.json with params: array ( 'delimited' => 'length', 'follow' => '12',)
Phirehose: Resolved host stream.twitter.com to 199.59.148.138
Phirehose: Connecting to ssl://199.59.148.138, port=443, connectTimeout=5
Phirehose: Connection established to 199.59.148.138
Phirehose: POST /1/statuses/filter.json HTTP/1.0
Phirehose: Host: stream.twitter.com:443
Phirehose: Content-type: application/x-www-form-urlencoded
Phirehose: Content-length: 26
Phirehose: Accept: */*
Phirehose: Authorization: Basic: dHdpenlvbjpUZW1wbzEyMw==
Phirehose: User-Agent: Phirehose/0.2.gitmaster +https://github.com/fennb/phireho
se
Phirehose:
Phirehose: delimited=length&follow=12
Phirehose:
c:\devel\stream>
If I use setTrack() with an array('keyword1', 'keyword2') instead, the same code works.
Am I using setFollow wrong?
Source: (StackOverflow)
I`m writing app that should work with twitter streaming api. For interacting with streaming api I use phirehose lib (https://github.com/fennb/phirehose). It runs great as separate php project. Php file with Phirehose lib is used as background process, that constantly consumes tweets, now I want save it in Db, so I need somehow make available all advantages work with DB of symfony framework in my simple php file that consumes tweets. Is there a way to make this, except put phirehose in vendor folder of symfony (I don't want to do this, cause in manuals that I find it is not easy, rename file and classes of lib, which can cause problems inside lib)?
Source: (StackOverflow)
Default Set Up
$stream->setTrack (array('cookie'));
Trial
$sql = "SELECT DISTINCT text FROM keywords"; $query = mysql_query($sql) or die($sql . ' - ' . mysql_error());
$keys = array();
while ($row = mysql_fetch_array($query)) {
$keys[] = $row[0];
}$stream->setTrack ($keys);
The default set up takes the written word and places it into an array
The trial set up should retrieve a set of keywords from a mysql table as an array and insert them into the function.
It doesn't work. What's wrong?
public function setTrack($trackWords)
{
$trackWords = ($trackWords === NULL) ? array() : $trackWords;
sort($trackWords); // Non-optimal, but necessary
if ($this->trackWords != $trackWords) {
$this->filterChanged = TRUE;
}
$this->trackWords = $trackWords;
}
Source: (StackOverflow)
I am trying to set up a 140dev Twitter server using the framework available on their site. I've followed all of the installation instructions and I've successfully created my database using the schema provided in the library.
When I attempt to run the two scripts get_tweets.php and monitor_tweets.php I get the following error in my CLI
Phirehose: HTTP failure 2 of 20 connecting to stream: HTTP ERROR 401: Authorizat ion Required (<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html ; charset=utf-8"/>\n<title>Error 401 Unauthorized</title></head><body><h2>HTTP E RROR: 401</h2><p>Problem accessing '/1.1/statuses/filter.json'. Reason:<pre> Unauthorized</pre></body></html>). Sleeping for 20 seconds.
Phirehose: Connecting to twitter stream: https://stream.twitter.com/1.1/statuses/filter.json with params: array ( 'track' => 'book,face,facebook,fb',)
Phirehose: Resolved host stream.twitter.com to 199.16.156.110
Phirehose: Connecting to ssl://199.16.156.110, port=443, connectTimeout=5
Phirehose: Connection established to 199.16.156.110
Phirehose: POST /1.1/statuses/filter.json HTTP/1.1
Host: stream.twitter.com:443
Connection: Close
Content-type: application/x-www-form-urlencoded
Content-length: 33
Accept: */*
Authorization: OAuth realm="",oauth_consumer_key="xxxxxxxx",oauth_nonce="xxxxxxxxxx",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1409748040",oauth_version="1.0A",oauth_token="%xxxxxxxx-xxxxxxxxxxxxx",oauth_signature="xxxx%xxxx"
User-Agent: Phirehose/1.0RC +https://github.com/fennb/phirehose
track=book%2Cface%2Cfacebook%2Cfb
I don't know where I am going wrong and would appreciate any input from people who have experience using the 140Dev Twitter Framework.
Source: (StackOverflow)
I want to run Phirehose, a php Twitter streaming API on Amazon EC2 to run continuously. I used the following code
<?php
// TweetStream.php
require_once "OauthPhirehose.php";
require_once "../ajax/common.php";
const table_name = "tweets";
set_time_limit(0);
class Consumer extends OauthPhirehose {
private $cobj; // mysqli object
protected $count = 0;
public function __construct($a, $b, $c) {
parent::__construct($a, $b, $c);
$this->cobj = new mysqli(db_host, db_user, db_pass, db_name);
}
public function enqueueStatus($status) {
$data = json_decode($status);
$cond = !isset($data->delete) && !isset($data->warning);
$cond = $cond && !($data->id == 0 || trim($data->user->name) == "" || trim($data->text) == "" || $data->geo == NULL);
if($cond) {
++$this->count;
echo str_pad($this->count, 12, " ", STR_PAD_LEFT).": Getting Tweet: ".$data->id_str." Inserting... ";
$id = $data->id_str;
$idi = $data->id;
$user = $data->user->name;
$coord = $data->coordinates->coordinates;
$lat = $coord[1];
$lng = $coord[0];
$tweet = $data->text;
$stmt = $this->cobj->prepare("insert into ".table_name." values(?, ?, ?, ?, ?)");
$stmt->bind_param("ssdds", $id, $user, $lat, $lng, $tweet);
$stmt->execute();
if($stmt->errno) echo "skipped: ".$stmt->error;
else echo "done. \n";
$stmt->close();
}
}
}
define("TWITTER_CONSUMER_KEY", "******");
define("TWITTER_CONSUMER_SECRET", "******");
define("OAUTH_TOKEN", "******");
define("OAUTH_SECRET", "********");
$sc = new Consumer(OAUTH_TOKEN, OAUTH_SECRET, Phirehose::METHOD_SAMPLE);
$sc->setLang("en");
$sc->consume();
?>
Once I upload this code to the EC2 instance I open ssh using Putty and execute following commands
$ cd /var/www/html/service
$ nohup php TweetStream.php &
After 5 minutes or so, when I run top -p $(pgrep -d',' php), I get the following
top - 06:38:52 up 1 day, 4:04, 1 user, load average: 64.61, 66.82, 65.65
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s):100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 606528k total, 599552k used, 6976k free, 1172k buffers
Swap: 0k total, 0k used, 0k free, 29660k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11902 ec2-user 20 0 438m 9096 1792 S 0.0 1.5 0:06.18 php
I can gather that it has run for 6 minutes or so and got into sleep state. Can I please know what am I missing?
How to make this process run continuously on Amazon EC2 Instance? Please help me friends.
By the way, The mysqli is connecting to RDS instance running in Amazon.
Source: (StackOverflow)
I am using the 140Dev Framework to connect to the Streaming API in Twitter and I've successfully managed to get it working on my web server to the point that I am pulling in tweets from all over the globe. However I've been trying to filter the results I am getting by limiting them to tweets in english and if the user has their location set. However it seems to be having no effect.
This is the code:
if ($tweet_object->lang <> 'en' &&
$user_object->location == 'ireland'){
continue;
}
Can anyone recommend what I might be doing wrong?
Source: (StackOverflow)
I am trying to create website to show a live twitter feed concerning a particular hash tag, say #baseball or from a particular user @twitter, my filter looks like #baseball OR @twitter
Now I have done implementation using Phirehose but I want to use AJAX instead so that i can contain the live stream in a fixed div on my website. I searched a lot, and guess many people have done this but I am not able to find any reference code or example of how to do it.
PHP code I am using:
require_once('../lib/Phirehose.php');
class FilterTrackConsumer extends Phirehose
{
public function enqueueStatus($status)
{
$data = json_decode($status, true);
if (is_array($data) && isset($data['user']['screen_name'])) {
echo $data['user']['screen_name'] . ': ' . urldecode($data['text']) . "\n";
}
}
}
// Start streaming
$sc = new FilterTrackConsumer('abc', 'abc', Phirehose::METHOD_FILTER);
$sc->setTrack(array('@twitter', '#baseball'));
$sc->consume();
I need something similar with Javascript & PHP. Or how to style the output here.
Help would be appreciated
Source: (StackOverflow)
I have a service where I list Twitter accounts and have a switch to enable/disable a Streaming API connection for them.
Currently I'm planning to have this flow:
- User enables a Twitter account for Streaming
- An email is sent to an admin with cli access on the server, who launches a Phirehose process for that account.
- (A cron job takes a list of Twitter accounts that have Streaming enabled and checks if a process exists for them. If not, it sends an email to the admin who can relaunch it.)
I would like to launch a Phirehose process directly to run in the background when a user wants a Twitter account to be enabled for Streaming. Also I would like to kill the process directly if a user wants to stop Streaming for that account.
There are several risks and concerns with executing the Phirehose process and especially killing it in an automated way.
Any tips on the best way to implement this with maximal automation and minimal admin interference?
Source: (StackOverflow)
I'm using Phirehose to pull tweets using the Twitter Streaming API. I'm wanting to pull geotagged tweets with certain keywords, like so:
$sc->setLocations(array(array(-180,-90,180,90))); //any geotagged tweet
$sc->setTrack($sc->getKeywords());
getKeywords class function looks like:
public function getKeywords()
{
$array = array('hurricane', 'flood', 'tornado','tsunami','earthquake');
return $array;
}
The problem is that setLocation() seems to cause setTrack() to not pull tweets with those keywords. If I remove setLocation() altogether, setTrack() pulls keywords fine. Is there any way I can pull geotagged tweets containing those keywords?
Source: (StackOverflow)
I have run the examples scripts from phirehose through terminal and seen the live/active ptint_r of my test tweets. I've loaded up the ghetto queue files and they execute their logging with resounding success. But I can't seem to find where the data went. What file is it actually writing to? The two example files ghetto-queue-collect.php and ghetto-queue-consume.php lead me to believe it is the /tmp directory relative to where the scripts are executing but I see nothing. Any suggestions?
Here are some of the lines from the logs
[03-Mar-2014 02:44:43 America/New_York] Phirehose: Opening new active status stream: /tmp/.phirehose-ghettoqueue.current
[03-Mar-2014 02:45:12 America/New_York] Phirehose: Successfully rotated active stream to queue file: /tmp/phirehose-ghettoqueue.20140303-024512.queue
----- and ------
[03-Mar-2014 03:41:58 America/New_York] Processing file: /tmp/phirehose-ghettoqueue.20140303-024102.queue
[03-Mar-2014 03:41:59 America/New_York] Successfully processed 1 tweets from /tmp/phirehose-ghettoqueue.20140303-024102.queue - deleting.
-- The bits of code in question, I think --
/**
* Subclass specific constants
*/
const QUEUE_FILE_PREFIX = 'phirehose-ghettoqueue';
const QUEUE_FILE_ACTIVE = '.phirehose-ghettoqueue.current';
public function __construct($username, $password, $queueDir = '/tmp', $rotateInterval = 10)
// Set subclass parameters
$this->queueDir = $queueDir;
// Construct stream file name, log and open
$this->streamFile = $this->queueDir . '/' . self::QUEUE_FILE_ACTIVE;
$this->log('Opening new active status stream: ' . $this->streamFile);
$this->statusStream = fopen($this->streamFile, 'a'); // Append if present (crash recovery)
Source: (StackOverflow)
I'm trying to put in a check for Phirehose to stop running after 10 seconds or 100 tweets...basically, I want to be able to stop the script.
I was told I could customize the statusUpdate() function or the heartBeat() function, but I'm uncertain how to do that. Right now, I'm just testing with the filter-track.php example.
How do I customize the functions, and where should I call them in the class?
class FilterTrackConsumer extends OauthPhirehose
{
/**
* Enqueue each status
*
* @param string $status
*/
public function enqueueStatus($status)
{
/*
* In this simple example, we will just display to STDOUT rather than enqueue.
* NOTE: You should NOT be processing tweets at this point in a real application, instead they should be being
* enqueued and processed asyncronously from the collection process.
*/
$data = json_decode($status, true);
if (is_array($data) && isset($data['user']['screen_name'])) {
print $data['user']['screen_name'] . ': ' . urldecode($data['text']) . "\n";
}
}
public function statusUpdate()
{
return "asdf";
}
}
// The OAuth credentials you received when registering your app at Twitter
define("TWITTER_CONSUMER_KEY", "");
define("TWITTER_CONSUMER_SECRET", "");
// The OAuth data for the twitter account
define("OAUTH_TOKEN", "");
define("OAUTH_SECRET", "");
// Start streaming
$sc = new FilterTrackConsumer(OAUTH_TOKEN, OAUTH_SECRET, Phirehose::METHOD_FILTER);
$sc->setLang('en');
$sc->setTrack(array('love'));
$sc->consume();
Source: (StackOverflow)
I use Twitter streaming API (POST https://stream.twitter.com/1.1/statuses/filter.json) to monitor tweets. It worked up to 01/13/2014, and sinse that time it returns 403 error; from documentation I see that this error code means "The connecting account is not permitted to access this endpoint." I tried different applications and accounts, as well as 2 different servers (to make sure that it is not IP ban for any reason), but this error appears always. I know about API movement from HTTP to HTTPS, but it shouldn't be a problem, because I use HTTPS anyway.
Does anyone use filter.json successfully now and what may be a reason of this issue?
Source: (StackOverflow)
I am working on a project that is using Phirehose to collect and consume the Twitter Streaming API. The Phirehose library is designed to be run from the command line, preferably as a daemon or cron job.
I created a daemon and placed it in the library folder. Bootstrap.php has been updated to autoload the custom library. So, the application itself has no problem seeing my daemon.
My issue is how to integrate it properly with Zend Framework. I need to be able to call the daemon file directly to start it from the command line or using a tool such as Upstart, but in doing so the Zend application doesn't load which means I don't have access to my models.
I could create a controller to start it, but I don't want to add the security issue of somebody being able to control the daemon from a web interface. I could also write the PDO to manually connect to the database, but I'm trying to avoid it for scaling reasons. I'd prefer all database connection data reside in application.ini.
Is there a way, within my daemon class, to initialize my Zend Framework application so I can use the models?
Source: (StackOverflow)