cubism
Cubism.js: A JavaScript library for time series visualization.
Cubism.js
I like the user experience of cubism, and would like to use this on top of a backend we have.
I've read the API doc's and some of the code, most of this seems to be extracted away. How could I begin to use other data sources exactly?
I have a data store of about 6k individual machines with 5 minute precision on around 100 or so stats.
I would like to query some web app with a specific identifier for that machine and then render a dashboard similar to cubism via querying a specific mongo data store.
Writing the webapp or the querying to mongo isn't the issue.
The issue is more in line with the fact that cubism seems to require querying whatever data store you use for each individual data point (say you have 100 stats across a window of a week...expensive).
Is there another way I could leverage this tool to look at data that gets loaded using something similar to the code below?
var data = [];
d3.json("/initial", function(json) { data.concat(json); });
d3.json("/update", function(json) { data.push(json); });
Source: (StackOverflow)
I am currently playing around with Cube . I have been able to send events to an evaluator and can query these via HTTP GET.
How do I go about making a customized dashboard to visualize the events/queries? I saw this website http://corner.squareup.com/2011/09/cube.html has a video for creating a "dashboard in 60 seconds" but I do not know where to find this.
Source: (StackOverflow)
Can someone point me in the direction of what I should be tweaking to get deterministic color band? For example, if they array of colors contains 6 colors, I would like the values 10-30 to always go to the 5th color band, 50-80 to the 6th, etc.
(in case someone is wondering why I need such an odd thing, right now the value 30 shows up in several different colors, depending on what other values are in the set -- can be pretty confusing for a user over time). I thought of restricting the min and max values in the set to achieve stable colors but that would distort the lables. Also, ideally, I would like to specify the mapping of input data value to color myself
Source: (StackOverflow)
I saw the cubism graphs and they are simply amazing. I have a big JSON file with 1000 entries that have a timestamp and a value (integer). Can Cubism graph those or not?! I can't seem to find documentation on this...
Source: (StackOverflow)
My problem is that the ruler doesn't exactly land on the cursor as in the screenshot and I'm not sure where to start looking in order to fix it.
Any advice?
Source: (StackOverflow)
I'm using cubism.js to graph some static data from a JSON object. We want to be able to display a years worth of data, with a point for each day. I have the solution working partially but I'd like to be able to set the scale independent of today's date, i.e. I'd like to graph data from yesterday to the corresponding day last year. I've tried the following without success:
context.scale(d3.time.scale().domain([start, end]).range([0,96]))
where start and end come from the JSON object.
Is it possible to set the cubism scale to behave in this way ?
Many thanks,
Michael
Source: (StackOverflow)
I'm plotting some metrics that I have stored in a backbone collection that is being updated every second. I'm not seeing the correct metrics being plotted on cubism. For each metric, I only get the one metric plotted (with an incorrect) value. I don't really see how I can debug this. Here is my code:
var models = dataSet.models;
//aggregate 'metrics'. Each component has a 'metric' which contains its stats over time
for(model in models){
var attributes = models[model].attributes;
var metricData = metricCache.get(models[model].id);
var metrics = metricData['attributes']['metrics'];
if(!attributes['name'] || attributes['type']== "FLOW" || attributes['type'] == "SERVER"){
continue;
}
if(attributes['name'] == null){
continue;
}
var name = attributes['name'];
var type = attributes['type'];
var metName = name;
/* GETTING METRIC. Using js closure to pass correct values to metric call */
if(metricData != null){
//var curContext = getMetric(metName, metrics);
var curContext = (function(val1, val2){
return getMetric(val1, val2);
})(metName,metrics);
}
statsList.push(curContext);
}
'getMetric' takes the metric's name and 'metrics', which is an array of objects where each object has a timestamp and a value. I'm simply trying to plot each value with its corresponding timestamp on the horizon chart. Here is my code for 'getMetric'.
function getMetric(name, metricData){
var format = d3.time.format("%I-%M-%S");
return context.metric(function(start, stop, step, callback){
var statValues = [];
var lookup = {},
i = start.getTime();
//console.log('startTime', i);
var curStat = null;
for(stat in metricData){
curStat = metricData[stat];
curStat.value = +curStat.value;
var curDate = new Date(curStat.timestamp);
curDate = format(curDate);
lookup[curDate] = curStat;
}
var lastValue;
while((i+=step) < stop){
var key = format(new Date(i));
var curVal = key in lookup ? lookup[key].value : NaN;
console.log(name,curVal);
statValues.push(curVal);
}
//console.log(statValues);
callback(null,statValues);
}, name);
}
Here is a screenshot of what I see being plotted, which is each metric plotting the same values (which I verified is incorrect data).
Source: (StackOverflow)
I'm trying to render real-time data using Cubism. Every 5 seconds I'm using a custom metric to poll my own data source and I'm displaying the last 2 hours of data. I'm building my context with:
var context = cubism.context().step(5000).size(1440);
I start out by plotting the past 2 hours of data, then start adding on in real-time. I don't need to show any axes, just the horizon chart by itself.
The catch is I need to render this graph within 216px of screen real-estate. When I render it, it doesn't seem to be adjusting for the width of the graph. Instead, the canvas always has a width of "1440", and (when I turn off the container's width) the chart goes all the way across the page.
On top of that, the horizon bars look terrible. They are very disjointed, and the dark vs. light is very scattered. See the screenshot for an idea of what I'm seeing:
Also, here is a gist containing the values rendered in this chart.
How do I get Cubism to render this chart properly using the correct width?
Thanks!
Source: (StackOverflow)
I want to gain experience using cubism.js
I found this dataset nyc real-time traffic I want to graph.
I need some tips on how to get this done.
I know I can use python to query the txt file every N minutes. I would then store the last M minutes of traffic data server-side.
When the user visits the site, the server will serve the last M minutes of data. But then what? The client queries the server via cubism.js? or does the server push updates to the user?
I've read through the cubism.js wiki but there aren't any examples on how to do this without cube or graphite.
Also, this would be my first project since my jsp days, so any tips on that front are welcome as well.
Source: (StackOverflow)
I am trying to make an axis of 24 hours with a separation of 1 hour in cubism.
var context = cubism.context()
.serverDelay(30*1000)
.step(10 * 60 * 1000) //The step size is 10 mins
.size(window_width - 200);
d3.select("#viz").selectAll(".axis")
.data(["top", "bottom"])
.enter().append("div")
.attr("class", function(d) { return d + " axis"; })
.each(function(d) { d3.select(this).call(context.axis().ticks(d3.time.hours, 1).orient(d)); });
This line context.axis().ticks(d3.time.hours, 1).orient(d) is somehow wrong. Since the axis text gets overlapped and nothing is visible.
I tried many combinations like 60 mins, 1 Day but nothing seems to work.
Can anyone please help me.
Source: (StackOverflow)
In a test environment setup, I'm using diamond to send server stats and metrics to statsd and then using statsd's elasticsearch backend to send them to elasticsearch.
The elasticsearch data looks like this:
{
"hits" : {
"hits" : [ {
"_source":{"ns":"servers","grp":"server1","tgt":"vmstat","act":"pswpout","val":"0","@timestamp":"1438565055000"}
}, {
"_source":{"ns":"servers","grp":"server1","tgt":"vmstat","act":"pgpgin","val":"0","@timestamp":"1438565055000"}
}, {
"_source":{"ns":"servers","grp":"server1","tgt":"vmstat","act":"pswpin","val":"0","@timestamp":"1438565055000"}
}, {
"_source":{"ns":"servers","grp":"server1","tgt":"cpu","act":"total.nice","val":"0","@timestamp":"1438565055000"}
}, {
"_source":{"ns":"servers","grp":"server1","tgt":"cpu","act":"total.irq","val":"0","@timestamp":"1438565055000"}
}, {
"_source":{"ns":"servers","grp":"server1","tgt":"cpu","act":"total.guest","val":"0","@timestamp":"1438565055000"}
}, {
"_source":{"ns":"servers","grp":"server1","tgt":"diskspace","act":"_logs.byte_used","val":"209944576","@timestamp":"1438565055000"}
}, {
"_source":{"ns":"servers","grp":"server1","tgt":"diskspace","act":"_logs.byte_free","val":"887513440256","@timestamp":"1438565055000"}
}, {
"_source":{"ns":"servers","grp":"server1","tgt":"diskspace","act":"_logs.byte_avail","val":"842419666944","@timestamp":"1438565055000"}
}, {
"_source":{"ns":"servers","grp":"server1","tgt":"diskspace","act":"_logs.inodes_used","val":"11","@timestamp":"1438565055000"}
} ]
}
}
How would I go about having this data rendered using cubism? I can extract the data using elasticsearch.js, but just generally stumped about how I would go about getting this rendered, so that, I can visualize the time-series data for each of the different actions, for instance.
Source: (StackOverflow)
So my goal is to plot some real-time metrics that are stored in a backbone collection. I have a backbone collection that stores metrics, it's updated every second when I poll the backend server. My collection has an array of historical metrics and a field 'most-recent-data' which is the latest data to come from the backend. Every second, 'most-recent-data' is updated to a different value. I pass a reference of the backbone collection to the function that creates the cubism metrics.
When the view containing the Cubism.js view is rendered, it should do a historical load of all the data points in the stored in the collection to the horizon chart. This is the part I've had success on. However, when the metric function does a callback, it is not plotting new/correct points from the 'most-recent-data' field which is updated every second.
Here is my code that interacts with the DOM (I don't believe this part is causing a problem):
var context = cubism.context()
.serverDelay(0)
.clientDelay(0)
.step(250)
.size(1116);
//Getting the metrics from the backbone collection
var models = this.dataSet.models;
console.log('models in metric view',models);
//aggregate 'metrics'. Each component has a 'metric' which contains its stats over time
for(model in models){
var attributes = models[model].attributes;
if(!attributes['name'] || attributes['type']== "FLOW" || attributes['type'] == "SERVER"){
continue;
}
if(attributes['name'] == null){
continue;
}
var name = attributes['name'];
var type = attributes['type'];
var serverName = attributes['serverName'];
var metName = name.concat(serverName);
console.log(metName);
//Getting the cubism metric for each stat in the backbone collection
//Passing in a reference to the current model in the backbone collection (this.dataSet.models[model])
var curContext = getMetric(metName, this.dataSet.models[model]);
statsList.push(curContext);
}
d3.select(this.loc).selectAll(".axis")
.data(["top", "bottom"])
.enter().append("div")
.attr("class", function(d) { return d + " axis"; })
.each(function(d) { d3.select(this).call(context.axis().ticks(12).orient(d)); });
//create rule
d3.select(this.loc).append("div")
.style("position", "fixed")
.style("top", 0)
.style("bottom", 0)
.style("width", "1px")
.style("pointer-events", "none")
.call(context.rule());
d3.select(this.loc).selectAll(".horizon")
.data(statsList)
.enter().insert("div", ".bottom")
.attr("class", "horizon")
.call(context.horizon().height(35));
It's essentially iterating through all the stats in the backbone collection, then calling 'getMetric' passing in a reference to the current stat in the backbone collection. 'getMetric' returns a cubism metric.
Here's the code that handles each metric (probably causing the problem):
/* Keep a map of intialized metrics
When the visualization is initialized, we load the historical data which is in the
'attributes' field of the model. The model is an individual set metrics */
var initializedMetrics = {};
function getMetric(name, model){
var format = d3.time.format("%I-%M-%S");
return context.metric(function(start, stop, step, callback){
var statValues = [];
if(initializedMetrics[name]){
/* If the metric has already been initialized and we loaded the historical data
from 'attributes' field, plot the newest data which is stored in 'most-recent-data'
This field should be updated every second on every API call to the server */
while(start<stop){
start+=step;
console.log(model.attributes['most-recent-data']['rate']);
statValues.push(model.attributes['most-recent-data']['rate']);
}
} else{
/* Metric has not been initalized, so we load all the historical data in 'all-data'
for this stat and plot it over time*/
initializedMetrics[name]=true;
var lookup = {},
i = start.getTime();
//console.log('startTime', i);
var curStat = null;
var metricData = model.attributes['all-data']
for(stat in metricData){
curStat = metricData[stat];
//console.log(name, curStat);
var curDate = new Date(curStat['timeStamp']);
curDate = format(curDate);
lookup[curDate] = curStat;
}
var lastValue;
while((i+=step) < stop){
start+=step;
var key = format(new Date(i));
if(key in lookup){
lastValue = lookup[key]['valueLong'];
}
var curVal = key in lookup ? lookup[key]['valueLong'] : lastValue;
//console.log(name,curVal);
statValues.push(curVal);
}
}
/* Callback with statValues*/
callback(null,statValues);
}, name);
}
I'm having success loading all the historical data (everything inside the else bracket). However, when a metric is initialized, I try to load new data which is stored 'most-recent-data'. This field is updated every second with the API calls, but with each callback, it's ONLY plotting the initial data stored in 'most-recent-data'. I can confirm that most-recent-data is indeed being updated in the backbone collection itself, but the reference being passed to cubism is not being updated at each callback. When I log 'most-recent-data' in the cubism code, it never updates to the latest value.
Is this a closure issue? Am I missing some cubism syntax for polling new data? Or would I need to pass in the backbone collection reference differently? Any hints would be very helpful. Thank you.
Source: (StackOverflow)
I have a few devices that emit time series data:
[deviceID],[time],[value]
I am using graphite to keep track of this data but the question applies to other databases as well.
I have defined my data retention/precision to be 5 seconds - so each device will only have one value per 5 seconds which is the average of all the observations it had made during this period. For example if these are the real measurements:
device1 1/1/2012 08:00:00 12
device1 1/1/2012 08:00:01 10
device2 1/1/2012 08:00:01 2
device1 1/1/2012 08:00:02 14
Then the data saved will be:
device1 1/1/2012 08:00:00 12
device2 1/1/2012 08:00:00 2
How could I query for the average value across both devices in this time period? I can't just take their average over the saved data (=7) since it is biased down because it does not consider that device1 had more measurements. Do I need to keep track of the avg for every device pair/trio? Maybe it is best not to do aggregations at all and get maximum flexibility? Or is it accepted to not allow such cross-device queries if this is just a nice to have feature?
Source: (StackOverflow)
I'm just wondering if d3.text is faster than d3.json?
The reason behind my question is that I'm reading the source code behind cubism.js and I'm just curious to know if it's done with d3.text because it's faster?
Source: (StackOverflow)