62 Commits

Author SHA1 Message Date
4c6a6732bf Bump version to 0.5.0 2012-04-18 23:30:23 +01:00
737cd342ba Change wording slightly and kill a whitespace in readme 2012-04-18 23:28:50 +01:00
eaca48ca76 Keep the attribute writer together with the method definition 2012-04-18 23:26:18 +01:00
20321650bd Merge pull request #14 from czarneckid/configurable_group_separator
Allow for configurable group separator
2012-04-18 15:22:50 -07:00
David Czarnecki
fa182e618d Allow for configurable group separator 2012-04-18 14:44:36 -04:00
a9cf3938cf Bump version to 0.4.0 - Add key expiry support 2012-04-18 16:49:52 +01:00
7684e0bebc Merge branch 'key-expiry' 2012-04-18 16:43:22 +01:00
41f53b9fba Print a warning to STDERR if Redis version is less than 2.1.3
Prior to v2.1.3, any writes to keys with an expire/TTL set were
completely deleted before the write took place.
2012-04-18 16:42:27 +01:00
c0ecf4bc84 Add information about key expiry to readme 2012-04-18 16:19:03 +01:00
a8de80e69e Remove rbx from .travis.yml
Proper support for Rubinus will added in due time.
2012-04-18 15:15:14 +01:00
6429f07d5b Update .travis.yml to cover more Ruby versions 2012-04-18 15:11:13 +01:00
e15f637603 Make Model.expire act like the other options, and add tests for it 2012-04-18 15:09:36 +01:00
fccf0db68a Options are optional 2012-04-18 12:33:19 +01:00
e25ac3b85a Merge branch 'master' into key-expiry 2012-04-18 11:41:07 +01:00
ea68a91a58 Update, and add travis-ci status to readme 2012-04-18 11:33:39 +01:00
ce21b839f2 Rewrote Redistat::Sychronize rspecs
Partly thanks to Travis-CI failing for an unexplained reason, but also
cause they needed to be updated.
2012-04-18 11:23:52 +01:00
4024887b58 Small tweak to better deal with value caching 2012-04-18 11:22:27 +01:00
9a084d28a0 Attempt to fix unexplainably failing specs on travis-ci 2012-04-18 10:00:55 +01:00
e5b0aa32ed Add simplecov to tests 2012-04-18 09:55:30 +01:00
273d6cda24 Initial work to get key expiry working 2012-04-17 17:02:57 +01:00
5087f4ef45 Improve Buffer's unique key identifier 2012-04-17 16:31:31 +01:00
6502fc0f62 Wrap readme at 78 colomns 2012-04-17 13:11:34 +01:00
7b14b9b5ab Clean up whitespace 2012-04-17 13:03:38 +01:00
4d5998af91 updated readme to syntax highlight Ruby code examples 2012-02-17 10:01:30 +00:00
07bb9e4fab removed rcov development dependency 2012-01-24 11:41:59 +00:00
d34d820a8a removed .rvmrc file 2012-01-24 11:34:52 +00:00
35c9cabb00 Merge pull request #13 from sguha00/patch-1
Update README.md
2012-01-24 03:32:37 -08:00
sguha00
74e2a86680 Update README.md 2012-01-23 15:39:33 -08:00
d4289400b6 Merge travis-ci.org related changes from dev. Doesn't effect code, so I'm not bothering with a new gem version. 2011-06-22 14:43:32 +01:00
77c6db0d4e added custom config for travis-ci.org 2011-06-22 14:39:15 +01:00
39dc4d90e8 added rake to development dependencies 2011-06-22 14:37:46 +01:00
a609b19ad2 updated Buffer spec comments and pending test with new info about JRuby, 1.8.x, and 1.9.x 2011-04-19 09:48:54 +01:00
91272dfe6a Merge branch 'release/v0.3.0' into dev 2011-04-18 14:28:02 +01:00
01a39b1b20 Merge branch 'release/v0.3.0' 2011-04-18 14:27:59 +01:00
ae5a391012 started release v0.3.0 2011-04-18 14:27:40 +01:00
c53c7116dd updated Connection TODO comment 2011-04-18 14:26:52 +01:00
d9a8aefcc5 updated readme with thread_safe and buffer info 2011-04-18 14:25:38 +01:00
0ec2f5bd14 Merge branch 'feature/buffer' into dev 2011-04-18 14:11:06 +01:00
b2c31a0e87 ensure buffer size value is read/written to in a thread-safe manner 2011-04-18 12:40:49 +01:00
b13da6f332 create a flush buffer #at_exit callback to ensure any buffered messages are flushed to Redis on process exit 2011-04-18 12:37:24 +01:00
7b5c308960 model spec updated to test write buffer 2011-04-18 12:36:43 +01:00
eb1d607a61 a number of issues fixed with Buffer class, and specs updated accordingly 2011-04-18 12:36:24 +01:00
b129074cd7 make Buffer#queue a private method as it's never supposed to be modified or read from outside of the Buffer object 2011-04-18 10:00:22 +01:00
4b06513813 additional specs for Redistat::Buffer, still a few more needed 2011-04-15 17:42:48 +01:00
2ca5aae4b8 require required libraries, just cause 2011-04-15 16:46:12 +01:00
6c63843cd5 updated Redistat::Summary to incorporate use of write Buffer 2011-04-15 16:45:56 +01:00
3a25fcc788 created Redistat::Buffer, mainly feature complete, still needs a few more specs 2011-04-15 16:45:21 +01:00
61231a8b57 updated Redistat::Summary to make it easier to plugin the buffer interception code 2011-04-15 14:14:17 +01:00
a197a04ce8 moved all internal mixin modules to lib/redistat/mixins to tidy up the file structure a bit 2011-04-15 14:10:51 +01:00
5d92c1dbae created Redistat::Synchronize mixin to help with thread-safety 2011-04-15 14:03:26 +01:00
0a7abe935e thread-safe connection handler 2011-04-14 16:53:29 +01:00
f155f6db05 cleaned up Connection spec a bit 2011-04-14 16:50:43 +01:00
9afd5b6ccf Merge branch 'release/v0.2.6' 2011-04-13 10:29:02 +01:00
eb0c461aa7 Merge branch 'release/v0.2.6' into dev 2011-04-13 10:29:02 +01:00
f89ccc2514 started release v0.2.6 2011-04-13 10:25:52 +01:00
ef7b74df81 Fixed an issue caused by smembers returning nil rather than an empty Array in some older versions of Redis. 2011-04-13 10:25:13 +01:00
8106677561 Merge branch 'release/v0.2.5' into dev 2011-03-16 00:58:52 +00:00
00e0015ac2 Merge branch 'release/v0.2.5' 2011-03-16 00:58:43 +00:00
7e82246662 started release v0.2.5 2011-03-16 00:58:31 +00:00
84f3bf26b5 updated readme 2011-03-16 00:57:18 +00:00
2e2d3273cc set #scope rather than #class_name as the preferred method for setting the scope value within a Redistat::Model 2011-03-16 00:56:59 +00:00
a983e554c6 Merge branch 'release/v0.2.4' into dev 2011-03-14 11:14:06 +00:00
48 changed files with 1587 additions and 659 deletions

1
.rvmrc
View File

@@ -1 +0,0 @@
rvm gemset use redistat

8
.travis.yml Normal file
View File

@@ -0,0 +1,8 @@
language: ruby
rvm:
- 1.8.7
- 1.9.2
- 1.9.3
- jruby-18mode
- jruby-19mode
- ree

342
README.md
View File

@@ -1,74 +1,339 @@
# Redistat # Redistat [![Build Status](https://secure.travis-ci.org/jimeh/redistat.png)](http://travis-ci.org/jimeh/redistat)
A Redis-backed statistics storage and querying library written in Ruby. A Redis-backed statistics storage and querying library written in Ruby.
Redistat was originally created to replace a small hacked together statistics collection solution which was MySQL-based. When I started I had a short list of requirements: Redistat was originally created to replace a small hacked together statistics
collection solution which was MySQL-based. When I started I had a short list
of requirements:
* Store and increment/decrement integer values (counters, etc) * Store and increment/decrement integer values (counters, etc)
* Up to the second statistics available at all times * Up to the second statistics available at all times
* Screamingly fast * Screamingly fast
Redis fits perfectly with all of these requirements. It has atomic operations like increment, and it's lightning fast, meaning if the data is structured well, the initial stats reporting call will store data in a format that's instantly retrievable just as fast. Redis fits perfectly with all of these requirements. It has atomic operations
like increment, and it's lightning fast, meaning if the data is structured
well, the initial stats reporting call will store data in a format that's
instantly retrievable just as fast.
## Installation ## Installation
gem install redistat gem install redistat
If you are using Ruby 1.8.x, it's recommended you also install the `SystemTimer` gem, as the Redis gem will otherwise complain. If you are using Ruby 1.8.x, it's recommended you also install the
`SystemTimer` gem, as the Redis gem will otherwise complain.
## Usage ## Usage (Crash Course)
The simplest way to use Redistat is through the model wrapper. view\_stats.rb:
class VisitorStats ```ruby
require 'redistat'
class ViewStats
include Redistat::Model include Redistat::Model
end end
Before any of you Rails-purists start complaining about the model name being plural, I want to point out that it makes sense with Redistat, cause a model doesn't exactly return a specific row or object. But I'm getting ahead of myself. # if using Redistat in multiple threads set this
# somewhere in the beginning of the execution stack
To store statistics we essentially tell Redistat that an event has occurred with a label of X, and statistics of Y. So let's say we want to store a page view event on the `/about` page on a site: Redistat.thread_safe = true
```
VisitorStats.store('/about', {:views => 1})
In the above case "`/about`" is the label under which the stats are grouped, and the statistics associated with the event is simply a normal Ruby hash, except all values need to be integers, or Redis' increment calls won't work.
To later retrieve statistics, we use the `fetch` method:
stats = VisitorStats.fetch('/about', 2.hour.ago, Time.now)
# stats => [{:views => 1}]
# stats.total => {:views => 1}
The fetch method requires 3 arguments, a label, a start time, and an end time. Fetch returns a `Redistat::Collection` object, which is normal Ruby Array with a couple of added methods, like total shown above.
For more detailed usage, please check spec files, and source code till I have time to write up a complete readme.
## Some Technical Details ### Simple Example
To give a brief look into how Redistat works internally to store statistics, I'm going to use the examples above. The store method accepts a Ruby Hash with statistics to store. Redistat stores all statistics as hashes in Redis. It stores summaries of the stats for the specific time when it happened and all it's parent time groups (second, minute, hour, day, month, year). The default depth Redistat goes to is hour, unless the `depth` option is passed to `store` or `fetch`. Store:
In short, the above call to `store` creates the following keys in Redis: ```ruby
ViewStats.store('hello', {:world => 4})
ViewStats.store('hello', {:world => 2}, 2.hours.ago)
```
VisitorStats//about:2010 Fetch:
VisitorStats//about:201011
VisitorStats//about:20101124
VisitorStats//about:2010112401
Each of these keys in Redis are a hash, containing the sums of each statistic point reported for the time frame the key represents. In this case there's two slashes, cause the label we used was “`/about`”, and the scope (class name when used through the model wrapper) and the label are separated with a slash. ```ruby
ViewStats.find('hello', 1.hour.ago, 1.hour.from_now).all
#=> [{'world' => 4}]
ViewStats.find('hello', 1.hour.ago, 1.hour.from_now).total
#=> {'world' => 4}
ViewStats.find('hello', 3.hour.ago, 1.hour.from_now).total
#=> {'world' => 6}
```
When retrieving statistics for a given date range, Redistat figures out how to do the least number of calls to Redis to fetch all relevant data. For example, if you want the sum of stats from the 4th till the last of November, the full month of November would first be fetched, then the first 3 days of November would be fetched and removed from the full month stats.
### Advanced Example
Store page view on product #44 from Chrome 11:
```ruby
ViewStats.store('views/product/44', {'count/chrome/11' => 1})
```
Fetch product #44 stats:
```ruby
ViewStats.find('views/product/44', 23.hours.ago, 1.hour.from_now).total
#=> { 'count' => 1, 'count/chrome' => 1, 'count/chrome/11' => 1 }
```
Store a page view on product #32 from Firefox 3:
```ruby
ViewStats.store('views/product/32', {'count/firefox/3' => 1})
```
Fetch product #32 stats:
```ruby
ViewStats.find('views/product/32', 23.hours.ago, 1.hour.from_now).total
#=> { 'count' => 1, 'count/firefox' => 1, 'count/firefox/3' => 1 }
```
Fetch stats for all products:
```ruby
ViewStats.find('views/product', 23.hours.ago, 1.hour.from_now).total
#=> { 'count' => 2,
# 'count/chrome' => 1,
# 'count/chrome/11' => 1,
# 'count/firefox' => 1,
# 'count/firefox/3' => 1 }
```
Store a 404 error view:
```ruby
ViewStats.store('views/error/404', {'count/chrome/9' => 1})
```
Fetch stats for all views across the board:
```ruby
ViewStats.find('views', 23.hours.ago, 1.hour.from_now).total
#=> { 'count' => 3,
# 'count/chrome' => 2,
# 'count/chrome/9' => 1,
# 'count/chrome/11' => 1,
# 'count/firefox' => 1,
# 'count/firefox/3' => 1 }
```
Fetch list of products known to Redistat:
```ruby
finder = ViewStats.find('views/product', 23.hours.ago, 1.hour.from_now)
finder.children.map { |child| child.label.me }
#=> [ "32", "44" ]
finder.children.map { |child| child.label.to_s }
#=> [ "views/products/32", "views/products/44" ]
finder.children.map { |child| child.total }
#=> [ { "count" => 1, "count/firefox" => 1, "count/firefox/3" => 1 },
# { "count" => 1, "count/chrome" => 1, "count/chrome/11" => 1 } ]
```
## Terminology
### Scope
A type of global-namespace for storing data. When using the `Redistat::Model`
wrapper, the scope is automatically set to the class name. In the examples
above, the scope is `ViewStats`. Can be overridden by calling the `#scope`
class method on your model class.
### Label
Identifier string to separate different types and groups of statistics from
each other. The first argument of the `#store`, `#find`, and `#fetch` methods
is the label that you're storing to, or fetching from.
Labels support multiple grouping levels by splitting the label string with `/`
and storing the same stats for each level. For example, when storing data to a
label called `views/product/44`, the data is stored for the label you specify,
and also for `views/product` and `views`. You may also configure a different
group separator using the `Redistat.group_separator=` method. For example:
```ruby
Redistat.group_separator = '|'
```
A word of caution: Don't use a crazy number of group levels. As two levels
causes twice as many `hincrby` calls to Redis as not using the grouping
feature. Hence using 10 grouping levels, causes 10 times as many write calls
to Redis.
### Input Statistics Data
You provide Redistat with the data you want to store using a Ruby Hash. This
data is then stored in a corresponding Redis hash with identical key/field
names.
Key names in the hash also support grouping features similar to those
available for Labels. Again, the more levels you use, the more write calls to
Redis, so avoid using 10-15 levels.
### Depth (Storage Accuracy)
Define how accurately data should be stored, and how accurately it's looked up
when fetching it again. By default Redistat uses a depth value of `:hour`,
which means it's impossible to separate two events which were stored at 10:18
and 10:23. In Redis they are both stored within a date key of `2011031610`.
You can set depth within your model using the `#depth` class method. Available
depths are: `:year`, `:month`, `:day`, `:hour`, `:min`, `:sec`
### Time Ranges
When you fetch data, you need to specify a start and an end time. The
selection behavior can seem a bit weird at first when, but makes sense when
you understand how Redistat works internally.
For example, if we are using a Depth value of `:hour`, and we trigger a fetch
call starting at `1.hour.ago` (13:34), till `Time.now` (14:34), only stats
from 13:00:00 till 13:59:59 are returned, as they were all stored within the
key for the 13th hour. If both 13:00 and 14:00 was returned, you would get
results from two whole hours. Hence if you want up to the second data, use an
end time of `1.hour.from_now`.
### The Finder Object
Calling the `#find` method on a Redistat model class returns a
`Redistat::Finder` object. The finder is a lazy-loaded gateway to your
data. Meaning you can create a new finder, and modify instantiated finder's
label, scope, dates, and more. It does not call Redis and fetch the data until
you call `#total`, `#all`, `#map`, `#each`, or `#each_with_index` on the
finder.
This section does need further expanding as there's a lot to cover when it
comes to the finder.
## Key Expiry
Support for expiring keys from Redis is available, allowing you too keep
varying levels of details for X period of time. This allows you easily keep
things nice and tidy by only storing varying levels detailed stats only for as
long as you need.
In the below example we define how long Redis keys for varying depths are
stored. Second by second stats are available for 10 minutes, minute by minute
stats for 6 hours, hourly stats for 3 months, daily stats for 2 years, and
yearly stats are retained forever.
```ruby
class ViewStats
include Redistat::Model
depth :sec
expire \
:sec => 10.minutes.to_i,
:min => 6.hours.to_i,
:hour => 3.months.to_i,
:day => 2.years.to_i
end
```
Keep in mind that when storing stats for a custom date in the past for
example, the expiry time for the keys will be relative to now. The values you
specify are simply passed to the `Redis#expire` method.
## Internals
### Storing / Writing
Redistat stores all data into a Redis hash keys. The Redis key name the used
consists of three parts. The scope, label, and datetime:
{scope}/{label}:{datetime}
For example, this...
```ruby
ViewStats.store('views/product/44', {'count/chrome/11' => 1})
```
...would store the follow hash of data...
```ruby
{ 'count' => 1, 'count/chrome' => 1, 'count/chrome/11' => 1 }
```
...to all 12 of these Redis hash keys...
ViewStats/views:2011
ViewStats/views:201103
ViewStats/views:20110315
ViewStats/views:2011031510
ViewStats/views/product:2011
ViewStats/views/product:201103
ViewStats/views/product:20110315
ViewStats/views/product:2011031510
ViewStats/views/product/44:2011
ViewStats/views/product/44:201103
ViewStats/views/product/44:20110315
ViewStats/views/product/44:2011031510
...by creating the Redis key, and/or hash field if needed, otherwise it simply
increments the already existing data.
It would also create the following Redis sets to keep track of which child
labels are available:
ViewStats.label_index:
ViewStats.label_index:views
ViewStats.label_index:views/product
It should now be more obvious to you why you should think about how you use
the grouping capabilities so you don't go crazy and use 10-15 levels. Storing
is done through Redis' `hincrby` call, which only supports a single key/field
combo. Meaning the above example would call `hincrby` a total of 36 times to
store the data, and `sadd` a total of 3 times to ensure the label index is
accurate. 39 calls is however not a problem for Redis, most calls happen in
less than 0.15ms (0.00015 seconds) on my local machine.
### Fetching / Reading
By default when fetching statistics, Redistat will figure out how to do the
least number of reads from Redis. First it checks how long range you're
fetching. If whole days, months or years for example fit within the start and
end dates specified, it will fetch the one key for the day/month/year in
question. It further drills down to the smaller units.
It is also intelligent enough to not fetch each day from 3-31 of a month,
instead it would fetch the data for the whole month and the first two days,
which are then removed from the summary of the whole month. This means three
calls to `hgetall` instead of 29 if each whole day was fetched.
### Buffer
The buffer is a new, still semi-beta, feature aimed to reduce the number of
Redis `hincrby` that Redistat sends. This should only really be useful when
you're hitting north of 30,000 Redis requests per second, if your Redis server
has limited resources, or against my recommendation you've opted to use 10,
20, or more label grouping levels.
Buffering tries to fold together multiple `store` calls into as few as
possible by merging the statistics hashes from all calls and groups them based
on scope, label, date depth, and more. You configure the the buffer by setting
`Redistat.buffer_size` to an integer higher than 1. This basically tells
Redistat how many `store` calls to buffer in memory before writing all data to
Redis.
## Todo ## Todo
* Proper/complete readme. * More details in Readme.
* Documentation. * Documentation.
* Anything else that becomes apparent after real-world use. * Anything else that becomes apparent after real-world use.
## Credits ## Credits
[Global Personals](http://globalpersonals.co.uk/) deserves a thank you. Currently the primary user of Redistat, they've allowed me to spend some company time to further develop the project. [Global Personals](http://globalpersonals.co.uk/) deserves a thank
you. Currently the primary user of Redistat, they've allowed me to spend some
company time to further develop the project.
## Note on Patches/Pull Requests ## Note on Patches/Pull Requests
@@ -77,8 +342,9 @@ When retrieving statistics for a given date range, Redistat figures out how to d
* Make your feature addition or bug fix. * Make your feature addition or bug fix.
* Add tests for it. This is important so I don't break it in a * Add tests for it. This is important so I don't break it in a
future version unintentionally. future version unintentionally.
* Commit, do not mess with rakefile, version, or history. * Commit, do not mess with rakefile, version, or history. (if you want to
(if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull) have your own version, that is fine but bump version in a commit by itself I
can ignore when I pull)
* Send me a pull request. Bonus points for topic branches. * Send me a pull request. Bonus points for topic branches.

View File

@@ -3,6 +3,7 @@ require 'rubygems'
require 'date' require 'date'
require 'time' require 'time'
require 'digest/sha1' require 'digest/sha1'
require 'monitor'
# Active Support 2.x or 3.x # Active Support 2.x or 3.x
require 'active_support' require 'active_support'
@@ -15,12 +16,15 @@ require 'time_ext'
require 'redis' require 'redis'
require 'json' require 'json'
require 'redistat/options' require 'redistat/mixins/options'
require 'redistat/mixins/synchronize'
require 'redistat/mixins/database'
require 'redistat/mixins/date_helper'
require 'redistat/connection' require 'redistat/connection'
require 'redistat/database' require 'redistat/buffer'
require 'redistat/collection' require 'redistat/collection'
require 'redistat/date' require 'redistat/date'
require 'redistat/date_helper'
require 'redistat/event' require 'redistat/event'
require 'redistat/finder' require 'redistat/finder'
require 'redistat/key' require 'redistat/key'
@@ -33,6 +37,7 @@ require 'redistat/version'
require 'redistat/core_ext' require 'redistat/core_ext'
module Redistat module Redistat
KEY_NEXT_ID = ".next_id" KEY_NEXT_ID = ".next_id"
@@ -47,6 +52,26 @@ module Redistat
class << self class << self
def buffer
Buffer.instance
end
def buffer_size
buffer.size
end
def buffer_size=(size)
buffer.size = size
end
def thread_safe
Synchronize.thread_safe
end
def thread_safe=(value)
Synchronize.thread_safe = value
end
def connection(ref = nil) def connection(ref = nil)
Connection.get(ref) Connection.get(ref)
end end
@@ -66,5 +91,16 @@ module Redistat
connection.flushdb connection.flushdb
end end
def group_separator
@group_separator ||= GROUP_SEPARATOR
end
attr_writer :group_separator
end end
end end
# ensure buffer is flushed on program exit
Kernel.at_exit do
Redistat.buffer.flush(true)
end

110
lib/redistat/buffer.rb Normal file
View File

@@ -0,0 +1,110 @@
require 'redistat/core_ext/hash'
module Redistat
class Buffer
include Synchronize
def self.instance
@instance ||= self.new
end
def size
synchronize do
@size ||= 0
end
end
def size=(value)
synchronize do
@size = value
end
end
def count
@count ||= 0
end
def store(key, stats, depth_limit, opts)
return false unless should_buffer?
to_flush = {}
buffkey = buffer_key(key, opts)
synchronize do
if !queue.has_key?(buffkey)
queue[buffkey] = { :key => key,
:stats => {},
:depth_limit => depth_limit,
:opts => opts }
end
queue[buffkey][:stats].merge_and_incr!(stats)
incr_count
# return items to be flushed if buffer size limit has been reached
to_flush = reset_queue
end
# flush any data that's been cleared from the queue
flush_data(to_flush)
true
end
def flush(force = false)
to_flush = {}
synchronize do
to_flush = reset_queue(force)
end
flush_data(to_flush)
end
private
# should always be called from within a synchronize block
def incr_count
@count ||= 0
@count += 1
end
def queue
@queue ||= {}
end
def should_buffer?
size > 1 # buffer size of 1 would be equal to not using buffer
end
# should always be called from within a synchronize block
def should_flush?
(!queue.blank? && count >= size)
end
# returns items to be flushed if buffer size limit has been reached
# should always be called from within a synchronize block
def reset_queue(force = false)
return {} if !force && !should_flush?
data = queue
@queue = {}
@count = 0
data
end
def flush_data(buffer_data)
buffer_data.each do |k, item|
Summary.update(item[:key], item[:stats], item[:depth_limit], item[:opts])
end
end
# depth_limit is not needed as it's evident in key.to_s
def buffer_key(key, opts)
# covert keys to strings, as sorting a Hash with Symbol keys fails on
# Ruby 1.8.x.
opts = opts.inject({}) do |result, (k, v)|
result[k.to_s] = v
result
end
"#{key.to_s}:#{opts.sort.flatten.join(':')}"
end
end
end

View File

@@ -1,30 +1,43 @@
require 'monitor'
module Redistat module Redistat
module Connection module Connection
REQUIRED_SERVER_VERSION = "1.3.10" REQUIRED_SERVER_VERSION = "1.3.10"
MIN_EXPIRE_SERVER_VERSION = "2.1.3"
# TODO: Create a ConnectionPool instance object using Sychronize mixin to replace Connection class
class << self class << self
# TODO: clean/remove all ref-less connections
def get(ref = nil) def get(ref = nil)
ref ||= :default ref ||= :default
synchronize do
connections[references[ref]] || create connections[references[ref]] || create
end end
end
def add(conn, ref = nil) def add(conn, ref = nil)
ref ||= :default ref ||= :default
synchronize do
check_redis_version(conn) check_redis_version(conn)
references[ref] = conn.client.id references[ref] = conn.client.id
connections[conn.client.id] = conn connections[conn.client.id] = conn
end end
end
def create(options = {}) def create(options = {})
#TODO clean/remove all ref-less connections synchronize do
options = options.clone
ref = options.delete(:ref) || :default ref = options.delete(:ref) || :default
options.reverse_merge!(default_options) options.reverse_merge!(default_options)
conn = (connections[connection_id(options)] ||= connection(options)) conn = (connections[connection_id(options)] ||= connection(options))
references[ref] = conn.client.id references[ref] = conn.client.id
conn conn
end end
end
def connections def connections
@connections ||= {} @connections ||= {}
@@ -36,9 +49,12 @@ module Redistat
private private
def check_redis_version(conn) def monitor
raise RedisServerIsTooOld if conn.info["redis_version"] < REQUIRED_SERVER_VERSION @monitor ||= Monitor.new
conn end
def synchronize(&block)
monitor.synchronize(&block)
end end
def connection(options) def connection(options)
@@ -46,10 +62,19 @@ module Redistat
end end
def connection_id(options = {}) def connection_id(options = {})
options.reverse_merge!(default_options) options = options.reverse_merge(default_options)
"redis://#{options[:host]}:#{options[:port]}/#{options[:db]}" "redis://#{options[:host]}:#{options[:port]}/#{options[:db]}"
end end
def check_redis_version(conn)
raise RedisServerIsTooOld if conn.info["redis_version"] < REQUIRED_SERVER_VERSION
if conn.info["redis_version"] < MIN_EXPIRE_SERVER_VERSION
STDOUT.puts "WARNING: You MUST upgrade Redis to v2.1.3 or later " +
"if you are using key expiry."
end
conn
end
def default_options def default_options
{ {
:host => '127.0.0.1', :host => '127.0.0.1',

View File

@@ -53,9 +53,10 @@ module Redistat
end end
def children def children
db.smembers("#{scope}#{LABEL_INDEX}#{@label}").map { |member| members = db.smembers("#{scope}#{LABEL_INDEX}#{@label}") || [] # older versions of Redis returns nil
members.map { |member|
child_label = [@label, member].reject { |i| i.nil? } child_label = [@label, member].reject { |i| i.nil? }
self.class.new(self.scope, child_label.join(GROUP_SEPARATOR), self.date, @options) self.class.new(self.scope, child_label.join(Redistat.group_separator), self.date, @options)
} }
end end

View File

@@ -13,7 +13,7 @@ module Redistat
def self.join(*args) def self.join(*args)
args = args.map {|i| i.to_s} args = args.map {|i| i.to_s}
self.new(args.reject {|i| i.blank? }.join(GROUP_SEPARATOR)) self.new(args.reject {|i| i.blank? }.join(Redistat.group_separator))
end end
def initialize(str, opts = {}) def initialize(str, opts = {})
@@ -48,16 +48,16 @@ module Redistat
end end
def me def me
self.to_s.split(GROUP_SEPARATOR).last self.to_s.split(Redistat.group_separator).last
end end
def groups def groups
return @groups unless @groups.nil? return @groups unless @groups.nil?
@groups = [] @groups = []
parent = "" parent = ""
self.to_s.split(GROUP_SEPARATOR).each do |part| self.to_s.split(Redistat.group_separator).each do |part|
if !part.blank? if !part.blank?
group = ((parent.blank?) ? "" : "#{parent}#{GROUP_SEPARATOR}") + part group = ((parent.blank?) ? "" : "#{parent}#{Redistat.group_separator}") + part
@groups << Label.new(group) @groups << Label.new(group)
parent = group parent = group
end end

View File

@@ -0,0 +1,52 @@
require 'monitor'
module Redistat
module Synchronize
class << self
def included(base)
base.send(:include, InstanceMethods)
end
def monitor
@monitor ||= Monitor.new
end
def thread_safe
monitor.synchronize do
return @thread_safe unless @thread_safe.nil?
@thread_safe = false
end
end
def thread_safe=(value)
monitor.synchronize do
@thread_safe = value
end
end
end # << self
module InstanceMethods
def thread_safe
Synchronize.thread_safe
end
def thread_safe=(value)
Synchronize.thread_safe = value
end
def monitor
Synchronize.monitor
end
def synchronize(&block)
if thread_safe
monitor.synchronize(&block)
else
block.call
end
end
end # InstanceMethods
end
end

View File

@@ -39,12 +39,20 @@ module Redistat
# #
option_accessor :depth option_accessor :depth
option_accessor :class_name option_accessor :scope
option_accessor :store_event option_accessor :store_event
option_accessor :hashed_label option_accessor :hashed_label
option_accessor :label_indexing option_accessor :label_indexing
alias :scope :class_name alias :class_name :scope
def expire(exp = nil)
if !exp.nil?
options[:expire] = exp.is_a?(Hash) ? exp : Hash.new(exp)
else
options[:expire]
end
end
def connect_to(opts = {}) def connect_to(opts = {})
Connection.create(opts.merge(:ref => name)) Connection.create(opts.merge(:ref => name))
@@ -62,7 +70,7 @@ module Redistat
alias :redis :connection alias :redis :connection
def name def name
options[:class_name] || (@name ||= self.to_s) options[:scope] || (@name ||= self.to_s)
end end
end end

View File

@@ -1,3 +1,5 @@
require 'active_support/core_ext/hash/indifferent_access'
module Redistat module Redistat
class Result < HashWithIndifferentAccess class Result < HashWithIndifferentAccess

View File

@@ -2,56 +2,77 @@ module Redistat
class Summary class Summary
include Database include Database
def self.default_options class << self
{ :enable_grouping => true,
def default_options
{
:enable_grouping => true,
:label_indexing => true, :label_indexing => true,
:connection_ref => nil } :connection_ref => nil,
:expire => {}
}
end end
def self.update_all(key, stats = {}, depth_limit = nil, opts = {}) def buffer
Redistat.buffer
end
def update_all(key, stats = {}, depth_limit = nil, opts = {})
stats ||= {} stats ||= {}
return nil if stats.size == 0 return if stats.empty?
options = default_options.merge((opts || {}).reject { |k,v| v.nil? }) options = default_options.merge((opts || {}).reject { |k,v| v.nil? })
depth_limit ||= key.depth depth_limit ||= key.depth
if options[:enable_grouping] update_through_buffer(key, stats, depth_limit, options)
end
def update_through_buffer(*args)
update(*args) unless buffer.store(*args)
end
def update(key, stats, depth_limit, opts = {})
if opts[:enable_grouping]
stats = inject_group_summaries(stats) stats = inject_group_summaries(stats)
key.groups.each do |k| key.groups.each do |k|
update_key(k, stats, depth_limit, options[:connection_ref]) update_key(k, stats, depth_limit, opts)
k.update_index if options[:label_indexing] k.update_index if opts[:label_indexing]
end end
else else
update_key(key, stats, depth_limit, options[:connection_ref]) update_key(key, stats, depth_limit, opts)
end end
end end
private private
def self.update_key(key, stats, depth_limit, connection_ref) def update_key(key, stats, depth_limit, opts = {})
Date::DEPTHS.each do |depth| Date::DEPTHS.each do |depth|
update(key, stats, depth, connection_ref) update_fields(key, stats, depth, opts)
break if depth == depth_limit break if depth == depth_limit
end end
end end
def self.update(key, stats, depth, connection_ref = nil) def update_fields(key, stats, depth, opts = {})
stats.each do |field, value| stats.each do |field, value|
db(connection_ref).hincrby key.to_s(depth), field, value db(opts[:connection_ref]).hincrby key.to_s(depth), field, value
end
if opts[:expire] && !opts[:expire][depth].nil?
db(opts[:connection_ref]).expire key.to_s(depth), opts[:expire][depth]
end end
end end
def self.inject_group_summaries!(stats) def inject_group_summaries!(stats)
summaries = {} summaries = {}
stats.each do |key, value| stats.each do |key, value|
parts = key.to_s.split(GROUP_SEPARATOR) parts = key.to_s.split(Redistat.group_separator)
parts.pop parts.pop
if parts.size > 0 if parts.size > 0
sum_parts = [] sum_parts = []
parts.each do |part| parts.each do |part|
sum_parts << part sum_parts << part
sum_key = sum_parts.join(GROUP_SEPARATOR) sum_key = sum_parts.join(Redistat.group_separator)
(summaries.has_key?(sum_key)) ? summaries[sum_key] += value : summaries[sum_key] = value (summaries.has_key?(sum_key)) ? summaries[sum_key] += value : summaries[sum_key] = value
end end
end end
@@ -59,9 +80,10 @@ module Redistat
stats.merge_and_incr!(summaries) stats.merge_and_incr!(summaries)
end end
def self.inject_group_summaries(stats) def inject_group_summaries(stats)
inject_group_summaries!(stats.clone) inject_group_summaries!(stats.clone)
end end
end end
end
end end

View File

@@ -1,3 +1,3 @@
module Redistat module Redistat
VERSION = "0.2.4" VERSION = "0.5.0"
end end

View File

@@ -24,7 +24,8 @@ Gem::Specification.new do |s|
s.add_runtime_dependency 'redis', '>= 2.1.0' s.add_runtime_dependency 'redis', '>= 2.1.0'
s.add_runtime_dependency 'time_ext', '>= 0.2.9' s.add_runtime_dependency 'time_ext', '>= 0.2.9'
s.add_development_dependency 'rake', '>= 0.8.7'
s.add_development_dependency 'rspec', '>= 2.1.0' s.add_development_dependency 'rspec', '>= 2.1.0'
s.add_development_dependency 'rcov', '>= 0.9.9'
s.add_development_dependency 'yard', '>= 0.6.3' s.add_development_dependency 'yard', '>= 0.6.3'
s.add_development_dependency 'simplecov', '>= 0.6.1'
end end

159
spec/buffer_spec.rb Normal file
View File

@@ -0,0 +1,159 @@
require "spec_helper"
describe Redistat::Buffer do
before(:each) do
@class = Redistat::Buffer
@buffer = Redistat::Buffer.instance
@key = mock('Key', :to_s => "Scope/label:2011")
@stats = {:count => 1, :views => 3}
@depth_limit = :hour
@opts = {:enable_grouping => true}
end
# let's cleanup after ourselves for the other specs
after(:each) do
@class.instance_variable_set("@instance", nil)
@buffer.size = 0
end
it "should provide instance of itself" do
@buffer.should be_a(@class)
end
it "should only buffer if buffer size setting is greater than 1" do
@buffer.size.should == 0
@buffer.send(:should_buffer?).should be_false
@buffer.size = 1
@buffer.size.should == 1
@buffer.send(:should_buffer?).should be_false
@buffer.size = 2
@buffer.size.should == 2
@buffer.send(:should_buffer?).should be_true
end
it "should only flush buffer if buffer size is greater than or equal to buffer size setting" do
@buffer.size.should == 0
@buffer.send(:queue).size.should == 0
@buffer.send(:should_flush?).should be_false
@buffer.send(:queue)[:hello] = 'world'
@buffer.send(:incr_count)
@buffer.send(:should_flush?).should be_true
@buffer.size = 5
@buffer.send(:should_flush?).should be_false
3.times { |i|
@buffer.send(:queue)[i] = i.to_s
@buffer.send(:incr_count)
}
@buffer.send(:should_flush?).should be_false
@buffer.send(:queue)[4] = '4'
@buffer.send(:incr_count)
@buffer.send(:should_flush?).should be_true
end
it "should force flush queue irregardless of result of #should_flush? when #reset_queue is called with true" do
@buffer.send(:queue)[:hello] = 'world'
@buffer.send(:incr_count)
@buffer.send(:should_flush?).should be_true
@buffer.size = 2
@buffer.send(:should_flush?).should be_false
@buffer.send(:reset_queue).should == {}
@buffer.instance_variable_get("@count").should == 1
@buffer.send(:reset_queue, true).should == {:hello => 'world'}
@buffer.instance_variable_get("@count").should == 0
end
it "should #flush_data into Summary.update properly" do
# the root level key value doesn't actually matter, but it's something like this...
data = {'ScopeName/label/goes/here:2011::true:true' => {
:key => @key,
:stats => @stats,
:depth_limit => @depth_limit,
:opts => @opts
}}
item = data.first[1]
Redistat::Summary.should_receive(:update).with(@key, @stats, @depth_limit, @opts)
@buffer.send(:flush_data, data)
end
it "should build #buffer_key correctly" do
opts = {:enable_grouping => true, :label_indexing => false, :connection_ref => nil}
@buffer.send(:buffer_key, @key, opts).should ==
"#{@key.to_s}:connection_ref::enable_grouping:true:label_indexing:false"
opts = {:enable_grouping => false, :label_indexing => true, :connection_ref => :omg}
@buffer.send(:buffer_key, @key, opts).should ==
"#{@key.to_s}:connection_ref:omg:enable_grouping:false:label_indexing:true"
end
describe "Buffering" do
it "should store items on buffer queue" do
@buffer.store(@key, @stats, @depth_limit, @opts).should be_false
@buffer.size = 5
@buffer.store(@key, @stats, @depth_limit, @opts).should be_true
@buffer.send(:queue).should have(1).item
@buffer.send(:queue)[@buffer.send(:queue).keys.first][:stats][:count].should == 1
@buffer.send(:queue)[@buffer.send(:queue).keys.first][:stats][:views].should == 3
@buffer.store(@key, @stats, @depth_limit, @opts).should be_true
@buffer.send(:queue).should have(1).items
@buffer.send(:queue)[@buffer.send(:queue).keys.first][:stats][:count].should == 2
@buffer.send(:queue)[@buffer.send(:queue).keys.first][:stats][:views].should == 6
end
it "should flush buffer queue when size is reached" do
key = mock('Key', :to_s => "Scope/labelx:2011")
@buffer.size = 10
Redistat::Summary.should_receive(:update).exactly(2).times.and_return do |k, stats, depth_limit, opts|
depth_limit.should == @depth_limit
opts.should == @opts
if k == @key
stats[:count].should == 6
stats[:views].should == 18
elsif k == key
stats[:count].should == 4
stats[:views].should == 12
end
end
6.times { @buffer.store(@key, @stats, @depth_limit, @opts).should be_true }
4.times { @buffer.store(key, @stats, @depth_limit, @opts).should be_true }
end
end
describe "Thread-Safety" do
it "should read/write to buffer queue in a thread-safe manner" do
# Setting thread_safe to false only makes the spec fail with
# JRuby. 1.8.x and 1.9.x both pass fine for some reason
# regardless of what the thread_safe option is set to.
Redistat.thread_safe = true
key = mock('Key', :to_s => "Scope/labelx:2011")
@buffer.size = 100
Redistat::Summary.should_receive(:update).exactly(2).times.and_return do |k, stats, depth_limit, opts|
depth_limit.should == @depth_limit
opts.should == @opts
if k == @key
stats[:count].should == 60
stats[:views].should == 180
elsif k == key
stats[:count].should == 40
stats[:views].should == 120
end
end
threads = []
10.times do
threads << Thread.new {
6.times { @buffer.store(@key, @stats, @depth_limit, @opts).should be_true }
4.times { @buffer.store(key, @stats, @depth_limit, @opts).should be_true }
}
end
threads.each { |t| t.join }
end
it "should have specs that fail on 1.8.x/1.9.x when thread_safe is disabled"
end
end

View File

@@ -3,33 +3,34 @@ include Redistat
describe Redistat::Connection do describe Redistat::Connection do
before(:each) do
@redis = Redistat.redis
end
it "should have a valid Redis client instance" do it "should have a valid Redis client instance" do
Redistat.redis.should_not be_nil Redistat.redis.should_not be_nil
end end
it "should have initialized custom testing connection" do it "should have initialized custom testing connection" do
redis = Redistat.redis @redis.client.host.should == '127.0.0.1'
redis.client.host.should == '127.0.0.1' @redis.client.port.should == 8379
redis.client.port.should == 8379 @redis.client.db.should == 15
redis.client.db.should == 15
end end
it "should be able to set and get data" do it "should be able to set and get data" do
redis = Redistat.redis @redis.set("hello", "world")
redis.set("hello", "world") @redis.get("hello").should == "world"
redis.get("hello").should == "world" @redis.del("hello").should be_true
redis.del("hello").should be_true
end end
it "should be able to store hashes to Redis" do it "should be able to store hashes to Redis" do
redis = Redistat.redis @redis.hset("hash", "field", "1")
redis.hset("hash", "field", "1") @redis.hget("hash", "field").should == "1"
redis.hget("hash", "field").should == "1" @redis.hincrby("hash", "field", 1)
redis.hincrby("hash", "field", 1) @redis.hget("hash", "field").should == "2"
redis.hget("hash", "field").should == "2" @redis.hincrby("hash", "field", -1)
redis.hincrby("hash", "field", -1) @redis.hget("hash", "field").should == "1"
redis.hget("hash", "field").should == "1" @redis.del("hash")
redis.del("hash")
end end
it "should be accessible from Redistat module" do it "should be accessible from Redistat module" do
@@ -58,4 +59,9 @@ describe Redistat::Connection do
Redistat.connect(:port => 8379, :db => 15) Redistat.connect(:port => 8379, :db => 15)
end end
# TODO: Test thread-safety
it "should be thread-safe" do
pending("need to figure out a way to test thread-safety")
end
end end

View File

@@ -194,13 +194,13 @@ describe Redistat::Finder do
def create_example_stats def create_example_stats
key = Redistat::Key.new(@scope, @label, (first = Time.parse("2010-05-14 13:43"))) key = Redistat::Key.new(@scope, @label, (first = Time.parse("2010-05-14 13:43")))
Redistat::Summary.update(key, @stats, :hour) Redistat::Summary.send(:update_fields, key, @stats, :hour)
key = Redistat::Key.new(@scope, @label, Time.parse("2010-05-14 13:53")) key = Redistat::Key.new(@scope, @label, Time.parse("2010-05-14 13:53"))
Redistat::Summary.update(key, @stats, :hour) Redistat::Summary.send(:update_fields, key, @stats, :hour)
key = Redistat::Key.new(@scope, @label, Time.parse("2010-05-14 14:52")) key = Redistat::Key.new(@scope, @label, Time.parse("2010-05-14 14:52"))
Redistat::Summary.update(key, @stats, :hour) Redistat::Summary.send(:update_fields, key, @stats, :hour)
key = Redistat::Key.new(@scope, @label, (last = Time.parse("2010-05-14 15:02"))) key = Redistat::Key.new(@scope, @label, (last = Time.parse("2010-05-14 15:02")))
Redistat::Summary.update(key, @stats, :hour) Redistat::Summary.send(:update_fields, key, @stats, :hour)
[first - 1.hour, last + 1.hour] [first - 1.hour, last + 1.hour]
end end

View File

@@ -38,6 +38,21 @@ describe Redistat::Label do
label.to_s.should == 'email/message/public' label.to_s.should == 'email/message/public'
end end
it "should allow you to use a different group separator" do
include Redistat
Redistat.group_separator = '|'
label = Label.join('email', 'message', 'public')
label.should be_a(Label)
label.to_s.should == 'email|message|public'
label = Label.join(Label.new('email'), Label.new('message'), Label.new('public'))
label.should be_a(Label)
label.to_s.should == 'email|message|public'
label = Label.join('email', '', 'message', nil, 'public')
label.should be_a(Label)
label.to_s.should == 'email|message|public'
Redistat.group_separator = Redistat::GROUP_SEPARATOR
end
describe "Grouping" do describe "Grouping" do
before(:each) do before(:each) do
@name = "message/public/offensive" @name = "message/public/offensive"

View File

@@ -25,6 +25,7 @@ end
class ModelHelper4 class ModelHelper4
include Redistat::Model include Redistat::Model
class_name "FancyHelper" scope "FancyHelper"
expire :hour => 24*3600
end end

View File

@@ -37,7 +37,8 @@ describe Redistat::Model do
ModelHelper2.depth.should == :day ModelHelper2.depth.should == :day
ModelHelper2.store_event.should == true ModelHelper2.store_event.should == true
ModelHelper2.hashed_label.should == true ModelHelper2.hashed_label.should == true
ModelHelper2.class_name.should be_nil ModelHelper2.scope.should be_nil
ModelHelper2.expire.should be_nil
ModelHelper1.depth.should == nil ModelHelper1.depth.should == nil
ModelHelper1.store_event.should == nil ModelHelper1.store_event.should == nil
@@ -55,8 +56,9 @@ describe Redistat::Model do
ModelHelper1.store_event.should == nil ModelHelper1.store_event.should == nil
ModelHelper1.hashed_label.should == nil ModelHelper1.hashed_label.should == nil
ModelHelper4.class_name.should == "FancyHelper" ModelHelper4.scope.should == "FancyHelper"
ModelHelper4.send(:name).should == "FancyHelper" ModelHelper4.send(:name).should == "FancyHelper"
ModelHelper4.expire.should == {:hour => 24*3600}
end end
it "should store and fetch stats" do it "should store and fetch stats" do
@@ -156,4 +158,44 @@ describe Redistat::Model do
stats.total[:weight].should == 617 stats.total[:weight].should == 617
end end
describe "Write Buffer" do
before(:each) do
Redistat.buffer_size = 20
end
after(:each) do
Redistat.buffer_size = 0
end
it "should buffer calls in memory before committing to Redis" do
14.times do
ModelHelper1.store("sheep.black", {:count => 1, :weight => 461}, @time.hours_ago(4))
end
ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1)).total.should == {}
5.times do
ModelHelper1.store("sheep.black", {:count => 1, :weight => 156}, @time)
end
ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1)).total.should == {}
ModelHelper1.store("sheep.black", {:count => 1, :weight => 156}, @time)
stats = ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1))
stats.total["count"].should == 20
stats.total["weight"].should == 7390
end
it "should force flush buffer when #flush(true) is called" do
ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1)).total.should == {}
14.times do
ModelHelper1.store("sheep.black", {:count => 1, :weight => 461}, @time.hours_ago(4))
end
ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1)).total.should == {}
Redistat.buffer.flush(true)
stats = ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1))
stats.total["count"].should == 14
stats.total["weight"].should == 6454
end
end
end end

View File

@@ -2,6 +2,12 @@
$LOAD_PATH.unshift(File.dirname(__FILE__)) $LOAD_PATH.unshift(File.dirname(__FILE__))
$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib')) $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
require 'simplecov'
SimpleCov.start do
add_filter '/spec'
add_filter '/vendor'
end
# require stuff # require stuff
require 'redistat' require 'redistat'
require 'rspec' require 'rspec'

View File

@@ -10,28 +10,43 @@ describe Redistat::Summary do
@date = Time.now @date = Time.now
@key = Redistat::Key.new(@scope, @label, @date, {:depth => :day}) @key = Redistat::Key.new(@scope, @label, @date, {:depth => :day})
@stats = {"views" => 3, "visitors" => 2} @stats = {"views" => 3, "visitors" => 2}
@expire = {:hour => 24*3600}
end end
it "should update a single summary properly" do it "should update a single summary properly" do
Redistat::Summary.update(@key, @stats, :hour) Redistat::Summary.send(:update_fields, @key, @stats, :hour)
summary = db.hgetall(@key.to_s(:hour)) summary = db.hgetall(@key.to_s(:hour))
summary.should have(2).items summary.should have(2).items
summary["views"].should == "3" summary["views"].should == "3"
summary["visitors"].should == "2" summary["visitors"].should == "2"
Redistat::Summary.update(@key, @stats, :hour) Redistat::Summary.send(:update_fields, @key, @stats, :hour)
summary = db.hgetall(@key.to_s(:hour)) summary = db.hgetall(@key.to_s(:hour))
summary.should have(2).items summary.should have(2).items
summary["views"].should == "6" summary["views"].should == "6"
summary["visitors"].should == "4" summary["visitors"].should == "4"
Redistat::Summary.update(@key, {"views" => -4, "visitors" => -3}, :hour) Redistat::Summary.send(:update_fields, @key, {"views" => -4, "visitors" => -3}, :hour)
summary = db.hgetall(@key.to_s(:hour)) summary = db.hgetall(@key.to_s(:hour))
summary.should have(2).items summary.should have(2).items
summary["views"].should == "2" summary["views"].should == "2"
summary["visitors"].should == "1" summary["visitors"].should == "1"
end end
it "should set key expiry properly" do
Redistat::Summary.update_all(@key, @stats, :hour,{:expire => @expire})
((24*3600)-1..(24*3600)+1).should include(db.ttl(@key.to_s(:hour)))
[:day, :month, :year].each do |depth|
db.ttl(@key.to_s(depth)).should == -1
end
db.flushdb
Redistat::Summary.update_all(@key, @stats, :hour, {:expire => {}})
[:hour, :day, :month, :year].each do |depth|
db.ttl(@key.to_s(depth)).should == -1
end
end
it "should update all summaries properly" do it "should update all summaries properly" do
Redistat::Summary.update_all(@key, @stats, :sec) Redistat::Summary.update_all(@key, @stats, :sec)
[:year, :month, :day, :hour, :min, :sec, :usec].each do |depth| [:year, :month, :day, :hour, :min, :sec, :usec].each do |depth|
@@ -48,7 +63,7 @@ describe Redistat::Summary do
it "should update summaries even if no label is set" do it "should update summaries even if no label is set" do
key = Redistat::Key.new(@scope, nil, @date, {:depth => :day}) key = Redistat::Key.new(@scope, nil, @date, {:depth => :day})
Redistat::Summary.update(key, @stats, :hour) Redistat::Summary.send(:update_fields, key, @stats, :hour)
summary = db.hgetall(key.to_s(:hour)) summary = db.hgetall(key.to_s(:hour))
summary.should have(2).items summary.should have(2).items
summary["views"].should == "3" summary["views"].should == "3"
@@ -121,12 +136,41 @@ describe Redistat::Summary do
summary["visitors/us"].should == "8" summary["visitors/us"].should == "8"
end end
it "should store label-based grouping enabled stats using a different group separator" do
Redistat.group_separator = '|'
stats = {"views" => 3, "visitors|eu" => 2, "visitors|us" => 4}
label = "views|about_us"
key = Redistat::Key.new(@scope, label, @date)
Redistat::Summary.update_all(key, stats, :hour)
key.groups[0].label.to_s.should == "views|about_us"
key.groups[1].label.to_s.should == "views"
child1 = key.groups[0]
parent = key.groups[1]
label = "views|contact"
key = Redistat::Key.new(@scope, label, @date)
Redistat::Summary.update_all(key, stats, :hour)
key.groups[0].label.to_s.should == "views|contact"
key.groups[1].label.to_s.should == "views"
child2 = key.groups[0]
summary = db.hgetall(child1.to_s(:hour))
summary["views"].should == "3"
summary["visitors|eu"].should == "2"
summary["visitors|us"].should == "4"
summary = db.hgetall(child2.to_s(:hour))
summary["views"].should == "3"
summary["visitors|eu"].should == "2"
summary["visitors|us"].should == "4"
summary = db.hgetall(parent.to_s(:hour))
summary["views"].should == "6"
summary["visitors|eu"].should == "4"
summary["visitors|us"].should == "8"
Redistat.group_separator = Redistat::GROUP_SEPARATOR
end
end end

125
spec/synchronize_spec.rb Normal file
View File

@@ -0,0 +1,125 @@
require "spec_helper"
module Redistat
describe Synchronize do
let(:klass) { Synchronize }
describe '.included' do
it 'includes InstanceMethods in passed object' do
base = mock('Base')
base.should_receive(:include).with(klass::InstanceMethods)
klass.included(base)
end
end # included
describe '.monitor' do
it 'returns a Monitor instance' do
klass.monitor.should be_a(Monitor)
end
it 'caches Monitor instance' do
klass.monitor.object_id.should == klass.monitor.object_id
end
end # monitor
describe '.thread_safe' do
after { klass.instance_variable_set('@thread_safe', nil) }
it 'returns value of @thread_safe' do
klass.instance_variable_set('@thread_safe', true)
klass.thread_safe.should be_true
end
it 'defaults to false' do
klass.thread_safe.should be_false
end
it 'uses #synchronize' do
klass.monitor.should_receive(:synchronize).once
klass.thread_safe.should be_nil
end
end # thread_safe
describe '.thread_safe=' do
after { klass.instance_variable_set('@thread_safe', nil) }
it 'sets @thread_safe' do
klass.instance_variable_get('@thread_safe').should be_nil
klass.thread_safe = true
klass.instance_variable_get('@thread_safe').should be_true
end
it 'uses #synchronize' do
klass.monitor.should_receive(:synchronize).once
klass.thread_safe = true
klass.instance_variable_get('@thread_safe').should be_nil
end
end # thread_safe=
describe "InstanceMethods" do
subject { SynchronizeSpecHelper.new }
describe '.monitor' do
it 'defers to Redistat::Synchronize' do
klass.should_receive(:monitor).once
subject.monitor
end
end # monitor
describe '.thread_safe' do
it ' defers to Redistat::Synchronize' do
klass.should_receive(:thread_safe).once
subject.thread_safe
end
end # thread_safe
describe '.thread_safe=' do
it 'defers to Redistat::Synchronize' do
klass.should_receive(:thread_safe=).once.with(true)
subject.thread_safe = true
end
end # thread_safe=
describe 'when #thread_safe is true' do
before { subject.stub(:thread_safe).and_return(true) }
describe '.synchronize' do
it 'defers to #monitor' do
subject.monitor.should_receive(:synchronize).once
subject.synchronize { 'foo' }
end
it 'passes block along to #monitor.synchronize' do
yielded = false
subject.synchronize { yielded = true }
yielded.should be_true
end
end # synchronize
end # when #thread_safe is true
describe 'when #thread_safe is false' do
before { subject.stub(:thread_safe).and_return(false) }
describe '.synchronize' do
it 'does not defer to #monitor' do
subject.monitor.should_not_receive(:synchronize)
subject.synchronize { 'foo' }
end
it 'yields block' do
yielded = false
subject.synchronize { yielded = true }
yielded.should be_true
end
end # synchronize
end # when #thread_safe is false
end
end # Synchronize
end # Redistat
class SynchronizeSpecHelper
include Redistat::Synchronize
end