Bump version to 0.4.0 - Add key expiry support

Prior to v2.1.3, any writes to keys with an expire/TTL set were
completely deleted before the write took place.

.gitignore

@@ -16,11 +16,12 @@ tmtags
 ## PROJECT::GENERAL
 coverage
 rdoc
-pkg
+pkg/*
+*.gem
+.bundle
+Gemfile.lock
 
 ## PROJECT::SPECIFIC
-.bundle/*
 .yardoc/*
 spec/db/*
-doc
-redistat.gemspec
+doc/*

.travis.yml

@@ -0,0 +1,8 @@
+language: ruby
+rvm:
+  - 1.8.7
+  - 1.9.2
+  - 1.9.3
+  - jruby-18mode
+  - jruby-19mode
+  - ree

Gemfile

@@ -1,12 +1,4 @@
 source 'http://rubygems.org/'
 
-gem 'activesupport', '>= 2.3.0'
-gem 'json', '>= 1.0.0'
-gem 'redis', '>= 2.0.0'
-gem 'time_ext', '>= 0.2.6'
-
-group :development do
-  gem 'rspec', '>= 2.0.1'
-  gem 'yard', '>= 0.6.1'
-  gem 'i18n'
-end
+# Specify your gem's dependencies in redistat.gemspec
+gemspec

Gemfile.lock

@@ -1,31 +0,0 @@
-GEM
-  remote: http://rubygems.org/
-  specs:
-    activesupport (3.0.3)
-    diff-lcs (1.1.2)
-    i18n (0.4.2)
-    json (1.4.6)
-    redis (2.1.1)
-    rspec (2.1.0)
-      rspec-core (~> 2.1.0)
-      rspec-expectations (~> 2.1.0)
-      rspec-mocks (~> 2.1.0)
-    rspec-core (2.1.0)
-    rspec-expectations (2.1.0)
-      diff-lcs (~> 1.1.2)
-    rspec-mocks (2.1.0)
-    time_ext (0.2.6)
-      activesupport (>= 2.3.0)
-    yard (0.6.3)
-
-PLATFORMS
-  ruby
-
-DEPENDENCIES
-  activesupport (>= 2.3.0)
-  i18n
-  json (>= 1.0.0)
-  redis (>= 2.0.0)
-  rspec (>= 2.0.1)
-  time_ext (>= 0.2.6)
-  yard (>= 0.6.1)

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2010 Jim Myhrberg.
+Copyright (c) 2011 Jim Myhrberg.
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

README.md

@@ -1,28 +1,351 @@
-# Redistat
+# Redistat [![Build Status](https://secure.travis-ci.org/jimeh/redistat.png)](http://travis-ci.org/jimeh/redistat)
 
 A Redis-backed statistics storage and querying library written in Ruby.
 
-## Early Beta
+Redistat was originally created to replace a small hacked together statistics
+collection solution which was MySQL-based. When I started I had a short list
+of requirements:
 
-Currently this is an early beta release. Readme and documentation is forthcoming.
+* Store and increment/decrement integer values (counters, etc)
+* Up to the second statistics available at all times
+* Screamingly fast
 
-For now, please check `spec/model_spec.rb` and `spec/model_helper.rb` to get started with how to use Redistat.
+Redis fits perfectly with all of these requirements. It has atomic operations
+like increment, and it's lightning fast, meaning if the data is structured
+well, the initial stats reporting call will store data in a format that's
+instantly retrievable just as fast.
+
+## Installation
+
+    gem install redistat
+
+If you are using Ruby 1.8.x, it's recommended you also install the
+`SystemTimer` gem, as the Redis gem will otherwise complain.
+
+## Usage (Crash Course)
+
+view\_stats.rb:
+
+```ruby
+require 'redistat'
+
+class ViewStats
+  include Redistat::Model
+end
+
+# if using Redistat in multiple threads set this
+# somewhere in the beginning of the execution stack
+Redistat.thread_safe = true
+```
+
+
+### Simple Example
+
+Store:
+
+```ruby
+ViewStats.store('hello', {:world => 4})
+ViewStats.store('hello', {:world => 2}, 2.hours.ago)
+```
+
+Fetch:
+
+```ruby
+ViewStats.find('hello', 1.hour.ago, 1.hour.from_now).all
+  #=> [{'world' => 4}]
+ViewStats.find('hello', 1.hour.ago, 1.hour.from_now).total
+  #=> {'world' => 4}
+ViewStats.find('hello', 3.hour.ago, 1.hour.from_now).total
+  #=> {'world' => 6}
+```
+
+
+### Advanced Example
+
+Store page view on product #44 from Chrome 11:
+
+```ruby
+ViewStats.store('views/product/44', {'count/chrome/11' => 1})
+```
+
+Fetch product #44 stats:
+
+```ruby
+ViewStats.find('views/product/44', 23.hours.ago, 1.hour.from_now).total
+  #=> { 'count' => 1, 'count/chrome' => 1, 'count/chrome/11' => 1 }
+```
+
+Store a page view on product #32 from Firefox 3:
+
+```ruby
+ViewStats.store('views/product/32', {'count/firefox/3' => 1})
+```
+
+Fetch product #32 stats:
+
+```ruby
+ViewStats.find('views/product/32', 23.hours.ago, 1.hour.from_now).total
+  #=> { 'count' => 1, 'count/firefox' => 1, 'count/firefox/3' => 1 }
+```
+
+Fetch stats for all products:
+
+```ruby
+ViewStats.find('views/product', 23.hours.ago, 1.hour.from_now).total
+  #=> { 'count'           => 2,
+  #     'count/chrome'    => 1,
+  #     'count/chrome/11' => 1,
+  #     'count/firefox'   => 1,
+  #     'count/firefox/3' => 1 }
+```
+
+Store a 404 error view:
+
+```ruby
+ViewStats.store('views/error/404', {'count/chrome/9' => 1})
+```
+
+Fetch stats for all views across the board:
+
+```ruby
+ViewStats.find('views', 23.hours.ago, 1.hour.from_now).total
+  #=> { 'count'           => 3,
+  #     'count/chrome'    => 2,
+  #     'count/chrome/9'  => 1,
+  #     'count/chrome/11' => 1,
+  #     'count/firefox'   => 1,
+  #     'count/firefox/3' => 1 }
+```
+
+Fetch list of products known to Redistat:
+
+```ruby
+finder = ViewStats.find('views/product', 23.hours.ago, 1.hour.from_now)
+finder.children.map { |child| child.label.me }
+  #=> [ "32", "44" ]
+finder.children.map { |child| child.label.to_s }
+  #=> [ "views/products/32", "views/products/44" ]
+finder.children.map { |child| child.total }
+  #=> [ { "count" => 1, "count/firefox" => 1, "count/firefox/3" => 1 },
+  #     { "count" => 1, "count/chrome"  => 1, "count/chrome/11" => 1 } ]
+```
+
+
+## Terminology
+
+### Scope
+
+A type of global-namespace for storing data. When using the `Redistat::Model`
+wrapper, the scope is automatically set to the class name. In the examples
+above, the scope is `ViewStats`. Can be overridden by calling the `#scope`
+class method on your model class.
+
+### Label
+
+Identifier string to separate different types and groups of statistics from
+each other. The first argument of the `#store`, `#find`, and `#fetch` methods
+is the label that you're storing to, or fetching from.
+
+Labels support multiple grouping levels by splitting the label string with `/`
+and storing the same stats for each level. For example, when storing data to a
+label called `views/product/44`, the data is stored for the label you specify,
+and also for `views/product` and `views`.
+
+A word of caution: Don't use a crazy number of group levels. As two levels
+causes twice as many `hincrby` calls to Redis as not using the grouping
+feature. Hence using 10 grouping levels, causes 10 times as many write calls
+to Redis.
+
+### Input Statistics Data
+
+You provide Redistat with the data you want to store using a Ruby Hash. This
+data is then stored in a corresponding Redis hash with identical key/field
+names.
+
+Key names in the hash also support grouping features similar to those
+available for Labels. Again, the more levels you use, the more write calls to
+Redis, so avoid using 10-15 levels.
+
+### Depth (Storage Accuracy)
+
+Define how accurately data should be stored, and how accurately it's looked up
+when fetching it again. By default Redistat uses a depth value of `:hour`,
+which means it's impossible to separate two events which were stored at 10:18
+and 10:23. In Redis they are both stored within a date key of `2011031610`.
+
+You can set depth within your model using the `#depth` class method. Available
+depths are: `:year`, `:month`, `:day`, `:hour`, `:min`, `:sec`
+
+### Time Ranges
+
+When you fetch data, you need to specify a start and an end time. The
+selection behavior can seem a bit weird at first when, but makes sense when
+you understand how Redistat works internally.
+
+For example, if we are using a Depth value of `:hour`, and we trigger a fetch
+call starting at `1.hour.ago` (13:34), till `Time.now` (14:34), only stats
+from 13:00:00 till 13:59:59 are returned, as they were all stored within the
+key for the 13th hour. If both 13:00 and 14:00 was returned, you would get
+results from two whole hours. Hence if you want up to the second data, use an
+end time of `1.hour.from_now`.
+
+### The Finder Object
+
+Calling the `#find` method on a Redistat model class returns a
+`Redistat::Finder` object. The finder is a lazy-loaded gateway to your
+data. Meaning you can create a new finder, and modify instantiated finder's
+label, scope, dates, and more. It does not call Redis and fetch the data until
+you call `#total`, `#all`, `#map`, `#each`, or `#each_with_index` on the
+finder.
+
+This section does need further expanding as there's a lot to cover when it
+comes to the finder.
+
+
+## Key Expiry
+
+Support for expiring keys from Redis is available, allowing you too keep
+varying levels of details for X period of time. This allows you easily keep
+things nice and tidy by only storing varying levels detailed stats only for as
+long as you need.
+
+In the below example we define how long Redis keys for varying depths are
+stored. Second by second stats are available for 10 minutes, minute by minute
+stats for 6 hours, hourly stats for 3 months, daily stats for 2 years, and
+yearly stats are retained forever.
+
+```ruby
+class ViewStats
+  include Redistat::Model
+
+  depth :sec
+
+  expire \
+    :sec => 10.minutes.to_i,
+    :min => 6.hours.to_i,
+    :hour => 3.months.to_i,
+    :day => 2.years.to_i
+end
+```
+
+Keep in mind that when storing stats for a custom date in the past for
+example, the expiry time for the keys will be relative to now. The values you
+specify are simply passed to the `Redis#expire` method.
+
+
+## Internals
+
+### Storing / Writing
+
+Redistat stores all data into a Redis hash keys. The Redis key name the used
+consists of three parts. The scope, label, and datetime:
+
+    {scope}/{label}:{datetime}
+
+For example, this...
+
+```ruby
+ViewStats.store('views/product/44', {'count/chrome/11' => 1})
+```
+
+...would store the follow hash of data...
+
+```ruby
+{ 'count' => 1, 'count/chrome' => 1, 'count/chrome/11' => 1 }
+```
+
+...to all 12 of these Redis hash keys...
+
+    ViewStats/views:2011
+    ViewStats/views:201103
+    ViewStats/views:20110315
+    ViewStats/views:2011031510
+    ViewStats/views/product:2011
+    ViewStats/views/product:201103
+    ViewStats/views/product:20110315
+    ViewStats/views/product:2011031510
+    ViewStats/views/product/44:2011
+    ViewStats/views/product/44:201103
+    ViewStats/views/product/44:20110315
+    ViewStats/views/product/44:2011031510
+
+...by creating the Redis key, and/or hash field if needed, otherwise it simply
+increments the already existing data.
+
+It would also create the following Redis sets to keep track of which child
+labels are available:
+
+    ViewStats.label_index:
+    ViewStats.label_index:views
+    ViewStats.label_index:views/product
+
+It should now be more obvious to you why you should think about how you use
+the grouping capabilities so you don't go crazy and use 10-15 levels. Storing
+is done through Redis' `hincrby` call, which only supports a single key/field
+combo. Meaning the above example would call `hincrby` a total of 36 times to
+store the data, and `sadd` a total of 3 times to ensure the label index is
+accurate. 39 calls is however not a problem for Redis, most calls happen in
+less than 0.15ms (0.00015 seconds) on my local machine.
+
+
+### Fetching / Reading
+
+By default when fetching statistics, Redistat will figure out how to do the
+least number of reads from Redis. First it checks how long range you're
+fetching. If whole days, months or years for example fit within the start and
+end dates specified, it will fetch the one key for the day/month/year in
+question. It further drills down to the smaller units.
+
+It is also intelligent enough to not fetch each day from 3-31 of a month,
+instead it would fetch the data for the whole month and the first two days,
+which are then removed from the summary of the whole month. This means three
+calls to `hgetall` instead of 29 if each whole day was fetched.
+
+### Buffer
+
+The buffer is a new, still semi-beta, feature aimed to reduce the number of
+Redis `hincrby` that Redistat sends. This should only really be useful when
+you're hitting north of 30,000 Redis requests per second, if your Redis server
+has limited resources, or against my recommendation you've opted to use 10,
+20, or more label grouping levels.
+
+Buffering tries to fold together multiple `store` calls into as few as
+possible by merging the statistics hashes from all calls and groups them based
+on scope, label, date depth, and more. You configure the the buffer by setting
+`Redistat.buffer_size` to an integer higher than 1. This basically tells
+Redistat how many `store` calls to buffer in memory before writing all data to
+Redis.
+
+
+## Todo
+
+* More details in Readme.
+* Documentation.
+* Anything else that becomes apparent after real-world use.
+
+
+## Credits
+
+[Global Personals](http://globalpersonals.co.uk/) deserves a thank
+you. Currently the primary user of Redistat, they've allowed me to spend some
+company time to further develop the project.
 
 
 ## Note on Patches/Pull Requests
- 
+
 * Fork the project.
 * Make your feature addition or bug fix.
 * Add tests for it. This is important so I don't break it in a
   future version unintentionally.
-* Commit, do not mess with rakefile, version, or history.
-  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
+* Commit, do not mess with rakefile, version, or history.  (if you want to
+  have your own version, that is fine but bump version in a commit by itself I
+  can ignore when I pull)
 * Send me a pull request. Bonus points for topic branches.
 
 
 ## License and Copyright
 
-Copyright (c) 2010 Jim Myhrberg.
+Copyright (c) 2011 Jim Myhrberg.
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

Rakefile

@@ -1,29 +1,11 @@
-require 'rubygems'
-require 'rake'
-
-begin
-  require 'jeweler'
-  Jeweler::Tasks.new do |gem|
-    gem.name = 'redistat'
-    gem.summary = 'A Redis-backed statistics storage and querying library written in Ruby.'
-    gem.description = 'A Redis-backed statistics storage and querying library written in Ruby.'
-    gem.email = 'contact@jimeh.me'
-    gem.homepage = 'http://github.com/jimeh/redistat'
-    gem.authors = ['Jim Myhrberg']
-    gem.add_dependency 'activesupport', '>= 2.3.0'
-    gem.add_dependency 'json', '>= 1.0.0'
-    gem.add_dependency 'redis', '>= 2.0.0'
-    gem.add_dependency 'time_ext', '>= 0.2.6'
-    gem.add_development_dependency 'rspec', '>= 2.0.1'
-    gem.add_development_dependency 'yard', '>= 0.6.1'
-  end
-  Jeweler::GemcutterTasks.new
-rescue LoadError
-  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
-end
+require 'bundler'
+Bundler::GemHelper.install_tasks
 
 
+#
 # Rspec
+#
+
 require 'rspec/core/rake_task'
 RSpec::Core::RakeTask.new(:spec) do |spec|
   spec.pattern = 'spec/**/*_spec.rb'
@@ -32,14 +14,16 @@ end
 RSpec::Core::RakeTask.new(:rcov) do |spec|
   spec.pattern = 'spec/**/*_spec.rb'
   spec.rcov = true
+  spec.rcov_opts = ['--exclude', 'spec']
 end
 
-task :spec => :check_dependencies
-
 task :default => [:start, :spec, :stop]
 
 
+#
 # Start/stop Redis test server
+#
+
 REDIS_DIR = File.expand_path(File.join("..", "spec"), __FILE__)
 REDIS_CNF = File.join(REDIS_DIR, "redis-test.conf")
 REDIS_PID = File.join(REDIS_DIR, "db", "redis.pid")
@@ -60,7 +44,10 @@ task :stop do
 end
 
 
-# YARD Documentation
+#
+# Yard
+#
+
 begin
   require 'yard'
   YARD::Rake::YardocTask.new
@@ -69,3 +56,14 @@ rescue LoadError
     abort "YARD is not available. In order to run yardoc, you must: sudo gem install yard"
   end
 end
+
+
+#
+# Misc.
+#
+
+desc "Start an irb console with Redistat pre-loaded."
+task :console do
+  exec "irb -r spec/spec_helper"
+end
+task :c => :console

VERSION

@@ -1 +0,0 @@
-0.0.2

lib/redistat.rb

@@ -1,104 +1,101 @@
 
 require 'rubygems'
-require 'active_support'
-require 'active_support/hash_with_indifferent_access' if !Hash.respond_to?(:with_indifferent_access) # Active Support 2.x and 3.x
-require 'redis'
 require 'date'
 require 'time'
-require 'time_ext'
-require 'json'
 require 'digest/sha1'
+require 'monitor'
 
+# Active Support 2.x or 3.x
+require 'active_support'
+if !{}.respond_to?(:with_indifferent_access)
+  require 'active_support/core_ext/hash/indifferent_access'
+  require 'active_support/core_ext/hash/reverse_merge'
+end
+
+require 'time_ext'
+require 'redis'
+require 'json'
+
+require 'redistat/mixins/options'
+require 'redistat/mixins/synchronize'
+require 'redistat/mixins/database'
+require 'redistat/mixins/date_helper'
+
+require 'redistat/connection'
+require 'redistat/buffer'
 require 'redistat/collection'
-require 'redistat/database'
 require 'redistat/date'
 require 'redistat/event'
 require 'redistat/finder'
-require 'redistat/finder/date_set'
 require 'redistat/key'
 require 'redistat/label'
 require 'redistat/model'
 require 'redistat/result'
 require 'redistat/scope'
 require 'redistat/summary'
+require 'redistat/version'
+
+require 'redistat/core_ext'
 
-require 'redistat/core_ext/date'
-require 'redistat/core_ext/time'
-require 'redistat/core_ext/fixnum'
 
 module Redistat
-  
+
   KEY_NEXT_ID = ".next_id"
   KEY_EVENT = ".event:"
-  KEY_LEBELS = "Redistat.lables:"
+  KEY_LABELS = "Redistat.labels:" # used for reverse label hash lookup
   KEY_EVENT_IDS = ".event_ids"
-  
+  LABEL_INDEX = ".label_index:"
+  GROUP_SEPARATOR = "/"
+
   class InvalidOptions < ArgumentError; end
+  class RedisServerIsTooOld < Exception; end
+
+  class << self
+
+    def buffer
+      Buffer.instance
+    end
+
+    def buffer_size
+      buffer.size
+    end
+
+    def buffer_size=(size)
+      buffer.size = size
+    end
+
+    def thread_safe
+      Synchronize.thread_safe
+    end
+
+    def thread_safe=(value)
+      Synchronize.thread_safe = value
+    end
+
+    def connection(ref = nil)
+      Connection.get(ref)
+    end
+    alias :redis :connection
+
+    def connection=(connection)
+      Connection.add(connection)
+    end
+    alias :redis= :connection=
+
+    def connect(options)
+      Connection.create(options)
+    end
+
+    def flush
+      puts "WARNING: Redistat.flush is deprecated. Use Redistat.redis.flushdb instead."
+      connection.flushdb
+    end
 
-  # Provides access to the Redis database. This is shared accross all models and instances.
-  def redis
-    threaded[:redis] ||= connection(*options)
   end
-
-  def redis=(connection)
-    threaded[:redis] = connection
-  end
-
-  def threaded
-    Thread.current[:redistat] ||= {}
-  end
-
-  # Connect to a redis database.
-  #
-  # @param options [Hash] options to create a message with.
-  # @option options [#to_s] :host ('127.0.0.1') Host of the redis database.
-  # @option options [#to_s] :port (6379) Port number.
-  # @option options [#to_s] :db (0) Database number.
-  # @option options [#to_s] :timeout (0) Database timeout in seconds.
-  # @example Connect to a database in port 6380.
-  #   Redistat.connect(:port => 6380)
-  def connect(*options)
-    self.redis = nil
-    @options = options
-  end
-
-  # Return a connection to Redis.
-  #
-  # This is a wapper around Redis.new(options)
-  def connection(*options)
-    Redis.new(*options)
-  end
-
-  def options
-    @options = [] unless defined? @options
-    @options
-  end
-
-  # Clear the database.
-  def flush
-    redis.flushdb
-  end
-  
-  module_function :connect, :connection, :flush, :redis, :redis=, :options, :threaded
-
 end
 
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+# ensure buffer is flushed on program exit
+Kernel.at_exit do
+  Redistat.buffer.flush(true)
+end

lib/redistat/buffer.rb

@@ -0,0 +1,110 @@
+require 'redistat/core_ext/hash'
+
+module Redistat
+  class Buffer
+    include Synchronize
+
+    def self.instance
+      @instance ||= self.new
+    end
+
+    def size
+      synchronize do
+        @size ||= 0
+      end
+    end
+
+    def size=(value)
+      synchronize do
+        @size = value
+      end
+    end
+
+    def count
+      @count ||= 0
+    end
+
+    def store(key, stats, depth_limit, opts)
+      return false unless should_buffer?
+
+      to_flush = {}
+      buffkey = buffer_key(key, opts)
+
+      synchronize do
+        if !queue.has_key?(buffkey)
+          queue[buffkey] = { :key         => key,
+                             :stats       => {},
+                             :depth_limit => depth_limit,
+                             :opts        => opts }
+        end
+
+        queue[buffkey][:stats].merge_and_incr!(stats)
+        incr_count
+
+        # return items to be flushed if buffer size limit has been reached
+        to_flush = reset_queue
+      end
+
+      # flush any data that's been cleared from the queue
+      flush_data(to_flush)
+      true
+    end
+
+    def flush(force = false)
+      to_flush = {}
+      synchronize do
+        to_flush = reset_queue(force)
+      end
+      flush_data(to_flush)
+    end
+
+    private
+
+    # should always be called from within a synchronize block
+    def incr_count
+      @count ||= 0
+      @count += 1
+    end
+
+    def queue
+      @queue ||= {}
+    end
+
+    def should_buffer?
+      size > 1 # buffer size of 1 would be equal to not using buffer
+    end
+
+    # should always be called from within a synchronize block
+    def should_flush?
+      (!queue.blank? && count >= size)
+    end
+
+    # returns items to be flushed if buffer size limit has been reached
+    # should always be called from within a synchronize block
+    def reset_queue(force = false)
+      return {} if !force && !should_flush?
+      data = queue
+      @queue = {}
+      @count = 0
+      data
+    end
+
+    def flush_data(buffer_data)
+      buffer_data.each do |k, item|
+        Summary.update(item[:key], item[:stats], item[:depth_limit], item[:opts])
+      end
+    end
+
+    # depth_limit is not needed as it's evident in key.to_s
+    def buffer_key(key, opts)
+      # covert keys to strings, as sorting a Hash with Symbol keys fails on
+      # Ruby 1.8.x.
+      opts = opts.inject({}) do |result, (k, v)|
+        result[k.to_s] = v
+        result
+      end
+      "#{key.to_s}:#{opts.sort.flatten.join(':')}"
+    end
+
+  end
+end

lib/redistat/collection.rb

@@ -1,16 +1,20 @@
 module Redistat
   class Collection < ::Array
-    
+
     attr_accessor :from
     attr_accessor :till
     attr_accessor :depth
     attr_accessor :total
-    
+
     def initialize(options = {})
       @from = options[:from] ||= nil
       @till = options[:till] ||= nil
       @depth = options[:depth] ||= nil
     end
-    
+
+    def total
+      @total ||= {}
+    end
+
   end
-end
+end

lib/redistat/connection.rb

@@ -0,0 +1,89 @@
+require 'monitor'
+
+module Redistat
+  module Connection
+
+    REQUIRED_SERVER_VERSION = "1.3.10"
+    MIN_EXPIRE_SERVER_VERSION = "2.1.3"
+
+    # TODO: Create a ConnectionPool instance object using Sychronize mixin to replace Connection class
+
+    class << self
+
+      # TODO: clean/remove all ref-less connections
+
+      def get(ref = nil)
+        ref ||= :default
+        synchronize do
+          connections[references[ref]] || create
+        end
+      end
+
+      def add(conn, ref = nil)
+        ref ||= :default
+        synchronize do
+          check_redis_version(conn)
+          references[ref] = conn.client.id
+          connections[conn.client.id] = conn
+        end
+      end
+
+      def create(options = {})
+        synchronize do
+          options = options.clone
+          ref = options.delete(:ref) || :default
+          options.reverse_merge!(default_options)
+          conn = (connections[connection_id(options)] ||= connection(options))
+          references[ref] = conn.client.id
+          conn
+        end
+      end
+
+      def connections
+        @connections ||= {}
+      end
+
+      def references
+        @references ||= {}
+      end
+
+      private
+
+      def monitor
+        @monitor ||= Monitor.new
+      end
+
+      def synchronize(&block)
+        monitor.synchronize(&block)
+      end
+
+      def connection(options)
+        check_redis_version(Redis.new(options))
+      end
+
+      def connection_id(options = {})
+        options = options.reverse_merge(default_options)
+        "redis://#{options[:host]}:#{options[:port]}/#{options[:db]}"
+      end
+
+      def check_redis_version(conn)
+        raise RedisServerIsTooOld if conn.info["redis_version"] < REQUIRED_SERVER_VERSION
+        if conn.info["redis_version"] < MIN_EXPIRE_SERVER_VERSION
+          STDOUT.puts "WARNING: You MUST upgrade Redis to v2.1.3 or later " +
+            "if you are using key expiry."
+        end
+        conn
+      end
+
+      def default_options
+        {
+          :host => '127.0.0.1',
+          :port => 6379,
+          :db => 0,
+          :timeout => 5
+        }
+      end
+
+    end
+  end
+end

lib/redistat/core_ext.rb

@@ -0,0 +1,5 @@
+require 'redistat/core_ext/bignum'
+require 'redistat/core_ext/date'
+require 'redistat/core_ext/fixnum'
+require 'redistat/core_ext/hash'
+require 'redistat/core_ext/time'

lib/redistat/core_ext/bignum.rb

@@ -0,0 +1,8 @@
+class Bignum
+  include Redistat::DateHelper
+
+  def to_time
+    Time.at(self)
+  end
+
+end

lib/redistat/core_ext/date.rb

@@ -1,8 +1,8 @@
 class Date
   include Redistat::DateHelper
-  
+
   def to_time
     Time.parse(self.to_s)
   end
-  
+
 end

lib/redistat/core_ext/fixnum.rb

@@ -1,8 +1,8 @@
 class Fixnum
   include Redistat::DateHelper
-  
+
   def to_time
     Time.at(self)
   end
-  
+
 end

lib/redistat/core_ext/hash.rb

@@ -0,0 +1,23 @@
+class Hash
+
+  def merge_and_incr(hash)
+    self.clone.merge_and_incr!(hash)
+  end
+
+  def merge_and_incr!(hash)
+    raise ArgumentError unless hash.is_a?(Hash)
+    hash.each do |key, value|
+      self[key] = value unless self.set_or_incr(key, value)
+    end
+    self
+  end
+
+  def set_or_incr(key, value)
+    return false unless value.is_a?(Numeric)
+    self[key] = 0 unless self.has_key?(key)
+    return false unless self[key].is_a?(Numeric)
+    self[key] += value
+    true
+  end
+
+end

lib/redistat/database.rb

@@ -1,10 +0,0 @@
-module Redistat
-  module Database
-    def self.included(base)
-      base.extend(Database)
-    end
-    def db
-      Redistat.redis
-    end
-  end
-end

lib/redistat/date.rb

@@ -1,6 +1,6 @@
 module Redistat
   class Date
-    
+
     attr_accessor :year
     attr_accessor :month
     attr_accessor :day
@@ -9,9 +9,9 @@ module Redistat
     attr_accessor :sec
     attr_accessor :usec
     attr_accessor :depth
-    
+
     DEPTHS = [:year, :month, :day, :hour, :min, :sec, :usec]
-    
+
     def initialize(input, depth = nil)
       @depth = depth
       if input.is_a?(::Time)
@@ -22,14 +22,16 @@ module Redistat
         from_string(input)
       elsif input.is_a?(::Fixnum)
         from_integer(input)
+      elsif input.is_a?(::Bignum)
+        from_integer(input)
       end
     end
-    
+
     def to_t
       ::Time.local(@year, @month, @day, @hour, @min, @sec, @usec)
     end
     alias :to_time :to_t
-    
+
     def to_d
       ::Date.civil(@year, @month, @day)
     end
@@ -39,7 +41,7 @@ module Redistat
       to_time.to_i
     end
     alias :to_integer :to_i
-    
+
     def to_s(depth = nil)
       depth ||= @depth ||= :sec
       output = ""
@@ -55,9 +57,9 @@ module Redistat
       output
     end
     alias :to_string :to_s
-    
+
     private
-    
+
     def from_time(input)
       DEPTHS.each do |k|
         send("#{k}=", input.send(k))
@@ -72,22 +74,15 @@ module Redistat
         send("#{k}=", 0)
       end
     end
-    
+
     def from_integer(input)
       from_time(::Time.at(input))
     end
-    
+
     def from_string(input)
       input += "19700101000000"[input.size..-1] if input =~ /^\d\d\d[\d]+$/i
       from_time(::Time.parse(input))
     end
-    
-  end
-  
-  module DateHelper
-    def to_redistat(depth = nil)
-      Redistat::Date.new(self, depth)
-    end
-    alias :to_rs :to_redistat
+
   end
 end

lib/redistat/event.rb

@@ -1,54 +1,58 @@
 module Redistat
   class Event
     include Database
-    
+    include Options
+
     attr_reader :id
     attr_reader :key
-    
+
     attr_accessor :stats
     attr_accessor :meta
-    attr_accessor :options
-    
-    def initialize(scope, label = nil, date = nil, stats = {}, options = {}, meta = {}, is_new = true)
-      @options = default_options.merge(options)
+
+    def default_options
+      { :depth => :hour,
+        :store_event => false,
+        :connection_ref => nil,
+        :enable_grouping => true,
+        :label_indexing => true }
+    end
+
+    def initialize(scope, label = nil, date = nil, stats = {}, opts = {}, meta = {}, is_new = true)
+      parse_options(opts)
       @key = Key.new(scope, label, date, @options)
       @stats = stats ||= {}
       @meta = meta ||= {}
       @new = is_new
     end
 
-    def default_options
-      { :depth => :hour, :store_event => false }
-    end
-    
     def new?
       @new
     end
-    
+
     def date
       @key.date
     end
-    
+
     def date=(input)
       @key.date = input
     end
-    
+
     def scope
       @key.scope
     end
-    
+
     def scope=(input)
       @key.scope = input
     end
-    
+
     def label
       @key.label
     end
-    
+
     def label_hash
       @key.label_hash
     end
-    
+
     def label=(input)
       @key.label = input
     end
@@ -56,10 +60,10 @@ module Redistat
     def next_id
       db.incr("#{self.scope}#{KEY_NEXT_ID}")
     end
-    
+
     def save
       return false if !self.new?
-      Summary.update_all(@key, @stats, depth_limit)
+      Summary.update_all(@key, @stats, depth_limit, @options)
       if @options[:store_event]
         @id = self.next_id
         db.hmset("#{self.scope}#{KEY_EVENT}#{@id}",
@@ -74,21 +78,21 @@ module Redistat
       @new = false
       self
     end
-    
+
     def depth_limit
       @options[:depth] ||= @key.depth
     end
-    
+
     def self.create(*args)
       self.new(*args).save
     end
-    
+
     def self.find(scope, id)
       event = db.hgetall "#{scope}#{KEY_EVENT}#{id}"
       return nil if event.size == 0
       self.new( event["scope"], event["label"], event["date"], JSON.parse(event["stats"]),
-                JSON.parse(event["meta"]), JSON.parse(event["options"]), false )
+                JSON.parse(event["options"]), JSON.parse(event["meta"]), false )
     end
-    
+
   end
-end
+end

lib/redistat/finder.rb

@@ -1,34 +1,166 @@
+require 'redistat/finder/date_set'
+
 module Redistat
   class Finder
     include Database
-    
+
+    class << self
+      def find(*args)
+        new.find(*args)
+      end
+
+      def scope(scope)
+        new.scope(scope)
+      end
+
+      def label(label)
+        new.label(label)
+      end
+
+      def dates(from, till)
+        new.dates(from, till)
+      end
+      alias :date :dates
+
+      def from(date)
+        new.from(date)
+      end
+
+      def till(date)
+        new.till(date)
+      end
+      alias :untill :till
+
+      def depth(unit)
+        new.depth(unit)
+      end
+
+      def interval(unit)
+        new.interval(unit)
+      end
+    end
+
     attr_reader :options
-    
-    def initialize(options = {})
-      @options = options
+
+    def initialize(opts = {})
+      set_options(opts)
     end
-    
-    def valid_options?
-      return true if !@options[:scope].blank? && !@options[:label].blank? && !@options[:from].blank? && !@options[:till].blank?
-      false
+
+    def options
+      @options ||= {}
     end
-    
-    def find(options = {})
-      @options.merge!(options)
+
+    def all(reload = false)
+      @result = nil if reload
+      @result ||= find
+    end
+
+    def total
+      all.total
+    end
+
+    def each(&block)
+      all.each(&block)
+    end
+
+    def map(&block)
+      all.map(&block)
+    end
+
+    def each_with_index(&block)
+      all.each_with_index(&block)
+    end
+
+    def parent
+      @parent ||= self.class.new(options.merge(:label => options[:label].parent)) unless options[:label].nil?
+    end
+
+    def children
+      build_key.children.map { |key|
+        self.class.new(options.merge(:label => key.label.to_s))
+      }
+    end
+
+    def connection_ref(ref = nil)
+      return options[:connection_ref] if ref.nil?
+      reset! if options[:connection_ref] != ref
+      options[:connection_ref] = ref
+      self
+    end
+
+    def scope(input = nil)
+      return options[:scope] if input.nil?
+      reset! if !options[:scope].nil? && options[:scope].to_s != input.to_s
+      options[:scope] = Scope.new(input)
+      self
+    end
+
+    def label(input = nil)
+      return options[:label] if input.nil?
+      reset! if options.has_key?(:label) && options[:label].to_s != input.to_s
+      options[:label] = (!input.nil?) ? Label.new(input) : nil
+      self
+    end
+
+    def dates(start, finish)
+      from(start).till(finish)
+    end
+    alias :date :dates
+
+    def from(date = nil)
+      return options[:from] if date.nil?
+      reset! if options[:from] != date
+      options[:from] = date
+      self
+    end
+
+    def till(date = nil)
+      return options[:till] if date.nil?
+      reset! if options[:till] != date
+      options[:till] = date
+      self
+    end
+    alias :until :till
+
+    def depth(unit = nil)
+      return options[:depth] if unit.nil?
+      reset! if options[:depth] != unit
+      options[:depth] = unit
+      self
+    end
+
+    def interval(unit = nil)
+      return options[:interval] if unit.nil?
+      reset! if options[:interval] != unit
+      options[:interval] = unit
+      self
+    end
+
+    def find(opts = {})
+      set_options(opts)
       raise InvalidOptions.new if !valid_options?
-      if @options[:interval].nil? || !@options[:interval]
+      if options[:interval].nil? || !options[:interval]
         find_by_magic
       else
         find_by_interval
       end
     end
-    
-    def find_by_interval(options = {})
-      @options.merge!(options)
+
+    private
+
+    def set_options(opts = {})
+      opts = opts.clone
+      opts.each do |key, value|
+        self.send(key, opts.delete(key)) if self.respond_to?(key)
+      end
+      self.options.merge!(opts)
+    end
+
+    def find_by_interval
       raise InvalidOptions.new if !valid_options?
       key = build_key
-      col = Collection.new(@options)
-      col.total = Result.new(@options)
+      col = Collection.new(options)
+      col.total = Result.new(options)
       build_date_sets.each do |set|
         set[:add].each do |date|
           result = Result.new
@@ -42,13 +174,12 @@ module Redistat
       end
       col
     end
-    
-    def find_by_magic(options = {})
-      @options.merge!(options)
+
+    def find_by_magic
       raise InvalidOptions.new if !valid_options?
-      key = Key.new(@options[:scope], @options[:label])
-      col = Collection.new(@options)
-      col.total = Result.new(@options)
+      key = build_key
+      col = Collection.new(options)
+      col.total = Result.new(options)
       col << col.total
       build_date_sets.each do |set|
         sum = Result.new
@@ -60,15 +191,25 @@ module Redistat
       end
       col
     end
-    
+
+    def reset!
+      @result = nil
+      @parent = nil
+    end
+
+    def valid_options?
+      return true if !options[:scope].blank? && !options[:label].blank? && !options[:from].blank? && !options[:till].blank?
+      false
+    end
+
     def build_date_sets
-      Finder::DateSet.new(@options[:from], @options[:till], @options[:depth], @options[:interval])
+      Finder::DateSet.new(options[:from], options[:till], options[:depth], options[:interval])
     end
-    
+
     def build_key
-      Key.new(@options[:scope], @options[:label])
+      Key.new(options[:scope], options[:label])
     end
-    
+
     def summarize_add_keys(sets, key, sum)
       sets.each do |date|
         db.hgetall("#{key.prefix}#{date}").each do |k, v|
@@ -77,7 +218,7 @@ module Redistat
       end
       sum
     end
-    
+
     def summarize_rem_keys(sets, key, sum)
       sets.each do |date|
         db.hgetall("#{key.prefix}#{date}").each do |k, v|
@@ -86,80 +227,10 @@ module Redistat
       end
       sum
     end
-    
-    class << self
-    
-      def find(*args)
-        new.find(*args)
-      end
-      
-      def scope(scope)
-        new.scope(scope)
-      end
-      
-      def label(label)
-        new.label(label)
-      end
-      
-      def dates(from, till)
-        new.dates(from, till)
-      end
-      alias :date :dates
-      
-      def from(date)
-        new.from(date)
-      end
-      
-      def till(date)
-        new.till(date)
-      end
-      alias :untill :till
-      
-      def depth(unit)
-        new.depth(unit)
-      end
-      
-      def interval(unit)
-        new.interval(unit)
-      end
-      
+
+    def db
+      super(options[:connection_ref])
     end
-    
-    def scope(scope)
-      @options[:scope] = scope
-      self
-    end
-    
-    def label(label)
-      @options[:label] = label
-      self
-    end
-    
-    def dates(from, till)
-      from(from).till(till)
-    end
-    alias :date :dates
-    
-    def from(date)
-      @options[:from] = date
-      self
-    end
-    
-    def till(date)
-      @options[:till] = date
-      self
-    end
-    alias :until :till
-    
-    def depth(unit)
-      @options[:depth] = unit
-      self
-    end
-    
-    def interval(unit)
-      @options[:interval] = unit
-      self
-    end
-    
+
   end
-end
+end

lib/redistat/finder/date_set.rb

@@ -1,7 +1,7 @@
 module Redistat
   class Finder
     class DateSet < Array
-      
+
       def initialize(start_date = nil, end_date = nil, depth = nil, interval = false)
         if !start_date.nil? && !end_date.nil?
           find_date_sets(start_date, end_date, depth, interval)
@@ -9,6 +9,10 @@ module Redistat
       end
 
       def find_date_sets(start_date, end_date, depth = nil, interval = false)
+        if depth.nil? && interval.is_a?(Symbol)
+          depth = interval
+          interval = true
+        end
         start_date = start_date.to_time if start_date.is_a?(::Date)
         end_date = end_date.to_time if end_date.is_a?(::Date)
         if !interval
@@ -42,7 +46,7 @@ module Redistat
         return find_start_year_for(start_date, end_date, lowest_depth) if unit == :year
         index = Date::DEPTHS.index(unit)
         nunit = Date::DEPTHS[(index > 0) ? index-1 : index]
-        if start_date < start_date.round(nunit) || start_date.next(nunit).beginning_of(nunit) > end_date.beginning_of(nunit)
+        if start_date < start_date.beginning_of_closest(nunit) || start_date.next(nunit).beginning_of(nunit) > end_date.beginning_of(nunit)
           add = []
           start_date.beginning_of_each(unit, :include_start => lowest_depth).until(start_date.end_of(nunit)) do |t|
             add << t.to_rs.to_s(unit) if t < end_date.beginning_of(unit)
@@ -59,7 +63,7 @@ module Redistat
         index = Date::DEPTHS.index(unit)
         nunit = Date::DEPTHS[(index > 0) ? index-1 : index]
         has_nunit = end_date.prev(nunit).beginning_of(nunit) >= start_date.beginning_of(nunit)
-        nearest_nunit = end_date.round(nunit)
+        nearest_nunit = end_date.beginning_of_closest(nunit)
         if end_date >= nearest_nunit && has_nunit
           add = []
           end_date.beginning_of(nunit).beginning_of_each(unit, :include_start => true, :include_end => lowest_depth).until(end_date) do |t|
@@ -67,7 +71,7 @@ module Redistat
           end
           { :add => add, :rem => [] }
         elsif has_nunit
-          { :add => [end_date.beginning_of(nunit).to_rs.to_s(nunit)], 
+          { :add => [end_date.beginning_of(nunit).to_rs.to_s(nunit)],
             :rem => end_date.map_beginning_of_each(unit, :include_start => !lowest_depth).until(end_date.end_of(nunit)) { |t| t.to_rs.to_s(unit) } }
         else
           { :add => [], :rem => [] }
@@ -89,7 +93,7 @@ module Redistat
           { :add => [], :rem => [] }
         end
       end
-      
+
     end
   end
-end
+end

lib/redistat/key.rb

@@ -1,54 +1,84 @@
 module Redistat
   class Key
-    
-    attr_accessor :scope
-    attr_accessor :date
-    attr_accessor :options
-    
-    def initialize(scope, label_name = nil, time_stamp = nil, options = {})
-      @options = default_options.merge(options || {})
-      @scope = scope
+    include Database
+    include Options
+
+    def default_options
+      { :depth => :hour }
+    end
+
+    def initialize(scope, label_name = nil, time_stamp = nil, opts = {})
+      parse_options(opts)
+      self.scope = scope
       self.label = label_name if !label_name.nil?
       self.date = time_stamp ||= Time.now
     end
-    
-    def default_options
-      { :depth => :hour, :hashed_label => false }
-    end
-    
+
     def prefix
       key = "#{@scope}"
-      key << "/#{label}" if !label.nil?
+      key << "/#{label.name}" if !label.nil?
       key << ":"
       key
     end
-    
+
     def date=(input)
       @date = (input.instance_of?(Redistat::Date)) ? input : Date.new(input) # Redistat::Date, not ::Date
     end
-    
+    attr_reader :date
+
     def depth
-      @options[:depth]
+      options[:depth]
     end
-    
-    def label
-      @label.name
+
+    # def scope
+    #   @scope.to_s
+    # end
+
+    def scope=(input)
+      @scope = (input.instance_of?(Redistat::Scope)) ? input : Scope.new(input)
     end
-    
-    def label_hash
-      @label.hash
-    end
-    
+    attr_reader :scope
+
     def label=(input)
       @label = (input.instance_of?(Redistat::Label)) ? input : Label.create(input, @options)
     end
-    
+    attr_reader :label
+
+    def label_hash
+      @label.hash
+    end
+
+    def parent
+      @parent ||= self.class.new(self.scope, @label.parent, self.date, @options) unless @label.parent.nil?
+    end
+
+    def children
+      members = db.smembers("#{scope}#{LABEL_INDEX}#{@label}") || [] # older versions of Redis returns nil
+      members.map { |member|
+        child_label = [@label, member].reject { |i| i.nil? }
+        self.class.new(self.scope, child_label.join(GROUP_SEPARATOR), self.date, @options)
+      }
+    end
+
+    def update_index
+      @label.groups.each do |label|
+        parent = (label.parent || "")
+        db.sadd("#{scope}#{LABEL_INDEX}#{parent}", label.me)
+      end
+    end
+
+    def groups
+      @groups ||= @label.groups.map do |label|
+        self.class.new(@scope, label, self.date, @options)
+      end
+    end
+
     def to_s(depth = nil)
       depth ||= @options[:depth]
       key = self.prefix
       key << @date.to_s(depth)
       key
     end
-    
+
   end
-end
+end

lib/redistat/label.rb

@@ -1,34 +1,69 @@
 module Redistat
   class Label
     include Database
-    
-    attr_reader :raw
-    
-    def initialize(str, options = {})
-      @options = options
+    include Options
+
+    def default_options
+      { :hashed_label => false }
+    end
+
+    def self.create(name, opts = {})
+      self.new(name, opts).save
+    end
+
+    def self.join(*args)
+      args = args.map {|i| i.to_s}
+      self.new(args.reject {|i| i.blank? }.join(GROUP_SEPARATOR))
+    end
+
+    def initialize(str, opts = {})
+      parse_options(opts)
       @raw = str.to_s
     end
-    
+
+    def to_s
+      @raw
+    end
+
     def name
-      @options[:hashed_label] ? hash : @raw
+      @options[:hashed_label] ? hash : self.to_s
     end
-    
+
     def hash
-      @hash ||= Digest::SHA1.hexdigest(@raw)
+      @hash ||= Digest::SHA1.hexdigest(self.to_s)
     end
-    
+
     def save
-      @saved = (db.set("#{KEY_LEBELS}#{hash}", @raw) == "OK") if @options[:hashed_label]
+      @saved = db.hset(KEY_LABELS, hash, self.to_s) if @options[:hashed_label]
       self
     end
-    
+
     def saved?
+      return true unless @options[:hashed_label]
       @saved ||= false
     end
-    
-    def self.create(name, options = {})
-      self.new(name, options).save
+
+    def parent
+      @parent ||= groups[1] if groups.size > 1
     end
-    
+
+    def me
+      self.to_s.split(GROUP_SEPARATOR).last
+    end
+
+    def groups
+      return @groups unless @groups.nil?
+      @groups = []
+      parent = ""
+      self.to_s.split(GROUP_SEPARATOR).each do |part|
+        if !part.blank?
+          group = ((parent.blank?) ? "" : "#{parent}#{GROUP_SEPARATOR}") + part
+          @groups << Label.new(group)
+          parent = group
+        end
+      end
+      @groups.reverse!
+    end
+
   end
-end
+end

lib/redistat/mixins/database.rb

@@ -0,0 +1,11 @@
+module Redistat
+  module Database
+    def self.included(base)
+      base.extend(Database)
+    end
+    def db(ref = nil)
+      ref ||= @options[:connection_ref] if !@options.nil?
+      Redistat.connection(ref)
+    end
+  end
+end

lib/redistat/mixins/date_helper.rb

@@ -0,0 +1,8 @@
+module Redistat
+  module DateHelper
+    def to_redistat(depth = nil)
+      Redistat::Date.new(self, depth)
+    end
+    alias :to_rs :to_redistat
+  end
+end

lib/redistat/mixins/options.rb

@@ -0,0 +1,41 @@
+module Redistat
+  module Options
+
+    def self.included(base)
+      base.extend(ClassMethods)
+    end
+
+    module ClassMethods
+      def option_accessor(*opts)
+        opts.each do |option|
+          define_method(option) do |*args|
+            if !args.first.nil?
+              options[option.to_sym] = args.first
+            else
+              options[option.to_sym] || nil
+            end
+          end
+        end
+      end
+    end
+
+    def parse_options(opts)
+      opts ||= {}
+      @raw_options = opts
+      @options = default_options.merge(opts.reject { |k,v| v.nil? })
+    end
+
+    def default_options
+      {}
+    end
+
+    def options
+      @options ||= {}
+    end
+
+    def raw_options
+      @raw_options ||= {}
+    end
+
+  end
+end

lib/redistat/mixins/synchronize.rb

@@ -0,0 +1,52 @@
+require 'monitor'
+
+module Redistat
+  module Synchronize
+
+    class << self
+      def included(base)
+        base.send(:include, InstanceMethods)
+      end
+
+      def monitor
+        @monitor ||= Monitor.new
+      end
+
+      def thread_safe
+        monitor.synchronize do
+          return @thread_safe unless @thread_safe.nil?
+          @thread_safe = false
+        end
+      end
+
+      def thread_safe=(value)
+        monitor.synchronize do
+          @thread_safe = value
+        end
+      end
+    end # << self
+
+    module InstanceMethods
+      def thread_safe
+        Synchronize.thread_safe
+      end
+
+      def thread_safe=(value)
+        Synchronize.thread_safe = value
+      end
+
+      def monitor
+        Synchronize.monitor
+      end
+
+      def synchronize(&block)
+        if thread_safe
+          monitor.synchronize(&block)
+        else
+          block.call
+        end
+      end
+    end # InstanceMethods
+
+  end
+end

lib/redistat/model.rb

@@ -1,58 +1,77 @@
 module Redistat
   module Model
-    
+    include Database
+    include Options
+
     def self.included(base)
       base.extend(self)
     end
-    
-    def store(label, stats = {}, date = nil, meta = {}, opts = {})
-      Event.new(name, label, date, stats, options.merge(opts), meta).save
+
+
+    #
+    # statistics store/fetch methods
+    #
+
+    def store(label, stats = {}, date = nil, opts = {}, meta = {})
+      Event.new(self.name, label, date, stats, options.merge(opts), meta).save
     end
     alias :event :store
-    
+
     def fetch(label, from, till, opts = {})
-      Finder.find({
-        :scope => name,
-        :label => label,
-        :from  => from,
-        :till  => till
-      }.merge(options.merge(opts)))
+      find(label, from, till, opts).all
     end
-    alias :find :fetch
-    
-    def hashed_label(boolean = nil)
-      if !boolean.nil?
-        options[:hashed_label] = boolean
+    alias :lookup :fetch
+
+    def find(label, from, till, opts = {})
+      Finder.new( { :scope => self.name,
+                    :label => label,
+                    :from  => from,
+                    :till  => till }.merge(options.merge(opts)) )
+    end
+
+    def find_event(event_id)
+      Event.find(self.name, event_id)
+    end
+
+
+    #
+    # options methods
+    #
+
+    option_accessor :depth
+    option_accessor :scope
+    option_accessor :store_event
+    option_accessor :hashed_label
+    option_accessor :label_indexing
+
+    alias :class_name :scope
+
+    def expire(exp = nil)
+      if !exp.nil?
+        options[:expire] = exp.is_a?(Hash) ? exp : Hash.new(exp)
       else
-        options[:hashed_label] || nil
+        options[:expire]
       end
     end
-    
-    def depth(depth = nil)
-      if !depth.nil?
-        options[:depth] = depth
-      else
-        options[:depth] || nil
-      end
+
+    def connect_to(opts = {})
+      Connection.create(opts.merge(:ref => name))
+      options[:connection_ref] = name
     end
-    
-    def store_event(boolean = nil)
-      if !boolean.nil?
-        options[:store_event] = boolean
-      else
-        options[:store_event] || nil
-      end
+
+
+    #
+    # resource access methods
+    #
+
+    def connection
+      db(options[:connection_ref])
     end
-    
-    def options
-      @options ||= {}
-    end
-    
-    private
-    
+    alias :redis :connection
+
     def name
-      @name ||= self.to_s
+      options[:scope] || (@name ||= self.to_s)
     end
-    
+
   end
-end
+end

lib/redistat/result.rb

@@ -1,23 +1,18 @@
+require 'active_support/core_ext/hash/indifferent_access'
+
 module Redistat
-  class Result < ::ActiveSupport::HashWithIndifferentAccess
-    
+  class Result < HashWithIndifferentAccess
+
     attr_accessor :from
     attr_accessor :till
-    
+
     alias :date :from
     alias :date= :from=
-    
+
     def initialize(options = {})
       @from = options[:from] ||= nil
       @till = options[:till] ||= nil
     end
-    
-    
-    def set_or_incr(key, value)
-      self[key] = 0 if !self.has_key?(key)
-      self[key] += value
-      self
-    end
-    
+
   end
-end
+end

lib/redistat/scope.rb

@@ -1,18 +1,18 @@
 module Redistat
   class Scope
     include Database
-    
+
     def initialize(name)
       @name = name.to_s
     end
-    
+
     def to_s
       @name
     end
-    
+
     def next_id
       db.incr("#{@name}#{KEY_NEXT_ID}")
     end
-    
+
   end
-end
+end

lib/redistat/summary.rb

@@ -1,24 +1,89 @@
 module Redistat
   class Summary
     include Database
-    
-    def self.update_all(key, stats = {}, depth_limit = nil)
-      stats ||= {}
-      depth_limit ||= key.depth
-      return nil if stats.size == 0
-      Date::DEPTHS.each do |depth|
-        update(key, stats, depth)
-        break if depth == depth_limit
+
+    class << self
+
+      def default_options
+        {
+          :enable_grouping => true,
+          :label_indexing  => true,
+          :connection_ref  => nil,
+          :expire          => {}
+        }
       end
-    end
-    
-    private
-    
-    def self.update(key, stats, depth)
-      stats.each do |field, value|
-        db.hincrby key.to_s(depth), field, value
+
+      def buffer
+        Redistat.buffer
       end
+
+      def update_all(key, stats = {}, depth_limit = nil, opts = {})
+        stats ||= {}
+        return if stats.empty?
+
+        options = default_options.merge((opts || {}).reject { |k,v| v.nil? })
+
+        depth_limit ||= key.depth
+
+        update_through_buffer(key, stats, depth_limit, options)
+      end
+
+      def update_through_buffer(*args)
+        update(*args) unless buffer.store(*args)
+      end
+
+      def update(key, stats, depth_limit, opts = {})
+        if opts[:enable_grouping]
+          stats = inject_group_summaries(stats)
+          key.groups.each do |k|
+            update_key(k, stats, depth_limit, opts)
+            k.update_index if opts[:label_indexing]
+          end
+        else
+          update_key(key, stats, depth_limit, opts)
+        end
+      end
+
+      private
+
+      def update_key(key, stats, depth_limit, opts = {})
+        Date::DEPTHS.each do |depth|
+          update_fields(key, stats, depth, opts)
+          break if depth == depth_limit
+        end
+      end
+
+      def update_fields(key, stats, depth, opts = {})
+        stats.each do |field, value|
+          db(opts[:connection_ref]).hincrby key.to_s(depth), field, value
+        end
+
+        if opts[:expire] && !opts[:expire][depth].nil?
+          db(opts[:connection_ref]).expire key.to_s(depth), opts[:expire][depth]
+        end
+      end
+
+      def inject_group_summaries!(stats)
+        summaries = {}
+        stats.each do |key, value|
+          parts = key.to_s.split(GROUP_SEPARATOR)
+          parts.pop
+          if parts.size > 0
+            sum_parts = []
+            parts.each do |part|
+              sum_parts << part
+              sum_key = sum_parts.join(GROUP_SEPARATOR)
+              (summaries.has_key?(sum_key)) ? summaries[sum_key] += value : summaries[sum_key] = value
+            end
+          end
+        end
+        stats.merge_and_incr!(summaries)
+      end
+
+      def inject_group_summaries(stats)
+        inject_group_summaries!(stats.clone)
+      end
+
     end
-    
   end
-end
+end

lib/redistat/version.rb

@@ -0,0 +1,3 @@
+module Redistat
+  VERSION = "0.4.0"
+end

redistat.gemspec

@@ -0,0 +1,31 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "redistat/version"
+
+Gem::Specification.new do |s|
+  s.name        = "redistat"
+  s.version     = Redistat::VERSION
+  s.platform    = Gem::Platform::RUBY
+  s.authors     = ["Jim Myhrberg"]
+  s.email       = ["contact@jimeh.me"]
+  s.homepage    = "http://github.com/jimeh/redistat"
+  s.summary     = %q{A Redis-backed statistics storage and querying library written in Ruby.}
+  s.description = %q{A Redis-backed statistics storage and querying library written in Ruby.}
+
+  s.rubyforge_project = "redistat"
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  s.add_runtime_dependency 'activesupport', '>= 2.3.6'
+  s.add_runtime_dependency 'json', '>= 1.4.0'
+  s.add_runtime_dependency 'redis', '>= 2.1.0'
+  s.add_runtime_dependency 'time_ext', '>= 0.2.9'
+
+  s.add_development_dependency 'rake', '>= 0.8.7'
+  s.add_development_dependency 'rspec', '>= 2.1.0'
+  s.add_development_dependency 'yard', '>= 0.6.3'
+  s.add_development_dependency 'simplecov', '>= 0.6.1'
+end

spec/_redistat_spec.rb

@@ -1,34 +0,0 @@
-require "spec_helper"
-
-describe Redistat do
-  include Redistat::Database
-  
-  before(:each) do
-    db.flushdb
-  end
-  
-  it "should have a valid Redis client instance" do
-    db.should_not be_nil
-  end
-  
-  it "should be connected to the testing server" do
-    db.client.port.should == 8379
-    db.client.host.should == "127.0.0.1"
-  end
-  
-  it "should be able to set and get data" do
-    db.set("hello", "world")
-    db.get("hello").should == "world"
-    db.del("hello").should be_true
-  end
-  
-  it "should be able to store hashes to Redis" do
-    db.hset("key", "field", "1")
-    db.hget("key", "field").should == "1"
-    db.hincrby("key", "field", 1)
-    db.hget("key", "field").should == "2"
-    db.hincrby("key", "field", -1)
-    db.hget("key", "field").should == "1"
-  end
-  
-end

spec/buffer_spec.rb

@@ -0,0 +1,159 @@
+require "spec_helper"
+
+describe Redistat::Buffer do
+
+  before(:each) do
+    @class  = Redistat::Buffer
+    @buffer = Redistat::Buffer.instance
+    @key = mock('Key', :to_s => "Scope/label:2011")
+    @stats = {:count => 1, :views => 3}
+    @depth_limit = :hour
+    @opts = {:enable_grouping => true}
+  end
+
+  # let's cleanup after ourselves for the other specs
+  after(:each) do
+    @class.instance_variable_set("@instance", nil)
+    @buffer.size = 0
+  end
+
+  it "should provide instance of itself" do
+    @buffer.should be_a(@class)
+  end
+
+  it "should only buffer if buffer size setting is greater than 1" do
+    @buffer.size.should == 0
+    @buffer.send(:should_buffer?).should be_false
+    @buffer.size = 1
+    @buffer.size.should == 1
+    @buffer.send(:should_buffer?).should be_false
+    @buffer.size = 2
+    @buffer.size.should == 2
+    @buffer.send(:should_buffer?).should be_true
+  end
+
+  it "should only flush buffer if buffer size is greater than or equal to buffer size setting" do
+    @buffer.size.should == 0
+    @buffer.send(:queue).size.should == 0
+    @buffer.send(:should_flush?).should be_false
+    @buffer.send(:queue)[:hello] = 'world'
+    @buffer.send(:incr_count)
+    @buffer.send(:should_flush?).should be_true
+    @buffer.size = 5
+    @buffer.send(:should_flush?).should be_false
+    3.times { |i|
+      @buffer.send(:queue)[i] = i.to_s
+      @buffer.send(:incr_count)
+    }
+    @buffer.send(:should_flush?).should be_false
+    @buffer.send(:queue)[4] = '4'
+    @buffer.send(:incr_count)
+    @buffer.send(:should_flush?).should be_true
+  end
+
+  it "should force flush queue irregardless of result of #should_flush? when #reset_queue is called with true" do
+    @buffer.send(:queue)[:hello] = 'world'
+    @buffer.send(:incr_count)
+    @buffer.send(:should_flush?).should be_true
+    @buffer.size = 2
+    @buffer.send(:should_flush?).should be_false
+    @buffer.send(:reset_queue).should == {}
+    @buffer.instance_variable_get("@count").should == 1
+    @buffer.send(:reset_queue, true).should == {:hello => 'world'}
+    @buffer.instance_variable_get("@count").should == 0
+  end
+
+  it "should #flush_data into Summary.update properly" do
+    # the root level key value doesn't actually matter, but it's something like this...
+    data = {'ScopeName/label/goes/here:2011::true:true' => {
+      :key => @key,
+      :stats => @stats,
+      :depth_limit => @depth_limit,
+      :opts => @opts
+    }}
+    item = data.first[1]
+    Redistat::Summary.should_receive(:update).with(@key, @stats, @depth_limit, @opts)
+    @buffer.send(:flush_data, data)
+  end
+
+  it "should build #buffer_key correctly" do
+    opts = {:enable_grouping => true, :label_indexing => false, :connection_ref => nil}
+    @buffer.send(:buffer_key, @key, opts).should ==
+      "#{@key.to_s}:connection_ref::enable_grouping:true:label_indexing:false"
+    opts = {:enable_grouping => false, :label_indexing => true, :connection_ref => :omg}
+    @buffer.send(:buffer_key, @key, opts).should ==
+      "#{@key.to_s}:connection_ref:omg:enable_grouping:false:label_indexing:true"
+  end
+
+  describe "Buffering" do
+    it "should store items on buffer queue" do
+      @buffer.store(@key, @stats, @depth_limit, @opts).should be_false
+      @buffer.size = 5
+      @buffer.store(@key, @stats, @depth_limit, @opts).should be_true
+      @buffer.send(:queue).should have(1).item
+      @buffer.send(:queue)[@buffer.send(:queue).keys.first][:stats][:count].should == 1
+      @buffer.send(:queue)[@buffer.send(:queue).keys.first][:stats][:views].should == 3
+      @buffer.store(@key, @stats, @depth_limit, @opts).should be_true
+      @buffer.send(:queue).should have(1).items
+      @buffer.send(:queue)[@buffer.send(:queue).keys.first][:stats][:count].should == 2
+      @buffer.send(:queue)[@buffer.send(:queue).keys.first][:stats][:views].should == 6
+    end
+
+    it "should flush buffer queue when size is reached" do
+      key = mock('Key', :to_s => "Scope/labelx:2011")
+      @buffer.size = 10
+      Redistat::Summary.should_receive(:update).exactly(2).times.and_return do |k, stats, depth_limit, opts|
+        depth_limit.should == @depth_limit
+        opts.should == @opts
+        if k == @key
+          stats[:count].should == 6
+          stats[:views].should == 18
+        elsif k == key
+          stats[:count].should == 4
+          stats[:views].should == 12
+        end
+      end
+      6.times { @buffer.store(@key, @stats, @depth_limit, @opts).should be_true }
+      4.times { @buffer.store(key, @stats, @depth_limit, @opts).should be_true }
+    end
+  end
+
+  describe "Thread-Safety" do
+    it "should read/write to buffer queue in a thread-safe manner" do
+
+      # Setting thread_safe to false only makes the spec fail with
+      # JRuby. 1.8.x and 1.9.x both pass fine for some reason
+      # regardless of what the thread_safe option is set to.
+      Redistat.thread_safe = true
+
+      key = mock('Key', :to_s => "Scope/labelx:2011")
+      @buffer.size = 100
+
+      Redistat::Summary.should_receive(:update).exactly(2).times.and_return do |k, stats, depth_limit, opts|
+        depth_limit.should == @depth_limit
+        opts.should == @opts
+        if k == @key
+          stats[:count].should == 60
+          stats[:views].should == 180
+        elsif k == key
+          stats[:count].should == 40
+          stats[:views].should == 120
+        end
+      end
+
+      threads = []
+      10.times do
+        threads << Thread.new {
+          6.times { @buffer.store(@key, @stats, @depth_limit, @opts).should be_true }
+          4.times { @buffer.store(key, @stats, @depth_limit, @opts).should be_true }
+        }
+      end
+
+      threads.each { |t| t.join }
+    end
+
+    it "should have specs that fail on 1.8.x/1.9.x when thread_safe is disabled"
+
+  end
+
+end

spec/collection_spec.rb

@@ -1,13 +1,20 @@
 require "spec_helper"
 
 describe Redistat::Collection do
-  
-  it "should should initialize properly" do
+
+  it "should initialize properly" do
     options = {:from => "from", :till => "till", :depth => "depth"}
     result = Redistat::Collection.new(options)
     result.from.should == options[:from]
     result.till.should == options[:till]
     result.depth.should == options[:depth]
   end
-  
-end
+
+  it "should have a total property" do
+    col = Redistat::Collection.new()
+    col.total.should == {}
+    col.total = {:foo => "bar"}
+    col.total.should == {:foo => "bar"}
+  end
+
+end

spec/connection_spec.rb

@@ -0,0 +1,67 @@
+require "spec_helper"
+include Redistat
+
+describe Redistat::Connection do
+
+  before(:each) do
+    @redis = Redistat.redis
+  end
+
+  it "should have a valid Redis client instance" do
+    Redistat.redis.should_not be_nil
+  end
+
+  it "should have initialized custom testing connection" do
+    @redis.client.host.should == '127.0.0.1'
+    @redis.client.port.should == 8379
+    @redis.client.db.should == 15
+  end
+
+  it "should be able to set and get data" do
+    @redis.set("hello", "world")
+    @redis.get("hello").should == "world"
+    @redis.del("hello").should be_true
+  end
+
+  it "should be able to store hashes to Redis" do
+    @redis.hset("hash", "field", "1")
+    @redis.hget("hash", "field").should == "1"
+    @redis.hincrby("hash", "field", 1)
+    @redis.hget("hash", "field").should == "2"
+    @redis.hincrby("hash", "field", -1)
+    @redis.hget("hash", "field").should == "1"
+    @redis.del("hash")
+  end
+
+  it "should be accessible from Redistat module" do
+    Redistat.redis.should == Connection.get
+    Redistat.redis.should == Redistat.connection
+  end
+
+  it "should handle multiple connections with refs" do
+    Redistat.redis.client.db.should == 15
+    Redistat.connect(:port => 8379, :db => 14, :ref => "Custom")
+    Redistat.redis.client.db.should == 15
+    Redistat.redis("Custom").client.db.should == 14
+  end
+
+  it "should be able to overwrite default and custom refs" do
+    Redistat.redis.client.db.should == 15
+    Redistat.connect(:port => 8379, :db => 14)
+    Redistat.redis.client.db.should == 14
+
+    Redistat.redis("Custom").client.db.should == 14
+    Redistat.connect(:port => 8379, :db => 15, :ref => "Custom")
+    Redistat.redis("Custom").client.db.should == 15
+
+    # Reset the default connection to the testing server or all hell
+    # might brake loose from the rest of the specs
+    Redistat.connect(:port => 8379, :db => 15)
+  end
+
+  # TODO: Test thread-safety
+  it "should be thread-safe" do
+    pending("need to figure out a way to test thread-safety")
+  end
+
+end

spec/core_ext/hash_spec.rb

@@ -0,0 +1,30 @@
+require "spec_helper"
+
+describe Hash do
+
+  it "should #set_or_incr values" do
+    hash = {:count => 1}
+    hash.set_or_incr(:sum, 3).should be_true
+    hash.should == {:count => 1, :sum => 3}
+    hash.set_or_incr(:count, 4).should be_true
+    hash.should == {:count => 5, :sum => 3}
+    hash.set_or_incr(:count, 'test').should be_false
+    hash.set_or_incr(:view, 'test').should be_false
+    hash.should == {:count => 5, :sum => 3}
+    hash[:view] = 'test'
+    hash.set_or_incr(:view, 3).should be_false
+  end
+
+  it "should #merge_and_incr hashes" do
+    hash = { :count => 1, :city => 'hell', :sum => 3, :name => 'john' }
+
+    new_hash = { :count => 3, :city => 'slum', :views => 2 }
+    hash.clone.merge_and_incr(new_hash).should == { :count => 4, :city => 'slum', :views => 2,
+                                                    :sum   => 3, :name => 'john' }
+
+    new_hash = { :count => 'six', :city => 'slum', :views => 2, :time => 'late' }
+    hash.clone.merge_and_incr(new_hash).should == { :count => 'six', :city => 'slum', :views => 2,
+                                                    :sum   => 3, :name => 'john', :time => 'late' }
+  end
+
+end

spec/database_spec.rb

@@ -0,0 +1,10 @@
+require "spec_helper"
+
+describe Redistat::Database do
+  include Redistat::Database
+
+  it "should make #db method available when included" do
+    db.should == Redistat.redis
+  end
+
+end

spec/date_spec.rb

@@ -1,14 +1,14 @@
 require "spec_helper"
 
 describe Redistat::Date do
-  
+
   it "should initialize from Time object" do
     now = Time.now
     [Redistat::Date.new(now), now.to_rs].each do |rdate|
       Redistat::Date::DEPTHS.each { |k| rdate.send(k).should == now.send(k) }
     end
   end
-  
+
   it "should initialize from Date object" do
     today = Date.today
     [Redistat::Date.new(today), today.to_rs].each do |rdate|
@@ -16,7 +16,7 @@ describe Redistat::Date do
       [:hour, :min, :sec, :usec].each { |k| rdate.send(k).should == 0 }
     end
   end
-  
+
   it "should initialize from Fixnum object (UNIX Timestamp)" do
     now = Time.now.to_i
     time = Time.at(now)
@@ -24,13 +24,13 @@ describe Redistat::Date do
       [:year, :month, :day, :hour, :min, :sec].each { |k| rdate.send(k).should == time.send(k) }
     end
   end
-  
+
   it "should initialize from String object" do
     now = Time.now
     rdate = Redistat::Date.new(now.to_s)
     [:year, :month, :day, :hour, :min, :sec].each { |k| rdate.send(k).should == now.send(k) }
   end
-  
+
   it "should initialize from Redistat date String" do
     now = Time.now
     rdate = Redistat::Date.new(now.to_s)
@@ -38,25 +38,25 @@ describe Redistat::Date do
       rdate.to_s(k).should == Redistat::Date.new(rdate.to_s(k)).to_s(k)
     }
   end
-  
+
   it "should convert to Time object" do
     now = Time.now
     rdate = Redistat::Date.new(now)
     rdate.to_time.to_s.should == now.to_s
   end
-  
+
   it "should convert to Date object" do
     today = Date.today
     rdate = Redistat::Date.new(today)
     rdate.to_date.to_s.should == today.to_s
   end
-  
+
   it "should convert to Fixnum object (UNIX Timestamp)" do
     now = Time.now
     rdate = Redistat::Date.new(now)
     rdate.to_i.should == now.to_i
   end
-  
+
   it "should convert to string with correct depths" do
     today = Date.today
     now = Time.now
@@ -71,25 +71,25 @@ describe Redistat::Date do
       end
     end
   end
-  
+
   it "should add helper methods to Date, Time and Fixnum classes" do
     Date.today.to_time.should == Time.parse(Date.today.to_s)
     Time.now.to_i.to_time.should == Time.at(Time.now.to_i)
     Date.today.to_rs.to_date.should == Date.today
   end
-  
+
   it "should have a depth property" do
     now = Time.now
-    
+
     date = Redistat::Date.new(now)
     date.depth.should be_nil
     date.to_s.should == now.to_rs(:sec).to_s
     date.to_s.should == now.to_rs.to_s(:sec)
-    
+
     date = Redistat::Date.new(now, :hour)
     date.depth.should == :hour
     date.to_s.should == now.to_rs(:hour).to_s
     date.to_s.should == now.to_rs.to_s(:hour)
   end
-  
-end
+
+end

spec/event_spec.rb

@@ -2,25 +2,25 @@ require "spec_helper"
 
 describe Redistat::Event do
   include Redistat::Database
-  
+
   before(:each) do
     db.flushdb
     @scope = "PageViews"
     @label = "about_us"
     @label_hash = Digest::SHA1.hexdigest(@label)
-    @stats = {:views => 1}
-    @meta = {:user_id => 239}
+    @stats = {'views' => 1}
+    @meta = {'user_id' => 239}
     @options = {:depth => :hour}
     @date = Time.now
     @event = Redistat::Event.new(@scope, @label, @date, @stats, @options, @meta)
   end
-  
+
   it "should initialize properly" do
     @event.id.should be_nil
-    @event.scope.should == @scope
-    @event.label.should == @label
+    @event.scope.to_s.should == @scope
+    @event.label.to_s.should == @label
     @event.label_hash.should == @label_hash
-    @event.date.to_time.should == @date
+    @event.date.to_time.to_s.should == @date.to_s
     @event.stats.should == @stats
     @event.meta.should == @meta
     @event.options.should == @event.default_options.merge(@options)
@@ -28,20 +28,20 @@ describe Redistat::Event do
 
   it "should allow changing attributes" do
     # date
-    @event.date.to_time.should == @date
+    @event.date.to_time.to_s.should == @date.to_s
     @date = Time.now
     @event.date = @date
-    @event.date.to_time.should == @date
+    @event.date.to_time.to_s.should == @date.to_s
     # label
-    @event.label.should == @label
+    @event.label.to_s.should == @label
     @event.label_hash.should == @label_hash
     @label = "contact_us"
     @label_hash = Digest::SHA1.hexdigest(@label)
     @event.label = @label
-    @event.label.should == @label
+    @event.label.to_s.should == @label
     @event.label_hash.should == @label_hash
   end
-  
+
   it "should increment next_id" do
     event = Redistat::Event.new("VisitorCount", @label, @date, @stats, @options, @meta)
     @event.next_id.should == 1
@@ -49,7 +49,7 @@ describe Redistat::Event do
     @event.next_id.should == 2
     event.next_id.should == 2
   end
-  
+
   it "should store event properly" do
     @event = Redistat::Event.new(@scope, @label, @date, @stats, @options.merge({:store_event => true}), @meta)
     @event.new?.should be_true
@@ -59,15 +59,17 @@ describe Redistat::Event do
     keys.should include("#{@event.scope}#{Redistat::KEY_EVENT}#{@event.id}")
     keys.should include("#{@event.scope}#{Redistat::KEY_EVENT_IDS}")
   end
-  
+
   it "should find event by id" do
     @event = Redistat::Event.new(@scope, @label, @date, @stats, @options.merge({:store_event => true}), @meta).save
     fetched = Redistat::Event.find(@scope, @event.id)
-    @event.scope.should == fetched.scope
-    @event.label.should == fetched.label
+    @event.scope.to_s.should == fetched.scope.to_s
+    @event.label.to_s.should == fetched.label.to_s
     @event.date.to_s.should == fetched.date.to_s
+    @event.stats.should == fetched.stats
+    @event.meta.should == fetched.meta
   end
-  
+
   it "should store summarized statistics" do
     2.times do |i|
       @event = Redistat::Event.new(@scope, @label, @date, @stats, @options, @meta).save
@@ -79,5 +81,5 @@ describe Redistat::Event do
       end
     end
   end
-  
-end
+
+end

spec/finder/date_set_spec.rb

@@ -1,11 +1,11 @@
 require "spec_helper"
 
 describe Redistat::Finder::DateSet do
-  
+
   before(:all) do
     @finder = Redistat::Finder::DateSet.new
   end
-  
+
   it "should initialize properly" do
     t_start = Time.utc(2010, 8, 28, 22, 54, 57)
     t_end = Time.utc(2013, 12, 4, 22, 52, 3)
@@ -20,506 +20,508 @@ describe Redistat::Finder::DateSet do
       { :add => ["2011", "2012"],                         :rem => []             }
     ]
   end
-  
+
   it "should find date sets by interval" do
     t_start = Time.utc(2010, 8, 28, 18, 54, 57)
-    
+
     t_end = t_start + 4.hours
     result = Redistat::Finder::DateSet.new.find_date_sets(t_start, t_end, :hour, true)
     result[0][:add].should == ["2010082818", "2010082819", "2010082820", "2010082821", "2010082822"]
     result[0][:rem].should == []
-    
+    result.should == Redistat::Finder::DateSet.new(t_start, t_end, nil, :hour)
+
     t_end = t_start + 4.days
     result = Redistat::Finder::DateSet.new.find_date_sets(t_start, t_end, :day, true)
     result[0][:add].should == ["20100828", "20100829", "20100830", "20100831", "20100901"]
     result[0][:rem].should == []
+    result.should == Redistat::Finder::DateSet.new(t_start, t_end, nil, :day)
   end
-  
+
   it "should find start keys properly" do
-    
+
     #
     # Simple fetching
     # Dates: 22:54, 26th August, 2010 --> 22:52, 14th December, 2010
     #
-    
+
     t_start = Time.utc(2010, 8, 26, 22, 54, 57)
     t_end = Time.utc(2013, 12, 14, 22, 52, 3)
-  
+
     result = @finder.send(:find_start_keys_for, :sec, t_start, t_end)
     result[:add].should == ["20100826225458", "20100826225459"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end)
     result[:add].should == ["201008262255", "201008262256", "201008262257", "201008262258", "201008262259"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :hour, t_start, t_end)
     result[:add].should == ["2010082623"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :day, t_start, t_end)
     result[:add].should == ["20100827", "20100828", "20100829", "20100830", "20100831"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :month, t_start, t_end)
     result[:add].should == ["201009", "201010", "201011", "201012"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :year, t_start, t_end)
     result[:add].should == ["2011", "2012"]
     result[:rem].should == []
-    
+
     #
     # Reverse / Inteligent fetching
     # Dates: 5:06, 4th April, 2010 --> 22:52, 14th February, 2011
     #
-    
+
     t_start = Time.utc(2010, 4, 4, 5, 6, 4)
     t_end = Time.utc(2011, 2, 14, 22, 52, 3)
-  
+
     result = @finder.send(:find_start_keys_for, :sec, t_start, t_end)
     result[:add].should == ["201004040506"]
     result[:rem].should == ["20100404050600", "20100404050601", "20100404050602", "20100404050603", "20100404050604"]
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end)
     result[:add].should == ["2010040405"]
     result[:rem].should == ["201004040500", "201004040501", "201004040502", "201004040503", "201004040504", "201004040505", "201004040506"]
-    
+
     result = @finder.send(:find_start_keys_for, :hour, t_start, t_end)
     result[:add].should == ["20100404"]
     result[:rem].should == ["2010040400", "2010040401", "2010040402", "2010040403", "2010040404", "2010040405"]
-    
+
     result = @finder.send(:find_start_keys_for, :day, t_start, t_end)
     result[:add].should == ["201004"]
     result[:rem].should == ["20100401", "20100402", "20100403", "20100404"]
-    
+
     result = @finder.send(:find_start_keys_for, :month, t_start, t_end)
     result[:add].should == ["2010"]
     result[:rem].should == ["201001", "201002", "201003", "201004"]
-    
+
     result = @finder.send(:find_start_keys_for, :year, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
   end
-  
+
   it "should find end keys properly" do
-    
+
     #
     # Simple fetching
     # Dates: 22:04, 26th December, 2007 --> 5:06, 7th May, 2010
     #
-    
+
     t_start = Time.utc(2007, 12, 26, 22, 4, 4)
     t_end = Time.utc(2010, 5, 7, 5, 6, 3)
-  
+
     result = @finder.send(:find_end_keys_for, :sec, t_start, t_end)
     result[:add].should == ["20100507050600", "20100507050601", "20100507050602"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end)
     result[:add].should == ["201005070500", "201005070501", "201005070502", "201005070503", "201005070504", "201005070505"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :hour, t_start, t_end)
     result[:add].should == ["2010050700", "2010050701", "2010050702", "2010050703", "2010050704"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :day, t_start, t_end)
     result[:add].should == ["20100501", "20100502", "20100503", "20100504", "20100505", "20100506"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :month, t_start, t_end)
     result[:add].should == ["201001", "201002", "201003", "201004"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :year, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     #
     # Reverse / Inteligent fetching
     # Dates: 22:04, 26th December, 2009 --> 22:56, 27th October, 2010
     #
-    
+
     t_start = Time.utc(2009, 12, 26, 22, 4, 4)
     t_end = Time.utc(2010, 10, 27, 22, 56, 57)
-  
+
     result = @finder.send(:find_end_keys_for, :sec, t_start, t_end)
     result[:add].should == ["201010272256"]
     result[:rem].should == ["20101027225657", "20101027225658", "20101027225659"]
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end)
     result[:add].should == ["2010102722"]
     result[:rem].should == ["201010272256", "201010272257", "201010272258", "201010272259"]
-    
+
     result = @finder.send(:find_end_keys_for, :hour, t_start, t_end)
     result[:add].should == ["20101027"]
     result[:rem].should == ["2010102722", "2010102723"]
-    
+
     result = @finder.send(:find_end_keys_for, :day, t_start, t_end)
     result[:add].should == ["201010"]
     result[:rem].should == ["20101027", "20101028", "20101029", "20101030", "20101031"]
-    
+
     result = @finder.send(:find_end_keys_for, :month, t_start, t_end)
     result[:add].should == ["2010"]
     result[:rem].should == ["201010", "201011", "201012"]
-    
+
     result = @finder.send(:find_end_keys_for, :year, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
   end
-  
+
   it "should fetch start/end keys with limits" do
-    
+
     #
     # Simple fetching with Limits
     #
-    
+
     # seconds
     t_start = Time.utc(2010, 8, 26, 20, 54, 45)
     t_end = t_start + 4.seconds
-    
+
     result = @finder.send(:find_start_keys_for, :sec, t_start, t_end)
     result[:add].should == ["20100826205446", "20100826205447", "20100826205448"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :sec, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     t_start = Time.utc(2010, 8, 26, 20, 54, 4)
     t_end = t_start + 4.seconds
-    
+
     result = @finder.send(:find_start_keys_for, :sec, t_start, t_end)
     result[:add].should == ["20100826205405", "20100826205406", "20100826205407"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :sec, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     # minutes
     t_start = Time.utc(2010, 8, 26, 20, 54)
     t_end = t_start + 4.minutes
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end)
     result[:add].should == ["201008262055", "201008262056", "201008262057"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     t_start = Time.utc(2010, 8, 26, 20, 4)
     t_end = t_start + 4.minutes
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end)
     result[:add].should == ["201008262005", "201008262006", "201008262007"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     # hours
     t_start = Time.utc(2010, 8, 26, 20, 54)
     t_end = t_start + 2.hours
-  
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end)
     result[:add].should == ["201008262055", "201008262056", "201008262057", "201008262058", "201008262059"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :hour, t_start, t_end)
     result[:add].should == ["2010082621"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :hour, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end)
     result[:add].should == ["2010082622"]
     result[:rem].should == ["201008262254", "201008262255", "201008262256", "201008262257", "201008262258", "201008262259"]
-    
+
     t_start = Time.utc(2010, 8, 26, 4, 54)
     t_end = t_start + 5.hours
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end)
     result[:add].should == ["201008260455", "201008260456", "201008260457", "201008260458", "201008260459"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :hour, t_start, t_end)
     result[:add].should == ["2010082605", "2010082606", "2010082607", "2010082608"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :hour, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end)
     result[:add].should == ["2010082609"]
     result[:rem].should == ["201008260954", "201008260955", "201008260956", "201008260957", "201008260958", "201008260959"]
-    
+
     # days
     t_start = Time.utc(2010, 8, 26, 20, 54)
     t_end = t_start + 2.day
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end)
     result[:add].should == ["201008262055", "201008262056", "201008262057", "201008262058", "201008262059"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :hour, t_start, t_end)
     result[:add].should == ["2010082621", "2010082622", "2010082623"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :day, t_start, t_end)
     result[:add].should == ["20100827"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :day, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :hour, t_start, t_end)
     result[:add].should == ["20100828"]
     result[:rem].should == ["2010082820", "2010082821", "2010082822", "2010082823"]
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end)
     result[:add].should == ["2010082820"]
     result[:rem].should == ["201008282054", "201008282055", "201008282056", "201008282057", "201008282058", "201008282059"]
-    
+
     t_start = Time.utc(2010, 8, 6, 20, 54)
     t_end = t_start + 2.day
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end)
     result[:add].should == ["201008062055", "201008062056", "201008062057", "201008062058", "201008062059"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :hour, t_start, t_end)
     result[:add].should == ["2010080621", "2010080622", "2010080623"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :day, t_start, t_end)
     result[:add].should == ["20100807"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :day, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :hour, t_start, t_end)
     result[:add].should == ["20100808"]
     result[:rem].should == ["2010080820", "2010080821", "2010080822", "2010080823"]
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end)
     result[:add].should == ["2010080820"]
     result[:rem].should == ["201008082054", "201008082055", "201008082056", "201008082057", "201008082058", "201008082059"]
-    
+
     # months
     t_start = Time.utc(2010, 8, 26, 20, 54)
     t_end = t_start + 3.months
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end)
     result[:add].should == ["201008262055", "201008262056", "201008262057", "201008262058", "201008262059"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :hour, t_start, t_end)
     result[:add].should == ["2010082621", "2010082622", "2010082623"]
     result[:rem].should == []
-        
+
     result = @finder.send(:find_start_keys_for, :day, t_start, t_end)
     result[:add].should == ["20100827", "20100828", "20100829", "20100830", "20100831"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :month, t_start, t_end)
     result[:add].should == ["201009", "201010"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :month, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :day, t_start, t_end)
     result[:add].should == ["201011"]
     result[:rem].should == ["20101126", "20101127", "20101128", "20101129", "20101130"]
-    
+
     result = @finder.send(:find_end_keys_for, :hour, t_start, t_end)
     result[:add].should == ["20101126"]
     result[:rem].should == ["2010112620", "2010112621", "2010112622", "2010112623"]
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end)
     result[:add].should == ["2010112620"]
     result[:rem].should == ["201011262054", "201011262055", "201011262056", "201011262057", "201011262058", "201011262059"]
-    
+
     t_start = Time.utc(2010, 4, 26, 20, 54)
     t_end = t_start + 3.months
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end)
     result[:add].should == ["201004262055", "201004262056", "201004262057", "201004262058", "201004262059"]
     result[:rem].should == []
-        
+
     result = @finder.send(:find_start_keys_for, :hour, t_start, t_end)
     result[:add].should == ["2010042621", "2010042622", "2010042623"]
     result[:rem].should == []
-        
+
     result = @finder.send(:find_start_keys_for, :day, t_start, t_end)
     result[:add].should == ["20100427", "20100428", "20100429", "20100430"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :month, t_start, t_end)
     result[:add].should == ["201005", "201006"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :month, t_start, t_end)
     result[:add].should == []
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :day, t_start, t_end)
     result[:add].should == ["201007"]
     result[:rem].should == ["20100726", "20100727", "20100728", "20100729", "20100730", "20100731"]
-    
+
     result = @finder.send(:find_end_keys_for, :hour, t_start, t_end)
     result[:add].should == ["20100726"]
     result[:rem].should == ["2010072620", "2010072621", "2010072622", "2010072623"]
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end)
     result[:add].should == ["2010072620"]
     result[:rem].should == ["201007262054", "201007262055", "201007262056", "201007262057", "201007262058", "201007262059"]
-    
+
   end
-  
+
   it "should find inclusive keys on lowest depth" do
-    
+
     #
     # Simple start fetching
     # Dates: 22:54, 26th August, 2010 --> 22:52, 14th December, 2010
     #
-    
+
     t_start = Time.utc(2010, 8, 26, 22, 54, 57)
     t_end = Time.utc(2013, 12, 14, 22, 52, 3)
-  
+
     result = @finder.send(:find_start_keys_for, :sec, t_start, t_end, true)
     result[:add].should == ["20100826225457", "20100826225458", "20100826225459"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end, true)
     result[:add].should == ["201008262254", "201008262255", "201008262256", "201008262257", "201008262258", "201008262259"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :hour, t_start, t_end, true)
     result[:add].should == ["2010082622", "2010082623"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :day, t_start, t_end, true)
     result[:add].should == ["20100826", "20100827", "20100828", "20100829", "20100830", "20100831"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :month, t_start, t_end, true)
     result[:add].should == ["201008", "201009", "201010", "201011", "201012"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_start_keys_for, :year, t_start, t_end, true)
     result[:add].should == ["2011", "2012", "2013"]
     result[:rem].should == []
-    
+
     #
     # Reverse / Inteligent start fetching
     # Dates: 5:06, 4th April, 2010 --> 22:52, 14th February, 2011
     #
-    
+
     t_start = Time.utc(2010, 4, 4, 5, 6, 4)
     t_end = Time.utc(2013, 2, 14, 22, 52, 3)
-  
+
     result = @finder.send(:find_start_keys_for, :sec, t_start, t_end, true)
     result[:add].should == ["201004040506"]
     result[:rem].should == ["20100404050600", "20100404050601", "20100404050602", "20100404050603"]
-    
+
     result = @finder.send(:find_start_keys_for, :min, t_start, t_end, true)
     result[:add].should == ["2010040405"]
     result[:rem].should == ["201004040500", "201004040501", "201004040502", "201004040503", "201004040504", "201004040505"]
-    
+
     result = @finder.send(:find_start_keys_for, :hour, t_start, t_end, true)
     result[:add].should == ["20100404"]
     result[:rem].should == ["2010040400", "2010040401", "2010040402", "2010040403", "2010040404"]
-    
+
     result = @finder.send(:find_start_keys_for, :day, t_start, t_end, true)
     result[:add].should == ["201004"]
     result[:rem].should == ["20100401", "20100402", "20100403"]
-    
+
     result = @finder.send(:find_start_keys_for, :month, t_start, t_end, true)
     result[:add].should == ["2010"]
     result[:rem].should == ["201001", "201002", "201003"]
-    
+
     result = @finder.send(:find_start_keys_for, :year, t_start, t_end, true)
     result[:add].should == ["2011", "2012", "2013"]
     result[:rem].should == []
-    
+
     #
     # Simple fetching
     # Dates: 22:04, 26th December, 2007 --> 5:06, 7th May, 2010
     #
-    
+
     t_start = Time.utc(2007, 12, 26, 22, 4, 4)
     t_end = Time.utc(2010, 5, 7, 5, 6, 3)
-  
+
     result = @finder.send(:find_end_keys_for, :sec, t_start, t_end, true)
     result[:add].should == ["20100507050600", "20100507050601", "20100507050602", "20100507050603"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end, true)
     result[:add].should == ["201005070500", "201005070501", "201005070502", "201005070503", "201005070504", "201005070505", "201005070506"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :hour, t_start, t_end, true)
     result[:add].should == ["2010050700", "2010050701", "2010050702", "2010050703", "2010050704", "2010050705"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :day, t_start, t_end, true)
     result[:add].should == ["20100501", "20100502", "20100503", "20100504", "20100505", "20100506", "20100507"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :month, t_start, t_end, true)
     result[:add].should == ["201001", "201002", "201003", "201004", "201005"]
     result[:rem].should == []
-    
+
     result = @finder.send(:find_end_keys_for, :year, t_start, t_end, true)
     result[:add].should == ["2010"]
     result[:rem].should == []
-    
+
     #
     # Reverse / Inteligent fetching
     # Dates: 22:04, 26th December, 2009 --> 22:56, 27th October, 2010
     #
-    
+
     t_start = Time.utc(2009, 12, 26, 22, 4, 4)
     t_end = Time.utc(2010, 10, 27, 22, 56, 57)
-  
+
     result = @finder.send(:find_end_keys_for, :sec, t_start, t_end, true)
     result[:add].should == ["201010272256"]
     result[:rem].should == ["20101027225658", "20101027225659"]
-    
+
     result = @finder.send(:find_end_keys_for, :min, t_start, t_end, true)
     result[:add].should == ["2010102722"]
     result[:rem].should == ["201010272257", "201010272258", "201010272259"]
-    
+
     result = @finder.send(:find_end_keys_for, :hour, t_start, t_end, true)
     result[:add].should == ["20101027"]
     result[:rem].should == ["2010102723"]
-    
+
     result = @finder.send(:find_end_keys_for, :day, t_start, t_end, true)
     result[:add].should == ["201010"]
     result[:rem].should == ["20101028", "20101029", "20101030", "20101031"]
-    
+
     result = @finder.send(:find_end_keys_for, :month, t_start, t_end, true)
     result[:add].should == ["2010"]
     result[:rem].should == ["201011", "201012"]
-    
+
     result = @finder.send(:find_end_keys_for, :year, t_start, t_end, true)
     result[:add].should == ["2010"]
     result[:rem].should == []
-    
+
   end
-  
-end
+
+end

spec/finder_spec.rb

@@ -2,7 +2,7 @@ require "spec_helper"
 
 describe Redistat::Finder do
   include Redistat::Database
-  
+
   before(:each) do
     db.flushdb
     @scope = "PageViews"
@@ -10,56 +10,70 @@ describe Redistat::Finder do
     @date = Time.now
     @key = Redistat::Key.new(@scope, @label, @date, {:depth => :day})
     @stats = {"views" => 3, "visitors" => 2}
+    @two_hours_ago = 2.hours.ago
+    @one_hour_ago = 1.hour.ago
   end
-  
-  it "should initialize properly" do
-    two_hours_ago = 2.hours.ago
-    one_hour_ago = 1.hour.ago
-    options = {:scope => "PageViews", :label => "Label", :from => two_hours_ago, :till => one_hour_ago, :depth => :hour, :interval => :hour}
-    
-    finder = Redistat::Finder.new(options)
-    finder.options.should == options
 
-    finder = Redistat::Finder.dates(two_hours_ago, one_hour_ago).scope("PageViews").label("Label").depth(:hour).interval(:hour)
-    finder.options.should == options
-                                    
-    finder = Redistat::Finder.scope("PageViews").label("Label").from(two_hours_ago).till(one_hour_ago).depth(:hour).interval(:hour)
-    finder.options.should == options
-    
-    finder = Redistat::Finder.label("Label").from(two_hours_ago).till(one_hour_ago).depth(:hour).interval(:hour).scope("PageViews")
-    finder.options.should == options
-    
-    finder = Redistat::Finder.from(two_hours_ago).till(one_hour_ago).depth(:hour).interval(:hour).scope("PageViews").label("Label")
-    finder.options.should == options
-    
-    finder = Redistat::Finder.till(one_hour_ago).depth(:hour).interval(:hour).scope("PageViews").label("Label").from(two_hours_ago)
-    finder.options.should == options
-    
-    finder = Redistat::Finder.depth(:hour).interval(:hour).scope("PageViews").label("Label").from(two_hours_ago).till(one_hour_ago)
-    finder.options.should == options
-    
-    finder = Redistat::Finder.interval(:hour).scope("PageViews").label("Label").from(two_hours_ago).till(one_hour_ago).depth(:hour)
-    finder.options.should == options
-    
+  it "should initialize properly" do
+    options = {:scope => "PageViews", :label => "Label", :from => @two_hours_ago, :till => @one_hour_ago, :depth => :hour, :interval => :hour}
+
+    finder = Redistat::Finder.new
+    finder.send(:set_options, options)
+    finder.options[:scope].should be_a(Redistat::Scope)
+    finder.options[:scope].to_s.should == options[:scope]
+    finder.options[:label].should be_a(Redistat::Label)
+    finder.options[:label].to_s.should == options[:label]
+    finder.options.should == options.merge(:scope => finder.options[:scope], :label => finder.options[:label])
+
+    finder = Redistat::Finder.scope("hello")
+    finder.options[:scope].to_s.should == "hello"
+    finder.scope.to_s.should == "hello"
+
+    finder = Redistat::Finder.label("hello")
+    finder.options[:label].to_s.should == "hello"
+    finder.label.to_s.should == "hello"
+
+    finder = Redistat::Finder.dates(@two_hours_ago, @one_hour_ago)
+    finder.options[:from].should == @two_hours_ago
+    finder.options[:till].should == @one_hour_ago
+
+    finder = Redistat::Finder.from(@two_hours_ago)
+    finder.options[:from].should == @two_hours_ago
+    finder.from.should == @two_hours_ago
+
+    finder = Redistat::Finder.till(@one_hour_ago)
+    finder.options[:till].should == @one_hour_ago
+    finder.till.should == @one_hour_ago
+
+    finder = Redistat::Finder.depth(:hour)
+    finder.options[:depth].should == :hour
+    finder.depth.should == :hour
+
+    finder = Redistat::Finder.interval(true)
+    finder.options[:interval].should be_true
+    finder.interval.should be_true
+    finder = Redistat::Finder.interval(false)
+    finder.options[:interval].should be_false
+    finder.interval.should be_false
   end
-  
+
   it "should fetch stats properly" do
     first_stat, last_stat = create_example_stats
-    
+
     stats = Redistat::Finder.find({:from => first_stat, :till => last_stat, :scope => @scope, :label => @label, :depth => :hour})
     stats.from.should  == first_stat
     stats.till.should  == last_stat
     stats.depth.should == :hour
-    
+
     stats.total.should == { "views" => 12, "visitors" => 8 }
     stats.total.from.should == first_stat
     stats.total.till.should == last_stat
     stats.first.should == stats.total
   end
-  
+
   it "should fetch data per unit when interval option is specified" do
     first_stat, last_stat = create_example_stats
-    
+
     stats = Redistat::Finder.find(:from => first_stat, :till => last_stat, :scope => @scope, :label => @label, :depth => :hour, :interval => :hour)
     stats.from.should == first_stat
     stats.till.should == last_stat
@@ -75,38 +89,119 @@ describe Redistat::Finder do
     stats[4].should == {}
     stats[4].date.should == Time.parse("2010-05-14 16:00")
   end
-  
+
   it "should return empty hash when attempting to fetch non-existent results" do
     stats = Redistat::Finder.find({:from => 3.hours.ago, :till => 2.hours.from_now, :scope => @scope, :label => @label, :depth => :hour})
     stats.total.should == {}
   end
-  
+
   it "should throw error on invalid options" do
     lambda { Redistat::Finder.find(:from => 3.hours.ago) }.should raise_error(Redistat::InvalidOptions)
   end
-  
-  
+
+  describe "Grouping" do
+    before(:each) do
+      @options = {:scope => "PageViews", :label => "message/public", :from => @two_hours_ago, :till => @one_hour_ago, :depth => :hour, :interval => :hour}
+      @finder = Redistat::Finder.new(@options)
+    end
+
+    it "should return parent finder" do
+      @finder.instance_variable_get("@parent").should be_nil
+      @finder.parent.should be_a(Redistat::Finder)
+      @finder.instance_variable_get("@parent").should_not be_nil
+      @finder.parent.options[:label].to_s.should == 'message'
+      @finder.label('message')
+      @finder.instance_variable_get("@parent").should be_nil
+      @finder.parent.should_not be_nil
+      @finder.parent.options[:label].should be_nil
+      @finder.parent.parent.should be_nil
+    end
+
+    it "should find children" do
+      Redistat::Key.new("PageViews", "message/public/die").update_index
+      Redistat::Key.new("PageViews", "message/public/live").update_index
+      Redistat::Key.new("PageViews", "message/public/fester").update_index
+      members = db.smembers("#{@scope}#{Redistat::LABEL_INDEX}message/public") # checking 'message/public'
+      @finder.children.first.should be_a(Redistat::Finder)
+      subs = @finder.children.map { |f| f.options[:label].me }
+      subs.should have(3).items
+      subs.should include('die')
+      subs.should include('live')
+      subs.should include('fester')
+    end
+  end
+
+  describe "Lazy-Loading" do
+
+    before(:each) do
+      @first_stat, @last_stat = create_example_stats
+
+      @finder = Redistat::Finder.new
+      @finder.from(@first_stat).till(@last_stat).scope(@scope).label(@label).depth(:hour)
+
+      @match = [{}, {"visitors"=>"4", "views"=>"6"},
+                    {"visitors"=>"2", "views"=>"3"},
+                    {"visitors"=>"2", "views"=>"3"}, {}]
+    end
+
+    it "should lazy-load" do
+      @finder.instance_variable_get("@result").should be_nil
+      stats = @finder.all
+      @finder.instance_variable_get("@result").should_not be_nil
+
+      stats.should == @finder.find # find method directly fetches results
+      stats.total.should == @finder.total
+      stats.total.should == { "views" => 12, "visitors" => 8 }
+      stats.total.from.should == @first_stat
+      stats.total.till.should == @last_stat
+      stats.first.should == stats.total
+
+      @finder.all.object_id.should == stats.object_id
+      @finder.from(@first_stat + 2.hours)
+      @finder.instance_variable_get("@result").should be_nil
+      @finder.all.object_id.should_not == stats.object_id
+      stats = @finder.all
+      stats.total.should == { "views" => 6, "visitors" => 4 }
+    end
+
+    it "should handle #map" do
+      @finder.interval(:hour)
+      @finder.map { |r| r }.should == @match
+    end
+
+    it "should handle #each" do
+      @finder.interval(:hour)
+
+      res = []
+      @finder.each { |r| res << r }
+      res.should == @match
+    end
+
+    it "should handle #each_with_index" do
+      @finder.interval(:hour)
+
+      res = {}
+      match = {}
+      @finder.each_with_index { |r, i| res[i] = r }
+      @match.each_with_index { |r, i| match[i] = r }
+      res.should == match
+    end
+
+  end # "Lazy-Loading"
+
+
   # helper methods
-  
+
   def create_example_stats
     key = Redistat::Key.new(@scope, @label, (first = Time.parse("2010-05-14 13:43")))
-    Redistat::Summary.update(key, @stats, :hour)
+    Redistat::Summary.send(:update_fields, key, @stats, :hour)
     key = Redistat::Key.new(@scope, @label, Time.parse("2010-05-14 13:53"))
-    Redistat::Summary.update(key, @stats, :hour)
-    key = Redistat::Key.new(@scope, @label, Time.parse("2010-05-14 14:32"))
-    Redistat::Summary.update(key, @stats, :hour)
+    Redistat::Summary.send(:update_fields, key, @stats, :hour)
+    key = Redistat::Key.new(@scope, @label, Time.parse("2010-05-14 14:52"))
+    Redistat::Summary.send(:update_fields, key, @stats, :hour)
     key = Redistat::Key.new(@scope, @label, (last = Time.parse("2010-05-14 15:02")))
-    Redistat::Summary.update(key, @stats, :hour)
+    Redistat::Summary.send(:update_fields, key, @stats, :hour)
     [first - 1.hour, last + 1.hour]
   end
-  
+
 end
-
-
-
-
-
-
-
-
-

spec/key_spec.rb

@@ -1,23 +1,26 @@
 require "spec_helper"
 
 describe Redistat::Key do
-  
+  include Redistat::Database
+
   before(:each) do
+    db.flushdb
     @scope = "PageViews"
     @label = "about_us"
     @label_hash = Digest::SHA1.hexdigest(@label)
     @date = Time.now
     @key = Redistat::Key.new(@scope, @label, @date, {:depth => :hour})
   end
-  
+
   it "should initialize properly" do
-    @key.scope.should == @scope
-    @key.label.should == @label
+    @key.scope.to_s.should == @scope
+    @key.label.to_s.should == @label
     @key.label_hash.should == @label_hash
+    @key.groups.map { |k| k.instance_variable_get("@label") }.should == @key.instance_variable_get("@label").groups
     @key.date.should be_instance_of(Redistat::Date)
     @key.date.to_time.to_s.should == @date.to_s
   end
-  
+
   it "should convert to string properly" do
     @key.to_s.should == "#{@scope}/#{@label}:#{@key.date.to_s(:hour)}"
     props = [:year, :month, :day, :hour, :min, :sec]
@@ -25,39 +28,102 @@ describe Redistat::Key do
       @key.to_s(props.last).should == "#{@scope}/#{@label}:#{@key.date.to_s(props.last)}"
       props.pop
     end
+    key = Redistat::Key.new(@scope, nil, @date, {:depth => :hour})
+    key.to_s.should == "#{@scope}:#{key.date.to_s(:hour)}"
   end
-  
+
   it "should abide to hashed_label option" do
     @key = Redistat::Key.new(@scope, @label, @date, {:depth => :hour, :hashed_label => true})
     @key.to_s.should == "#{@scope}/#{@label_hash}:#{@key.date.to_s(:hour)}"
     @key = Redistat::Key.new(@scope, @label, @date, {:depth => :hour, :hashed_label => false})
     @key.to_s.should == "#{@scope}/#{@label}:#{@key.date.to_s(:hour)}"
   end
-  
+
   it "should have default depth option" do
     @key = Redistat::Key.new(@scope, @label, @date)
     @key.depth.should == :hour
   end
-  
+
   it "should allow changing attributes" do
     # scope
-    @key.scope.should == @scope
+    @key.scope.to_s.should == @scope
     @scope = "VisitorCount"
     @key.scope = @scope
-    @key.scope.should == @scope
+    @key.scope.to_s.should == @scope
     # date
-    @key.date.to_time.should == @date
+    @key.date.to_time.to_s.should == @date.to_s
     @date = Time.now
     @key.date = @date
-    @key.date.to_time.should == @date
+    @key.date.to_time.to_s.should == @date.to_s
     # label
-    @key.label.should == @label
+    @key.label.to_s.should == @label
     @key.label_hash == @label_hash
     @label = "contact_us"
     @label_hash = Digest::SHA1.hexdigest(@label)
     @key.label = @label
-    @key.label.should == @label
+    @key.label.to_s.should == @label
     @key.label_hash == @label_hash
   end
-  
-end
+
+  describe "Grouping" do
+    before(:each) do
+      @label = "message/public/offensive"
+      @key = Redistat::Key.new(@scope, @label, @date, {:depth => :hour})
+    end
+
+    it "should create a group of keys from label group" do
+      label = 'message/public/offensive'
+      result = [ "message/public/offensive",
+                 "message/public",
+                 "message" ]
+
+      key = Redistat::Key.new(@scope, label, @date, {:depth => :hour})
+
+      key.groups.map { |k| k.label.to_s }.should == result
+    end
+
+    it "should know it's parent" do
+      @key.parent.should be_a(Redistat::Key)
+      @key.parent.label.to_s.should == 'message/public'
+      Redistat::Key.new(@scope, 'hello', @date).parent.should be_nil
+    end
+
+    it "should update label index and return children" do
+      db.smembers("#{@scope}#{Redistat::LABEL_INDEX}#{@key.label.parent}").should == []
+      @key.children.should have(0).items
+
+      @key.update_index                                                  # indexing 'message/publish/offensive'
+      Redistat::Key.new("PageViews", "message/public/die").update_index  # indexing 'message/publish/die'
+      Redistat::Key.new("PageViews", "message/public/live").update_index # indexing 'message/publish/live'
+
+      members = db.smembers("#{@scope}#{Redistat::LABEL_INDEX}#{@key.label.parent}") # checking 'message/public'
+      members.should have(3).item
+      members.should include('offensive')
+      members.should include('live')
+      members.should include('die')
+
+      key = @key.parent
+      key.children.first.should be_a(Redistat::Key)
+      key.children.should have(3).item
+      key.children.map { |k| k.label.me }.should == members
+
+      members = db.smembers("#{@scope}#{Redistat::LABEL_INDEX}#{key.label.parent}") # checking 'message'
+      members.should have(1).item
+      members.should include('public')
+
+      key = key.parent
+      key.children.should have(1).item
+      key.children.map { |k| k.label.me }.should == members
+
+      members = db.smembers("#{@scope}#{Redistat::LABEL_INDEX}") # checking ''
+      members.should have(1).item
+      members.should include('message')
+
+      key.parent.should be_nil
+      key = Redistat::Key.new("PageViews")
+      key.children.should have(1).item
+      key.children.map { |k| k.label.me }.should include('message')
+    end
+  end
+
+end

spec/label_spec.rb

@@ -2,27 +2,70 @@ require "spec_helper"
 
 describe Redistat::Label do
   include Redistat::Database
-  
+
   before(:each) do
     db.flushdb
     @name = "about_us"
     @label = Redistat::Label.new(@name)
   end
-  
+
   it "should initialize properly and SHA1 hash the label name" do
     @label.name.should == @name
     @label.hash.should == Digest::SHA1.hexdigest(@name)
   end
-  
+
   it "should store a label hash lookup key" do
     label = Redistat::Label.new(@name, {:hashed_label => true}).save
     label.saved?.should be_true
-    db.get("#{Redistat::KEY_LEBELS}#{label.hash}").should == @name
-    
+    db.hget(Redistat::KEY_LABELS, label.hash).should == @name
+
     name = "contact_us"
     label = Redistat::Label.create(name, {:hashed_label => true})
     label.saved?.should be_true
-    db.get("#{Redistat::KEY_LEBELS}#{label.hash}").should == name
+    db.hget(Redistat::KEY_LABELS, label.hash).should == name
   end
-  
-end
+
+  it "should join labels" do
+    include Redistat
+    label = Label.join('email', 'message', 'public')
+    label.should be_a(Label)
+    label.to_s.should == 'email/message/public'
+    label = Label.join(Label.new('email'), Label.new('message'), Label.new('public'))
+    label.should be_a(Label)
+    label.to_s.should == 'email/message/public'
+    label = Label.join('email', '', 'message', nil, 'public')
+    label.should be_a(Label)
+    label.to_s.should == 'email/message/public'
+  end
+
+  describe "Grouping" do
+    before(:each) do
+      @name = "message/public/offensive"
+      @label = Redistat::Label.new(@name)
+    end
+
+    it "should know it's parent label group" do
+      @label.parent.to_s.should == 'message/public'
+      Redistat::Label.new('hello').parent.should be_nil
+    end
+
+    it "should separate label names into groups" do
+      @label.name.should == @name
+      @label.groups.map { |l| l.to_s }.should == [ "message/public/offensive",
+                                                   "message/public",
+                                                   "message" ]
+
+      @name = "/message/public/"
+      @label = Redistat::Label.new(@name)
+      @label.name.should == @name
+      @label.groups.map { |l| l.to_s }.should == [ "message/public",
+                                                   "message" ]
+
+      @name = "message"
+      @label = Redistat::Label.new(@name)
+      @label.name.should == @name
+      @label.groups.map { |l| l.to_s }.should == [ "message" ]
+    end
+  end
+
+end

spec/model_helper.rb

@@ -1,16 +1,31 @@
 require "redistat"
 
-class ModelHelper
+class ModelHelper1
   include Redistat::Model
-  
-  
+
+
 end
 
 class ModelHelper2
   include Redistat::Model
-  
+
   depth :day
   store_event true
   hashed_label true
-  
-end
+
+end
+
+class ModelHelper3
+  include Redistat::Model
+
+  connect_to :port => 8379, :db => 14
+
+end
+
+class ModelHelper4
+  include Redistat::Model
+
+  scope "FancyHelper"
+  expire :hour => 24*3600
+
+end

spec/model_spec.rb

@@ -3,64 +3,199 @@ require "model_helper"
 
 describe Redistat::Model do
   include Redistat::Database
-  
+
   before(:each) do
-    db.flushdb
+    @time = Time.utc(2010, 8, 28, 12, 0, 0)
+    ModelHelper1.redis.flushdb
+    ModelHelper2.redis.flushdb
+    ModelHelper3.redis.flushdb
+    ModelHelper4.redis.flushdb
   end
-  
+
   it "should should name itself correctly" do
-    ModelHelper.send(:name).should == "ModelHelper"
+    ModelHelper1.send(:name).should == "ModelHelper1"
     ModelHelper2.send(:name).should == "ModelHelper2"
   end
-  
+
+  it "should return a Finder" do
+    two_hours_ago = 2.hours.ago
+    one_hour_ago  = 1.hour.ago
+    finder = ModelHelper1.find('label', two_hours_ago, one_hour_ago)
+    finder.should be_a(Redistat::Finder)
+    finder.options[:scope].to_s.should == 'ModelHelper1'
+    finder.options[:label].to_s.should == 'label'
+    finder.options[:from].should  == two_hours_ago
+    finder.options[:till].should  == one_hour_ago
+  end
+
+  it "should #find_event" do
+    Redistat::Event.should_receive(:find).with('ModelHelper1', 1)
+    ModelHelper1.find_event(1)
+  end
+
   it "should listen to model-defined options" do
     ModelHelper2.depth.should == :day
     ModelHelper2.store_event.should == true
     ModelHelper2.hashed_label.should == true
-    
-    ModelHelper.depth.should == nil
-    ModelHelper.store_event.should == nil
-    ModelHelper.hashed_label.should == nil
-    ModelHelper.depth(:hour)
-    ModelHelper.depth.should == :hour
-    ModelHelper.store_event(true)
-    ModelHelper.store_event.should == true
-    ModelHelper.hashed_label(true)
-    ModelHelper.hashed_label.should == true
-    ModelHelper.options[:depth] = nil
-    ModelHelper.options[:store_event] = nil
-    ModelHelper.options[:hashed_label] = nil
-    ModelHelper.depth.should == nil
-    ModelHelper.store_event.should == nil
-    ModelHelper.hashed_label.should == nil
+    ModelHelper2.scope.should be_nil
+    ModelHelper2.expire.should be_nil
+
+    ModelHelper1.depth.should == nil
+    ModelHelper1.store_event.should == nil
+    ModelHelper1.hashed_label.should == nil
+    ModelHelper1.depth(:hour)
+    ModelHelper1.depth.should == :hour
+    ModelHelper1.store_event(true)
+    ModelHelper1.store_event.should == true
+    ModelHelper1.hashed_label(true)
+    ModelHelper1.hashed_label.should == true
+    ModelHelper1.options[:depth] = nil
+    ModelHelper1.options[:store_event] = nil
+    ModelHelper1.options[:hashed_label] = nil
+    ModelHelper1.depth.should == nil
+    ModelHelper1.store_event.should == nil
+    ModelHelper1.hashed_label.should == nil
+
+    ModelHelper4.scope.should == "FancyHelper"
+    ModelHelper4.send(:name).should == "FancyHelper"
+    ModelHelper4.expire.should == {:hour => 24*3600}
   end
-  
+
   it "should store and fetch stats" do
-    ModelHelper.store("sheep.black", {:count => 6, :weight => 461}, 4.hours.ago)
-    ModelHelper.store("sheep.black", {:count => 2, :weight => 156})
-    
-    stats = ModelHelper.fetch("sheep.black", 2.hours.ago, 1.hour.from_now)
+    ModelHelper1.store("sheep.black", {:count => 6, :weight => 461}, @time.hours_ago(4))
+    ModelHelper1.store("sheep.black", {:count => 2, :weight => 156}, @time)
+
+    stats = ModelHelper1.fetch("sheep.black", @time.hours_ago(2), @time.hours_since(1))
     stats.total["count"].should == 2
     stats.total["weight"].should == 156
     stats.first.should == stats.total
-    
-    stats = ModelHelper.fetch("sheep.black", 5.hours.ago, 1.hour.from_now)
+
+    stats = ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1))
     stats.total[:count].should == 8
     stats.total[:weight].should == 617
     stats.first.should == stats.total
-    
-    ModelHelper.store("sheep.white", {:count => 5, :weight => 393}, 4.hours.ago)
-    ModelHelper.store("sheep.white", {:count => 4, :weight => 316})
-    
-    stats = ModelHelper.fetch("sheep.white", 2.hours.ago, 1.hour.from_now)
+
+    ModelHelper1.store("sheep.white", {:count => 5, :weight => 393}, @time.hours_ago(4))
+    ModelHelper1.store("sheep.white", {:count => 4, :weight => 316}, @time)
+
+    stats = ModelHelper1.fetch("sheep.white", @time.hours_ago(2), @time.hours_since(1))
     stats.total[:count].should == 4
     stats.total[:weight].should == 316
     stats.first.should == stats.total
-    
-    stats = ModelHelper.fetch("sheep.white", 5.hours.ago, 1.hour.from_now)
+
+    stats = ModelHelper1.fetch("sheep.white", @time.hours_ago(5), @time.hours_since(1))
     stats.total[:count].should == 9
     stats.total[:weight].should == 709
     stats.first.should == stats.total
   end
-  
-end
+
+  it "should store and fetch grouping enabled stats" do
+    ModelHelper1.store("sheep/black", {:count => 6, :weight => 461}, @time.hours_ago(4))
+    ModelHelper1.store("sheep/black", {:count => 2, :weight => 156}, @time)
+    ModelHelper1.store("sheep/white", {:count => 5, :weight => 393}, @time.hours_ago(4))
+    ModelHelper1.store("sheep/white", {:count => 4, :weight => 316}, @time)
+
+    stats = ModelHelper1.fetch("sheep/black", @time.hours_ago(2), @time.hours_since(1))
+    stats.total["count"].should == 2
+    stats.total["weight"].should == 156
+    stats.first.should == stats.total
+
+    stats = ModelHelper1.fetch("sheep/black", @time.hours_ago(5), @time.hours_since(1))
+    stats.total[:count].should == 8
+    stats.total[:weight].should == 617
+    stats.first.should == stats.total
+
+    stats = ModelHelper1.fetch("sheep/white", @time.hours_ago(2), @time.hours_since(1))
+    stats.total[:count].should == 4
+    stats.total[:weight].should == 316
+    stats.first.should == stats.total
+
+    stats = ModelHelper1.fetch("sheep/white", @time.hours_ago(5), @time.hours_since(1))
+    stats.total[:count].should == 9
+    stats.total[:weight].should == 709
+    stats.first.should == stats.total
+
+    stats = ModelHelper1.fetch("sheep", @time.hours_ago(2), @time.hours_since(1))
+    stats.total[:count].should == 6
+    stats.total[:weight].should == 472
+    stats.first.should == stats.total
+
+    stats = ModelHelper1.fetch("sheep", @time.hours_ago(5), @time.hours_since(1))
+    stats.total[:count].should == 17
+    stats.total[:weight].should == 1326
+    stats.first.should == stats.total
+  end
+
+  it "should connect to different Redis servers on a per-model basis" do
+    ModelHelper3.redis.client.db.should == 14
+
+    ModelHelper3.store("sheep.black", {:count => 6, :weight => 461}, @time.hours_ago(4), :label_indexing => false)
+    ModelHelper3.store("sheep.black", {:count => 2, :weight => 156}, @time, :label_indexing => false)
+
+    db.keys("*").should be_empty
+    ModelHelper1.redis.keys("*").should be_empty
+    db("ModelHelper3").keys("*").should have(5).items
+    ModelHelper3.redis.keys("*").should have(5).items
+
+    stats = ModelHelper3.fetch("sheep.black", @time.hours_ago(2), @time.hours_since(1), :label_indexing => false)
+    stats.total["count"].should == 2
+    stats.total["weight"].should == 156
+    stats = ModelHelper3.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1), :label_indexing => false)
+    stats.total[:count].should == 8
+    stats.total[:weight].should == 617
+
+    ModelHelper3.connect_to(:port => 8379, :db => 13)
+    ModelHelper3.redis.client.db.should == 13
+
+    stats = ModelHelper3.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1), :label_indexing => false)
+    stats.total.should == {}
+
+    ModelHelper3.connect_to(:port => 8379, :db => 14)
+    ModelHelper3.redis.client.db.should == 14
+
+    stats = ModelHelper3.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1), :label_indexing => false)
+    stats.total[:count].should == 8
+    stats.total[:weight].should == 617
+  end
+
+  describe "Write Buffer" do
+    before(:each) do
+      Redistat.buffer_size = 20
+    end
+
+    after(:each) do
+      Redistat.buffer_size = 0
+    end
+
+    it "should buffer calls in memory before committing to Redis" do
+      14.times do
+        ModelHelper1.store("sheep.black", {:count => 1, :weight => 461}, @time.hours_ago(4))
+      end
+      ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1)).total.should == {}
+
+      5.times do
+        ModelHelper1.store("sheep.black", {:count => 1, :weight => 156}, @time)
+      end
+      ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1)).total.should == {}
+
+      ModelHelper1.store("sheep.black", {:count => 1, :weight => 156}, @time)
+      stats = ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1))
+      stats.total["count"].should == 20
+      stats.total["weight"].should == 7390
+    end
+
+    it "should force flush buffer when #flush(true) is called" do
+      ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1)).total.should == {}
+      14.times do
+        ModelHelper1.store("sheep.black", {:count => 1, :weight => 461}, @time.hours_ago(4))
+      end
+      ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1)).total.should == {}
+      Redistat.buffer.flush(true)
+
+      stats = ModelHelper1.fetch("sheep.black", @time.hours_ago(5), @time.hours_since(1))
+      stats.total["count"].should == 14
+      stats.total["weight"].should == 6454
+    end
+  end
+
+end

spec/options_spec.rb

@@ -0,0 +1,36 @@
+require "spec_helper"
+
+describe Redistat::Options do
+
+  before(:each) do
+    @helper = OptionsHelper.new
+    @helper.parse_options(:wtf => 'dude', :foo => 'booze')
+  end
+
+  it "should #parse_options" do
+    @helper.options[:hello].should == 'world'
+    @helper.options[:foo].should == 'booze'
+    @helper.options[:wtf].should == 'dude'
+    @helper.raw_options.should_not have_key(:hello)
+  end
+
+  it "should create option_accessors" do
+    @helper.hello.should == 'world'
+    @helper.hello('woooo')
+    @helper.hello.should == 'woooo'
+  end
+
+end
+
+class OptionsHelper
+  include Redistat::Options
+
+  option_accessor :hello
+
+  def default_options
+    { :hello => 'world',
+      :foo => 'bar' }
+  end
+
+
+end

spec/result_spec.rb

@@ -1,14 +1,14 @@
 require "spec_helper"
 
 describe Redistat::Result do
-  
+
   it "should should initialize properly" do
     options = {:from => "from", :till => "till"}
     result = Redistat::Result.new(options)
     result.from.should == "from"
     result.till.should == "till"
   end
-  
+
   it "should have set_or_incr method" do
     result = Redistat::Result.new
     result[:world].should be_nil
@@ -17,5 +17,5 @@ describe Redistat::Result do
     result.set_or_incr(:world, 8)
     result[:world].should == 11
   end
-  
-end
+
+end

spec/scope_spec.rb

@@ -2,20 +2,20 @@ require "spec_helper"
 
 describe Redistat::Scope do
   include Redistat::Database
-  
+
   before(:all) do
     db.flushdb
   end
-  
+
   before(:each) do
     @name = "PageViews"
     @scope = Redistat::Scope.new(@name)
   end
-  
+
   it "should initialize properly" do
     @scope.to_s.should == @name
   end
-  
+
   it "should increment next_id" do
     scope = Redistat::Scope.new("Visitors")
     @scope.next_id.should == 1
@@ -23,5 +23,5 @@ describe Redistat::Scope do
     @scope.next_id.should == 2
     scope.next_id.should == 2
   end
-  
-end
+
+end

spec/spec_helper.rb

@@ -2,11 +2,17 @@
 $LOAD_PATH.unshift(File.dirname(__FILE__))
 $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
 
+require 'simplecov'
+SimpleCov.start do
+  add_filter '/spec'
+  add_filter '/vendor'
+end
+
 # require stuff
 require 'redistat'
 require 'rspec'
 require 'rspec/autorun'
 
 # use the test Redistat instance
-Redistat.connect({:port => 8379, :db => 15})
-Redistat.flush
+Redistat.connect(:port => 8379, :db => 15, :thread_safe => true)
+Redistat.redis.flushdb

spec/summary_spec.rb

@@ -2,7 +2,7 @@ require "spec_helper"
 
 describe Redistat::Summary do
   include Redistat::Database
-  
+
   before(:each) do
     db.flushdb
     @scope = "PageViews"
@@ -10,28 +10,43 @@ describe Redistat::Summary do
     @date = Time.now
     @key = Redistat::Key.new(@scope, @label, @date, {:depth => :day})
     @stats = {"views" => 3, "visitors" => 2}
+    @expire = {:hour => 24*3600}
   end
-  
+
   it "should update a single summary properly" do
-    Redistat::Summary.update(@key, @stats, :hour)
+    Redistat::Summary.send(:update_fields, @key, @stats, :hour)
     summary = db.hgetall(@key.to_s(:hour))
     summary.should have(2).items
     summary["views"].should == "3"
     summary["visitors"].should == "2"
-    
-    Redistat::Summary.update(@key, @stats, :hour)
+
+    Redistat::Summary.send(:update_fields, @key, @stats, :hour)
     summary = db.hgetall(@key.to_s(:hour))
     summary.should have(2).items
     summary["views"].should == "6"
     summary["visitors"].should == "4"
-    
-    Redistat::Summary.update(@key, {"views" => -4, "visitors" => -3}, :hour)
+
+    Redistat::Summary.send(:update_fields, @key, {"views" => -4, "visitors" => -3}, :hour)
     summary = db.hgetall(@key.to_s(:hour))
     summary.should have(2).items
     summary["views"].should == "2"
     summary["visitors"].should == "1"
   end
-  
+
+  it "should set key expiry properly" do
+    Redistat::Summary.update_all(@key, @stats, :hour,{:expire => @expire})
+    ((24*3600)-1..(24*3600)+1).should include(db.ttl(@key.to_s(:hour)))
+    [:day, :month, :year].each do |depth|
+      db.ttl(@key.to_s(depth)).should == -1
+    end
+
+    db.flushdb
+    Redistat::Summary.update_all(@key, @stats, :hour, {:expire => {}})
+    [:hour, :day, :month, :year].each do |depth|
+      db.ttl(@key.to_s(depth)).should == -1
+    end
+  end
+
   it "should update all summaries properly" do
     Redistat::Summary.update_all(@key, @stats, :sec)
     [:year, :month, :day, :hour, :min, :sec, :usec].each do |depth|
@@ -45,28 +60,80 @@ describe Redistat::Summary do
       end
     end
   end
-  
+
+  it "should update summaries even if no label is set" do
+    key = Redistat::Key.new(@scope, nil, @date, {:depth => :day})
+    Redistat::Summary.send(:update_fields, key, @stats, :hour)
+    summary = db.hgetall(key.to_s(:hour))
+    summary.should have(2).items
+    summary["views"].should == "3"
+    summary["visitors"].should == "2"
+  end
+
+  it "should inject stats key grouping summaries" do
+    hash = { "count/hello" => 3, "count/world"   => 7,
+             "death/bomb"  => 4, "death/unicorn" => 3,
+             :"od/sugar"   => 7, :"od/meth"      => 8 }
+    res = Redistat::Summary.send(:inject_group_summaries, hash)
+    res.should == { "count" => 10, "count/hello" => 3, "count/world"   => 7,
+                    "death" => 7,  "death/bomb"  => 4, "death/unicorn" => 3,
+                    "od"    => 15, :"od/sugar"   => 7, :"od/meth"      => 8 }
+  end
+
+  it "should properly store key group summaries" do
+    stats = {"views" => 3, "visitors/eu" => 2, "visitors/us" => 4}
+    Redistat::Summary.update_all(@key, stats, :hour)
+    summary = db.hgetall(@key.to_s(:hour))
+    summary.should have(4).items
+    summary["views"].should == "3"
+    summary["visitors"].should == "6"
+    summary["visitors/eu"].should == "2"
+    summary["visitors/us"].should == "4"
+  end
+
+  it "should not store key group summaries when option is disabled" do
+    stats = {"views" => 3, "visitors/eu" => 2, "visitors/us" => 4}
+    Redistat::Summary.update_all(@key, stats, :hour, {:enable_grouping => false})
+    summary = db.hgetall(@key.to_s(:hour))
+    summary.should have(3).items
+    summary["views"].should == "3"
+    summary["visitors/eu"].should == "2"
+    summary["visitors/us"].should == "4"
+  end
+
+  it "should store label-based grouping enabled stats" do
+    stats = {"views" => 3, "visitors/eu" => 2, "visitors/us" => 4}
+    label = "views/about_us"
+    key = Redistat::Key.new(@scope, label, @date)
+    Redistat::Summary.update_all(key, stats, :hour)
+
+    key.groups[0].label.to_s.should == "views/about_us"
+    key.groups[1].label.to_s.should == "views"
+    child1 = key.groups[0]
+    parent = key.groups[1]
+
+    label = "views/contact"
+    key = Redistat::Key.new(@scope, label, @date)
+    Redistat::Summary.update_all(key, stats, :hour)
+
+    key.groups[0].label.to_s.should == "views/contact"
+    key.groups[1].label.to_s.should == "views"
+    child2 = key.groups[0]
+
+    summary = db.hgetall(child1.to_s(:hour))
+    summary["views"].should == "3"
+    summary["visitors/eu"].should == "2"
+    summary["visitors/us"].should == "4"
+
+    summary = db.hgetall(child2.to_s(:hour))
+    summary["views"].should == "3"
+    summary["visitors/eu"].should == "2"
+    summary["visitors/us"].should == "4"
+
+    summary = db.hgetall(parent.to_s(:hour))
+    summary["views"].should == "6"
+    summary["visitors/eu"].should == "4"
+    summary["visitors/us"].should == "8"
+  end
+
 end
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-

spec/synchronize_spec.rb

@@ -0,0 +1,125 @@
+require "spec_helper"
+
+module Redistat
+  describe Synchronize do
+
+    let(:klass) { Synchronize }
+
+    describe '.included' do
+      it 'includes InstanceMethods in passed object' do
+        base = mock('Base')
+        base.should_receive(:include).with(klass::InstanceMethods)
+        klass.included(base)
+      end
+    end # included
+
+    describe '.monitor' do
+      it 'returns a Monitor instance' do
+        klass.monitor.should be_a(Monitor)
+      end
+
+      it 'caches Monitor instance' do
+        klass.monitor.object_id.should == klass.monitor.object_id
+      end
+    end # monitor
+
+    describe '.thread_safe' do
+      after { klass.instance_variable_set('@thread_safe', nil) }
+
+      it 'returns value of @thread_safe' do
+        klass.instance_variable_set('@thread_safe', true)
+        klass.thread_safe.should be_true
+      end
+
+      it 'defaults to false' do
+        klass.thread_safe.should be_false
+      end
+
+      it 'uses #synchronize' do
+        klass.monitor.should_receive(:synchronize).once
+        klass.thread_safe.should be_nil
+      end
+    end # thread_safe
+
+    describe '.thread_safe=' do
+      after { klass.instance_variable_set('@thread_safe', nil) }
+
+      it 'sets @thread_safe' do
+        klass.instance_variable_get('@thread_safe').should be_nil
+        klass.thread_safe = true
+        klass.instance_variable_get('@thread_safe').should be_true
+      end
+
+      it 'uses #synchronize' do
+        klass.monitor.should_receive(:synchronize).once
+        klass.thread_safe = true
+        klass.instance_variable_get('@thread_safe').should be_nil
+      end
+    end # thread_safe=
+
+    describe "InstanceMethods" do
+      subject { SynchronizeSpecHelper.new }
+
+      describe '.monitor' do
+        it 'defers to Redistat::Synchronize' do
+          klass.should_receive(:monitor).once
+          subject.monitor
+        end
+      end # monitor
+
+      describe '.thread_safe' do
+        it ' defers to Redistat::Synchronize' do
+          klass.should_receive(:thread_safe).once
+          subject.thread_safe
+        end
+      end # thread_safe
+
+      describe '.thread_safe=' do
+        it 'defers to Redistat::Synchronize' do
+          klass.should_receive(:thread_safe=).once.with(true)
+          subject.thread_safe = true
+        end
+      end # thread_safe=
+
+      describe 'when #thread_safe is true' do
+        before { subject.stub(:thread_safe).and_return(true) }
+
+        describe '.synchronize' do
+          it 'defers to #monitor' do
+            subject.monitor.should_receive(:synchronize).once
+            subject.synchronize { 'foo' }
+          end
+
+          it 'passes block along to #monitor.synchronize' do
+            yielded = false
+            subject.synchronize { yielded = true }
+            yielded.should be_true
+          end
+        end # synchronize
+      end # when #thread_safe is true
+
+      describe 'when #thread_safe is false' do
+        before { subject.stub(:thread_safe).and_return(false) }
+
+        describe '.synchronize' do
+          it 'does not defer to #monitor' do
+            subject.monitor.should_not_receive(:synchronize)
+            subject.synchronize { 'foo' }
+          end
+
+          it 'yields block' do
+            yielded = false
+            subject.synchronize { yielded = true }
+            yielded.should be_true
+          end
+        end # synchronize
+      end # when #thread_safe is false
+
+    end
+
+  end # Synchronize
+end # Redistat
+
+class SynchronizeSpecHelper
+  include Redistat::Synchronize
+end

spec/thread_safety_spec.rb

@@ -0,0 +1,39 @@
+require "spec_helper"
+
+describe "Thread-Safety" do
+  include Redistat::Database
+
+  before(:each) do
+    db.flushdb
+  end
+
+  #TODO should have more comprehensive thread-safe tests
+
+  it "should incr in multiple threads" do
+    threads = []
+    50.times do
+      threads << Thread.new {
+        db.incr("spec:incr")
+      }
+    end
+    threads.each { |t| t.join }
+    db.get("spec:incr").should == "50"
+  end
+
+  it "should store event in multiple threads" do
+    class ThreadSafetySpec
+      include Redistat::Model
+    end
+    threads = []
+    50.times do
+      threads << Thread.new {
+        ThreadSafetySpec.store("spec:threadsafe", {:count => 1, :rand => rand(5)})
+      }
+    end
+    threads.each { |t| t.join }
+    result = ThreadSafetySpec.fetch("spec:threadsafe", 5.hours.ago, 5.hours.from_now)
+    result.total[:count].should == 50
+    result.total[:rand].should <= 250
+  end
+
+end