#Redis Failover

This is simple #failover code for our Redis master/slave config that we would want to wrap with some sort of monitoring to detect when to initiate a failover for a given machine. In that condition, we would both promote the appropriate slaves to master (the Redis command is: slaveof no one) and call failover_host on the bad host.

# This is a function we can execute to manually initiate a failover for an individual node.
def failover_node(node)
  node_index = $master_arr.index(node)
  # There is no remove_node for redis::distributed, so we must rebuild the ring
  #  but first swap slave/master.
  $master_arr[node_index] = $slave_arr[node_index]
  $slave_arr[node_index] = node
  $redis = Redis::Distributed.new $master_arr
end

# This function will failover an entire host machine.
#  e.g. failover_host('localhost')
def failover_host(host)
  $master_arr.each {|x|
    if x[:host] == host
      failover_node(x)
    end
  }
end

Again, do keep a close eye on the Redis Cluster with Sentinel projects, which will likely be the best supported version of HA for distributed servers in the future.

Advertisements

Redis Master/Slave Config

While there is the #Redis Sentinel project, which is your best bet for a more complete HA (High Availability) solution, we wanted a simple solution that would work for our distributed Redis nodes we recently described, which unfortunately will not work with Sentinel. We recommend you have slave nodes (ideally in a different site than the master Redis hosts to allow for better HA) that stream copies of any changes made in the master. Fortunately, Redis already has a concept of master/slave replication. In your config for these slave nodes, you simply have to specify: slaveof [master host] [master port].

This code assumes that the way we set up these slaves is in a top level range of ports above the master nodes on each machine. For example, if we had 3 processes per node and 3 slaves on that same node for 2 hosts:

# Set up pointers to the slave processes
#
# A slave will exist on a different host (if available) at opposite ports
#  so slave_port = (first_port+procs_per_host*2)-(master_port)-1
#  e.g. master1 = host1, port 1
#       slave1  = host2, port 6
#       master2 = host1, port 2
#       slave2  = host2, port 5
#       master3 = host1, port 3
#       slave3  = host2, port 4
#       master4 = host2, port 1
#       slave4  = host1, port 6
#       master5 = host2, port 2
#       slave5  = host1, port 5
#       master6 = host2, port 3
#       slave6  = host1, port 4
$slave_arr = Array.new(REDIS_NUMPROCS) {|i|
  {
    :host => REDIS_HOST[i/redis_procs_per_host],
    :db => 0,
    :port => REDIS_FIRST_PORT + redis_procs_per_host + i%redis_procs_per_host
  }
}.reverse

We can then use these pointers for executing failovers, which we will describe tomorrow.

Distributed Redis

We love #Redis on #Rails here at Nowcado. We also love #scalability. Redis has a distributed flavor that automagically shards your keys across multiple nodes (though of course keep an eye on Redis Cluster which looks exciting and hopefully will be stable soon!)

This is a rough layout of how our Redis initializer for Rails looks so that in our app we can reference all of our distributed Redis nodes with $redis. It is creating however many processes we define evenly distributed across however many hosts we define in our Redis config.

This is a snippet of our redis initializer to handle the distributed redis nodes defined in our config YAML which is set up to support development, test, and production environments.

# REDIS_NUMPROCS defines how many redis processes we will use.
# REDIS_HOST is the array of available redis server hosts.
# REDIS_FIRST_PORT is the first sequential TCP port for the redis server array.

require 'redis/distributed'
redis_procs_per_host = REDIS_NUMPROCS / REDIS_HOST.length
# Set up the initial master processes
$master_arr = Array.new(REDIS_NUMPROCS) {|i|
  {
    :host => REDIS_HOST[i/redis_procs_per_host],
    :db => 0,
    :port => REDIS_FIRST_PORT + i%redis_procs_per_host,
    :driver => :hiredis
  }
}
if $redis.nil?
  # set up the global $redis variable
  $redis = Redis::Distributed.new $master_arr
end

Make sure that all your redis nodes are set up in the manner this is expecting, which means you should have redis_procs_per_host number of process running on each host with ports ranging from REDIS_FIRST_PORT to REDIS_NUMPROCS / REDIS_HOST.length. Note that we use the hiredis driver to improve performance.

From an application perspective this becomes dead simple to work with (just have to reference $redis as mentioned), though having all of these distributed nodes means high availability becomes very important. Stay tuned for when we cover how we set up site-resilient slave nodes tomorrow!

National Tell A Story Day

As today is National #TellAStory Day, we have written a short story – enjoy!

A long time ago in a college town far, far away…

War on Univosia! The rebel Shoperians and imperial Rotsians are caught in a firefight in the neutral zone. Upset by the devastation caused to their home turf, the local and impartial Lartunes have no choice but to take action.

The leader of the rebels (Simon) meets with the head of the Lartune collective (Neil)…

Simon: We come to convince you to join our cause.
Neil: What cause is that?
Simon: The greatest cause – ending the tyranny and oppression that the Rotsians have forced upon us. We are unable to decide for ourselves any aspect of our lives, but must instead live by their code, even when it conflicts with our own.
Neil: I sympathize with your plight, but we are a small group here and don’t have much to offer in way of reinforcements.
Simon: I understand. We are hoping for any assistance that you might be able to provide, though, as your people are known for inventing clever solutions to tricky problems.
Neil: I will discuss with the other members of the collective to see if we might develop some solution to your problem.

The next day, they meet again.

Neil: We think we have an idea how we might save your people and avoid needless bloodshed in the process. We will meet with the Rotsian Council and devise a system that disallows them from controlling you, yet at the same time makes them happy so they have no reason to fight.
Simon: I cannot imagine them agreeing to any such deal. Rotsians want nothing more than total power over all things.
Neil: We do not believe that to be true. What they actually want is much simpler – to sustain themselves, just as you do. Give us time with the Council. We are confident they will come to our way of understanding and leave you alone.
Simon: As you wish.

Neil goes to meet with the Council and lays out the Lartunes’ plan, codenamed “Nowcado.” He describes a system where the Rotsians can still oversee the security of the planet, yet have no ability to totally control it, nor to spy on every citizen. The Shoperians can further gain additional data on the Rotsians as they see fit. The Rotsians seem hesitant at first to the power being granted to the Shoperians, but agree that this solution will prevent further rebel attacks and still satisfy their core goal of securing Univosia.

Shortly after, an ambassador of the Council and Simon meet to work out the terms of the new world order, and Univosia enters into a new golden age.

OLW (One Liner of the Week): Convert To Int Array

Another #rubyoneliner – we use this one all the time in our backend controllers when dealing with parameters that should be arrays.string_or_array.to_s.gsub(/[\[\]”‘\\]/,””).split(“,”).map {|val| val.strip.to_i}

It is particularly interesting because it can take any string or array of strings/ints/arrays/etc and convert it into an array of ints, as long as the elements are separated by commas.

“I’m angrier than a dragon trying to blow out candles!”

#RoR NoSQL Migrations: Part 3

Here is how a #nosql migration for #elasticsearch might look using the rake task we built up previously in part 1 and part 2. In this case, we are creating a new mapping and saving all our SQL models which have been configured to update their ES indices when the underlying SQL data is saved.

begin
  $es.indices.create  index: 'items',
    body: {
      mappings: {
        item: {
          _id: { store: true },
          properties: {                            
            name: { type: 'string' }
          }
        }
      }
    }

  Item.all.each {|i|
    i.save
  }
rescue
  abort("Migration failed.")
end

Notice the begin – rescue block, where on errors we write out the failure “Migration failed” to the STDERR stream which will cause the wrapping system command in our rake task to be false. We could potentially view the STDERR stream and see the details of that failure with any detail we choose to include in the abort call.