Moving offices....again

Another office move! I love these things, actually I don’t but they’re not that bad. Ok so there is a lot that can go wrong and it’s a lot of hours which no-one notices you doing but at the you’ve created something!

My company is, for the 6th time since I started working there, moving it’s offices around. I actually have 2 to do within the next 6 months so I will be pretty busy trying to juggle that with all the other stuff.

There are some interesting elements to this particular office move in which I am hoping to “do away” with the server/comms room in its traditional form as it consumes too much space; but more of that later. In this first little post I am just going to say that I am going to write it up! Hopefully some of the considerations and workings out will be useful to someone when they have to plan their first office move and start to panic about more or less everything.

More later…..

Increased costs for all leads to more home working?

Been a while since I have written one of these but a lot has been going on recently so it’s time to write some of it up!

As anyone who lives in the UK and has to drive to work knows fuel duty and VAT both went up recently pushing fuel prices to triple what they were in 1990.  Accompanying that are the calls for static salaries, salary reductions or even redundancies within companies who are trying all they can to cut costs.

Cost cutting tends to focus on the “big numbers” and office spends are a big number for any organisation with any real return on investment.

From both the employer and the employee, therefore, “home working” becomes an attractive option to save money in both office and travel costs. Reduction in travel costs for the employee and office space for the employer seems attractive but masks the true cost of someone working from their home environment and the implications for both the employee and employer.

From an IT perspective there are further costs and considerations which need to be addressed in a move towards home working.

In no particular order these could be:

• Home equipment - what are these people actually going to use? If they are coming into the office and hot desking then likely these will be laptops. You may also wish to use docking stations and screens for home use. This adds up quickly to become multiple times more expensive than a normal desktop.
• Security - a bit like the above. How are you going to ensure the security of your network? What about computer theft from the home and consequent data loss?
• Home internet connections - how are these going to be provisioned? There are a lot of options here depending on your set-up but lets assume for now we are talking about an ADSL line. Are they going to be supplied by the company and therefore maintained by the company or are you going to run it all through expenses?
• Infrastructure - All these people now working from home are going to place demands on the existing firewalls which might not be able to handle the multiple VPN sessions or the increased throughput. Use of back office applications that are needed throughout the organisation may suffer in performance and thus decrease productivity. Investment in technology to bring LAN-like connectivity to the home workers is needed which costs money and resources to run them.
• Telephony - How is your organisation going to communicate? Are you replacing the cost of an office with the cost of having everyone travel to meetings and claiming it all back on expenses? Video conferencing and VoIP are good technologies but can put stress on existing networks and, again, require resources to implement and manage them.
• Hot desks and meeting rooms - Ultimately you are going to have people coming into the office for meetings. When in the office they are going to use hot desks and meeting rooms; does the organisation have enough of either?
• Remote workers - you might already have a set of these but remember they are often coming back into the office or making allowances for the things they cant do remotely. They NEED to be out of the office (Sales people for instance) and their productivity would decrease further if there access methods were under performing due to increase use from home workers.
• Training - All this new technology that everyone is going to have to use to do their jobs (such as VPNs and Citrix) are your staff going to know how to use it or are they going to require training?

When a CEO or CFO starts talking about home working I can’t help but think of the infrastructure upgrades that are neccessary to facilitate this change. Failure in IT to provide a similar service to that experienced in an office environment could easily lead to a loss in productivity and confidence with the IT department. Uptime and redundancy within the infrastructure also costs money and a higher level of technical skill to maintain. Outsourcing is a option to overcome so of these scale issues but that similarly pushes costs up.

So; does people working home actually save an organisation money? Probably not and it really shouldn’t be your primary business driver for change.

Acquisition time!

This morning the company I work for bought its 5th company in the last 3 years taking our total headcount to around 350 and firmly pushing us into the middle of SME. In that time it has diversified from a single product offering of a document management system to 6 (maybe 7 now) main products and pure on-line businesses. The recent purchase is another one of diversification which takes us into the pharmaceutical and oil/gas market; which is somewhere we have never been before!
This is a big change for us not least because of the different requirements from these new customers and their expectations of their suppliers but also simple things like terminology and ethos.

It used to be that our IT department was closely aligned to the business’ products and were able to deliver those products internally through a close understanding of their development. More recently, however, we have only been able to deliver frameworks in which the business units themselves can deliver those products.
Largely the reason for this is the business rate of change. With that much increase in product and that many different markets it is difficult to see how we could understand each of the products particular foibles to the point we can provide effective consultancy on them. So instead our role becomes one of a generic back office role trying to support those individuals who can deliver each of the software components.

The more the company diversifies the more we become generic and I can’t help but feel that our IT department is turning into an internal outsourced unit where we provide services to the business as a whole and then set SLAs for them to work against. Our role becomes focused on trying to provide a flexible, affordable and resilient infrastructure to feed business units increasingly hungry for more resource.

I have been promoting the idea of a “brand” for IT internally for sometime now, trying to show everyone what we do and allow them to see the performance of our systems and rate our level of work. I quite like the idea of making us even more of an internal outsourcing dept but can’t help but think it is exposing us to a large risk of being outsourced ourselves!

So here we are and we have a choice to make. The business as a whole is changing quickly (as it has to) and we need to change with it. We might expose ourselves to risk but show me a role that is ever secure! By providing a quality and open service it might quicken the road to our departure but equally it might also give others confidence in what we do and appreciate the value we add to their business unit and the group as a whole.

Security or productivity?

Like many of my colleagues I am often presented with a lesser-of-two-evils choice and  security is often one of those choices. In today’s consolidating market, with many SMEs facing a downturn in sales, a company with an acquisition strategy is like the proverbial kid in the sweet shop.

For me, working in one such organisation, the merging of companies into the ever growing group structure is now a regular occurrence.

Most mergers are about getting more for less and requires integration of business processes  into the new, larger, group as quickly as possible. Once the merger is completed the number of people maintaining processes can be reduced and the profitability of activities increased.

In broad strokes this makes sense, but only if you inject initial capital into merging those systems and processes. Compromises are often made where that capital does not cover the cost of integrating those services, or indeed even exist. At this point your security-vs-productivity kicks in and you would be hard pushed to find a board outside the financial services sector that would choose security over profit.

The problem can be particularly acute when faced with the acquisition of a smaller company where there is no formal IT department, or a one man band who was also doing 3 other jobs at the same time. Usually IT security would have suffered in an effort to ‘just make it work’, often in pressured circumstances. These situations lead to a lack of knowledge and make integration even more difficult and prone to outages.

When presented with a new company’s infrastructure and bolting it onto an existing company’s environment you could end up with any number of the following:

• Multiple domains
• Weak or inconsistent password policies
• Inconsistent remote access
• Unsecured user-published services

Really I would not be surprised if you ended up with all of those and more.

As an example:

• We currently have 6 production domains (yes 6), only one of which is destined to survive
• Each domain has a different password policy; we cannot implement the same across all of them
• Fortunately we have managed to get consistent remote access and that is compatible with the domain that is staying
• We still have user-published services that are all part of the security-vs-productivity balancing act

I would hope we have started in the right place by sorting out the remote access first. I would further hope that doing so tips about even on the security-vs-productivity scales.

Business led change or IT led change

“The time for passwords is gone because they can be captured easily by password sniffers, no matter how long and complex they are…”
Matthijs Van der Wel - head of EMEA forensics team at Verizon Business

This is a strong statement by Mr Van der Wel but it does ring true. The complexity of delivery mechanisms and the inability of Anti Virus software to protect against day zero or near day zero attacks leaves many computers infected. Where a machine is compromised there could be huge implications for the company or individual concerned.  What could help stop this is multiple for factor authentication; secureID dongles or biometric logons but these are more the domain of the enterprise. With multiple domains or a badly maintained permission domain underpinning an infrastructure, the implementation of enhanced security could at least be expensive, or worse not work at all!

Moreover, should an IT department be pushing through these changes to a business that does not understand or want them? Obviously this is where the risk log would come in to demonstrate offsetting the risk and cost of a security breach against the cost of implementing tougher security.  Even with a risk log conveying the ideas to a non-IT literate board could prove difficult if they view it in a similar way to a natural disaster. Yet if a river ran next to every office in the land how many would have flood defences just in case?

With security moving ahead rapidly in the enterprise through multi form authentication, Network Access Control and even down to limiting administrators access, the SME is starting to get left behind.  Yet SMEs are just as much in the firing line as enterprises and could be increasingly so as enterprises step up their security operations. The data protection act is there to protect the individual and the ICO can certainly impose heavy fines on any organisation in breech of it, but by then the damage is already done.

It is important to mention that integrated security is not just about defence, it can increase productivity with less downtime requesting access to resources and simplification of systems for IT staff and users alike. Expanding out active directory domains across the company makes it a powerful information repository to which systems can refer with confidence.

Closing thoughts

There are two subjects here; the integration of companies to achieve security and the security of the whole. The landscape is changing so quickly it is tough to keep up with threats, methods of working and even just the accessibility now demanded by users.  Traditional IT has led us down a route of a company infrastructure hidden behind firewalls in a secure network. If you rip that concept up and start to treat internal and external as insecure, each LAN and wireless network as a security threat, then you’re probably looking in the right place for a breach.

Should I even try to consolidate the IT infrastructure? Maybe 6 domains is the right number to ensure that any breach of security is isolated to that 1 domain.  Either way you wont be thanked for decreasing productivity and maybe you wont have that security breach…..

Avaya "desktop video device"

The new Avaya video desktop device and “Flare experience” is something I was recently introduced to at the UK executive briefing centre of Avaya.

The device itself is an Android powered 11.6 inch HD touch screen tablet, wireless capable with an inbuilt camera and a stand (and yes, I did ask if it had that specifically!). There are other bits and bobs of connectivity but its rather ancillary and not really critical to the primary function of the device.

The primary function!

Yes the primary function; it possibly has a secondary function as a phone but it’s designed with one thing in mind  - Video Conferencing.

Video conferencing has been around for a while and is now used heavily in business; presented in a variety of forms and integrated into phone systems this device takes that integration one step further. By using the same contacts list as the phone system it can facilitate video conferencing in the same way as you would make a voice call. To do this it runs an operating system called “Flare” and there is now a “Flare experience” (see YouTube clip) all about how this works.

The drag and drop create and manipulation of video conferences is they key selling point here and this might be the true selling point of the device. The Flare application sitting on top of the Android OS makes this platform very productive for the things it supports but with obvious limitations:

Shall I get one?

It’s a good product which should certainly be on the radar of any IT manager who has, or is going to implement, a company wide video conferencing system. Not necessarily to buy one but to see the new products in the market and the direction technology is taking.

At $2000 a time (what’s the chances this turns into £2000 when its sold over here?) it’s a very expensive single function device. It needs an existing video conferencing system to plug into, which wont be that common, and is only any use against that video conferencing system. It also has a very limited audience; ask yourself which person you can see in your company using it and is it anyone other than a director or CEO? Actually that’s a genuine question - can you see someone using it in a production capacity in your company?

For me that’s always been the problem with the video conferencing - we don’t actually like doing it! The technology to do this sort of thing has been around for over 10 years yet no-one has pushed to roll it out. In the same time period mobile phones have been hugely successful and developed into multi function devices which were nothing like the mobile phones of old. Yet here we are re-hashing the same concept when it’s been proved (over time) that no-one actually wants it!

I’m not a researcher, and perhaps I lack the insight of the Avaya board, but i don’t think this is going to go anywhere in its current form. It does represent a new development in video conferencing to a mobile device and that is a step forward - wonder what the next step will be….

Choosing a data centre

When I set out to choose a data centre I knew I had a lot to learn to make an informed choice. Plenty of research later, and an amount of time studying Electrical Engineering at University, helped me reach a decision. This article is a reflection on those decisions and how I came to them.

Data centres themselves are complex monoliths representing huge capital investments for businesses who need to sell rack space to recoup that investment.  Starting to choose one brings you many decisions to make based on the business you are trying to support. Your Board might have given you a budget, or perhaps you have to set one, either way they are going to think there servers are now in an indestructible fortress so you had better provide one for the right price.

So what does a data centre provide? Well typically a data centre will provide some or all of the following:

• Power
• Cooling
• Physical security
• Fire suppression
• IP Transit (read: networking)
• Physical redundancy of systems
• Remote or “Intelligent” hands
• Managed Hosting
• 24/7 access
• Location Location Location

Some of these maybe be crucial to your business but all of them need consideration if your going to work out which facility is appropriate your company.

Power

Power consumption ultimately will determine the price you will pay for your hosting it also might rule out some data centres which can’t provide you with what you need. As a comparison between data centres this is the thing you should be looking at.

The first thing you need to do is work out what exactly you are going to put into the data centre and what power requirements it will have. Be careful here that you don’t look at the peak power of your server/switch/unit as that will not tell you what it will draw usually but what it will draw under extreme load.  Adding this up and coming to a working total is a judgement call, you may choose to always have enough power if every unit peaks at the same time, you may choose to average out but this will ultimately be down to how much money you have to spend in your monthly rental budget.

Once you have worked out exactly how much power you need you can start to ask data centres what they can provide. Typically they will provide x AMPS to a rack for £y per month. So as a comparison (real world figures btw) i have the following:

Which clearly shows data centre A as being cheaper than B but C is not as clear. This is something that sales people will rely on when talking to you and discussing their rates “ahh but we give you 22 AMPS for that price not 16!” well yes you will but how much does it cost me per AMP???

So now it is clear that Data Centre C is very expensive in comparison to the others.

In my case we wanted to put a blade centre into our racks and that thing can really consume the amps. We worked out that top load is around 22 amps so started to ask each of the data centres if they could accommodate this. What’s key here is that if they are not setup for providing the power to the rack then they are bodging it. It may well work but you are going to be pushing more power to the rack than it is designed to take which could well provide complications later (see cooling).

In terms of what to look for in a data centre you should be thinking in terms of two feeds into your rack from diverse feeds and SLAs against the provided power. Servers are often dual PSU so they can work off either feed and you would plug one PSU into the A feed and one into the B feed. The data centre might also recommend that you look at Power Distribution Unit (PDU) to plug in your equipment and monitor your power. It’s out of the scope of this blog but we very happily run APC PDUs which allow us to turn plugs on and off remotely and check how power we are drawing.  If the data centre can’t offer you practical advice on this or doesn’t tell you in too much detail how the power works then I would start to worry.

Although you could go into this in great detail it is worth remembering your Service Level Agreement. If your SLA says “100% power” then you can be relatively confident that they know what they are doing, and if they don’t then they are going to be paying you your money back! I wouldn’t accept anything other than 100% power in an SLA document from a data centre.

Cooling

We take this for granted really, a server room will be adequately cooled. How well is this actually managed though and is it environmentally sound? The data centre in question should be considering how much it will have to cool per rack based on how much power it is supplying to that rack. If a data centre is willing to put more power to a rack than is in the design specifications for the data centre (by say moving power from a nearby rack) alarm bells would start to ring for me. By doing so they create “hot-spots” where the heat being generated is too much for the cooling system to handle and as a result equipment overheats.

For the new generation of data centres there is a push to have them totally air-cooled by increasing the footprint of the entire data centre such that there are no hot-spots. Do remember that cooling is merely the conduction of heat from inside the building to outside in the most efficient way possible, too closely packed servers stress the system that is doing that work and increase the potential for hot-spots and overheating servers.

You should check the SLA document again and ensure its another 100% and start to ask questions on how they manage the cooling. Very en-vogue is the idea of using hot and cold aisles which work exactly how they sound - cool air to the front of the racks (where its get drawn into the servers) and then hot air where all the exhaust air is pumped out. Management of the floor space like this is a good indication that the data centre know what they are doing and have it under control.

Physical Security

This is up to you and how important your company data is. You have a number of things to consider here; how easy is it to get into the building and the suite in which your racks are located? (If you ever wander round Telehouse you will know what i mean) Do you need more security than just a rack? Do you need remote monitoring of your racks? Are there enough security cameras around the site? (in the UK - are you kidding!) Does everyone going and out of the building have to sign in and out?

Whether or not you need these are up to you but could have further financial implications to your rack rental cost.

You will need to be sure that the data centre has the appropriate ISO accreditation (27001 - UK only)  and that they practice what they preach! Some of this can only be down to experience; if someone is blase about letting you in when you visit then they are blase when you are not there and those are your servers in there. Of course the only way you can ever test the physical security of a site is by going and having a look and I would ask them to show you the security features whilst you are there.

A security cage is another good option to protect your servers; essentially a large metal cage surrounding your racks with some sort of coded entry system. Whilst this might sound like a good idea it occupies floor space which they data centre would otherwise use for more racks so if you want it your going to have to pay for it….

Ultimately, should the worst happen, you might have to reclaim on someones insurance and you should ensure that either you or the data centre has some to the appropriate level.

Fire suppression

It just has to be there! Hopefully there will be huge red canisters on the wall indicating a gas fire suppression system and it wont be something involving sprinklers!  If a data centre had a sprinkler system I would be out of there, I don’t care how good your early detection is if the fall back is to destroy all my servers then I just am not interested.

Ask the question, ask how it works and then think of the what ifs; What if a nearby room catches fire? What if my rack catches fire? What if someone burns some toast?

IP Transit

This is the second part of the data centre comparison matrix. How much is your data centre going to charge you to move your data around? You will want your networking guys involved in this as this is a complex subject which is not fully explored in this particular blog.

What this comes down to is how much data you think you will be transmitting at the same time and not the volume. As this will be most of your servers in the data centre you can check on the switches using SNMP in your currently location and find out how much they are pushing backwards and forwards. Typically there will be a single up-link into the switch that governs all your servers or, if there is not one, its probably a good idea to make one just for the purposes of gathering this data. Use a product capable of recording the information presented through SNMP by the switch (a product such as PRTG) and you can find out your average and peak data rates.

A data centre will sell you as much bandwidth as you need and are usually in a monopoly to provide IP transit (bandwidth) around their own facility as well as a peering point to connect you to the rest of the Internet. You could of course run your own cables between your racks and other carriers but if you need to do this then I doubt you are reading this blog!

So having retrieved the price for (in this example) 20MB/s CDR you can continue to fill in the comparison table:

The unavailable for data centre C was due to them wanting us to peer with a carrier rather than supply us themselves so it would not be a direct comparison.

You can see clearly that data centre A is expensive in bandwidth terms compared to data centre B and, if you are using  a lot of bandwidth, this could well be a problem.

It is worth pointing out that if you are going to be a high bandwidth user then you will have to think very carefully about your physical location before committing to a data centre. The further you move out of London the more expensive it becomes. This isn’t a problem for a usual company but where you are a media company (or have similar high requirements) the costs of bandwidth could eclipse your rack charges, as I have no experience of high bandwidth usages I will comment no further.

Physical redundancy of systems

Data centres are built to a specification or “tier”. Each tier is defined by the facilities which the data centre offers and the level of redundancy of those systems. Typical UK data centres offer a N+1 redundancy meaning that you need N number of air conditioning units/power units/etc to make everything work so you put in N with 1 spare meaning N+1.

The N+1 is an example of a tier 3 data centre. For tier 4 data centres everything becomes N+N, so every system has a standby version of itself - highly redundant.

The choice here is one of risk and expense. How much risk is your business willing to take on their systems potentially going down should there be a double failure in the data centre? If the answer is “not at all” then you might start looking for a tier 4 data centre. Be aware that there aren’t that many of these around so they will charge a bit more for their services.

For a fuller description on tiers and some more information on how data centers are operated the Wikipedia page does well: http://en.wikipedia.org/wiki/Data_center

Remote or Intelligent hands

Remote or Intelligent hands are often offered by the data centre to carry out tasks on your behalf in your absence. Typically these are things like powering a server on and off, changing a backup tape, installing a piece of hardware or plugging a cable in. The limit here is what you feel comfortable with and what the data centre feels comfortable with in both time and experience.

Of the data centre we use they offer 45 minutes per day of Intelligent hands for free. In order to use those we have to log a call and they undertake the task. There is often the question of “do we feel comfortable asking them to do this?” and sometimes its yes sometimes no. Obviously where it’s something we want one of our members of staff to do it costs us much more to send them.

You should be looking to design your setup with the view of never returning and using those remote hands as a last resort, but you need to check they are available otherwise you might come unstuck.

Managed Hosting

The other way you could go is managed hosting. In this model the data centre takes a much more active role with your servers and manages them on your behalf. This is an expensive but often great service where the hosting company will manage all aspects of your servers up to and including the applications they serve. An Exchange or a public facing website is a good example of why you might wish to use this service.If, for instance, you are a small company with a large on-line presence and do not have the IT staff in house to run a 24/7 hosting service then managed hosting might be the route for you.

Just check: 24/7 access

Might sound obvious but can you get to your servers night and day without any restrictions? You may take this for granted but its worth asking the question before you sign on the dotted line. What about deliveries of equipment? What about building your servers on-site, is there a build room? Practical questions might change your perception of your supplier and could give you an insight into the way they work and think.

Just check: Location Location Location

The last thing to check is where the data centre physically is and what kind of risks its location poses. This may sound like a “oh no terrorists” type comment but is about measuring up risk. Three particular scenarios spring to mind which you might just fire up google earth to have a look around before you make a site visit:

• Terrorist Activity
• Accident
• Force Majeure

Terrorist Activity

OK this is unlikely but why expose yourself to a potential risk? Do you need to be in Canary Wharf (surely a prime target for disruption of uk business)? If you do need to be there does everything need to be there or can you split it up?

Accident

Remember Buncefield anyone? Northgate’s UK HQ was around 50metres away from the oil storage depot when it exploded. This more or less destroyed the building (see photos). Is your data centre next to something that could go bang? (For more information on the Buncefield disaster the official investigation site has a wealth of information)

Force Majeure

Are you on a flood plain? Believe me this is something i checked! Sounds daft but you bet the Vodafone data centre guys in Turkey have this on their list now:

Data Center security camera recording - Vodafone Istanbul, Turkey

Due Diligence

This is the same when working with any supplier for the first time:

• References
• Financial reports
• Director checks
• Credit rating
• Contract negotiation

These should be pretty standard so I’ll leave these alone.

Final thought - which one do you choose?

There are still other technical aspects of a data center decision that each IT leader must choose for themselves.  Most revolve around the kind of environment required by the business and what kind of IT environment that you want to deploy.  For a large SME (of which I currently work for) the choice is clear; get out of London and exploit the lower cost of data centres outside the M25 and the rapidly decreasing cost of bandwidth to get the most for your money.

Oh and if your interested in seeing how a data centre is built have a look here: http://blog.gyron.net/

Mobile Working

Something I have been considering for a while is how people work remotely in a safe and secure manner.
People want to be mobile, they want to be out there chatting with clients, they want to have few barriers to productivity.
This isn’t a fully formed blog post like it should be but some thoughts:

Mobile phones should be used to access content but they should no unique content on them. If you are going to use a mobile phone start with the permise that it is going to be lost and work out what would you do in that situation. So not having unique content on them means they are disposable, oh and they should be easily wiped and secure.

Dial up networking using mobiles: to ensure that people can gain access to what they need try to ensure that they can tether the mobile they are using to a laptop - means they don’t have to connect to some horrid wifi (if it is even available).

VPN offerings away from firewalls. Outside a corporate network employees need to be connected back via VPN to access resources. This should be painless. easy to control, stable and the default for every device that is in use. Avoid licenced products if you can as that will move you away from the “default” mindset.

Voip. Connecting employees to the internal phone system via voip does not mean carrying around a handset. There are softphones for laptops and for mobile phones - use them!!

Blog for IT things

Before starting to think about this blog I started to realise quite how many times I had researched a piece and formed an opinion only to forget how I ended up there.

I guess the intention of this site is to formalise some of my thinking and ultimately share it.

It’s interesting to both have your views challenged and to challenge others, you may still disagree but you may also understand someone else’s viewpoint.