Adventures of a Technical MasterMind

New blog location

2011-03-31T22:14:00.000-04:00

I am putting this blog to rest. I will be writing about upcoming ideas, research, and industry happenings on my new work blog located here : http://blogs.gartner.com/jonah-kowall

Of course you can always find more relaxed, less structed Jonah on twitter at http://twitter.com/jkowall

Look forward to having you all follow me over to my new location, and hopefully you will enjoy my writing and insight.

Off to a new adventure

2011-03-14T12:37:00.000-04:00

I am just starting a new position with Gartner as a research director in the IT Operations space : http://www.gartner.com/it/products/research/asset_129494_2395.jsp

I am very excited to be making this career change, and I believe I will be able to contribute heavily to the content and direction of the areas that I have a true passion for.

I will be moving my blog to a Gartner blog in the coming month, and writing there instead. I will post a link to my new blog here once its online. Please be patient as I move over to the new systems, and get acclimated with my new position.

Patching and updating for home and corporate

2011-02-26T10:37:00.000-05:00

We all are well aware of the Microsoft patches and windows update. Same goes for those of us who use itunes and iOS devices, we know Apple Software Update. Some of us may even patch our Adobe products, which we should since they have been the largest attack vector (http://goo.gl/bOQ3D) for the past 2 years hands down. This is just at home.... How do you expect the security experts to keep on top of all of these patches in a corporate environment. The number of patches for Oracle alone is daunting to understand and analyze.

There are ways to do this, you can use some clever software which I will outline below, or you can read ~25 RSS feeds and analyze vendor security bulletins. I do enjoy doing some of this, but I don't have time to keep on top of all the releases. Here is some software for home and corporate use to help manage this.

Corporate Patch Management:

Microsoft WSUS and SCCM - This is free and a no brainer for patching your desktops in a corporate environment. If you need tighter control you can implement System Center Configuration Manager (SCCM, formerly SMS) which does a good job. Microsoft has invested heavily in these products and brought them a lot way from the old SMS days.
Shavlik HFnetchk pro - Great product, but the price hasn't fallen as the competition has heated up. This product does a good job with other products outside of Microsoft, but doesn't support Linux or other *NIX variants.
GFI Languard - Cheap tool and does a good job with patching servers or desktops. Normally I recommend this for servers since it does a good job with general auditing too. Works well on Linux as well.
Lumension Patch Management (formerly Patchlink) - Used this previously, its a mixture of online services and software in house. Its a good product, and can handle multiple operating systems. Cost is medium in range. I haven't used the other components of the suite, but I would be willing to test them out. When I was using the product it did a great job on Solaris, Linux, and Windows.
Manageengine Security Manager Plus - This product was missing some key features in managing the patch lists, but overall it worked well and it was inexpensive. I love some of the other tools by these guys who also bring us the zoho products. The product does support Linux as well as Windows.

Wish list : I wish spiceworks would get into this space!

Home Patch Management:

Windows Update - Turn it on, and use it at home. It also manages updates for other Microsoft products if you follow these directions to enable this feature : http://technet.microsoft.com/en-us/magazine/ff642466.aspx
Secunia PSI - If you are a moderately advanced user this product does a great job managing updates to all of your other system software. Highly recommend using this product. I haven't tested the commercial versions of the products, I would be willing to test them if someone from Secunia contacts me.
Mozilla Firefox/Thunderbird - These products and plugins do a good job with updating themselves, but when there is a new release, they don't auto update. This is something that Secunia would handle, for example moving from Firefox 3.5 to 3.6... or soon from 3.6 to 4.0. The extensions update themselves well on any of the products from Mozilla.
Chrome - This product has the smartest best update system of any. Since the application resides not in program files, but within the users home directory it doesn't have restrictions on what it can do to its own files. This is both a blessing and curse in the security realm. The downside is that if there is a security issue it could compromise the browser binary itself, versus something which is installed in a "secured" location such as program files. The product will notify you, via a small icon that it has downloaded a new version. When you close the browser and start it again its switched to it.

Antivirus and Antispyware for the Home and Corporate user

2011-02-26T09:29:00.000-05:00

Since endpoint protection is of growing importance in the security industry, you will see some major technical advanced this year. I expect this to be one of the hot buttons for acquisition as technologies such as whitelisting and more connected heuristics create a more advanced software landscape.

Since I have quite a bit of experiance and I do keep on top of the industry I believe that I should outline some of the tools I recommend in both cases.

One side note, I do not use MacOS so I cannot recommend any software for Macs. If Apple can apply some of the success they have had with iOS to OSX then I may have to spend more time thinking about them on the PC front.

Corporate AV/Antispyware:

Mcafee - Does a good job with a suite, but ePO is still a bloated tool which is hard to manage effectively without creating gaps.
Symantec - I don't recommend the products for endpoint protection, in my experience I see too many missed viruses with the product.
Trend Micro - Does a good job at a lower price point than the big two in AV
GFI Vipre - My personal favorite, this is a cheap easy endpoint antivirus solution which works very well. There are a few gaps in the product they should shore up in the next 6 months, but overall I would recommend this product in a corporate environment.

I have more I would recommend, but I am not going to cover them. Products I would like to test for corporate use: Avast and Microsoft. If I have time I will try both of them in the next couple weeks.

Personal AV/Antispyware:

Microsoft Security Essentials - This is what I would tell my mom to use, it works, and its maintenance free! Sometimes this is the best option for a typical end user.
Avast - This is what I recommend and run. In version 6.0 they have added a lot of new features and improvements.
AVG - Another very good option, this would probably be my second choice personally
Microsoft Defender - Its free and does a decent job preventing spyware
Spybot S&D - Free and works great for spyware infections
CCleaner - Free, and this is a great overall system cleanup tool. It will fix spyware issues, registry issues, and other problems. If you don't run this type of tool every month or so, your windows installation will slow down in a much shorter amount of time.

Time to move on, and time to learn

2011-02-08T09:05:00.000-05:00

Sorry its been a while since I have posted last. I decided to move on from my previous position. I learned at a lot there, but it was getting somewhat slow the last 6 months. My skills are better used for more complex problems that the IT world is facing. I have had some pretty interesting discussions and interviews in the last week. I was hoping to enjoy my unemployment a little bit more.... maybe after my offer is signed.

Some non-technical notes, since I have a diverse set of skills I find my resume geared towards one type of career path and not another. This limited my opportunities to "get in the door" for some positions I think I would be well suited for. I have started by making some more detailed and targeted versions of my resume which are geared towards some of the other fields that I have experience in.

I wish there was a better way to manage and express all of your skills without having a 15 page resume, or 4 versions that are mostly the same. If anyone has comments or ideas please leave them!

If anyone wants to discuss any openings with me, there is a digsby chat here, or you can email me from the blog.

Now for some technology projects and learning:

Since I have some free time and feel that I have been off my Oracle game for the last 2-3 years I am evaluating and testing several products. Expect another post later this week on these products:

1. Oracle 11g and Weblogic (used these, but it was 3-4 years ago)
2. Oracle Enterprise Manager - Specifically around the VM management parts
3. Oracle Linux
4. Oracle VM
5. Oracle Real User Experience Insight - Never heard of this before, but it looks interesting!

I am looking at these to see where Oracle/Sun have gone the last couple years, and just as a general educational experience for me.

Sonicwall 5.8 New Features

2011-01-19T12:07:00.000-05:00

I'm very impressed by all the new features that Sonicwall has added to v5.8. I spoke to Jock Breitwieser and Eric Crutchlow who gave us a nice overview of some of the features. I wanted to make sure I didn't miss anything as there were quite a few changes. There are even more NG (Next Generation) Firewall features coming this year in v6.0. Sonicwall has been a great vendor for us, I am very happy we moved to their platforms.

Here is some of the new features (screenshots) with some explanation above the screen capture.

Before you can turn on the netflow sender (and built in netflow collector) you need to consider the impact this will have on your firewall. You should keep an eye on your CPU and ensure you aren't running above 50%, or you will probably have CPU contention issues. You also need to be aware that the memory usage of the netflow database and visualization will reduce your peek connections. This model is a NSA 3500 (http://www.sonicwall.com/us/products/NSA_3500.html) which previous to netflow could handle 35,000 stateful connections can only handle 49,152 when I have all of the DPI (Deep packet inspection) and Netflow enabled. This includes gateway antivirus, content filtering, IDS/IPS, and more. As you can see we don't go over 4000 which makes this a non-issue. Most firewalls can handle much higher workloads for connections and CPU than people are using.

First you have to turn on flow reporting. You can turn it on internally and or file it off to another netflow collector :

http://www.networkuptime.com/tools/netflow/
http://www.solarwinds.com/downloads/ - Solarwinds makes a nice free netflow collector as well.

Once you have netflow turned on you can see the applications below being collected, there are many views, graphs, and other options:

Here is a user list with our AD users and how much traffic, connections, and throughput they are doing. You will need to setup SSO for this to work (more on that later):

Here is the realtime monitor, this is the default view, which shows the main applications, you can filter by interface or by application and see realtime usage:

Here is a view of just some of the applications I picked : evernote, twitter, wikipedia, google, twitter:

Now for the SSO, you need to setup a small program on a windows host which will get requests from the firewall and lookup the users via active directory in order to correlate the DHCP (we use windows DHCP) address with the username. Here is the SSO setup screen:

When you click configure next to the "SonicWALL SSO Agent" then you get the following screen where you can setup the agent and view the stats to the agent process:

Sonicwall has also added in application control, which are rules which allow for much more customized actions based on the application detection engine. Before you can use these rules you need to turn App Control on (below), and then turn it on for a zone. After that you can control the applications. I have attached some additional applications below as well:

You can also define you own custom applications as well which can be used in rules :

The Content Filter can use Users/Zones (legacy) or App Rules which are far more flexible. They can be restricted and bandwidth controlled:

Email addresses and content can also be included in App Rules:

Here are the Action objects (what you want to do if an App Rule is matched). You can also define your own. Note that you have options such as Bandwitch Management, Blocking, Ignoring DPI, or enabling a packet capture session (for debugging those difficult to find issues).

If you have any question please leave a comment or hit me up on twitter or instant messenger. I hope you found this useful!

Dealing with Information Overload

2010-12-30T15:51:00.000-05:00

I follow a lot of different sites. I tend to read my news every other day. I'm always trying the best tools to deal with my data on multiple devices (iphone, ipad) and multiple computers (google crome OS, Windows). I have found the best combination to be the following:

1. Master feeds on google reader (works great on any device)
2. Feeddemon (Windows)

FeedDemon is a great app, it will sync with my google feeds and give me the same view of what I have read and what I have not. You have a better user interface for getting through all that data. I also customize the views to work for the way that I want to fly through all my news. These kinds of user interface tweaks aren't possible with google reader yet, but it will improve. Google reader is a great app and works great on mobile or desktop browsers.

I always use google chrome as my primary browser, it wins the speed race hands down, and supports the critical extensions I need. I have moved of Firefox as my primary browser over 6 months ago. I love the fact that my extensions and data (most of them) sync across every system and even the google netbook.

List of extensions I use:

AdBlock - Version: 2.2.19
Browser Button for AdBlock - Version: 0.0.12
gCast Weather - Version: 2.1.2
Google Translate - Version: 1.2.3.1
Google Voice (by Google) - Version: 2.2.3.4
IE Tab - Version: 1.4.30.4
LastPass - Version: 1.70.11
Lazarus: Form Recovery - Version: 3.0.1
PriceBlink - Version: 2.1
Tweetbeat Firsthand - Version: 0.6.3
Weather Window Beta by WeatherBug - Version: 1.0.5
Woot! - Version: 1.1
Xmarks Bookmark and Password Sync - Version: 0.9.0
Yoono Web - Version: 1.0.0.1

Google Chrome Netbook

2010-12-10T17:04:00.000-05:00

I came home today to find a strange looking box with odd shapes on the outside of it. Once I opened it I didn’t expect to find a small netbook there shipped from google. This is the Chrome OS Netbook they shipped to me for free. It’s a nice form factor, not too large and not too small. Once the OS booted up, you just hook it up to wifi, login with your google account, snap a profile picture, and you are off. After some time it upgraded itself, similar to chrome. It required a reboot after the update. It runs very well.

It does a good job with flash and other web media. It took quite a while to get my full extension sync from the PC, but most of my extensions worked without a hitch. The major one which doesn’t work is lastpass, which I really need! I tried many sites on it, and everything looks like it works well. I also installed several “apps” from the store which make for easy access to my google products and other sites I use a lot. It will make a good ipad type tool, something to grab and use. I haven’t setup the broadband yet, but I will soon.

The physical design is nice and very “black” I really like the additional of reload, forward, back, and search buttons. They are handy to have. I would have loved to have a “@” key. There is no need to have caps lock anymore, so it was nice to see it gone. The keyboard keyout is a little “off” for my touch typing, but I will get used to it. The control and alt keys are very large. If you multitouch on the trackpad it scrolls which is pretty handy once you get used to it.

The downsides to the design are the mouse, which allows for left and right click but doesn’t use buttons. It uses the click type pad that the macs use, which is somewhat annoying since I am used to having two hands on the trackpad. One hand for clicking and one hand for moving. If you try to do this with the trackpad it screws it all up. I would love to have page up and page down keys, as well as home/end keys. I am a power user and I use the page up and page down keys even more than the arrow keys when I am using the web.

Overall it’s a cool device and I’m looking forward to using it more extensively.

Great freeware system admin tools

2010-12-10T17:03:00.000-05:00

Thanks to the awesome community and software over at spiceworks - www.spiceworks.com I found these great free tools from Netwrix - www.netwrix.com that are superb for any system admin dealing with windows systems. I could have used this fileserver monitor freeware in the past for basic audits. I wish I had known about it sooner. The other really useful tools we are using are the following freeware tools. Essentially the free versions just email you at 3am every night with changes or reports. You can change the schedule using the scheduled task control panel:

Password Expiration Notifier - http://www.netwrix.com/password_expiration_notifier_freeware.html
This tool emails the end users when their passwords are expiring, its good for us because we have some remote users and mac users who do not get notified. This should prevent the lockouts we see when passwords expire.

AD Change Reporter - http://www.netwrix.com/active_directory_change_reporting_freeware.html
This product will show you changes in Active Directory, Exchange, and Group Policy. This is something I have been wanting to have for a while, but it was always too expensive. Now you can have it for free!

I emailed the company a few days ago to get pricing on the commercial products, but

Using AWS for larger business

2010-11-24T09:34:00.000-05:00

Netflix is one of those really secretive companies, there have beens some interesting articles how they run the operations for the disc delivery, but not much on the way the deliver digital content. I came across this really cool article on how they use AWS:

http://www.readwriteweb.com/cloud/2010/11/why-netflix-switched-its-api-a.php

Pretty interesting read. Not sure I agree with some of the statements about less system admin and less database folks when they use AWS. I can understand less datacenter staff, but managing virtual or cloud infrastructure is just as much work. Obviously this is only the case when its running customized software and databases built internally (such as netflix). You still need to release software, manage the databases, and handle the same problems you would if you were doing it all in house.

The only items you don't need to worry about would be the following:

Backups
Provisioning new hardware (which is pretty simple if you run your own vmware in house)

This is another good read on a website I read. They moved to EC2, and wrote a review 1 year later:

http://4sysops.com/archives/4sysops-one-year-in-the-cloud-part-1-costs/

The costs seem to be higher than using a normal colo server, so I'm not sure what the ROI would be for companies moving to EC2 or AWS. It would be good to see some more detailed comparisons of how companies use the services, and what the ROI is.

PCI compliance and SSLv2

2010-11-03T17:30:00.000-04:00

So I am doing a PCI audit, and one of the requirements is that there must not be weak cipher support enabled on systems which collect credit cards from the web. I started doing some testing around some of the larger ecommerce sites out there, and it had some pretty startling findings. SSLv3 has been in Browsers since 1996 (think mozilla 2.0... way before we had firefox).

http://blog.zenone.org/2009/03/pci-compliance-disable-sslv2-and-weak.html

From my testing these sites to have SSLv2 disabled: google, paypal, delta, etrade

These sites don’t have SSLv2 disabled, this is strictly against PCI: Home depot, bank of America, Scottrade, Microsoft, Amazon, QVC, Dell, Orbitz

Really concerning that these big commerce sites allow something like that to slip by the auditors. Time to hire me to fix your compliance :)

Akamai user conference

2010-10-16T11:44:00.000-04:00

Had a good time in Miami for a few days this week, got a lot of good content from the conference. I'm going to go over my notes some more next week, but there are some of the highlights from the show:

New offering
Fraud detection and scoring - Akamai does tokenization removing PCI scope, and they can build a profile on the end users and given them scores.

Cybersource - they are one of the processors of the akamai pci solution.

Edge encryption - Encrypts the data at the edge all the way to the database Only privledged systems can decrypt the data from the database. (might be useful, not sure)

Siteshield - ACL only allowing requests from specific akamai servers (protects against ddos)

ADS - Predictive analytics
Shows the proper ads based on what they are looking at across all sites
Look across 500+ shopping sites, on 160M users
Don't use pixels to allocate ads (slows down the site)

Akamai - Velocitude mobile reformatting on the fly - http://www.akamai.com/html/about/press/releases/2010/press_061010.html
Special tagging system which takes content and displays it for the proper device. Resized images and content done on the fly. Includes all data.

Dnssec
Verisign:
2010 - .NET
2011 - .COM

Akamai implement at end of 2010. Ga early 2011.
signed by Akamai. No need to manage it.
Need to look at godaddy. (they do some of our DNS along with Akamai)
Kpi. You make your own key and keep private key.
Look into internally. Microsoft dns support.

Ipv6
Need to start looking into routing ipv6. Check on firewalls (security support for ips and others) lb.
Geolocation for v6?
Look at ipv6 mtu issues
Idea Use Akamai edsgescape for geolocation versus what we do now.
Google support for ga and other tools? We use. Reporting infrastructure.

Q4.
Whitepaper
Roadmap

Q1 2011
Tech preview

Q1 2012
Limited availability

Akamai will nat to v4 for you.

HP Upgrades – QC,QTP,PC

2010-10-05T15:17:00.002-04:00

Before I go into these 3 suites of tools, when is HP finally going to update BAC, RUM, or Diagnostics? It seemed like these tools have been really stagnate the last 3 years. I’m not going to HP Software Universe anymore, and I keep getting new account reps, so I have no idea what the roadmap is these days. That’s enough HP bashing, onto the good stuff.

We did the yearly HP upgrade over the last couple weeks, here is the rundown on the technologies and what was involved in each:

1. QTP – Very easy upgrade, did this on our terminal server and a desktop. The license server is now supported on 64-bit machines, so we moved the license server off an older 2003 box onto a 2008R2 system. No issues with the upgrade, and there seems to be a lot of improvements. Still waiting for feedback from our QA team on the improvements.

2. PC – Ended up building a couple new VMs for this, as we moved it onto Windows 2008R2 (64-bit) as well. There were no issues with the reinstall or moving our scripts and data over to the new systems. The tool itself didn’t have a lot of changes, but the fact that it runs on 64-bit is a good step to us getting rid of our 2003 systems.

3. QC – This one is the problem child. Initially we were going to be doing a larger rollout of QC10, so we built 2 VMs. One was for the DB and one for the app. The DB needed to be on an older OS and such, which was annoying. I ended up reinstalling it onto a Windows 2008 (32-bit) system along with moving to SQL 2008. They don’t yet support SQL 2008 R2 or Windows 2008 R2 from looking at the documentation. It was not all that clear. I have a case open with HP, as during the setup it doesn’t seem to want to connect to the database. I have checked the SQL Server TCP settings, and verified the login/password both locally and over the network. More on this one as HP helps me with the issues.

Sorry for the lack of updates, I should have a few posts coming up now.

Password management

2010-09-01T09:28:00.001-04:00

Personally I am a big fan of proper password management procedures. For my personal data I always used the open source tool Keepass (http://keepass.info/) for my passwords, but it was always missing two items:

Better browser integration (I know you can use the form filling plugins, but they aren't very well done or supported)
Distribution (I know you can use dropbox or something else and it works fine)

For my encrypted data I always built hidden share volumes using another open source tool Truecrypt (http://www.truecrypt.org/). The product works great, but I find that I need to encrypt less and less of my data these days.

I just replaced them all with Lastpass (http://lastpass.com/), which is a very impressive product. It integrates with pretty much every major browser out there and it's all centralized and allows for web access for all of your data. It allows for import from pretty much any browser database, or product (such as keepass). Its $12 per year for premium, I buy products like this because the value is high and the cost is low. If we don't support companies like this then they are not around for us.

For our enterprise I have used several nice distributed products in the past, but one always stands out as a cheap and well built solution. The product we use is Password Manager Pro (http://www.manageengine.com/products/passwordmanagerpro/), we don't use the enterprise products from them which allow for centralized password reset and such. All systems, regardless of if they are Linux, or windows use active directory for authentication (thanks winbind).

The product is a great secure repository, and it allows us to share relevant passwords with finance, HR, Marketing, Development, or the Database teams. It allows for dynamic groupings which are very flexible based on the content of the resources defined.

OpenSolaris

2010-08-23T19:02:00.001-04:00

At my previous company we were a heavy user of Solaris, and we also had a lot of legacy specific SCO systems as well. 3-4 years ago, some person (who shall remain nameless) was pushing Opensolaris as "the future", personally I thought the guy was way off base. He did deploy some of it, and it worked well, the problem would be the support and future for another player in the x86/64 market. There was no future, I saw it, but apparently other people in management didn't. I then read this article:

http://www.infoworld.com/d/open-source/requiem-os-opensolaris-board-closes-shop-961?source=rss_infoworld_news

I love being right J

Sorry to see you go, but it's for the better. Oracle hopefully will invest more resources into Linux which it hasn't been doing as much since the Sun purchase.

Thoughts on the McAfee Intel purchase

2010-08-21T11:03:00.001-04:00

I've been waiting for quite a while for a major security firm to be purchased by one of the big boys. I am glad that Intel was the first one to start this trend, because they are generally only a hardware player. If security were embedded at that level it would create a differentiator from other competitors, weather they are x64 based or other chips (Oracle, IBM). Security has become very commoditized and consolidated over the last several years.

You haven't seen much innovation in several years either. Is that because we've solved the problem? I think not… Is that because there isn't capital in this market? Nope… I think the main reason is due to the massive consolidation and the work needed to integrate all of these smaller companies together. You are also seeing players like Microsoft developing a larger security portfolio, as well as network vendors integrating more security features and products into their appliances. If you look back 5 years ago, there wasn't much as far at UTM (Unified Threat Management) devices, now every firewall vendor has one, you can find hundreds of products both commercial and open source in this area.

Some other thoughts on how security is being embedded across the stack are by Bruce Schneier, who is a superb writer and author as well as a great cryptographer.

http://www.schneier.com/blog/archives/2010/08/intel_buys_mcaf.html

http://www.schneier.com/essay-196.html

http://www.schneier.com/news-060.html

ESXi 4.0 – 4.1 Planning

2010-08-05T11:53:00.001-04:00

Writing this thanks to gogo wireless… Love this service.

I'm going to start upgrading our hosts to 4.1 next weekend probably. Going to try update manager even though it crashed and burned on my 3.5-4.0 upgrade and I ended up using the host utility. I know the command line upgrader works.

I read this on the spiceworks message board:

Resolution:

This is worth knowing as it's a bug and definate gotcha.

VMWare said:

"I had one of the escalation engineers for Update Manager look at the log and here is what he said

The customer imported the pre-upgrade offline bundle which is NOT needed and in fact causes problems.

[2010-07-28 13:36:55:781 'DownloadOfflinePatchTask.DownloadOfflinePatchTask{9}' 3700 INFO] [vciTaskBase, 530] Task started...

[2010-07-28 13:36:55:781 'DownloadOfflinePatchTask.DownloadOfflinePatchTask{9}' 3700 INFO] [downloadOfflinePatchTask, 123] Upload offline bundle: C:\Windows\TEMP\vum8205261630712700578.pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip

We are working on correcting the situation so this doesn't happen. The existing pre-upgrade bundle will be replaced and we are working on a KB for those that have gotten into the situation (iKB 1024805). Unfortunately, there is no easy workaround once VUM is in this situation. A reinstall/DB reinit is suggested."

The end result is that VUM didn't want to reinstall cleanly so I had to nuke my Vcenter server and rebuild it from scratch.

So, DON'T APPLY THE PRE-UPGRADE PACKAGE IF YOU ARE USING UPDATE MANAGER TO UPGRADE YOUR ESX HOSTS!

Maybe useful next weekend J

Vmware 4.1

2010-07-26T18:05:00.001-04:00

Let me start off, by just saying that this site is great, its always entertaining and filled with great data : http://get-admin.com/blog/

I'm very happy that Vmware released 4.1 recently. After the horror stories I read about 4.0U1 we decided to skip them and wait for 4.1. There wasn't a lot in the updates we cared for anyways. There are quite a few interesting features in 4.1, and one of the good things that Vmware has done is finally killed off ESX (after this release). ESXi has been great for us over the last couple years, and I haven't had any complaints with switching over to it from ESX in previous years. I found using update manager was not the most reliable when moving from 3.x to 4.x on the physical systems themselves, so we opted to use the host upgrade tool. This is not a supported method to move from 4.x to 4.1. We will probably have to give the update manager another run, which concerns me. At least its not as complex as upgrading Hyper-v J

I will probably start upgrading our enterprise (not production) systems to 4.1 in the next couple weeks, and I will post my findings on the blog as I go.

Thoughts on IBM BigFix Purchase

2010-07-06T12:57:00.001-04:00

Bigfix makes some excellent products, and they have been moving in great directions over the last couple years. They have moved out of pure remediation and into configuration management and control. I would have loved to have purchased them for use at my current company, but the pricing was a bit higher than I'm using for Shavlik HFNetchk Protect, which is another good product, but is far more limited. I wanted to have one tool to patch Linux and Windows systems.

IBM has been really struggling to provide a good provisioning and patch management tool for years and years. First they were pushing TPM which is probably the worst product I have seen IBM release. Unfortunately a company I worked for previously was obsessed with using this product that most Tivoli enterprise customers get for free and completely disregard. I spent a good amount of time looking at the product and its capabilities, or lack thereof. I'm concluding my rant now, but its happy to see IBM adding a superb replacement for TPM and adding additional security related products they will acquire with the Bigfix purchase.

I was also quite surprised at the cost of the purchase at $400m. I know Bigfix has a lot of customers, and they sell a service, which makes it nice for both operating business as well as the customers who can bill this against opex versus capex. I would have assumed they would have had to pay more for the company. It will be interesting to see what features IBM takes from them and puts into Tivoli, and which other ones become part of the ISS portfolio over time.

Simplify and Automate

2010-07-05T10:31:00.001-04:00

Now that the workload has reduced a bit over the last month or so we can spend time doing project work. It's always been my philosophy to simplify as much as possible, this is normally because I end up having to fix messes, which are normally caused by undue complexity. Complexity can affect performance, availability, and manageability. Automation can often create complexity, as can requests by various people in the business who don't necessarily plan the projects or requests they make of others (especially development and operations).

This being said I often get blocked when I try to simplify things, because people want to build things out in a more redundant manner than is required for the business needs. There are a lot of ways to create a redundant system without creating complexity, you just have to step back and look at the overall configuration and requirements to come up with the best solution.

We get a lot of requests from our QA team to reload various Resin app servers, and other processes. What we are doing now is creating a web based interface for them to do the reloads on their own. This eliminates the need for operations to run the scripts, and saves time and resources.

JMS, Endeca

2010-05-22T21:21:00.001-04:00

We are building a services tier which will be based on ActiveMQ JMS, and our standard Resin webserver/app servers. We are building this with 2 nodes sharing each on the nodes. They will use a shared file system. We have a few services to start with, which are internal only at first. Should be interesting.

We are also working on some new products, and we are pretty close to selecting Endeca search for the indexing engine and SEO engine for it. More on that as we get along with development and implementation. The product looks pretty cool, so it should be fun.

Upgrade Land for Microsoft - Sharepoint / Exchange 2010, and JIRA 4.1

2010-04-30T15:28:00.001-04:00

I have been using Office 2010 for a while, and moved from the preview, to beta, and now I am finally on the release. We decided corporately to stick to 32 bit even when we are on 64 bit windows 7 on our newer systems. The main reason for staying on 32 bit was the all of the addons in the market are written for 32 bit only. When I was testing on 64 bit, I wished I had just stuck to 32. The released version has been stable for the last few days, but I didn't have much issues with the beta release either.

Now that we have office underway we are beginning upgrades to the other 2010 products we use from Microsoft. The first one was Sharepoint, which we are MOSS 2007 right now. The migration was slightly painful, and here are some of the pointers that I found helpful in the migration.

Run Powershell as admin
- http://manish-sharepoint.blogspot.com/2009/11/error-while-working-with-powershell.html
Batch Upgrade Visual Styles (the new look and feel for sharepoint)
- http://technet.microsoft.com/en-us/library/ff607998(office.14).aspx#section4

The next step is a bunch of testing, and hopefully cutting over next weekend to the new version (5/8/10). We avoided any custom components on our sharepoint, which made the migration much simpler. We have yet to have any complaints with the migrated test data. The new interface is awesome, and works great in Chrome as well. Great job to the Microsoft Team on this product!

We are in process of an exchange 2010 upgrade as well, we are building out some new VMs and we will migrate the mailboxes over. The work is still initial on that project, so I will post more on that as we go. My colleague is the main lead on that project.

On another side note, I moved us from JIRA 4.0 to JIRA 4.1. The upgrade was somewhat manual and required some work and planning. The new JIRA interface is very nice, and its good to see them finally changing the old reliable interface they have had for many years. Now if they would only fix the UI for the admin section so I could stop scrolling on a huge list that would be great!

Tridion Upgrade 2009 SP1

2010-04-30T15:13:00.001-04:00

We have decided after some pain to give Tridion another go over here. We have some really sharp guys helping us from the firm, and they have helped us immensely. We just upgraded to the newest version, and after the struggle to get it running initially it's gone very smoothly and simply. Within a couple hours we moved everything over to the new version and its working flawlessly. It was very simple and good to see the quality of the installers. They handled pretty much everything without any additional manual steps. We are looking forward to moving to the new version later this summer as we beta test for them.

Lots of the issues with the product were due to the implementation that was designed for us. We will be redoing our site and building it properly using the new version. I think with proper guidance and a good technical team we will not have the issues from the past. We are also moving a lot of our custom code from the current codebase into a web services layer that will isolate the our code from the main Tridion content. I am looking forward to the project.

There are lots of other things going on today, new database server swap for our performance testing, and a bunch of other project work. Its good that its quiet in the office as far as non-project work goes.

Week in Geneva

2010-04-19T09:07:00.001-04:00

Just wrapping up a week of pretty intense work here in our datacenter, here is a list of some of the fun projects we accomplished.

Disk upgrades to netapp
1. Netapp locally here in Switzerland went out of their way to fix issues caused by my purchase in the US. Last time I buy in US and ship overseas.
2. Netapp also looked over the system and made some very good corrections and suggestions, much appreciated the great customer support.
Reconfigured network
1. Move 10GE to other subnets
2. Change netapp network config
3. Several other additional cables and infrastructure was built out
Firewall Upgrades
F5 Upgrades from OSv9.4.3 to OSv10.1
Install 3 New VM Servers
Install memory in systems (DB, VM)
Cleanup of office, and build other infrastructure
Major failover testing of netapp, firewall, and loadbalancers

Now we are trying to get home with the volcanic ash situation in Europe. It looks like we will be driving our rental car to Barcelona, and taking a flight from there. Should be an interesting little side trip.

More fun later, glad to have a little break after working crazy hours the last week. J

Finally a way to block those pesky bots stealing content

2010-04-13T14:09:00.001-04:00

We've been using a product over at MFG which is sort of like an invisible captcha tool. The beauty of the product is the end user doesn't even know its running, but the accuracy and technology which is used is very unique and cutting edge. We first started speaking with Pramana – www.pramana.com about over a year ago, initially there was issues with the technology, but it had progressed quickly and become rock solid. I was unable to get false positives in all my testing and scripting.

We implemented the technology (Pramana HumanPresent - www.pramana.com/human-present/) based on issues with competitors which sell databases and information about manufacturing companies essentially stealing our content. They use various methods, including screen scraping, and seo scraping bots. This has been observed in many occasions, and we even had one company who wanted to sell out to us, while they were stealing our data! (somewhat legally)

The product is not super simple to implement, but the benefits are great. They have SDKs for a bunch of languages (for us we use Java, which is more complex than the PHP API or others they have). The SDKs give you all kinds of granular control.

We are a paying customer of Pramana, and they got the great idea of letting users use the service for free (Called BotAlert - http://www.pramana.com/botalert/) in order to detect and measure the bots (you get pretty daily reports from them), if you want to block the bots then you have to pay. The cost is very reasonable considering it doesn't inconvenience users, and it can also allow search engine crawlers to index content, but homebuilt screen scrapers to be blocked.

F5 Persistence and my 6 week battle with support

2010-03-25T09:36:00.001-04:00

We've been having issues with persistence on our F5's since we launched our new product. We have tried many different ways of trying to get our clients to stick on a server. Of course the first step was using a standard cookie persistence which the F5 was injecting. All of our products which use SSL is being terminated on the F5, which makes cookie work fine even for SSL traffic. After we started seeing clients going to many servers, we figured it would be safe to use a JSESSIONID cookie which is a standard Java application server cookie that is always unique per session. We implemented the following Irule (slightly modified in order to get more logging):

http://devcentral.f5.com/Default.aspx?tabid=53&view=topic&postid=1171255 (registration is free)

when HTTP_REQUEST {

# Check if there is a JSESSIONID cookie

if {[HTTP::cookie "JSESSIONID"] ne ""}{

# Persist off of the cookie value with a timeout of 2 hours (7200 seconds)

persist uie [string tolower [HTTP::cookie "JSESSIONID"]] 7200

# Log that we're using the cookie value for persistence and the persistence key if it exists.

log local0. "[IP::client_addr]:[TCP::client_port]: Request to [HTTP::uri] on server [LB::server] with cookie: [HTTP::cookie value JSESSIONID]"

} else {

# Parse the jsessionid from the path

set jsess [findstr [string tolower [HTTP::path]] "jsessionid=" 11]

# Use the jsessionid from the path for persisting with a timeout of 2 hours (7200 seconds)

if { $jsess != "" } {

persist uie $jsess 7200

# Log that we're using the path jessionid for persistence and the persistence key if it exists.

log local0. "[IP::client_addr]:[TCP::client_port]: Request to [HTTP::uri] on server [LB::server] used persistence record from path: [persist lookup uie $jsess]"

}

when HTTP_RESPONSE {

# Check if there is a jsessionid cookie in the response

if {[HTTP::cookie "JSESSIONID"] ne ""} {

# Persist off of the cookie value with a timeout of 2 hours (7200 seconds)

persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 7200

# Log Response

log local0. "[IP::client_addr]:[TCP::client_port]: Request to server [LB::server] with cookie: [HTTP::cookie value JSESSIONID]. Added persistence record from cookie: [persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]]]"

}

when LB_SELECTED {

log "From [IP::client_addr] to physical server [LB::server] the cookie JSESSIONID is [HTTP::cookie "JSESSIONID"] URI JESSIONID is [findstr [string tolower [HTTP::path]] "jsessionid=" 11] "

}

We've replicated and done 3 rounds of packet captures, and you can always see the issue in the logging from the irule above:

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63231: Request to /images/mfg/icons/search_cross.png on server -http-pool x.x.x.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63229: Request to /images/mfg/icons/icon_largemessages.png on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63228: Request to /images/mfg/icons/icon_clock.png on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63231: Request to /images/mfg/icons/icon_largequotes.png on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63233: Request to /images/mfg/icons/icon_largendas.png on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63230: Request to /images/mfg/icons/icon_largebluestar.png on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63232: Request to /mfg/scripts/search/search.js on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63235: Request to /favicon.ico on server -http-pool 0 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: 01220002:6: Rule JSESSION_iRule_withlogging <LB_SELECTED>: From CLIENTIP to physical server -http-pool BACKENDSUBNET.19 80 the cookie JSESSIONID is abcND0QYKjeOCczB8c_Ds URI JESSIONID is

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63236: Request to /servlet/mfg.Controller?time=1269130074065&pmId=1001&act=1154 on server -http-pool 0 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63224: Request to /mfg/contactHome.jsp?time=1269130079475&pmId=1154 on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:31 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /servlet/mfg.Controller?time=1269130079686&pmId=1154&act=supplierDisplayAgent&aid=904564&dgrdv=1 on server -http-pool 0 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:31 tmm tmm[1629]: 01220002:6: Rule JSESSION_iRule_withlogging <LB_SELECTED>: From CLIENTIP to physical server -http-pool BACKENDSUBNET.20 80 the cookie JSESSIONID is abcND0QYKjeOCczB8c_Ds URI JESSIONID is

Mar 21 01:15:31 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /mfg/contactHome.jsp?time=1269130145070&pmId=1016 on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:43 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /images/mfg/modalbox/close.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:44 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /servlet/mfg.Controller?time=1269130145287&pmId=1016&act=modal&mtId=800&mLoad=true&aid=904564 on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:44 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /images/bo/design/spinner.gif on server -http-pool 0 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:44 tmm tmm[1629]: 01220002:6: Rule JSESSION_iRule_withlogging <LB_SELECTED>: From CLIENTIP to physical server -http-pool BACKENDSUBNET.20 80 the cookie JSESSIONID is abcND0QYKjeOCczB8c_Ds URI JESSIONID is

Mar 21 01:15:44 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /images/mfg/icons/search_cross.png on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:44 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /images/mfg/icons/doubleDownArrow.png on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:45 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /images/mfg/combo/comboover.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:45 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /images/mfg/combo/combopress.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /servlet/mfg.ajaxProvider.GetDisciplineProvider;jsessionid=abcND0QYKjeOCczB8c_Ds?time=1269130158059&pmId=1016 on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /servlet/mfg.ajaxProvider.GetRfqBuyerLocationProvider?time=1269130158059&pmId=1016&sImg=false&sCwor=false on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /images/mfg/icons/dhtmlTree/iconUnCheckAll.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /servlet/mfg.ajaxProvider.GetMaterialProvider?time=1269130158060&pmId=1016&sImg=false on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /images/mfg/combo/combonormal.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /servlet/mfg.ajaxProvider.GetIndustryProvider?time=1269130158060&pmId=1016&ids= on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /servlet/mfg.ajaxProvider.GetLanguageProvider?time=1269130158060&pmId=1016&ids= on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:47 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /images/mfg/icons/dhtmlTree/folderOpen.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

The support team cannot figure out why this is, and it's been going on for very long. I will keep updating this as it goes on. The latest saga is that they are blaming it on the SNAT we are using.

http://devcentral.f5.com/Default.aspx?tabid=53&view=topic&postid=813179

http://devcentral.f5.com/Default.aspx?tabid=53&forumid=5&tpage=1&view=topic&postid=86374

F5 can be implemented as the inline gateway or outside of the gateway as a NAT device. We opted for the latter in order to avoid sending unnecessary traffic through the device. We may have to do some testing with it as the gateway and see if it fixes it. It clearly looks like a bug to me. We are running BIG-IP 9.4.5 Build 1049.10 Final, and we're planning on moving to V10 soon.

The Battle of the CMS

2010-03-10T17:33:00.003-05:00

My company was paying a lot for an expensive CMS system which wasn't working properly, so I had mentioned we should look at Joomlah and Drupal, as some of the popular systems out on the internet. Of course IT wasn't as involved as we should have been, and marketing is essentially forcing us to use Drupal. Then we starting giving some of our requirements and some of the lack of integrated core functionality is pretty disappointing for a proper CMS. I still have yet to see a full requirements list, but I do have a list of half a dozen or so on the operations side surrounding deployment, rollback, and environment management. I hope someone puts together a proper requirements list so we know where the technology will work well and where it will fail.

Moving to the Cloud and Packing up DR

2010-03-10T17:33:00.001-05:00

In other news on a side consulting gig I am doing we moved the company from exchange to Google apps. It's been a bit painful, but it will be more efficient in terms of cost and support. With DR being very important to the firm, this is a perfect fit, especially with the Postini archiving solutions. With such a small firm it made a lot of sense, and should prove to be a perfect fit. Also we are re-architecting the overall infrastructure from a dual location (DR) setup with clustering to a single location. In the process we are moving from Windows Server 2008 to Windows Server 2008 R2. I haven't done a lot of Hyper-v, but I have done a lot of VM work, Windows, and iSCSI. This should prove to be an interesting project both on technology and moving to cloud based resources, as well as the future direction of the company.

Expect more soon!

JMS queues and Spring problems

2010-02-10T17:32:00.001-05:00

We're looking at implementing JMS, which is something we sorely need in order to break our monolithic codebase into small portable non-interdependent (where it makes sense) modules. I leave the code and software architecture to the experts on the Development team, but of course my team has to deal with supporting whatever is designed and implemented, as well as monitoring and managing the services and associated technologies.

Of the products out there, we are leaning towards the open source queues due to cost and the fact that JSM has been around long enough and is a reliable and commonly used technology. One of the major ones we are looking at is ActiveMQ, but we're also looking at Sun, and other alternatives. Any suggestions would be awesome!

http://en.wikipedia.org/wiki/Java_Message_Service

Sorry for the lack of posts recently, been busy with work and doing all that not so fun internal administrative stuff. I hope to be posting more regularly now that we have a lot of new interesting projects going. Had a great time in New Orleans last weekend, nice job Saints!

Open source config management and AV

2010-02-10T17:29:00.001-05:00

I have a new engineer on my team this week, and a couple of the first projects he's working on are open source config mgmt. (for basic config files) as well as open source AV.

On the config management front, we're leaning towards Puppet. I have a good friend who uses cfengine on a big server farm, and he loves it. From what I've read Puppet seems to be a newer more modern version, and we don't have a huge farm to manage so I think it would work perfectly for us. Looking forward to learning and implementing it!

http://en.wikipedia.org/wiki/Comparison_of_open_source_configuration_management_software

With AV we are runnin some tests with Clamwin, and we'll see how well it can pickup incoming viruses uploaded via our Resin application servers. We shall see.

Next project will be bringing a CentOS yum repo locally and patching over the LAN versus the internet, another project I've wanted to get done, but haven't yet completed.

Cloud usage

2010-01-14T11:39:00.001-05:00

Could, cloud cloud, the cloud is HERE! I love articles like this http://alan.blog-city.com/has_amazon_ec2_become_over_subscribed.htm# Really interesting read, and makes a lot of sense. The problem is that you don't have visibility into what they are running and how well its running. Running my business on a completely unknown infrastructure would be a no no. I understand the internet is a big shared infrastructure, but wouldn't you rather know the specifics about the performance and capacity of your infrastructure.

This is an interesting thing to think about before you move apps all over the cloud, and have to manage a mess. I do like some of the monitoring which is being done by products like Zenoss : http://mediasrc.zenoss.com/documents/Using+Zenoss+to+Manage+Cloud.pdf . I like Zenoss a lot, but haven't done a full deployment of it yet. I might have the opportunity to do that soon!

Storage upgrades - Netapp

2010-01-06T16:44:00.001-05:00

We're doing some storage upgrades, basically I need to build out a load testing environment, and since our application is pretty much database trapped I need to invest a bunch of cash in more big iron and more spindles. I need quad – 6 core db servers, which is what we have in production right now. I also want to get 24 spindles of disk, which is what we have in production as well for the database. The disks will mostly be used for the loadtesting environment, but they will also be used for some of the other projects.

It's too bad our netapp FAS 3040s have all the slots full (10g, PAM, then the rest FC cards). I need to buy FC disks and DS14 shelves versus buying the newer SAS disks which I would rather have, pretty annoying. Anyways I am buying 28 disks, and its costing me around $54k for the disks. So if you add up the disks that's 8.4TB raw (but the spindle count is what really matters), which means I am at about $6.30/G of data, which is pretty poor. That doesn't even include the support costs for the new hardware. Retail price of these disks is $280, lets tack on another 25% so they are $350 per disk, or around $1.10/G. The cost of the shelfs is costing me roughly 5x the price for the same disks. Can you see why I am pissed? I can understand 100% markup, but 500%... really? Who says these storage companies aren't like the lawyers J

Back from vacation and diving in head first!

2009-12-15T16:13:00.001-05:00

We had an awesome cruise in the Caribbean with my wife for the last 10 days. I am feeling very good and back in the swing of things. I missed my great colleagues, house, and cats.

1^st day back and we ended up seeing some odd packet loss around 6pm in our office. We ended up finding a couple of bad switches, and essentially rewired the whole office. It was a good time. We also put an order in for a couple of better Dell switches that support STP, management, and Spanning. These features are something we really wanted, but we had these crappy web managed Dell switches which cannot do much of anything. It was a fun 16 hour day yesterday, and thank you to Jamie for working so late with me to fix the problem. We still have 1 issue left to fix with the router -> firewall connection, but it should be done tonight late or tomorrow morning.

We have a software push tonight to production systems, should go smoothly, but we keep risking not letting the code settle long enough before we push it.

Also our VM environment is ever growing; supporting our enterprise product is really becoming a major drag on the infrastructure. Still have a lot of cleanup to do on the legacy environments, but It's mostly being pushed back as far as the dates go. We are starting to be too risky on this side of things for my comfort.

Been seeing major Database growth as well, we are doing a online volume expansion using Snapdrive and the iSCSI LUNs we host on the Netapps. Done this before without issue, but there is always some risk involved.

Hope all the readers are doing well, and I will update soon with what's been going on

Phone system

2009-11-25T12:06:00.001-05:00

We elected to trial one office upgrade with a new phone system, on much different technology than myself or my colleagues are used to. We decided to go with a Fonality system. The support seems great, the product is very advanced, and the price is VERY good. We are doing our first SIP turnup and phone system install in the Paris office this weekend. I hope it goes well. If this works out we'll be rolling out Fonality across the company over time. Munich is next on the list.

It's essentially a modified asterix system. Fonality also manages the development of Trixbox, which is the largest distribution of Asterix on the market. Looking forward to seeing it in action!

Really amazing features and I like the fact that we can use almost any supported phone provider. (Cisco, Polycom, Grandstream, Aastra)

Fonality PBXtra - http://pbxtra.fonality.com/

Office 2010

2009-11-25T12:01:00.001-05:00

I have been using Office 2010 for about 6 months now. It was a bit rough the last 4 months or so. I was also very sad that I didn't have my favorite addins such as plaxo, xobni, and others. I have been running them on a desktop I can remote into if I need them or need to sync my outlook data via these extensions. I got the new build installed after having to mess with the uninstall process. It was really annoying but it worked out. The new beta build is very nice. It's more stable, and there are several items added which didn't exist previously. You can also see in outlook how they are starting to integrate social media into email, something xobni has been doing very well for quite a while. The Microsoft social media integration isn't complete or even really working, while the xobni version is very advanced and works great.

I wish that Microsoft built some kind of wrapper to allow the previous addins to work with outlook, but I guess that's not going to happen.

Xobni - www.xobni.com

Plaxo - www.plaxo.com

Windows 7 UAC articles

2009-11-05T19:00:00.001-05:00

This is really upsetting me. I keep seeing this as I read my news tonight:

http://www.betanews.com/article/Sophos-study-suggests-Windows-7-UACs-default-setting-is-selfdefeating/1257455306

I was one of the only ones who seemed to think Vista was a good user interface and OS upgrade from XP, of course it could have been more optimized and even better, which is what windows 7 is. I also found the UAC feature in Vista to be very good, and similar to those of us who use unix are used to working. You su to root when you need to do something elevated, otherwise you operate at user level. The typical end user complained "it keeps asking me to elevate so often, I don't understand what this means". On windows 7, Microsoft decided to elevate only under certain cases (by default), and of course the inconvenience of the extra click, otherwise known as security, was removed essentially. This makes Windows 7 in its default setting much less secure than vista.

Being a systems and infrastructure guy, we get the same Vista feature in Windows Server 2008 (based on Vista), and R2 (based on 7). They kept the same escalation we had in Vista enabled out of the box on both platforms. This is especially good for a server OS. I have been seeing some of the admins (not in my group, but DBAs) disable this feature, and I always implore them to turn it back on. I explain the reason it's there, and it will save them, either from doing something by accident, or by something running in their session they aren't aware of.

Then you get other poorly designed software such as HP's Quicktest Professional which still cannot run with any level of UAC enabled. It takes 4 years to make your application work with UAC? Really?

So basically, user feedback promoted Microsoft to reduce the nags (otherwise known as security), and then the press and AV vendors are touting Windows is less secure? Seems like a catch 22 for Microsoft, they want to sell operating systems, but they also need to placate people like me who would like a secure OS. I understand they are shipping the servers hardened, and the clients less so, but is that a good idea? I think my mom will thank them J

We did it

2009-11-05T12:09:00.001-05:00

We finally launched the new platform, its been pretty difficult both pre and post launch supporting the business, developers, QA folks, doing perf testing. Lots of stuff to fix, and I'm really looking forward to the cleanup part. Its always fun to recover space and processing power which is not needed on the new platform. Things are progressing well.

http://www.techcrunch.com/2009/11/02/mfg-com-takes-off-the-cuffs-with-manufacturing-marketplace-redesign/

We have to move a couple offices in the near future, and we're trying to open one in India. All of that planning and work is keeping us busy as well.

I am very happy to have a new global helpdesk manager onboard. Great addition to our team!

Ipmonitor, spiceworks, and vendor maintenance

2009-10-21T15:31:00.001-04:00

I'm happy that solarwinds has released a major upgrade to ipmonitor. Too bad they didn't notify me my maintenance expired about a month ago. I'm renewing it now, and looking forward to v10. This product is excellent, cheap, and does a great job with agentless monitoring. You can also tweak it to monitor pretty much anything as needed. Such a good deal for a great all around product.

Speaking of maintenance, I also just investigated and found out that our F5 Big-IP maintenance expired in April. Glad we have a HA pair in case of issues in the next little while, but I don't understand why vendors and resellers don't keep on top of customers. It's essentially free money they aren't going to get if they don't chase folks about it.

Vendors who do a good job with maintenance:

IBM
Oracle
Sybase
HP Software
Cisco

Vendors who are horrible with maintenance in my experience:

HP hardware
Dell
F5

What cisco does that's really cool is they integrate maintenance into other tools so when you have inventory such as HP Network Automation System (Formerly opsware nas) as well as Cisco's own Ciscoworks.

Another thing we do because we are cheap and love good free software is we leverage this awesome product called Spiceworks. I can't believe what you can do with the product, we've been using it for over a year now, and it's completely free:

System inventory
Hardware details and changes
Software installs and changes
Up/down monitoring
Event log monitoring
Disk monitoring (only used as a blanket monitor, non production)
Exchange monitoring
Antivirus definitions monitoring
Active Directory integration
Interface graphing (firewalls, routers, switches)
Network mapping (relationships of devices and switches)

Its very easy to find a users system based on the login, its very easy to see changes in software and hardware, its also easy using this script:

http://community.spiceworks.com/how_to/show/197

This awesome script populates the inventory with the warranty expiration on dell devices. (including servers, printers, switches, desktops, and laptops)

Right now we have 5 collectors feeding one instance, so I can do global scanning and aggregate the results in a single repository.

Netapp SATA perf

2009-09-16T17:07:00.001-04:00

I decided that when we built out the netapp gear that we would put low volume and QA data on the SATA disks and save the FC disk for databases, VMware, and other intensive stuff. Now looking at performance on the QA VMware boxes with the SATA disks and I'm thinking I shouldn't have done that. It's been quite good in general, but when there is a backup running or other disk intensive actions occur it grinds to a halt. I really need to figure out a way to move onto a FC aggregate at some point.

Vmware issues resolution

2009-09-16T16:40:00.001-04:00

I had support on the line for a while to fix my errors with the reports. They finally fixed it, it was some obscure bug which is fixed the the next major patch for Vsphere 4. It was a complex fix they had to do, but it works finally. Nice job by support.

Sonicwall Sonicpoints

2009-08-28T16:31:00.001-04:00

These things are still having issues. We got them stable for 2 weeks, but now one of them is on the fritz again. The N access points seem to be more problems than the G ones. I have another service case open with Sonicwall, their support is pretty unresponsive in general. Annoying.

Too bad, because the product is great!

Vsphere server issues and upgrade progress

2009-08-28T16:29:00.001-04:00

So I found out that using the host update tool versus Vcenter update manager is much easier and more reliable when moving from ESXi 3.5 to 4.0. Before I was using the update manager and it wasn't working all that reliably. So far I haven't had any issues using the host update tool. I've done many upgrades now, and I only have 4 left, 3 of which I am doing this weekend.

Whenever I speak to vmware they always think I'm using ESX, when I prefer and expect that people should move to the more appliance model of ESXi. With 4.0 they are pretty much on par, and I'm going to stick with ESXi.

On one of my vsphere 4.0 servers (virtualcenter) its doing this annoying thing when I try to use the performance overview:

Perf Charts service experienced and internal error.

Message:

Report application initialization is not completed successfully. Retry in 60 seconds.

In my stats.log I see this.

[28 Aug 09, 22:28:07] [ERROR] com.vmware.vim.stats.webui.startup.StatsApplicationLauncher : Task execution produced an error. Re-initialization attempt #26 will startup after 60 seconds...

java.util.concurrent.ExecutionException: java.lang.IllegalStateException: com.vmware.vim.stats.webui.StatsReportException: Unable to open VC DataSource.

at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:205)

at java.util.concurrent.FutureTask.get(FutureTask.java:80)

at com.vmware.vim.stats.webui.startup.StatsApplicationLauncher$1.run(Unknown Source)

at java.lang.Thread.run(Thread.java:595)

Caused by: java.lang.IllegalStateException: com.vmware.vim.stats.webui.StatsReportException: Unable to open VC DataSource.

at com.vmware.vim.stats.webui.startup.StatsApplicationLauncher$1$1.run(Unknown Source)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)

at java.util.concurrent.FutureTask.run(FutureTask.java:123)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

... 1 more

Caused by: com.vmware.vim.stats.webui.StatsReportException: Unable to open VC DataSource.

at com.vmware.vim.stats.webui.startup.VcDataSourceInitializer.init(Unknown Source)

at com.vmware.vim.stats.webui.startup.StatsReportInitializer.createInitializers(Unknown Source)

at com.vmware.vim.stats.webui.startup.StatsReportInitializer.init(Unknown Source)

... 9 more

Caused by: org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (The connection to the named instance has failed. Error: java.net.SocketTimeoutException: Receive timed out.)

at org.apache.tomcat.dbcp.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1225)

at org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)

at com.vmware.vim.stats.webui.startup.VcDataSourceInitializer.openVcDataSource(Unknown Source)

... 12 more

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The connection to the named instance has failed. Error: java.net.SocketTimeoutException: Receive timed out.

at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown Source)

at com.microsoft.sqlserver.jdbc.SQLServerConnection.getInstancePort(Unknown Source)

at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(Unknown Source)

at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(Unknown Source)

at org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)

at org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:294)

at org.apache.tomcat.dbcp.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1247)

at org.apache.tomcat.dbcp.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1221)

... 14 more

I have an open case with vmware right now. See if I can get it fixed.

Ahh my favorite

2009-08-14T08:22:00.000-04:00

So we pay a LOT for support on some software, you are talking about 30-50k. I just got this email from one of my vendors. They happen to be one I am not fond of which I have commented on before:

Only 2 months and 10 days for a ticket reply for a new license.

Date of email reply - 8/20/2009 3:37am

Dear Jonah

Please find attached the license requested

Kind regards,

From: support.emea@somecompany.com [mailto:support.emea@somecompany.com]
Sent: Wednesday, June 10, 2009 03:22
To: Support EMEA
Subject: INC000000007516 License-request

Uverse hell

2009-07-28T15:38:00.001-04:00

Day one - Thursday

I got my uverse installed, install went okay, took about 5 hours to get it running. Once the tech left I switched from my Clear Wimax connection to the uverse. Looked good, then I got on the PTPP connection at work, and it started dropping me every 5-9 minutes. I called them and worked with support for 2 hours and tried lots of settings on the router. These included lowering the MTU and upping the timeout. Nothing fixed it at all, they were keeping my case open in case I wanted to call back. Then I was playing with it later on and lowered the timeout and it seemed to fix the issue. The router has a very small connection table which is a known issue, I think with the 1 day timeout there were lots of open connections which weren't being cleaned up.

Day two - Saturday

Turned on the TV to start setting up some of the DVR recordings, and the DVR wasn't working. I called ATT and the box was dead, we tried to do a factory reset and OS reload several times and the box was toast. They promptly sent out a tech a few hours later. He tried with 4 different boxes and was unable to get it working. He was there for about 3 hours, and he was out of ideas. It was 8:30PM on Saturday. He also reset my router since the TV connects to the router and uses the line that is shared.

Day Three – Sunday

I reset up the router with the same settings I had before, but I started once again to get PTPP issues and drops. I messed with it for a while and gave up again.

Day Four – Monday

I turned on the DVR, and what do you know, it started working perfectly, makes no sense to me. I called support once again on my VPN issues, I gave them my case number and they told me they don't work on VPN issues and regardless of my case and the tech I worked with I should go away. I spoke to the shift supervisor and she told me the same thing, but was nicer about it. I told them I would disconnect service if I wasn't able to use a simple VPN. They were happy to lose me as a customer, and transferred me to the disconnection line. When I got someone there, I asked them if I could have TV only, and they said I need to have internet as well. I told them I would work on it some more, and let them know if I wanted to disconnect.

We happen to have firewalls which give us SSL VPN as a free option, so one of my co-workers set it up, and it worked perfectly. No drops, and no issues. At least I have a fix for now, so I can work at home if needed.

I called customer service and got a credit for my outage time and a discounted rate for the next 6 months. All in all I've wasted about 20 hours on this mess, but glad its working now. The customer support has been good, aside from the lady who refused to help me.

Service impressions

The base 6down/512up is decent, but no better than the clear wimax service. The TV seems very good, and the DVR is better than others I have seen, but I still miss my Tivo HD XL. Nothing beats Tivo!

I wish I had my FIOS like I had back in Boston L

Move - clear wimax - uverse

2009-07-22T23:36:00.000-04:00

I haven't been posting becuase I just moved, and I'm also getting married in the next few weeks, but enough of the personal stuff. Lets get into some good tech talk :)

I have a verzion wireless card in my laptop which comes in handy when I don't have other ways to access the internet. Its a good connection, and usable for working as needed. The main issue is there is some lag/latency in the connection.

We have a clear Wimax device at work which is our backup internet connection if we lose the fiber link. I decided to borrow this connection during my move to deal with my lack of internet. All I have to say is that the connection quality and speed is amazing. The lag is almost non-existant, and the throughput is superb. I can even download a lot on it without an issue.

In my new location I can get ATT uverse, which is similar to the Verizon FIOS connection I had back in Boston. I loved FIOS, and was very sad when I had to move down to Atlanta and get on Comcast. I had a stuggle getting the FIOS installed, but eventually it worked and was rock solid for the duration of my service. It was also about $35 per month less than comcast.

Uverse costs as much as comcast, which is fine if the quality is there. I've had a small struggle so far getting uverse installed, but tomorrow is the big day when my place will be fully provisioned hopefully. I will post more on the quality of the connection and the TV capability compared to Comcast, Verizon (FIOS/Wireless), and Clear.

Another major difference is that on FIOS they ran standard cable from the Fiber demarcation point to the TV, so I used my Tivo HD XL box with cablecards, which was perfect! On Uverse they use a full IPTV device, which means my TIVO isn't going to work, but the picture quality and features should be better. The Uverse box also supports recording 4 channels at once, versus my TIvo's capability to record 2 channels at once. http://www.att.com/Common/totalhomedvr/

We'll see if it works well, more on that this weekend.

So I should be posting more, telling you all how it is to deal with vendors and technology we use in every day work and home life.

Leave comments, I like a good dialog!

Vmware ESXi upgrades from 3.0 to 4.0

2009-07-22T23:26:00.000-04:00

I've been trying to move hosts over to 4.0 from 3.0, but i've run into countless issues on each migration. I'm trying to get a good method do this consistently, but its just not really possible from what I've been seeing.

When I do speak to support, they keep confusing esxi with esx, and ask me to do things that cannot be done on esxi. I've also been studying on if I should run esx or esxi. For the way we deploy and the lightweight nature of esxi it makes more sense, and I believe its the proper model. They still need to develop a proper CLI/SDK or some way to get into the boxes better in a controlled manner without using the "unsupported" login method that support always asks you to do. That method requires console access which we don't have on lots of our hosts.

So far we have 2 hosts upgraded, and quite a few to do somehow or other. I'll keep plugging away.

Vmware support is very good, and responsive, good company to deal with, and great technology. They just haven't figured out the upgrades.

Microsoft print spooler i hate you

2009-07-22T23:21:00.000-04:00

I've been debugging crashes on the print spooler on a Windows 2008 x64 domain controller which is a print server. With all of the advances Microsoft has put into Vista and Windows 7 on the client side to help debug and diagnose issues such as the automated collection technology inside Performance and Reliability monitor you'd figure that they could figure out a way to isolate a fault down to a vendor driver. Thats not the case, I've had to jump through hoops for about 8 weeks to figure out what the causes of the crashes are. I think its partially related to running x64, and x64 drivers, but having mostly x86 clients printing. Its kind of a mess, but its been improving with patches and the countless dumps and other files I've been sending them.

Issues with Tridion

2009-07-22T23:15:00.002-04:00

Updated on 7/24:

We use the tridion CMS, which is a high end CMS product, we have had a lot of trouble with the product in the past. (www.tridion.com) The content manager is a strange beast which uses a combination of vb, .net, and other technologies. Its always breaking, and is not reliable. We need to debug messed up stuff on it on a regular basis. We have the systems under change control, but it still seems to manage to break easily. The good news is that it spits out generated jsp pages, which seem very reliable running on Resin application servers.

The support is always excellent and responsive, which helps us deal with code issues that we have with development. In the last case they actually went out of their way to get our content database and replicate the issue pointing to our code. This is something that few vendors would do for a small customer as we are. It still doesn't make up for the strange design of the system. I am not sure if this is related to our implementation or a problem with the product itself.

Here is a snippet from my emails with support:

It is not supported to run other versions of .Net on the Content Management server - only the version listed in section 2.2.d of the "SDL Tridion R5 Product Prerequisites 5.3.pdf" (Microsoft .NET Framework version 2.0) document are supported. Please uninstall all versions of .Net Framework, reboot, and install the supported version.

Last time I checked Microsoft made .NET fully backwards compatible, and even keeps the frameworks in different folders:

For example on my machine:

Directory of C:\Windows\Microsoft.NET\Framework

04/22/2009 02:17 AM v1.0.3705
04/22/2009 02:17 AM v1.1.4322
06/29/2009 03:14 PM v2.0.50727
04/22/2009 05:01 AM v3.0
05/18/2009 02:58 PM v3.5
06/29/2009 11:08 AM VJSharp

Since my last post was a bit too harsh, and I didn't credit where I meant to credit I now have to meet with the Tridion folks on Monday... what did I get myself and my poor colleagues into. Oops...

Sorry for lack of updates, lots of stuff

2009-07-01T16:43:00.000-04:00

Have been in Geneva for the last 3 weeks now. Been busy over here getting things in order and making sure we are ready for a major product launch over here. I am also working on some staffing and support issues for here in Europe.

Our CEO was just on CNBC, and he did an awesome job, yay Mitch!
http://www.cnbc.com/id/15840232?video=1169688483&play=1

Upgraded our Vcenter servers up to Vsphere 4. The upgrade of the servers themselves went well, but I tried to do a hypervisor upgrade, which failed. I have a case open with Vmware on the upgrade. I'm also have some strange connectivity issue on one of my physical systems. The new product is very cool, the new alarming system is a much needed improvement. Overall very happy with it so far. I am looking forward to testing the performance with the new hypervisor.

Some cool stuff my team has done. Integrated all of our linux boxes with winbind to AD. Its much easier to admin them and control access now, very good stuff. We can give developers granular access to QA and Dev systems now.

We are also building a syslog infrastructure which is similar to the one I built in my previous company. Essentially its a syslog-ng frontend to splunk, where we can forward and log streams as needed. The cool thing is that my sharp engineers at work are figuring it out on their own and learning new stuff in the process. This is the kind of thing that makes doing infrastructure and systems cool.

Working with Microsoft on a spooler issue thats been annoying, printers are so fussy.

We closed a deal with HP to get a bunch of new software. My team will be working on implementing some new technologies to add onto our existing HP investments. We currently use HP Performance Center, HP Quality Center, and HP Quicktest Pro. We added more users into Performance Center. We also puchased HP Real User Monitor (RUM, which requires BAC... I thought I got away from BAC at my last gig). We also will be implementing HP Diagnostics for Java, which should be a very useful tool for us leading up to our new product release.

That should do it for now :)

In Geneva

2009-06-16T08:52:00.000-04:00

Just trying to wrap up a few things, and get some more help on my team here. With the language barrier I face in Geneva, meeting and spending time face to face really helps a lot. I'm headed up to France for my brothers wedding for a few days, then a week back in the office and headed home. I will post more info on my trip and some details of the projects and such I have going on when I am back (or flying across the ocean and not sleeping). Been working long days, and long nights. I have been here almost a week now, but it seems like 3 days. I miss home :)

Production Java Profilers

2009-05-27T17:17:00.003-04:00

I have been looking at java profiling tools. Mostly focusing on the following vendors:

Precise – Good on java, Superb on the database, no RUM capability
1. Would make a good tool for us, to help upgrade our current toolset of Idera SQLdm (which is a great product as well, but more limited).
Dynatrace – Great on java, no DB, Superb RUM capability
1. Coradiant brought me into this tool, which I had discounted due to a previous bad experience (about 2 years ago) when the product was very new into the .NET space. Product has a lot of similarities to the way that the Identify Appsight works, where you export a file and the developers open the file and do the deeper analysis.
2. I also see they just announced the partnership today here : http://www.prweb.com/releases/Performance_Management/solution/prweb2464644.htm
Wily – Best on java, no DB, not sure on RUM
1. Put together a great deal for us, but I'm not sure how well the RUM products work, since they are new to the game.
2. Not sure I want to give up my Coradiant…
HP Diagnostics – Good on java, no DB, no RUM
1. Best pricing, since we already do business with them, but I didn't see a lot of advancements in the tool since HP purchased them.

So that's the current project, along with a new helpdesk/bugtracking system (JIRA) we are building. We are also doing some additional work on our SEO. That sums it up other than that just the standard engineering work, support, and smaller projects.

Cool post from Alistair

2009-05-27T17:17:00.001-04:00

Big fan of what Alistair has been up to, and he has a new site here: http://www.rednod.com/ I really loved this post, and the overview of the process and software needed to build software : http://www.rednod.com/index.php/2008/12/07/testing-and-launching-a-web-app-what-every-startup-needs-to-know/

Lots of what big companies do in this space are overly complex, or missing key parts to the process. Using these types of tools will save time and create better software.

Been a good week – Windows 7 and Engineering Update

2009-05-13T16:35:00.001-04:00

Sorry I haven't been posting as much as usual. Last weekend, I had some free time, and ended up re-installing one of my desktops at home with Windows 7. I liked the OS so much I ended up doing the same on my laptop. The OS is very responsive, and although I do like Vista, I agree that Windows 7 is a major improvement in many areas. The new taskbar took some getting used to, but the search and overall responsiveness are welcomed.

In more fun server news, I have been working on lots of different issues/engineering tasks:

Tape backup drive is hosed, and I've been eating lots of time with Dell on this issue, I wish they would just swap it out. Annoying, but the service is much improved from what it was before when I was dealing with them previously.
Redoing a bunch of stuff on the netapp to get ready for the new product release, its all coming together now J
1. Having some odd issues with multi-protocol permissions and such, also having an issue with some of our NFS exports not showing up in the web gui (Filerview) but showing up on the CLI. Strange one.
Put up a MSSQL cluster, but had a strange issue. Debugged it with Microsoft for a few days.
1. Issue creating failover cluster - Getting Event ID: 1570.
2. Tried to turn on the firewall as per microsoft's recommendation. I don't see why this matters, but we did it anyways. I had to make some custom ports for UDP 3343 and TCP 3343.
3. Ended up having a user in AD with the same name as the computer account. This was giving us the access denied error. Renamed the user, and the cluster was fine.
4. Was on the phone and email for too long with the engineer, but happy it got resolved.
Getting a new conference calling solution for cheaper calls than what we pay now with BT. We use the conference calling internally since we have offices in Europe and Asia. We prefer and try to use Skype as much as possible.

Have to head back to Geneva next month for 3-4 weeks, which should be a good useful trip! Looking forward to working with my company over there, my main engineer has been there for a week, and will be there for another 2 weeks. He has been really fixing things up in the office, which are all much needed improvements.

Last item… Go Celtics! Go Bruins J

Been a bit crazy

2009-04-20T22:02:00.000-04:00

I have been travelling quite a bit for work, I spent some productive time in Geneva, and Munich. I have been gone for about 3 weeks total. I had a nice weekend in Amsterdam as well.

The weeks consisted of a datacenter move, some engineering and upgrades, and fixing some issues in our other offices as well. Overall the trip was good, but a bit too long of course. Made some good progress communicating with our core development team, and nailing down a lot of things on the table. I'm happy to be back, and looking forward to the big push coming to a new product launch. There is always a big project on the horizon that we work towards.

I've also been trying to beef up and implement for centralized storage in order to properly utilize our ESXi implementations. We have a homegrown NAS for NFS in one office, an older AX150i in another office, and I'm looking to get some more storage in the other office. I wish we could afford to put netapps everywhere and replicate... maybe next year :)

Speaking of Netapps, it seems that the upgrade about 11 days ago from 7.3.1 to 7.3.1P3 has finally fixed the weekly crashing issues we have been having for the last 4 months. It took a long time to get it fixed, but glad it seems to be fixed.

I have one last sonicwall I am trying to get upgraded with the sonicwall trade-in program, then i'll be fully single firewall platform finally. Should make things easier to manage and to train the staff on. They are also great products and nice to work with. UTM is the way to go.

Anyone have any comments on good replacement tools for acrobat pro (editing PDFs). Please spare me the mac user comments.

In Geneva

2009-04-06T17:09:00.001-04:00

Finished some moving around of servers and one of our products this past weekend. Also did some cleanup so we can vacate some older cabinet space. It runs much faster on the new environment, and we are doing lots to optimize it even more. We have plans to VM a lot of it, which should help with management and scalability. Everything went really well over here, we had some issues but not a ton. Learned a lot about the other environments and how we can streamline a lot of it.

I’m going to spend the rest of the week meeting with various people, and spending time with my engineer over here. He’s pretty excited about a lot of what’s going on, and that’s great. Its always nice to teach eager people.

I’m headed to Munich office next week to fix some ongoing connectivity issues, we have a DSL line which is over ISDN (Annex B). Its been very problematic, and we’ve had connectivity issues. I’m also installing a new firewall to get them onto the corporate standards, as well as deploying a new access point.

Been working non-stop since I arrived last Thursday morning, I hope the rest of this week isn’t as long days. I think I worked 28 hours last weekend alone.

Cool Skype Addons

2009-03-25T17:54:00.002-04:00

We use a lot of Skype at work, and it’s just nice and easy especially with the Skype Polycom speakerphones (http://www.amazon.com/gp/product/B000GG0EFY), I love them.

I was searching for two nice apps for use with skype. We really like Zoho meeting (free 1 on 1), but I wanted something with better features similar to the glance application we use in sales over here. I found this app Oneeko (http://www.oneeko.com) which is really slick, and has excellent skype integration. It’s by far the coolest skype/screen sharing type app I have used.

I also wanted to record my skype conversations to mp3 versus having to pay $20-$30 to buy an app, I found a pretty decent app called Callgraph (http://callgraph.biz) which works very well and puts all my conversations and meetings into mp3.
Now I am all set with my new tools :)

Released new CMS, Wireless

2009-03-15T16:01:00.002-04:00

A couple weeks late, but we finally launched the new CMS for our corporate site. Its good marketing is in control of the site now, but the CMS has some major issues. I’m not going to get into the details, but it doesn’t seem like the right technology for most people who want to implement a CMS. We changed our main domain, and further split the marketing site from the application, which makes my team’s lives much easier in operations. Glad to have Coradiant in place, its very handy for these kinds of moves. Also happy that we put in Omniture with this new CMS release. I would like to test Coradiant Edge and the Omniture integration at some point.

We have been putting in sonicpoints to replace the hackjob dd-wrt I tried to put into place. I really like that product, and its very nice for my home use, but its got some issues with the N radios I guess, not too sure. Anyways the sonicpoints allow us to control them from our NSA appliances from Sonicwall. Makes security and implementation easier and faster. They are also very cheap, so that’s another positive. Trying to get them rolled out fully this week. Bought some for our Geneva office too, I’m headed there in a couple weeks.

I also implemented some new Netapp monitoring and such which is also pretty cool.

I’m headed to London on Tuesday night for some vacation and to spend time with my brother and his fiancé. Will be back a week from tomorrow (Monday), so you probably will not get another post for at least 10 days.
Have a good week.

Updates – Netapp, HP, Coradiant

2009-03-05T10:14:00.002-05:00

Our netapp (one of the nodes) is still crashing even after upgrading from ONTAP 7.3 to 7.3.1 . So far there is no advice from netapp on the issues. Obviously I am not happy since this has been going on for 7 weeks now. I just had a meeting with my Netapp rep, and he introduced and assured me the support would be better. I hope this is the case as its been pretty bad so far. Speaking of bad support the HP support is pretty much the worst support of any company on the market. I’ve had so many support issues just trying to get licenses and upgrades for all the HP (Mercury) software that we own for QA. Its been very painful. I’m trying to switch off HP support (aside from software upgrades) but we shall see.

Finally got the Coradiant box hooked up in Geneva, and I just did an upgrade this morning to the newest OS. Its working well. I have no idea what’s in the release notes, but I will soon find out 

Lots of updates, projects, running and ducking

2009-02-20T20:33:00.000-05:00

Sorry for the lack of posting, I have been travelling and working a lot trying to get a bunch of projects out the door. We have about 6 weeks left of craziness until there is a break before some other larger projects. At least most of the infrastructure and underpinnings will be done for that which should make things less crazy.

We are close to moving our marketing site over to Tridion (beta launch today, and release next Friday), so marketing can deal with the content independently of the code and products. Tridion is a crappy product to deal with, it breaks a lot, and its hard to keep the publishing system working. The upside is that it's very flexible. As far as CMS's go its very overpriced, the support is sub-par (compared to higher end tools I have dealt with) and not something I would recommend to most companies.

I am headed to Geneva in 1.5 weeks to finish the rest of some migrations we are doing. Essentially moving some servers and integrating some products together.

Our Netapp is still crashing weekly due to a bug with the 10G card. Netapp is having a hard time debugging the core files we have provided. I'm pretty surprised this has taken 6 weeks, and I've escalated to my sales folks to hopefully get them moving.

I'm trying to sell a bunch of surplus gear that we pulled from our old datacenter, which we moved out of on Sunday. Bunch of servers and other random hardware. Curious to see what the lot will get.

Things are good, we have a lot of projects to deliver, but we are progressing, learning, and advancing to the new product launch. Hope all of my readers are doing well and you enjoy the weekend.

Netapp Issues

2009-02-04T16:50:00.010-05:00

We’ve had a few Netapp issues since we went live. One of them was that we were shipped the wrong card, and so we had to reconfigure the nodes with the proper 10GE fiber cards right before we went live. We weren’t aware of the cluster requirements that you had to set the Partner IP. Last night we rebooted both nodes and did a failover test after adding in the partner IPs into the multiple interfaces we are using. The failover worked great, and everything is good.

Every Sunday we have also been seeing a degrading performance issue on one of the filers, it starts out by some pack loss over the LAN, and cascades down to the filer eventually either being rebooted or going down to ping. This effects the 1g interfaces and the 10g interfaces as well. This filer is serving NFS for vmware, as well as CIFS for standard fileserving to a farm of webservers.

I’ve had a case open with Netapp, but the response time of the engineers has been lackluster, which is surprising since we have 1 outage per week on this node. I just noticed yesterday that we were seeing errors on the filer 10g interface (only 1 of them) but after the reboot there were none. The switch wasn’t seeing any errors, only the filer.

Data has been changed below (Network and Address):
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Collis Queue
e0a* 1500 none none 0 0 0 0 0 0
e0b 1500 10.10.5/24 *SNIP* 1m 0 1m 0 0 0
e0c* 1500 none none 0 0 0 0 0 0
e0d 1500 10.10.3/24 *SNIP* 25m 0 16m 0 0 0
e2a 1500 10.10.5/24 *SNIP* 38m 2m 3m 0 0 0
e2b 1500 10.10.2/24 *SNIP* 65k 0 18m 0 0 0
lo 8160 127 localhost 30k 0 30k 0 0 0

Keeping Busy – Akamai changes, Change windows, and QA

2009-01-29T18:30:00.004-05:00

Been super busy cleaning up from the move, and trying to make progress on the QA environment buildout and other projects. We need to get QA up so we can shut down the old datacenter, and stop a bunch of replication scripts we wrote. We’ve been building out a lot of VMs, and messing with the Netapp Flexclones and such, pretty useful. Should be done with QA later next week hopefully, but with the late start I’m not sure. It also depends how the weekend release goes.

I am trying to provision a bunch of new properties with Akamai and the China CDN. It’s always slow going getting approval from the government when we provision, but it works out well. Kind of annoyed I bought a Akamai SSL certificate (up to ten domains) and now they need professional services when I want to use a domain off it. Its like nothing is ever simple with them, too bad they own the market, if I don’t like it I can’t go elsewhere

Need to nail down some better maintenance windows and communications about releases and timing, this should help the sanity of all the IT folks, not just my area (Techops). I wish I had a good change management system that was simple and good for maintenance. Every solution I look for for simple notifications and change management and notification is complex, expensive, and overkill for my needs.

QA is using a terminal server to do testing, which avoids them having to do hostfile hacks. It should help the testing accuracy, and we can much better control how they do their testing. Simple fix for an annoying problem!

Datacenter cutover this weekend

2009-01-14T09:52:00.000-05:00

We are putting the finishing touches on a project which will move us from an old design (many 1U boxes, and some DBs with DAS arrays) to a modern, flexible infrastructure. In the new design there is NAS, Vmware, and much less hardware. It should also allow us to scale much better while fully utilizing storage and server hardware. We have plans to launch a new product later this year, so this prepares us to better launch the product. We still have some loadtesting to do, but we are in pretty good shape.

After this project is over we have a couple of other large projects to complete before March to help merge together environments in our hosting facility. Should be a very long weekend, but looking forward to being on the new environment.

Akamai EDNS

2008-12-14T14:19:00.001-05:00

My team is running yet another move from internally hosted DNS to Akamai EDNS. I've done the same at CCBN and Thomson on other projects previously. The product is great, and the design is superb. We should start moving on Tuesday for web forwarding (from godaddy) and using Akamai EDNS on Wednesday. I look for improves response times and much better DNS resiliency than we had before. More when its implemented J

Been in Geneva doing a datacenter build out

2008-12-14T14:16:00.002-05:00

Built out our new datacenter, I was in Geneva for the last 12 days. We are getting ready to move our US hosting facility to Geneva to get ready to merge the MFG and sourcingparts environments. Its going to take several months overall, but the first steps are going to occur in the next few weeks. Everything went really well, we installed and moved a lot of gear, servers, F5 Big-ips, (Unknown Vendor) HA firewalls, and a nice Netapp 3040 dual head cluster.

The only issue was that we got a quad 1gig card in it versus the 10G copper card we wanted. Netapp only makes a 10G fiber card, so we had to buy different switch modules as well. Netapp ate the extra cost on the cards for us, which was nice. We also can't run LACP on the 10G fiber cards, so we lose switch redundancy.

The Netapp is currently serving Vmware ESXi over NFS, and it screams. We also are using iSCSI for our MSSQL clusters. The speeds on Vmware and iSCSI are very good even with the current 4G we are running until we get the new cards in. Working perfectly, and very happy about it.

The new network and server design is nice and clean, and its working perfectly as well. Still have a lot to build in the 1^st environment we are working on, as well as the other environments we have to build out. Going to be back in Geneva to finish the last physical moves in Feb or so, but we'll be doing things remotely, and moving as much as possible as well. My team over there has been doing great, as well as the US team who was responsible for most of the implementation and design. Great job to all parties involved, I'm very excited by the progress, and the flexibility we'll have with our new 70 disk Netapp system, networking improvements, and the larger Vmware environment.

More later.

Trip to Geneva

2008-12-03T18:31:00.001-05:00

I arrived in lovely Geneva on Tuesday morning, and let some of the guys, as well as one of the engineers who works for me here. Great cool city, reminds me of Banff. We spent some time in the datacenter preparing, and getting acquainted with the staff and facility. We unpacked all of the gear, around 4 tons total, it was a lot of work. Today we started racking and configuring after working on some stuff in the office first. Trying to install some new DCs, access points, CA, and RADIUS authentication for 802.1x using dd-wrt. Still need to finish this work tomorrow, we also have Netapp coming in to finish the configuration of the new FAS3040s we bought.

Had a great dinner with some of the guys here in France, and doing a bunch of work tonight to catch up. More as we progress, I'll be here for another 8 days working on the datacenter.

Nite.

Week in review – Hosting Move planning, and other projects

2008-11-22T10:29:00.001-05:00

Had a productive week, and didn't kill myself. I write this from the office on Saturday, I am working on fixing our fileserver which isn't configured properly (on the disk config). I and a colleague booked a trip to Geneva for 12/2 for 10 days. We finished getting everything out the door for that work, aside from the f5 cluster, which we are still working on. We swapped our two production boxes out for a loaner on Wednesday, that went ok. We are still working on our new hosting contract in Geneva. Everyone wants no downtime, but we need time to build the environment. I didn't have the cash to buy everything, so we have to do some shuffling of equipment. Much of that is repurposing machines with ESXi and moving the old machines to VMs.

Other projects we are working on include a Microsoft licensing deal, new web conferencing, office videoconferencing, a new ticketing system, and some project planning.

Bad bad week, kinda…

2008-11-14T19:12:00.001-05:00

From REALLY BAD to AWESOME J

REALLY BAD : We had a little exchange issue this week, which was not recovered from so well. That's fixed going forward.

SOMEWHAT BAD : On top of that my biggest project is starting to be realized, which is a datacenter move. That's going okay, but we have some interesting swapping to do. I need to move my f5s this week, and ship a whole bunch of networking gear. I have a Netapp, and lots of servers ready to be installed.

BAD: The issue is that my new contract isn't done yet, which is another thing I have to do. I also just realized we aren't going to do managed backups due to the $5k a month they want to charge us, so now I'm ordering a library as well.

BAD : Found out we opened a German office without any IT signoff, no firewall, connectivity issues, and the staff there speaks no English. Trying to get my hands on that, but it's a bad situation. Need to order some gear.

BAD : We have 4 web conferencing tools in use here, we waste a LOT of money. I want to get everyone on 1 or 2 web conferencing tools. Another annoying project to deal with.

GOOD : Just wrapped up signing a new and much better Akamai contract, which is pretty cool. Also got a LOT of purchases pushed through. Probably about 10 purchases this week for the company.

GOOD : Redoing and reassessing all of our backups due to the exchange issue in all of our offices.

KICKASS : Installed a new helpdesk app that I love that we are evaluating – www.cerb4.com its awesome. I hope to get off the crappy ACE system we are using which is like pulling teeth out of a rabid Rottweiler.

Changes at my Company

2008-11-14T19:03:00.001-05:00

I'm not running the group as of 2 weeks ago. That means all operations, enterprise, production, networks, systems, etc etc. I have a good crew of guys, and looking to better augment some of our staff. Happy to have been given the opportunity, and looking forward to making an even larger impact than I have up until this point. Lots of things to get in order which are eating my time aside from all the project management, purchasing, and other engineering I also have to do. Its been very crazy. I'll be posting more on some of the highlights of this week as well.

Monitoring, Reporting, and other stuff :)

2008-10-31T16:39:00.001-04:00

Over the last week, I was sick a couple days. I used some of that time to build a cool management dashboard based on data coming from various tools:

Coradiant – E2E, Host, Status Codes, Sessions, good versus errored
Google Analytics – Pageviews
ACE – Ticket metrics

We are also evaluating the Dejaclick product from Alertsite to enhance the basic remote reporting we are using. The tool is very slick, and the pricing is pretty reasonable for what you get. We are also going to be looking at Coradiant Truesight Edge next week to get more data from our Akamaized traffic (which most of it is).

Other than that just ordering a bunch of RAM for some boxes we are turning into ESXi machines, and also ordering some new Dell switches to run 10Gig Ethernet to the new Netapps we are putting in place.

Domain fun

2008-10-22T17:43:00.001-04:00

I have been busy moving about 400 domains to godaddy. It's been a lot of fun (NOT) consolidating across our current 14 registrars. Once this is done we can start to do web forwarding and migrating onto our new DNS infrastructure. The new one is fully split, where we have a proper external, internal, and update system running. Good design, and should serve us much better.

Enterprise Fun

2008-10-22T17:22:00.001-04:00

I am rewiring a bunch of servers, and reconfiguring a large fileserver on Friday. We need to repartition a 8TB volume into smaller slices. I'm going to use the knoppix CD and some of the tools on that distro to move and recut the NTFS partition so that we can run VSS.

Progressing on the Exchange 2007 migration, everything seems to be 100% now, all we have to do is get some of the mailboxes small enough so we can move them over properly. We are shooting for under 500MB, too bad some of our bad users have 7G mailboxes L

Pain.

Outages

2008-10-08T17:09:00.001-04:00

Our site has been having a lot of issues, so we offloading the image serving onto its own 2 boxes. We also added another 15 spindles on the production DB server, which seems to be helping a lot. We are seeing higher traffic volumes on our Coradiant reports, and we are also seeing lower latency. Good work by the whole team to fix the environment.

Licensing

2008-10-08T16:58:00.001-04:00

Working on some licensing for Redhat as well for our new environment. I don't really like dealing with Redhat, but in order to have a good supported OS we have no choice. They are already using it in some places. We were deciding if we were going to switch to CentOS, but figured it was smarter to have the fallback on a vendor.

Vmware and Firewalls

2008-10-08T16:57:00.001-04:00

We've been pushing more esxi here, and we are going to run our remote offices off a single esxi host running a domain controller, exchange server, and vpn server. We did the same setup for both Shanghai and Geneva offices. We are going to ship them with the new firewalls which should be a good setup, and easy to manage.

Tons of progress this week – VM, Exchange, Firewalls, Security, Storage/NAS

2008-09-26T13:33:00.001-04:00

Had a very productive, and busy week. Balancing out server crashes taking down the production site, and the build-out project we are working on the team was very busy. I built over a dozen VMs, and 3 ESXi boxes.

I also worked on some of the config to finish out the main exchange 2007 implementation. We are waiting for ESXi boxes to ship to the remote offices, which will house a mailbox server, a domain controller, and possible a client access server.

One of my colleagues (who knows the setup of the WAN well) is working on the configuration and testing of our new firewall infrastructure. We are putting in all Sonicwall NSA series appliances. We have a 2400 for Shanghai, and 3500's for Atlanta and Geneva. The production environment will be off a pair of HA 4500 series boxes. The features, ease of setup, and price were excellent. We also have good resources from the reseller and the vendor if we need them.

We are also running a pilot (which has been going quite slow) of the Q1labs qradar product. I have used this product in previous companies, and its been a great tool for security, network analysis, and troubleshooting. The problem is running the pilot with these huge projects that need to be done over the next few months isn't really feasible. I hope to invest more time in testing it, but my priorities are dictating that not to be the case.

I also got rid of a couple older boxes this week by doing P2V using VMware converter starter edition. Now the enterprise ESXi boxes are pretty much full, so I need more disk space in order to do anymore work. That should be happening as we get the NAS implemented.

We wrapped up the Bluearc v. Netapp stuff, we are going with a couple of Netapps. The solution looks very good, and we should have it up in a couple weeks. I'm looking forward to building out some Exchange, MSSQL, and ESXi clusters using it!

Talk to you all soon, have a nice weekend. Loving the weather in Georgia J

Updates on some projects – NAS/Firewalls

2008-09-16T13:52:00.001-04:00

Senior management has given us a date of December to have our current production hosting here in Atlanta (colo, downtown) moved to Geneva (hosting facility we have there). In order to do this we'll need to virtualize a lot, and build out a much better network and storage infrastructure. This will also help us in the future as we grow the business as well.

We pretty much decided at this point that Bluearc is a better solution for us. It was very close between them and Netapp, but it was a matter of performance over more advanced software. We are looking to wrap up the deal soon and get the purchase completed.

We have also decided to go with a sonicwall solution at 5 locations. The product looks very good, and will enable us to have a proper VPN mesh between all sites. We will also be replacing some of our web and spam standalone filters with the new NSA UTM device.

Making some plans to move our Sharepoint and Spam filter from our colo to the corporate office.

Not much else going on at the moment…. That's the update.

Week updates

2008-08-29T15:12:00.001-04:00

Been working on a F5 which is having issues. I was going to reload the OS, but I think I can fix it. Still working on it. Waiting for a reply from support.
Finally figured out a way to export from sharepoint collection, and import them into the root collection. Something I'll need to do before we can reorganize the structure of our sharepoint here. I should have this finished early next week. Have to schedule some downtime to do the export/import.
Got my first ESXi machine, with the embedded hypervisor. Me and Jamie will rack it up and get it running on Monday, they we should be good to start loading the backlog of about 5 VMs to start.
Should be making a decision on NAS and Firewalls next week. Have to nail down the specific configs and pricing.
Threw SQL 2008 (DB, SSAS, Report server) on my development box, seems pretty cool so far.
Completed the implementation of DB monitoring and machine monitoring on production and enterprise hosts.

I think that covers this week's major project work. Still also doing some of day-to-day work, fixing things as needed.

Wireless networks using certificates painful

2008-08-20T17:20:00.001-04:00

Another project which is being annoying is hooking up a few hacked dd-wrt boxes to my Active Directory CA by using IAS. Still doing some testing, and I'll post my findings and details once the team figures out why Windows XP doesn't seem to want to work (we are trying SP3 now). Works like a charm on Vista J

Love Vista.

Sharepoint annoyances

2008-08-20T17:18:00.001-04:00

Have a bit of a time trying to fix up this mess of a sharepoint site. First thing we got our 3GB database off SQL express. Now we are on standard, I need to fix up the database itself. Myself and a co-worker (Steve) are having a tuff time figuring out how to restructure the site, since it was setup in a very odd manner. I think we have some good ideas, so lets hope we can get a good structure to it. I would post the hierarchy we are moving towards here, but I probably shouldn't J

Storage/NAS Evals

2008-08-20T17:15:00.001-04:00

We are looking for a strong NAS system, I'm leaning towards a clustered scale up/out system versus buying a box that I have to replace every 4 years. I think Isilon is the proven leader in this space, and we've selected them to go up against the king (Netapp). Here are the criteria we used to get it down to these two. We are looking into them in depth now.

Requirement	Sub-Feature	Weight
Architecture
	Centralized Management	6
	Clustered Device (NAS, Controller, and Power)	10
	Appliance Model	8
	Add storage with no downtime	10
	Add bandwidth and nodes without downtime	10
	Add cache without downtime (increase IO)	8
	Auto balance IO across disks and connections	10
Security
	AD Integration	10
Data Management
	Tiered design 2 or 3 tier	8
	Migration of data based on usage and other vars	8
	Content aware	5
	Ability to snapshot multiple times, and replicate a snap	10
	Rapid snapshot	6
	NDMP support	5
	Backup Exec Support	10
Connectivity
	CIFS	10
	NFS	10
	Replication at file/block	8
	Optimization for WAN replication	10
	Namespace virtualization and migration of namespaces	7
Management
	Reporting via Web	8
	Email reporting	6
	Usage/compliance reporting per user (AD integration)	6
	Monitoring and alerting of issues	8
	Thin provisioning	5
General
	Ease of administration	10
	Ease of use	10
	Configuration and setup	8
	Documentation quality	5
	Speed of client	8
	Resources used by client	8
Company
	Viability	10
	Support	8
	Price	10
*Total Score*

Things happening - Firewalls

2008-08-20T16:54:00.001-04:00

We decided to evaluate Sonicwall and Cisco based on our assessment. We are digging into depth in the next week or so. Here is the criteria. These are needed for both hosting/collocation and for our 3 major offices:

Requirement	Sub-Feature	Weight
Architecture
	Centralized Management	10
	Clustered Device	8
	Appliance Model	8
	QOS management	6
	Appliance must support up to 2G (external) and 3G (internal)	10
Security
	Stateful Firewall	10
	Full packet inspection and content filtering	10
	IPSEC VPN Support	10
	SSL VPN Support	8
	OpenVPN Support	8
	PTPP Support	9
	AD integration for authentication	6
	Anti virus/Auti Spam/Anti spyware	4
	IDS/IPS	9
	Enforce desktop patchlevel and AV, Quarantine user	4
	Wireless security	4
	Behavioral analysis	7
Management
	Reporting via Web	8
	Email reporting/Monitoring and alerting of issues	8
	Usage/compliance reporting per user (AD integration)	6
	Backup/Restore	8
	Upgrades	8
General
	Ease of administration	10
	Ease of use	10
	Configuration and setup	8
	Documentation quality	5
	Speed of client	8
	Resources used by client	8
Company
	Viability	10
	Support	8
	Price	10
*Total Score*

Missed a couple things

2008-08-13T18:42:00.001-04:00

I'm moving us onto SQL Standard from SQL Express which is what the corporate intranet site is running. That upgrade was tested, and I'm doing production today. I've also setup some proper backup jobs for the database.

Starting tomorrow, I (and another engineer here) are redoing the Sharepoint structure, permissions, etc. It should be a lot more clear, and easier to understand what people are up to.

I haven't done a lot of Sharepoint administration, so it should be a good learning experience!

Recap of week 1.25 :)

2008-08-13T17:15:00.001-04:00

So first week on the new job, and making some good progress. I am learning the infrastructure and some issues that have been bothering us. We have done the following items:

Monitoring
1. Redid the Coradiant Truesight setup to better catch items and view backend information. Got visibility to additional network areas.
2. Implemented Solarwinds IPMonitor. We are installing it at the colo and at our enterprise office.
3. Testing Idera DM, deciding if it will work for us. We need better DB monitoring and diagnostics.
Infrastructure planning
1. Did initial grading of clustered scale up/out NAS solutions. I will post more details as the project progresses.
2. Did requirements for new firewall solutions, still have yet to nail this down and grade them.
3. Built plan around fixing exchange, and moving to a multi-site international infrastructure on Windows 2008 and Exchange 2007.
4. Started planning a DNS revamp, and proper split domain configurations.
5. Working on a new wireless implementation as we speak, using dd-wrt and integration into AD with WPA.
Ops
1. Debugged issues going on with production website.
2. Reviewed and did some updates on the Akamai configuration.
3. Implemented VMware environment for HP Quality Center testing, have yet to have QA fully test the buildout.

I learned about the platform, and the upcoming new version of the platform. We also started looking at NBA/IDS systems such at Mazu and Q1Labs. Its not high priority, but it would help a lot with security, and the ability to diagnose network issues, and non-http issues. I really wish Coradiant would view SQL response time…. One can dream.

The fun begins

2008-08-05T10:01:00.001-04:00

I had a smooth 15.5 hour drive down to Atlanta from Boston on Saturday. I have to do it again with my girl, cats, and other car in a couple weeks. Just working on renting my place in Boston, should be done in a couple days.

I got my place on Sunday, which is awesome. It's on Grant Park, and It's huge and empty. The city is great, and I'm doing a reverse commute.

I've been digging in on some things (yes its only day 2)

Looked at some network issues, but I haven't determined anything really yet.
Sharepoint migration plan from SQL Express to SQL Standard.
1. Temporary VM setup for testing
2. Purchasing a proper VMware box for real dev VM
Looking at some NAS vendors, and putting together a requirements plan. I will post that once we nail it down today or tomorrow.
1. Support for Virtualization (ESX clusters)
2. Exchange clustering
3. SQL clustering
4. Located in Geneva development, Colocation, and Corporate office for DR/replication of data.
Looking at deploying www.dd-wrt.com and integrating our wireless into AD with enterprise WPA. Should be fun and easy to do.

Funny stuff for today

2008-07-31T11:56:00.001-04:00

Love the failblog!

http://failblog.org/2008/07/31/grammar-check-fail/

NAS NAS NAS

2008-07-24T09:05:00.000-04:00

I've onlyr eally delat with a couple scalable NASes, and the EMC Celera was a nightmere for many reasons. I always did like Netapp, they work well and are easy to manage. Now there are the "next gen" NASes out there, at the new gig we need something good, which can scale easily and run clusters off MS and Linux technologies. I was looking at the following "cool" vendors. Has anyone used these before?

These are in order of which ones I think are coolest:

Isillon - I know we have some at my current company, and they work well. Nice product, no idea on costs.
Onstor - Seen these guys when I was looking at 3par, product looked good, cheap, and solid.
Netapp - old faithful is something thats good with storage :) Easy to find people who've used it. Who hasn't?
Bluearc - seems good
Ibrix - Looks interesting since you don't need to buy hardware, but might be too "out there"
Pillar - looks okay...

News time- Moving jobs and moving locations

2008-07-18T12:07:00.000-04:00

Now that I told most of the people who I work with, and the vendors who I deal with often. I've decided to accept a position at a very exciting startup. The company is located in Atlanta, so I'll be moving down south. It should be interesting since with the high growth Atlanta is not as "southern" as it could be.

The company name is MFG.com (www.mfg.com) and its a very interesting business. They match up companies who want to have parts manufactured with manufacturers. The kicker is that they have a good presence in europe, and a large presence in China. It allows for smaller shops to get the benefits of the big companies regarding leveraging the global economy. It has some very high profile investors, and has a profitable, and fast growing business.

I will be doing all kinds of infrastructure work there. There is a hot list of major issues which I hope to start to tackle right away, but I'll be moving back into a role further from the tools and technology and into general IT issues. I will use my skills to focus on system issues, application problems, network issues, and security problems. I already have to tackle some problems around exchange, sharepoint, and the hosting environment. There is a lack of specific kinds of tools there, and I hope to take a best of breed approach with a combination of open source, and small company products (Sorry HP and IBM). There are a lot of cool products out there I have tested and wished I had a smaller enterprise to deal with. Now that i'll be working in a smaller environment in terms of production and enterprise it will be a good chance to use these tools.

Since my current employer is large, public, and sensitive, I've avoided naming them. I will be leaving there on 8/1, and starting my new position on 8/4.

Please keep reading, I will post anything that comes up in the meantime, and should resume in early August.

Thanks to you all, please leave comments, IMs, or email me!

A bit crazy

2008-07-15T17:03:00.001-04:00

Sorry I haven't updated you all. I will be updating on Friday. Mostly its to announce something which I am going to be undertaking. It should be interesting, and a bit of a shift back to my roots.

More on Friday!

HP Software Universe

2008-06-24T16:32:00.001-04:00

I attended and spoke at HPSU this year. It was a good conference, and HP is making a lot of progress on the products which we are very interested in. I'm going to go into each of these and explain some of what I saw, and what we are up to with each product.

HP NAS
- New UI is coming down the road.
- Additional of bare metal provisioning.
- Internally
  - We are moving from 7.0 to 7.2 here in the next few weeks.
  - We are just finishing up a MS-SQL Multimaster setup with the UK this week. It's been painful, and taken us 4 attempts now, but we are close.
HP SAS
- Some interesting stuff with other customers, best practices, and other tips on it.
- Internally
  - We are moving towards a small SAS deployment in 2008, probably 200-400 systems. Just firming up some budgets, since we are splitting the cost among a couple budgets.
HP OO
- Tons of new content, which makes OO above and beyond.
- Multimaster is coming down the pipe, which should be really nice. Since the platform of OO is very similar to NAS it should be easily done.
HP BAC/RUM
- Lots of progress in RUM, especially generic TCP Monitoring.
- Correlation of alarms coming into BAC.
- Baselining and auto thresholding – REALLY GOOD TO SEE THIS IN BPM!
  - They have taken this another level higher, and they show you how it would have worked if you use the suggested thresholds.
- Problem management and workflow ideas which should help the usage.
- Integration of OM into UCMDB, and other feeds to and from the UCMDB.
HP BPM
- Interesting customer presentation of a large distributed BPM they deploy at branch offices. Very good use case, and well managed. They deal with BPM problems in a good way.
HP OM
- Sitescope integration in OM 8.1.
- New enhancements around usability and reporting.
- HTTP agent is excellent, and much easier to deal with than the old agent.

HP Software Universe

2008-06-19T13:04:00.003-04:00

I'm at HPSU this week. I will summarize the great stuff happening over here this weekend when I'm back east.

HP Software Universe 2008

2008-06-06T12:33:00.001-04:00

I'm giving a talk at HPSU on our conversion off OVIS onto BAC, BPM, and Sitescope. Should be pretty good, and straightforward, it's taken a while to get it all baked and integrated, but things are progressing well now.

Are any readers planning on attending the show? It would be good to hear from you guys who are, and we can meet up.

Thanks!

HP – Real User Monitoring (RUM)

2008-06-06T12:25:00.001-04:00

I like what I see from HP RUM, my issue is that the pricing and model for it don't work in a shared environment. I can't nail down each application, webserver, application server, or environment. I prefer to deploy it at the edge of the datacenter and just use it as needed. This is how we deploy and use our current real user monitoring from Coradiant. HP has been trying to be flexible, and create a pricing model that works for us over the last 6 months, but we haven't made the kind of progress that I need to see.

We can't test or deploy a solution which isn't cost effective compared to what I am getting from my current vendor. I don't want to POC the product if it doesn't make financial sense. Hopefully at HP Software Universe we'll make more progress, and enable us to test the product. Ultimately the solution is very compelling and fits nicely with our usage of Business Availability Center (BAC) and related products.

Tool Replacements

2008-06-06T12:17:00.001-04:00

I have a couple of short term tool replacements we are gearing up to normalize toolsets across the company. We are a HP NNM shop, and we've decided to finally get rid of it, due to the numerous problems we've been dealing with over the past several years. The amount of work that this tool needs is immense. This is why HP is doing a ground up rewrite of it for 8.0. We can't wait for it, since its not mature enough.

We are implementing IBM ITNM (Formerly Precision). We are also going forward with enhancing our event management by using IBM Impact. Both of these products will provide a lot of value for us in terms of efficiency and increasing the effectiveness of the event management layer we already use from IBM.

We are also looking to get HP SAS in for a small install which will scale out. It's looking like 400 systems to start with. More on that as we nail down the project and timelines. More on the HP side, we are also looking to scale out HP OO and start using it across the company for larger projects.

IBM Pulse 2008 - Review

2008-05-23T16:42:00.001-04:00

I spent Monday-Wednesday at IBM Pulse in Orlando. It was a good show, but quite a few of the sessions were full when I arrived. It was frustrating because they didn't offer them more than once. The morning sessions were mostly pie in the sky, and not very useful to me. I got to spend a lot of time with senior people in engineering, architecture, and acquisitions/strategy. I also got to meet people I knew from online or other dealings with IBM. Overall, the show was a good use of my time, and I found it enjoyable.

Here are some of my highlights:

ITM 6.2.1 improvements including agentless capabilities and such.
New reporting framework based on BIRT which will be rolling forward.
New UI which is being pushed and was on display from TBSM 4.2.
Hearing about what other customers are up to (mostly bad decisions from what I've seen).
Affirmation of ITNM (Precision) as a best of breed tool, with a excellent roadmap.

Some things which are bad and make no sense:

Focus on manufacturing (due to MRO).
Pushing the MRO platform as IT, when it's clearly not ready for prime time.
Pushing TPM as datacenter automation. TPM is a worst of breed product in every way, and is years behind HP SAS and Bladelogic. The product will never catch up.
Allowing people to abuse Omnibus by turning it into a monitoring tool, and not a MOM or event manager.
Lack of clarity in strategy for Webtop vs TBSM (which customers seem to build BSM on Webtops).
ITCAM is total junk, and is years behind competing products from HP (Mercury).

Week in review

2008-05-23T16:18:00.001-04:00

Been working out some organizational stuff with my counterpart of the company we purchased. I've also been spending time figuring out the future toolset we will be using for server/app monitoring, event management, network monitoring and management. It will be a combination of our toolsets based on what makes sense. Automation is a huge goal for us of course.

Need to put together some documentation on what we have, and then complete a strategy in the next 4-5 weeks.

Conference Updates

2008-05-14T09:06:00.001-04:00

I am speaking at HP Software Universe in June in Vegas. It should be interesting. I am talking about our conversion off OVIS onto HP Sitescope, HP BPM, and HP BAC. We are very happy with the new products, and most of all my monitoring customers absolutely love the visibility and clarity these new tools give them.

I am attending IBM Pluse as well in Orlando this weekend. This is my first IBM show, and I'm very interested in ITNM (which we are moving to from NNM in the next 12 months), Omnibus, Webtop, and some of the other IBM tools we use. The show doesn't seem as well put together as shows from CA, HP, or EMC. We'll see how it runs. The venue is definitely questionable… I'm not a big fan of the Disney junk J

Been a little out of it – Updates on my job

2008-05-14T09:02:00.001-04:00

Once again I've been quite busy dealing with the integration of a large company we purchased. They have a lot of major toolset issues which need to be worked through. I feel there are a lot of architecture problems with what they are using for monitoring. Since the project is 18 months old, and not progressing as well they are still sinking money into a sinking ship. Should be interesting to see when they want to scale this back, or actually design a portfolio which is best of breed, versus trying to implement a full solution from a single large vendor. Should be a matter of time…. Then they will ask for my help and I'll be willing to assist. I don't want to be the architect when I can't make major changes to the design which has a lot of flaws.

I am interested in potentially running a transformation group, which would include global reach of automation, virtualization, etc. This would be a new area for us, and we need to get serious with products like HP OO, SAS, and NAS. Leveraging automation across the board would be a huge cost and time saver.

Other than that, not sure where I am off to.

Several close friends, and excellent co-workers have been leaving my company. My thoughts are with you in your ventures, they sound like excellent moves which will promote your learning and horizons. I hope to work with you again, and we'll be in touch on LinkedIn or Plaxo!!!

I also have had several people I work with at vendors who's software has been a great asset and learning experience who have also moved onto smaller upcoming companies. I wish them luck, and I hope to work with them in the future again.

Travels

2008-04-25T12:49:00.001-04:00

Been travelling a bit the last week, and I am off to London for most of next week to work on strategy and organizational structure. I was offered a seat at IBM Pulse, but I don't think I'm going to make it. I will still be attending HP Software Universe in mid-June, which should be interesting given the advances in the HP products from Mercury and Opsware integrations.

Toolset Changes

2008-04-25T12:48:00.001-04:00

As many of you know, we just completed a large purchase of another company, so in the next few weeks I will know what I am going to be doing. In the meantime we are figuring out how to converge our toolsets. There will be more activity here as we work through the issues and determine the final set of products we are going to be pushing out across the universe of the company. Based on some of the deals we make with IBM and other vendors it could also create other areas where we can standardize.

More to come in this area.

IBM POTs

2008-04-25T12:45:00.001-04:00

This week we were offsite at the tech center in NYC for a day trip. We looked at IBM Tivoli Provisioning Manager (TPM), the provisioning, deployment product. It is one of the products we are considering standardizing on. I wanted to get clear picture of what it can and cannot do versus Opsware SAS. The product looks good, but I still need to write up the full gap analysis. It definitely would meet most of our patching, inventory, and deployment requirements, but it doesn't fill the system administration, or complex audit and control requirements we are given due to customer audits and regulatory compliance.

Last week IBM brought us a POC for IBM Tivoli Monitoring (ITM), which is the monitoring platform. It compares to HP Openview Operations (OVO). We are going to bring it in house to do more testing, but upon the initial 1 day with the product, we found the following comparison to be true:

Issues in POC:

Multiple times the agent died, and the server died. There was no indication of the error aside from a manual restart.
Did not go over agent installation.

Environment:

Pros to ITM:

Reporting is nicer, and based on open standards.
Multiple server roll into a single TEMS easier than OVO.
More flexible on operating system, database and platform the components can run on.
IBM is quicker to support new component versions (OS, Application server, etc)

Cons to ITM:

Email management for notifications outside of event escalation are not manageable aside from using command line calls with emails as arguments.
Scenarios applied to groups are not easily manageable, meaning you have to manage the policy in a lot of notifications.
UI is not as easy to use, there are fewer wizards to guide the engineer with the workflow of making a change or implementing something new.
Everything seems to run as a separate agent. So you will have a Windows OS agent, a Universal Agent, and a Custom Agent etc, with all of them running as separate services and processes.