Vmware 4.1

Let me start off, by just saying that this site is great, its always entertaining and filled with great data : http://get-admin.com/blog/

I'm very happy that Vmware released 4.1 recently. After the horror stories I read about 4.0U1 we decided to skip them and wait for 4.1. There wasn't a lot in the updates we cared for anyways. There are quite a few interesting features in 4.1, and one of the good things that Vmware has done is finally killed off ESX (after this release). ESXi has been great for us over the last couple years, and I haven't had any complaints with switching over to it from ESX in previous years. I found using update manager was not the most reliable when moving from 3.x to 4.x on the physical systems themselves, so we opted to use the host upgrade tool. This is not a supported method to move from 4.x to 4.1. We will probably have to give the update manager another run, which concerns me. At least its not as complex as upgrading Hyper-v J

I will probably start upgrading our enterprise (not production) systems to 4.1 in the next couple weeks, and I will post my findings on the blog as I go.

Thoughts on IBM BigFix Purchase

Bigfix makes some excellent products, and they have been moving in great directions over the last couple years. They have moved out of pure remediation and into configuration management and control. I would have loved to have purchased them for use at my current company, but the pricing was a bit higher than I'm using for Shavlik HFNetchk Protect, which is another good product, but is far more limited. I wanted to have one tool to patch Linux and Windows systems.

IBM has been really struggling to provide a good provisioning and patch management tool for years and years. First they were pushing TPM which is probably the worst product I have seen IBM release. Unfortunately a company I worked for previously was obsessed with using this product that most Tivoli enterprise customers get for free and completely disregard. I spent a good amount of time looking at the product and its capabilities, or lack thereof. I'm concluding my rant now, but its happy to see IBM adding a superb replacement for TPM and adding additional security related products they will acquire with the Bigfix purchase.

I was also quite surprised at the cost of the purchase at $400m. I know Bigfix has a lot of customers, and they sell a service, which makes it nice for both operating business as well as the customers who can bill this against opex versus capex. I would have assumed they would have had to pay more for the company. It will be interesting to see what features IBM takes from them and puts into Tivoli, and which other ones become part of the ISS portfolio over time.

Simplify and Automate

Now that the workload has reduced a bit over the last month or so we can spend time doing project work. It's always been my philosophy to simplify as much as possible, this is normally because I end up having to fix messes, which are normally caused by undue complexity. Complexity can affect performance, availability, and manageability. Automation can often create complexity, as can requests by various people in the business who don't necessarily plan the projects or requests they make of others (especially development and operations).

This being said I often get blocked when I try to simplify things, because people want to build things out in a more redundant manner than is required for the business needs. There are a lot of ways to create a redundant system without creating complexity, you just have to step back and look at the overall configuration and requirements to come up with the best solution.

We get a lot of requests from our QA team to reload various Resin app servers, and other processes. What we are doing now is creating a web based interface for them to do the reloads on their own. This eliminates the need for operations to run the scripts, and saves time and resources.

JMS, Endeca

We are building a services tier which will be based on ActiveMQ JMS, and our standard Resin webserver/app servers. We are building this with 2 nodes sharing each on the nodes. They will use a shared file system. We have a few services to start with, which are internal only at first. Should be interesting.

We are also working on some new products, and we are pretty close to selecting Endeca search for the indexing engine and SEO engine for it. More on that as we get along with development and implementation. The product looks pretty cool, so it should be fun.

Upgrade Land for Microsoft - Sharepoint / Exchange 2010, and JIRA 4.1

    I have been using Office 2010 for a while, and moved from the preview, to beta, and now I am finally on the release. We decided corporately to stick to 32 bit even when we are on 64 bit windows 7 on our newer systems. The main reason for staying on 32 bit was the all of the addons in the market are written for 32 bit only. When I was testing on 64 bit, I wished I had just stuck to 32. The released version has been stable for the last few days, but I didn't have much issues with the beta release either.

    Now that we have office underway we are beginning upgrades to the other 2010 products we use from Microsoft. The first one was Sharepoint, which we are MOSS 2007 right now. The migration was slightly painful, and here are some of the pointers that I found helpful in the migration.

• Run Powershell as admin
  
  • http://manish-sharepoint.blogspot.com/2009/11/error-while-working-with-powershell.html
    
• Batch Upgrade Visual Styles (the new look and feel for sharepoint)
  
  • http://technet.microsoft.com/en-us/library/ff607998(office.14).aspx#section4
    

    The next step is a bunch of testing, and hopefully cutting over next weekend to the new version (5/8/10). We avoided any custom components on our sharepoint, which made the migration much simpler. We have yet to have any complaints with the migrated test data. The new interface is awesome, and works great in Chrome as well. Great job to the Microsoft Team on this product!

    We are in process of an exchange 2010 upgrade as well, we are building out some new VMs and we will migrate the mailboxes over. The work is still initial on that project, so I will post more on that as we go. My colleague is the main lead on that project.

    On another side note, I moved us from JIRA 4.0 to JIRA 4.1. The upgrade was somewhat manual and required some work and planning. The new JIRA interface is very nice, and its good to see them finally changing the old reliable interface they have had for many years. Now if they would only fix the UI for the admin section so I could stop scrolling on a huge list that would be great!

 

Tridion Upgrade 2009 SP1

We have decided after some pain to give Tridion another go over here. We have some really sharp guys helping us from the firm, and they have helped us immensely. We just upgraded to the newest version, and after the struggle to get it running initially it's gone very smoothly and simply. Within a couple hours we moved everything over to the new version and its working flawlessly. It was very simple and good to see the quality of the installers. They handled pretty much everything without any additional manual steps. We are looking forward to moving to the new version later this summer as we beta test for them.

    Lots of the issues with the product were due to the implementation that was designed for us. We will be redoing our site and building it properly using the new version. I think with proper guidance and a good technical team we will not have the issues from the past. We are also moving a lot of our custom code from the current codebase into a web services layer that will isolate the our code from the main Tridion content. I am looking forward to the project.

    There are lots of other things going on today, new database server swap for our performance testing, and a bunch of other project work. Its good that its quiet in the office as far as non-project work goes.

Week in Geneva

Just wrapping up a week of pretty intense work here in our datacenter, here is a list of some of the fun projects we accomplished.

1. Disk upgrades to netapp
   
   1. Netapp locally here in Switzerland went out of their way to fix issues caused by my purchase in the US. Last time I buy in US and ship overseas.
      
   2. Netapp also looked over the system and made some very good corrections and suggestions, much appreciated the great customer support.
      
2. Reconfigured network
   
   1. Move 10GE to other subnets
      
   2. Change netapp network config
      
   3. Several other additional cables and infrastructure was built out
      
3. Firewall Upgrades
   
4. F5 Upgrades from OSv9.4.3 to OSv10.1
   
5. Install 3 New VM Servers
   
6. Install memory in systems (DB, VM)
   
7. Cleanup of office, and build other infrastructure
   
8. Major failover testing of netapp, firewall, and loadbalancers
   

Now we are trying to get home with the volcanic ash situation in Europe. It looks like we will be driving our rental car to Barcelona, and taking a flight from there. Should be an interesting little side trip.

More fun later, glad to have a little break after working crazy hours the last week. J

Finally a way to block those pesky bots stealing content

We've been using a product over at MFG which is sort of like an invisible captcha tool. The beauty of the product is the end user doesn't even know its running, but the accuracy and technology which is used is very unique and cutting edge. We first started speaking with Pramana – www.pramana.com about over a year ago, initially there was issues with the technology, but it had progressed quickly and become rock solid. I was unable to get false positives in all my testing and scripting.

We implemented the technology (Pramana HumanPresent - www.pramana.com/human-present/) based on issues with competitors which sell databases and information about manufacturing companies essentially stealing our content. They use various methods, including screen scraping, and seo scraping bots. This has been observed in many occasions, and we even had one company who wanted to sell out to us, while they were stealing our data! (somewhat legally)

The product is not super simple to implement, but the benefits are great. They have SDKs for a bunch of languages (for us we use Java, which is more complex than the PHP API or others they have). The SDKs give you all kinds of granular control.

We are a paying customer of Pramana, and they got the great idea of letting users use the service for free (Called BotAlert - http://www.pramana.com/botalert/) in order to detect and measure the bots (you get pretty daily reports from them), if you want to block the bots then you have to pay. The cost is very reasonable considering it doesn't inconvenience users, and it can also allow search engine crawlers to index content, but homebuilt screen scrapers to be blocked.

F5 Persistence and my 6 week battle with support

We've been having issues with persistence on our F5's since we launched our new product. We have tried many different ways of trying to get our clients to stick on a server. Of course the first step was using a standard cookie persistence which the F5 was injecting. All of our products which use SSL is being terminated on the F5, which makes cookie work fine even for SSL traffic. After we started seeing clients going to many servers, we figured it would be safe to use a JSESSIONID cookie which is a standard Java application server cookie that is always unique per session. We implemented the following Irule (slightly modified in order to get more logging):

http://devcentral.f5.com/Default.aspx?tabid=53&view=topic&postid=1171255 (registration is free)

when HTTP_REQUEST {

# Check if there is a JSESSIONID cookie

if {[HTTP::cookie "JSESSIONID"] ne ""}{

# Persist off of the cookie value with a timeout of 2 hours (7200 seconds)

persist uie [string tolower [HTTP::cookie "JSESSIONID"]] 7200

# Log that we're using the cookie value for persistence and the persistence key if it exists.

log local0. "[IP::client_addr]:[TCP::client_port]: Request to [HTTP::uri] on server [LB::server] with cookie: [HTTP::cookie value JSESSIONID]"

} else {

# Parse the jsessionid from the path

set jsess [findstr [string tolower [HTTP::path]] "jsessionid=" 11]

# Use the jsessionid from the path for persisting with a timeout of 2 hours (7200 seconds)

if { $jsess != "" } {

persist uie $jsess 7200

# Log that we're using the path jessionid for persistence and the persistence key if it exists.

log local0. "[IP::client_addr]:[TCP::client_port]: Request to [HTTP::uri] on server [LB::server] used persistence record from path: [persist lookup uie $jsess]"

}

}

}

when HTTP_RESPONSE {

# Check if there is a jsessionid cookie in the response

if {[HTTP::cookie "JSESSIONID"] ne ""} {

# Persist off of the cookie value with a timeout of 2 hours (7200 seconds)

persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 7200

            # Log Response

log local0. "[IP::client_addr]:[TCP::client_port]: Request to server [LB::server] with cookie: [HTTP::cookie value JSESSIONID]. Added persistence record from cookie: [persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]]]"

}

}

when LB_SELECTED {

log "From [IP::client_addr] to physical server [LB::server] the cookie JSESSIONID is [HTTP::cookie "JSESSIONID"] URI JESSIONID is [findstr [string tolower [HTTP::path]] "jsessionid=" 11] "

}

 

We've replicated and done 3 rounds of packet captures, and you can always see the issue in the logging from the irule above:

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63231: Request to /images/mfg/icons/search_cross.png on server -http-pool x.x.x.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63229: Request to /images/mfg/icons/icon_largemessages.png on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63228: Request to /images/mfg/icons/icon_clock.png on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63231: Request to /images/mfg/icons/icon_largequotes.png on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63233: Request to /images/mfg/icons/icon_largendas.png on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63230: Request to /images/mfg/icons/icon_largebluestar.png on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63232: Request to /mfg/scripts/search/search.js on server -http-pool BACKENDSUBNET.19 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63235: Request to /favicon.ico on server -http-pool 0 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: 01220002:6: Rule JSESSION_iRule_withlogging <LB_SELECTED>: From CLIENTIP to physical server -http-pool BACKENDSUBNET.19 80 the cookie JSESSIONID is abcND0QYKjeOCczB8c_Ds URI JESSIONID is

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63236: Request to /servlet/mfg.Controller?time=1269130074065&pmId=1001&act=1154 on server -http-pool 0 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:14:25 tmm tmm[1629]: 01220002:6: Rule JSESSION_iRule_withlogging <LB_SELECTED>: From CLIENTIP to physical server -http-pool BACKENDSUBNET.19 80 the cookie JSESSIONID is abcND0QYKjeOCczB8c_Ds URI JESSIONID is

Mar 21 01:14:25 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63224: Request to /mfg/contactHome.jsp?time=1269130079475&pmId=1154 on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:31 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /servlet/mfg.Controller?time=1269130079686&pmId=1154&act=supplierDisplayAgent&aid=904564&dgrdv=1 on server -http-pool 0 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:31 tmm tmm[1629]: 01220002:6: Rule JSESSION_iRule_withlogging <LB_SELECTED>: From CLIENTIP to physical server -http-pool BACKENDSUBNET.20 80 the cookie JSESSIONID is abcND0QYKjeOCczB8c_Ds URI JESSIONID is

Mar 21 01:15:31 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /mfg/contactHome.jsp?time=1269130145070&pmId=1016 on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:43 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /images/mfg/modalbox/close.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:44 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /servlet/mfg.Controller?time=1269130145287&pmId=1016&act=modal&mtId=800&mLoad=true&aid=904564 on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:44 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /images/bo/design/spinner.gif on server -http-pool 0 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:44 tmm tmm[1629]: 01220002:6: Rule JSESSION_iRule_withlogging <LB_SELECTED>: From CLIENTIP to physical server -http-pool BACKENDSUBNET.20 80 the cookie JSESSIONID is abcND0QYKjeOCczB8c_Ds URI JESSIONID is

Mar 21 01:15:44 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /images/mfg/icons/search_cross.png on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:44 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /images/mfg/icons/doubleDownArrow.png on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:45 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /images/mfg/combo/comboover.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:45 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /images/mfg/combo/combopress.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /servlet/mfg.ajaxProvider.GetDisciplineProvider;jsessionid=abcND0QYKjeOCczB8c_Ds?time=1269130158059&pmId=1016 on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /servlet/mfg.ajaxProvider.GetRfqBuyerLocationProvider?time=1269130158059&pmId=1016&sImg=false&sCwor=false on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /images/mfg/icons/dhtmlTree/iconUnCheckAll.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /servlet/mfg.ajaxProvider.GetMaterialProvider?time=1269130158060&pmId=1016&sImg=false on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /images/mfg/combo/combonormal.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /servlet/mfg.ajaxProvider.GetIndustryProvider?time=1269130158060&pmId=1016&ids= on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:46 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63250: Request to /servlet/mfg.ajaxProvider.GetLanguageProvider?time=1269130158060&pmId=1016&ids= on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

Mar 21 01:15:47 tmm tmm[1629]: Rule JSESSION_iRule_withlogging <HTTP_REQUEST>: CLIENTIP:63247: Request to /images/mfg/icons/dhtmlTree/folderOpen.gif on server -http-pool BACKENDSUBNET.20 80 with cookie: abcND0QYKjeOCczB8c_Ds

 

The support team cannot figure out why this is, and it's been going on for very long. I will keep updating this as it goes on. The latest saga is that they are blaming it on the SNAT we are using.

http://devcentral.f5.com/Default.aspx?tabid=53&view=topic&postid=813179

http://devcentral.f5.com/Default.aspx?tabid=53&forumid=5&tpage=1&view=topic&postid=86374

F5 can be implemented as the inline gateway or outside of the gateway as a NAT device. We opted for the latter in order to avoid sending unnecessary traffic through the device. We may have to do some testing with it as the gateway and see if it fixes it. It clearly looks like a bug to me. We are running BIG-IP 9.4.5 Build 1049.10 Final, and we're planning on moving to V10 soon.

The Battle of the CMS

My company was paying a lot for an expensive CMS system which wasn't working properly, so I had mentioned we should look at Joomlah and Drupal, as some of the popular systems out on the internet. Of course IT wasn't as involved as we should have been, and marketing is essentially forcing us to use Drupal. Then we starting giving some of our requirements and some of the lack of integrated core functionality is pretty disappointing for a proper CMS. I still have yet to see a full requirements list, but I do have a list of half a dozen or so on the operations side surrounding deployment, rollback, and environment management. I hope someone puts together a proper requirements list so we know where the technology will work well and where it will fail.

I