Fixing the Blackberry Enterprise Server Administration Service

Our workplace has deployed Blackberry phones to around 40% of its workforce. This is a pretty high figure for NSW Government agencies, I believe - if nothing else, a reasonable proportion of staff need to stay connected all the time. While my own IT team have moved on to Android devices, Blackberry continues to make a lot of sense for the general business, but power users like IT tend to get more benefit out of more fully featured smartphones.

Our Blackberry Enterprise Server (BES) 5.02 deployment developed a fault where my team were unable to login to the Blackberry Administration Service (BAS) at all. Equally, the Blackberry Web Desktop Manager wouldn't work with AD logins. I'll plead some culpability in terms of configuring only AD logins to the server - you really should always have a BAS account defined for just such a reason.

That said, it was a problem which needed a solution, and having resolved it, it seemed like a good opportunity to share. 

Our BES is installed on a VMWare VSphere VM, running Windows 2008 R2 SP1. The fault seemed entirely unrelated to a recent upgrade of our VSphere farm from 4.0 to 4.1, and the Nexus 1000v switch to 4.2(1) SV1(4). Equally, it didn't seem related to either Windows 2008 R2 SP1, or to a recent issue I'd had on some physical 2008 R2 servers. Having ruled these out, there were 3 things which I saw as needing attention;

1. Establishing a working login to the BAS
2. Troubleshooting the fault with AD logins
3. Upgrading the BES installation to SP3, having recently been released

In investigating the SP3 prerequisites, I found that the MDS Integration Service needed to be uninstalled before upgrading. This in itself necessitated a working BAS account, so it was clear that my priority had to be the login to BAS.

Knowing that the BAS users were stored in the BESMgmt user database, I decided to convert an existing AD account to a BAS account rather than adding a new one. The table used for BAS users is called, funnily enough, BASUsers. It's easy enough to browse the table from SQL Server Management Studio, but I knew that we had a user with a displayname containing "administrator", so a simple query would suffice;

This gave me the essential information I needed - the UserId (75 in this case) - but also confirmed that the LoginName and LoginPassword were set to NULL.

First, I needed to set a LoginName and LoginPassword. A quick BingGoogle gave me the password hash I needed to set a LoginPassword of blackberry.

So now having set this, I just needed to establish what tells BAS that it's an AD or BAS account. I found that this was controlled by the BASUserAuthenticators table. An AuthenticatorTypeId of 0, and AuthenticationInstanceId of 0, indicates a BAS login (AuthenticatorTypeId 1 and AuthenticationInstanceId 111, respectively, indicate AD).

Now, I was essentially set to login. But I wanted to make sure my newly converted user had complete control over the BAS. Normal admin users aren't given permission to add new administrators. I established that the BASUserRoles table controlled this ability - a RoleId of 1, rather than the 2 that was already set, would give me the control needed.

I now finally had complete admin rights, and a viable user to login. Testing showed that this was fully operational, so one big problem down.

Next, I was able to review the issue with AD. Having access to the BAS meant that I could confirm the correct settings for AD were set, and I could do more effective testing using the logfiles to guide me.

Reviewing the [SERVER]_BBAS_AS_01_[DATE]_[SEQUENCE].TXT files, which are located under %ProgramFiles%\Research In Motion\Blackberry Enterprise Server\Logs\[Date]\, I could establish that we were getting entries like;

Not the most meaningful error - telling us that it had a problem doing an LDAP search. There were no indicative log entries elsewhere, such as on the LDAP server, to support it. Equally, there didn't seem to be a match among the myriad of possible causes that RIM had posted on their knowledge base. It wasn't until I traced back earlier that I found this gem;

Now it started to make sense. The LDAP search performed by BAS needs the rootDSE attributes to perform a search against AD. Even if you want to configure this manually as the search base, it needs to auto discover it initially in order to verify it. In this case, it was picking up another server - one which, in fact, had been recently decomissioned by my staff without being cleaned up properly.

In short, BES was taking the first server it found from a DNS lookup for the Global Catalog, failing to connect, and not attempting to use any others. Here was the root cause of the problem.

It's a relatively simple task to kill failed or decommissioned domain controllers from AD - I've done it plenty of times. A quick run through the process and a restart of the BAS services later, and I had fixed the issue altogether.

I could then go through the MDS integration service removal, SP3 upgrade, and the following maintenance release. I even enabled single sign on. I also took the opportunity to prepare the server for OCS 2007 R2/Lync clients via the Blackberry Collaboration Service, now that RIM have finally introduced support for them. All nice and smooth sailing from there on in.

But as with so many features of Blackberry, I had to go through the pain first to get to the good stuff. It seems ironic that Blackberry has the superior email experience over other smartphone platforms - and even has a brilliant solution for unlimited data over the BES connection - but involves so much more hassle. But hopefully what I've found will be of assistance to others encountering similar issues!

I sing my tweets in the shower: @OhCrap