Last night we managed to close a PMR that had been open for a few weeks on a strange issue with Community Widgets.
Rewind 3 or 4 weeks – One our amazing customers made some security changes which in turn required some reconfiguration of Connections. We switched to a secure LDAP connection, reconfigured WebSphere global security and changed the LDAP realm to fix and LTPA issue we had seen and also just for good measure switched the WebSphere Admin user to be an LDAP user – a few changes to implement at once, but not rocket science.
The only bit of reconfiguration that needs a bit of work is the changing of the connections admin user – previously we had just used the standard wasadmin user, Connections application security was switched (using the community scripts – thank you Christoph), changing references to administrative credentials was completed and the lovely task of updating the messaging bus was completed.
Everything worked exactly as expected except for 2 tiny issues – Adding the blogs and surveys widgets inside a community thew a nasty error.
The errors in the logs were not actually helping much – just saying the new admin user did not have the authority, which didn’t really make much sense as all the other widgets were adding to communities without issue and the blogs app worked perfectly outside of Communities – also any exiting blog widgets were also working as expected.
After some very intensive investigation from IBM Support, especially Justin Cornell who ploughed through much logs, traces, fiddler records and a live debugging session where he saw the problem live – we were still no closer to getting to the bottom of the issue. Justin then brought up the issue in a team meeting and another college mentioned that he had seen something similar.
Waltz and Sonata were recognizing the new admin user but something else was still not quite right – the suggestion was to remove and remap the admin user for the applications – forcing the remapping of the security and flushing any old info relating to the old admin user. We also re-synced the new admin user against all the connections applications just to be sure using the Application.MemberService.syncMemberExtIdByEmail(“emailhere“) command. After this was done we still had the same problem. By now this was becoming extremely frustrating – it made no sense, it should just work!
It was a public holiday in the UK on Monday so I didn’t notice the mail Justin had sent me straight away – he suggested that before this goes up yet another level of the support chain that we remap the widget admins again, just to make sure that it had actually added the new admin user correctly. After quickly jumping on to the Deployment manager machine and remapping the widget admin user for the blogs application and restarting the blogs app we attempted to add the widget to a community again – AND IT WORKED !! Re-mapping the admin user for the Forms Experience Builder (surveys) app and restarted that application also resolved the issue with that widget too.
It appears that re-mapping these individually and saving and restarting flushed out any cached or old admin information, when it was done as part of a script or mass change it didn’t seem to clear out the old info.
This is one of the most frustrating and strangest issues we have seen with Connections – as theoretically nothing has changed. The new admin user was mapped correctly previously – but re-mapping it fixed it.
Thanks IBM Support and the very tenacious Justin who was just as frustrated and determined to get to the bottom of the issue.