deep dive adminp process - admin and infrastructure track at uklug 2012

ADMINP DEEP DIVE

Olaf Boerner, BCC

UKLUG 2012 Cardiff 4.9.2012

UKLUG 2012: AdminP Deep Dive, Olaf Boerner, BCC

Speaker introduction

CEO and founder of BCC in 1996

Working with Lotus Notes since Version 3 in 1993

• focused on Domino infrastructure

• CLP certification since Release 3

I am working

• with large enterprise customers as Senior Architect and

Project Manager

• to optimize Lotus Domino Infrastructure Managements

• with customers to enhance BCC products


AdminP History

AdminP was a major breakthrough in Release 4

Inspired by enterprise customers like Deutsche Bank who

had developed similar Server AddIn tasks for their

administration

• Domino Directory Management

• Central PKI Management with User IDs on Lotus Notes

• Tasks to change fields in databases

• Support Distributed Systems

• Better performance than agents

Continuous improvement in each Domino version


Architecture – Admin4

Admin4 Database

• Replica on each server (automatic deployment) • Storage for Task documents and logs • Users need access right to create documents in admin4.nsf (Notes Client

creates documents with users rights) - Archivar

How does a server know that he has to execute a task

• Check AdminP settings in server document • Check for new task document in admin4.nsf • Checks for its name or Wildcard

How does a server know that he has executed this task

• Keep in Memory • Each server can write a log document • Write a log document as response document to task document

Own Task for housekeeping (Delete Obsolete Change Requests)


Architecture AdminP Server task

AdminP Server Entry in ACL defines AdminP Server for this Database

• Only one AdminP Server for each Database Replica

• Every Server can be AdminP Server

• Define “Administration Server for Databases” (next slide)

AdminP Options

• Do not modify names

• Modify all readers and authors fields

• Modify all names fields -> DO NOT USE for Mailfiles


Architecture AdminP Server task

Domino Directory ACL (SPECIAL)

• AdminP Server Entry defines your Directory Server in

your Domain

• Every adminp tasks changing documents in Domino

Directory is executed on that server

• Changes must be replicated !

• Do not change this if you have “open” adminp

request documents in admin4 ! DR procedure needs define how to handle AdminP Server of DD

• Using cluster member is not a good idea


AdminP Task execution & replication

Server which performs AdminP tasks :

• AdminServer for Domino Directory

• Users Homeserver

• AdminP Server of each Database -> Wildcard

Requests

Task documents are distributed with admin4 replication

or direct deposit „replication“ in R8.x


AdminP Task execution & replication

AdminP will do changes just once !

Example

• Change ACL

• Executed at Database AdminP Server

•AdminP Server replicates ACL change to all

replicas

• Change of field entries

• Executed only at Database AdminP Server

•Replicate modified documents to all replicas


How to define “Administration Server for

Databases” Dedicated Server vs. Multi purpose server

• Group Applications to same AdminP Server (AdminP

Hub)

• Define a dedicated AdminP Server for all

Applications Extended Administration servers ?

• Idea: Split up workload to multiple servers

• Requires extended ACL

• Do not do this !!!


AdminRequest Document

One Standard form for all requests

All Fields start with Proxy...

• ProxyAction: contains current actioncode

• ProxyServer: server to perform the action

• ProxyAuthor: who has requested

• ...

Field ProxyAction

• Contains a list of all AdminP Request

• Field contains request numbers


AdminP and Security

AdminP is fully integrated within Domino Security

• ACL – even if AdminP is using local access

• Reader

• Encrypted and signed documents

How does adminp server task know that he has a "real" task document ?

• You might copy and modify a task document

• "misused" server tasks might be dangerous


AdminP Security

Well we have a great PKI built in

AdminP Security relies on Signatures (Private Key)

• AdminP Documents are signed

• Signature will ensure "correct" task documents

• Modification will break signature

• Documents with broken signature will not be

executed !


AdminP Security Check

AdminP Security will check two fields :

• Name to perform the action on: User, Database or

Server

• Action requested by: User or Servername

• Entry must match signature !

• Entry will be checked with ACL and security

settings Error Handling

• “You are not authorized to create new replica

databases on this server.”

• Check settings in server documents and ACL


Sidestep: Why your server ID needs a

password ?

Server ID can

• sign adminp documents

• Agents signed with server id can Create adminp docs

• Server ID can create „fake“ adminp requests Runing ID Vault you need to secure your Domino Server ID • http://www-10.lotus.com/ldd/dominowiki.nsf/dx/securing-your-notes-id-vault-

server

• See Paul Mooneys 2012 AdminBlast Tip #42

http://www-10.lotus.com/ldd/dominowiki.nsf/dx/securing-your-notes-id-vault-server















AdminP and Security

Do never ever modify documents in adminP database !!!

Public key in person/server document must match with

key pair in idfile


AdminP Request Document

How to create AdminP Request Document

• Lotus AdminClient ->> 90%

• Script Agent – AdminP Class

• Server Tasks – AdminP API Manually with Script / API

• Create a sample request

• Do some reengineering (field and values)

• Create a document and set all fields manually

• Sign the document !!! Why do you need this ?

• Automation and batch processing


AdminP Interaction with Notes Client

Some tasks need interaction with Users

Interaction is done due to fields in person documents

and/or creating documents in admin4.nsf

• AdminP changes fields in person document

• Lotus Notes creates „response“ document in

admin4.nsf


AdminP Interaction with Notes Client

Example: Rename User

• Rename User > AdminP changes Field and Public key

in person document

• Lotus Notes Client checks at login for these field and

execute internal procedures inside Lotus Notes Client

• Notes Clients creates

• a „done successfull“ log document in admin4.nsf


AdminP Statistics

AdminP statistics reported to statrep.nsf

Useful to compare servers to see where AdminP activity

is high

Statistics (Sample from Domino Admin Help)

• ACLsModified

• ReaderAuthorModified

• ProfilesModified (mailfile)

• AppointmentsModified

• DirectoryDocumentsDeleted

• DirectoryDocumentsModified


AdminP Monitoring (even more important)

Monitoring • How do you know when your AdminP task has completely

finished? • Remember AdminP usually runs per User, Database etc !!!

Possible Solutions • Create Monitoring Agent (run on server)

• which scans AdminP Request for response documents • Create a report per Object

• Realtime “Scan” using Notes C API • Analyzing Extension Manger Events before/after each

adminp execution • Execute a monitoring action / log etc.

• Use Domino Domain Monitoring


AdminP – Monitoring „Enhanced Log“

Using DEBUG parameter for more useful information

about what AdminP is currently doing

• “DEBUG_ADMINP_REQUEST_PROCESSING=1”

• “DEBUG_ADMINP_REQUEST_PROCESSING=2”

DEBUG Output can be directed to text file

• “DEBUG_OUTFILE=<output file path>

Can be set using „set config“ at server console

Cross Domain AdminP


Cross Domain AdminP

Most AdminP processes are only working inside a domain which is the same admin4.nsf

• Not clear why ! Cross Domin AdminP Tasks are

• Rename User

• Delete User

• Rename Server

• Delete Server

• Create Replica


Cross Domain AdminP: How it works

Architecture

• AdminP will be sent “mails” from the source domain

to the target domain.

• mail will be created at the administration server of

the source domain

• Mail will be delivered directly to the admin4.nsf in

the target domain

• Mail will be processed as a adminp request document Security

• Still relies on PKI and „Signature Validation“


Cross Domain AdminP: How to setup

Domino Directory • Create cross certificate documents. Identify all required

certifiers !

• Create connection document to allow server to connect

to other domain

• Edit Domino Directory Profile: Who are allowed to create

Cross Domain Configuration in admin4.nsf

Admin4 Database • Create Cross Domain Configuration document

• For each domain to import and

• For each domain to export request

Best Practice using AdminP

Or how to deal with Mass Recertification


Project: Mass recertification

Move a number of user to new Org Certifier

• Rename company name

• Recreate Certifier due to security issues

• Integrate a new company

• Split off company Move in hierarchy adminP for name change

• Two approvals for each user

• Response documents might be an issue or

nightmare

• No view update for admin4.nsf


AdminP limitations -> „Renames“

AdminP-Process Expiration

• Enlarge the interval for user to accept the name

change request. Default interval is 21 days. (can be

configured from 14 – 60 days)

• it is strictly necessary that User connects to his

server during that period to start the AdminP

• If a name change request expires, the user will be

reverted to it’s old username! Same behaviour with ID Vault ! Error in Documentation.


AdminP Rename

What happens after User accept rename request ? Notes Client is changing User Name in current ID File ID File get synchronized with ID Vault What happened with old user name

• It is still there !!!

• User ID contains old and new user name

• User can access Database which still have ist old

names in ACL

• Old User name get removed after expiration date

• You will not receive Help Desk Calls before



Manual interaction required

• Admin must confirm execution,

•Move Certifier

•Move Mailfile

• User must "confirm" execution

• Login / Access to server

•No pass thru server or replication access !!! Same behaviour with ID Vault ! Error in Documentation


AdminP – Project Troubleshooting

User currently not working in Lotus Notes (21 – 60 days expiration)

• Avoid absent User: In average 15% - 20% of all users

are not taking part in the daily working process.

• Define a Workaround for absent users with your

Audit Department or write an server tasks (C-API) User is using a wrong ID (public key does not match to AdminP request)



ACL Settings „Modify / Do not modify names“ in each database must be set properly Solution

• New request: “Rename Person in Calendar Entries

and Profiles in Mail File Extended

•Overwrites ACL Setting

•Renaming users in ACLs, Calendar profiles, C&S

documents



AdminP does not handle text fiedls

• Check your application using text field for application logic !

AdminP will not modify profile documents

• Check applications for profile documents using Reader / Author / Names fields

AdminP does not modify wildcards (*USR/BCC)

• Check applications for use of wildcards in Reader / Author / Name fields

• adjusted manually or by agent

The Administration Process can not modify encrypted documents.

• Reader / Author / Names fields in encrypted documents must be adjusted manually by the user, who has encrypted the document.



Default: AdminP scans all documents for reader, author or names fields in a Database Creating an AdminP View in an application with name $AdminP • Only documents which appear in that view will be

considered and processed

• Be careful

AdminP in R8.x is using namelist for Rename • namelist contains all users in that database

• Requires ODS 48

• If AdminP does not find the username in the namelist, it

does not search that database


Mass Recertification – admin4 size issues

Domain size consideration belong to AdminP Size • AdminP Database can grow to enormous sizes

• Number of documents are an issue

• Response documents slow down indexer tasks

Local AdminP Tasks and response documents will be replicated to all admin4 databases • User in Tokio will change ACL of Mailfile

• User Creates ACL Change Request in admin4 on his

current mail server

• Tokio Server will execute AdminP task document and

creates log document

• Documents will replicate to whole domain


Mass Recertification – admin4 size issues

Recertification tasks are part of the ordinary user management in Domino Issues start with mass data / batch requests Admin4.nsf database size • admin4.nsf with 300.000 documents (1,5 – 2 GB size)

will have performance issues

• Replicator tasks requires index update

• Example “Move User in Hierarchy”:

Example „Move User in Hierarchy“ • The request requires 11 requests documents

• 20.000 users

• 50 Servers


Mass Recertification – admin4 size issues Request Log Docs for 50

Server

Server Timing

Move Person's Name in Hierarchy 1 Directory Server Requires administrator approval in Administration Requests database

Initiate Rename in Domino Directory 1 AdminP Server Interval

Rename Person in Domino Directory 1 AdminP Server Interval

Rename in Person Documents 1 AdminP Server Execute once a day requests at

Rename Person in Unread List 50 One per Server Execute once a day requests at

Rename in Access Control List 50 One per Server Interval

Rename in Design Elements 50 One per Server Delayed

Rename Person in Free Time Database 1 Mail Server Immediate

Rename Person in Calendar Entries and

Profiles in Mail File

1 Mail Server Immediate

Rename in Reader / Author Fields 50 One per Server Start Executing On

Start Executing At

Rename Person in Address Book 1 AdminP Server Multi Domain Configuration

Summary per User 207 20.000 User

4.140.000 documents!!!


Mass Recertification – Replication Issues

Replication of names.nsf and admin4.nsf are critical !

• Domino Directory has to replicate before Administration

Database !!!

• Otherwise you may get errors that have to be corrected

manually (i.e. “Rename Person in Domino Directory” fails

because Domino Directory was not updated)

In the replication settings the value to purge documents shall

be set to 7 days on all replicas (not more than 14 days)

Prevent replication to all servers using replication formula:

• select (Form='AdminRequest') |

(ProxyServername=@username)


Mass Recertification – Replication Issues

R8 is using Direct Deposit Feature by default

• Automatically „replicate“ requests

• AdminP requests can be directly deposited to „target

server“ admin4.nsf

• Wildcard requests must be replicated

Also enabled at the client

• Example: Change HTTP Password in Domino Directory

• You need direct access to the target server

Disable with notes.ini parameter ADMINP_DONT_ATTEMPT_DIRECT_DEPOSIT=1


Mass Recertification – Performance

AdminP Tasks carried out on every server • Rename in Reader/Author fields • Rename in Access Control List • Rename in Design Element

Time consuming tasks and will have performance impact Performance Problems while processing the AdminP • Indexing admin4 • Searching fields in Databases

Check AdminP Threads settings • Default 3 • Check if you have idle tasks and CPU time • Increase to 10 Threads max


Best Practice performance issues

Servertask configuration • Change “daily” and “delayed” request execution to “non

working times”.

• Use suspend AdminP at when you see performance

issues on mail servers

• Reduce the amount of (log) documents. A server that

has nothing done during the rename process should not

report. (server task configuration)

Split up threads in Domino 8 (max 10) • ADMINP_IMMEDIATE_THREAD=X

• ADMINP_INTERVAL_THREAD=X


Best Practice performance issues

Change AdminP Task execution • ADMINP_IMMEDIATE_OVERRIDE= x, x, x • ADMINP_INTERVAL_OVERRIDE=X, X, X • ADMINP_DAILY_OVERRIDE=X • ADMINP_DELAYED_OVERRIDE=X

Example (see Admin Help) • Rename in Access Control List

• Interval • Number 1.00

• Rename in Reader/Author Fields • Delayed • Number 20.00

Be careful !!!


Best Practice to avoid performance issues

Kepp Admin4 small

• Plan renaming “waves”

• Do not rename all user at the same day Clean-up Admin4

• reduce the amount of Admin4 documents.

• User that has been renamed successfully should not

stay in admin4.nsf Replication

• Check Use of selective replication formula

• Ensure fast and reliable replication

Questions ?

THANK YOU !

deep dive adminp process - admin and infrastructure track at uklug 2012

Technology

adminp server adminp

adminp history adminp

dedicated adminp server

adminp server of dd

adminp tasks

directory server

server changes

server document check