According to the Lightweight CMP profile:
recipient REQUIRED
-- SHOULD be the name of the intended recipient; otherwise, the
-- NULL-DN MUST be used
-- In the first message of a PKI management operation: SHOULD be
-- the subject DN of the CA the PKI management operation is
-- requested from
-- In all other messages: SHOULD contain the value of the sender
-- field of the previous message in the same PKI management
-- operation
So also the client should mirror whatever it received from the server.
When doing so (which is already done by the server side),
not only the directoryName choice of type X509_NAME, but any GENERAL_NAME should be supported.