Userlist access is based off a view access level only. Best I can suggest is maybe using CB Auto Actions to redirect away from a userlist they don't have access to. You should be able to act on onBeforeDisplayUsersList to do this. Its variables are as follows.
Code:
$_PLUGINS->trigger( 'onBeforeDisplayUsersList', array( &$row, &$users, &$columns, &$fields, &$input, $row->listid, &$search, &$Itemid, 1 ) );
To condition against a specific userlist you'd condition against [var6] which contains the userlist id. Aside from that I recommend considering adding a usergroup based off field value using CB Auto Actions so you can do ACL based checks instead of just field based.