Facing issue with device discovery. Got 2 sites that has been discovered without any issues. But when tried to add another one got "cli validation failure". There is no problem with routing nor ACL - snmp part all works fine. Have checked the logs from TACACS+ server, and there was no attempt to log in from APIC-EM server. Logged in to linix console and tried with account used by APIC-EM and it worked. Captured tcpdump on physical interface and run discovery process. Noticed, that there was no ssh attempt.
APIC-EM Version 188.8.131.52.
10.10.10.10 is switch that does not work, 10.10.20.20 is APIC-EM server.
$ sudo tcpdump -n -i eth0 host 10.10.10.10
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:51:13.749932 IP 10.10.20.20.57388 > 10.10.10.10.161: F=apr U=apicem [!scoped PDU]5e_8e_9b_a2_d4_1f_2b_28_01_46_d2_2d_e3_84_8 0_6b_60_1d_b8_2f_77_64_20_a6_c2_6c_7b_e9_6f_ff_ce_e4_b4_c5_db_f0_b6_2f_08_6c_29_61_ce_27_fa_d4_bb_10_1f_97_b0_e8_c5_c1_e0_4a_2a_c4_ bc_f1_6e_4f_ed_a6_6a
tried 2 different switch boxes and different model, but have the same result. Also rebooted APIC-EM server.
Check for the SNMP step. This could be an issue in SNMP v3 configuration. Check where you were seeing SNMP success and CLI failure message.
Tried to discovery using Discovery App. Devices with all green has both TACACS+ and correct SNMP settings. Devices with failed SNMP does not have SNMP but has TACACS+ configured. Device marked in red box has TACACS+ enabled (CLI access works, tried from apic-em server with credentials used by apic-em server) but has SNMPv3 not correctly configured.
SNMP on "red" box switch is:
p access-list standard ACL
snmp-server user apicem_user apic-group v3 auth md5 AUTH priv aes 128 PRIV
snmp-server host 184.108.40.206 traps version 3 priv apicem_user
snmp-server group apic-group v3 priv access ACL
it works like a charm.