Design Recommendation for APIC-EM Network

Building out a proof of concept for zero touch provisioning and would like to focus on APIC-EM.  We have a temp server that we are using for a single APIC-EM VM.  Its an ESXi 5.5 host. Along with that we have a MS DHCP server configured to provide option 43.  We are running 1.3.1.9 as a standalone server.

 Eth0 of the host goes to an external network with services (NTP etc.), Eth1 goes to the private "Provisioning Network".  The APIC-EM VM is similarly configured so that it is accessible for scripting etc on its first interface and has access to the private network on the second interface.

 Everything seems to be working as expected except that we can't seem to successfully use the PnP application.  I've tried disabling the "external interface" so that only the private network is available thinking the network interfaces were more 'HA' /NIC Teaming but that did not make any difference.

Tried a variety of devices which all meet the minimum requirements for hardware and software

2901 ISR (Gen2)

2960S Switch

3650 Switch

 Tried projects as well as just seeing they will be "discovered" without success.

 The devices do get an IP address and they start the AutoInstall process and the APIC-EM never recognizes them, pre-provisioined or not.

 Here is the log from the 2960S

 ```

*Mar  1 00:02:26.098: %SYS-5-RESTART: System restarted --

Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(58)SE2, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2011 by Cisco Systems, Inc.

Compiled Thu 21-Jul-11 02:22 by prod_rel_team

*Mar  1 00:02:27.634: %LINK-3-UPDOWN: Interface FastEthernet0, changed state to down

to up

*Mar  1 00:02:29.107: %USB_CONSOLE-6-MEDIA_RJ45: Console media-type is RJ45.

*Mar  1 00:02:30.444: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up

*Mar  1 00:02:58.446: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up

*Mar  1 00:03:09.215: AUTOINSTALL: Vlan1 is assigned 192.0.2.102 got vend id vend spec. info ret: succeed

*Mar  1 00:03:19.224: AUTOINSTALL: Obtain siaddr 192.0.2.100 (as config server) <--.100 is the DHCP server

%Error opening tftp://192.0.2.100/network-confg (Timed out)

%Error opening tftp://192.0.2.100/cisconet.cfg (Timed out)

%Error opening tftp://192.0.2.100/router-confg (Timed out)

%Error opening tftp://192.0.2.100/ciscortr.cfg (Timed out)

%Error opening tftp://192.0.2.100/network-confg (Timed out)

%Error opening tftp://192.0.2.100/cisconet.cfg (Timed out)

%Error opening tftp://192.0.2.100/router-confg (Timed out)

Any suggestions on where to look for issues would be very welcome!

First off what application are you using- PnP or IWAN.

From what you posted it looks to me like post provisioning, the device does not have a route back to 192.0.2.100. Is that APIC-EM or a different tftp server.. If different, did you verify that tftp is running and that no ACL or firewall rules drop the traffic.

Some troubleshooting tips (I will probably blogs these if there is interest).

1) Connect to the PnP switch and determine the following:

• the switch has a valid IP address
• the switch can ping the controller (by the appropriate IP address -- internal vs external)
• that the PnP profile has been established on the switch (which discovery mechanism are you using) - "show run | inc pnp" should show a "pnp profile pnp-zero-touch"
• If there is an issue with discovery, you can create the pap-profile manually to test the rest of the process..you can then use the "debug pnp all" command to get great insight into what is going on

 pnp profile manual-test

transport http ipv4 x.x.x.x port 80

On the 2960S switch, even though it running the minimum code it does not look like pnp is on there, so that may explain that.

On the ISR Gen2 2901, also running > recommended ios, it has ccp ans so Nick mentioned that that will cause it to fail.  I'm deleting and retrying.

On the 3650s I may have had an incorrect SN so I'm about to verify and try that again.

Switch Ports Model              SW Version            SW Image               

------ ----- -----              ----------            ----------             

*    1 52    WS-C2960S-48LPS-L  12.2(58)SE2           C2960S-UNIVERSALK9-M

 BTW:

• the switch has a valid IP address - Yes it does so DHCP is working TBD on the Opt 43
• the switch can ping the controller (by the appropriate IP address -- internal vs external)  It can on the internal
• I will check these in a few:

• that the PnP profile has been established on the switch (which discovery mechanism are you using) - "show run | inc pnp" should show a "pnp profile pnp-zero-touch"
• If there is an issue with discovery, you can create the pap-profile manually to test the rest of the process

On the Gen2 ISR 2901:

 There is no pnp bootstrap and its trying to use DNS to go to the cloud server I believe.

 *Dec  8 22:10:29.439: %SYS-5-RESTART: System restarted --

Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.4(3)M1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2014 by Cisco Systems, Inc.

Compiled Sat 25-Oct-14 03:34 by prod_rel_team

*Dec  8 22:10:30.123: %SYS-6-BOOTTIME: Time taken to reboot after reload =  670 seconds

*Dec  8 22:10:30.735: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

*Dec  8 22:10:30.735: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

*Dec  8 22:10:30.735: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF

*Dec  8 22:10:30.735: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF

%Error opening tftp://192.0.2.100/router-confg (Timed out)

*Dec  8 22:10:49.551: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://devicehelper.cisco.com/pnp/HELLO

Router>en

Router#sh run | inc pnp

Router#

%Error opening tftp://192.0.2.100/ciscortr.cfg (Timed out)

*Dec  8 22:11:27.551: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server http://devicehelper.cisco.com/pnp/HELLO

%Error opening tftp://255.255.255.255/network-confg (Timed out)

 What discovery mechanism are you attempting to use for your testing?  I just read your earlier post.... looks like option 43.

 That should create the pnp profile.

 "show  pnp trace" is another useful command too.

PnP protocol will be using "time-pnp.cisco.com" and/or "pool.ntp.org"  for time sync.   Some lab networks block public NTP access, for such DNS mappings to local NTP server is needed.

Does this apply to your case CLaudia? If yes, can you do the needful in your setup

If thetwork is totally isolated.  If sending the NTP server in the DHCP offer does not work what would you suggest in the isolated lab.  dummy DNS A records.

ip host time-pnp.cisco.com <IP of your choice>

Similar entry for pool.ntp.org I believe.

This way, you can resolve name queries for time-pnp.cisco.com and/or pool.ntp.org with an IP of your choice

option 43 needs to be

an ascii string - "5A1N;B2;K4;Iw.x.y.z.;J80"

where "w.x.y.z" is the IP address of your controller.

root cause seems to be the Microsoft DHCP server.

Took the 2901 ISR router (which worked if configured the pnp profile manually) and turned it into a DHCP server with the following configuration:

 ip dhcp excluded-address 192.168.2.1 192.168.2.100

!

ip dhcp pool PNP_Pool

network 192.168.2.0 255.255.255.0

default-router 192.168.2.1

option 43 ascii "5A1D;B2;K4;I192.0.2.2;J80"

 Has anyone run across instructions on how to set up the Option 43 on an MS DHCP server for PnP.

 Test ISR G2 2901

Works with manually configured pnp profile after removing ccp files/directory and now in use as DHCP server so no longer a test client

 Test 2960S  WS-C2960S-48LPS-L  12.2(58)SE2

- Still not working even though it is running the right code and other devices are now getting the right info from the new IOS DHCP server

 Test 3650 #2 "Out of the Box"

Now working in that on boot up I see it in the "Unplanned Devices" section

Note: Had to up the SNMP timeout to 30 seconds before it started working (after I fixed DHCP)

 Test 3650 #1 "Out of the Box"

This one is interesting...I had a pre-provisioned record for the 2960S and I initially mistakenly put in the Serial Number of 3650#1.  I then went back and fixed it but only after the APIC-EM PnP saw it and started talking to it.  I disconnected once i saw that since I didn't want to load a 2960 image onto a 3650.  So even now, with the correct SN of the 2960S when I bring up this 3650#1 APIC-EM still sees it as the 2960 and tries to provision it per the "state" I defined in the pre-provisioned tool. . That is, even though I now have a pre-privision record for a 2960S with the right SN when I bring up the 3650 whose SN I had mistakenly put in it tries to provision that 3650 per the 2960S rules I defined.

 the APIC dashboard is "stuck"  in "Deploying Image" and I can't delete that record from the APIC-EM.

 Note: after it failed,  was able to remove the entire project.

1) for the 2960S, can you manually create a pnp profile?  does the option 43 get received by the device?

Have upgraded it to 15 code and now the pnp commands are there (no debug) but when I manually configure it just keeps trying the APIC-EM but never gets a response.  It does get an IP from the IOS DHCP server so I'm assuming it gets DHCP option 43 as the other devices seem to be now but I've not done any packet captures.

2) 3650#2.  SNMP should not be required until after the PnP process is complete (the controller adds it to the inventory).  Are you using VLAN 1 for management, and what version of code on the 3650.

using Vlan1 - trying to keep it very simple.

Not sure why changing the SNMP to 30sec should have made a difference. Possible I didn't wait long enough....I tried it with the default value and it never came up in APIC-EM. Once I updated that it seemed to come right up

3) 3650#1.  A rule binds a serial number to an image/config file.  if the deployment fails you need to wait for an error timeout (16mins for config).  Once this occurs you can remove the rule and start again.

that is consistent with what is observerd

Switch(config)#

Switch(config)#pnp profile manual-test

Switch(config-pnp-init)#transport http ipv4 192.0.2.2 port 80

Switch(config-pnp-init)#end

Switch#d

Mar 30 01:35:35.988: %SYS-5-CONFIG_I: Configured from console by consolee

% Ambiguous command:  "de"

Switch#debug pnp ?

% Unrecognized command

Switch#debug pnp

              ^

% Invalid input detected at '^' marker.

 Switch#show pnp profile

 Initiator Profile manual-test: 0 open connections: 0 closing connections

  Encap: pnp

  WSSE header is not required. Configured authorization level is 1

  Max message (RX) is 50 Kbytes

  XEP Faults are sent

  Idle timeout infinite

  Keepalive not configured

  Reconnect time 60 seconds

  Primary transport: http to host 192.0.2.2, port 80, URL onplusops/WORK-REQUEST

  Not connected, next reconnect attempt in 41 seconds

  Switch#

 PnP release notes require the following min version of code for 2960S 15.2.2E3, 15.2.3E2, 15.2.4E1

 There is a c2960s-universalk9-mz.152-2.E5a.bin (Oct 2016) that should work well (and have debugging)!

 unfortunately, you have a "bootstrap" issue.  you need to upgrade to 15.2(2)E5a before you can test pnp.

The base apps are discovery/topology/path trace.

 PnP has it's own set of release notes as the PnP feature requires specific IOS support for the PnP agent.  We can make base apps work on most versions of IOS, PnP is quite specific.

 Release Notes for Cisco Network Plug and Play, Release 1.3x - Cisco

glad it is under control.  Good catch on three common issues. CCP is also a common gotcha.

 There are quite a few on different pnp deployment models

For more details - Refer -- My Blog Index