Troubleshooting Device Connectivity

Network Device Connectivity Troubleshooting in rConfig V8

Device connectivity troubleshooting in rConfig V8 addresses the challenge of diagnosing and resolving connection failures that prevent configuration downloads from network devices. Understanding systematic diagnostic procedures enables operations teams to rapidly identify root causes—whether network connectivity, authentication, prompt detection, or protocol-specific issues—and implement effective resolutions.

Organizations can leverage rConfig’s built-in diagnostic tools, debug capabilities, and comprehensive logging to troubleshoot device connectivity efficiently, minimizing downtime and ensuring reliable automated configuration management across diverse network infrastructures.

Jump to Device Debug Section

Tip

This guide assumes devices are already added to rConfig. For initial device setup and configuration, see Device Management. For device prompt configuration, see Device Prompts.

Understanding Device Connectivity Issues

Common Failure Scenarios

Device connectivity failures manifest in several ways:

Blank configuration files: Configuration download completes without errors, but downloaded files contain no content. This typically indicates prompt detection failures where rConfig cannot identify when device output completes.

Timeout errors: Configuration download fails with timeout messages after 60 seconds (default). Indicates rConfig did not receive expected prompt within timeout period, often due to slow device response or incorrect prompt configuration.

Authentication failures: Connection to device fails during login phase with “authentication failed” or “access denied” errors. Caused by incorrect credentials, disabled accounts, or authentication protocol mismatches.

Connection refused: Network connection to device fails immediately, indicating firewall blocking, device unreachable, or service not listening on configured port.

Partial configuration downloads: Some configuration sections download successfully while others fail or are truncated. Often caused by timeout issues, prompt mismatches, or command-specific failures.

Diagnostic Approach

Effective troubleshooting follows systematic layered approach:

Layer 1: Network connectivity - Verify basic IP connectivity, routing, and firewall rules permit communication between rConfig server and device.

Layer 2: Service availability - Confirm SSH/Telnet/SNMP services are running on device and accepting connections on configured ports.

Layer 3: Authentication - Validate credentials are correct, account is active, and authentication method (password, SSH keys) matches device configuration.

Layer 4: Protocol interaction - Verify prompt detection, command execution, and output capture work correctly for device’s specific protocol behavior.

Layer 5: rConfig configuration - Ensure device settings in rConfig (prompts, commands, timeouts) accurately reflect actual device behavior.

Diagnosing issues in sequence from network through rConfig configuration efficiently identifies root cause without skipping fundamental prerequisites.

Basic Connectivity Verification

Network Layer Testing

Before troubleshooting rConfig-specific issues, verify network connectivity from the rConfig server:

Step 1: Ping device (if ICMP permitted):

ping -c 4 192.168.1.1

Successful ping confirms:

• Device is powered on and connected to network
• Routing exists between rConfig server and device
• Firewall permits ICMP (if configured)

Note

ICMP (ping) is not required for rConfig to download configurations. Some networks block ICMP for security reasons. If ping fails but you know the device is reachable, proceed to next step.

Step 2: Test port connectivity:

Test SSH port (22):

telnet 192.168.1.1 22
nc -zv 192.168.1.1 22

Test Telnet port (23):

telnet 192.168.1.1 23
nc -zv 192.168.1.1 23

Test custom SSH port (e.g., 2222):

nc -zv 192.168.1.1 2222

Successful port test displays “Connection succeeded” or prompt from device. Connection refused indicates:

• Service not running on device
• Firewall blocking port
• Wrong port configured
• Device not listening on tested interface

Step 3: Verify routing and traceroute:

traceroute 192.168.1.1
mtr 192.168.1.1

Trace output identifies where packets stop if connectivity fails. Look for:

• Timeouts at specific hops indicating routing issues
• Firewall ACLs blocking traffic
• Asymmetric routing causing return traffic loss

Resolution for network connectivity failures:

• Verify device management interface has IP address and is administratively up
• Check firewall rules on all devices between rConfig server and target device
• Confirm routing tables include routes to device management networks
• Verify VLANs and VRFs configured correctly for management traffic
• Review access control lists (ACLs) on intermediate devices

Service Availability Testing

After confirming network connectivity, verify the management protocol service is accessible:

SSH availability test:

ssh [email protected]

Successful SSH test displays:

• SSH version banner
• Host key fingerprint
• Password or authentication prompt

SSH failures may indicate:

• SSH service not enabled on device
• SSH version mismatch (SSHv1 vs SSHv2)
• Host key issues
• Cipher or algorithm negotiation failures

Telnet availability test:

telnet 192.168.1.1

Successful Telnet test displays:

• Login prompt
• Banner or message of the day

Telnet failures indicate:

• Telnet service disabled
• VTY lines not configured
• Access class restrictions

SNMP availability test (for SNMP-enabled devices):

snmpwalk -v 2c -c public 192.168.1.1 system

Successful SNMP test returns system information. Failures indicate:

• SNMP not enabled
• Community string incorrect
• SNMP ACLs blocking access
• Wrong SNMP version configured

Authentication Verification

Testing Credentials Interactively

The most common cause of connectivity failures is incorrect credentials. Verify credentials work through interactive login:

SSH credential verification:

ssh [email protected]

Enter password when prompted. Successful login confirms:

• Username is correct
• Password is correct
• Account is not disabled or locked
• Authentication method is compatible

Telnet credential verification:

telnet 192.168.1.1

Enter username and password when prompted. Successful login confirms credentials are valid.

Enable password verification (for devices requiring privilege escalation):

After logging in, attempt privilege escalation:

enable

Enter enable password when prompted. Successful escalation confirms enable password is correct.

Common Authentication Issues

Case sensitivity: Device usernames and passwords are typically case-sensitive. Verify exact case matches configured credentials in rConfig.

Special characters: Passwords containing special characters ($, !, @, #, etc.) may require escaping or cause parsing issues. Test passwords without special characters first to isolate issue.

Account lockouts: Repeated failed login attempts may trigger account lockout policies. Check device logs for lockout events and wait for lockout timer to expire or manually unlock account.

Privilege levels: Some devices require specific privilege levels for configuration access. Ensure credentials provide adequate privileges to execute show commands and retrieve configurations.

Authentication methods: SSH supports multiple authentication methods (password, public key, keyboard-interactive). Ensure device and rConfig are configured for compatible authentication methods.

Password expiration: Expired passwords prevent login. Check device for password expiration policies and update expired passwords.

Credential Configuration in rConfig

After confirming credentials work interactively, verify they are correctly configured in rConfig:

1. Navigate to Device Credentials management interface
2. Locate the credential set assigned to the device
3. Verify username and password match tested credentials exactly (including case)
4. Verify enable password is configured (if device requires privilege escalation)
5. Test credentials using device connection test functionality
6. Review Application Log for authentication failure messages providing specific error details

For credential management details, see Device Credentials.

Device Prompt Troubleshooting

Device prompts are the second most common cause of connectivity issues. Incorrect prompt configuration prevents rConfig from detecting when command output completes, leading to timeouts or blank configurations.

Understanding Prompt Matching

rConfig uses prompts to determine when device command output is complete:

How prompt matching works:

1. rConfig connects to device and authenticates
2. Device displays its command prompt (e.g., Router#)
3. rConfig sends configuration command (e.g., show running-config)
4. Device returns configuration output line by line
5. When device finishes output, it displays prompt again
6. rConfig detects prompt reappearance, considers output complete, moves to next command

Prompt matching failures:

• rConfig waits indefinitely for prompt that never appears (timeout)
• rConfig detects false prompt match in command output, truncating configuration
• rConfig cannot match prompt due to regex syntax errors

Common Prompt Configuration Issues

Using overly generic prompts:

Problem: Prompt configured as # (too generic)

Impact: Device output contains # characters in configuration:

Fortigate-Device # show
#config-version=FG200E-6.2.3-FW-build1066-191218:opmode=1:vdom=0:user=someadmin
#conf_file_ver=22474973413141345234794

rConfig sees first # after command and considers output complete, truncating remainder of configuration.

Resolution: Use complete, specific prompt: Fortigate-Device #

Prompt includes dynamic elements:

Problem: Prompt changes based on mode or configuration:

Router(config)#
Router(config-if)#
Router(config-router)#

Impact: Prompt configured for one mode doesn’t match other modes, causing failures when device changes modes.

Resolution: Use regex to match multiple prompt variations:

Router.*#

This matches Router#, Router(config)#, Router(config-if)#, etc.

Regex syntax errors:

Problem: Prompt contains regex special characters not properly escaped:

Router[1]#

Impact: Square brackets are regex special characters. Without escaping, regex interprets [1] as character class matching single digit, not literal [1].

Resolution: Escape special characters:

Router\[1\]#

Prompt includes whitespace:

Problem: Device prompt includes trailing space not accounted for in configuration:

Router#

(note space after #)

Impact: Configured prompt Router# doesn’t match actual prompt Router# (with trailing space), causing detection failure.

Resolution: Include whitespace in configured prompt or use regex to match optional whitespace:

Router#\s*

Testing and Correcting Prompts

Step 1: Capture actual prompt:

Log in to device interactively and copy the exact prompt including all characters and whitespace:

ssh [email protected]
Router#

Select and copy the prompt text exactly as displayed. Paste into text editor to examine for hidden characters or whitespace.

Step 2: Configure prompt in rConfig:

1. Navigate to device edit page
2. Locate Prompt field
3. Paste exact prompt captured from device
4. For devices with multiple modes, use regex to match all variations

Step 3: Test prompt with debug tool:

Use rConfig CLI debug command (detailed below) to verify prompt detection works correctly. Debug output shows when rConfig detects prompts and how output is captured.

Step 4: Refine if necessary:

If prompt detection still fails:

• Try more specific prompt (include hostname)
• Use regex for flexible matching
• Check for vendor-specific prompt behaviors
• Review device documentation for prompt customization options

For comprehensive prompt configuration guidance, see Device Prompts.

Using rConfig CLI Diagnostic Tools

rConfig provides powerful command-line tools for detailed device connectivity troubleshooting. These tools provide verbose output showing exact protocol interactions, timing, and where failures occur.

Download Device Debug Command

The primary diagnostic tool is the artisan rconfig:download-device command with debug flag:

Basic syntax:

cd /var/www/html/rconfig8/current
php artisan rconfig:download-device <device_id> -d

Examples:

Debug single device (ID 1001):

cd /var/www/html/rconfig8/current
php artisan rconfig:download-device 1001 -d

Debug multiple devices:

php artisan rconfig:download-device 1001 1002 1003 -d

Debug without verbose output (no -d flag):

php artisan rconfig:download-device 1001

Obtaining Device ID

Multiple methods to obtain device ID for debug command:

Method 1: Device list page:

1. Navigate to Devices list
2. Device ID appears in leftmost column of devices table
3. Note the ID for the device to troubleshoot

Method 2: Device detail page:

1. Navigate to specific device detail page
2. Device ID appears in page header
3. Click Copy Debug Command icon to copy complete debug command to clipboard

Method 3: Device URL: Device ID is the number in the URL when viewing device details:

https://rconfig.company.com/devices/1001
                                       ^^^^
                                    Device ID

Interpreting Debug Output

Debug output provides detailed protocol interaction showing exactly what rConfig sends to devices and receives back:

SSH debug output structure:

Connecting to device...
Authentication successful
Detecting prompt...
Prompt detected: Router#
Sending command: show running-config
[command output appears here line by line]
Prompt detected: Router#
Command complete
Sending command: show version
[command output appears here]
Prompt detected: Router#
Command complete
Configuration saved to: /path/to/config/file

Telnet debug output structure:

Connecting to device...
Username: admin
Password: ********
Login successful
Detecting prompt...
Prompt detected: Router>
Sending command: enable
Enable Password: ********
Prompt detected: Router#
Sending command: show running-config
[command output appears here]
Prompt detected: Router#

Key Information in Debug Output

Connection timing: How long each step takes. Long delays indicate slow device response or network latency.

Prompt detection: Shows when prompts are detected and what prompt text matched. If prompt never detected, output shows timeout after 60 seconds.

Command execution: Displays each command sent and full output received. Verify commands execute successfully and return expected output.

Authentication flow: Shows authentication steps including username submission, password submission, and privilege escalation. Authentication failures display specific error messages from device.

Error messages: Any protocol errors, timeouts, or exceptions appear in debug output with detailed context about when and why they occurred.

Common Debug Output Patterns

Timeout pattern:

Connecting to device...
Authentication successful
Detecting prompt...
[60 second pause]
Error: Timeout waiting for prompt

Indicates: Prompt never detected, likely due to incorrect prompt configuration or device not displaying expected prompt.

False prompt match pattern:

Sending command: show running-config
#config-version=...
Prompt detected: #
Command complete

Indicates: rConfig detected false prompt match in command output (the # at start of config line), truncating remainder of configuration.

Authentication failure pattern:

Connecting to device...
Username: admin
Password: ********
Error: Authentication failed
Permission denied

Indicates: Incorrect username, password, or account disabled on device.

Hung connection pattern:

Connecting to device...
Authentication successful
Detecting prompt...
[indefinite hang with no output]

Indicates: Device not responding after login, possibly due to banner prompts, interactive messages, or device expecting additional input.

Debug Command Best Practices

Run debug from rConfig server: Execute debug command directly on rConfig server, not from remote system. This ensures network path and timing match actual configuration download operations.

Debug one device at a time: While multiple device IDs can be specified, debugging single device produces cleaner output easier to analyze. Debug additional devices individually after resolving first device’s issues.

Save debug output: Redirect debug output to file for detailed analysis and sharing with support:

php artisan rconfig:download-device 1001 -d > /tmp/device-1001-debug.txt 2>&1

Compare working and failing devices: If some devices work correctly, debug both working and failing devices to identify differences in protocol interaction, prompt detection, or timing.

Test during different times: Network latency and device load vary throughout day. Test during different time periods to identify performance-related issues.

Advanced Troubleshooting Techniques

Analyzing Timeout Issues

Timeouts occur when rConfig doesn’t receive expected prompt within timeout period (default 60 seconds):

Diagnosis approach:

1. Run debug command and observe where timeout occurs:

   • During connection: Network or service availability issue
   • During authentication: Slow authentication provider (TACACS, RADIUS, LDAP)
   • During prompt detection: Prompt mismatch or device not displaying prompt
   • During command execution: Command takes longer than timeout allows

2. Identify timeout cause from debug output timing:

   • Connection takes 30+ seconds: Network latency or routing issues
   • Authentication takes 20+ seconds: External authentication provider slow
   • Prompt detection never completes: Prompt configuration incorrect
   • Command execution exceeds timeout: Large configuration or slow device

Resolution strategies:

Increase timeout (for legitimately slow devices):

Edit device connection settings to increase timeout from default 60 seconds to 120 or 180 seconds for devices with large configurations or slow processors.

Optimize commands (for command-specific timeouts):

Some commands return excessive output causing timeouts. Replace verbose commands with more specific alternatives:

• Instead of show tech-support, use show running-config
• Instead of show log, use show running-config only
• Remove diagnostic commands from command sets

Fix prompt configuration (for prompt detection timeouts):

If timeout occurs waiting for prompt that never appears, prompt configuration is incorrect. Capture actual prompt from interactive session and update device configuration.

Reduce authentication latency (for authentication timeouts):

If external authentication (TACACS+, RADIUS) is slow:

• Use local device accounts for rConfig instead of centralized authentication
• Configure fallback to local authentication on devices
• Optimize authentication server performance and network connectivity

Handling Stalled Connections

Connections stall when protocol interaction stops without timeout or error:

Symptom: Debug output shows connection progressing normally, then stops producing output without timeout error.

Common causes:

Interactive prompts: Device displays prompt requiring user input:

The system has detected that you are logging in for the first time.
Would you like to enter basic configuration? [yes/no]:

rConfig waits for this prompt to match configured prompt, but device expects different input.

Resolution: Disable initial configuration prompts on device or configure device to suppress them for automated sessions.

Banner or MOTD requiring acknowledgment:

*** WARNING: Unauthorized access prohibited ***
Press any key to continue...

Device expects keypress before displaying prompt, but rConfig cannot detect this requirement.

Resolution: Remove interactive banners from device configuration or configure banners that don’t require acknowledgment.

Privilege escalation prompts: Device requires additional authentication for privileged mode, but credentials not configured:

Password:

Resolution: Configure enable password in device credentials, or use credentials with adequate privilege level not requiring escalation.

Protocol-Specific Issues

Different protocols exhibit different failure modes:

SSH-specific issues:

Host key verification failures: When connecting to device for first time, SSH requires accepting host key. rConfig automatically accepts unknown host keys, but strict host key checking or corrupted known_hosts file can cause failures.

Resolution: Remove device entry from known_hosts file:

ssh-keygen -R 192.168.1.1

Cipher negotiation failures: Some devices support only legacy SSH ciphers not enabled in modern SSH clients.

Resolution: Configure SSH client to allow legacy ciphers, or upgrade device firmware to support modern ciphers. Consult device documentation for supported cipher suites.

Key exchange algorithm incompatibility: Older devices may not support key exchange algorithms in modern SSH clients.

Resolution: Configure SSH to allow legacy key exchange algorithms or upgrade device firmware.

Telnet-specific issues:

Telnet negotiation failures: Telnet protocol negotiation can fail with some device implementations causing connection hangs or garbled output.

Resolution: Telnet is inherently less reliable than SSH. Migrate to SSH whenever possible for better reliability and security.

Character encoding issues: Telnet may misinterpret special characters or control codes in device output.

Resolution: Review command output for special characters causing parsing issues. Modify commands to avoid problematic characters.

Vendor-Specific Behaviors

Different vendors implement management protocols differently:

Cisco IOS:

• Typically reliable SSH and Telnet implementations
• Watch for privilege level issues requiring enable password
• VTY line configuration affects concurrent session limits

Juniper Junos:

• Default CLI is shell-style, may require configuration for operational mode
• Prompts change based on hierarchy level in configuration mode
• Use | display set format for consistent configuration output

Palo Alto Networks:

• API access preferred over SSH for configuration retrieval
• SSH sessions may time out on large configurations
• Commit operations affect configuration download timing

Fortinet FortiGate:

• Configuration output format varies by FortiOS version
• Prompts include spaces and special formatting requiring careful regex
• Some commands require confirmation or interactive input

Arista EOS:

• Generally excellent SSH compatibility
• JSON output format available for some commands
• Prompt customization common, requiring flexible regex patterns

Consult vendor-specific documentation and rConfig’s vendor configuration guides for detailed troubleshooting of vendor-specific issues.

Troubleshooting Workflow

Follow this systematic workflow for efficient device connectivity troubleshooting:

Phase 1: Basic verification (5 minutes)

1. Verify device is reachable (ping, port test)
2. Verify interactive login works with configured credentials
3. Verify enable password works (if required)
4. Confirm device displays expected prompt

Phase 2: rConfig configuration review (5 minutes)

1. Verify device configured with correct IP address and protocol
2. Verify correct credential set assigned to device
3. Verify prompt configuration matches actual device prompt
4. Verify command set appropriate for device vendor/model

Phase 3: Debug command analysis (10 minutes)

1. Run debug command for failing device
2. Identify where connectivity fails (connection, authentication, prompt, command)
3. Note timing information and any error messages
4. Compare with debug output from working device if available

Phase 4: Targeted resolution (variable)

1. Based on Phase 3 findings, apply appropriate resolution:
   • Network: Fix firewall, routing, or service issues
   • Authentication: Correct credentials or privilege levels
   • Prompt: Update prompt configuration to match actual device
   • Timeout: Increase timeout or optimize commands
   • Protocol: Address SSH/Telnet-specific issues

Phase 5: Validation (5 minutes)

1. Run debug command again to verify resolution
2. Execute actual configuration download task
3. Verify downloaded configuration is complete and accurate
4. Monitor subsequent scheduled downloads for continued success

Best Practices

Preventive Measures

Standardize device configurations: Use consistent hostname formats, prompt styles, and authentication mechanisms across device fleet. Standardization simplifies rConfig configuration and reduces troubleshooting complexity.

Document exceptions: When devices require non-standard configurations (custom prompts, extended timeouts, specific command sets), document these exceptions with business justification and configuration details.

Test before production: When adding new device types or vendors to rConfig, test thoroughly in lab environment before deploying to production. Identify and resolve connectivity issues before managing production devices.

Monitor connection success rates: Track configuration download success rates across device inventory. Declining success rates indicate emerging issues requiring investigation before failures become widespread.

Maintain device firmware: Keep device firmware updated to modern versions supporting secure, reliable SSH implementations. Legacy firmware often has protocol bugs causing connectivity issues.

Diagnostic Practices

Establish baselines: For correctly functioning devices, capture debug output as baseline reference. When troubleshooting failures, compare against baseline to identify differences.

Isolate variables: When troubleshooting, change one variable at a time. Changing multiple variables simultaneously makes it difficult to identify which change resolved the issue.

Document resolutions: When resolving connectivity issues, document the symptom, diagnosis, root cause, and resolution. Build internal knowledge base of device-specific issues and solutions.

Engage vendor support: For persistent device-specific issues, engage device vendor technical support. Protocol implementation bugs in device firmware sometimes require vendor intervention or firmware updates.

Leverage community resources: rConfig community forums and vendor communities often have solutions to common connectivity issues. Search for similar problems before extensive troubleshooting.

Operational Practices

Schedule testing during maintenance windows: When troubleshooting production devices, coordinate with network operations to schedule testing during maintenance windows. Debug operations may impact device performance or logging.

Use read-only commands: During troubleshooting, use only read-only commands (show commands). Never send configuration changes while diagnosing connectivity issues.

Implement staged rollout: When applying connectivity fixes to multiple similar devices, implement on small subset first. Validate resolution works consistently before applying to entire device population.

Monitor after changes: After resolving connectivity issues, monitor affected devices for several scheduled download cycles. Ensure resolution is stable and doesn’t cause new issues.

Maintain emergency procedures: Document procedures for rapidly disabling or modifying device configurations if rConfig operations cause unexpected device impact. Include contacts, access methods, and rollback steps.

Escalation to Support

When to Contact rConfig Support

Contact rConfig support ([email protected]) when:

• Connectivity issues persist after following all troubleshooting steps
• Debug output indicates rConfig software issues rather than configuration problems
• Vendor-specific protocol behaviors require rConfig software adaptation
• Multiple devices exhibit same connectivity failure suggesting systemic issue
• Suspected bug in rConfig connection handling or protocol implementation

Information to Provide Support

When opening support tickets for connectivity issues, include:

Environment details:

• rConfig version (display in system dashboard)
• Operating system and version
• PHP version
• Network architecture overview (rConfig location relative to managed devices)

Device details:

• Device vendor, model, and software version
• Management protocol (SSH, Telnet) and port
• Sample device prompt
• Device configuration relevant to management access (SSH, VTY, user accounts)

Diagnostic information:

• Complete debug command output: php artisan rconfig:download-device <id> -d
• Application Log entries related to device connection attempts
• System Log entries with timestamps matching connection failures
• Screenshots of device configuration in rConfig (credentials redacted)

Reproduction steps:

• Detailed steps to reproduce the connectivity failure
• Frequency of failure (every attempt, intermittent, time-of-day pattern)
• Comparison with working devices if available

What you’ve tried:

• Troubleshooting steps already attempted
• Temporary workarounds or partial solutions
• Configuration changes that had no effect

Comprehensive diagnostic information accelerates support resolution by eliminating back-and-forth requests for additional details.

Related Documentation

• Device Management - Adding and configuring devices
• Device Credentials - Managing authentication credentials
• Device Prompts - Configuring device prompts and regex patterns
• Command Sets - Configuring commands for configuration downloads
• Application Log - Reviewing connection attempt logs
• System Logs - Detailed protocol-level logging

Summary

Effective device connectivity troubleshooting follows systematic diagnostic approach progressing from network layer through authentication, prompt detection, and protocol-specific behaviors. Organizations can leverage rConfig’s CLI debug tools, comprehensive logging, and vendor-specific knowledge to rapidly identify and resolve connectivity issues across diverse network device populations.

Key takeaways for successful troubleshooting:

• Verify basics first: Network connectivity and interactive authentication before investigating complex issues
• Use CLI debug tools: The artisan rconfig:download-device -d command provides detailed protocol interaction visibility essential for diagnosis
• Focus on prompts: Prompt configuration is the second most common failure cause after credentials, requiring careful matching of actual device prompts
• Isolate variables: Change one configuration element at a time to identify specific cause of connectivity failures
• Document patterns: Build internal knowledge base of device-specific issues and resolutions for future reference

Mastering device connectivity troubleshooting enables operations teams to maintain reliable automated configuration management, minimize manual intervention, and ensure comprehensive visibility across network infrastructure through consistent, successful device polling and configuration downloads.