PSOD (Purple Screen of Death)

Example of a PSOD kernel stack trace screen

A PSOD (Purple Screen of Death) is the VMware ESX version of a Windows BSOD (Blue Screen of Death).  This occurs when the kernel panics and can no longer function.  There most common causes for a PSOD are:

• Hardware failure
• Out of memory
• Hung CPU conditions
• Misbehaving drivers (null pointers, invalid memory access, etc)
• NMI (Non Maskable Interrupts)

When a PSOD occurs, one should collect the following:

1. Screenshot of PSOD kernel stack trace screen (if possible)
2. Support logs from the vm-support command
3. Kernel log(should be included in vm-support, but better safe then sorry)
4. Kernel core dump (only needed if a developer asks for it)

If the cause of the PSOD isn't obvious from the PSOD kernel stack trace screen, then the kernel log is the second best place to look for the cause of a kernel panic. To manually collect the kernel log:

## Kernel log - will output: vmkernel-log.1
# esxcfg-dumppart -L /vmfs/devices/disks/$( \
  esxcfg-dumppart --get-active | awk '{print $1}' )

To manually collect the kernel core dump: (if developer asks for it)

## Kernel core dump - will output: vmkernel-zdump.1
## Note: ESXi 5.x will put kernel dump here:
##     /scratch/core/vmkernel-zdump.*
# esxcfg-dumppart -C -D /vmfs/devices/disks/$( \
  esxcfg-dumppart --get-active | awk '{print $1}' )

For testing purposes, one can manually trigger a PSOD:

# vsish -e set /reliability/crashMe/Panic

VMware ESX Versions

VMware ESXi

VMware ESXi 5.5 GA (Build 1331820) (22 September 2013)

VMware ESXi 5.1 Update 1 (Build 1065491) (25 April 2013)
VMware ESXi 5.1 GA (Build 799733) (10 September 2012)

VMware ESXi 5.0 Update 3 (Build 1311175) (17 October 2013)
VMware ESXi 5.0 Update 2 (Build 914586) (20 December 2012)
VMware ESXi 5.0 Update 1 (Build 623860) (15 March 2012)
VMware ESXi 5.0 GA (Build 469512) (24 August 2011)

VMware ESXi 4.1 Update 3 (Build 800380) (30 August 2012)
VMware ESXi 4.1 Update 2 (Build 502767) (27 October 2011)
VMware ESXi 4.1 Update 1 (Build 348481) (10 February 2011)
VMware ESXi 4.1 GA (Build 260247) (13 July 2010)

VMware ESXi 4.0 Update 4 (Build 504850) (17 November 2011)
VMware ESXi 4.0 Update 3 (Build 398348) (5 May 2011)
VMware ESXi 4.0 Update 2 (Build 261974) (10 June 2010)
VMware ESXi 4.0 Update 1 (Build 208167) (9 December 2009)
VMware ESXi 4.0 GA (Build 164009) (21 May 2009)

VMware ESX

VMware ESX 4.1 Update 3 (Build 800380) (30 August 2012)
VMware ESX 4.1 Update 2 (Build 502767) (27 October 2011)
VMware ESX 4.1 Update 1 (Build 348481) (10 February 2011)
VMware ESX 4.1 GA (Build 260247) (13 July 2010)

VMware ESX 4.0 Update 4 (Build 504850) (17 November 2011)
VMware ESX 4.0 Update 3 (Build 398348) (5 May 2011)
VMware ESX 4.0 Update 2 (Build 261974) (10 June 2010)
VMware ESX 4.0 Update 1 (Build 208167) (19 November 2009)
VMware ESX 4.0 GA (Build 164009) (21 May 2009)

Data Source: VMware ESX - Wikipedia - http://en.wikipedia.org/wiki/VMware_ESX

How To Disable ESXi Password Restrictions

The following instructions will allow you to change the "root" user's password to something simple like "password".

• ESXi 5.0, ESXi 5.1 and ESXi 5.5
• ESXi 4.1
• ESXi 4.0
• ESX 4.1
• ESX 4.0

ESXi 5.0, ESXi 5.1 and ESXi 5.5

1. Edit /etc/pam.d/passwd and add "enforce=none" to the end of the "password   requisite" line:

esx# vi /etc/pam.d/passwd
password  requisite  /lib/security/$ISA/pam_passwdqc.so \
        retry=3 min=8,8,8,7,6 enforce=none

2. Change password:

esx# passwd

ESXi 4.1

1. Edit /etc/pam.d/system-auth and add "enforce=none" to the end of the "password   requisite" line:

esx# vi /etc/pam.d/system-auth
password  requisite  /lib/security/$ISA/pam_passwdqc.so \
        retry=3 min=8,8,8,7,6 enforce=none

Note: You will need to force the write with an exclamation ":wq!".

2. Change password:

esx# passwd

Note: The enforce setting will be lost on the next reboot, so change your password now.

ESXi 4.0

1. Edit /etc/pam.d/common-password and add "enforce=none" to the end of the "password   requisite" line:

esx# vi /etc/pam.d/common-password
password  requisite  /lib/security/$ISA/pam_passwdqc.so \
        retry=3 min=8,8,8,7,6 enforce=none

2. Change password:

esx# passwd

ESX 4.1

1. Edit /etc/pam.d/system-auth and add "enforce=none" to the end of the "password   required" line:

esx# vi /etc/pam.d/system-auth
password  required  /lib/security/$ISA/pam_passwdqc.so \
        min=8,8,8,7,6 similar=deny match=0 enforce=none

2. Change password:

esx# passwd

ESX 4.0

1. Edit /etc/pam.d/system-auth-generic and add "enforce=none" to the end of the "password   required" line:

esx# vi /etc/pam.d/system-auth-generic
password  required  /lib/security/$ISA/pam_passwdqc.so \
        min=8,8,8,7,6 similar=deny match=0 enforce=none

2. Change password:

esx# passwd

How To Enable ESXi SSH Access

The following instructions will allow you to enable root ssh access to an ESXi server:

• ESXi 5.0 and ESXi 5.1
• ESXi 4.1
• ESXi 4.0
• ESX 4.1 and ESX 4.0

ESXi 5.0 and ESXi 5.1

1. From the DCUI (Direct Console User Interface), press "[F2]" and login.

2. Navigate down to "Troubleshooting Options", and press enter.

3. Navigate down to "Enable SSH" and press enter.

4. To disable the configuration warning "Configuration Issues: SSH for the host has been enabled" in the vSphere Client, change the following:

• Configuration -> Software (section) -> Advanced Settings -> UserVars -> SuppressShellWarning = 0

ESXi 4.1

1. From the DCUI (Direct Console User Interface), press "[F2]" and login.

2. Navigate down to "Troubleshooting Options", and press enter.

3. Navigate down to "Enable Remote Tech Support (SSH)" and press enter.

ESXi 4.0

1. From the DCUI (Direct Console User Interface), switch to the hidden shell login screen with "[Alt]+[F1]".

2.  Type in "unsupported" (the characters will not be echoed back) and press enter.  Now enter in the root user's password and press enter.

3. Edit /etc/inetd.conf and uncomment the "ssh stream tcp" line:

esx# vi /etc/inetd.conf
ssh  stream tcp nowait root /sbin/dropbearmulti dropbear  ...

4. Reboot the server:

esx# reboot

ESX 4.1 and ESX 4.0

1. From the terminal screen, switch to the shell login screen with "[Alt]+[F1]" and login as root.

2. Edit /etc/ssh/sshd_config and change the 'PermitRootLogin' line to allow root login:

esx# vi /etc/ssh/sshd_config
PermitRootLogin yes

3. Restart the SSH service:

esx# service sshd restart

How To Manage Kernel Module Load Settings on ESXi

The following are instructions for managing kernel modules (drivers) on ESX(i).

List all loaded kernel modules

# vmkload_mod -l    # --list

Show kernel module information

# vmkload_mod -s some_module    # --showinfo

Note: This works in the same manner as modinfo on Linux

List module load parameters for on boot

# esxcfg-module -g some_module    # --get-options

Set module load parameters for on boot

# esxcfg-module -s 'parameter_a=1 parameter_b=test' some_module

Note: This does not affect parameters on manual module load

Clear module load parameters for on boot

# esxcfg-module -s '' some_module    # --set-options

Disable module on boot

# esxcfg-module -d some_module    # --disable

## write boot config, extra step for ESX 4.x only 
# esxcfg-boot -b

Enable module on boot

# esxcfg-module -e some_module    # --enable

NOTE: On ESXi 5.1 you will need to re-enable the module to even manually load the driver.

Manually load module

# vmkload_mod some_module
# vmkload_mod some_module parameter_a=1 parameter_b=test

Note: This works in the same manner as modprobe/insmod on Linux

Manually unload module

# vmkload_mod -u some_module    # --unload

Note: This works in the same manner as modprobe/rmmod on Linux

How To Reset ESXi Trial License

WARNING: This is for education/informational testing/development purposes only, and should not be used on a production server.

WARNING: This trick will only work with an ESX(i) stand alone server.  It will not work if the ESX(i) server is connected to a vCenter Server, as the vCenter Server knows better than to let you do this.  (you can always remove and readd the ESX(i) server to vCenter.)

To reset your ESX 4.x, ESXi 4.x and ESXi 5.x 60 day evaluation license:

1. Login to the TSM through SSH or Shell
2. Remove the following two files:
1. /etc/vmware/vmware.lic
2. /etc/vmware/license.cfg
3. Reboot server

If your ESX server is connected to a vCenter server, please remove the ESX server first.  Once the steps above are completed, you can add it back to the vCenter server.

Sample commands:

rm -f /etc/vmware/vmware.lic /etc/vmware/license.cfg
reboot

For ESXi 5.1 and ESXi 5.5, you may need to continually remove the license files as the server reboots for this to work.  The following should do this quite nicely:

rm -f /etc/vmware/vmware.lic /etc/vmware/license.cfg
reboot ; while true ; do
    rm -f /etc/vmware/vmware.lic /etc/vmware/license.cfg
done

---

An alternative shows that restarting the services should works just as well as rebooting the server:

# For ESXi 5.0
rm -f /etc/vmware/vmware.lic /etc/vmware/license.cfg
services.sh restart

# For ESXi 5.1
rm -r /etc/vmware/license.cfg
cp /etc/vmware/.#license.cfg /etc/vmware/license.cfg
/etc/init.d/vpxa restart

The alternative also shows a method for resetting the trial license while connected to vCenter server.  I still think removing and re-adding the ESXi server is cleaner.

How To Setup ESXi SSH Keys for Passwordless Access

Generate Your Keys

Linux

From the remote Linux server:

ssh-keygen

Note: Make sure to leave the password empty, or you will defeat this whole passwordless exercise.

This will generate ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub.  You can setup your default identity with:

cp ~/.ssh/id_rsa.pub ~/.ssh/identity.pub

Next we will use a helpful tool to copy our public key to the appropriate location on the target server.  The "ssh-copy-id" tool will copy the public key to ~/.ssh/authorized_keys and set the appropriate permissions for us.  This can be done manually, if needed.


The general method to copy the public key to the target ESX server's authorized list is using ssh-copy-id:

ssh-copy-id root@[SERVER]

If you don't have the default identity.pub setup, you can specify your public key with:

ssh-copy-id -i ~/.ssh/id_rsa.pub root@[SERVER]

WARNING: Make sure to include 'root@' or it will try to copy to the same user name as you are currently logged in as.

Note, each version of ESX/ESXi has a few adjustments to make it work.

Windows

For Windows servers you will need to use PUTTY and PUTTYgen.  Within PUTTYgen, simply click "Generate", move your mouse a lot over the blank area, and wait for the keys to be generated.

The text within the "Public key for pasting into OpenSSH authorized_key file" area is the key that needs to go into the authorized_keys file on your server.  Click the "Save private key" button and save this private_key.ppk file to a secure location.  This is the key you will load into Putty to connect to the server.

 

ESXi 5.x

ESXi 5.x looses the keys on reboot unless you take a few extra steps.

1. Copy public key:

linux# ssh-copy-id -i ~/.ssh/id_rsa.pub root@[SERVER]

2. On the ESXi server:

esx# cp /.ssh/authorized_keys  /etc/ssh/keys-root/authorized_keys

3. Reboot ESXi server, so the script is saved to the boot bank. (Do not power cycle)

esx# reboot

4. Done. Verify passwordless access:

linux# ssh root@[SERVER] 

ESXi 4.x

ESXi 4.x looses the keys on reboot unless you take a few extra steps.

1. Copy public key:

linux# ssh-copy-id -i ~/.ssh/id_rsa.pub root@[SERVER]

2. On the ESXi server:

esx# cp -r /.ssh /scratch

3. On the ESXi server, add the following to /etc/rc.local:

esx# cp -r /scratch/.ssh /

3. Reboot ESXi server, so the script is saved to the boot bank. (Do not power cycle)

esx# reboot

4. Done. Verify passwordless access:

linux# ssh root@[SERVER]

ESX 4.x

1. Copy public key:

linux# ssh-copy-id -i ~/.ssh/id_rsa.pub root@[SERVER]

2. Done. Verify passwordless access:

linux# ssh root@[SERVER]

How To VMware VMDirectPath (PCI Passthrough)

How To setup VMware ESXi VMDirectPath (aka PCI Passthrough)

VMware's PCI Passthrough solution is by far the best I have used.  The other virtualization platforms (eg. Microsoft Hyper-V, Xen, Citrix XenServer, Oracle VM, KVM, etc) provide little, or no, PCI Passthrough support.  I have found that ESXi 5.0 had the best PCI Passthrough support, so stick with that version.  (ESXi 5.1 was not quite as stable in this area)

VMDirectPath I/O Summary

VMDirectPath I/O, aka PCI Passthrough, is a method of giving a VM direct access to a PCI device.
This VMware KB article describes configuring a device for PCI Passthrough:

• Configuring VMDirectPath I/O pass-through devices on an ESX host - http://kb.vmware.com/kb/1010789

As there is direct access to the PCI device, the only setup required is to create a VM and then assign the desired PCI device to the VM, using the direction in the KB article. No driver need be installed on the ESX host, only within the VM's guest OS.  To configure the VM follow the standard procedures for the target operating system, as though it were a physical system.

Requirements

The server's CPU must support and and BIOS must have enabled: "Intel® Virtualization Technology for Directed I/O (VT-d)" or "AMD I/O Virtualization Technology (IOMMU)"

Limitations

• Not all ESX servers have full support for VMDirectPath, and on some servers not all PCIe slots do either. (especially the desktop class systems)
• The server CPU/BIOS must support Intel VT-d or AMD IOMMU.
• Limited to a maximum of 8 pass through devices powered on simultaneously, per host
• Limited to passing through a maximum of 6 devices per VM

WARNING: When updating firmware through VMDirectPath, when fio-update-iodrive requests a reboot, this means a reboot of the physical hardware, not just the VM.

VMDirectPath Procedure

Note: These same procedures work for ESX/ESXi 4.x and ESXi 5.x.

1. Export out PCI devices through "Advanced Configuration".  The names may not be what you would recognize, so it is best to match on "Device ID" and "Vendor ID".

2. Reboot ESX/ESXi server.

3. Add PCI Device to VM's configuration.

4. Set Memory Reservation.  PCI Passthrough requires all allocated memory to be reserved for the VM.  Set the VM's Memory Reservation to match the VM's Memory Allocation.  Simply drag the Memory Reservation slider all the way to the right to the orange triangle.  This will also need to be done anytime the memory, for the VM, is adjusted in the future.

5. Treat device, in VM, as you would a physical server (install the OS specific driver within the target VM).

VMworld 2013 Conference Highlights

CEO General Session

Speaker: VMware’s CEO Patt Galsinger

“The Software Defined Data Center”

22,000 attendees this year to VMworld 2013!

VMware’s 3 Imperatives:

1. Virtualize all the things
2. IT management gives way to automation
3. Hybrid cloud will be ubiquitous

4 Pillars of a Software Defined Data Center:

1. Compute
2. Storage
3. Network
4. Management and Automation

Product Announcements:

• Compute:
  • ESXi 5.5 (vSphere 5.5)
• Storage:
  
  • Virtual SAN (vSAN)
  • Virtual Volumes
  • vFlash Read Cache
  • Virsto
• Network:
  
  • VMware NSX (major topic for VMworld this year)
• Management/Automation:
  
  • vCloud Hybrid Services

New in ESXi 5.5:

• 2x (cores and other limits raised)
• App Aware H.A.
• Big Data Extensions
• Hadoop

Software Defined Storage:

• Policy driven control plane
• Virtual data plane
• Application centric data services

New with vSAN:

• Simplifies Storage
• distributed software datastore
• Resilient
• Policy driven
• Elastic
• High performance
• * Release in 1H 2014 *

New in Networking with NSX:

• Software Defined Networking (SDN)
• What ESX was to hardware, NSX is to networking
• Network virtualization platform

Summary: VMware is hell bent on removing all IT admins (Storage/Server/Network/Management) from the data center and replacing them with Virtualization Admins!

VMworld 2014 – Save the Date – August 24^th - 28^th, 2014

COO General Session

Speaker: VMware COO, Carl Eschenbach

ITaaS (IT as a Service)

• Self service
• Transparent Pricing
• Governance
• Automation

All about “having the perception that you have control”

Focus: East and West Fabric

• There is a major shift of traffic volume in the data center from North-South (client-server) to East-West (server-server).

New Product Summary:

• vSAN – Simplifies storage, distributed software datastore
• Virtual Vol – VVOLS – VM centric, snapshot VMDK from storage
• Virstro – Appliance that aggregates storage pool as NFS data store
• Horizon View – Desktop Virtualization Product (renamed from “VMware View”)

War Room – designated for critical issues

Virtual San

Presenter: Kiran Madnani

• On Beta forms, for vSAN, have had conversations with him before

Virtual Data Plane – pooling hardware like a hypervisor does – “hypervisor based pooling of heterogeneous hardware”

Virtual SAN (vSAN):

• Single datastore called “vsanDatastore” by default, can be renamed.
• Optimized for resources that are a mix of expensive and inexpensive
• Scale out
• Software solution
• Cluster disks and cache
• High performance – flash based distributed read / write cache
• VM centric management
• Automated SLA management
• High resiliency – distributed RAID
• Dynamic capacity scaling
• Rebuilds happen in parallel
• Different sizes of servers – including magnetic disks
• Storage policy specification
  • Capacity, availability, performance
  • Distribution, RAID and SLA are on a per VM level
  • Works with vMotion, storage vMotion, DRS, SRM, etc

Requirements:

• 3 x ESXi 5.5 Hosts
• vCenter 5.5
• 1 x SSD empty
• 1 x HDD empty
• Network: 1 Gbps, 10 Gbps (preferred)
• SAS/SATA controller working in passthru/JBOD/HBA mode

Performance: 80 – 100K IOPS ???  (I have not seen this level of performance from my tests)

Use Cases:

• VDI
• Test/Dev
• Data Center Consolidation
• Big Data
• Private Cloud
• Disaster Recovery

Join the vSAN beta: http://vsanbeta.com

Power CLI Best Practices

Strong PowerCLI community.  Google “PowerCLI Community”

Based off of PowerShell

Works off of objects (can pipe objects too!)

Uses verb-noun construct in plain English

Easy help: get-help

New in 5.5:

• Same: core, image, auto deploy, license, vds, cloud, tenants
• Tagging
• Open-vmconsole - awesome

Best Practices:

• Store results in variables
• Help and Examples
• -ShowWindow switch
• Functions breakdown
• Splatting (put parameters into hash table)
• Make it simple
• Try, catch, throw
• Filter on the left
• Steal from the best
• Don’t use write-host

Check out VMware Flings – free projects that are very useful!

Exchange Messages Between Guest and Host using RPC

Cool stuff, might be useful for vCenter Plugin?

Uses a combination of Guest API and VMware Tools

Fling Guest API Library – http://labs.vmware.com

VAM (Virtual Machine Monitor)

Good for monitoring an application and posting to vSphere Client

vSAN Panel

Discussion:

• Be able to track trends (request from audience)
• Provision without having to pester IT
• CapEx driven
• Direct Attach – host pinning – get around
• Moves responsibility from Storage Team to virtualization team, but can they handle the extra load?

Join the vSAN beta: http://vsanbeta.com

Ask the Experts

Experts:

• Duncan Epping, VMware – Yellow Bricks blog
• William Lam, VMware – Virtually Ghetto blog
• Scott Lowe, VMware – scottlowe.org
• Vaughn Steward, NetApp – Virtual Storage Guy.com

Awesome blogs, add to favorites!

What about Linux side Automation – PowerCLI seems to be the focus now

• pySphere – soap suds (recommended by William Lam)

vSAN Best Practices

Use Cases:

• Best used for VDI
• Tier 2 & Tier 3 Test / Dev Private Cloud
• DR Target (SRM)
• Branch Office making use of replication

vSAN is just a cluster object/property

vSAN Workflow:

• Setup Network and vmk interfaces
• Enable vSAN
• Set vSAN Auto Mode or Manual Mode
  • Manual Mode: Pick disks

Best practice: have each host similarly configured for best results

Storage policies are like Storage Profiles v2

Policy Capabilities:

• Stripes per object
• Failures to tolerate
• Space reservation
• Flash read cache reservation
• Force provisioning

Performance Monitoring:

• Esxtop
• Performance manager UI
• RVC & Observer (where does one get these from??)

Performance Group Discussion

NUMA:

• Trust VMware NUMA default settings, the experts know better.  Modify at your own peril!
• App must be NUMA aware
• NUMA allows scheduling RAM on same CPU

Virtual CPUs:

• Virtual Cores only added for licensing reasons.  Using Virtual Sockets vs Virtual Cores makes no difference to VM.
• Match Hardware and Virtual CPU to actual need for best results
• Max, don’t allocate more Virtual CPUs than system has of cores (ignore HT and the like)

Latency:

• If latency sensitive – don’t over subscribe!  Simple as that.
• Monitor with vCenter Operations
• No silver bullet
• New “latency” property in 5.5 for Network latency related issues (reserves CPU time)

Storage:

• Split out disks (luns) is still a good practice for database
• Qdepth, queues, and paths

PCIe Hotplug

Supported with Native Driver – another good reason to switch to native driver model

VMware Ready

VMware: Joe Taylor, Pat Richards, Matt Stander

Matt Stander – runs Dev Center (mstandar@vmware.com, or is that mstander@vmware.com?)

All in the Ecosystems Engineering department

Dev Center & VMware Ready – One stop development shop for partners

VMware has 20,000 partners, and 3,000 technical partners.

Contact: tapalliance@vmware.com

Resources: Dev Center, Partner Central, Dev Community, vmdev.net

Workbench IS – for integrating directly into existing eclipse installs

SDK UI Integration docs and guides

Questions:

• Community forums for developers?
  • Coming
• How to get better dev support?
  • Sales Force pay style coming
• How to get in as inbox driver?
  
  • One off requests
• TAM or SDK support (SLA)?
  
  • Ecosystems Engineering
• IOVP Recert 5.5?
  • Will better communicate
• vSAN Cert on Dev Center?
  
  • Coming
• Native Driver?
  
  • Slowly migrate and deprecate Linux
• Workbench – why SLES?
  
  • Legal reasons, getting unified with other departments
• Need packaging repo
  
  • Conference for Dev or more technical dev sessions at VMworld?

Have we tried the new Test Manager that can help automate IOVP process?

• Workbench – top left tutorials
• CLI / Log browser

DDV – Dynamic Driver Verifier on Dev Center

ROSA – will migrate to Dev Center eventually

Dev Support SLA – can hire a guy for a week through Dev Consulting at about $11K a week.  Managed through “Sandy”.  Contact Martin Lister / Joe Taylor for more information (Ecosystem Engineering)

Recommend using Ecosystems Engineering over TAM.  (TAM account manager-> GSS -> Product Management)

vCenter Deep Dive

Web Client now in Simple Install

Custom install: for distributed setup

New in Web Client 5.5:

• Now includes Mac OSX Support
• Filter by tags
• Recent Objects Tab
• Missing: VUM, Host Client, SRM
• Performance still sucks
• Database “shattered” into smaller ones to help with performance

SSO:

• Dropped database
• Dropped master password

How To Redirect a ESX Linux VM's Console Output to a File

This How To will redirect the Linux console output to a file.  Especially useful when you want to collect the full boot log, or full stack trace of a Linux Kernel Panic.

Add VM Serial Port

1. Edit VM Settings and select "Add.." to add new hardware:

2. Select "Serial Port" from the Device Type:

3.Select "Output to file" for Serial Port Type:

4. Specify the file name to output as, on the desired Datastore: (I recommend an NFS share for easy access)

5. Finish saving your changes. Now ready to configure the VM.  Example of a finished Serial Port configuration:

Redirect Linux Console to Serial

1. Test serial out with: (should appear in serial output file specified above)

echo "hello" > /dev/ttyS0 

2. Modify Grub to have Linux Kernel also send console output to Serial: (/boot/grub/grub.conf)

• Comment out splash screen (optional)
• Add serial and terminal settings below "hidden menu" (optional)
• Add kernel options (this is the important part)

# splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
serial --unit=1 --speed=19200
terminal --timeout=8 console serial

title CentOS
  ...
  kernel ... console=tty0 console=ttyS0,19200n8

3. (Optional) Create serial virtual console, append to /etc/inittab: (useful for login through serial)

S0:23:respawn:/sbin/agetty -h -L ttyS0 19200 vt100

• Note: This would be useful if you change the output type from file to something bidirectional (physical serial, pipe, network) that you could login from (if needed)

4. Reboot.  Console output should now appear in the serial output file.

Test a Kernel Panic

To quickly test a kernel panic:

echo c > /proc/sysrq-trigger

Auto Answer Append/Replace Question

To auto answer the append/replace question that occurs when you now power on this VM: (http://kb.vmware.com/kb/1027096http://kb.vmware.com/kb/1027096)

1. Power off the virtual machine.
2. In the vSphere Client, right-click the virtual machine and choose Edit Settings.
3. Click the Options tab.
4. Under Advanced, select General and then click Configuration Parameters.







5. Click Add Row.
6. In the new row, enter "answer.msg.serial.file.open" for the Name and enter a Value of either "Append" or "Replace".

answer.msg.serial.file.open = Append

7 .Click OK in the Configuration Parameters window, then click OK in the Virtual Machine Properties window.

References:

• Add a Serial Port to a Virtual Machine in the vSphere Client - http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.vm_admin.doc_50%2FGUID-A779C84D-4324-4D32-AF43-BBD962ABC9E3.html
• VMware vSphere 4 - ESX and vCenter Server - Add an Output Serial Port - http://pubs.vmware.com/vsphere-4-esx-vcenter/index.jsp?topic=/com.vmware.vsphere.webaccess.doc_40/adding_hardware_to_a_virtual_machine/t_add_an_output_serial_port.html
• Linux: Configure / Enable Serial Console By Editing GRUB Boot Loader - http://www.cyberciti.biz/faq/linux-serial-console-howto/
• VMware KB: Powering on a virtual machine pauses at 95% while waiting for a question to be answered - http://kb.vmware.com/kb/1027096

I'm too ESXi for my shirt

Couldn't have said it better.  ;-)