System Configuration:

Hardware: Dell R710, 2xE5620 CPUs, 12x2GB UDIMMs, RAID1 via onboard controller

Host OS: Windows Server 2008 R2 Enterprise 64bit running the Hyper-V role, fully patched as of testing.

Guest OSes: 2 VMs both running Windows Server 2008 R2 Enterprise 64bit, 1CPU, 4GB RAM, 1 NIC, dynamic VHD on IDE Controller 0 (127GB max) each, fully patched as of testing.

Troubleshooting

Further troubleshooting using perfmon in Windows showed that VMs boot fine on cores 4-7, but not on 0-3.  I had previously disabled logical processors in the bios so I didn’t have to troubleshoot 16 logical CPUs.  At this point, I thought CPU0, or maybe just the hardware virtualization part of it, was bad.

I ran Prime95 (64bit) and the Intel Processor Diagnostic tool to test the CPU, but neither showed an issue with either CPU.  I also ran Memtest86+ v4.20 which showed the memory to be fine.  My line of thinking was that if Prime95 and the Intel CPU diag don’t access the hardware virtualization calls in the CPU, they may not find the fault I’m experiencing.

Resolution

After a brief online text chat with Greg Whiteman at Dell hardware support, he suggested that I disable Power Management (set it to Maximum Performance).  That single change resolved the issue.

Conclusion

I changed the following setting from “Active Power Controller” to “Maximum Performance” and I was able to launch multiple VMs at once without Hyper-V crashing.  I changed back to “Active Power Controller” and I could immediately reproduce the problem (Hyper-V crashing).  I’m not sure where the bug lies (Windows Hyper-V implementation or hardware/firmware implementation), but this setting change fixed it.

Finally, to fully test, I set the following which I believe to be my final settings, and everything works fine.

Final Dell R710 BIOS Settings Used

Processor Settings\Logical Processor: “Enabled”
Processor Settings\Virtualization Technology: “Enabled”
Processor Settings\Execute Disable: “Enabled”
Processor Settings\Turbo Mode: “Disabled”
Processor Settings\C1E: “Disabled”
Processor Settings\C States: “Disabled”
Power Management: “Maximum Performance”

Full Event Details:

Log Name:      Microsoft-Windows-Hyper-V-Worker-Admin

Source:        Microsoft-Windows-Hyper-V-Worker

Date:          3/15/2011 5:50:02 PM

Event ID:      18560

Task Category: None

Level:         Critical

Keywords:

User:          NETWORK SERVICE

Computer:      WIN-L4DCENF56H0

Description:

'testvm1' was reset because an unrecoverable error occurred on a
virtual processor that caused a triple fault. If the problem persists,
contact Product Support.
(Virtual machine ID 8CB72529-261D-4685-A118-F781F34CE986)

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="Microsoft-Windows-Hyper-V-Worker"
Guid="{51DDFA29-D5C8-4803-BE4B-2ECB715570FE}" />

<EventID>18560</EventID>

<Version>0</Version>

<Level>1</Level>

<Task>0</Task>

<Opcode>0</Opcode>

<Keywords>0x8000000000000000</Keywords>

<TimeCreated SystemTime="2011-03-15T22:50:02.132334800Z" />

<EventRecordID>370</EventRecordID>

<Correlation />

<Execution ProcessID="1148" ThreadID="3736" />

<Channel>Microsoft-Windows-Hyper-V-Worker-Admin</Channel>

<Computer>WIN-L4DCENF56H0</Computer>

<Security UserID="S-1-5-20" />

</System>

<UserData>

<VmlEventLog xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events"
xmlns="http://www.microsoft.com/Windows/Virtualization/Events">

<VmName>testvm1</VmName>

<VmId>8CB72529-261D-4685-A118-F781F34CE986</VmId>

</VmlEventLog>

</UserData>

</Event>

Important note (this will come into play later): I have VERY fast home-grown ZFS-based NAS/SAN that stores my VDI (and most other) files that I use on a regular basis.  In most cases, this proves to be faster than running virtual machines off of my local disk.

I wanted to compact my Sun (now Oracle) VirtualBox vdi file.  The vdi file was created with version 3.2.12r68302 of VirtualBox, and was a clean, patched base install of Windows 7 that I wanted to compact as much as possible.  After reviewing the user’s manual that comes with the product (C:\Program Files\Oracle\VirtualBox\doc\UserManual.pdf), page 99 gave me the command line to use (vboxmanage.exe modifyhd –compact file.vdi), and page 116 references the sdelete tool from Microsoft required to zero out blank space.  After deleting unnecessary files, deleting system restore points, emptying recycle bin, etc., I ran the sdelete tool (which took about 2 hours), then shut the machine down.  So far, so good.  Now it was time to run the “vboxmanage.exe modifyhd –compact” command, which presented a problem that most people probably won’t run into.

Symptom

C:\Program Files\Oracle\VirtualBox>VBoxManage.exe modifyhd –compact y:\Base_OS_Installs\win7base.vdi
Oracle VM VirtualBox Command Line Management Interface Version 3.2.12
(C) 2005-2010 Oracle Corporation
All rights reserved.

ERROR: Could not find a hard disk with location ‘y:\Base_OS_Installs\win
7base.vdi’ in the media registry (‘C:\Users\palermo/.VirtualBox\VirtualBox.xml’)

Details: code VBOX_E_OBJECT_NOT_FOUND (0x80bb0001), component VirtualBox, interface IVirtualBox, callee IUnknown
Context: “FindHardDisk(Bstr(FilenameOrUuid), hardDisk.asOutParam())” at line 430
of file VBoxManageDisk.cpp

Resolution

Apparently when you add a vdi file to the media registry, it expands mapped drives to the UNC name.  Using the UNC name instead of the mapped drive letter fixed the problem:

0%…10%…20%…30%…40%…50%…60%…70%…80%…90%…100%

This took my VDI file from 12.4GB to 8.4GB.  Still big, but almost 33% less than its original size.

The computer crashes and reboots for no apparent reason, and now users get an error message when trying to access any of the file or printer shares on the server/computer.  This is related to a registry setting, the Security Event Log filling up, and a possible bug in 2008 R2 that doesn’t allow users to access the server even after the Security Event Log issue has been resolved.

Upon further inspection, the Windows Security Event Log is full and HKLM\SYSTEM\CurrentControlSet\Control\Lsa\CrashOnAuditFail = 1.  You resolve this issue by right-clicking the Security event log, selecting properties, then doing one of the following:

1. Increase the “Maximum log size”
2. Allow the Security event log to “Overwrite events as needed (oldest events first)”
3. “Archive the log when full, do not overwrite events”

This has solved the Security Event Log issue, but now users report they can no longer access \\computername, or any of the shares or shared printers on that machine.   You (a member of the administrators group on the machine), and other administrators on the machine can access shares and shared printers without a problem.

Regular users receive this error:

\\computername is not accessible.  You might not have permission to use this network resource.  Contact the administrator oft his server to find out if you have access permissions.

Logon failure: user account restriction.  Possible reasons are blank passwords not allowed, logon hour restrictions, or a policy restriction has been enforced.

Cause:

The Windows Security Event Log has filled up, causing the server to crash.  This was caused by the following registry value:

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\CrashOnAuditFail = 1

This could have been set by a security template, group policy, or manual registry change.  In any case, the server crashed because it could no longer log audit-able events to the security log.  Users can not access this server until this issue is resovled (see steps 1-3 above).  But even after the Security log is happy again, users may still receive the “\\computername is not accessible” error.

Resolution:

1. Increase the “Maximum log size” to 1GB or more
2. Allow the Security event log to “Overwrite events as needed (oldest events first)”
3. Set HKLM\System\CurrentControlSet\Control\Lsa\LimitBlankPasswordUse = 0
4. Reboot
5. Set HKLM\System\CurrentControlSet\Control\Lsa\LimitBlankPasswordUse = 1
6. Reboot

This took hours to figure out.  Hopefully this post saves someone else some time.

Related Event Log IDs

<Event ID 4625>
Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          4/30/2010 4:20:59 PM
Event ID:      4625
Task Category: Logon
Level:         Information
Keywords:      Audit Failure
User:          N/A
Computer:      <computer where this event was logged>
Description:
An account failed to log on.

Subject:
Security ID:         NULL SID
Account Name:        -
Account Domain:      -
Logon ID:       0×0

Logon Type:                3

Account For Which Logon Failed:
Security ID:         NULL SID
Account Name:        <username>
Account Domain:      <domain>

Failure Information:
Failure Reason:      Unknown user name or bad password.
Status:              0xc000006e
Sub Status:          0×0

Process Information:
Caller Process ID:   0×0
Caller Process Name: -

Network Information:
Workstation Name:    -
Source Network Address:    <IP address of client>
Source Port:         3089

Detailed Authentication Information:
Logon Process:       Kerberos
Authentication Package:    Kerberos
Transited Services:  -
Package Name (NTLM only):  -
Key Length:          0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
</Event ID 4625>

<Event ID 4776>
Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          4/30/2010 4:20:55 PM
Event ID:      4776
Task Category: Credential Validation
Level:         Information
Keywords:      Audit Failure
User:          N/A
Computer:      <computer where this event was logged>
Description:
The computer attempted to validate the credentials for an account.

Authentication Package:    MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account:  <username>
Source Workstation:  <client computername>
Error Code:     0xc0000064
</Event ID 4776>

Related Documentation

Microsoft TechNet article about CrashOnAuditFail

13%: Amount of printed pages wasted by banner pages: 13% (1 out of 8 pages is usually thrown away recycled).

1,050,000 pages: Average yearly print volume in my dept (yes, over a million)

$932: Money wasted on paper to print banner pages each year.  That’s just the paper cost, and does not include toner, printer maintenance, etc.

Call them what you want: banner pages, separator pages, job sheet, etc.  These things are a waste of money.  I’m looking for solutions to eliminate banner pages from my department.  So far, I’ve found two options.

The on-printer option

The Lexmark TS6545dn is relatively cheap ($1500), fast (55ppm), and has a function called “Reserve Print” that allows printouts to be held on the printers hard drive until they are printed via the printer’s LCD interface.

The HP P4515n is also relatively cheap (~$1500), fast (62ppm), and has “Job Storage Features”, which include “Personal Jobs”.  This feature is a bit nicer in that it automatically uses the username (you don’t have to set it in the driver config like the Lexmark), and automatically deletes the job after it prints.  The HP is a bit worse in that there are 3-4 more steps necessary to retrieve printouts.  A pin (4-digit code) is still necessary, but can be set to a default like “0000″, which is acceptable for my uses.  HP’s Web Jetadmin may be able to work around the quirks (allowing multiple copies at the printer).  If this is true, HP will be the clear winner.  I’ll updated this post as more details become available.

Pros

1. Free!  The feature is built in to the printer and requires minimal additional hardware (an internal hard drive is recommended for print job storage).
2. Easy to use!  Walk up to the printer, find your username, find the job you want to print and press go.

Cons

1. At the printer, you can print multiple copies of your document, even if you just sent 1 copy to the printer.  I support student printing and need a way to account for who prints what and how much.  This detraction is present on both Lexmark and HP printers.
2. Lexmark only: Print job is not automatically deleted after you print it.  This may not be a “con” in your case, but it is in mine.

The add-on option

HP, CZ, and others have release station software.  This runs on a computer near the printer and is used to release print jobs from the queue on an as-needed basis.  The best implementation I’ve seen uses an all-in-one touch screen computer hung on the wall above the printer.  Of course that can get expensive.

Pros

1. Use it with any printer since it isn’t printer specific.
2. Easy to use.  Touch the print job you want and press release.
3. Use to load-balance printers.  Choose which printer you want to release your job to (ex: if there are 3 printers in the same room)

Cons

1. Expensive.  The touch screen all-in-one PC, plus related software licenses quickly tops $1000.

Do you have/use an option I don’t know about?  Please share your knowledge by leaving a comment below.!

Environment

200 PCs (staff and lab machines) running Windows XP Pro, all with at least 40GB free.

Goal

Utilize more of the storage we have already paid for.  Most workstations are online 24×7, and have unused disk space (20GB to 200GB).  I would like to pool the storage in a highly redundant fashion (ex: 4 machines contain the same data so 1 can be powered off with no impact on data accessibility) to present a large block of usable space.

Possible technologies to use

· DFS

· iSCSI+ZFS

· iSCSI+software RAID

Possible uses of combined, redundant, pooled storage

· CIFS shares

· VTL storage

· large temporary storage for student use

· large DB backend (probably not a great use for this, but given the number of spindles, it has potential)

Benefits

· highly redundant

· LOTS of spindles

Possibly helpful links
http://www.faqs.org/patents/app/20080320097
http://www.datacore.com/
http://msdn.microsoft.com/en-us/library/bb524801(VS.85).aspx

http://www.gluster.org

iSCSI Target software for Windows
http://www.iscsicake.com/
http://www.emboot.com/products_sanFly.htm
http://www.kernsafe.com/Default.aspx

Installing Windows 2003 Server’s Network Monitor on Windows XP

1. Install Network Monitor on a Windows 2003 server if it is not already installed.
   1. Start -> Control Panel -> Add/Remove Programs -> Add/Remove Windows Components -> Management and Monitoring Tools -> Details -> [check] Network Monitor Tools -> OK -> Next -> Finish
2. On the XP workstation, install the network capture driver
   1. Start -> Run -> CMD -> OK -> netcap -> the capture driver is now installed, hit the space bar to stop the packet capture
3. Copy the netmon folder from the System32 folder on the server (%windir%\system32\netmon) to the System32 folder on the workstation (%windir%\system32\netmon).
   1. Be sure to grant system and administrator change permissions on the netmon folder and all the files/folders it contains, otherwise netmon.exe won’t start. 
   2. Be sure the “parsers” folder is included when you copy the netmon folder. Drag-and-drop copy keeps this, as does xcopy, but regular “copy” from the command prompt does not.
4. Copy nmsupp.dll from the server (%windir%\system32\nmsupp.dll) to the netmon folder on the workstation.
5. Run netmon.exe

1. On VMware virtual machine, set Exchange and SMTP services to manual
2. Shut down virtual machine
3. Create a backup of the VMDK and related files for OS (C:\) partition
4. Add and IDE disk to the virtual machine to enable IDE drivers in Windows and make the Hyper-V converted disk easier to boot.
5. Remove Exchange’s mail store VMDK disk (E:\) from the virtual machine’s configuration to ensure no data is changed on that partition.
6. Disable networking in VMware console so that any changes made are confined to the local machine.
7. Boot working copy of Exchange OS partition
8. Log on as local admin
9. Uninstall VMware tools and reboot VM when prompted
10. Cancel “Found new hardware wizard” and do not reboot when asked
11. Verify IDE drive is present & drivers loaded
12. Append C:\windows\system32\hal.dll and ntoskrnl.exe with .bak extensions and copy versions from C:\windows\servicepackfiles\i386 to replace the versions in the system32 folder.
13. Run prepvm.vbs created by Chris Wolf
14. Shut down the server
15. Copy modified VMDK files for OS partition (one 20GB and one 1KB) to LUN on IBM SAN for Hyper-V server (arch-host-09) to use
16. Convert VMDK file to VHD file using the VMDK to VHD converter from vmtoolkit.com
17. Link VHD to VM in Hyper-V and boot the file. The following steps were used for troubleshooting.
    1. Booted to Windows 2003 server OS CD to run recovery console
    2. Added a line in boot.ini with arguments “/bootlog /sos /safeboot:minimal” which enabled the system to boot into safe mode.
    3. Rebooted server and chose normal boot-up
    4. Copied halaacpi.dll from C:\windows\servicepackfiles\i386 to overwrite C:\windows\system32\hal.dll
    5. Reboot server normally
    6. Opened “System” in Control Panel to remove reference to the old hal.dll in the Computer properties.
    7. Rebooted system
18. Installed Hyper-V add-ons and rebooted.

Surprisingly, that’s all it took.  Step 17.4 took a little while to figure out because I kept getting an error message about ACPI-compatible hal.dll when trying to install the Hyper-V add-ons.

1. Teamviewer – absolutely my #1 pick for supporting friends & family. Free for personal use, very fast, easily bypasses most firewalls, routers, NATs, etc. 
   My implementation for home use: http://www.aaronpalermo.com/help
2. VNC is a free remote control software package. Many versions exist (UltraVNC, TightVNC, RealVNC, etc.) The main drawback here is TCP port 5900 has to be open and routed to the requestor’s PC. This can be a real hassle if the requestor is using NAT, and/or a cable modem router/firewall, and/or a 3^rd party software firewall. The easiest way I have found to get around this is using UltraVNC’s SingleClick add-on. It is a lot easier to ensure you can listen on port 5500 than trying to talk someone else through configuring their router(s) and firewall(s), especially if they are the type of person who has to call you for help in the first place.
   My implementation for work: http://staff.cs.tamu.edu/palermo/help
3. Remote Assistance built into XP and Vista works well if you have it preconfigured. TCP port 3389 has to be listening and routed to the requestor’s PC, and you have to have a remote assistance invitation, or be a admin on the same domain as the requestor. Also, you have to know the requestor’s IP address (yes, a quick visit to www.whatismyip.com fixes this, but just one more thing to talk a person through)
4. MSN messenger, Netmeeting (called Windows Meeting Space in Vista), Skype, and other instant messenger programs have remote assistance (also called desktop sharing) functions built in, but if you don’t already use one of those, then do you really want to create yet another login & password to remember?
5. Other commercial options like webex and GoToMeeting exist for remote support, but they range from slightly pricy to price gouging. These may be excellent options for enterprise level support, but not for personal or small business use.