System Configuration:

Hardware: Dell R710, 2xE5620 CPUs, 12x2GB UDIMMs, RAID1 via onboard controller

Host OS: Windows Server 2008 R2 Enterprise 64bit running the Hyper-V role, fully patched as of testing.

Guest OSes: 2 VMs both running Windows Server 2008 R2 Enterprise 64bit, 1CPU, 4GB RAM, 1 NIC, dynamic VHD on IDE Controller 0 (127GB max) each, fully patched as of testing.

Troubleshooting

Further troubleshooting using perfmon in Windows showed that VMs boot fine on cores 4-7, but not on 0-3.  I had previously disabled logical processors in the bios so I didn’t have to troubleshoot 16 logical CPUs.  At this point, I thought CPU0, or maybe just the hardware virtualization part of it, was bad.

I ran Prime95 (64bit) and the Intel Processor Diagnostic tool to test the CPU, but neither showed an issue with either CPU.  I also ran Memtest86+ v4.20 which showed the memory to be fine.  My line of thinking was that if Prime95 and the Intel CPU diag don’t access the hardware virtualization calls in the CPU, they may not find the fault I’m experiencing.

Resolution

After a brief online text chat with Greg Whiteman at Dell hardware support, he suggested that I disable Power Management (set it to Maximum Performance).  That single change resolved the issue.

Conclusion

I changed the following setting from “Active Power Controller” to “Maximum Performance” and I was able to launch multiple VMs at once without Hyper-V crashing.  I changed back to “Active Power Controller” and I could immediately reproduce the problem (Hyper-V crashing).  I’m not sure where the bug lies (Windows Hyper-V implementation or hardware/firmware implementation), but this setting change fixed it.

Finally, to fully test, I set the following which I believe to be my final settings, and everything works fine.

Final Dell R710 BIOS Settings Used

Processor Settings\Logical Processor: “Enabled”
Processor Settings\Virtualization Technology: “Enabled”
Processor Settings\Execute Disable: “Enabled”
Processor Settings\Turbo Mode: “Disabled”
Processor Settings\C1E: “Disabled”
Processor Settings\C States: “Disabled”
Power Management: “Maximum Performance”

Full Event Details:

Log Name:      Microsoft-Windows-Hyper-V-Worker-Admin

Source:        Microsoft-Windows-Hyper-V-Worker

Date:          3/15/2011 5:50:02 PM

Event ID:      18560

Task Category: None

Level:         Critical

Keywords:

User:          NETWORK SERVICE

Computer:      WIN-L4DCENF56H0

Description:

'testvm1' was reset because an unrecoverable error occurred on a
virtual processor that caused a triple fault. If the problem persists,
contact Product Support.
(Virtual machine ID 8CB72529-261D-4685-A118-F781F34CE986)

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="Microsoft-Windows-Hyper-V-Worker"
Guid="{51DDFA29-D5C8-4803-BE4B-2ECB715570FE}" />

<EventID>18560</EventID>

<Version>0</Version>

<Level>1</Level>

<Task>0</Task>

<Opcode>0</Opcode>

<Keywords>0x8000000000000000</Keywords>

<TimeCreated SystemTime="2011-03-15T22:50:02.132334800Z" />

<EventRecordID>370</EventRecordID>

<Correlation />

<Execution ProcessID="1148" ThreadID="3736" />

<Channel>Microsoft-Windows-Hyper-V-Worker-Admin</Channel>

<Computer>WIN-L4DCENF56H0</Computer>

<Security UserID="S-1-5-20" />

</System>

<UserData>

<VmlEventLog xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events"
xmlns="http://www.microsoft.com/Windows/Virtualization/Events">

<VmName>testvm1</VmName>

<VmId>8CB72529-261D-4685-A118-F781F34CE986</VmId>

</VmlEventLog>

</UserData>

</Event>

Important note (this will come into play later): I have VERY fast home-grown ZFS-based NAS/SAN that stores my VDI (and most other) files that I use on a regular basis.  In most cases, this proves to be faster than running virtual machines off of my local disk.

I wanted to compact my Sun (now Oracle) VirtualBox vdi file.  The vdi file was created with version 3.2.12r68302 of VirtualBox, and was a clean, patched base install of Windows 7 that I wanted to compact as much as possible.  After reviewing the user’s manual that comes with the product (C:\Program Files\Oracle\VirtualBox\doc\UserManual.pdf), page 99 gave me the command line to use (vboxmanage.exe modifyhd –compact file.vdi), and page 116 references the sdelete tool from Microsoft required to zero out blank space.  After deleting unnecessary files, deleting system restore points, emptying recycle bin, etc., I ran the sdelete tool (which took about 2 hours), then shut the machine down.  So far, so good.  Now it was time to run the “vboxmanage.exe modifyhd –compact” command, which presented a problem that most people probably won’t run into.

Symptom

C:\Program Files\Oracle\VirtualBox>VBoxManage.exe modifyhd –compact y:\Base_OS_Installs\win7base.vdi
Oracle VM VirtualBox Command Line Management Interface Version 3.2.12
(C) 2005-2010 Oracle Corporation
All rights reserved.

ERROR: Could not find a hard disk with location ‘y:\Base_OS_Installs\win
7base.vdi’ in the media registry (‘C:\Users\palermo/.VirtualBox\VirtualBox.xml’)

Details: code VBOX_E_OBJECT_NOT_FOUND (0x80bb0001), component VirtualBox, interface IVirtualBox, callee IUnknown
Context: “FindHardDisk(Bstr(FilenameOrUuid), hardDisk.asOutParam())” at line 430
of file VBoxManageDisk.cpp

Resolution

Apparently when you add a vdi file to the media registry, it expands mapped drives to the UNC name.  Using the UNC name instead of the mapped drive letter fixed the problem:

0%…10%…20%…30%…40%…50%…60%…70%…80%…90%…100%

This took my VDI file from 12.4GB to 8.4GB.  Still big, but almost 33% less than its original size.

The computer crashes and reboots for no apparent reason, and now users get an error message when trying to access any of the file or printer shares on the server/computer.  This is related to a registry setting, the Security Event Log filling up, and a possible bug in 2008 R2 that doesn’t allow users to access the server even after the Security Event Log issue has been resolved.

Upon further inspection, the Windows Security Event Log is full and HKLM\SYSTEM\CurrentControlSet\Control\Lsa\CrashOnAuditFail = 1.  You resolve this issue by right-clicking the Security event log, selecting properties, then doing one of the following:

1. Increase the “Maximum log size”
2. Allow the Security event log to “Overwrite events as needed (oldest events first)”
3. “Archive the log when full, do not overwrite events”

This has solved the Security Event Log issue, but now users report they can no longer access \\computername, or any of the shares or shared printers on that machine.   You (a member of the administrators group on the machine), and other administrators on the machine can access shares and shared printers without a problem.

Regular users receive this error:

\\computername is not accessible.  You might not have permission to use this network resource.  Contact the administrator oft his server to find out if you have access permissions.

Logon failure: user account restriction.  Possible reasons are blank passwords not allowed, logon hour restrictions, or a policy restriction has been enforced.

Cause:

The Windows Security Event Log has filled up, causing the server to crash.  This was caused by the following registry value:

HKLM\SYSTEM\CurrentControlSet\Control\Lsa\CrashOnAuditFail = 1

This could have been set by a security template, group policy, or manual registry change.  In any case, the server crashed because it could no longer log audit-able events to the security log.  Users can not access this server until this issue is resovled (see steps 1-3 above).  But even after the Security log is happy again, users may still receive the “\\computername is not accessible” error.

Resolution:

1. Increase the “Maximum log size” to 1GB or more
2. Allow the Security event log to “Overwrite events as needed (oldest events first)”
3. Set HKLM\System\CurrentControlSet\Control\Lsa\LimitBlankPasswordUse = 0
4. Reboot
5. Set HKLM\System\CurrentControlSet\Control\Lsa\LimitBlankPasswordUse = 1
6. Reboot

This took hours to figure out.  Hopefully this post saves someone else some time.

Related Event Log IDs

<Event ID 4625>
Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          4/30/2010 4:20:59 PM
Event ID:      4625
Task Category: Logon
Level:         Information
Keywords:      Audit Failure
User:          N/A
Computer:      <computer where this event was logged>
Description:
An account failed to log on.

Subject:
Security ID:         NULL SID
Account Name:        -
Account Domain:      -
Logon ID:       0×0

Logon Type:                3

Account For Which Logon Failed:
Security ID:         NULL SID
Account Name:        <username>
Account Domain:      <domain>

Failure Information:
Failure Reason:      Unknown user name or bad password.
Status:              0xc000006e
Sub Status:          0×0

Process Information:
Caller Process ID:   0×0
Caller Process Name: -

Network Information:
Workstation Name:    -
Source Network Address:    <IP address of client>
Source Port:         3089

Detailed Authentication Information:
Logon Process:       Kerberos
Authentication Package:    Kerberos
Transited Services:  -
Package Name (NTLM only):  -
Key Length:          0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
</Event ID 4625>

<Event ID 4776>
Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          4/30/2010 4:20:55 PM
Event ID:      4776
Task Category: Credential Validation
Level:         Information
Keywords:      Audit Failure
User:          N/A
Computer:      <computer where this event was logged>
Description:
The computer attempted to validate the credentials for an account.

Authentication Package:    MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account:  <username>
Source Workstation:  <client computername>
Error Code:     0xc0000064
</Event ID 4776>

Related Documentation

Microsoft TechNet article about CrashOnAuditFail

13%: Amount of printed pages wasted by banner pages: 13% (1 out of 8 pages is usually thrown away recycled).

1,050,000 pages: Average yearly print volume in my dept (yes, over a million)

$932: Money wasted on paper to print banner pages each year.  That’s just the paper cost, and does not include toner, printer maintenance, etc.

Call them what you want: banner pages, separator pages, job sheet, etc.  These things are a waste of money.  I’m looking for solutions to eliminate banner pages from my department.  So far, I’ve found two options.

The on-printer option

The Lexmark TS6545dn is relatively cheap ($1500), fast (55ppm), and has a function called “Reserve Print” that allows printouts to be held on the printers hard drive until they are printed via the printer’s LCD interface.

The HP P4515n is also relatively cheap (~$1500), fast (62ppm), and has “Job Storage Features”, which include “Personal Jobs”.  This feature is a bit nicer in that it automatically uses the username (you don’t have to set it in the driver config like the Lexmark), and automatically deletes the job after it prints.  The HP is a bit worse in that there are 3-4 more steps necessary to retrieve printouts.  A pin (4-digit code) is still necessary, but can be set to a default like “0000″, which is acceptable for my uses.  HP’s Web Jetadmin may be able to work around the quirks (allowing multiple copies at the printer).  If this is true, HP will be the clear winner.  I’ll updated this post as more details become available.

Pros

1. Free!  The feature is built in to the printer and requires minimal additional hardware (an internal hard drive is recommended for print job storage).
2. Easy to use!  Walk up to the printer, find your username, find the job you want to print and press go.

Cons

1. At the printer, you can print multiple copies of your document, even if you just sent 1 copy to the printer.  I support student printing and need a way to account for who prints what and how much.  This detraction is present on both Lexmark and HP printers.
2. Lexmark only: Print job is not automatically deleted after you print it.  This may not be a “con” in your case, but it is in mine.

The add-on option

HP, CZ, and others have release station software.  This runs on a computer near the printer and is used to release print jobs from the queue on an as-needed basis.  The best implementation I’ve seen uses an all-in-one touch screen computer hung on the wall above the printer.  Of course that can get expensive.

Pros

1. Use it with any printer since it isn’t printer specific.
2. Easy to use.  Touch the print job you want and press release.
3. Use to load-balance printers.  Choose which printer you want to release your job to (ex: if there are 3 printers in the same room)

Cons

1. Expensive.  The touch screen all-in-one PC, plus related software licenses quickly tops $1000.

Do you have/use an option I don’t know about?  Please share your knowledge by leaving a comment below.!

For example, 250MB of storage space for students on the departmental server used to be status quo.  A while back, we increase that by 800%, and will continue to grow to meet the needs of our students.

Another example, a 2MB mailbox for plain text email, using a client such as pine (or elm, mail, mailx, etc.) used to be the status quo.  Now, anyone can go online and get a 5GB+ mailbox, rich text email, and much, much more for free from places like Gmail, Yahoo!, and Hotmail.  Email (let’s use a more general term like electronic collaboration) has become a commodity.  Lots of people/companies do it better than I do for cheaper than I can, so why would I bother hosting my own server?  That’s a good question!

In an academic environment, things typically change slowly (if at all).  If you want to do something new, cool, and cutting edge, there are always rules and regulations to hold you back comply with (anyone ever dealt with IRB?).  So I started the long tedious process of outsourcing email about a year ago.  Within the past couple of years, electronic collaboration has become much more than email, and lots of companies are doing it (some are even doing it right).  Even better, electronic collaboration has become a commodity, and with commodities come competition, even if the service is free!  Did you notice how your old 2MB hotmail account jumped to 250MB seemingly overnight (you know, after Gmail debuted), and then again to 5GB?  That’s commodity competition at work!

So, with all the new cool features (or if you’re providing the serivce: complexity and development time) and increased reliance (or supporting the service: customer intolerance for downtime) on electronic collaboration, why not let someone else do it?  They can, in most cases, do it better, cheaper, faster, and offer those new features that I can’t (or am not willing to) roll out on my own email server.

The next question is “Who does it best?”  Well, you have to define “best”, or rather, you have to define your requirements.  The following lists detail my hard and soft requirements for an electronic collaboration solution.

Hard Requirements

• Email
• Calendar
• Contacts
• Shared folders that contain items listed above
• Mobile sync
• Web browser accessible
• Easy to use
• OS independent
• Offline capability

Soft Requirements

• Document collaboration
  • Text docs
  • Spreadsheets
  • Presentations
• LOTS of space
• Email filters/rules
• Chat/IM
• Free (or close to it)

Conclusion

For my money, no one (currently) does electronic collaboration better than Google.  Google Apps for Education meets all the hard AND soft requirements.  It also provides video conferencing, fast searching, and is really, REALLY easy to use.  BUT (yeah, you knew it was coming), here’s why I can’t use them in my department:

1. Their support is email-only
2. I have never been able to talk to anyone other than a salesperson regarding the purchase or negotiating Terms of Service.
3. Data can be stored anywhere in the world, which doesn’t give the lawyers warm fuzzies (and may actually be against the law in my particular situation).
4. Google doesn’t actively recruit Aggies.

Microsoft’s Live@Edu comes in a close second for the following reasons.  Items that are struck through have been resolved in more recent versions of Live@Edu.

1. Web browser accessible – well, not ALL web browsers.  IE, Firefox, and Safari are the only 3 supported.  Yeah, that’s the majority, but I like Google Chrome.  I also know people who use Opera, Konqueror, and others.
2. OS independent - again, if your OS runs (and you are willing to use) IE, Firefox, or Safari, you’re ok.
3. Document collaboration – This is HUGE!  To share and collaborate on documents, you MUST have Microsoft Office installed.  You can share lists and text docs online, but to do anything fancy/creative, you must have Microsoft Office.  That was a deal breaker for me.  I can’t go telling Dr. Linux Evangelist that they need MS Office (not available for Linux) to use our new cool electronic collaboration suite.  With the availability of Office 2010 beta, this service is actually better than Google’s offering.
4. Assumes it is authoritative for your domain!  If you set up Live@Edu for domain.edu and then try to send an email to userx@domain.edu (who still has an email account on the domain.edu email server), it’ll bounce.  This was a little annoying.  There are work-arounds, and they work.  There’s a work-around for this, but it’s still annoying.
5. MORE storage than Google!!  10GB for FREE, 25GB for <$20/year (academic pricing).
6. They heavily recruit A&M graduates, especially Computer Science grads!

Questions/Comments?

There’s a box down there that says “Leave a comment.”  If you have a valid question or concern, I’ll do my best to get a satisfactory answer for you.

For even more details, view my presentation on this subject (slides may change at any time)

Environment

200 PCs (staff and lab machines) running Windows XP Pro, all with at least 40GB free.

Goal

Utilize more of the storage we have already paid for.  Most workstations are online 24×7, and have unused disk space (20GB to 200GB).  I would like to pool the storage in a highly redundant fashion (ex: 4 machines contain the same data so 1 can be powered off with no impact on data accessibility) to present a large block of usable space.

Possible technologies to use

· DFS

· iSCSI+ZFS

· iSCSI+software RAID

Possible uses of combined, redundant, pooled storage

· CIFS shares

· VTL storage

· large temporary storage for student use

· large DB backend (probably not a great use for this, but given the number of spindles, it has potential)

Benefits

· highly redundant

· LOTS of spindles

Possibly helpful links
http://www.faqs.org/patents/app/20080320097
http://www.datacore.com/
http://msdn.microsoft.com/en-us/library/bb524801(VS.85).aspx

http://www.gluster.org

iSCSI Target software for Windows
http://www.iscsicake.com/
http://www.emboot.com/products_sanFly.htm
http://www.kernsafe.com/Default.aspx

Installing Windows 2003 Server’s Network Monitor on Windows XP

1. Install Network Monitor on a Windows 2003 server if it is not already installed.
   1. Start -> Control Panel -> Add/Remove Programs -> Add/Remove Windows Components -> Management and Monitoring Tools -> Details -> [check] Network Monitor Tools -> OK -> Next -> Finish
2. On the XP workstation, install the network capture driver
   1. Start -> Run -> CMD -> OK -> netcap -> the capture driver is now installed, hit the space bar to stop the packet capture
3. Copy the netmon folder from the System32 folder on the server (%windir%\system32\netmon) to the System32 folder on the workstation (%windir%\system32\netmon).
   1. Be sure to grant system and administrator change permissions on the netmon folder and all the files/folders it contains, otherwise netmon.exe won’t start. 
   2. Be sure the “parsers” folder is included when you copy the netmon folder. Drag-and-drop copy keeps this, as does xcopy, but regular “copy” from the command prompt does not.
4. Copy nmsupp.dll from the server (%windir%\system32\nmsupp.dll) to the netmon folder on the workstation.
5. Run netmon.exe

1. On VMware virtual machine, set Exchange and SMTP services to manual
2. Shut down virtual machine
3. Create a backup of the VMDK and related files for OS (C:\) partition
4. Add and IDE disk to the virtual machine to enable IDE drivers in Windows and make the Hyper-V converted disk easier to boot.
5. Remove Exchange’s mail store VMDK disk (E:\) from the virtual machine’s configuration to ensure no data is changed on that partition.
6. Disable networking in VMware console so that any changes made are confined to the local machine.
7. Boot working copy of Exchange OS partition
8. Log on as local admin
9. Uninstall VMware tools and reboot VM when prompted
10. Cancel “Found new hardware wizard” and do not reboot when asked
11. Verify IDE drive is present & drivers loaded
12. Append C:\windows\system32\hal.dll and ntoskrnl.exe with .bak extensions and copy versions from C:\windows\servicepackfiles\i386 to replace the versions in the system32 folder.
13. Run prepvm.vbs created by Chris Wolf
14. Shut down the server
15. Copy modified VMDK files for OS partition (one 20GB and one 1KB) to LUN on IBM SAN for Hyper-V server (arch-host-09) to use
16. Convert VMDK file to VHD file using the VMDK to VHD converter from vmtoolkit.com
17. Link VHD to VM in Hyper-V and boot the file. The following steps were used for troubleshooting.
    1. Booted to Windows 2003 server OS CD to run recovery console
    2. Added a line in boot.ini with arguments “/bootlog /sos /safeboot:minimal” which enabled the system to boot into safe mode.
    3. Rebooted server and chose normal boot-up
    4. Copied halaacpi.dll from C:\windows\servicepackfiles\i386 to overwrite C:\windows\system32\hal.dll
    5. Reboot server normally
    6. Opened “System” in Control Panel to remove reference to the old hal.dll in the Computer properties.
    7. Rebooted system
18. Installed Hyper-V add-ons and rebooted.

Surprisingly, that’s all it took.  Step 17.4 took a little while to figure out because I kept getting an error message about ACPI-compatible hal.dll when trying to install the Hyper-V add-ons.