A request to allocate an ephemeral port number from the global TCP port space has failed due to all such ports being in use.

This particular issue made my life miserable for over a 2 week period and affected 3 of my backup servers! So what would happen is that, randomly some of my backup servers would somehow lose connection and drop of the network. I mean they would still be pingable and online however when I would try to login, I would get a "No logon servers available" message and at that point the only way to fix this was to restart the server! This made my backups to fail as well. I have like over 50 Tb to backup on each of my locations and because of this issue they would never complete. So after digging around I found a couple of warning messages in my System log as you see below.

tcp1.JPG

So looks like my backup servers were getting their TCP connections exhausted and hence would drop connections to the DC and the general connectivity was affected as well.

Resolution

  • I opened up my iSCSI initiator on the server and noticed a redundant LUN stuck in a reconnecting state. I had introduced a lun from my Netapp to this server to act as a D drive, on which I have my Netbackup installation. So  I removed that reconnecting lun from the iSCSI window as it was no longer needed and also removed it from my Favorite targets window within the iSCSI initiator window.
  • After that just to be safe I modified some registry entries as well to increase the TCP buffer, I dont think his was necessary after I fix the reconnecting LUN in the iSCSI initiator but I was fed up with my backups not completing so I put these keys in anyways.

Go to HKLM\System\CurrentControlSet\Services\Tcpip\Parameters and under that make 4 DWORD vaules as you see below and restart the server!

Once you do this, hopefully the server will no longer lose the TCP connection and you will not get the "No Logon Server Available" message when you try to logon.  I believe this was an affect of a Microsoft patch that was released recently as I have read this online. I am sure what the patch was or the number though.

Hope this helps!

-Alstar

The host does not have Netbackup Host-ID based security certificate installed

I recently upgraded my Netbackup 7.6.1 installation to 8.0, purely because the older version is end of life and not supported by Veritas. I absolutely love  the earlier 7.6.1 version, it was light and had a very good user interface. The newer 8.0 is a horrible java based bullshit what they have done in my opinion and ruined a simple application. Anyway its's enough of my rant about the 8.0 version as you guys are not reading this post for that!

So after performing an in-place upgrade from 7.6.1 to 8.0, when I tried to open up my console I was presented with the below error.

I clicked the HELP button as you see above and its talks about generating a new certificate and then installing it. My initial thought was, "Ok this looks simple enough". However it soon turned into a nightmare! I just couldn't get the first command to work to get a certificate and it would just hang on me and also give me crazy client/server handshake errors. One of the tech notes online mentioned to check if the Netbackup Web Management Console service is running and it was but no luck.

bin>nbcertcmd.exe -getCACertificate

After working with Veritas for a day and a half we figured out a solution. It does ease the pain as it bypasses the certificate authentication and lets you inside the console. So here is what you need to do which involves creating a string value in the server registry. If you think you are bold enough to venture into the registry follow the below steps!

  • Open Registry by typing Regedit
  • Navigate to HKLM\Software\Veritas\Netbackup\CurrentVersion\Config
  • Create a String value  under Config called BPJAVA_ALLOW_LEGACY_CERT_FALLBACK
  • Set the value to 1
  • Restart your Netbackup services

Hopefully after you restart and then try and open up the god-forsaken 8.0 admin console and enter in your credentials you should no longer get that certificate error and it should let you in the admin console.

PS: I hate Netbackup 8.0 and I hate JAVA!

-Alstar 

Unable to update Vmtools on a VM - Vix Error Code = 21009

When updating the vmtools on a virtual machine, if you encounter the below message it most likely means you have issues with the security permissions. I have seen users also talk about how the VmwareToolsUpgraded.exe under C:\Windows\Temp has read only permissions and that causes it. In my case this wasnt the issue.

veer.JPG

What you need to do is shut the VM down and go to the below settings

  • Right click edit settings
  • Click on the Options tab on top
  • Select the General option from the below table
  • Then click on Configuration Parameters
  • At the end add a Row by clicking on a button called "Add Row" and enter the below configuration line and then start the VM.

-Alstar

Fun with Powershell - Retrieve space utilization from Netapp systems

Alright, so my PowerShell adventures continue and this time, its taken me to the Netapp world. Now I am by no means a Netapp guru or claim to be one. This report was requested by my storage team and I wanted to explore this with PowerShell!

First in order to run any netapp related commands, you need to down the Netapp Tool Kit for Powershell which add a bunch of cmdlets. Click here to download that!

Once that is done, all you need to do is run the below script in PowerShell against your Netapp controller and you should see a neat html report generated in  the location specified in the script. I have also included the CSS to make it look pretty!

I have highlighted the values in bold that you will have to change as per your environment and once done, you should be one your way.

Import-Module DataONTAP

Hello, World!

$controller = Get-Content C:\powershell\Scripts\Netapp\controller.txt

$vservers = Get-Content C:\powershell\Scripts\Netapp\vserver.txt

$report = "\\server1\d$\wamp\www\main.html"

ConvertTo-Html –title "NDMP REPORTS" –body "<H1>NDMP REPORTS</H1>" -head "<link rel='stylesheet' href='http://meassets/css/style.css'>" | Out-File -Encoding unicode $report

ConvertTo-Html –title "NDMP REPORTS" –body "<H4>Date and time     $(get-date)</H4>" | Out-File -Encoding unicode -Append $report

$password = ConvertTo-SecureString -AsPlainText -Force "hello123"

$credential = new-object management.automation.pscredential "admin", $password

$one = foreach($control in $controller)

{

Connect-NcController $control -Credential $credential

ConvertTo-Html –body "<H2> $($control.ToUpper()) </H2>" | Out-File -Encoding unicode -Append $report

ConvertTo-Html -Body "<H4>$(Get-NcSystemVersion)</H4>" | Out-File -Encoding unicode -Append $report

$main = get-NcAggr |Select-Object Name,State,@{n="Total Space in TB";e {[math]::round($_.TotalSize / 1TB,2)}},

@{n="Available Space Remaining in TB";e={[math]::round($_.Available / 1TB,2)}},@{n="Utilized Percentage %";e={($_.Used)}}| ConvertTo-Html | Out-File -Encoding unicode -Append $report

$main2 = get-ncaggr| Measure-Object -Property Available,TotalSize -Sum | select-object @{name="Name";e={$_.Property}},@{name="Size in TB";e={[math]::round($_.sum / 1TB,2)}} | ConvertTo-Html –body "<H2>Summary: </H2>" |  Out-File -Encoding unicode -Append $report

ConvertTo-Html - Body "<br>================================================================================================</br>" | Out-File -Encoding unicode -Append $report

$global:CurrentNcController = $null

}

$one

Send-MailMessage -SmtpServer mail.smtp.com -To  you@yourcompany.com -Subject "AWESOME NDMP REPORTS" -From  you@yourcompany.com -Priority High -Attachments $report

-Alstar

Fun with Powershell - Find serial number of physical servers

I have to do this quite a lot for my physical HP servers, and previously it was quite a time consuming task. So here is a small script which might make things a little easier.

$servers = Get-Content C:\powershell\Scripts\SerialNumber\servers.txt

foreach ($server in $servers) {
Get-WmiObject win32_bios -ComputerName $server -ErrorAction SilentlyContinue                         -ErrorVariable ProcessError | Select-Object pscomputername,serialnumber | Sort-Object PSComputername 
if($ProcessError) {
Write-Warning -Message "Something went wrong, with the server $server!!"
}
}

The good  thing about this is that, its got error handling in place. So if a script cannot find the information of a server, it will display a warning message with the server name, instead of spitting about that ugly red message in powershell by default.

-Alstar