Oh Shit!!! The Data is Gone…. Or Is It?

Yep, I screwed up, I made assumptions, didn’t double and triple check things, and made a mess of something I was working on, professionally none the less. I did fix it, but it was a stupid screw up none the less. The irony is not lost on me about how I harp on about backups regularly to everyone.

The other day I ended up with one of those Oh!! SHIT moments, I was migrating an older 2012 R2 file server to 2016 and whilst I was doing this I decided to kick the old server that was due to return for the leasing company from the Failover Cluster it was in. As standard I paused the node, removed all the references from it and I then hit the evict function on the node to evict it from the cluster, and it was from this that it all went to shit, doing this whilst doing migrations, what the first part of my mistake. What happened the Failover Cluster borked itself and crashed on the remaining servers, and it would not restart, to this day I have not got it to restart.

After spending an hour or so trying to get the cluster to restart, I relented and went to the backups to restore the offending server. Hitting the backups I go to the server I want to store, and notice that its only 16GB WTF!!!! the server should be several TB in size (it is a file server after all).

Upon further investigation it seems that I was missreading the backup reports and the old server, which has the same name on the old Hyper-V Cluster as the new server does on the new implementation was not getting backed up, it was the new one. I misread the report and assumed that it was backing up the old server, mistake number 0 (this had been happening for the 6 weeks before the backup failure) and the old restore points being more than our retention limit were gone. Ok I will hit the long term off-site backups might take a while but the data is safe, well it was not or so it seems, the other technican at the offsite location had removed the offsite backups for the fileserver from the primary site. Why, because they were taking up too much space on that site’s primary backup disk (The storage at each site is partitioned to provide onsite backup for each site, with the second partition being the offsite backup for the other site).

Damn, so this copy of the data is the only one.

Ok so I killed the cluster server that everything was on, and using the old evicted node I rebuild a single node “cluster” and mounted the CSV, mounted the VHDX and everything appeared as it should. Whoo Hoo access to the data, well not so fast there buddy.

After moving some data an error popped up stating that the data was inaccessible, ok no problem loss of a single file is not a real issue. Then it popped up again, then again.. the second Oh! Shit! moment within several hours.

2017-02-02 - Dedupe Error

I recovered and moved the data I could access leaving me purely with data I couldn’t. I tried chkdsk and other tools and after several hours I took a break from it, needing to clear my mind.

Coming back to it later I looked at the error, looked at what was happening, and recalled seeing an article on another blog about Data Deduplication corrupting files on Server 2016. With this I began wondering if it had effected Server 2012 R2, then the lightning struck deduplication, this process leaves redirects in place and essentially has a database of files that it links to for the deduplication. The server the VHDX was mounted did not know about the Deduplication, the database or how to access it.

Up until now I had only mounted the Data VHD. Now I rebuilt the server utilising the original Operating System VHDX to run the server. I let it install the new devices and boot.

Upon the server booting I opened a file I could not access before, and it instantly popped onto my screen. Problem Solved

Note to remember If you are doing Data Recovery or trying to copy data from a VHDX (or other disk, virtual or physical) that was part of a deduplicated file server, you need to do it from the server due to the deduplication database. You may be able to import the database to another server, I really have no idea, and I am not going to try to find out.

Enabling Data Deduplication on Server 2012 R2


Data deduplication (or dedupe for short) is a process which by the system responsible for the deduplication scans the files in one or more specific locations for duplicates, and where duplicates are found it replaces all the duplicate data with a reference to the “original” data. This in essence is a data compression technique designed to save space by reducing the data actually stored, as well as aiming to provide single-instance data storage (storing only one copy of the data, no matter how many places its located in).

The way this is achieved is dependent on the system used, it can be done but it can be done on block level, file level or other levels, again depending on the system and how it is implemented.

What we are going to do in this article is we are going to enable deduplication on a Windows Server 2012 R2 Server. Keep in mind this is changing data and quite possibly going to cause data damage or loss, as such make sure you have a working backup BEFORE continuing.

Firstly we need to access the server that you are planning to configure deduplication on, I will leave it up to you how you achieve that. Once you have access to the server we can begin.

On the server open “Server Manager” if it is not already open

2014-09-19-01-ServerManager

If it gives you the default splash page, simply click next (and I suggest telling it to skip that page in future by use of the checkbox) Once we are in the “Installation Type” page we need to select “Role-based or feature-based installation” and click “Next”

2014-09-19-02-AddRoleorFeature

In the “Server Selection” page select the server you want to install the service on (commonly the one your using), Click “Next”

2014-09-19-03-SelectServer

 

Next up is the “Server Roles” page, here is where the configuration changes need to take place. In the right had list of checkboxes (titled “Roles”) scroll down till you see “File And Storage Services” then open “File and iSCSI Services” then further down the page check the “Data Duplication” checkbox. Click “Next”, accepting any additional features it wants to install.

2014-09-19-04-SelectService

In the “Features” page simply click “Next”

2014-09-19-05-IgnoreFeatures

On the “Confirmation” page check you are installing what is required and click “Install”

2014-09-19-06-Install

Wait for the system to install, and exit the installer control panel, restart if your server requires it.

Upon completion of the install and any tasks associated with the installation re-open “Server Manager” and in the left hand column select “File and Storage Services”

2014-09-19-07-ServerManager

This will change the screen in “Server Manager” to a three column layout, in the middle column select “Volumes”

2014-09-19-08-ServerVolumes

With the volumes now displaying in the right hand of the three columns, right click on the volume you want to configure deduplication on and select “Configure Data Deduplication”

2014-09-19-09-ServerVolumesRightClick

This will bring up the “Deduplication Settings” screen for the volume you right clicked on. Unless Data Deduplication has been configured before, the “Data deduplication” will be “Disabled”.

2014-09-19-10-DuplicationSettings-Initial

As I am configuring this on a file server, I am going to select the “General purpose file server” option, and leave the rest as defaults. I am then going to click on the “Set Deduplciation Schedule” button

2014-09-19-11-DuplicationSettings-Enable

The “Deduplication Schedule” will now open. I suggest checking the “Enable background optimization” checkbox as this will allow the server to optimise data in the background. I also elected to create schedules to allow for more aggressive use of system resources, the first one allows for it to be done after most people have left for the day, and before the servers scheduled backup, the second one allows it to run all weekend but again stops for backups. Please note that these settings are SYSTEM settings and apply to all data deduplication jobs on the system, and are not unique to each individual deduplication job

Click “Apply” on the “Deduplication Schedule” screen, and then “Apply” on the “Deduplication Settings” screen, this will drop you back to the “File and Storage Services > Volumes” screen, and you are now done, Data deduplication is configured.

Have fun, and don’t forget that backup

Justin

Using Internet Information Services (IIS) to Redirect HTTP to HTTPS on a Web Application Proxy (WAP)Server

For those of you who do not know, Microsoft’s Web Application Proxy (WAP) is a reverse HTTPS proxy used for redirecting  HTTPS requests from multiple incoming domains (or subdomains) to internal servers. it does however not handle HTTP at any point, which is a failure in itself, I mean it would not be hard to add a part of the system where if enabled it redirects HTTP to HTTPS itself, rather than having to use a workaround, come on Microsoft stay on the ball here, but I digress.

As I stated the main issue here is it does not within the WAP itself redirect a HTTP request to the equivalent HTTPS address. I have played with multiple possible solutions for this including a Linux server running Apache 2 using PHP to read the requested URL and redirect it to the HTTPS equivalent. None of these however have the simple elegance of this solution which includes the HTTP to HTTPS redirect on the same box as the WAP system itself.

First of all you need to log into the WAP server and install the Internet Information Services role. Once done open the management console and you should get a window similar to below.

01-OpenIISManager

Now navigate to the required server by clicking on it, and on the right hand side click “Get New Web Platform Components”.

02-GetNewWebPlatformComponents

This will open a new web browser window as shown below, when it does simply select “Free Download”.if you have issues with not being able to download the file due to a security warning, you should see the earlier blog here to see how to enable the downloads. Download and install the software via your chosen method.

03-FreeDownload

Once it is installed a new page will appear, this is the main splash page of the Web Platform Installer

04-WebPlatformInstaller5.0HomeScreen

Using the search box (which at the time of writing, using Web Platform Installer 5.0, is in the top right hand corner) search for the word “Rewrite”. This will then display a “URL Rewrite” result with the version number appended to the end (which at time of writing this article is 2.0) and click the “Add” button to the right of the highlighted “URL Rewrite” line,

05-URLRewriteAdd

This will change the text on the button to “Remove” and activate the “Install” button the the lower right of the screen, click the install button.

06-URLRewriteInstall

Clicking this install button will bring up a licensing page, click the “I Accept” button (assuming of course you do accept the T’s & C’s)

07-LicenceAcceptance

You will then get an install progress page

08-RewriteInstallProcess

Which will change to a completed page after it is done, so click the “Finish” button in the lower right hand corner

09-RewriteInstallFinish

This will drop you back to the same original splash screen of the Web Platform Installer, click “Exit

10-WPI-Finish

You will now need to close and re-open the IIS Manager and reselect the server you were working on. You should now see two new options, the first being “Web Platform Installer” which we do not need to concern ourselves with any further, the second is “URL Rewrite”,

11-IISManager-NewModule

Double click on “URL Rewrite” and open up the URL Rewrite management console, on the right hand side of this console in the “Actions” pane, click “Add Rule”.

12-AddRewriteRule

This opens up a box of possible rewrite rules, what we want to create is an “Inbound Rule” as our requests are coming into the server from an external source. Select “Blank Rule” and click the “OK” button

13-NewRule-BlankRule(Inbound)

In the new page that opens, in the “Name” field type the name that you want to give the rule, I use and suggest HTTP to HTTPS Redirect, as this tells you exactly what it does at a glance

14-NewRule-NameRule

In the next section, “Match URL” set “Requested URL” to “Matches the Pattern” (default), “Using” to “Regular Expressions” (default) and most importantly “Pattern” to “(.*)” (without the quotes). I suggest you take this opportunity to test the pattern matching.

15-NewRule-Regex Match

In the “Conditions” section, ensure that the “Logical grouping” is set to “Match All” (default) and click the “Add” button.

16.01-NewRule-AddCondition

In the new box that appears enter the following, in the “Condition input” field type “{HTTPS}” (again without the quotes, and yes those are curly braces, not brackets). Change the “Check if input string” dropdown to “Matches the Pattern” and in the “Pattern” box below type “^OFF$” (again, no quotes), and “Ignore case” should be checked. With this one I do not suggest testing the pattern, as even though this system works fine for me, this test ALWAYS fails. Click the “OK” button (mine is not highlighted here as I had already clicked it away and had to re-open the box)

16.02-NewRule-ConditionSettings

This will take you back to the new rule screen, check the conditions match as shown and then we can move on.

16.03NewRule-ConditionComplete

This is the part where we now tell it what we want to do when it matches the previous conditions, in the Action pane change the “Action type” to “Redirect”, Set the “Redirect URL” to “https://{HTTP_HOST}/{R:1}” (again, they are curly braces and of course no quotes), you can select whether “Append query string” is checked or not, but I highly recommend leaving it checked, as if someone has emailed out a URL with a query on it, but not put in the protocol headers (http:// and https:// being the ones we are concerned about) we want the query string to be appended to the end of the redirected URL so they end up where they intended to be. Finally make the “Redirect type” dropdown read “Permanent (301)” (default).

17-NewRule-ActionConfiguration

Restart the server service for good measure and there you have it you now have HTTP being redirected to HTTPS which in theory at least is on the same server. Ensure that you have ports 80 (HTTP) and 443 (HTTPS) redirected from your router to the server and the firewalls (and any other intermediaries) on both the router and server set to allow the traffic as required

Enjoy and as always have fun

Justin

Internet Explorer Cannot Download a File on Server 2012 R2

So you have just set up a new Server 2012 (R2) server, and gone to download that fine you need for the next step, only to be shown a nasty message stating that you cannot do that, as file downloads have been disabled.

NoFileDownload

Well the good thing to know is that its an easy fix, simply open up “Internet Options” go to the “Security” tab, select the “Internet” zone and Select the “Custom level…” button

InternetOptions-SecurityTab-CustomLevel

This opens up a “Security Settings – Internet Zone” window. In the main section of the windows scroll down to where it says “Downloads”, and the the subsection of “File download” (as of this writing the setting is just above half way down the options list) and simply change it from “Disableto “Enable”. Click ok and drop back to the main screen and retry that download again

EnableDownloads

If you get a warning, as shown below, simply OK it and continue on

Warning

Have fun

Justin

%d bloggers like this: