Update on WHS2SmugMug

Back in January i said I’d get back to writing this Windows Home Server Add-In.

Its now June, 6 months later.  For 3 of those months my camera was back at Nikon being repaired. So I took exactly zero pictures during that time. Its now back and I’m bracing myself for the flood of pictures. I carry 26Gb’s worth of memory cards with me, so I nearly always end up over doing it.

Which brings me to the Add-In. I’ve set up a Codeplex page here. And I’ve made a few check-ins. This is not even pre-alpha code. Let me explain.

A few weeks ago, I asked, via Twitter,  Omar Shahine if I could use his Smugmug Uploader code. now I’m a great fan of the Uploader. I’ve used it for every single uploaded to SmugMug.

So Omar kindly emailed me his code.

So what you will find in the Check-ins, should you be brave enough to take look is Omar’s back-end code sans any UI as part of a WCF service. The WCF service is hosted by a Windows Service (imaginatively called “UploadService”). My plan ( cunning or not) is to have the UI in the Console communicate with the uploader process via WCF. There are other methods, but WCF gives me incredible latitude when it comes to moving data back and forth.

So the Home Server Console tab will simply be a UI for uploading stuff. Instead of Remoting in and using Omar’s uploader. This is an intermediate goal.

My ultimate goal is actually to have a “Smugmug” folder and under it a folder for every Smugmug Album in your account. the above mentioned service will monitor those folders for changes and replicate those changes to Smugmug.

So I’m building now with such a convoluted architecture with a view to where i want to get this Add-in to.

So hopefully I can get the Add-in working soon.

I’m a pretty good test case for this, but I will need testers for it.

Watch my twitter account or my FriendFeed account for updates

PS If you’re asking why I’ve not moved blogs yet, I’m waiting for the next Oxite Release first.

Apple Tv – How It Fits In…..

So i got me an Apple Tv last week, more on impulse than as part of any rational thought, as such things go.  i was planning on getting one, but only later this summer.

But such is life….

So its sitting there on my Tv cupboard, replacing the DVD player that does not work any way…. surrounded by my impressive collection of DVDs. Which do me no good sitting on the shelf instead of on my Apple Tv hard drive….

The solution to this is, of course to back up your DVDs to disk. I use DVDShrink 3.2 (the last version ever released), or DVD Decryptor. both of which leave you with a folder full of .vob files ( usually  a VIDEO_TS folder).  This will strip copy protection from the files.

These files do us no good. Apple Tv will not take .vob files. But, and I must admit my irritation here, Windows Media Center does ( serves me right for being a turncoat ).

So the next step is to use Handbrake ( for Mac and Windows) to take your backed up files and turn them into Apple Tv readable m4v or mp4 files. These files can be read across iTunes, Apple Tv, and your video capable iPods and iPhones.

Now this conversion process is eyewateringly long. I’ve resurrected an old laptop ( Intel Celeron M 1.4 ghz, 256Mb RAM). The average time is about 10 hours – at 4.3 frames per second. My 3ghz Pentium 4 with 2Gb RAM is actually only slightly faster, confusingly enough. in any case, i need the desktop and no the laptop, so the old and busted laptop is doing the conversions.

The 10 hours conversion does not bother me much. I set it up to go in the morning, and its usually almost done my the time I’m back home in the evening.

All this media is being stored on my Windows Home Server, with a normal install of iTunes being the server. I’m not sure if Firefly Media Server will work with Apple TV, but if it is, I’ll install that instead.

Finally, I use MetaX 1.0 for Windows (theres a Mac version, too) to add in metadata – pic, sypnosis, direcots, actor, etc…. This is amazingly useful and saved me loads of time….

Now my music library is full of duplicates (my fault for leaving the “Copy to iTunes library” option on when i import stuff), So I’m slowly going through it and deleting the duplicates (and saving space at the same time).

My Photos are spread across a hundred folders, so i have to get some sort of organisation going so i can sync them to Apple Tv and show them off on my Tv .

I have SageTv running on the same server. Their conversion process for Apple Tv takes just as long and won’t work – sigh. If I could just get Sage to save the files in m4v format, i could sync my recorded tv directly to my apple Tv without any (hours long) conversion in between.

Update 14-10-09: The latest version of SageTv does indeed convert to m4v perfectly. Works wonders, but the time is still rather long. I reccomend turning off everything, defrags, AV runs (and temporarily, backups) if you have a long queue of files to convert. Right now I have 16 files to convert all of which are about 2- 3 hours long. Usually two a day. So for this week, I have turned off the defrag passes. SageTv actuall scales its CPU usage remarkabely well in these situations.

If i could, i’d have uploaded everything to EC2 for encoding

Thanks to the guys over on Friendfeed for help with this.

Dvd backup software thread: here.

Metadata thread: here.

Windows 7

Yep. I’m writing this from my Windows 7 VM (on Virtual PC 2007 SP2).

Performance wise, The setup inside of the Vm is making it sluggish. But of the gig of RAM its got, its only using 32%. Which is notable. Vista beta 2, on the other hand) on the same machine in a dual boot configuration used up 80% (of one gig of RAM) standing still.

Talking of performance, I’ve backed the VM up to Windows Home Server. It took all of 20 minutes. Which frankly surprised me. given the fact that this was a new OS running under a VM.

So I’m inclined to wonder exactly how similar to Vista is 7, file wise? Since WHS only copies to the server files which it does not have a copy of (or a version of). Or, it could be that 7 is optimised for WHS to backup (Which makes sense on a number of levels, but not to the European Union).

The other thing i notice is the new taskbar. I’ve grown used to the Vista taskbar for some reason or other, but this is a pleasant change. The fact that the  task bar items can be configured to show application names or not, is really neat.

They do, however get confused with the buttons in the Quick Launch bar quite easily.

The UAC logo has changed colour, to yellow and blue, in keeping with the OS colour scheme. The UAC prompts themselves are worded differently.

The absence of a sidebar is nice. And I hope that the performance hit that running Sidebar produced is gone too. Gadgets are still there, just in the background and way less conspicuous.

Its quite a please feel to the whole OS. Does it feel like Vista?? A little. Its familiar territory. But In truth, I’ve yet to explorer the OS thoroughly. So that answer will have to wait.

One thing that is defiantly different is that Google Chrome 1.0 looks different.its a dark Blue instead of alight blue.

Talking of web browsers, i decided to install IE8. Which didn’t install. It didn’t recognize the OS for some strange reason. Must try again cause I hear that a few people have managed to do it.

I must say that I’m impressed enough to be considering upgrading one of my Vista machines to Windows 7.

This Beta 1 makes me look to Beta 2 and Release with a lot of hope that Microsoft have learned their lesson of the Vista Release debacle.

The one thing that no ones said anything much about is the WinFS file system that Vista was supposed to ship.

With Sun’s ZFS redundant file system, Microsoft are lagging behind. Even OSX has ZFS built in ( it has to be enabled with some obscure command line tricks, but its there).

Even if Microsoft released a separate beta version with WinFS, I’d be happy. 

NTFS is old. Time to innovate it.

Back and ready to rock

So I’m back from holiday (Florida – no ride queues and was great).

And I’m raring to go .

To start the year off, WHS4Smugmug development will resume ( after a year of being busy and feeling guilty). There’s a great series of posts on Add-In development on the tentacleBlog:

• Part 1
• Part 2
• Part 3

And I’ll be using them to bootstrap development and hopefully get moving.

One area I’m worries about is the file/folder structure that it’ll pull the photos from and send them to Smugmug.

So I’ll crowdsource this problem. Please leave a comment on how you organise your photo folder. Thanks.

This’ll help me with the 8000 photos I took in Florida.

Hard Drive Troubles

This picture was taken using WinDirStat to image the drive. Great utility, by the way. Its part of my Software Keychain now.

See that yellow area over there? Its 3.5Gbs of space that are used, but not accessible via Exploder and thus the Windows API.

Before you ask, I already tried RootkitRevealer from SysInternals, but it doesn’t work on Vista.

It was 63Gbs yesterday. Here’s what I did:

Then I decided that if I restore from a Windows Home Server backup, the unknown space shouldn’t be there as its effectively invisible to the Backup service at the time of making a backup.

Which I did. After booting up, it didn’t work.

However, this morning chkdsk started as part of the boot sequence, no idea why. And the hole in my hard drive is gone  reduced.

Whew. I was contemplating a complete rebuild of my laptop.

Out of curiosity I ran WinDirStat on my Desktop.

Here’s what I got:

Again, there’s a 30Gb hole in the hard drive.

I ran chkdsk.exe and got a slightly tidier picture, but with the 30Gb hole still there.

Looks like I’m going to have to restore it from WHS as well.

WHS Add-In: WHS2SmugMug – Update

I’ve neglected this project for a while mainly due to me being so busy with other stuff.

The hiatus has actually done the project good as feature creep was threatening to de-rail the thing the last time I had a look at it.

So I’ve cleaned up the requirements for the data to be stored locally. I’ve eliminated just about everything I can pull from SmugMug leaving me with a nice, clean object model to work with.

I was also struck by the fragmenting of the project into three – scheduled, service and WHS Console Add-in.

While this seems logical, it is a bit over the top. So I’m dropping the scheduled uploader and having uploads handled by the service.

Work is progressing nicely and I hope to have a working  service app soon ( if not a console add-in).

I’m now using the SmugMugAPIWrapper from Codeplex. Its MIT Licenced so WHS2SugMug will have to be too. This library is one that I can actually use without looking at the source as its built to use the current SmugMug API, so no worries there.

As with my Windows Live Writer add-in, I’ll host the project on Codeplex as soon as there is a release-ready codebase.

WHS: Virtual Server

As I promised, I’m posting a How To for installing Virtual Server on WHS.

A word to the wise:

I’ve a 2.8Ghz Celeron D with 2GB RAM to run this on. Virtual Server can provision processor usage to an extent, but it uses RAM like there’s no tomorrow. Don’t forget that it has to play nicely with WHS (specifically DEmigrator.exe that burns CPU cycles) and other stuff like defrag passes and anti-virus.

Since WHS is built on top of Small Business Server,  the underlying OS is essentially the same. Hence no compatibility issues.

Installing Virtual Server is relatively straight forward.

First, download Virtual Server 2005 from here.

Then Download the Service Pack from here.

Open a Remote Desktop Session or use the Advanced Admin console tab to access the WHS desktop.

Now, its up to you whether you want to install the program files to C drive or D drive. Its worth noting that the Virtual Machines are stored separately in a location you specify on a per VM basis.

Once the install( including that of the service pack) is completed go to Start -> All Programs -> Microsoft Virtual server and hit the Administration Website shortcut.

You get this:

Note: I was accessing this remotely so had to prefix the user name with “server\”.

Otherwise, these are your WHS credentials that you use to access the console or remote in to WHS.

This is the webpage you get to (click for a larger version):

As you can see I have two virtual machines listed. One of which is currently  running.

I find that its easier to manage the server remotely, so copy the web site shortcut from the Start Menu to a network share. This now allows you to reach the web site from any connected PC.

If you’ve done this, close Remote Desktop and try it. The Default IIS settings that the install configures for you should be OK.

Setting up a Virtual Machine couldn’t be simpler:

First we want to set the default location of our Virtual Machines. Go to Server Properties at the bottom of the Sidebar and click on Search Paths. Change the Default virtual machine configuration folder to your desired location. I’ve been using a network share with replication turned off.

You can also set default paths for ISO’s that you will use. These will show up when you configure the VM’s Cd/DVD drive.

And press Ok when you’re done.

Hit Create under the Virtual machines section of the sidebar:

Enter all the info on the screen.

The RAM that the Virtual Machine is assigned is occupied as soon as the VM starts up. So be careful when doing this.

You have a choice of creating a new Virtual Hard Drive or attaching an existing one. make sure that the size of the hard drive is enough for your needs. Expanding it later can be a little difficult.  And choose a SCSI bus if you intend to have multiple VHDs attached to the same machine.

The VHD actual file size increases as you add data to it. It stops at the logical size of the VHD. So a 80Gb VHD can no be larger than 80Gb on disk.

You can also choose to create the VM without an attached hard disk.

Pressing “create” takes us to the Vm config page:

You can also reach this page by selecting your VM from the Configure menu under the Virtual Machines Section of the Sidebar.

This is where you change items such as memory, hard disks, CD/DVD ROM drives, Networking, SCSI Adaptors, COM and  LRP ports.

Each option takes you to a new page were you configure settings specific to that area.

Its worth noting here that the VM needs to be Shutdown for some operations. But you can still change the location of the media that the CD/DVD ROM drive captures while the VM is running.

You can also set the VM to start up automatically with WHS by going to the Server properties.

With my Windows Server 2003 VM running constantly, the WHS automatic restarts could be a problem. All I do is check the box, input the account details, set a delay ( in my case, 600 seconds) and tell Virtual server to save the VM’s state when WHS shuts down. The delay is actually a pretty nifty feature as it allows WHS to initialise itself, bringing all its processes online, before starting the VM.

One more thing I have to cover for running this on WHS is Resource Allocation:

You’ll find it the bottom of the sidebar.

As you can see my VM has 50% Max Processor capacity to play with. This protects the WHS processes from being starved of resources.

Once you’ve got your VM set up and you’ve installed you software all you have to do is remote in using the Remote Control facility the web site provides you with ( its good for the initial setup such as enabling Remote desktop Connections and so forth). All you do is double click on the VM icon  on the front page.

I also recommend installing VM Additions (the ISO for it comes with Virtual Server and is a default option for the CD/DVDROM drive) that will improve the way the VM behaves within the Virtual Server environment.

Additionally, this post probably scratches the surface of what’s possible with this.

And I’m sure I’ve got a few things (unintentionally) wrong along the way so its not fool proof, so YMMV ( Your Mileage May Vary).

Happy Hacking…

Subversion Source Control

This post has been bouncing around my head for sometime since finding the time to write has been a little hard.

The fact is that the whole Subversion vs Git vs TFS vs [insert preferred system here] debate can get to the point where its quasi-religious (as are numerous other tech debates – windows vs mac being one of the notable ones).

Now I, bad developer that I am, have only recently started using Subversion. The need to sync source code between my desktop and my laptop outgrew the copy/paste via Windows Home Server share approach.

The fact that Subversion is free is also a big deal maker ( compared to, say Team Foundation Server).

I have been using it on another free product, namely Netbeans 6.01 for my Java development. Its nothing major, only university projects. 

The fact is that Netbeans comes with Subversion and CVN support baked into the IDE. Additionally, Netbeans uses its own Local History feature to keep track of your files wether you’re using formal source control or not. Every time you build your project it makes a commit to your Local History. This means that one has both Local history for the small changes and Subversion for the big changes.

The fact is that having it baked into the IDE makes all the difference – it allows one to interact with the code in revisions directly. Let me explain. One can make line-by-line rollbacks from your previous versions wether you are using Local history or Subversion or both. Powerfully, the rollbacks count as changes to your code and are committed right back to the database.

(I plan to do a post on the value of a good IDE soon as well)

In the (Subversion) case above, the local copy is out of date (on the right) and the remote changes (on the left) are shown in blue. The changes were made on my laptop, committed and I’m comparing them against the desktop copy of the files.

By clicking on the blue arrow one can insert the changes into the local copy with out inserting all the changes. The same applies for replacements (in cases where the lines have been modified rather than outright replaced).

The green area in the local copy no longer exists and by clicking on the red cross one can delete the highlighted lines.

Local History works in exactly the same way.

This is a small example and one needs to actually use it to understand the power of the concept.

Now, the fact is that the vast majority of my coding gets done in Visual Studio. And Visual Studio , the TFS edition aside, does not support source control. Period.

However, VisualSVN actually has an Add-In for Visual Studio that brings this functionality into the IDE. The problem is  that the Add-In isn’t free. Its $49 per licence. Personal licenses are restricted to one per order ( but not corporate licenses) and open source projects qualify for free licenses. And, no, I haven’t yet decided whether to get a license.

I agree with Jeff Atwood that software such as Firefox should take the most popular Add-Ins and fold them in the main code base. Which gives rise to the question that if Source Control is so important and popular (is it for non-TFS Visual Studio users????), why doesn’t Microsoft add it in?

I mean its only Subversion. Surely Microsoft must have some pretty good selling points related to why TFS is, like, totally better than Subversion ( ). That takes care of the “They already have source control” pundits.

Source Control is simply good programming practice and if Microsoft is serious about attracting people to its platform (and the freely available express editions shows that they are somewhat serious), they should provide it.

Contrast Visual Studio to Netbeans and that’s enough said on the subject.

Now the easiest way to install Subversion is to head over to Jeff’s blog post about it. If you don’t subscribe to Jeff’s blog, i suggest that you do – its invaluable.

I went the VisualSVN server route which set everything up with the addition of  the ability to browse the server using a web browser (it does this by using Apache). To avoid Apache conflicting with Windows Home Server, its running on a Virtual Server virtual machine ( that image runs a few other odds and ends). Setting up Virtual Server on WHS is itself the subject of another blog post. It took me literally 2 minutes.

Before letting you go, Scott Hanselman has a great podcast on Subversion vs Git for Source Control and it comes highly recommended.

Windows Home Server: Resource Deconfliction

As more and more Windows Home Server Add-Ons are introduced, WHS becomes more and more like an appliance than a piece of software (and hardware).

More and more Add-ons mean that we ask more and more of our systems. These demands mean that finite resources have to allocated and shared with the WHS software itself.

DEMigrator.exe comes to mind ( the magic behind folder duplication). Since DeMigrator does not actually have a front end ( short of turning off folder duplication), it is impossible to pause or stop it when its running in favour of something more urgent. Granted we could change our backup window, but this is not always convenient or possible.

What WHS needs is some way of managing resources on a much more granular level than process priorities. By that I mean that WHS makes  a logical guess as to what process(es) need to run now  and what processes are less immediate.

So if I use SageTV to record show x at time y and a defrag ( or other processor intensive program) is scheduled to run at the same time, we need resource deconfliction to kick in and sort it out. We can do this in one of two ways: either throttle back the proccessor intensive process or re schedule it ( if the drive isn’t very fragmented a missed defrag pass wont make much of a difference).

Naturally, we can’t expect this souped up task scheduler to be able to handle every occurrence of every program. this means that WHS would simply notify the offending process(es) of the situation and it would be up to the program to implement a responsible and reasonable strategy to handle that.

If you’ve got a high end system running WHS, this discussion isn’t very dramatic. But between backups, defrags, virus scans, DeMigrator, SageTV  and others ad nauseam  ( even automatic Windows Update needs to be able to safely restart) jockeying for resources, something needs to manage this safely and well.

Essentially, this is bringing WHS closer to the headless system originally envisioned. It would save me a lot of Remoteing in every day.

Before we finish, let take a look at the specs for the WHS systems commercially available from HP, etc to get an idea of exactly what resources are available.

The Microsoft minimum spec is 1Ghz and 512Mb RAM and 1x 70Gb drive.

The recommended spec is 64-bit Compatible Intel Pentium 4, AMD x64 or newer with 512Mb Ram and 2x hard drives with a 300GB primary disk.

	CPU	RAM	Hard Drive
HP Media Smart	AMD 1.8 GHZ 64-bit Sempron 3400+ processor	512Mb	2x 500Gb
Norco DS-520	Intel Celeron M 1GHz	1Gb	1x 250Gb
Piranha Home Server	Intel Celeron 430 (1.8GHz, 512KB, Conroe)	1GB	2×250Gb
T2-WHS-A3 Harmony Home Server	Intel Celeron 220 1.2GHz	512Mb (1Gb Optional)	1 x 500Gb (1Tb/2Tb Optional)
T7-HSA Harmony Home Server	Via C7M “Esther” 1.5Ghz	512Mb (1Gb Optional)	1 x 500Gb (1Tb Optional)
My own homebuilt system (Dell  Poweredge SC440)	Intel Celeron D 2.8Ghz	2GB	1×160Gb 1×400Gb 2×750Gb

I think this is a pretty representative sample of the entire range. You can get the reviews on these servers and others from We Got Served Hardware page.

NB. The extra possibilities of multi-core  64 bit machines allowing true concurrent execution are mind boggling.

Hardware Upgrades- Vista and WHS

I just upgraded my main PC from XP to Vista  Business. And stuck a new graphics card in.

Before we discuss the Windows Experience Index, the only down side of the upgrade is that there are no Vista drivers for the front ports ( USB, SD, etc)  or the built in sound card from Compaq. None. Their driver downloads page for the model ( SR1629 UK) essentially says “best of luck” for those upgrading to Vista.

In fact, I suspect that the model number of my PC is different to that on the page as the picture does not show the front card reader and the drivers don’t show up for it. Any Suggestions???

This is extremely inconvenient, as you might imagine.I’ve lost 3 USB ports and  a 9-in one card reader. The sound issue was fixed by buying a cheap SoundBlaster Creative card.

I bought a  Radeon HD 3650 PCI- Express card.  512MB of GDDR3 RAM (that has a 1.73GHZ clock). A 790Mhz engine clock. Crossfire X support as well as full HD. The graphics that come out of this thing are amazing.

I bought it mainly to play Flight Simulator X on it and it rocks ( though I’m still adjusting the settings to get the best combination of  graphics and speed/playability).

Here’s the System information page with the Windows Experience Index:

And here’s the breakdown:

I must say I was expecting a dramatic improvement, but not by THAT much. Can’t remember what the original score was before I put the card in, but it was pretty dramatic.

There is a pretty in-depth discussion of the Index on the Vista Team blog here.

Moving on.

My Windows Home Server has been screaming for the last week that the USB drive had failed ( its connected and turned on and has been tried on multiple USB ports on multiple PCs). I’m not sure precisely what happened but I’m a hard drive short and have 200Gbs of space left.

So I’ve ordered two Samsung SpinPoint F1 750GB SATA-II drives. One to replace the failed USB hard drive and the other to replace the aging 80Gb IDE drive. Since there is the hard drive replication feature, I’ve gone down the path of more drives rather than higher drive capacity.  This should take my total drive capacity to about 2 TB, which is plenty. I go through hard drive space rather fast, mainly cause I use SageTv to record Tv and virtual server to run the occasional VM ( I have a 44 GB VM, to give you an idea of size).

I got all this great hardware from a UK company called Overclockers UK. Great customer service. They have no problem dealing with returned items (I had to return a AGP card because my system was PCI-E). They are quite reasonably priced and have some really amazing specials quite often.

My hard drives were shipped 40 minutes after my confirmation arrived in my inbox. And that really is amazing service. If i need hard ware, they’re my first port of call. Well done guys!!