Building a Hyper-V Cluster on two W2K8 Nodes

ServerCare home

Home

ServerCare home

10/4/2008

Building a Hyper-V Cluster on two W2K8 Nodes

In this first part we'll focus on getting the infrastructure in place. We'll talk about the systems making up the nodes and the shared storage we need to build. Once that is in place, we'll focus in the next article on how to get Hyper-V clustered on this infrastructure. Note that we are not building a cluster –in- Hyper-V, as many people have written about; we are building a physical Fail-over cluster with two nodes, and on that we run Hyper-V.

My goals

The moment I read that Hyper-V was cluster-aware, I knew I would not rest until I had tried building a failover cluster running Hyper-V. It's one of those exciting technologies you just hàve to test out for yourself, not being satisfied with simply reading about it. (yes; that is a hint ;-) In fact, I had the sneaky feeling that Robert Larson (the MS blog in the link above) in his ten-step plan may have cut a few corners to expedite things.

First of all let's take a moment to realize the importance of hyper-V being cluster aware… In the past you would have several virtual machines running one some big iron, but once the big iron failed, being the single point of failure; all your machines would go down hard. As tempting as virtual machines are, this is completely unacceptable to most people or businesses. Fail-over clustering adds this nice and secure feeling of having a backup system, removing that single point of failure.

It must be nice for guys like Robert Larson who works for Microsoft consulting services, to have all this hardware readily available to build his labs with. Unfortunately I don't have such spare hardware, and no big budgets to spend. Still, for this project some dedicated hard and software is needed and I set my budget not to go over 1500 Euro, and use whatever spare or free equipment or software I can get, without compromising too much for performance reasons.

Figure 1. The bill (and configuration details) of one of the nodes

I believe that technologies such as clustering and virtual machines are ready to reach out from the enterprise level into the mainstream. You could argue that I'm following Google's strategy of using combining cheaper hardware with intelligent software; quantity in hardware for redundancy, and quality in software for availability.

In order to prove that I want to build something I can actually use; not just a simple proof of concept. While doing that I will still attempt to remain within budget. With that in mind I decided to buy some hardware to build a few identical systems to create my cluster with (rather than buying pre-built –more expensive- systems). A bit risky, as at the time of writing it's still a bit of a guess if the motherboard you are buying has a bios supporting VT. More on how to figure that out can be found here. For that reason alone you may want to consider buying more expensive machines from the MS recommended hardware list (see previous link), if you are building this for a business. All in all I have not spent much more than 1400 Euro's on two nice dual code 2.66 Ghz systems with 6Gigs of memory, leaving me with 100 Euro to buy a Gigabit switch to build my SAN backbone with and still remain within budget.

Figure 2. The two nodes; MSI Neo2 FIR main boards, Core2 Duo E6750 CPU's

This should be sufficient to build my lab with, in fact a small size business would do pretty well running a similar setup. (It's no enterprise hardware, but carefully selected and much better than what I see at some small businesses!) I would like to be able to run 3 to 4 servers on this Hyper-V cluster. Of course the real interesting part will be finding out the stability and performance of these systems. Once I'm satisfied with the design I'll make some baseline measurements using various tools such as performance monitor and see how things evolve… this should be fun!

When money is a factor running several virtual machines on one cluster sounds like a great way to save some cash and still have redundancy. A few things have changed with Windows 2008 clustering though; where in the past it was possibly to simply share a SCSI hard-drive that option is now no longer available. For more details on the requirements for a Hyper-V cluster you might want to check out Microsoft's step-by-step guide. The article you are reading now however is written in a much more entertaining style, I even have pictures and everything! So be sure to return once you've read that, to get a more down-to-earth perspective on building a Hyper-V cluster. The shared storage options left include dedicated hardware solutions such as fiber based SAN or something a little bit closer to our budgetary needs: iSCSI

More on iSCSI.

iSCSI is often called an emerging technology, and while I'm writing this I wonder if you can still call it 'emerging'; it has been around for years now. Strangely enough this great technology has not gained as much traction as it should have, but rest assured; with Hyper-V and Windows 2008 Failover clustering now advocating this, and with iSCSI client (initiator)software natively available in Windows 2003 server, Windows 2008 server, Vista and separate downloadable Windows XP I'm sure in the next few years this technology is going to break-through widely. Before it was a nice technology available to us if you needed some LAN based storage, but now we actually are seeing some very compelling technology that gains greatly buy the use of iSCSI. As mentioned above, it competes with the much more expensive fiber channel which is used at the higher end of the market. The advantages for iSCSI are that it is using technology (Ethernet, preferably Gigabit), commonly available, this means most IT people already have a sound knowledge of the basic technologies needed to maintain the SAN. Not all IT people have similar experience with fiber channel Switched fabric, HBA's (the fiber channel Host Bus Adapter) and the enterprise solutions associated with that technology. As mentioned the initiator software is natively available in the latest Microsoft operating systems, making the entry level to SAN even more accessible. These are the factors that will drive iSCSI to become a mainstream technology. When building enterprise solutions however; Fiber Channel in combination with a dedicated (hardware solution) NAS is still king, but at a price smaller businesses will not be willing to pay. For them: iSCSI is the alternative.

So what is this iSCSI? Simply put, it's a fake hard drive that you can share from a server (target) on the network somewhere, and then link to from a client (initiator). On the client the initiator software makes the 'network disk' look as if it's a real hardware disk in your system. Instead of sharing a network drive on file or network level as you normally do with a file-server, you are now sharing a disk at block level, and this opens up a whole new realm of possibilities!

Figure 3. Would you believe these last two disks aren't real? The node sure thinks they are!

It's even possible for (for example) thin clients to boot from iSCSI drives, provided the right software is in place. Ideally one would have a NAS (Network Attached Storage) system with some RAID configuration for redundancy and/or performance. Even better: get the devices or servers sharing the iSCSI resources on their own fast (Gigabit) network and you'll have your very own SAN (Storage Area Network)… I could probably throw some more acronyms at you, but I'll leave it at this for now. I decided to build a small SAN on Gigabit Ethernet to allow some performance for my iSCSI drives. Gigabit Ethernet switches are cheap now, so it should not be a big drain on the budget. If you decide to go this route, make sure to read up on jumbo fames especially if you have decided to build something more permanent than a lab situation. For now, we need to build a SAN solution!

How about Open Source?

Since we know we need iSCSI to remain on budget, we simply have to figure out what NAS to get! There are quite some options, and NAS's can come as a hardware solution, or simply software installed on (usually) windows server. Typically a NAS shares its disks using several protocols, such as SMB, NFS etc. Since we only need iSCSI we might as well only look into that. For my solution I went the server with iSCSI software route, since I happen to have some spare (older) servers anyway. You'll find there are quite some iSCSI solutions out there, open as well as closed source.

I'll throw in the spoiler right here..: If you where thinking of using open source solutions like OpenFiler (current version 2.3) or FreeNAS (current version 0.69b3), think again. Starting with Windows 2008, the cluster software requires your iSCSI target software to support something called "persistent reservation commands". These are based on the SCSI Primary commands-3 specs (SPC-3), and most open source software I know of (with the possible exception of OpenSolaris) is based on SCSI-2, and does not support these reservation commands. OpenFiler for example is based on IET . The FreeNAS target is based on the NetBSD iSCSI target by Alistair Crooks and ported to FreeBSD, and –also- does not support these command. So far for Linux and BSD…

[update May 2009]

Good news! There are a few new options available now that support persistent reservervation. First of all, as you may have read from the comments on this blog the latest nightly builds of FreeNAS now seem to work fine. Additionally, if you happen to be an MSDN or Technet subscriber (as I am ;-) you can get the brand-new Windows Storage Server 2008, with the 3.2 iSCSI Target, which of course works fine too...

[/update]

[update Oct 2009]

FreeNAS 0.7.0 RC2 (as of writing just out) now uses ISTGT, and according to this page: http://www.peach.ne.jp/archives/istgt/

it supports SPC-3...!

[/update]

If you are simply looking for a nice NAS for home use, and would like to use iSCSI to only have a few systems connect to; look no further; OpenFiler as well as FreeNAS work admirably to help you accomplish this goal. For clustering purposes however, we need to look further…

Windows iSCSI Targets

Good news on the Microsoft based front, there are two options (unfortunately not free ones) that do support the persistent reservation commands and that also support concurrent connections to the shared disk (keep in mind that for a cluster several nodes need to have access to the disk!).

Windows based iSCSI targets that support SCSI-3 persistent reservation and concurrent connections are "StarWind" from Rocket Division Software, or Microsoft's own iSCSI Software Target 3.1.

Notice that although there is a free evaluation version (the personal version) offered by Rocket Division Software, this won't work when building a cluster, as it only allows one (single) connection. A single node cluster seems a bit of a waste of time… The alternative is a 30 day evaluation version that does support concurrent connections however and would be suitable... For our purposes the 'Server' version of StarWind would be sufficient, since it support two concurrent connections, and we have only 2 nodes anyway…

I opted to use the Microsoft 90 day evaluation version of the iSCSI Target, as it gives me more time to play around with my cluster. Both solutions support SCSI-3 persistent reservation, and both support multiple concurrent connections. Both implementations of the target software are very easy to install and configure, using a very intuitive interface. Jose Barreto has done an excellent job of describing how to configure the Microsoft software.

The Microsoft iSCSI software Target is hard to find, as it is a OEM solution that is sold along (as an option) with Windows Storage Server ; a Windows 2003 server version typically sold in combination with specialized hardware as a OEM solution by vendors such as HP, Dell etc. You can't even find it on Technet.

The 90 day Microsoft evaluation version can be downloaded from: http://files.download-ss.com/storageserver.iso however.

This is the software intended for Windows 2003 Storage server, but it seems to work quite well on Windows 2003 server SP2.

Figure 4. The Microsoft iSCSI Target Management interface

One thing that was not too intuitive was the way access is granted to the initiators, which was much more straightforward in StarWind.

[update October 4rth]

Rocket Division Software graciously sponsored this article by providing a licensed copy of their server version, which has now taken the place of the earlier mentioned 90 day Evaluation version of the Microsoft target. First thing I noticed was that the logging off and on to the Target is much faster. I have not done any speed tests between the two yet, but if I find some time I might just do that. The StarWind interface is very intuitive, I did however find a few options presented in the GUI that made me wonder if Rocket Division Software have read the MS User Interface Style Guide; maybe intuitive, but definitely not standard. The iSCSI engine however has been running without a hitch for the last few days, and right now; that's my only concern.

Figure 5. More options are possible with regards to what type of virtual storage is used and snapshots of such storage in the professional and enterprise version.

We have now basically built the following setup:

Figure 6. The NAS layout

As you can see the storage network is physically separated from the LAN. The hartbeat is nothing more than a simple crosscable between the two node servers.

Now that we have the iSCSI Target up and running, we need to connect the iSCSI client (the 'initiator'). So we go take a look at the control panel of the nodes.

Figure 7. The iSCSI initiator in the control panel!

Figure 8. Initiator as found in the control panel of the Windows 2008 node.

First thing we do is Discovery, simply provide the IP number of the iSCSI Target you have set up.

If the discovery went just fine on your initiator, but there are no targets, make sure to check the iSCSI Initiators tab on the properties of your Targets (on the 'server' or SAN); to test, simply add the FQDN (Fully Qualified Domain Name) of one of the initiator machines. For example: one of my nodes; node1.servercare.nl. You can also add the iSCSI Qualified Name (IQN) which is quite similar to a DNS name, with some extra numbers added. In fact if you want to go overboard; go and run the ISNS server that can be downloaded from Microsoft. It's not required for our purposes though.

Figure 9. The MS ISNS server kind of work like a DNS for iSCSI targets and initiators

Now that the Discovery has been done, the Targets tab should show you your targets, you can simply logon to get the 'drives' you need.

Figure 10. As you can see the targets are there now!

You'll find that now, when you go to the disk manager on de nodes there are new disks to be found. Simply make them active basic disks and format them using NTFS. Keep in mind that for this lab environment we have chosen to implement iSCSI software on a Windows 2003 server that may have other functionality. If you decide to build a more permanent solution I urge you to seriously consider a NAS that is of a more dedicated nature with all the enterprise redundancy (double power supplies, RAID levels, UPS etc) in place. Believe me: once you have a number of virtual machines running from this NAS you do not want to have this storage go down.. ever… Having to reboot it because of some other software running on it would be very annoying to say the least.

Validating the cluster environment

Now that we have added two new disks to the two Windows 2008 servers making up the nodes of what will become our cluster, next step is to get the Failover Clustering feature installed (note it is a feature, not a role!). Simply do this using the Server Manager.

Figure 11. Add Features in the Server Manager

Once this software is installed, you'll find the amongst the administrator tools you now can find the Failover Cluster Management MMC.

Figure 12. The Failover Cluster Manager as found in the administrative tools.

One of the options you can choose is Validate a Configuration… This will test both nodes.

Figure 13. This is what happens if you use FreeNAS or OpenFiler…

After the test you get a very lengthy report with all the conclusions. Below you can see the part that shows that at the second attempt we have used the Microsoft iSCSI Target software.

Figure 14. The Cluster Validation report.

Once you have a report looking like the above, you have been successful at creating a Hyper-V cluster infrastructure, we are now ready to build the cluster and get Hyper-V installed! More on how to do that in the next article….

[nov 2009] update: although the building process is identical as described here, building a Hyper-V cluster under Windows 2008 R2 brings the options of having Cluster Shared Storage and Live migration; more on that here: http://www.servercare.nl/Lists/Posts/Post.aspx?ID=89

Posted at 16:00 by Paul Weterings | Category: Microsoft | Permalink | Email this Post | Comments (27)

Comments

Missing Pieces, questions

Nice article, but there are som emissing hardware pieces in your example setup:
- each machine needs two additional NICs (10/100 as I read your write-up)
- your iSCSI server isn't specified at all
- you mention 6 Gigs of RAM per system, yet list 2 per system in the build list

Also, your configuration raises a question - why four low-capacity SATA drives (4x 160 Gig) on each machine? For a cluster test system, two larger mirrored drives would likely have cost less, used less power, and provided more storage per Euro...

Also, have you looked into MS newest offering, Hyper-V Server 2008 - a free (as in beer) OS for hosting Hyper-V machines? Would it cluster as well as the Full OS your example uses?

Finally, the cluster you describe is great for testing/development/training use, but for any kind of real use the cost escalates fast - by my count you would need two Windows Server 2008 Enterprise licenses (for each machine and up to four virtual machines), a copy of Windows Server 2003 to host the iSCSI fileshare, and the unknown cost of the iSCSI target software... The price of your hardware is trivial compared to the license costs...

But again, I understand and agree, for testing/development/training use this is great - the Windows Server 2008 Enterprise OS can be installed from a downloadable ISO from Microsoft for evaluation purposes, Windows Server 2003 is a bit trickier. Could the MS iSCSI target software be run under Server 2008?

at 11/5/2008 14:52

Missing pieces... you're quite right!

WOW, you have been paying attention! ;-)

In fact you are quite right. After I had received the hardware I initially ordered I decided to beef up the servers a bit more, with future projects in mind. Please note though that this didn't add much to the total bill, as prices have dropped quite a bit since the initial purchase. Besides memory I added two cheap Realtek 1 Gbit nics (the same, which is important for clustering)

The drives are in a RAID10 config, simply because I wanted to play with that, not too much tought went into it. I guess it could be cheaper. Funny thing is that I only use half of the space available. The other node has to live with a RAID5 config.

The free beer OS won't cluster....

The iSCSI server is one of my other lab W2K3 servers, I should have mentioned that.

The cluster should be usable (in fact it is) for me. but the hardware of course is, as you point out, not worth installing the OS on. This makes sense as a Lab environment, or for testing purposes. There are by the way some Microsoft customers that do get their licenses for silly prices, for them this may actually be a worthwile setup. In real (enterprise) life one would scale up the hardware, but from an installation/software point of view that would not change the concept of what is described here.

By the way, I have a Technet subscription that I used for the software. It's not too expensive, and well worth the money, if you, like me; love to tinker with this stuff...! ;-)

I haven't tested the MS iSCSI target on W2K8, but I don't see any reason why it would not work.

Hope this answers your questions!

Paul

at 11/5/2008 21:37

hello,very good!

ok,is good,good!

at 11/11/2008 4:15

Re: Building a Hyper-V Cluster on two W2K8 Nodes

I'm glad you liked it!

Paul

at 11/12/2008 0:21

Quick Question...

Hello and thank you for posting these articles, they are very informative and helpful!

Your hardware diagram shows the two servers, a 100mb switch, a 1GB switch and a "heartbeat" connection. What excactly does that consist of? I imagine it would be another 1GB switch (and NIC cards) to allow the Hyper V/Quorum to communicate, yes? If so, then a functional system would require each main server to have 3 NIC's and the SAN support computer to have one yes? Many thanks, Mac.

at 12/21/2008 21:17

Question on hartbeat

The hartbeat link is a crosscable, nothing more! Since you don't need much speed for it, a simple 100Mbit cable is sufficient!

Ideally a system would have three nic's (as advised by Ms) however, in a lab situation you'll find 2 work fine. Never use it like that in a production environment though!

The SAN; as you state, needs one. (teaming won't work (yet) so two does not make sense right now)

I hope this answers your question.

regards,

Paul

at 12/23/2008 22:56

core help

Have you seen any blogs or step by steps on this topic, but with Core instead of the Full version of 2008?

at 2/17/2009 19:57

re: core help

Appearantly setting up ISCSI on core is a b*tch... have you checked Jose's blog?

http://blogs.technet.com/josebda/archive/2008/04/14/snw-demo-windows-server-2008-core-hyper-v-and-failover-clustering-with-screenshots.aspx

Paul

at 2/17/2009 20:39

Been running clusters since 2000 myself.

I was actually brought to your article from looking for almost the same solution. However.. In a real world Enterprise application.

Current plans consist of 2 Dell servers ( unknown models at this point). I already run on equallogic SAN and LOVE IT !! So I will opt for another for this.

My plans are to utilize the hyper-V from Microsoft compared to VMWARE's offering which is VMWARE view.

I plan to use WYSE V10L thin clients and boot wireless to the Hyper-V image. Total nodes will be 50.

Why not just do the same simple setup I;ve done for years you ask? I was actually promted into beleiving I could cross cluster ( term used ) the system and do a dual failover type cenario. Meaning.....

I will actually be running 25 nodes on one machine and 25 on the other. If one fails the other will pick up the left over nodes.

Almost sounded rediculious to me however Im suppose to join in on a video conference tomorrow and a rep is going to have me vpn into his test lab to show that he is running it in this exact scenario.

Im sure there is more pieces to the puzzle but well soon see. Clustering and virtualizing has grown leaps and bounds in the last 10 years I've been doing it so Im sure someone has some tricks up their sleeves we haven't seen or heard of yet.

Stay tuned. Ill post back with more.

at 4/22/2009 2:59

Re: Building a Hyper-V Cluster on two W2K8 Nodes

Sounds interesting! Love to see the details!

Running VMWare at one of my customers also, and quite frankly, it's doing a great job. It will be interesting to see how both solutions compete going forward. So far I've put my money on Hyper-V...

Truely interested in your experiences:
If you find the time to get your findings into a word doc with some pics I'd be more that happy to put it up as an article, in your name of course.

cheers,

Paul

at 4/22/2009 8:01

iSCSI for free!

Just an update to the comment about openfiler/freenas not supporting the required SP-3 commands for Persistent Reservations. I've just pulled a copy of the nightly build of freenas (7.4216) and that support is now available. I've built my cluster machines and run the validations and everything is happy. I don't know what the target date is for the next release, but this appears to be capable of doing the task. Sooo, if you're comfortable with the pre-release code, and for testing that shouldn't be a major issue, now you do have an open source option.

at 5/20/2009 19:15

Excellent news!

Excellent news on openfiler, I've downloaded a copy of the nightly build... and things look good.

Could it be that they are using the opensolaris target?

Thanks for sharing!

Paul

at 5/21/2009 9:48

Using 2 single servers

Hello

I have 2 HP Servers with a SAS RAID10 array each. Can I use the internal array of each server to arm a 2 node failover cluster using Windows 2008 Server Enterprise? I need high availability of SQL Server 2008, if Server-1 fails, then Server-2 must work automatically...

at 5/29/2009 14:30

iSCSI and w2k3 vs w2k8

I tried to install Windows iSCSI Software target 3.1 on w2k8 32b and it was not possible i have red on MS sites that MS is currently shipping newer version with w2k8 and its verions 3.2 so i think u can't run iSCSI Soft. Target on w2k8 using versions below 3.2. Also you need Windows Server 2003 at least SP2 better R2 with patches to get all the functions it offers.

Add to last questions no you can do failover cluster on seperate arrays. the philosophy is to have "shared" volumes to keep the data consistent (including Witness/Quorum, Database..)

I your case you will not able to keep the databasa files at the same level of consistency so if one node fails it could switch the SQL server instance to another node but it'll be upon the old database without transaction files or redo logs === catastrophic scenario.. There for shared disk are "following" the cluster group while it switch to another node to provide the lates data (or where it stopped).....

at 6/10/2009 0:00

How do you configure the heartbeat cable

Do you build yet another network with each server having a different IP on that network? How does the cluster know to use that network for the heartbeat?

Thanks,

at 6/23/2009 19:14

Using 2 single servers

What shared storage would you use? You'd need an extra system for that...

Paul

at 7/2/2009 7:48

hartbeat

Hartbeat in my case (a two node setup) was a simple crosscable. If you have more servers then you may want to consider using a VLAN for it on your switch. Or if you don't have a managed switch, simply use the same switch used for connectivity to LAN or SAN, as long as all the servers can see each-other. In my case I simply used the 192.168.x.x network (the servers are in the 10.x.x.x range)

Paul

at 7/2/2009 7:52

Re: Building a Hyper-V Cluster on two W2K8 Nodes

install

at 7/4/2009 11:38

opinion

Where can I read more about this? I've found several tutorial at http://www.rapidsharemix.com , but they didn't help me much (

at 7/20/2009 8:51

re; opinion

Technet has some excellent articles available.

Paul

at 8/3/2009 10:20

Hi!!!!!!!

Thanks for the useful information!!!!!!!!!!!!

http://forum.maxsite.org/viewtopic.php?pid=28849

at 1/26/2010 19:42

Thank You

Very useful, thank you for sharing this info.

at 3/19/2010 3:24

quorum/witness disk

Excellent article and good threads as well. In going through various articles I have come across references to quorum and witness disk and file share witness and that somehow one or another of these has to be in place in any cluster which might in the quorum/witness case mean a separate physical disk just for that function - in other words the minimum disk configuration for a cluster is two physical disks on a SAN or some other shared storage device. Any elucidation on that would be appreciated. A good link that describes in detail the formatting process once the targets are identified would be nice.

at 4/3/2010 21:16

What I can If I Use FC not Iscsi

Reall I Have Fiber Chanell Connect two Nodes (hyper -v) but The validation test for cluster give me Failed to access sector 11 on disk with identifier f78e3300 from node2
I want to Know whate i have to find in Storage manager for both servers before validate the cluster

at 7/24/2010 8:18

mylacebags

Chanel is running an exhibition of the history and manufacture of their top [url=http://www.mylacebags.com/category-118-b0-Louis-Vuitton.html]Louis Vuitton replica[/url] in New York this May, running from the 8th to 16th May 2010. [url=http://www.mylacebags.com/category-122-b0-Thomas-Wylde.html]thomas wylde[/url] Here you are bound to find details of one of Chanel's most popular models, the Chanel 2.55 quilted XL jumbo/maxi caviar black [url=http://www.mylacebags.com/category-113-b0-Fendi.html]cheap fendi handbags[/url] with gold hardware, or A28600 as this model is known. This elegant handbag, with its authentic quality calf leather is a true classic and remains top of Chanel's very popular market in exquisite [url=http://www.mylacebags.com/category-124-b0-Versace.html]versace replica handbags[/url] for the discerning elite.?Chanel 2.55 [url=http://www.mylacebags.com/category-183-b0-Juicy-Corture.html]juciy couture[/url] The single flap A28600 authentic quality [url=http://www.mylacebags.com/category-195-b0-Paul-Smith-Bag.html]paul smith bag[/url] by Chanel is also known by its brand name of Chanel 2.55. The calf leather that is featured as the exterior of this divine [url=http://www.mylacebags.com/category-186-b0-Marc-jacob.html]marc jacobs replica[/url] is described as having a caviar finish. Basically, [url=http://www.mylacebags.com/category-124-b0-Versace.html]versace replica handbags[/url] this is calf leather which has been finished with a scratch resistant surface and given a grainy texture. [url=http://www.marisabags.com/]replica handbags[/url] This authentic quality [url=http://www.marisabags.com/category-105-b0-Bottega-Veneta.html]bottega veneta replica handbags[/url] is manufactured in black and measures 33cm x 23cm x 9cm. Inside, this authentic quality [url=http://www.marisabags.com/category-105-b0-Bottega-Veneta.html]replica bottega veneta[/url] is finished with a leather lining and has a single interior slit pocket, ideal for slotting your cell phone into.?Unique Manufacture As with other Chanel models of authentic quality [url=http://www.marisabags.com/category-115-b0-Hermes.html]hermes handbags[/url], the shoulder strap consists of a braided [url=http://www.thetotebag4u.com/category-104-b0-Balenciaga.html]balenciaga handbags[/url], interwoven with a leather strap slotted through it for additional security. The rear of this Chanel 2.55 [url=http://www.marisabags.com/category-109-b0-Chloe.html]replica chloe handbags[/url] features an open pocket: the whole Chanel [url=http://www.marisabags.com/category-174-b0-Cartier.html]replica cartier handbags[/url] is closed by a single flap dropping over the main compartment and clipping shut. Additional extras include the ubiquitous Chanel dust [url=http://www.marisabags.com/category-181-b0-Jp-Tods.html]jp tods handbag[/url] for storage, an authenticity card, together with the [url=http://www.thelacebags.com/]Cheap Handbags[/url] serial number as well as a hologram sticker whose number matches directly to the number features on the authenticity card. This Chanel 2.55 [url=http://www.thelacebags.com/category-117-b0-Loewe.html]loewe handbags[/url] retails at around $2,295 although a genuine authentic Chanel 2.55 [url=http://www.thelacebags.com/category-109-b2-Chloe.html]replica chloe handbags[/url] can be picked up for a starting price of around $1,200.?What is the Secret to [url=http://www.thetotebag4u.com/category-105-b0-Bottega-Veneta.html]bottega veneta replica handbags[/url] Name?Every Chanel 2.55 quilted XL jumbo/maxi caviar black [url=http://www.thelacebags.com/category-124-b3-Versace.html]versace replica handbags[/url] with gold hardware, [url=http://www.thelacebags.com/category-117-b1-Loewe.html]loewe replica handbags[/url] or A28600 as this model is known is finished to perfection, with the attachment of the gilt chain being the final step in its manufacture. To complete each Chanel 2.55 [url=http://www.thelacebags.com/category-186-b1-Marc-Jacob.html]marc jacobs replica[/url] takes three full lambskins. This Rolls Royce of all [url=http://www.thetotebag4u.com/]designer replica handbags[/url], the figurehead of the Chanel [url=http://www.thetotebag4u.com/category-195-b0-Paul-Smith.html]paul smith bag[/url] emporium, remains Chanel's well-accorded classic since its 1955 debut – and therein lays the secret of its name: the Chanel 2.55 [url=http://www.thetotebag4u.com/category-173-b0-Bally.html]bally replica handbags[/url] was first launched in February, 1955. Since February is the second month of the year of 1955, what else could this treasured [url=http://www.thetotebag4u.com/category-117-b0-Loewe.html]loewe replica[/url] 2.55 [url=http://www.thetotebag4u.com/category-113-b0-Fendi.html]fendi handbags[/url] be called other than 2.55?

at 7/30/2010 7:02

watches

I can not image myself without my <a href="http://www.thetotebag4u.com/">replica handbags</a> and I know that many other women agree with me. Woman surely needs her <a href="http://www.thetotebag4u.com/category-122-b0-Thomas-Wylde.html">thomas wylde bags</a> by her side to carry her things around and also to make a fashion statement. Along with her clothes, shoes, <a href="http://www.thetotebag4u.com/category-121-b0-Prada.html">prada replica</a> and jewelry, the right <a href="http://www.thetotebag4u.com/category-173-b0-Bally.html">bally handbags</a> complete the final look. Choosing the right <a href="http://www.thetotebag4u.com/category-178-b0-Dooney-Bourke.html">dooney & bourke handbags</a> is like an art form that many women learn while growing up. <a href="http://www.thetotebag4u.com/category-120-b0-Mulberry.html">mulberry bags</a> A lot of our fashion is influenced by our favorite celebrities and from advertisements we see on television. We then begin to develop our own unique style, which we modified from time to time, and carry this into our adulthood as we become more fashionable.Women's <a href="http://www.thetotebag4u.com/category-110-b0-Dior.html">dior replica handbags</a> choices are always changing as their age. They choose styles that are not only age-appropriate but also in line with their social status and lifestyle. Career women are more likely to go for a sleek structured shoulder <a href="http://www.thelacebags.com/">designer Handbags</a> for a business look. Many women carry styles of? <a href="http://www.thelacebags.com/category-112-b5-Dolce-Gabbana.html">dolce & gabbana replica handbags</a> famous designer <a href="http://www.thelacebags.com/category-118-b0-Louis-Vuitton.html">louis Vuitton handbags</a> including??inspired?replica designer <a href="http://www.thelacebags.com/category-114-b2-Gucci.html">replica gucci handbags</a>?if the budget is not for the big designer name?brands.?Other woman may choose a vintage looking <a href="http://www.thelacebags.com/category-122-b0-Thomas-Wylde.html">thomas wylde handbags</a>. Whether you are?on a night out on the town of just hanging out with friends, there are many <a href="http://www.thelacebags.com/category-115-b3-Hermes.html">hermes replica</a> styles to choose for that situation.Women know what looks good on them and what does not. Younger women may prefer shoulder style <a href="http://www.thelacebags.com/category-104-b2-Balenciaga.html">replica balenciaga</a> that are more in line with their age and lifestyle while older women may go for <a href="http://www.marisabags.com/category-113-b0-Fendi.html">fendi bags cheap</a> that are more formal and classic in designs.To see <a href="http://www.marisabags.com/category-118-b0-Louis-Vuitton.html">louis Vuitton replica</a> made of denim, quilted on some non leather materials are common with young women. There are many styles of <a href="http://www.marisabags.com/category-111-b0-Coach.html">coach replica</a> with countless choices to choose from market for any particular occasion.Woman's choice on <a href="http://www.marisabags.com/category-107-b0-Burberry.html">burberry replica</a>?is always personal. Choosing the right <a href="http://www.marisabags.com/category-112-b0-Dolce-Gabbana.html">dolce gabbana bags</a> is not limited to what looks good but how?she?feels about it like if she is comfortable with it or not.?Therefore, it's a matter of choosing the right <a href="http://www.marisabags.com/category-120-b0-Mulberry.html">mulberry bayswater</a>??for her. <a href="http://www.mylacebags.com/">replica handbags</a> If you are in the position of selecting a <a href="http://www.mylacebags.com/category-124-b0-Versace.html">versace bags</a> as a gift for woman you should try to remember the types of <a href="http://www.mylacebags.com/category-120-b0-Mulberry.html">mulberry replica handbags</a> you have seen her carrying usually.If you are a man who want to choose a <a href="http://www.mylacebags.com/category-113-b0-Fendi.html">authentic fendi handbags</a> for your girlfriend or wife. Try to remember if there is any point if she had mentioned?on <a href="http://www.mylacebags.com/category-174-b0-Cartier.html">cartier replica handbags</a> style that she is interested in.Or you can ask her friends for her opinion before you make your choice. If you are in a store, the easiest thing to do is to ask the sales clerk as they may have experience for recommending the right <a href="http://www.mylacebags.com/category-112-b0-Dolce-Gabbana.html">dolce & gabbana handbags</a>. Keep in mind that <a href="http://www.mylacebags.com/category-104-b0-Balenciaga.html">balenciaga handbags</a> is not just an accessory. It plays an import part in completing women's style.

at 7/30/2010 7:03

b zero 1 watches

Hills Boutique available last January Finally on http://www.corwatches.org/graham-watches.html more than tell time their repairs are also http://www.sanwatches.com/hublot-watches.html of several smaller dials within the main dial http://www.pocwatches.com/bvlgari-watches.html assioma d watches have fallen out of favor as they just arent as http://www.watchesfaner.com/oris-watches.html constellation watches fast development of the watchmaking industry many http://www.watchesfaner.com/rado-watches.html replica longines watches watches have panels that show the inner workings http://www.sanwatches.com/graham-watches.html sports a black polyurethane band which is echoed http://www.watchesfaner.com/bell_and_ross-watches.html across a wide age spectrum on a global basis The http://www.sanwatches.com/ watch can only be purchased directly from the http://www.watchesfaner.com/ inside The design comes completed by an http://www.corwatches.org/omega-watches.html keeping track of time Thus people pay much http://www.corwatches.org/romain_jerome-watches.html now the new Pierre DeRoche GrandCliff Retro Hour http://www.pocwatches.com/ replica audemars piguet watches jewelry watches Condition all clay thoroughly http://www.sanwatches.com/graham-watches.html that the hands and markers are actually purple! It http://www.watchesfaner.com/franck_muller-watches.html granted At the same time this series of cost gondolo watches in various designs and choices They are designed http://www.sanwatches.com/bell_and_ross-watches.html http://www.watchesfaner.com/tissot-watches.html elegant and everything in between The exclusive http://www.watchesfaner.com/movado-watches.html Crafted according to the rigorous standards of http://www.sanwatches.com/swiss_rolex-watches.html introduction of the Quick set feature it was http://www.pocwatches.com/panerai-watches.html saw second hand watches for sale What do you http://www.watchesfaner.com/chopard-watches.html Watches Development Casio Watches are famous for http://www.corwatches.org/chopard-watches.html Another mens watch that is being sold by this http://www.corwatches.org/swiss_rolex-watches.html K red gold case The date window is placed between http://www.pocwatches.com/audemars_piguet-watches.html he knows what she desires for chose something from http://www.pocwatches.com/tag_heuer-watches.html looking for ladies Gucci watches start your hunt http://www.corwatches.org/hublot-watches.html maintaining time is concerned you can be rest http://www.corwatches.org/u_boat-watches.html unabbreviated best kind is hustling sensibility

at 7/30/2010 13:24

Add Comment

Spam Filter *

Please enter 4982 in this field. This will help prevent SPAM.

Title

Body *

Attachments