whitepaper guidelines for local storage in vdi environments · 2020. 6. 27. · whitepaper...

Whitepaper

Guidelines for local storage in VDI environments

Author(s): Hugo Peters, Peter Sterk, Herco van Brug

Version: 1.0

Date: 1 August 2013

Whitepaper

locGuidelines for local storage in VDI environments

Versie 1.0 1 augustus 2013 Page i

© 2013 PQR, all rights reserved.

All rights reserved. Specifications are subject to change without notice. PQR, the PQR logo and its tagline 'Eenvoud

in ICT' (Simplicity in ICT) are trademarks or registered trademarks of PQR in the Netherlands and/or other coun-tries. All other brands or products mentioned in this document are trademarks or registered trademarks of their respective holders and should be treated as such.

Whitepaper


Versie 1.0 1 augustus 2013 Page ii

DOCUMENT OVERVIEW

CONSULTED DOCUMENTS

Reference Date Title

http://www.vmware.com/files/pdf/view/VMware-stateless-virtual-desktops-ref-arch.pdf

The VMware Reference Architecture for Stateless Virtual Desktops on Local Solid-State Storage with VMware View 5 (12th June 2012)

http://blogs.citrix.com/2011/11/04/tracking-the-pvs-write-cache-size/

Tracking the PVS Write Cache Size

http://blogs.citrix.com/2011/10/06/pvs-write-cache-sizing-considerations/

PVS Write Cache Sizing & Considerations

http://support.citrix.com/article/CTX129292 How to Disable the Dedicated Dump

File Feature in Provisioning Services 5.6

http://info.kraftkennedy.com/blog/bid/102199/Citrix-Provisioning-Server-Understanding-the-Limitations-of-Write-Cache-in-Target-Device-RAM

Citrix Provisioning Server - Understanding the Limitations of Write Cache in Target Device RAM







http://support.citrix.com/article/CTX129292




Whitepaper


Versie 1.0 1 augustus 2013 Page iii

TABLE OF CONTENTS

1. VDI: Local Storage and Sizing GuidelInes...........................................................................1

1.1 Objective .........................................................................................................................1 1.2 Challenge?! ......................................................................................................................1

2. VDI Design factors ...........................................................................................................2

2.1 Performance ....................................................................................................................2 2.2 Capacity ..........................................................................................................................2 2.3 Availability .......................................................................................................................2 2.4 Life expectancy ................................................................................................................3 2.5 Costs ...............................................................................................................................3 2.6 Optimization ....................................................................................................................3

3. VDI Sizing guidelines ........................................................................................................4

3.1 Type of users ...................................................................................................................4 3.2 CPU .................................................................................................................................5 3.3 Memory ...........................................................................................................................5 3.4 Storage performance ........................................................................................................6 3.5 Storage capacity ..............................................................................................................6

4. Storage capacity ..............................................................................................................7

4.1 Single Image Management ...............................................................................................7 4.2 Sizing of delta disk/files ....................................................................................................8

5. VDI & Storage Tiering .................................................................................................... 12

5.1 Storage tiering at the hypervisor level ............................................................................. 12 5.2 Storage tiering at the VDI level ....................................................................................... 12

6. Scenarios ....................................................................................................................... 14

6.1 VMware vSphere, VMware View and VMware ThinApp ...................................................... 14 6.2 VMware vSphere, stateless Citrix XenDesktop .................................................................. 16

7. VDI software.................................................................................................................. 19

7.1 Citrix XenDesktop 5.6 ..................................................................................................... 19 7.2 VMware View 5.1 ........................................................................................................... 20

8. Application virtualization ................................................................................................. 22

8.1 Microsoft App-V 4.6 ........................................................................................................ 22 8.2 VMware ThinApp ............................................................................................................ 22

9. About the authors .......................................................................................................... 24

Whitepaper


Versie 1.0 1 augustus 2013 Page 1/24

1. VDI: LOCAL STORAGE AND SIZING GUIDELINES

1.1 OBJECTIVE

The objective of this document is to provide a better insight into the key figures for resources that are applicable to design of a VDI solution. This document also describes the impact and the use of local storage for VDI. We not only discuss the required amount of IOPS, but also the amount of storage capacity that is required for a successfully executed VDI project.

1.2 CHALLENGE?!

The benefits that Virtual Desktop Infrastructure (VDI) can offer are numerous. It enables organisations to support BYO initiatives and to always work, everywhere, independent of the device. During the past few years, many workstation projects have been implemented with VDI. However, several of these projects have turned out to be rather disappointing. A key reason for this is the complete failure to assess the impact that VDI has on the storage. Although we were aware of this from the first project, we continue to see VDI projects start up in which incorrect assumptions have been made. Consequently, this significantly reduces the chances of success

for the end users as well as the IT organization.

Because the impact is so great, the storage component in the initial investment (CAPEX) can amount to more than 40% of the total solution. If the storage design is not optimal, the user experience will be disappointing and the acceptance of the VDI environment will present a major problem, from a technical as well as operational point of view.

However, the emergence of SSD seems to present a viable solution for this problem. SSD is known to be fast; SSD (or actually: Flash) solutions supply so many Input/Output Operations Per Second (IOPS) that VDI will no longer experience IO bottlenecks. The term ‘fast’ principally refers to the number of IOPS. The processing speed is often limited by the interface to which

the Flash cells are connected. Thus, a USB interface is many times slower than a SAS or SATA6 connection that are, in turn, much slower than a Flash solution based on PCIe.

SSD also has its downside; it is fast but also expensive and it breaks down relatively quickly (also see the white paper Spinning out of control). Current prices of Flash in enterprise storage solutions are in the region of € 15.-/GB, whilst this is in the region of € 0.50/GB for local hard disks in a server. In contrast, the price per IOPS for SSD is approximately € 1.-/IOPS, and in the region of € 5.-/IOPS for local hard disks. This means that there is no optimum balance in terms of a moderately high workload with a moderately large storage requirement. The most efficient,

especially in relation to VDI, seems to be local Flash solutions for servers. However, local storage in a VDI host also presents some challenges. This so-called VDI Local Storage Server (VDI LSS) makes it difficult to move workstations between hosts. Special measures must also be taken to ensure that data is secure in the virtual machines. Thus, VDI LSS solutions are primarily intended for stateless workstations that do not contain persistent information. But this is not the only problem. Moreover, finding the optimal balance between space and IOPS is a challenge for local storage.

Final remark: For the proposed solutions described in this document, it is assumed that different types of data are written to different storage locations. Currently, this is the only way

to optimally use the resources of all components of a node. In other words, 10 traditional hard disks or 4 SSD’s are not required. Although Citrix as well as VMware have stated on their list of future developments for their VDI products that this is possible, in the current versions (Q2/Q3-2013) this is not yet possible! In order to implement these proposed solutions, for the roll-out, a modification must be made to the location where, for example, temp files and page files end up. These modifications are easy to script, but make the solution more difficult to manage.

http://virtuall.eu/download-document/spinning-out-of-control-using-ssd-technology-in-enterprise-environments

Whitepaper



2. VDI DESIGN FACTORS

When designing a VDI Local Storage Server (VDI LSS), several factors must be taken into account. For many clients, the request for DI often entails a mix of stateless and stateful workstations.

a stateless desktop is restored to the original state when it is restarted. All changes that a user has made are lost; the user profile is often centrally stored and managed by User Environment Management solutions.

a stateful desktop immediately writes the changes for the user to the virtual file system.

Although a LSS does not seem to be logical, it can be an adequately available solution for those workstations. For example, just picture the backing-up of a (part of) the local storage of the servers at night. Moreover, traditional workstations have a 0% data loss in the event of outage; so why should this be the case for a central version?

2.1 PERFORMANCE

With regard to a device, due to the black-box approach, different specifications exist for performance than are assumed for traditional servers. In other words, it does not matter to the

user which CPU is used, how much memory it has, or even how many IOPS the system can supply. An appliance supplies the resources for a number of workstations. This is the only number that matters.

2.2 CAPACITY

The storage capacity of a Local Storage Server (LSS) is the next bottleneck. But, of course, it is a pity to be limited by the space in your storage if you still have additional CPU capacity. How can you obtain the right balance? When designing an environment and determining the required capacity, a number of scale factors must be taken into account;

Temp files and temporary Internet files Spooler vSwap (esx) or saved state file (Hyper-V) Page file Split profile into app data, local/roaming and settings

Application virtualization, sandboxes or caches

These factors are housed in a delta file of the stateless VDI solution. Thus, stateless workstations do not have the complete disk size required for their (Windows) workstation, but only the difference in relation to a basic image. By cleverly manipulating this basic image, the size of such a delta file can be limited to a few Gbytes.

Technologies that enable this include:

VMware Composer (linked clones)

Citrix Machine Creation Services (differencing disks) Citrix Provisioning Service (write cache) Microsoft Image Management for VDI

Other rules are applicable to stateful workstations where the basic OS is also included in the storage per user.

2.3 AVAILABILITY

The N+1 principle is often used for handling the outage of a host, i.e. adding an extra node to increase the capacity in order to handle the outage of a node.

Whitepaper



2.4 LIFE EXPECTANCY

Flash storage is often considered for a VDI LSS. However, the SSD life expectancy has resulted in it virtually being classed as a disposable article. Enterprise solutions expect a minimum life expectancy of 3 years, often as much as 5 years. However, for VDI workloads, this cannot be expected from all SSD’s. Without any intelligence such as IO Optimization, Write Coalescing or Write Ordering, it is possible that via Write Amplification1, much more data is written in the Flash cells of the SSD, than is sent as MB/s to the SSD. MLC SSD’s can thus fail within one or two years even though vendors promise that they, when full 1x per day, should last for 5 years.

Vendors often word the text of the warranty provisions so that this is not covered by the warranty. In other words, SSD must be regarded as a disposable article that must probably be replaced every year, thus significantly increasing the costs.

Stating the life expectancy is very important. Predicting when a SSD fails, can significantly increase the uptime of the VDI infrastructure.

2.5 COSTS

SSD or PCIe differ enormously in terms of price/GB, but also differ considerably in terms of, for example, performance and latency characteristics. So which solution is the best and when?

2.6 OPTIMIZATION

Optimization can take place in terms of space saving for Flash solutions, but also in terms of IO savings for spinning disk solutions, or a combination thereof; small partial high performance on Flash and static bulk data on HD (storage tiering).

When using Flash solutions, optimization of life expectancy is also important, especially when these are classified as disposable Flash.

1 http://www.pqr.com/spinning-out-of-control

http://www.pqr.com/spinning-out-of-control

Whitepaper



3. VDI SIZING GUIDELINES

A number of factors are important when sizing a VDI solution. The most restrictive resource in a server is the CPU. This is installed per 1, 2 or 4 sockets in an average server. The number of cores and the number of MHz can also be selected. Once determined, scaling is no longer possible. Thus, the CPU is the determining factor for the number of users on a host. Other resources such as network cards, hard disks or memory are scaled up, based on this.

The point of departure for the sizing guidelines is that an average equipped server is used. This means that for N+1 not equal, a much too expensive extra server is required, however the

density of users is so high that overhead for chassis and licences is restricted as much as possible. At the moment (H2-2013), this is achieved with a server with 2 sockets.

3.1 TYPE OF USERS

In terms of CPU sizing, it is important to know which type of users will use the host. From experience, we know that for government agencies, this is a mix of approximately 90% light and 10% heavy users. For care institutions, this is a mix of approximately 80% medium and 20% heavy users.

Defining what a light, medium or heavy user is, is nothing more than a classification of the

resources that they require. Light users are generally users who actually sit behind their workstation for less than half of their working hours. Light users have, at most, three applications open that are often standard Office Automation (OA) applications. Medium users sit behind their workstation for more than half of their working hours and have approximately 5 to 7 applications open that they actively use. Heavy users sit behind their workstation most of the day, and also have several applications open that interact to a great extent with the file system (logging, streaming, etc.).

The required resources are determined by making measurements at clients where VDI solutions are implemented, and via lab tests such as Project VRC. This data has been used to compile the

following table, based on the Intel Xeon E5 series of processors.

Type of user CPU (MHz) RAM (MB) ESX RAM (MB) Hyper-V IOPS [log on]

Light 270 1200 1500 4 [5] Medium 350 1400 1750 8 [10]

Heavy 460 1800 2000 12 [15] Table 1

The CPU load is calculated from measurements plotted against the GHz that the physical processors can supply. Sizing is often based on the number of users per core, however this presents a false picture because core speeds vary from 1.8GHz to more than 3GHz. In this table, the user load on the CPU is therefore calculated regressively to the required MHz.

Each (VM) user is allocated 2GB of memory, however thanks to technologies in the hypervisor, less physical memory is required per user than this 2GB. The amount of memory per user that is stated in Table 1 is based on overcommitting of VMware ESX. Memory that is not used and identical memory between VM’s, is only stored once in the physical memory. For Microsoft

Hyper-V, the point of departure is that each user receives at least 1GB and the rest is obtained from the dynamic pool. In practice, the pool appears to be divided as shown in Table 1.

For larger numbers, at IO level, the logon storm produces approximately 25% extra IO load. The lower the numbers are, the greater the (static!) deviation. On average, in a 1000-user environment, approximately 3% of the users log on at the same time at around 09:00. At storage level, the higher read load that is required for this produces 20-40% (average is approximately 25%) extra load, however the average Read/Write ratio usually remains around 15/85%. The IOPS load from these users usually accounts for 80-90% of the VDI environments. In the other environments, the deviation from this average is so great that twice

as much traditional storage (or even more!) is sometimes required to facilitate the required IO

http://www.projectvrc.com/

Whitepaper



load. The only way to know this beforehand is by making a measurement in the current infrastructure and identifying the load from the applications.

3.2 CPU

In order to determine the number of users on a host, a margin is determined so that space remains in the CPU for handling peak loads. If hyperthreading is used, a context switch is created more quickly, and more space remains for serving the users. These context switches are taken into account in Table 1. If hyperthreading is not enabled, this means that a user requires more MHz, because context switches take more time.

The new Intel Xeon E5-series of processors operate at a lower MHz than their comparable predecessors, but according to Intel, are between 25 and 45% faster under various load scenarios. In practice, for a workstation environment with the same user experience, 20% less MHz is required per user (see Table 1).

For the light/heavy user ratio of 90/10%, each user requires an average of 289MHz. In this document, this type of users are referred to as Type 1 users. The CPU load is, taking account of hyperthreading, determined as a maximum of 80%. The number of users that can then use a host with an average heavy CPU (Intel E5-2660 8-core 2.2GHz) is then:

2.2GHz * 8 cores * 2 CPU * 80% load / 289MHz = 97 users

For the medium/heavy user ratio of 80/20%, referred to from now on as Type 2 user, this is

calculated as follows:

2.2GHz * 8 cores * 2 CPU * 80% load / 372MHz = 75 users

3.3 MEMORY

The new Intel E-series of processors support quadchannel memory. In other words, reference can be made to the memory DIMM’s 4 at a time. This results in a processing speed of more than 50GB/s (here, we are talking about gigabytes, not gigabits!). The memory speed decreases proportionally to the number of DIMM’s, thus with two channels filled instead of four, the processing speed is decreased by half.

In addition to this, the motherboards on which the DIMM’s are installed, decrease the number

of GHz at which the memory operates if a channel has more than two DIMM’s. The ideal solution for an E5 system is to fill each channel with one or two DIMM’s. For a dual-socket system, DIMM’s are thus installed per 8 or per 16.

Preference is often given to 8GB DIMM’s because these are the cheapest per GB. However, 16GB DIMM’s are slightly more than twice as expensive as 8GB DIMM’s, thus for larger amounts of memory, 16GB DIMM’s are certainly an option.

3.3.1 ESX

For a Type 1 user, for 97 users, an average of

90% * 1.2GB + 10% * 1.8GB = 1.26GB

memory is required per user, thus a total of 122GB. For Type 2 users, this is

80% * 1.4GB + 20% * 1.8GB = 1.48GB

thus a total of 111GB for 75 users. Thus, both can be filled with 16x8GB DIMM’s.

3.3.2 Hyper-V

The same calculations for Hyper-V are as follows:

90% * 1.5GB + 10% * 2GB = 1.55GB

average memory per user for Type 1 for a total of 1.55 * 97 = 150GB and:

http://ark.intel.com/products/64584/Intel-Xeon-Processor-E5-2660-20M-Cache-2_20-GHz-8_00-GTs-Intel-QPI

Whitepaper



80% * 1.75GB + 20% * 2GB = 1.8GB

for Type 2 for a total of 75 * 1.8GB = 135GB. Both can be filled with 8x8GB + 8x16GB = 196GB of memory.

3.4 STORAGE PERFORMANCE

The number of front end IOPS that is required (already stated in Table 1)has, in almost all of the cases, the same Read/Write ratio of 15/85%. The high number of writes account for the impact that VDI has on storage environments (also see VDI & Storage: Deep Impact White Paper).

For Type 1, the average IO load is

90% * 5 + 10% * 15 = 6 IOPS.

In order to supply these numbers of IOPS in the correct R/W%, local 15,000 rpm SAS hard

disks thus require:

15% * 6 IOPS * 97 users / 160 Riops + 85% * 6 IOPS * 97 users / 80 Wiops =

6.7 disks.

Because these are placed in Raid 1(0), 8 disks are thus required.

For Type 2, per user, this is:

80% * 10 IOPS + 20% * 15 IOPS = 11 IOPS.

The number of disks that are required is then

15% * 11 IOPS * 75 users / 160 Riops + 85% * 11 IOPS * 75 users / 80 Wiops =

8.8 disks.

Or, in Raid 1(0), around 10 disks.

3.5 STORAGE CAPACITY

For different VDI solutions, a different amount of storage is required per workstation. The diversity in this context is further elaborated in the next chapter.

http://www.pqr.com/images/stories/Downloads/whitepapers/deep%20impact%20eng.pdf

Whitepaper



4. STORAGE CAPACITY

As previously stated in the aforementioned paragraphs, during the past few years, IOPS has been a key success factor in the implementation of a VDI infrastructure. By making the correct calculation and deploying adequate disks/spindles, it can be ensured that adequate performance is available for the virtual desktops. As a result of this, many GB’s became available for storing temporary information, and the required storage capacity was also often available.

Moreover, the availability of new fast Flash-based storage solutions has turned things around.

The IOPS no longer present a problem, and the performance of the desktops is superb. However, the choice of a Flash-based storage solution has an impact on the available storage capacity. Affordable solutions are available with a capacity varying from 320GB, 640GB to, for example, 1.2TB. IOPS is no longer a discussion point these solutions are so fast that other components have to do their utmost to keep pace. The storage capacity is however limited, thus oversizing of required storage capacity per VDI workstation can have an extremely negative impact on the total number of VDI workstations per host. Correct sizing of this required storage capacity is thus extremely important, but how much space is actually required per VDI workstation/user, and which options exist for optimising this?

This chapter provides an overview of technologies and factors that can affect the required storage capacity per VDI workstation. The stated numbers are only guidelines and can differ in practice; the exact numbers depend on many factors that can greatly differ per environment, for example, the application set used within the environment, and the type of users of the VDI workstation (light versus heavy users).

4.1 SINGLE IMAGE MANAGEMENT

Single image management (also referred to as Golden Image management) enables us to start several virtual desktops from a read-only image, so that it is no longer necessary to equip each

VDI workstation with its own disk with operating system and applications. The use of single image management can thus result in significant savings in required disk capacity.

Various technologies are available that enable single image management. A few examples of these are:

Linked Clones (VMware View Composer) Differencing Disks (Citrix Machine Creation Services) vDisks with Write cache (Citrix Provisioning Services)

Although the technical composition can differ, all of these solutions basically offer the same: supplying a single read-only image where writes to this read-only disk are captured and are saved in a delta disk or file.

A delta disk or file is created for each VDI workstation. The size of this directly depends on the number of disk blocks that are changed on the VDI workstation in relation to the basic image. Is the number of changes is large, than the size of the delta disk will also be large.

For correct storage sizing, it is also extremely important, already in the design phase, to accurately estimate how large the delta disk of a VDI workstation will be, i.e. how much data

will be changed in relation to the basic image. This is especially the case when using local Flash-based storage; the capacity of such storage is limited and the price per GB is high. Too generous sizing of the delta disk would mean that 20% to 50% fewer VDI workstations can be hosted.

If however, the sizing is too modest, the storage space on the delta disks can become full, with this having significant consequences for the availability and performance of the VDI environment; if the Windows system can no longer write changes, this will result in a system crash.

Whitepaper



In practice, the sizing of the required storage capacity for the delta disks is often a difficult step in the sizing process. A large number of factors exist that can affect the number of changes that a workstation and/or user makes on disk, and all the results of these factors are hardly ever known in the design phase. The following paragraphs provide an overview of the factors that must be taken into account.

4.2 SIZING OF DELTA DISK/FILES

The following paragraphs discuss of the factors that play a role when determining the required storage capacity for a VDI environment. These factors can be subdivided into three categories:

System User Hypervisor

These categories are explained in the following paragraphs.

4.2.1 System

These are all writes that are generated by the Windows operating system and are necessary so that the system can operate. This includes the following:

Log files Windows event logs other logs

Registry changes (policies, etc.) Page file

Monitoring / management tools (Anti-Virus, etc.)

A major advantage is that we can control the size of these data changes and can thus determine reasonably accurately how much storage capacity is required for these system-related issues. It is therefore prudent to reserve some space for unforeseen situations.

The maximum size of the event logs can be limited to a few MB’s. A reserve is also included for the other log files. 200MB is reserved for all of the log files.

The size of the page file can also be set to a fixed value (depending on the amount of memory allocated to the VDI workstation). In this document, we assume that VDI workstations are

equipped with 2 GB of allocated memory, and that 512 MB of space is reserved for the page file. The general philosophy behind this is that page file use must be limited as much as possible (IO limitation), and that unused page file space on expensive central storage makes the TCO for VDI difficult. In practice, the page file use for workstations during a workday is often less than 250MB. This justifies the choice of a 512MB page file with a fixed size.

A provision of 500 MB extra space is reserved for the ‘registry’ and ‘monitoring tools’ parts. This space also provides some reserve capacity for other parts.

In total, we require 1212 MB of disk space for the ‘system’ part.

4.2.2 User

In addition to the Windows operating system, the user of the VDI workstation also has significant control over the size of the delta disk.

USER PROFILE

As soon as the user logs on to the system, a profile is created or copied locally op the VDI workstation in which the personal settings of the user are saved. This data is thus written to the delta file. We must therefore take this into account when determining the required storage capacity.

Whitepaper



This profile can have various forms (local, roaming, hybrid or mandatory), depending on the profile management solution selected within the infrastructure. The size of a user profile can vary from several tens of MB’s to even GB’s.

The choice of type profile determines the required disk space. A mandatory profile will thus generally be smaller than, for example, a roaming profile, so that when using mandatory profiles, considerably less space is usually required compared with using roaming profiles. Moreover, when using mandatory profiles, the required space is easier to predict.

Currently, profiles are, to an increasing extent, composed as a hybrid profile, by using available tooling and/or User Environment Management solutions. A hybrid profile uses a local or

mandatory user profile where application settings are injected at the moment when the user starts the application. In this way, profiles can be optimized (clean, fast) and flexible (can be used for different operating systems). Adequate space must be available (depending on the number of applications that the user starts and the size of the profile files). Because hybrid profiles basically use a mandatory profile, the storage capacity required for this type of profile will also be limited and reasonably easy to predict.

Applications

In addition to the personal Windows settings, the profile is also used for saving application-

specific information. The application set used within the VDI environment can also affect the size of the user profile. For example, after a standard installation of Microsoft Office, the caching mode of Outlook is automatically active. This results in a local copy of the user mailbox being saved locally (in a so-called ost) in the profile. Depending on the size of the mailbox, the space required can increase significantly.

In practice, and in accordance with best practices, the caching functionality of Outlook will be disabled when used in a (non-persistent / stateless) VDI environment, and this illustrates how functionality within applications can affect the user profile.

Folder redirection

By using folder redirection, the space required for the user profile can be further limited. Folder redirection enables us to redirect certain folders from a user profile to a network location so that the space required can be limited; these are directly accessed from the network and thus not copied locally on the workstation (and thus to the delta disk).

Note: When using Folder Redirection, special attention must be paid to optimal design for the file servers because, in such a case, these have to cope with a heavier load. Moreover, communication for storage takes place via SMB, thus creating extra overhead. A difference is apparent when using Windows 2003, 2008 and 2012 as a file server.

Temporary files

In addition to application- and customizable settings, temporary files are also saved in the profile. The ‘temporary Internet files’ are a good example of this. This location is used to temporarily store a file to be downloaded before it, in its entirety, is copied to the correct folder (choice of the user!). How much space is required for this depends entirely on the file to be downloaded. This can be MB’s, but can even be GB’s!

Note: From Internet Explorer 9.0, a ‘partial’ file is created at the target location, and the ‘temporary Internet files’ location will not be used.

PRINT SPOOLER

The standard location of the Print spooler directory is “C:\Windows\system32\spool”. As soon as the user prints something, a temporary file will be placed in this directory before it is sent to the printer (server). A spool file can however be several times larger than the original file. A number of examples:

4 MB Powerpoint 2010 pptx is 110 MB; 18 MB Powerpoint 2010 pptx is 480 MB; 4.2 MB PDF file is 475 MB (!).

Whitepaper



Test system: Windows 7 x64, Office 2010

A number of the VDI solutions described in this document enable the print spooler folder to be placed outside the delta disk or -file. For solutions in which this is not possible, this data is placed in the delta disk and can thus affect the required storage capacity for this disk. The total space required depends on the number of simultaneous printer commands and the format of the files. However, in practice, this is difficult to predict.

CACHE FOR VIRTUAL APPLICATIONS

Application virtualization is frequently used to limit the number of read-only images required; by using application virtualization, an application can be made directly available to the user without

a local installation or change being required. Depending on the selected application virtualization solution, a local cache is created from which the application is started. The space required for this depends on the number of applications that the user starts (during a session) and how large the application packages are.

If the application virtualization solution requires a local cache, inclusion of the cache in the read-only image can be considered, so that virtual applications are directly started from this image and do not have to be cached on demand (when a user starts an application) on disk and thus be saved on the delta disk. In this case, only the updates for the application are

cached on the delta disk.

TOTAL USER

In all cases, there must however be a temporary ‘workspace’ for the user. Depending on the selected profile strategy, this must provide space for the following components:

User profile; Desktop; Documents;

Settings (via registry changes); Temporary (Internet) files; Application data;

Spooler; Caches for virtual applications (depending on presence/selected application virtualization

solution).

This presents a major challenge; The profile of the user is important, this provides the user with the Windows Desktop and application environment required for performing the daily work. Limiting the space available can possibly restrict the work of the user! It is thus recommended to retain something from a buffer.

In this document, we assume a VDI environment in which use is made of hybrid profiles and folder redirection. The hybrid profile basically consists of a mandatory profile with a set size of between the 0 and 1 MB. A provision of approximately 200 MB is reserved for the use of the

user settings (registry settings and profile files) on this mandatory profile.

The ‘temporary Internet cache’ is limited to 50 MB for users. As stated, large downloads that are temporarily saved in the profile must be taken into account. In this document, we assume 1 in 4 users download a 1 GB file once per day, i.e. 250 MB of space is reserved per user. The VDI workstation is restarted daily.

In total, the space required for the user profile is 500 MB per user.

For the Print Spooler, we assume that 1 in 2 users print a document daily where the size of the spool file is 500 MB. This space is made available after a document has been printed on the workstation and can thus be reused for other print jobs initiated by the same user.

Finally, we must reserve space for the virtual application cache. Virtual applications are pre-cached in the read-only image so that space only has to be reserved for the delta cache (updates of virtual applications that, after the pre-caching, are implemented within the application virtualization environment). For this, we reserve 1 GB of space.

Whitepaper



Finally, a buffer with 500 MB of disk space is included. In total, (500 MB (profile) + 500 MB (print) +500 MB (Virtual application cache) + 500 MB (buffer) = 2000 MB of disk space reserved for the user.

4.2.3 Hypervisor

Depending on the selected hypervisor, a temporary file is created at a storage level that is just as big as the allocated RAM memory of the Virtual Machine. This space is used by the hypervisor if enough physical memory is not available for the virtual machines.

In Vmware, this file is placed by default in the same data store as the virtual machine. It is possible to save this file to another data store. Hyper-V also uses such a file and places it in the

same folder as the VM files.

When using VMware vSphere or Hyper-V, we must then also reserve an amount of disk space equal to the amount of memory allocated to the VDI workstation for the Hypervisor swap file per VDI workstation. In this document, since we assume that VDI workstations are equipped with 2048 MB of memory, we must reserve 2048 MB of disk space per desktop.

4.2.4 Total storage required per VDI workstation

If we add up the disk space required per component (as in the aforementioned paragraphs), then approximately 5 GB of storage capacity is required per VDI workstation. The storage sizing

components are graphically depicted in the figure below:

5210 MB

VM SWAPFILE

Hypervisor

PAGEFILE

Systeem

LOGFILES

BUFFER

WINDOWS SWAPFILE

Gebruiker

GEBRUIKERSPROFIEL

Print Spooler

APP. VIRT. CACHE

BUFFER

Hypervisor

Systeem

Gebruiker

500 MB

500 MB

500 MB

200 MB

500 MB

200 MB

512 MB

2048 MB

+

TEMP. FILES 250 MB

Whitepaper



5. VDI & STORAGE TIERING

Storage tiering means that data, depending on a number of characteristics such as required performance and availability, is distributed across different types of storage within an environment. The movement of, for example, archive data from high performance to nearline storage enables us to use the available storage resources more efficiently. But does storage tiering also offer advantages in a VDI environment? And if so, do the present VDI solutions from Citrix and VMware enable storage tiering to be used?

If we look at a VDI environment, then, on the one hand, we require many IOPS to guarantee

optimal performance. We can solve this by using Flash-based storage, however this has the disadvantage of limiting the available disk space. On the other hand, traditional storage offers adequate storage capacity, but limited IOPS. Of course, it is also possible to use a combination of Flash-based and traditional storage, where the most disk I/O intensive parts of the VDI environment are placed on the fast Flash disks, and the traditional storage is used for the less I/O intensive tasks.

In order to be able to distribute these tasks across the two types of storage, the selected VDI solution and hypervisor must provide support for this. For example, the hypervisor must enable the swap file location to be configured to another volume, independent of the location of the

virtual machine files. Within the VDI solution, it must be possible to link more than one volume to a VDI workstation.

5.1 STORAGE TIERING AT THE HYPERVISOR LEVEL

When using VMware vSphere as a virtualization platform; it is possible to centrally state the location of the virtual machine swap files, which enables us to place the swap files on volumes other than where the virtual machine itself is located. However, Microsoft Hyper-V does not allow this; virtual machine swap files (.bin files) are always saved in the virtual machine.

5.2 STORAGE TIERING AT THE VDI LEVEL

If we look at the two most used VDI solutions, i.e. Citrix XenDesktop and VMware View, then the options for using storage tiering are extremely limited.

When using Machine Creation Services from Citrix XenDesktop, only one virtual disk can be linked per VDI workstation. Different types of storage cannot be used within a VDI workstation.

Citrix XenDesktop, where deployment of the workstations is performed by means of Provisioning Services, offers limited options for using storage tiering. Provisioning Services enables several virtual disks to be linked to a VDI workstation that can be placed on different hypervisor volumes. This, for example, enables us to refer the Windows page file to a volume

other than where the delta disk is saved (in the case of PVS, the write cache). However, you must also take into account the fact that these disks are persistent. Data on these disks is not saved in the delta file and thus also not deleted, also not in the case of a stateless desktop.

Whitepaper



DATASTORE 1FLASH STORAGE

DATASTORE 1TRADITIONELE STORAGE

E:\E:\D:\

ProfielApp-V Delta Cache

Systeem (Registry

etc)

PVS WriteCache

C:\ (Microsoft Windows)

Windows Pagefile

Print Spooler

Temp Files

ProfielApp-V Delta Cache

Systeem (Registry

etc)

Windows Pagefile

Print Spooler

Temp Files

HYPERVISOR

Citrix PVS

READ-ONLY DISK

(vDISK)

READ WRITE

Citrix XenDesktop & Provisioning ServerStrorage Tiering

VMWare View enables temporary data such as the page file and temp files to be saved separately from the delta disk in a so-called disposable disk, however the present versions do not allow these to be placed on a volume other than where the linked clone is saved.

Although the use of storage tiering in a VDI environment in which Flash-based storage is used could solve part of the capacity problem for Flash-based storage, the present solution only allows this to be implemented to a limited extent.

Whitepaper



6. SCENARIOS

This chapter discusses a number of scenarios where the required storage capacity is estimated per scenario.

6.1 VMWARE VSPHERE, VMWARE VIEW AND VMWARE THINAPP

In this scenario, Windows 7 VM’s with VMware View Composer are made available on a VMware vSphere Server. VMware View is used as VDI solution and VMware ThinApp for virtualization of applications.

Every virtual machine is equipped with 2 GB of RAM memory.

One virtual machine is equipped with Windows 7, Microsoft Office, and the required VDI software. A snapshot is made on this virtual machine so that this can be used as a golden image. The size of the golden image is 30 GB.

VMware View composer uses the snapshot to make a copy (the replica disk) of the golden image. The replica disk is copied to every data store where VDI workstations will run. All virtual machines have a reference to this replica disk and consist of the following disks:

Internal disk (unique data, name, computer account password: 20MB) Virtual Machine disk (reference to replica with all changes from when the virtual machine is

switched on, quick prepping, switching off and taking snapshot: approximately 100MB) Checkpoint disk: All changes from the moment that a user logs on. This disk is deleted and

re-created when a user logs off.

A diagram of this configuration is shown below:

Windows 7Microsoft Office

VM disk

Checkpoint disk

Internal disk

Replica diskVirtuele machine

configuratie

Whitepaper



6.1.1 Required amount of disk space per VDI workstation

The figure below shows how the different data sets can be distributed across the flash and traditional disks.

3330 MB

VM SWAPFILE

Hypervisor

Systeem

LOGFILES

BUFFER

WINDOWS SWAPFILE

Gebruiker

GEBRUIKERSPROFIEL

PRINT SPOOLER

APP. SANDBOX

BUFFER

Flash

500 MB

500 MB

500 MB

200 MB

500 MB

200 MB

+

512 MB

IDENTITY DISK

VMWare VIEW

VIRTUAL MACHINE DISK

20 MB

100 MB

TEMP FILES 250 MB

2048 MB

2048 MB

Traditioneel

6.1.2 Total required storage

To calculate the total required storage per vSphere host when using VMware View, in this scenario, the following formula can be used:

Total required storage = (number of VMs * (size Delta file) + Replica)

It should be noted that a minimum of once the disk space must be included within the VMware

View environment for saving the golden image.

6.1.3 Focus points for this configuration

TEMPORARY INTERNET FILES AND SPOOLER

For the storage of temporary Internet files (also downloads) and spooler directory, the expansion of the delta disk must be taken into account. If large files are placed in the virtual machine, the delta disk will expand. If these files are deleted, the delta disk will not become smaller. The “empty” space can however be used again for writing.

Whitepaper



6.2 VMWARE VSPHERE, STATELESS CITRIX XENDESKTOP

In this scenario, Windows 7 VM’s with Citrix Provisioning Services are made available on a VMware vSphere Server. Citrix XenDesktop is used as a VDI solution, and Microsoft App-V for the virtualization of applications. A virtualization host is deployed, equipped with a combination of traditional and solid state disks.

Every virtual machine is equipped with 2 GB of RAM memory.

Every virtual machine is equipped with Windows 7, standard applications and the required software components before a Provisioning Service vDisk is created. The size of the vDisk is

approximately 50 – 70 GB, depending on the number of virtual applications. Before the vDisk is first used, all virtual applications are saved pre-cached in the vDisk.

The virtual machine is allocated a partition that is saved in a local data store based on Flash storage. Because most writes end up in the PVS write cache, it is highly desirable to use this local data store. However, Citrix PVS tends to write the write cache in the largest available partition, and this is not configurable. Because the Flash-based data store has a limited capacity, an alternative method must be used. By configuring the partition larger than the 2nd partition, as non-persistent and thin-provisioned, this partition will still be used for the write cache and only take up the space actually used. Via non-persistent switching on, after a restart,

the used space is made available for the data store.

In addition, a 2nd partition is linked to the virtual machine that is saved in a local data store based on traditional storage. The Windows Page file and Print Spooler are configured to this. The used space thus becomes available again after a print command. All other data is written to the PVS Write cache, including updates to the virtualized applications.

A diagram of this configuration is shown below:

VDISKWindows 7

Standaard applicatiesMS App-v (pre-cached)

PERSISTENT DISK 1Type: Flash

WriteCache: Gebruikersprofiel, App-V Cache (Delta), Logfiles

PERSISTENT DISK 2Type: Traditioneel

Windows pagefile Print Spooler

Whitepaper



6.2.1 Required amount space per VDI workstation

The figure below shows how the different data sets can be distributed across the flash and traditional disks.

1900 MB

VM SWAPFILE

Vmware vSphere

Systeem

LOGFILES

BUFFER

WINDOWS SWAPFILE

Gebruiker

GEBRUIKERSPROFIEL

PRINT SPOOLER

APP. VIRT. CACHE

BUFFER

Flash

500 MB

500 MB

200 MB

500 MB

200 MB

+3310 MB

Traditioneel

512 MB

2048 MB

500 MB

TEMP. FILES 250 MB

Note: The components in the ‘Flash’ column are saved in the PVS write cache file. So that this can be used, the ‘Flash’ partition must be configured larger than the ‘Traditional’ partition.

6.2.2 Total storage required per host

When using Provisioning Server for the deployment of the VDI workstations, space is required for saving temporary data, such as PVS Write cache, Page file and profile data. The following formula can be used to calculate the total storage required per vSphere host when using Citrix XenDesktop ICM Provisioning Server:

Total required storage = (number of VMs * (Total data on Flash + Total data on

Traditional)

Additional space does not have to be reserved for the master image, since this only requires several tens of MB’s on the Provisioning Server.

6.2.3 Focus points for this configuration

APP-V CACHES

All App-V packages are saved pre-cached on the Provisioning Services vDisk. Thus, only the changed applications (deltas) are saved in the PVS Write cache on the persistent disk. If the number of changes increases in relation to the cache in the vDisk, this has an impact on the space required in the PVS Write cache. Regular updating of the App-V cache in the vDisk is thus necessary!

Whitepaper



BUFFER FOR TEMPORARY INTERNET FILES AND SPOOLER

A limited amount of space is reserved for the storage of temporary Internet files (also downloads) and spooler directory, per user. This is possible by assuming that not all users use this space daily or at the same time. If it is evident that this space is not adequate, additional space will have to be reserved.

DAILY RESTARTING

The daily restarting of the virtual machine is important in order to refresh the PVS Write cache. Only a correct restart will ensure that the PVS Write cache is refreshed; ‘hard’ switching off of a virtual machine is not sufficient.

Whitepaper



7. VDI SOFTWARE

One of the strengths of VDI is the ability to simplify the management of the workstation OS and revert to single image management. All workstations are derived from a master image. For this, stateless (pooled) desktops are often thin provisioned so that all identical parts of all workstation operating systems only have to be saved once or a few times.

At the time of writing this document, the latest versions of the two software solutions are VMware View 5.1 (http://www.vmware.com/products/view) and Citrix XenDesktop 5.6 (http://www.citrix.com/xendesktop).

7.1 CITRIX XENDESKTOP 5.6

A Citrix XenDesktop 5.6 infrastructure consists of a Delivery Controller that acts as a broker within the VDI infrastructure. The Controller keeps a record of which desktops are available and will switch on or switch off desktops, depending on policies configured by the administrator. In addition, the clients connect via the installed Citrix Receiver to the Controller, and are routed to an available desktop.

XenDesktop has two deployment mechanisms;

Provisioning Services (PVS)

Machine Creation Services (MCS).

For PVS, a central server is used to stream the read-only image via the network to the workstations. This is loaded as far as possible into the memory of the PVS server in order to limit the read IOPS. Because the workstations load the operating system from the network in the memory, the number of required IOPS is significantly lower than when using, for example, linked clone technology, so that the OS is read directly from disk. A virtual disk is coupled to every within the hypervisor layer, on which the delta file, the PVS write cache is placed. This

cache is saved in a file, and refreshed after a restart of the workstation. Because the coupled disk appears as a normal volume within Windows, it can also be used for the placing of, for example, page file and log files. It should be noted that this data is persistent, since this data is saved outside the write cache.

http://www.vmware.com/products/view

http://www.citrix.com/xendesktop

Whitepaper



MCS can basically be compared with how VMware’s linked clones operate. Workstations are rolled-out by creating a linked clone of a basic image. In addition, every workstation is given a 16MB identity disk that makes the workstation unique, and a differencing disk in which the writes from the workstation take place. In the case of non-persistent desktops, the differencing disk is refreshed after every restart, so that the workstation reverts to a ‘clean state’ after the restart. Because every desktop reads the OS directly from disk, when using MCS, the number of IOPS is approximately 65% higher than when PVS is used.

This size of the PVS Write cache and the MCS difference disk is always dynamic and can be 1 – 2 GB as a basis plus sometimes 1GB/day, depending on the selected application methodology

and modifications to the profile, temporary files and page files. For PVS installations, 1.5 - 1.7 x the number of IOPS is often observed for ‘native’ installed operating systems.

7.2 VMWARE VIEW 5.1

The connection Server forms the core of a VMware View infrastructure. A View client is installed on the endpoints so that end users can link to this connection server. An agent is installed in the virtual desktop that, in turn, links to the connection server. This enables the connection server to know which virtual desktop is available.

VMware Composer can be used to create virtual desktops. This is a part of VMware View

Premier, and this Composer simplifies the creation of virtual desktops, and single image management is possible. Only one desktop image has to be maintained, and an intelligent cloning technology enables savings to be made in terms of the required storage. VMware View only works in combination with VMware vSphere.

7.2.1 Local storage options

In View, a Golden Image is used to equip the workstation. One (of several) snapshot(s) are made of this image that subsequently serves as a basis for the workstations by making a replica of this base image snapshot per set of VM’s. This replica may appear at the same location as

Whitepaper



the VM’s, but also at its own location. The workstations can then be rolled-out by creating a linked clone of the replica (up to a maximum of 1000 linked clones). This clone may appear at the same location as the replica, but may also appear at a different location.

From the replica, it is possible to read (maximum of 2GB) the most read blocks in the memory of the host (ESX).

Page files and temporary files from the workstation can be saved on a separate disk, but these must appear at the same location as the linked clone. By placing these on a separate disk, it is possible to discard these in the event of a reboot, whilst changes in the linked clone will be retained.

The VM swap file may either be placed with the linked clone or on a local data store of the host. The size of this is always the non-reserved part of the memory of the VM. Because reserved memory in a host is no longer shared (Transparent Page Sharing or TPS), memory is often not reserved due to a higher VM density.

Whitepaper



8. APPLICATION VIRTUALIZATION

Another strength of VDI is the flexibility offered to the users for the diversity of applications while they still all use the same basic image. For this, application virtualization is used which uncouples the application from the underlying OS. It is thus possible to link applications to users. The application is made available on every workstation where they log on, without these applications having to be reinstalled.

The two most advanced solutions for application virtualization are Microsoft App-V (http://www.microsoft.com/nl-nl/windows/enterprise/products-and-

technologies/virtualization/app-v.aspx) and VMware ThinApp (http://www.vmware.com/products/thinapp). In several studies, both of these have been found to have the highest level of virtualization, but differ greatly in terms of management structure. Of course, other forms of application virtualization also exist. The differences are explained in more detail in the ‘Application Virtualization Smackdown’ publication (http://www.pqr.com/application-virtualization-smackdown).

8.1 MICROSOFT APP-V 4.6

Microsoft App-V (App-V) uses a central management platform that connects to the System

Center platform from Microsoft. It is part of the Microsoft Desktop Optimization Pack (MDOP) that can be purchased after concluding a Software Assurance (SA) agreement.

8.1.1 Impact on storage

App-V uses an agent installed in the desktop that is supplied with applications from the central infrastructure. These applications are saved in a local or central cache so that they can be restarted faster. A size of a local cache is often 1 to (sometimes) 4GB, depending on the number and size of the applications.

For VDI, the agent and often a large part of the application cache are also saved in the read-only image (preloading of App-V cache). After starting a VDI workstation, this prevents the

agent from loading all applications into the cache and ending up in the delta disk (write-cache or differencing disk) of the user. This prevents the delta disk from being unnecessarily large, and the entire virtual application cache from being created separately for every user. ‘On the fly’ addition or updating of virtual applications (App-V Active Upgrade technology) is still possible, updates of the cache can still take place, although these will be saved in the delta file of the VDI workstation. In order to prevent this from becoming too large, it is advisable to periodically update the App-V cache in the read-only image.

APP-V SHARED CACHE

From version 4.6, App-V has a shared cache option, also referred to as a read-only cache. The virtual applications are thus no longer locally cached on the VDI workstation, and the App-V client reads the cache from a network location. Because nothing is cached locally on the client, IOPS can be saved on the storage layer. However, there is one major drawback: newly published applications can only be started by a user after the central read-only cache has been updated. Because the cache is read-only, the cache is no longer automatically updated. When using a read-only cache, the updating or addition of applications is also more difficult.

With the release of App-V 5, the shared cache function is further optimized. The shared cache is no longer read-only, so that the limitations of App-V 4.6 are no longer applicable.

8.2 VMWARE THINAPP

ThinApp application packages are stand-alone packages. The agent is included in every package so that the application with all of the associated files can be distributed in a single executable. A central distribution mechanism is therefore not required and the applications can easily be

http://www.microsoft.com/nl-nl/windows/enterprise/products-and-technologies/virtualization/app-v.aspx

http://www.microsoft.com/nl-nl/windows/enterprise/products-and-technologies/virtualization/app-v.aspx

http://www.vmware.com/products/thinapp

http://www.pqr.com/application-virtualization-smackdown

Whitepaper



placed on a file server. It must however be ensured that users have access to their allocated applications.

8.2.1 Impact on storage

ThinApp does not have a local application cache and streams the applications directly into the memory of the VDI workstation. Since a local application cache does not exist when using ThinApp, the impact of ThinApp on the underlying storage layer is less than when, for example, App-V is used. This is the case for the required storage capacity as well as the number of required IOPS.

VMware ThinApp uses so-called sandboxes to save user settings and other unique application

data for that user. The size of the sandbox can vary from several to hundreds of MB’s, depending on the application and this is, by default, located in the app data folder in the profile of the user. Using Microsoft Windows folder redirection prevents these sandboxes from being copied with the profile during the logging on of a user, and ending up in the delta file of the VDI workstation. The extra load that is placed on the file server must also be taken into account.

Whitepaper



9. ABOUT THE AUTHORS

HUGO PETER

Hugo has is already worked in the IT sector for 15 years. Having held several positions, he has gained a great deal of experience in the public as well as the private sector. Since 2011, Hugo has worked as an Application & Desktop Delivery Consultant at PQR where he focuses on providing advice and the design and implementation of new workstation concepts. Hugo has gained a wide range of certifications including Citrix Certified Enterprise Engineer (CCEE), Microsoft Certified Solutions Associate (MCSA): Windows Server 2012, RES Software Certified

Professional (RCP) and VMware Certified Professional (VCP).

Contact: [email protected] and www.twitter.com/hpeme.

PETER STERK

Peter Sterk works as a solution architect at PQR. In this function, he supports clients by solving every technical ICT challenge, in accordance with PQR’s slogan “Simplicity in ICT”. Whilst he mainly focuses on Enterprise Mobility and Application and Desktop Delivery, Peter has an overview of the entire IT infrastructure. Peter active participates in the vision of PQR in relation to subjects such as Enterprise Mobility and Application and Desktop Delivery at various (inter)national events.

Contact: [email protected] or www.twitter.com/PeterSterk.

HERCO VAN BRUG

Herco has worked at PQR since 2006, currently as solution architect. Key focus area: business continuity in the datacentre. In addition to being a certified Microsoft, RedHat, Citrix and VMware specialist, Herco is also a VMware Authorized Consultant and VMware Certified Advanced Professional, and performs assignments for VMware Branded Professional Services. Herco regularly gives presentations at congresses and seminars and has published various articles in IT journals, mainly relating to the subjects of virtualization and storage.

Contact: [email protected] and www.twitter.com/brugh.

ABOUT PQR

PQR is dé specialist voor professionele ICT-infrastructuren met focus op het veilig en beheersbaar beschikbaar stellen van data, applicaties en werkplekken met optimale gebruikerservaring.

PQR staat voor Eenvoud, Vrijheid en Professionaliteit en voorziet haar klanten van innovatieve ICT-oplossingen, van on-premises tot cloud management, zonder dat processen complexer worden. Eenvoud in ICT, dat is waar PQR voor staat.

Naast aantoonbare referenties beschikt PQR over brede expertise op het vakgebied, getuige de

vele hoogste partnerstatussen en certificeringen. PQR is onder meer Citrix Platinum Solution Advisor en Elite Partner, HDS Platinum Partner, HP Converged Infrastructuur en HP GOLD Preferred Partner, meervoudig Microsoft Gold Partner, NetApp Star Partner, RES Platinum Reseller en VMware Premier Partner.

PQR richt zich op vier hoofdthema’s:

Data & System Availability Application & Desktop Delivery

Secure Access & Secure Networking Advanced IT Infrastructure & Management

PQR is opgericht in 1990, gevestigd in De Meern en telt ruim 100 medewerkers. In het boekjaar 2011/2012 is een omzet geboekt van € 94,9 miljoen en een nettowinst na belastingen van € 4,6 miljoen.

mailto:[email protected]

http://www.twitter.com/hpeme


http://www.twitter.com/PeterSterk


http://www.twitter.com/brugh

PQR B.V.

Rijnzathe 7

3454 PV De Meern

the Netherlands

Tel: +31 (0)30 6629729

Fax: +31 (0)30 6665905

E-mail: [email protected]

www.PQR.com