searching external data in sharepoint 2010 using business connectivity services

8/10/2019 Searching External Data in SharePoint 2010 Using Business Connectivity Services

http://slidepdf.com/reader/full/searching-external-data-in-sharepoint-2010-using-business-connectivity-services 1/33



A SharePoint 2010 web application with a site collection. For more information, see Create a

probably use Sharepoint - 80 for these examples.

•

Access to Microsoft SQL Server. You can use the same instance of SQL Server that SharePoint•

A sample database for Microsoft SQL Server (AdventureWorks will be used in these examples.

to install.)

•

These examples will be more meaningful if you have some general understanding of SQL queries an

helpful. Even without that experience, you should be able to successfully complete all the examples.

Step-by-Step Example of Searching External Data

In this section, I will show you how to quickly set up a connection to external data using SharePoint

you will see how to configure the search service to index that data and show results in a way that is

Verify that the web application has connections to the proper services.•

Create a profile page site and a search site.•

Configure the Business Data Connectivity Service.•

Create the External Content Type.•

Configure the Search Service and index the external data with a crawl.•

Test the search.•

As you work through the specific steps that follow, you will be using four main areas:

Business Data Connectivity in SharePoint 2010 Central Administration•

Search in SharePoint 2010 Central Administration•

SharePoint Designer•

The search site that you will be creating•

I suggest that once you have started working in one of those areas that you keep that window open.

Business Data Connectivity, open a new window to configure the search service.

The first step is to verify that the web application has a connection to the necessary services. If any

them.

Procedure: Verifying the Web Application Connections

Start SharePoint 2010 Central Administration.1.

Click Manage web applications under the Application Management heading.2.

Select the web application you will be using for these examples (e.g. Sharepoint - 80).3.

Click Service Connections in the ribbon.

Figure 1. The Web Applications ribbon

4.

You need to have checks by Business Data Connectivity Service and a search service, like Sear

service the web application uses so that you are configuring the correct one. (For example, yo

5.

of 33MSDN Blogs

1/25/2013http://blogs.msdn.com/b/ericwhite/archive/2010/04/27/searching-external-data-in-sharepoi...



are not checked, check them and click OK. If you cannot check them, use the drop down at th

Figure 2. Dialog for Web Applications connections

The next step is to configure the Business Data Connectivity Service. You will need to set the permis

are used to show the detailed information about an external data record. The search service will lin

set up a profile page, then the search results will contain a useless link. Each profile page will show t

need a different page for each type. You could host these pages on any site, but it is usually better t

collection for those pages so that they are not stored with regular site content. These instructions w

collection.

Procedure: Creating a site for profile pages

Open the site collection you will be using with Internet Explorer.1. On the Site Actions drop-down, click New Site.2.

Select the Blank Site template.3.

In the right pane, click the More Options button.4.

Type "Search Results Details" in the Title box.5.

Type "ProfilePages" in the URL name box.6.

In the Navigation section, click the No radio button for Display this site on the top link bar of 7.

Click Create.8.

of 33MSDN Blogs




The URL that you see is the one you will use for the next step. For example, if the URL for the profile

then you will specify the location of the profile pages as "http://server1/ProfilePages" as the host U

Procedure: Configuring the Business Data Connectivity Service

Start SharePoint 2010 Central Administration.1.

Click Manage service applications under the Application Management heading.2.

In the Name column, click on Business Data Connectivity Service.

Figure 3. Manage Service Applications list

3.

Click on the Edit tab.4.

In the ribbon, click on Configure in the Profile Pages group.5.

In the Host SharePoint site URL box, enter the full URL for the site you just created for profile

Figure 4. Configuring the ECT profile page host

6.

Click OK.7.

In the ribbon, click Set Metadata Store Permissions in the Permissions group.8.

of 33MSDN Blogs




If the account you will be using is not shown in the dialog, type the account name in the text b9.

Click Add.10.

In the bottom list, check all of the check boxes (Edit, Execute, Selectable In Clients and Set Pe11.

Check the box for Propagate permissions to all BDC Models, External Systems and External C12.

Click OK.13.

You must also create a simple search site for testing.

Procedure: Creating a Basic Search Site

Open the site collection you will be using with Internet Explorer. (Make sure you have navigat

not still viewing the site you created for the profile pages.)

1.

On the Site Actions drop-down, click New Site.2.

Select the Basic Search Center template.3.

In the right pane, type "BCS Search" for the Title and "BCSSearch" for the URL name.4.

Click Create.5.

Next, you will create an external content type (ECT) for the product and description view in the Adv

root of the site collection or any of the sites in that collection, including the one you just created. Thsystem. Then you will define two basic operations to make the ECT usable by the search service. The

from the ECT. The Read Item operation is used to get a specific record from the ECT. Both operation

identifies each record.

Procedure: Creating the External Content Type

On the Site Actions drop-down for the BCS Search site that you just created, click Edit in Shar

of the four windows open that I mentioned previously. Only the search service has not been o

1.

In the Navigation pane, click External Content Types.2.

In the New group of the ribbon, click External Content Type.

Figure 5. Creating a new External Content Type

3.

of 33MSDN Blogs




Figure 7. SQL Server Connection dialog

Enter "AdventureWorks" for the Database Name box.10.

Enter "AWProducts" for the Name (optional) box.11.

Click OK.

Note: This connection will use the search service account to connect to the database. In a pro

impersonated identities so that database access is controlled independently from that accoun

12.

Click on the expansion symbol next to AWProducts.13.

Click on the expansion symbol next to Views.14.

Right-click on vProductAndDescription to open a pop-up menu and click New Read Item Ope15.

of 33MSDN Blogs




Figure 8. Data Source Explorer for an External Content Type

Click the Next > button at the bottom of the Read Item dialog.16.

On the Input Parameters page, click ProductID in the Data Source Elements pane.17.

of 33MSDN Blogs




Figure 9. Read Item dialog

In the Properties pane, check the box for Map to Identifier.18.

Click the Next > button.19.

On the Return Parameter page, click ProductID in the Data Source Elements pane.20.


Click the Finish button.22.

Right-click on vProductAndDescription to open a pop-up menu and click New Read List Oper23.

Click the Next > button at the bottom of the Read List dialog.24.

Click the Next > button.25.On the Return Parameter page, click ProductID in the Data Source Elements pane.26.



In the ribbon, click on Summary View.29.

In the Fields group, click on Name in the list.30.

In the ribbon, click on Set as Title.31.

of 33MSDN Blogs




Figure 10. Completed External Content Type

Right-click on the Products tab and click Save on the pop-up menu. (You can also use the disk32.

In the ribbon, click Create Profile Page.33.

Toward the end of creating the ECT, you set the field that will be used as the title. That is the value t

don't specify a title, the search results will show cryptic text as the title that is likely to be confusing

Now you need to configure the search service to use this ECT while crawling and indexing. The searc

the content it needs to search. You will also need to set the account to use for the crawl operation.

set permissions in the Business Data Connectivity service and it must have read access to the Adven

Note: In a production configuration, a new account should be created and used only to control per

Procedure: Configuring the Search Service

Start SharePoint 2010 Central Administration. (If you open this as a new window, then you wil1.

Click Manage service applications under the Manage Applications heading.2.

In the Name column, click on Search Service Application (or the search service that is connect3.

Click the link to the right of Default content access account to change the account that will be

used to set permissions in the Business Data Connectivity service.

4.

In the navigation pane, under the Crawling heading, click Content Sources.5.

Click New Content Source.6.

of 33MSDN Blogs




Figure 11. Content Source configuration in the search service

Type "AWProductSearch" in the Name box.7.

Click on the Line of Business Data radio button.8.

Click on the Crawl selected external data source radio button.9.Check the box next to AWProducts.

Figure 12. New Content Source dialog

10.

Check the box next to Start full crawl of this content source.11.

Click OK.12.

of 33MSDN Blogs




The search service is now crawling through the new data to create search indexes for that content. I

millions of records, the crawl would take significantly longer. You can click on the Refresh link to upd

crawl is done.

Procedure: Testing the Search

Open the "BCS Search" site you created earlier.1.

Type "Jersey" in the search box and then Enter.2.Click on any of the results titles to see the profile page for that item.

Figure 13. Profile page from the search results

3.

Now you have a basic search working with external data. Next you will see how to control the appea

Customizing Search Results

There are two simple modifications you can make to the search using SharePoint Designer. First, yo

and their descriptions by modifying the Read Item operation. For example, the descriptions "Produc

friendly. Also, the CultureID field is not particularly useful.

If you have kept SharePoint Designer open, you can skip the first two steps below.

Procedure: Changing the Fields Shown on the Profile Page

In SharePoint Designer, click External Content Types in the navigation pane.1.

In the list, click on Products to edit that ECT.2.

In the External Content Type Operations group, double-click on the Read Item operation.3.

Click Next > twice.4.

In the Return Parameter page, uncheck the box by CultureID.5.

Click on ProductID.6.In the Properties pane, type "ID" in the Display Name box.7.

Click on ProductModel.8.

In the Properties pane, type "Model" in the Display Name box.9.

Click Finish.10.

Click on the disk icon to save the ECT.11.


Click Yes on the warning dialog.13.

of 33MSDN Blogs




Using Associations for Master/Detail Relationships

Business Connectivity Services also allows you to define a master/detail (aka parent-child) relationsh

master ECT will automatically be generated with a list of the detail records for that particular maste

handles this type of association.

To try this out, you will create an association from the detail ECT (Product) to the master ECT (Produ

the search to see how the results are generated for those. Then you will see how to modify the confiresults by treating the detail records as attachments to the master record.

Procedure: Creating an Association in an External Content Type

In the Navigation pane in SharePoint Designer, click External Content Types.1.

On the New section of the ribbon, click External Content Type.2.

In the External Content Type Information group, next to Name, click New external content ty

be replicated to the Display Name text.)

3.

In the External Content Type Information group, next to Display Name, click ProductModel.

profile page.)

4.

Next to External System, click Click here to discover external data sources and...5.Click on the Add Connection button.6.

In the External Data Source Type Selection dialog, choose SQL Server in the drop down.7.

Click OK.8.

In the SQL Server Connection dialog, enter "localhost" for the Database Server.9.

Enter "AdventureWorks" for the Database Name box.10.

Enter "AWModel" for the Name (optional) box.11.

Click OK.12.

Click on the expansion symbol next to AWModel.13.

Click on the expansion symbol next to Tables.14.

Right-click on ProductModel (you will probably have to scroll down to find this table) to open15.Click the Finish button.16.

Right-click on ProductModel to open a pop-up menu and click New Read List Operation.17.





Save the ECT by clicking on the disk icon.22.

In the Navigation pane in SharePoint Designer, click External Content Types.23.

On the New section of the ribbon, click External Content Type.24.

In the External Content Type Information group, next to Name, click New external content ty

replicated to the Display Name text.)

25.

Next to External System, click Click here to discover external data sources and...26.

Right-click on Product (in Tables within AWModel) to open a pop-up menu and click New Rea27.


Right-click on Product to open a pop-up menu and click New Read List Operation.29.


Right-click on Product to open a pop-up menu and click New Association.31.

of 33MSDN Blogs




Type "Products" in the box for the Association Display Name. (This will change the label for th

record.)

32.

Click the Browse button.

Figure 15. Choosing the External Content Type for an Association

33.

Click ProductModel to select it.34.

Click OK.35.Click Next >.36.

In the Data Source Elements pane on the Input Parameters page, click ProductModelID to sel37.



In the Data Source Elements pane on the Return Parameter page, uncheck all of the fields ex

and ProductModelID.

40.

If you used the top check box to uncheck all the fields, you will need to set the identifier again

then check Map to Identifier in the Properties pane.

41.

Click ProductID and change the text for the Display Name to "ID."42.

Click ProductNumber and change the text for the Display Name to "Product Number."43.Click ListPrice and change the text for the Display Name to "List Price."44.

of 33MSDN Blogs




Figure 16. Return Parameter page for association





Save the ECT using the disk icon.49.


Click the Model tab to see the first ECT.51.


You probably noticed that I had you create a new named connection for the same AdventureWorks

search will crawl all ECTs defined with the same data source. If you want to be able to completely co

they are crawled, you will need to create different data sources so that you can also create different

You may also have noticed that you did not have to set the identifiers for these ECTs. SharePoint De

tables, but not for views.

The associated ECTs are ready. Now you need to create a new search content source to index them.

you can see the results of the new indexing only.

of 33MSDN Blogs




Procedure: Crawling and Indexing the Associated ECTs

In the search service configuration in SharePoint Central Administration, click on Content Sou1.

Click on the drop-down button for the AWProductSearch content source. Click Delete.2.

Click OK to delete.3.

Click New Content Source.4.

Type "AWModelSearch" in the Name box.5.

Click on the Line of Business Data radio button.6.

Click on the Crawl selected external data source radio button.7.

Check the box next to AWModel.8.

Check the box next to Start full crawl of this content source.9.

Click OK.10.

When the crawl is done (click Refresh to update the status), you can search for "Jersey" and you will

(Product). The model results do not have sizes (e.g. S, M, L) and the link for them includes the words

one of those and you will see the master/detail view that is created by the association. Although thi

indexing information in the detail. You can see matches for the detail, but they are not based on the

"Dissolver," you will only see the detail in the results. To make these results better, you will first elithe ECT to "attach" the detail record information to the master record. The first change is one you h

the fields that can be indexed.

Figure 17. Profile Page for a Master/Detail Association

If you have left the tabs open for the ECTs you just defined, skip the first two steps.

Procedure: Changing the Field Values Indexed for Search


In the list, click on ProductDetail to edit that ECT.2.

of 33MSDN Blogs




In the External Content Type Operations group, double-click on the Read List operation.3.


In the Return Parameter page, uncheck the boxes for all of the fields in the Data Source Elem

you use the check box to uncheck all fields, you will need to check Map to Identifier for Produ

5.

Click Finish.6.

Save the ECT using the disk icon.7.

Follow the usual steps to run a full crawl after changing the model. Remember to delete the existing

you search for "Dissolver," there should be no results. You will now modify the model so that the as

indexing. This change is not supported by SharePoint Designer, so you will need to modify the mode

model to an XML file, edit it and then import the changed file.

Procedure: Exporting the BCS Model to an XML file


In the list, click AWModel next to ProductDetail to select that ECT.2.

In the ribbon, click Export BDC Model.3.

In the Export BDC Model dialog, type "ProductDetail" in the BDC Model Name box.4.

Click OK.5.

Change the Save as type to All Files.6.

Choose a destination folder for the file.7.

Change the File Name box to "ProductDetail.xml"8.

Click Save.9.

You can use any XML editor to change this file. I use SharePoint Designer by finding the file in a file b

SharePoint Designer from the right-click popup menu. You could also drag the file into SharePoint D

Modify the XML file to add the highlighted line shown below. Be sure you are in the MethodInstanc

the Read List or Read Item methods.

<MethodInstances> <Association Name="ProductNavigate Association" Type="AssociationNavigator" ReturnParameterName="ProductNavigate Association"

DefaultDisplayName="Products"> <Properties> <Property Name="AttachmentAccessor" Type="System.String"></Property> <Property Name="ForeignFieldMappings" ...>

</Properties>

Save the change and then follow the steps below to import the model with the change.

Procedure: Importing an XML file to Update a BCS Model

In the BDC service configuration in SharePoint Central Administration, click on the Edit tab.1.

In the View section of the ribbon, select External Content Types from the drop-down list.2.

Click on the drop-down menu for ProductDetail and click Delete.3.

Click OK.4.

In the BDC Models section of the ribbon, select Import.5.

In the BDC Model File box, type your XML file name or click the Browse button to browse for i6.

of 33MSDN Blogs




Click the Import button. You may see one or two warnings, but there should be no errors.7.

Click OK.8.

Note: When you exported the model, you chose a name for it that also happens to be the same as t

may seem like the same thing, they are not. If you had to import that same model file again, you wo

even if you had deleted the ECT. Whenever you are importing the same model a second time, you m

error.

Follow the usual steps to recreate the AWModelSearch content source and run a full crawl. Now wh

results. You can also try searching for "Jersey" and see that it only returns the shows the model reco

Explanation of the BCS Model XML

You made a modification to the XML version of the BCS Model in the last section. This section will ex

skip down to the next section about incremental crawls, if you are not that interested in these detail

examples. It will help if you are looking at an actual model XML file while reading through this.

The top-level element is the Model. It has a Name attribute that will appear as the name of the mod

may contain one or more LobSystem child elements. The LobSystem element has a Name attribute talso has a Type attribute for the type of data source. For these examples, the Type is "Database."

The actual connection information for that data source is defined as Property elements in the LobSy

LobSystemInstances. LobSystemInstances is a child of LobSystem. Property elements are very comm

Properties element that contains one or more Property elements. A Property element has a Name a

that identifies the .Net type of the value for that property. The value then appears in the body of th

LobSystemInstance are shown in the table below.

Table 1. Properties for LobSystemInstance

Name Type Example Value

AuthenticationMode System.String PassThrough

DatabaseAccessProvider System.String SqlServer

RdbConnection Data Source System.String localhost

RdbConnection Initial Catalog System.String AdventureWorks

RdbConnection Integrated

Security

System.String SSPI

RdbConnection Pooling System.String True

of 33MSDN Blogs




ShowInSearchUI System.String <anything>

You might have noticed that all the properties that start with "RdbConnection" are regular connecti

appears in a connection string can defined in this list of properties by prefixing it with "RdbConnecti

have value. Even though it is defined as a string, just having the property is all that is needed for the

appear in the user interface.

Another child of LobSystem is Entities. Although it may contain one or more Entity children, you will

equivalent to a single External Content Type. There are several attributes for an Entity, including the

an Entity element are shown in the table below.

Table 2. Properties for Entity


Title System.String Name

DefaultAction System.String View Profile

The Title property defines the field that will be used as the title in the search results. The DefaultActi

action will be used as the link from the search results. In these examples, that action is always set to

Some child elements for an Entity are AccessControlList, Identifiers, AssociationGroups, Actions and

The AccessControlList element may contain one or more AccessControlEntry elements that have a P

access. The child elements are Right elements that define the access rights for the principal. Each Ri

Execute, SetPermissions or SelectableInClients.

The Identifiers element contains an Identifier element with a Name attribute for the field name that

that contains the .Net data type of the identifier.

The AssociationGroups element is used to define associations between ECTs. The Actions element c

page. These are easily defined by SharePoint Designer, so I won't say much about them here.

The Methods element contains one or more Method elements. Each Method element has a Name a

entity. It also has an AccessControlList element that is identical to the one in the Entity element. The

below.

Table 3. Properties for Model


RdbCommandType System.String System.Data.CommandType ...

of 33MSDN Blogs




RdbCommandText System.String SELECT [ProductID],

[ProductModelID] FROM

[Production].[Product]

BackEndObjectType System.String SqlServerTable

BackEndObject System.String Product

Schema System.String Production

The RdbCommandText is very important since it allows you to greatly customize the SQL query used

to create a view on a particular database, you could still modify that text to do joins within the com

BackEndObjectType can be SqlServerView if the RdbCommandText refers to a view instead of table.

FilterDescriptors is a child of the Method element. It may contain one or more FilterDescriptor elemin its user interface, but they are not generally that useful for search methods where you have contr

SynchronizationCookie, that will be shown in the next section.

Parameters is a child of the Method element. It may contain one or more Parameter child elements.

uniquely identifies the parameter and Type attribute that can be In, InOut or Return. There can only

type. The number and types of parameters are determined by the operation that the method imple

SharePoint Designer, so it is easiest to create the operation there and then edit as necessary. Two o

Designer, ChangedIdEnumerator and DeletedIdEnumerator, will be defined in the next section.

The parameter is actually defined by the child TypeDescriptor element. This element has attributes

elements have simple types and are defined by a single TypeDescriptor element. Others can be veryTypeDescriptor elements and additional properties. Some of the properties for the TypeDescriptor e

Table 4. Properties for TypeDescriptor


Size System.Int32 25

RequiredInForms System.Boolean true

Decimal Digits System.Int32 4

For the most part, you should be able to copy and edit the parameters that are defined by SharePoi

The last important child of the Method element is MethodInstances. It contains a MethodInstance e

attribute may be Finder, SpecificFinder, AssociationNavigator, IdEnumerator and others. The Return

of 33MSDN Blogs




attribute that defines the return value for the operation. The Default attribute will usually be "true"

operation. If there were more than one Finder, for example, this would specify which one would be

MethodInstance and the DefaultDisplayName will be used whenever that operation name is display

properties as shown in the table below.

Table 5. Properties for TypeDescriptor


RootFinder System.String <anything>

AttachmentAccessor System.String <anything>

LastModifiedTimeStampField System.String ModifiedDate

The RootFinder marks the Finder method that the search crawler will use to enumerate all of the to

only one Finder method that is marked as the RootFinder. The AttachmentAccessor property was di

LastModifiedTimeStampField property is discussed in the next section.

If you want to see more documentation of the BCS Model, see the links in the Additional Resources

Modifying the External Content Type to Support Incremental Crawls

It's easy enough to run full crawls all the time for the numbers of records in these examples, but it d

records. Once the ECT and search is set up for that kind of volume, it is much better to run incremen

then update just those in the search index. SharePoint search supports two approaches for handling

advantages and disadvantages to each, but first you need to understand how they function.

Timestamp-based Incremental Crawls

In this approach, you define a field that contains the last modified timestamp for each record. If tha

indexes for that record are updated. For this example, you will be modifying the ECT based on a vie

the view in the AdventureWorks database.

Procedure: Modifying the vProductAndDescription View to Add the Modified Date

Start SQL Server Management Studio.1.

In the Connect to Server dialog, select Database Engine as the Server type.2.Type "localhost" in the Server name box.3.

Click Connect.4.

Expand the Databases item in the Object Explorer group.5.

Right-click on the AdventureWorks item and click New Query.6.

Execute the ALTER VIEW statement as shown below.7.

ALTER VIEW [Production].[vProductAndDescription]

WITH SCHEMABINDINGAS

of 33MSDN Blogs




SELECTp.[ProductID]

,p.[Name],pm.[Name] AS [ProductModel]

,pmx.[CultureID],pd.[Description],CASE WHEN p.ModifiedDate > pm.ModifiedDate THEN p.ModifiedDate ELSE pm.ModifiedDate END

FROM [Production].[Product] p

INNER JOIN [Production].[ProductModel] pmON p.[ProductModelID] = pm.[ProductModelID]INNER JOIN [Production].[ProductModelProductDescriptionCulture] pmx

ON pm.[ProductModelID] = pmx.[ProductModelID]INNER JOIN [Production].[ProductDescription] pdON pmx.[ProductDescriptionID] = pd.[ProductDescriptionID];

This adds one new field to the view. For this example, the ModifiedDate is the most recent modified

to include ProductDescription in that expression, you would just need to add another case to deter

Procedure: Adding the ModifiedDate to the ECT Operations


In the list, click on Products to edit that ECT.2.

In the External Content Type Operations group, double-click on the Read Item operation.3.


In the Return Parameter page, check the ModifiedDate field in the Data Source Elements pan5.

Click Finish.6.

In the External Content Type Operations group, double-click on the Read List operation.7.


In the Return Parameter page, check the ModifiedDate field in the Data Source Elements pan9.

In the Properties pane, check the box next to Timestamp Field.

Figure 18. Setting the Timestamp Field

10.

of 33MSDN Blogs




Click Finish.11.

Save the ECT by clicking on the disk icon.12.

By checking the box for the Timestamp Field, you have set that field to be used by the search service

crawl. Internally, the XML will now have a LastModifiedTimeStampField property that specifies the

Products and searched for the property, you would find it in the MethodInstances properties for the

You can now follow the usual steps to create the AWProductSearch content source and run a full cr

AWModelSearch to avoid seeing results from both sources. When the full crawl is done, start an inc

completed, you can now see if it worked.

Procedure: Examining the Results of the Incremental Crawl

In the search service configuration in SharePoint Designer, click on Crawl Log under Crawling i1.

Click Crawl History at the top.2.

You should see that the full crawl had something like 296 successes and the incremental crawl had o

incremental crawl did not have to re-index the records.

If you want to test the modified date, make a change to one or more of the records that includes up

and then run another incremental crawl. The successes will be higher than 2 (3 if you change just on

shows an example of that sort of change.

UPDATE Production.Product

SET Name='Classical Vest, M',ModifiedDate=GETDATE()WHERE ProductID=865

Changelog-based Incremental Crawls

In this approach, SharePoint search stores a token or cookie that we designate to track changes sinc

timestamp from the last time that a crawl was started. If there have been any changes since that ti

This process requires two new operations to be defined for the ECT, one for changed records and on

because SharePoint Designer does not have a way to create them. (If you look at the ECT in SharePo

Type Operations list, but you will not be able to edit them.) As before, you will need to export the B

Procedure: Exporting the BCS Model to an XML file


In the list, click AWProducts next to Products to select that ECT.2.

In the ribbon, click Export BDC Model.3.

In the Export BDC Model dialog, type "Products" in the BDC Model Name box.4.

Click OK.5.

Change the Save as type to All Files.6.

Choose a destination folder for the file.7.

Change the File Name box to "Products.xml"8.

Click Save.9.

In order to implement these two operations, you will need two SQL queries. One will return the rec

will return the records deleted after a particular timestamp. For the product table, I will use the Disc

of 33MSDN Blogs




effectively deleted and should no longer appear in search results. If the DiscontinueDate field is null,

table will need to be added to the view just like the ModifiedDate was added previously, though the

a simple added field (p.DiscontinuedDate).

Here are the new methods that you will need to add to the BCS model for the Products ECT within t

SynchronizationCookie filter type to track the last run date and time. The SQL query must return the

also update the last run date and time. The key SQL is highlighted in the XML for these two operatio

<Method IsStatic="false" Name="GetChangedProductIds"> <Properties>

<Property Name="RdbCommandType" Type="System.Data.CommandType, System.Data, Version=PublicKeyToken=b77a5c561934e089">Text</Property> <Property Name="RdbCommandText" Type="System.String">

if @LastRunDate != CAST('1900-1-1' as datetime)BEGIN SELECT [ProductID] FROM [Production].[vProductAndDescription] WHERE [ModifiedDate]END

SET @LastRunDate=CURRENT_TIMESTAMP </Property>

<Property Name="BackEndObjectType" Type="System.String">SqlServerView</Property>

<Property Name="BackEndObject" Type="System.String">vProductAndDescription</Property <Property Name="Schema" Type="System.String">Production</Property> </Properties>

<AccessControlList> <AccessControlEntry Principal="contoso\administrator"> <Right BdcRight="Edit" /> <Right BdcRight="Execute" />

<Right BdcRight="SetPermissions" /> <Right BdcRight="SelectableInClients" />

</AccessControlEntry>

</AccessControlList> <FilterDescriptors> <FilterDescriptor Name="LastRunDate" Type="InputOutput"> <Properties>

<Property Name="SynchronizationCookie" Type="System.String">ChangedItemCookie</P </Properties>

</FilterDescriptor> <FilterDescriptor Name="FilterDescriptor" Type="Timestamp" />

</FilterDescriptors> <Parameters> <Parameter Name="@LastRunDate" Direction="InOut">

<TypeDescriptor Name="LastRunDateTypeDescriptor" TypeName="System.DateTime" Associ <Interpretation>

<NormalizeDateTime LobDateTimeMode="Local" /> </Interpretation>

</TypeDescriptor> </Parameter> <Parameter Direction="Return" Name="ChangedProductIds">

<TypeDescriptor TypeName="System.Data.IDataReader, System.Data, Version=2.0.0.0, CIsCollection="true" Name="ChangedProductIds">

<TypeDescriptors> <TypeDescriptor TypeName="System.Data.IDataRecord, System.Data, Version=2.0.0.PublicKeyToken=b77a5c561934e089" Name="ChangedProductIdElement"> <TypeDescriptors>

<TypeDescriptor TypeName="System.Int32" ReadOnly="true" IdentifierName="Pr

of 33MSDN Blogs




<Properties> <Property Name="RequiredInForms" Type="System.Boolean">true</Property>

</Properties> </TypeDescriptor>

</TypeDescriptors> </TypeDescriptor> </TypeDescriptors>

</TypeDescriptor>

</Parameter> </Parameters> <MethodInstances>

<MethodInstance Name="GetChangedPrimaryKeysInstance" Type="ChangedIdEnumerator" Retu </MethodInstances></Method><Method IsStatic="false" Name="GetDeletedProductIds">

<Properties>

<Property Name="RdbCommandType" Type="System.Data.CommandType, System.Data, Version=PublicKeyToken=b77a5c561934e089">Text</Property>

<Property Name="RdbCommandText" Type="System.String">if @LastRunDate != CAST('1900-1-1' as datetime)BEGIN

SELECT [ProductID] FROM [Production].[vProductAndDescription] WHERE [DiscontinuedDENDSET @LastRunDate=CURRENT_TIMESTAMP </Property>

<Property Name="BackEndObjectType" Type="System.String">SqlServerView</Property> <Property Name="BackEndObject" Type="System.String">vProductAndDescription</Property <Property Name="Schema" Type="System.String">Production</Property> </Properties>

<AccessControlList> <AccessControlEntry Principal="contoso\administrator"> <Right BdcRight="Edit" />

<Right BdcRight="Execute" /> <Right BdcRight="SetPermissions" />

<Right BdcRight="SelectableInClients" /> </AccessControlEntry>

</AccessControlList> <FilterDescriptors> <FilterDescriptor Name="LastRunDate" Type="InputOutput"> <Properties>

<Property Name="SynchronizationCookie" Type="System.String">DeletedItemCookie</P </Properties> </FilterDescriptor>

<FilterDescriptor Name="FilterDescriptor" Type="Timestamp" /> </FilterDescriptors> <Parameters> <Parameter Name="@LastRunDate" Direction="InOut">

<TypeDescriptor Name="LastRunDateTypeDescriptor" TypeName="System.DateTime" Associ <Interpretation> <NormalizeDateTime LobDateTimeMode="Local" />

</Interpretation> </TypeDescriptor> </Parameter> <Parameter Direction="Return" Name="DeletedProductIds">

<TypeDescriptor TypeName="System.Data.IDataReader, System.Data, Version=2.0.0.0, CIsCollection="true" Name="DeletedProductIds"> <TypeDescriptors> <TypeDescriptor TypeName="System.Data.IDataRecord, System.Data, Version=2.0.0.

of 33MSDN Blogs




PublicKeyToken=b77a5c561934e089" Name="DeletedProductIdElement"> <TypeDescriptors>

<TypeDescriptor TypeName="System.Int32" ReadOnly="true" IdentifierName="Pr <Properties>

<Property Name="RequiredInForms" Type="System.Boolean">true</Property> </Properties> </TypeDescriptor>

</TypeDescriptors>

</TypeDescriptor> </TypeDescriptors> </TypeDescriptor>

</Parameter> </Parameters> <MethodInstances> <MethodInstance Name="GetDeletedPrimaryKeysInstance" Type="DeletedIdEnumerator" Retu

</MethodInstances>

</Method>

Although these methods appear very long, they are actually pretty simple and similar. The filter is d

timestamp containing both date and time. The return parameter is defined as a set of records contai

define these methods as the changed record and deleted record operations.

The SQL code is also similar. Both check to see if the "LastRunDate" timestamp is valid. If so, they re

the timestamp is updated to the current date and time so that the cookie will be stored with the ne

ModificationDate to the last timestamp and the deleted record method compares the Discontinued

There is one other change that should be made to correctly implement this approach. The SQL for t

no deleted records will be included in a full crawl, as shown below.

<Property Name="RdbCommandText" Type="System.String">SELECT [ProductID],[Name],[ProductModel],[CultureID],[Description]

FROM [Production].[vProductAndDescription]WHERE [DiscontinuedDate] IS NULL</Property>

Make these changes to the Products model, and you might also want to remove the LastModifiedTi

are doing the job. Import the modified model using the BDC service configuration in SharePoint Cen

Procedure: Importing an XML file to Update a BCS Model

In the BDC service configuration in SharePoint Central Administration, click on the Edit tab.1.

In the View section of the ribbon, select External Content Types from the drop-down list.2.

Click on the drop-down menu for Products and click Delete.3.Click OK.4.

In the BDC Models section of the ribbon, select Import.5.

In the BDC Model File box, type your XML file name or click the Browse button to browse for i6.

Click the Import button. You may see one or two warnings, but there should be no errors.7.

Click OK.8.

Now you can try it out by recreating the content source for search and then making changes to the r

see the results.

of 33MSDN Blogs




If the database really needs to delete records, rather than just mark them as "deleted," you will nee

The easiest is to create a new table for deleted IDs and add a trigger to the original table that adds r

table of deleted records could store just IDs and a timestamp. If your incremental crawls are run on

delete older records from the table.

Comparing the Two Approaches

The timestamp-based approach is much simpler to implement, but the changelog-based approach isrequires that the search service to store the timestamps for every record and then compare them al

-based approach only requires the search service to store a single timestamp. In addition, when the

the records that have changed. If you have a very large number of records, then the changelog-base

have very few records, then the incremental crawls aren't really necessary. If you aren't sure which

approach first. Then, if it seems to be too slow or be taking too much disk storage, try switching to t

have any control over the database, you may be forced to use the approach that works with the exis

What Ever Happened to IDEnumerator?

If you have used SharePoint search with external data since the Business Data Catalog was introduc

IDEnumerator instead of the Read List operation. The IDEnumerator method still works for searchesOne of the uses of IDEnumerator was also to allow for very large external lists that otherwise would

not constrained by the throttle settings, so there is no need to use "segmenting" techniques. If you

use it for search as well. For new ECT definitions, it is quicker to use the Read List operation in Share

Using the Same External Content Type for an External List

The use of ECTs for external lists is not really within the scope of this paper, but you might be tempt

many of the customizations for search could also affect the appearance of the external list. Also, yo

could interfere with the efficiency of the search crawls. In general, it is probably better to use differ

simple and less than a thousand records.

Additional Resources

For more information, see the following resources:

Enterprise Search administration

Business Data Connectivity services administration

Business Connectivity Services Schema Reference

Stereotyped Operations Supported By BDC

The Notion of Associations and the External Item Picker

Tooling Associations in SharePoint Designer 2010

Configure the Secure Store Service (SharePoint Server 2010)

of 33MSDN Blogs




DL

3 May 2010 4:58 PM

Thorough technical blog posts like this are vital! We appreciate the great efforts that have gone into th

Moe1 Jun 2010 8:37 AM

Thank you for this article Eric.

I was searching for this topic for some time now. I am trying to figure out which approach to use for pr records) to our users: providing search through a content source reading from an ECT or exposing theknow we could use both at the same time (i.e. search and the filtering through a web part), but we nee

I guess after reading this article, I will be going with the search option using the incremental crawls.

Thank you.

Sara

11 Jun 2010 8:31 AM

This is incredibly helpful. When I'm configuring the Business Data Connectivity Service, I get an error Access is denied by Business Data Connectivity, with a long and unhelpful correlation ID. Any ideas a

Eric White - Independant Consultant

11 Jun 2010 2:15 PM

Hi Sara,

I'm not sure why. Have you reviewed Rob Silver's permissions post?

blogs.msdn.com/.../permissions-in-business-connectivity-services.aspx

Also, if you use SPD to modify the ECT, you need to re-setup permissions:

blogs.msdn.com/.../access-denied-by-business-data-connectivity.aspx

-Eric

Kirk Liemohn

30 Jul 2010 6:38 AM

I have a few questions regarding incremental crawls and throttling:

1) Is it true that deleted items cannot be handled with timestamp-based incremental crawls? I didn't s Regardless, the better approach seems to be changelog-based if feasible

2) In your large XML snippet for changelog-based crawls you use @LastRunDate for the RdbCommafor the RdbCommandTExt in the GetDeletedProductIds method. Am I missing something? It's mentio

3) Is there a way to run queries in batches for throttling purposes? Say you are doing a full crawl andrecords 1000 at a time? How do you prevent a huge full crawl query from causing memory and resou

of 33MSDN Blogs




BobMcClellan

30 Jul 2010 10:34 AM

Kirk,

1. The search service handles deletions for you when using the timestamp-based incremental crawl. It

their current timestamp. If it does not find one of the old records in the new list, then it assumes that renot actually delete the records, you will need to manually modify the SQL for the Read List operation swith the same kind of change as I described for the changelog-based approach.)

2. You are not missing something, the article is. The correct SQL for the GetDeletedProductIds metho

If @LastRunDate != CAST('1900-1-1' as datetime)

BEGIN

SELECT [ProductID] FROM [Production].[vProductAndDescription]

WHERE [DiscontinuedDate] > @LastRunDate

END

SET @LastRunDate=CURRENT_TIMESTAMP

That was totally my fault. (Eric, perhaps you can replace that SQL and make it bold, just like the GetC

3. The way to prevent huge crawls is to use the changelog-based incremental crawl so that only the call the records the first time. If you want that huge index, it is going to take some time, but there is no ngroups as the search service is processing them. It shouldn't be a problem, but you will have a big stoSearch Results, you can remove fields from the Read List operation so that they won't be indexed and

-Bob

BobMcClellan

30 Jul 2010 10:38 AM

Hmm, in my previous comment, the post has changed my "& gt ;" (remove spaces) into a >.

Kirk Liemohn

30 Jul 2010 1:51 PM

Bob, thanks for the quick response - and thanks to you and Eric for an excellent posting!!!

What you have said makes sense. For the throttling part it sounds like I should go to search administr server will adjust accordingly. I have had a problem with the MOSS 2007 BDC where a crawl basicallyagainst an underpowered test environment, but I still want to be a good citizen for the rest of the Shar

Eric White - Independant Consultant

30 Jul 2010 3:20 PM

I've updated the post to contain the correct code. Thanks for the update, Bob.

of 33MSDN Blogs




Kirk Liemohn

5 Aug 2010 10:09 PM

A few more comments now that I have gone through this completely.

1. I believe there is a typo for the second time you are importing a BDC model. It specifies to delete th"Products" ECT.

2. For those that like to have it spelled out in black and white, here is the alter view statement needed

ALTER VIEW [Production].[vProductAndDescription]

WITH SCHEMABINDING

AS

SELECT

p.[ProductID]

,p.[Name]

,p.[DiscontinuedDate]

,pm.[Name] AS [ProductModel]

,pmx.[CultureID]

,pd.[Description]

,CASE WHEN p.ModifiedDate > pm.ModifiedDate THEN p.ModifiedDate ELSE pm.ModifiedDate END

FROM [Production].[Product] p

INNER JOIN [Production].[ProductModel] pm

ON p.[ProductModelID] = pm.[ProductModelID]

INNER JOIN [Production].[ProductModelProductDescriptionCulture] pmx

ON pm.[ProductModelID] = pmx.[ProductModelID]

INNER JOIN [Production].[ProductDescription] pd

ON pmx.[ProductDescriptionID] = pd.[ProductDescriptionID];

3. To test out a modification and deletion for the changelog-based incremental crawl, here is the SQL


SET Name='Classical Vest, M',ModifiedDate=GETDATE()

WHERE ProductID=865


SET DiscontinuedDate=GETDATE()

WHERE ProductID=980

Kirk Liemohn

of 33MSDN Blogs




6 Aug 2010 6:42 AM

One other note. Last night I was having problems seeing the deletes show up in the incremental crawdon't see it (maybe it is in moderation limbo). Regardless, I was wrong. You can see deletes just fine.expecting to see the count go up there for every deleted item. It does not (makes sense if you think a Chalk that up to working too late at night (and having my browser too small to see that column).

Kirk Liemohn

18 Aug 2010 2:52 PM

For your changelog-based incremental crawls you have T-SQL that sets @LastRunDate after performimissed that occurred during the change query? Would a better approach be to store @LastRunDate i@LastRunDate=CURRENT_TIMESTAMP, then run the query? Of course you'd then have to have an1900-1-1.

Kirk Liemohn

19 Aug 2010 9:00 PM

For what it's worth, I tried the following for my changelog-based incremental crawls to hopefully remov

DECLARE @NewRunDate datetime

if @LastRunDate != CAST('1900-1-1' as datetime)

BEGIN

SET @NewRunDate=CURRENT_TIMESTAMP

SELECT [ProductID] FROM [Production].[vProductAndDescription] WHERE [ModifiedDate] > @La

SET @LastRunDate=@NewRunDate

END

else

SET @LastRunDate=CURRENT_TIMESTAMP

macngif

8 Dec 2010 6:32 PM

Great post.

Successfully set up AdventureWorks example & got it going fine.

Then moved onto to my own ECTs & BDCs – again all fine – however when I implement a crawl using“AttachementAccessor” property) my crawls error with the following

Error while crawling LOB contents. ( Invalid object name 'dbo.myECTtable'. )

Error is occurring when the crawl attempts to execute the select query for the association - i.e. the sel“method”.

The major difference between my case and the AdventureWorks case is that my parent and child EC(hence separate “external systems”).

of 33MSDN Blogs




1 2

In both cases BDC is using a simple SQL query to grab the data.

Tried various different configurations in an attempt to resolve – however no luck as yet.

I am starting to wonder if it is even possible to configure an association that spans across external sys

nir oren24 Jan 2011 3:54 PM

This is a great post - 10X for that

One thinkg, what about LOB permissions. Let's say the LOB system managed it's own permissions ho

For example let's say that teh products in you example are not available for everyone and the permiss

of 33MSDN Blogs

searching external data in sharepoint 2010 using business connectivity services

Documents