Make Coveo For Sitecore work on Authenticated Sitecore URL

You have your Sitecore Instance URL protected.  Why? Many organizations do this to protect their under development site to be indexed or even worse seen by search engines and obviously you dont want any one who knows the URL to actually access your not so ready site or worse add a reference link or something.
So, yes, it is very common to have this kind of set up on lower environments while development is in progress.  It could be to share this URL with selected Clients for UAT or content load in some cases.
You have Coveo for Sitecore installed on this Sitecore instance that you protected using IIS authentication.
The Problem – “Coveo For Sitecore will not work – Diagnostics will fail 401, Index rebuilding will not work(401 again!) and obviously your renderings will not load as Indexes are not built yet”
Solution – While scrambling for a good solution and still leave the Sitecore Instance URL protected, we actually did find more than one solution through various unrelated sources and one from Coveo team as well . ๐Ÿ™‚

Hopefully, this will help some one who is still scrambling and unable to find a solution.  Select which way to implement and go with based on your usage of Coveo Indexes.

  1. If your project uses plain old Coveo renderings and not use Coveo Indexes from Sitecore content search – You can choose to add another IIS site on CD server and point that to the same site, use this URL(internally open and no auth) to manage your coveo indexes. 
  2. If your project accesses Coveo indexes from code and content search – #1 wont work and will cause 500(Coveo rest end point to pull index will error due to 401 again). In this case, you have two options 
    • Disable IIS basic auth and use IP whitelisting for security instead as our infrastructure thinks this is a possible
    •  Keep IIS Authentication enabled on all folders of the Sitecore instance except WebsiteCoveorest and Website/Sitecore Modules/Web/Coveo 
Have fun Sitecoring and Coveofying your search. (New word..lol)


New Items not Indexed on Coveo

New Items Not Showing Up on Coveo Indexes

You have set up everything alright.  All Coveo components are green, your search renderings show up fine on UI.  All is set! You are a rock star. 
You notice something really wierd, updates on sitecore content items show up fine on Coveo Indexes, but, new items added are not showing up on Indexes until you manually rebuild them using Indexing manager. Well, this is a problem we can not live with as we can not ask Content authors to actually rebuild indexes everytime they add a new item.  Bad solution!
Have to resolve this for sure, there is no option. 
Below are the check points I made – 
1. Check the logs for errors
2. Ensure on showconfig.aspx, Coveo index sync strategies are set up fine OOTB one’s are perfect in most cases. 
3. Check CES console for any errors
4.  No luck so far, everything looks perfect. 
5. Next resort was CES Admin Tool, I  used localhost:8081 as I was debugging on my local. 
6. Dates looked pretty off and outdated, so, I tried to turn off and turn on Live Monitoring
7. Re-opened CES console and added new item
8. Voila!!! new items now showed up on Master 
9. Published and yep showed up on Live site too. 
Problem Solved!!!!  Back to being relaxed. ๐Ÿ™‚
I posted this on Coveo Questions and Answers too so some one else in my shoes can probably benefit. 

Coveo Facet Slider Across Two Fields

Coveo Facet Slider Across Two Fields

We had a requirement that obviously could not be achieved using OOTB Facet Slider control. 
We were aware of that and did know we need to do some customization.  Just did not realize that there wont be any to none straight forward solution. 
I came up with the below solution though was not heavily confident while re-iterating this to the team, just  felt worth a shot. 
Good news is after a lot gymnastics it did work. 
Goal –  Use Coveo Facet Slider on two fields on the same item.
Challenge – Coveo does not allow number based Multi Value Field on their Index, hidden challenge I can not base Facet Slider on a Multi Value String field either.  So, I am at wall in both directions. 
Solution – Fluke at first and reality the next 
Below are the steps I did
1. Turn off Auto Max/Min Calculation on Coveo facet slider on the layout
              We can have some livable manual entries for Max/Min on layout as they are required if not auto-generated, I will see what I can do here.
2.       Next, grab Max/Min across all Neighborhoodโ€™s under the current metro page from back end
3.       I overwrite the max/min of Coveo controls with this back end loaded Max/Min patching on to their exposed events.
4.       Now, the filtering by default could be based on only one field(our challenge) โ€“โ€œIn our case Maximum Neighborhood Priceโ€
5    I will have to hijack the rest api call that Coveo makes on slider change to inject this expression along with others it does, so I do not interrupt the inner workings of Slider.  Additional Detail: Used Disjunction available from Coveo query builder expressions  
 

Coveo facets error

Coveo Facets Error 

So, you are on September release of coveo components and you do not have tab component on your search view.  You now want to add coveo facet or the view, which was added fine, but the facet though indexed and pointing to correct field, it would either not render fine on view, what would you do ??? 
I tried doing the below…
  • Checked conside and tried to debug error 
    •  Actual Error – ‘Value can not be null or empty ‘ 
  • What value is coveo asserting for ?checked call stack,  nothing important is missing 
  • Checked Presentation Details and ensured that everything is set correctly. 
  • Dropped facet component , no error any more …??
  • Slack conversation with Coveo team , no break through!!!! They asked to check if I am not hard coding ‘Model .ID’ like plain text, but, nope we were doing things the correct way. 

What else ??? Google should have been my first resort, but I like to do basic checks first, seems like I should follow the reverse order. 

Found below link and seemed very much related, so gave it a shot.  That’s it!!!
It started working. Yippeeee!
Hope, this would help some one battling this issue and unaware that it is a known issue on September release. 
Few Screen Shots of Error for reference 

Sitecore Tagging Surprises

Sitecore tagging is immensely powerful out of the box starting Sitecore 7+.  You can tag any items( including bucketed items)
The below links will get you started on the basic features –

https://visionsincode.wordpress.com/2014/11/16/tagging-is-fun-in-sitecore/

https://community.sitecore.net/technical_blogs/b/sitecore_7_development_team/posts/content-tagging-with-sitecore-7

https://doc.sitecore.net/sitecore_experience_platform/content_authoring/managing_items/general/add_a_tag_to_an_item

Now, coming to the surprises…

  1. While working with the basics,  the __Semantics field on my local sitecore instance did not look anything like the articles explain.  After some pondering,  it turns out my ‘raw’ view was checked.  So, for the multi-value search field to behave and allow selection as expected make sure this is unchecked.
  2. After adding some new tags into tag repository(),  when i go back to tagging section of my item, I did not see the newly added tags in the selection list.   This is because tagging depends and reads from indexes.  when i re-built my indexes, I could then see the new one’s
  3. Since, tagging is so dependent on indexing,  selecting a good indexing strategy is very important.  Doing this operation only as often as needed and not overdoing goes long way.
To tackle #3,  it would be handy to know the default indexing strategies on master and web databases.  For master, the default is syncMaster and for web database, the default strategy is onPublishEndAsync.   You can apply different one’s as needed, but, in our case the defaults would just do it. 
You can read more on indexing strategies available through Sitecore OOTB on the below article, Good reads. ๐Ÿ™‚

https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/search_and_indexing/indexing/index_update_strategies

https://community.sitecore.net/technical_blogs/b/sitecorejohn_blog/posts/sitecore-7-index-update-strategies

https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/search_and_indexing/indexing/rebuild_search_indexes

Sitecore URL Duplication

Sitecore is amazing when it comes to how it can resolve a URL seamlessly to an item and how we can tune that based on requirements and project demands.
In most of the blogs out there – Good reads
https://jammykam.wordpress.com/2015/07/13/seo-friendly-urls-in-sitecore-prevention-is-better-than-cure/  – My Fav
http://reinoudvandalen.nl/blog/using-replacement-characters-in-sitecore-the-right-way/
https://www.cmsbestpractices.com/add-seo-value-by-replacing-spaces-with-dashes-in-sitecore/

you will see how you can tackle LinkManager using  to ensure you always have a good SEO friendly URL using <encodeNameReplacements> and how to avoid the side effects of this by ensuring we do not allow hyphen in item names using “InvalidItemNameChars”

Now, there are other problems we will have to deal with.  Google does not like duplicate content, which means when two URL’s yield same result, in our case render same item content.  It means it is duplication.

But, though spaces are replaced by hyphens internally, it does not do anything when user or say some refferal link some where actually has spaces
For example – www.domain.com/test%20page and www.domain.com/test%20page would yield same page/content

Not Good…Google would not like it!

So, good SEO options are either do a 404 on space version(%20) above or do a 301 re-direct.  Canononical links might help a little if you cant do both of the above.

You could also you IIS re-write rules to replace spaces with ‘-‘ instead of %20

Wait for more…

  

Item Publish and Save Random Errors(Coveo)

Sitecore Item Publish and Save Errors

One day, though I did not do any specific config level changes, I started getting lot of errors, items were not saving, publish was not happening.  Same object reference errors on every single major operation on my sitecore instance.
I looked at the logs and see bunch of Coveo related errors, Yes, I had Coveo installed on this instance.  I ran the diagnostic tool of Coveo and see bunch of services having issues.  Even the indexing was broken.  It seems Coveo patches on to lot of Item pipelines like save, publish, etc.,
To debug this, there are few common steps you need to do and for most cases below should work.
In my case, I was using Windows Credentials for Coveo services and I had changed my password. ๐Ÿ˜‰
But, it could be other service related issues that could be causing it –  So, the traditional magic steps are –

1. Ensure Coveo Diagnostic is all green
2. Go to Windows Services and ensure all Coveo services are up and running.
  Note – Some times the services show up as running, but, infact they would be broken.  When issues persist, try to stop and restart service after ensuring credentials are correct.
3. Tip – wait for few seconds before starting the service after you stop it
4. Re-index all Coveo Indexes
5. Check Diagnostics again to ensure

These steps should help most of issues specifically when everything is installed correctly initially.
As always Coveo first time install is always rough on newbies.  But, slowly you will get good hang of it, once you know the components.

Good Reads –
https://developers.coveo.com/display/public/SitecoreV3/Troubleshooting+Problems+Using+the+Coveo+Diagnostic+Page;jsessionid=2E0BEDF7A39D390CB791743AAE57FF87

https://developers.coveo.com/display/public/SitecoreV3/Resolving+Coveo+for+Sitecore%27s+Most+Common+Issues

Multi line Field With Embed code Experience Editor Issues

Media Media Media!!!!

Every website has media like videos for example where the brand and the company want to showcase and put their best foot forward. 
To Enhance content author experience, we thought it was a good idea not to give them the whole RTE, but, instead to avoid any more thinking than needed and to avoid experimentation, we though multi line text field was best option.  So, the content author can grab the embed code from where ever and toss it on the field we would suggest to them. 
All was good, jumped on with implementation, only to see that this field with embed code behaves totally wierd on page editor/experience editor mode sitecore 8.1 /Update 2. 
I researched else where saw nothing about this and decided to log a sitecore support ticket, it  turns out that starting sitecore 7.2, Experience editor trims any tags on multi line text field for security purposes. Bummer!! Sitecore advised us to to RTE instead, but, due to third party we were using even that was not a possibility as the embed code had video tags and script tags.  
We moved towards using basic fields as parameters that we would need like Video ID, Account ID, etc and ended up constructing the html on partial view.  So far so good. 
More reference details on why multi line field would not work on latest editions of sitecore from support – 
Sitecore encodes all tags in the single-line/multi-line text fields when you edit it in Experience Editor application and it’s an expected behavior.
This behavior was implemented since Sitecore CMS 6.6:
-Page Editor
 In 6.6 Update-3, the <renderField> pipeline was modified to HTML encode the field value when rendering single-line/ multi-line text fields (ref. no. 327905). This did not work correctly in the Page Editor which displayed the encoded value. And if the user saved the page, the already encoded value would be HTML  encoded again. (384997)

TDS – Set up Deploy Steps

TDS Useful Feature 

For those who always would need to publish after deploy using your favorite tool out there, TFS/Team City etc.,  There is a out of box feature that TDS offers to publish all items included in the project.  This means you dont have to login and publish post deployment which is usually the case. 
I tested this out and it actually works pretty awesome. 
Unfortunately, there is lack of clear instructions on TDS site for this option, so I thought it is worth a blog post to provide exact steps you need to do to make this work. 
1. Go to Deploy tab of your TDS project, You should see the below options 
2. Check box as shown on the screenshot above and the most important one is step 3 
3. Ensure you have entered the string(name) of the publishing target you wish to push to post deployment.  It can have more than one as you might have realized. (Just comma separate them)
It is important to know what is the publishing target, it is not the database name as some of you might think, it is actually the name of the item(publishing target) that points to the database you wish to publish to. 
Default publishing target that comes out of the box from sitecore is called “Internet”.  You can add your own or find existing one through navigating to 
Yoursitedomain.com/sitecore/admin/showconfig.aspx and searching for target or simply go to the below section 
System -> Publishing targets -> Screenshot below 
Helpful read – 
http://hedgehogdevelopment.github.io/tds/chapter4.html

Coveo – Windows 10 CES install fatal error

CES Install failure on Windows 10

I am not sure if any one encountered this before, Coveo pre-requisites on latest June 2016 version of CES failed on my local(this used to work fine before), The only change I realized this time was I am using Windows 10.  I had .netframework 3.5 enabled already, but, Coveo installer was trying to install again causing fatal error.  I could not move forward with installation.. Upon research I found the below issue on Coveo Q&A site.  I followed it and it worked.  It’s kind of not fair that I need to do regedit to make CES install fine. 
I added my concern up on the below Coveo question and answers as well.  Let’s see if Coveo does something for us not to have to edit the windows registry to make their product work.

More details on the link below –

https://answers.coveo.com/questions/5344/ces-fails-to-install-net-35-sp1-when-its-already-installed

Screenshot for reference on what i did to make this work.
Note – Make sure you create a system restore point and back up your original key before modification, just in case if it causes issues you can revert your changes

Tapping on to Duplicate Item pipeline – Sitecore

Tapping Duplicate Item Pipeline


One of the latest requirements and challenging request was to tap on to duplicate item pipeline on latest version of Sitecore.  As of this writing, Sitecore 8.1, update 3.
Every time I have a problem on my hand, to come up with a solution, I clearly imprint on my brain what my goal is, this actually helps find different ways to achieve what you want the system on hand to do.
My goal was to tap on to duplicateitem pipeline somehow to do few renaming and change of few references post duplication of a specific item on the content tree.  The key here is specific item, I dont want this custom code to run if the duplication is done on any other item, so, it is important to know where and how I can patch this to successfully instruct sitecore. “Hey! If content author is duplicating this ‘xyz’ item only, perform a set of operations”
I ran across couple blogs to start understanding what happens behind the scenes.  Not much of help as either these are outdated and probably not close to what I was looking for….
So, time to write it up, so any one who is looking for some direction on how to tap to duplicate pipeline.

Trials and Failure


I went ahead and looked at my “showconfig.aspx”, found the below related config section under processors and this felt like perfect place I could patch on to –

<uiDuplicateItem>
<processor mode=”on type=”Sitecore.Shell.Framework.Pipelines.DuplicateItem,Sitecore.Kernel method=”CheckPermissions/>
<processor mode=”on type=”Sitecore.Shell.Framework.Pipelines.DuplicateItem,Sitecore.Kernel method=”GetName/>
<processor mode=”on type=”Sitecore.Buckets.Pipelines.UI.ItemDuplicate, Sitecore.Buckets method=”Execute patch:source=”Sitecore.Buckets.config/>
<processor mode=”on type=”Sitecore.Shell.Framework.Pipelines.DuplicateItem,Sitecore.Kernel method=”Execute/>
</uiDuplicateItem>

I tried patching to most apt looking processor which is “Sitecore.Shell.Framework.Pipelines.DuplicateItem”, did not work. ๐Ÿ™

It is because of introduction of buckets which I came across while reading related blogs out there. Great read if you have time.

http://jockstothecore.com/a-tale-of-three-friends-close-look-at-three-sitecore-pipelines/

Based on this article, I patched on to before the buckets related pipeline for duplicate 
that is ‘Sitecore.Buckets.Pipelines.UI.ItemDuplicate’

The challenge here was the client args in the info object had array of fields under the item that is to be duplicated which I had no way to overwrite. So, it does not achieve my goal, scrap that. 

Then, I thought I will tap SaveItem pipeline, but, I will loose the context of which item is being duplicated and either way’s duplicateitem does not trigger save item pipeline, so that would not work. 

I am hitting walls in every direction, but, I need to keep looking and trying…

Voila Moment!!!!!

I was curious what does Sitecore.Buckets Item Duplicate pipeline does with Client Args as that route made most sense.  It seems only thing the code uses from Client Args was the “Name” for the new item. 
While exploring the code on Kernel dll using reflector, I see that it directly calls Context.Workflow.DuplicateItem(item, args[“name”)).  This is getting interesting, seems like I see light at the end of the tunnel. 

Explored more deeper, chain of methods called from this point one ->

Item.Duplicate(copyName)
this.CopyTo(parent,copyName)
this.CopyTo(more params)
ItemManager.CopyItem(all params)
Provider.CopyItem(params related)

On derived type of copyItem of Provider, I see something that caught my eye. 
It calls GetDataEngine and proceeding with CopyItem on DataEngine.  Dug more deeper..
It calls CopyItemCommand’s Execute, Voila!!! I found my answer. Panicking with excitement, I tried extending CopyItemCommand’s prototype using the below kind of config edits 


<database id=”master” singleInstance=”true” type=”Sitecore.Data.Database, Sitecore.Kernel”>
        <Engines.DataEngine.Commands.CopyItemPrototype>       
          <obj type=”Yournamespace.ClassName, DLL” />
        </Engines.DataEngine.Commands.CopyItemPrototype>
      </database>

I followed similar syntax from Sitecore.Buckets.Config to make this work.  There is a catch, always make sure you are overriding CreateInstance method and returning your new class in that method, otherwise it will not consider your class as valid prototype and will take the default one. 
Now extend the class “Sitecore.Data.Engines.DataCommands.CopyItemCommand”

It worked!!!!

I could get the source item, destination item(parent), copy ID(new id to be created) and I can extend the Executed method now to add my own business logic.  

Make sure you you call base.Executed() to ensure the base functionality is still in place. 

Good Reads/Helpful Related articles
https://reasoncodeexample.com/2013/01/13/changing-sitecore-item-references-when-creating-copying-duplicating-and-cloning/

http://jockstothecore.com/a-tale-of-three-friends-close-look-at-three-sitecore-pipelines/

http://www.sitecore.net/learn/blogs/technical-blogs/john-west-sitecore-blog/posts/2010/11/intercepting-item-updates-with-sitecore.aspx

Another reason to get on latest update of VS 2015

Nuget Extension Issue – VS 2015 Update 1

Well, I think it is about time VS pushes notifications to it’s users to update to latest update available and recommended. ๐Ÿ™‚

I was having issues downloading Nuget’s dependent on my project template on VS 2015 Update 1. After my co-worker mentioned he is using a different update version of VS 2015 he is on.  I wanted to give it a try.  Updated to update 3 and boom it started working.

https://www.visualstudio.com/en-us/news/releasenotes/vs2015-update3-vs

May be it will help someone who is using old update of VS 2015 and having the below issue –

Error was -“Object reference not found”.

So, basically the project template was trying to download Nugets and each Nuget had multiple dependencies which VS update 1 was not able to deal well.

Other option is to download latest version of Nuget Package Extension, some told me even that worked.

Hope, this helps..

Purpose of Area field on Sitecore 8.1?

So, It all started with an un-answered question on sitecore community on why the ‘Area’ field does not seem to work on View Rendering.
https://community.sitecore.net/developers/f/8/t/3509

Field In Question

Human tendency is if we are taking pain to enter a value for a field on sitecore, it does do something dynamically.  The first thought is, if I fill in the Area field on Sitecore Rendering Item, It should auto pick the path to my *.cshtml by constructing it rather than depending on us to fill in full path of the view in the Path field.

But, if I tried that it caused error on my end that it can not find the view.
I started reading around to see what is the purpose of this Area field? May be the purpose is bigger/better? I mean sitecore added this in the latest release.

Good Reads Below –
http://www.nonlinearcreations.com/Digital/how-we-think/articles/2015/10/Sitecore-8-1-MVC-Improvements-Area-Support-and-MVC-5-2-3.aspx
https://ctor.io/new-in-sitecore-8-1-mvc-areas/
https://citizensitecore.com/2015/10/26/mvc-areas-in-sitecore-8-1-a-step-by-step-guide/

Based on the blogs, this new token or field value could be used for view resolutions inside your controller or view renderings, but, sitecore would still need full path in the path field.

So, the gist is you read this field as you read any other field on the code.  Only note is if you need to use this field value on code use it by name and not GUID(As it is unique obviously for each rendering).

Something like..Sitecore.Context.Site.Properties[“area”];

If you have your own Area Resolving strategy, which by the way you need to plugin properly for it to work, then in your class use the Area value when needed.

Got it! Some purposes that could pay off on adding this new field.  I am still curious to know the ground breaking reason if any that Sitecore added this field.

Anyone???

Sitecore 8.1 Update 3 – Bug on Scheduling Tasks

For my upcoming project, one of the requirement was to run a scheduled job every 24 hours.  Fair enough I hardly remember how to accomplish this as on my previous projects I was using console application with windows scheduler to do the needful.  This time around, I wanted to explore what sitecore offers out of the box for this demands.  It was cool, very simple and easy to do so.

Understanding – My first step 
I explored couple of blogs to gain understanding on how scheduled tasks and commands work on Sitecore.  Please see the references that helped me a lot.

https://sitecorebasics.wordpress.com/2014/08/09/scheduled-task-basics/
https://briancaos.wordpress.com/2011/06/28/run-sitecore-scheduled-task-at-the-same-time-every-day/
https://www.degdigital.com/insights/how-to-create-sitecore-scheduled-task/
For even more deeper understanding and if you have time refer to below as well –
http://www.sitecore.net/Learn/Blogs/Technical-Blogs/John-West-Sitecore-Blog/Posts/2010/11/All-About-Sitecore-Scheduling-Agents-and-Tasks.aspx

Debugging – Crucial One
For triggering the jobs manually, this is for debugging purpose and for us devs with out this, we can not successfully test the code that runs on command trigger.  There are multiple options to do this, I found the below module available on market place, it was very easy to install and fairly simple UI to deal with, I tried others, but, I could not make it to work or documentation was missing some steps.  I recommend below one –
http://sitecoreblog.blogspot.in/2011/07/new-module-on-trac.html
Note – After you install, many newbies who have not worked on Toolbox might not know where this UI get’s installed, it is in “All Programs -> ToolBox -> ScheduledTasksUtil .  Click and view all jobs on your instance and trigger any one you wish to debug manually.  Cool! Simple! Isn’t it.

Actual Bug  – 

I logged a support ticket for this, I will update this blog post based on sitecore team comments. Though the bug is not huge and has a workaround that we could do for new projects, for teams who might have this issue and if that goes un-recognized could be great deal of problems especially when syncing super important data feed or something.

Here goes the issue –

When I create a scheduled task, there is an option to set the schedule, for example if I need to set a command to run every 24 hours based on documentation by John West, we need to do below as format suggests HH:mm:ss
But, if I have 24 set in the hours, sitecore is reading that as 24 days instead of 24 hours, for any number lesser than 24, it reads correctly as hours.
It does not seem to be an intuitive behavior and looks like a bug while converting hours to days.

On our end, we can proceed by setting this number to less than 24 like 23 as that works fine, but, I wanted to report this regardless as it could be broken on sitecore instances where they had set up a schedule for every 24 hours as this will be running on their end for every 24 days instead of hours.
See some screenshots below.
Example Schedule that has unexpected assignment on schedule –




Loaded up Schedule Object Interval showing 24 days instead of hours. 

Work Around (Update from Sitecore team)

It seems this is how .Net above 3.5 deals with it on Time parse.  See more details below.
https://connect.microsoft.com/VisualStudio/feedback/details/764618/timespan-parse-24-00-00-returns-24-00-00-00-instead-of-throwing-an-overflowexception

On Sitecore side, the format that has to be used on latest versions of .net frame work would be to use below format for 24 hours.
Example – 20160714|21000714|127|1.00:00:00